What is model distillation?

Definition: Model distillation trains a compact student model to reproduce the behavior of a larger teacher model, keeping much of its quality on a target task while cutting inference cost and latency.

How distillation works

The teacher model generates outputs, or probability distributions, for a large set of task examples. The student is trained on those outputs instead of, or alongside, human labels. For language models this often means generating many high-quality responses from a frontier model and fine-tuning a more compact model on them.

When it helps

Distillation fits high-volume, narrow tasks such as classification, extraction or routing, where paying frontier-model prices for every request is hard to justify. It is less suited to open-ended work that needs broad knowledge.

Watch-outs

Check the teacher provider's terms of service, since some restrict using outputs to train competing models. The student inherits the teacher's mistakes, so teams evaluate it on held-out data and monitor it for model drift after launch.

How Ryz helps

Ryz fine-tuning engineers build distillation and evaluation pipelines in your cloud, working on your team.

Related

• LLM fine-tuning services
• Hire fine-tuning engineers
• LLM fine-tuning
• Model quantization