Distillation: teaching a small model to do a big model's job
Distillation trains a small student model to imitate a large teacher model. In LLM applications, it usually means: run the large model on many real inputs for your task, keep the outputs that pass your checks, and fine-tune a small model on those input and output pairs. It works well for narrow, high-volume tasks like classification, extraction, routing and short structured answers, where a small model can match the large one at a fraction of the cost and latency.
Why distil
A large model handles your ticket classification well, but at a million tickets a month the bill is too high and each call takes a second. You don't need the small model to be good at everything the large model can do. You need it to be good at your one task.
That is the bet distillation makes, and for narrow tasks, it often pays off.
Two kinds of distillation
Classic knowledge distillation
The original idea, from a 2015 paper by Hinton and colleagues, trains the student on the teacher's full probability distribution over answers (soft labels), not only the final answer. Those probabilities carry more information than the label alone, for example that an image of a 7 looks a bit like a 1.
DistilBERT is a well-known example: a smaller version of BERT trained this way, which its authors reported was 40% smaller and 60% faster while keeping 97% of BERT's language understanding performance on their benchmarks.
Distillation from outputs
With API models, you usually only see the generated text, not the probabilities. So in practice, LLM distillation often means supervised fine-tuning of a small model on the large model's outputs. Some providers offer this as a managed feature, where you store the outputs of a large model and fine-tune a smaller one on them.
Steps
- Pick one task with high volume and a clear definition of success.
- Collect real inputs from production, not invented ones, so the student learns your actual distribution of cases.
- Run the teacher model on them with your best prompt.
- Filter the outputs with your evals and, for a sample, with your domain expert. Don't train the student on the teacher's mistakes.
- Fine-tune the student (often with LoRA) on the filtered pairs.
- Evaluate the student against the teacher on a held-out test set.
- Deploy with a fallback: route inputs the student is unsure about to the teacher.
What to watch
- Terms of service. Some providers prohibit using their outputs to develop competing models. Read the terms for the model you use as a teacher.
- Coverage. If rare cases are missing from your inputs, the student won't learn them. Add them on purpose.
- Confidence. Give the student a way to signal uncertainty (a confidence score, an "unknown" class) so you can route those cases to the large model.
- Drift. When your inputs change, the student falls behind faster than a general model, so monitor its pass rate and retrain on fresh data.
Measuring it
The numbers that decide whether it worked:
- Student pass rate as a share of teacher pass rate on the same test set (for example 96% of the teacher's pass rate).
- Cost per 1,000 calls for each.
- Latency at your expected load.
- How much traffic falls back to the teacher.
From the floor
In the LLM training workflow demo, we fine-tuned DistilBERT, itself a distilled version of BERT, to classify whether a piece of code is valid or invalid, because it is light enough to train on a laptop CPU. That made the whole workflow visible without waiting hours: preparing the data as JSON with a code example and a label, training arguments with a batch size of 1 and a single epoch, saving the model, and serving predictions through a small Flask app.