What Is Model Distillation? — Teaching a Small Model to Think Like a Large One

Last week we covered quantization — reducing the precision of a model’s parameters to make it smaller and faster. This week, a related but different technique: model distillation. Both are about making AI models more efficient. But they approach the problem differently.

What is model distillation?

Model distillation is the process of training a smaller, simpler model — called the student — to mimic the behaviour of a larger, more capable model — called the teacher. The student learns not just from the original training data but from the teacher’s outputs — the probabilities and patterns the teacher produces when it processes examples. The result is a smaller model that captures much of the teacher’s capability at a fraction of the size and compute cost.

Why learn from the teacher’s outputs rather than just the data?

When a model produces an output, it does not just give a single answer — it produces a distribution of probabilities across all possible answers. For a classification task, it might be 87% confident the answer is A, 9% confident it is B, and 4% confident it is C. These ‘soft’ probability distributions carry more information than the hard label ‘A’ alone. They reveal how the teacher model thinks — which answers it considered, how confident it was, where it was uncertain. The student learns from this richer signal and develops better generalisation than training on the original data alone.

A simple analogy

Think of a master craftsman training an apprentice. Rather than having the apprentice learn from a textbook, the master demonstrates their technique, explains their reasoning, and shows the apprentice not just what they do but how they think about it. The apprentice learns from the master’s accumulated knowledge and judgement — not just from first principles. The apprentice may never match the master’s breadth of experience, but they develop capability far faster than starting from scratch.

Distillation vs quantization

Quantization takes an existing model and reduces the precision of its parameters. The architecture stays the same; the numbers get smaller. Distillation creates a new, architecturally smaller model and trains it to behave like the larger one. Quantization is faster to apply and works without retraining. Distillation produces a genuinely smaller architecture that can be more efficient for specific tasks. Both are commonly used together.

Where distillation is used

DistilBERT — a distilled version of BERT that is 40% smaller and 60% faster while retaining 97% of BERT’s performance on many tasks. Mobile and edge AI — distilling large models into versions small enough to run on phones and IoT devices. Cost reduction — serving inference from a smaller distilled model is significantly cheaper than serving the full model. Specialisation — distilling a general model into a smaller model specialised for a specific domain or task.

The simple rule

Large model: powerful, expensive. Distilled model: smaller, faster, cheaper, trained to mimic the large one. Most of the capability at a fraction of the cost. Quantization compresses the numbers. Distillation compresses the architecture.

🎁 Try MarineRef free for 1 year — use code FREE2026 at maritime.rangalabs.cloud

👉 Get one AI tip every day on WhatsApp — free. Join here: https://chat.whatsapp.com/DhaGgTuQ9GE67ykGVMgxXb