Follow up, 14 August 2026. The most common reply to this piece was that static quantisation or QAT would have been faster. We measured both on one machine: neither was.
CPU · batch 1 · median of 200 passes · nn.Linear only
What We'll Cover
Introduction
Every guide to shipping a smaller model lists the same four techniques: quantise it, prune it, distil it, export it. They are usually presented as a menu of wins, with the implication that you pick the ones you have time for.
We ran all four against the same model on the same machine and timed them. One of them made inference slower. One of them changed nothing we could bank. One of them lost accuracy for no gain at all. The fourth was worth more than the other three put together.
The point of writing this down is not that these techniques do not work. It is that which one works depends on hardware and runtime you have to measure, and the ordering everyone repeats is not the ordering we got.
The model we started with
A small convolutional network on MNIST. Nothing exotic, because the point is the arithmetic of making it cheaper, not the architecture.
| accuracy | size KB | latency ms | |
|---|---|---|---|
| the big model | 0.9775 | 3223.5 | 0.310 |
| 824458 parameters | |||
Three numbers to beat: accuracy of 0.9775, 3.2 MB on disk, and 0.310 ms per inference. Everything below is measured against these.
Quantisation made inference slower
Post-training dynamic quantisation of the linear layers to int8. The size result is exactly what the literature promises: 3223.5 KB down to 865.2 KB, a 73 percent reduction, for five ten-thousandths of accuracy.
| Pos | accuracy | size KB | latency ms | Behind leader | |
|---|---|---|---|---|---|
| 1 | float32 | 0.9775 | 3223.5 | 0.259 | leader |
| 2 | int8 linear layers | 0.9770 | 865.2 | 0.559 | +0.3000 |
| int8 also shrank the file 73 percent. That is real, it is just not speed. | |||||
The latency went the wrong way. 0.259 ms became 0.559 ms: not a small regression, but 2.2 times slower.
This is not a mystery once you look at what dynamic quantisation actually does. It stores weights as int8 and then, at every forward pass, works out the activation scale, quantises, does the integer matrix multiply, and dequantises the result. On hardware with int8 kernels that pipeline is a win. On a general-purpose CPU without them, you have added three steps around a matrix multiply that was already fast.
Quantisation is a memory technique that is often also a speed technique. Which one you get is a property of your hardware, and the only way to find out is to time it on the machine that will run it.
Pruning to 90 percent sparsity changed nothing we could bank
Magnitude pruning, removing the smallest weights and measuring accuracy as the sparsity climbs. The accuracy result is genuinely striking.
| sparsity | accuracy | zeros |
|---|---|---|
| 0.00 | 0.9775 | 0.0% |
| 0.50 | 0.9770 | 50.0% |
| 0.80 | 0.9790 | 80.0% |
| 0.90 | 0.9780 | 90.0% |
| 0.95 | 0.9290 | 95.0% |
| Zeros do not shrink a dense tensor. Size and latency were unchanged at every level. | ||
Nine out of ten weights set to zero and accuracy went up, from 0.9775 to 0.9780. The network was carrying an enormous amount of nothing. That is a real and interesting finding about how overparameterised these models are.
It is also, on its own, worth nothing operationally. A pruned tensor full of zeros is the same size on disk and takes the same time to multiply as a dense one, because a dense kernel does not know or care that most of the numbers are zero. Converting sparsity into speed or size needs a sparse storage format and a runtime with sparse kernels, and if you do not have both then you have measured something interesting and shipped nothing.
Sparsity is not compression until something downstream exploits it. Check that your serving runtime has sparse kernels before you spend a week on pruning.
Distillation lost, at every alpha, on both seeds
The idea is appealing: a small student trained on the teacher's softened probabilities learns more than one trained on hard labels, because the probabilities carry information about which classes look alike.
That extra structure is real. The teacher genuinely does say a seven looks somewhat like a two and a three. It did not help.
| Pos | accuracy | size KB | latency ms | Behind leader | |
|---|---|---|---|---|---|
| 1 | teacher | 0.9775 | 3223.5 | 0.271 | leader |
| 2 | student, labels only | 0.8905 | 101.6 | 0.028 | -0.0870 |
| 3 | student, distilled | 0.8730 | 101.6 | 0.031 | -0.1045 |
| The 32x size reduction came from the smaller architecture, not from the distillation attached to it. | |||||
The distilled student scored below the student trained on labels alone. Before concluding anything from one run, we swept the mixing weight and repeated on a second seed.
| Pos | alpha | seed 1 | seed 2 | Record | Behind leader |
|---|---|---|---|---|---|
| 1 | 0.0 | 0.8905 | 0.8920 | leader | |
| 2 | 0.3 | 0.8835 | 0.8795 | · | -0.0070 |
| 3 | 0.5 | 0.8835 | 0.8800 | · | -0.0070 |
| 4 | 0.7 | 0.8835 | 0.8785 | · | -0.0070 |
| 5 | 0.9 | 0.8820 | 0.8775 | · | -0.0085 |
| Labels only won all ten runs. Alpha 0 is labels only. | |||||
Ten runs. Labels-only won every one of them. The gap is small, roughly one point, and it is consistent in direction across every alpha and both seeds, which is what makes it a result rather than noise.
We are not claiming distillation does not work. It plainly does in the settings where it was developed, on harder problems with more classes and more room between teacher and student. We are claiming that it is not free, that it can lose, and that a paper's result is not a prediction about your problem.
The student was 32 times smaller and 10 times faster than the teacher, whether or not it was distilled. The compression came from choosing a smaller architecture. The distillation added a training pipeline and cost 1.75 points.
Export was the one that worked
Exporting to ONNX and serving with onnxruntime. No change to architecture, no retraining, no accuracy trade-off to argue about.
4.4 times faster, with predictions identical to within four millionths. The file is the same size, so this is not compression; it is that a graph compiled ahead of time and run by a dedicated inference runtime avoids the per-operation Python overhead that eager execution pays on every call.
This is the least discussed of the four techniques and it was the only one that made the model cheaper to run without costing anything.
What we would actually do
In the order the measurements suggest, rather than the order the guides list:
Export first. It cost nothing, changed no predictions, and was worth 4.4 times. If you do one thing, do this.
Choose a smaller architecture second. The student was 32 times smaller and 10 times faster than the teacher. That came from the architecture, not from the clever training method attached to it.
Quantise if you are memory-bound, and time it. A 73 percent size reduction is a genuine win when size is the constraint. Just do not assume it also buys speed until you have measured it on the target hardware.
Prune only if your runtime has sparse kernels. Otherwise you have a fascinating result about overparameterisation and an unchanged deployment.
Every one of these techniques works somewhere. None of them works everywhere, and the only thing that tells you which case you are in is a timer on the hardware you will actually deploy to.
Where these numbers come from. They come from week 16 of the Deep Learning programme, which spends a week making one model cheap enough to ship and measures each technique rather than recommending it. Every figure on this page was printed by code that was run, not written from what the result should have been. The course is free to read and the workbooks are free to download.