CPU · batch 1 · one thread · median of 300 passes · PyTorch 2.11.0+cpu · onednn
Contents
The original piece measured dynamic int8 quantisation and found it slower than float32. The reply, every time, was the same: dynamic is the weakest of the three options, use static quantisation or train with quantisation in the loop and it will be faster.
Fair objection. So we measured those too.
The objection, stated properly
There are three ways to get an int8 model in PyTorch and they differ in when the activation ranges are decided.
Dynamic quantises the weights ahead of time and works out the activation range on every single call. That per call work is real, and it is the obvious suspect for the original result.
Static observes the activation ranges once, on calibration data, and bakes them in. Nothing is computed per call. If the overhead was the range finding, static should win.
Quantisation aware training simulates the rounding during training, so the weights learn to survive it. It is the most expensive to produce and usually the most accurate.
What we measured
| accuracy | size KB | median ms | p90 ms | against float32 | |
|---|---|---|---|---|---|
| float32 | 0.9791 | 3223.5 | 0.0914 | 0.1103 | leader |
| dynamic int8 | 0.9788 | 812.7 | 0.2294 | 0.2687 | 2.5x slower |
| QAT int8 | 0.9793 | 830.7 | 0.2367 | 0.2790 | 2.6x slower |
| static int8 | 0.9786 | 830.7 | 0.2397 | 0.4150 | 2.6x slower |
| Every int8 path lands within 0.011 ms of the others. The choice between them did not matter; the choice to quantise at all did. | |||||
Static quantisation was not faster than dynamic. On this machine it was marginally slower, and the gap between the three int8 arms (0.0103 ms) is smaller than the run to run variation on any one of them.
All three int8 paths are roughly two and a half times slower than float32 on this CPU. The objection was reasonable and it turned out not to be the explanation.
What QAT actually bought
| accuracy | against float32 | |
|---|---|---|
| QAT int8 | 0.9793 | +0.0002 |
| float32 | 0.9791 | leader |
| dynamic int8 | 0.9788 | -0.0003 |
| static int8 | 0.9786 | -0.0005 |
| A spread of 0.0007 across four arms, on 10,000 samples. That is seven digits out of ten thousand and it is noise, not a ranking. | ||
Almost nothing, on this problem. QAT came top by 0.0002, which on 10,000 samples is two digits.
That is worth stating plainly because the first version of this experiment got it wrong in a way worth admitting. QAT was trained for five epochs and float32 for three, and QAT duly looked more accurate. It was not more accurate; it had two more epochs. Every arm in the table above trains for five. An unfair control is worse than no control, and it is an easy one to ship without noticing.
Why none of it was faster
The honest answer is that the win from int8 arithmetic has to be larger than the cost of moving between representations, and at this size it is not.
Every quantised layer converts its input to int8, does the matrix multiply, then converts back out. At batch 1 the multiply is small, so the conversion is a meaningful share of the work rather than a rounding error. int8 arithmetic is faster per operation and there are not enough operations here for that to pay.
That also explains why static did not help. Static removes the range calculation, and the range calculation was never the expensive part. The conversions are, and static does those too.
The file did shrink: 74 per cent smaller, from 3224 KB to 831 KB. If the constraint is memory or download size, quantisation delivers. If the constraint is latency at batch 1, on this hardware, it does not.
Where this result does not apply
Four limits, and they are load bearing.
Batch size 1. This is the latency sensitive setting. At larger batches the multiply grows and the conversion overhead is amortised. int8 generally does win there.
This model is small. 824,550 parameters, three linear layers. On a model where the weights do not fit comfortably in cache the picture changes, because int8 weights mean a quarter of the memory traffic.
One backend. This build of PyTorch exposes only
onednn. Results with fbgemm or qnnpack may differ, and anyone
seeing a different number on a different backend is not contradicting this
one.
Linear layers only. Quantised convolution takes different kernels. Nothing here says anything about a convolutional network.
The point is not that quantisation is useless. It is that "quantise it to make it faster" is a hypothesis about your model on your hardware at your batch size, and it takes about an hour to test.
Run it yourself
Same machine, same process, same seed, five epochs per arm, 300 timed passes after 50 warmup passes, threads pinned to one so the measurement is of the kernel rather than of the scheduler. Two independent runs agreed on the ordering and on the size of the gap.
If you are measuring this kind of thing on a system that has to go to production, the same discipline applies to agents and language models: fix the cases, fix the thresholds, and diff the runs. We wrote an open harness for that at r4agent.