1257 training images · 8x8 digits · no retraining between the two
What We'll Cover
Introduction
A held-out test set is supposed to be the honest number. You keep it separate, you touch it once, and what it says is what you get.
It is honest about exactly one thing: performance on data drawn the same way as the training data. Move the data slightly and it tells you nothing at all, and it will not warn you that it has stopped applying.
Here is what that looks like with numbers attached.
Six classifiers, one held-out set
Handwritten digits. Every classifier trained on the same 1,257 images and scored on the same held-out set.
| Pos | accuracy | Behind leader | |
|---|---|---|---|
| 1 | support vector machine | 0.9870 | leader |
| 2 | nearest neighbour | 0.9852 | -0.0018 |
| 3 | random forest | 0.9778 | -0.0092 |
| 4 | logistic regression | 0.9704 | -0.0166 |
| 5 | convolutional network | 0.8630 | -0.1240 |
| 6 | guess the commonest digit | 0.1019 | -0.8851 |
Against a 0.1019 baseline, these are strong results. The support vector machine gets 987 of every thousand digits right. On the evidence of this table you would ship it.
Then we moved the images
The same trained models, the same test images, translated by one pixel and then by two. Nothing else changed. No new data, no retraining, no noise added, no adversarial optimisation. The digits are still perfectly legible to a person.
| unchanged | shift 1 | shift 2 | |
|---|---|---|---|
| support vector machine | 0.9870 | 0.5074 | 0.1667 |
| convolutional | 0.8630 | 0.3407 | 0.1370 |
| convolutional, augmented | 0.6944 | 0.3352 | 0.1185 |
| No noise, no adversarial optimisation. The digits remain legible to a person at both shifts. | |||
The support vector machine falls from 0.9870 to 0.5074. One pixel cost it just under half of everything it appeared to know. At two pixels it is at 0.1667, which against a 0.1019 baseline is barely distinguishable from guessing.
The model never learned what a digit looks like. It learned which pixel positions tend to be dark in this particular dataset, which is a completely different thing that happens to produce the same test score.
Augmentation did not save it either
The obvious fix is to train on shifted copies. We did, and the third row is the result.
The augmented convolutional network scores 0.6944 on unchanged images, well below the 0.8630 it managed without augmentation. It paid 17 points on the clean data. In exchange, at shift 1 it scores 0.3352 against the unaugmented 0.3407, which is to say it bought nothing.
Augmentation is a real technique and it does work, but it is not a switch you flip. Done in this quantity on a dataset this small, it made the model worse at the task and no better at the failure it was meant to fix.
Why the test set could not warn you
The held-out images were drawn from the same collection as the training images, digitised the same way, centred the same way. Held-out is not the same as different. It answers "how does this do on more data exactly like the data I have", and that is a narrower question than anybody hears when the number comes out at 0.99.
Anything your collection process holds constant is invisible to the test set, because it is constant in the test set too.
The same failure with a different disguise
A related result from the text side of the material. A ticket classifier scored well on held-out data until we checked how many test rows also appeared in training.
| seen | unseen | overall | |
|---|---|---|---|
| 1-nearest neighbour | 0.9474 | 0.6580 | 0.6682 |
| logistic regression | 0.8947 | 0.9449 | 0.9432 |
| 38 of 1073 test rows also appeared in training. | |||
Thirty-eight duplicate rows out of 1,073, about three and a half percent. Nearest neighbour scores 0.9474 on the rows it had already seen and 0.6580 on the rows it had not. Its overall number is a blend of memory and ability, and it is much closer to the memory than the gap in the table suggests.
Logistic regression, notably, does slightly better on the unseen rows than the seen ones, which is what a model that generalises rather than memorises looks like. The comparison is only visible because somebody split the score by whether the row was a duplicate.
What to measure instead
Perturb the test set. Shift, rotate, crop, rescale, recompress. Anything your collection process held constant that the real world will not. If the score falls off a cliff, you have learned that before your users did.
Check for duplicates across the split. It took one line to find 38 of them, and they were quietly inflating one model's score by tens of points on the rows they touched.
Score by segment, not just overall. An average is the number most likely to hide the thing you need to know.
Report the run-to-run spread. In the deep learning capstone the seed-to-seed variation was 0.0147, which turned out wider than five of the six improvements being proposed. Five of those six ideas were indistinguishable from changing the random seed.
A single held-out number is the beginning of an evaluation, not the end of one. The useful question is never "how well does it score", it is "what would have to change for this number to stop being true".
Where these numbers come from. They come from weeks 9 and 13 of the Artificial Intelligence and Machine Learning programmes. Every figure on this page was printed by code that was run, not written from what the result should have been. The course is free to read and the workbooks are free to download.