Classifying Images

Week 9 of 14 · Vision · 7 days

Full curriculum
Week 09 · Vision

Classifying Images

Week 09 · Day 1 of 7

Baselines First, Again

Nearest neighbour, and how strong a method that does no training is

By 293 words

Week 8 turned images into numbers and found that raw pixels beat hand-designed edge features on this data. This week compares the actual classifiers, starting as always with what a system that does nothing clever would score.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
from sklearn.dummy import DummyClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression

print('%-34s %10s' % ('', 'accuracy'))
for name, model in [('guess the commonest digit',
DummyClassifier(strategy='most_frequent')),
('guess at random',
DummyClassifier(strategy='uniform',
random_state=0)),
('nearest neighbour',
KNeighborsClassifier(n_neighbors=3)),
('logistic regression',
LogisticRegression(max_iter=5000))]:
model.fit(flat[tr], y[tr])
print('%-34s %10.4f' % (name, model.score(flat[te], y[te])))
accuracy
guess the commonest digit 0.1019
guess at random 0.1111
nearest neighbour 0.9852
logistic regression 0.9704

Nearest neighbour is worth pausing on. It does no training at all: it stores every image and, when asked, finds the three most similar and takes a vote. On clean centred digits that is extremely strong, which is a useful corrective to the idea that a problem needs a sophisticated model.

Nearest neighbour does not scale, and it does not generalise

It keeps the entire training set, so prediction cost grows with the data, and it needs the new image to be genuinely similar pixel by pixel to a stored one. Shift everything by a pixel, as week 8 did, and it degrades badly. It is an excellent baseline and rarely a good production answer.

Week 09 · Day 2 of 7

Kernels Learned From Data

The model that works out its own filters, and what they look like

By 958 words

Week 8 promised a model that works out its own kernels rather than using the ones a person chose. Here it is, and it is a short piece of code.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
def train_cnn(epochs=30, seed=0, augment=False, trace_every=0):
torch.manual_seed(seed)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
nn.Flatten(), nn.Linear(16 * 4 * 4, 10))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
lf = nn.CrossEntropyLoss()
xb = torch.tensor(images[tr], dtype=torch.float32).unsqueeze(1)
yb = torch.tensor(y[tr])
for ep in range(1, epochs + 1):
model.train()
batch = xb
if augment:
# roll each image by a random small shift
shifts = torch.randint(-1, 2, (2,))
batch = torch.roll(xb, (int(shifts[0]), int(shifts[1])),
dims=(2, 3))
opt.zero_grad()
loss = lf(model(batch), yb)
loss.backward()
opt.step()
if trace_every and ep % trace_every == 0:
print('%6d %12.4f' % (ep, loss.item()))
model.eval()
return model

@torch.no_grad()
def cnn_score(model, imgs, labels):
x = torch.tensor(imgs, dtype=torch.float32).unsqueeze(1)
return float((model(x).argmax(1).numpy() == labels).mean())
model = train_cnn(epochs=1)
params = sum(p.numel() for p in model.parameters())
print(model)
print()
print('%d parameters, of which the first layer holds %d'
% (params, 8 * 1 * 3 * 3 + 8))
Sequential(
(0): Conv2d(1, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): ReLU()
(2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(3): Conv2d(8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(4): ReLU()
(5): Flatten(start_dim=1, end_dim=-1)
(6): Linear(in_features=256, out_features=10, bias=True)
)

3818 parameters, of which the first layer holds 80

The first layer is eight kernels of three by three, exactly the shape of the edge detector written by hand last week. The difference is that these eighty-one numbers start random and are adjusted by training.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
def train_cnn(epochs=30, seed=0, augment=False, trace_every=0):
torch.manual_seed(seed)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
nn.Flatten(), nn.Linear(16 * 4 * 4, 10))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
lf = nn.CrossEntropyLoss()
xb = torch.tensor(images[tr], dtype=torch.float32).unsqueeze(1)
yb = torch.tensor(y[tr])
for ep in range(1, epochs + 1):
model.train()
batch = xb
if augment:
# roll each image by a random small shift
shifts = torch.randint(-1, 2, (2,))
batch = torch.roll(xb, (int(shifts[0]), int(shifts[1])),
dims=(2, 3))
opt.zero_grad()
loss = lf(model(batch), yb)
loss.backward()
opt.step()
if trace_every and ep % trace_every == 0:
print('%6d %12.4f' % (ep, loss.item()))
model.eval()
return model

@torch.no_grad()
def cnn_score(model, imgs, labels):
x = torch.tensor(imgs, dtype=torch.float32).unsqueeze(1)
return float((model(x).argmax(1).numpy() == labels).mean())
print('%6s %12s' % ('epoch', 'loss'))
model = train_cnn(epochs=30, trace_every=5)
print()
print('accuracy on held out images %.4f'
% cnn_score(model, images[te], y[te]))
epoch loss
5 2.2540
10 2.1439
15 1.9455
20 1.6400
25 1.2500
30 0.8662

accuracy on held out images 0.8630

What it learned to look for

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
def train_cnn(epochs=30, seed=0, augment=False, trace_every=0):
torch.manual_seed(seed)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
nn.Flatten(), nn.Linear(16 * 4 * 4, 10))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
lf = nn.CrossEntropyLoss()
xb = torch.tensor(images[tr], dtype=torch.float32).unsqueeze(1)
yb = torch.tensor(y[tr])
for ep in range(1, epochs + 1):
model.train()
batch = xb
if augment:
# roll each image by a random small shift
shifts = torch.randint(-1, 2, (2,))
batch = torch.roll(xb, (int(shifts[0]), int(shifts[1])),
dims=(2, 3))
opt.zero_grad()
loss = lf(model(batch), yb)
loss.backward()
opt.step()
if trace_every and ep % trace_every == 0:
print('%6d %12.4f' % (ep, loss.item()))
model.eval()
return model

@torch.no_grad()
def cnn_score(model, imgs, labels):
x = torch.tensor(imgs, dtype=torch.float32).unsqueeze(1)
return float((model(x).argmax(1).numpy() == labels).mean())
model = train_cnn(epochs=30)
kernels = model[0].weight.detach().numpy()[:, 0]
for i in range(3):
k = kernels[i]
print('kernel %d:' % (i + 1))
for row in k:
print(' ' + ' '.join('%6.2f' % v for v in row))
print()
kernel 1:
0.01 0.17 -0.34
-0.34 -0.19 0.12
-0.09 0.36 0.07

kernel 2:
0.06 -0.17 -0.14
-0.39 -0.30 -0.20
-0.05 0.23 0.30

kernel 3:
-0.31 -0.24 0.20
0.28 0.02 0.33
0.04 0.13 0.39

Nobody chose those numbers. Compare them with week 8's edge detector, which had a negative column, a zero column and a positive column: several of these have a similar structure, arrived at from data alone because that is what is useful for telling digits apart.

Week 09 · Day 3 of 7

Six Classifiers, One Dataset

The fair comparison, and the convolutional network that does not win

By 451 words

A fair comparison of everything so far, on identical data, with the parameter count beside each so the cost is visible.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
def train_cnn(epochs=30, seed=0, augment=False, trace_every=0):
torch.manual_seed(seed)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
nn.Flatten(), nn.Linear(16 * 4 * 4, 10))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
lf = nn.CrossEntropyLoss()
xb = torch.tensor(images[tr], dtype=torch.float32).unsqueeze(1)
yb = torch.tensor(y[tr])
for ep in range(1, epochs + 1):
model.train()
batch = xb
if augment:
# roll each image by a random small shift
shifts = torch.randint(-1, 2, (2,))
batch = torch.roll(xb, (int(shifts[0]), int(shifts[1])),
dims=(2, 3))
opt.zero_grad()
loss = lf(model(batch), yb)
loss.backward()
opt.step()
if trace_every and ep % trace_every == 0:
print('%6d %12.4f' % (ep, loss.item()))
model.eval()
return model

@torch.no_grad()
def cnn_score(model, imgs, labels):
x = torch.tensor(imgs, dtype=torch.float32).unsqueeze(1)
return float((model(x).argmax(1).numpy() == labels).mean())
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.neural_network import MLPClassifier

print('%-34s %10s' % ('', 'accuracy'))
for name, model in [('nearest neighbour',
KNeighborsClassifier(n_neighbors=3)),
('logistic regression',
LogisticRegression(max_iter=5000)),
('random forest',
RandomForestClassifier(n_estimators=200,
random_state=0)),
('support vector machine', SVC()),
('dense neural network',
MLPClassifier(hidden_layer_sizes=(64,),
max_iter=800, random_state=0))]:
model.fit(flat[tr], y[tr])
print('%-34s %10.4f' % (name, model.score(flat[te], y[te])))
cnn = train_cnn(epochs=30)
print('%-34s %10.4f' % ('convolutional network',
cnn_score(cnn, images[te], y[te])))
accuracy
nearest neighbour 0.9852
logistic regression 0.9704
random forest 0.9778
support vector machine 0.9870
dense neural network 0.9778
convolutional network 0.8630

The convolutional network does not win here

On eight by eight centred digits, several much simpler methods match or beat it, and a support vector machine on raw pixels is extremely strong. The advantage of convolution is that it shares each kernel across every position, which matters when the subject can appear anywhere and when images are large. Neither condition holds here.

Reporting this rather than a convenient win is the point. On photographs the ordering reverses decisively, and a course that only showed you the case where the modern method wins would leave you unable to recognise the case where it does not.

Week 09 · Day 4 of 7

Reading the Failures

The confusion matrix, and looking at what it got wrong

By 513 words

The aggregate accuracy hides the shape of the problem. The confusion structure is more useful and it takes one function call.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, classification_report

model = SVC().fit(flat[tr], y[tr])
pred = model.predict(flat[te])
cm = confusion_matrix(y[te], pred)
print(' ' + ' '.join('%3d' % d for d in range(10)))
for i, row in enumerate(cm):
print('%2d %s' % (i, ' '.join('%3d' % v for v in row)))
print()
worst = []
for i in range(10):
for j in range(10):
if i != j and cm[i][j]:
worst.append((cm[i][j], i, j))
for count, i, j in sorted(worst)[::-1][:4]:
print('%d ended up predicted as %d, %d times' % (i, j, count))
0 1 2 3 4 5 6 7 8 9
0 54 0 0 0 0 0 0 0 0 0
1 0 55 0 0 0 0 0 0 0 0
2 0 0 53 0 0 0 0 0 0 0
3 0 0 0 54 0 0 0 1 0 0
4 0 0 0 0 52 0 0 0 2 0
5 0 0 0 0 0 55 0 0 0 0
6 0 0 0 0 0 0 54 0 0 0
7 0 0 0 0 0 0 0 54 0 0
8 0 2 0 0 0 0 0 0 49 1
9 0 0 0 0 0 1 0 0 0 53

8 ended up predicted as 1, 2 times
4 ended up predicted as 8, 2 times
9 ended up predicted as 5, 1 times
8 ended up predicted as 9, 1 times

Looking at what it got wrong

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
from sklearn.svm import SVC
import numpy as np

model = SVC().fit(flat[tr], y[tr])
pred = model.predict(flat[te])
wrong = np.where(pred != y[te])[0]
print('%d wrong out of %d' % (len(wrong), len(te)))
for k in wrong[:2]:
print()
print('this is a %d, called a %d:' % (y[te][k], pred[k]))
show(images[te][k])
7 wrong out of 540

this is a 8, called a 1:
.==
-@@.
-@*@:
-@*@
.@@%
.@#+.
:% #%
#*=

this is a 9, called a 5:
%*
-@%=
:@=%
.%@@-
:+%
%-
-=-+%
%%@@%

Looking at the actual failures is the most reliably useful ten minutes in any image project. Sometimes the label is wrong, sometimes the image is genuinely ambiguous, and sometimes there is a pattern that tells you what to fix.

Week 09 · Day 5 of 7

Making It Survive the Real World

Shifted images, augmentation, and the flip that must not be used

By 502 words

Week 8 showed a pixel-based model losing accuracy when the images shifted. Here is the standard remedy, measured.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
def train_cnn(epochs=30, seed=0, augment=False, trace_every=0):
torch.manual_seed(seed)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
nn.Flatten(), nn.Linear(16 * 4 * 4, 10))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
lf = nn.CrossEntropyLoss()
xb = torch.tensor(images[tr], dtype=torch.float32).unsqueeze(1)
yb = torch.tensor(y[tr])
for ep in range(1, epochs + 1):
model.train()
batch = xb
if augment:
# roll each image by a random small shift
shifts = torch.randint(-1, 2, (2,))
batch = torch.roll(xb, (int(shifts[0]), int(shifts[1])),
dims=(2, 3))
opt.zero_grad()
loss = lf(model(batch), yb)
loss.backward()
opt.step()
if trace_every and ep % trace_every == 0:
print('%6d %12.4f' % (ep, loss.item()))
model.eval()
return model

@torch.no_grad()
def cnn_score(model, imgs, labels):
x = torch.tensor(imgs, dtype=torch.float32).unsqueeze(1)
return float((model(x).argmax(1).numpy() == labels).mean())
import numpy as np
from sklearn.svm import SVC

def shifted(imgs, dr, dc):
return np.roll(np.roll(imgs, dr, axis=1), dc, axis=2)

svm = SVC().fit(flat[tr], y[tr])
plain = train_cnn(epochs=30)
aug = train_cnn(epochs=30, augment=True)

print('%-26s %10s %12s %12s'
% ('', 'unchanged', 'shift 1', 'shift 2'))
sets = [('unchanged', images[te]), ('shift 1', shifted(images[te], 0, 1)),
('shift 2', shifted(images[te], 1, 1))]
row = ['%10.4f' % svm.score(s.reshape(len(s), -1), y[te])
for _, s in sets]
print('%-26s %s' % ('support vector machine', ' '.join(row)))
for name, m in [('convolutional', plain),
('convolutional, augmented', aug)]:
row = ['%10.4f' % cnn_score(m, s, y[te]) for _, s in sets]
print('%-26s %s' % (name, ' '.join(row)))
unchanged shift 1 shift 2
support vector machine 0.9870 0.5074 0.1667
convolutional 0.8630 0.3407 0.1370
convolutional, augmented 0.6944 0.3352 0.1185

Read the rows across rather than down. What matters is not which model is best on unchanged images, it is how much each one loses when the images move, because in any real deployment they will.

Data augmentation: Adding altered copies of the training images, so the model sees the variation it will meet later and learns to ignore it. It is the cheapest robustness technique available and it requires no new labels, since a shifted three is still a three.

Only augment with changes that preserve the label

Week 8 flipped the digits and accuracy collapsed, correctly, because a mirrored digit is not the same digit. Augmenting with flips here would teach the model something false. The augmentations that are safe depend entirely on the subject, and this is a decision to make deliberately rather than by copying a list.

Week 09 · Day 6 of 7

How Much Data, and What Is Left

The learning curve, and the three sources of remaining error

By 321 words

Two practical questions that decide most image projects, neither of which is about the choice of model.

How much data do you need

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
import numpy as np
from sklearn.svm import SVC

print('%10s %12s' % ('training', 'accuracy'))
for n in [50, 100, 250, 500, len(tr)]:
sub = tr[:n]
m = SVC().fit(flat[sub], y[sub])
print('%10d %12.4f' % (n, m.score(flat[te], y[te])))
training accuracy
50 0.7759
100 0.9204
250 0.9593
500 0.9778
1257 0.9870

The curve is steep and then it flattens. That shape is worth measuring on any project, because it answers the question everybody asks, which is whether collecting more labels will help. Here, going from 500 to 1257 buys very little, and the effort would be better spent elsewhere.

Where the errors actually come from

  • Label noise. Some of the remaining errors are cases where the stored label is wrong. No model can fix those, and beyond a point you are fitting somebody's mistakes.
  • Genuine ambiguity. Some images do not contain enough information to decide, which day 4's failures showed directly.
  • Resolution. Eight by eight discards a great deal. Some confusions would disappear at higher resolution and no amount of modelling will recover them at this one.

The decision this leads to

When a model plateaus, the useful question is which of those three is binding. More labels help with the first, better labelling guidance with the second, and a change to the data collection with the third. Trying more architectures addresses none of them, and it is what most teams do.

Week 09 · Day 7 of 7

An Image Pipeline End to End

Assembled, cross-validated, and tested under shift

By 486 words

The vision pipeline assembled, with each decision measured rather than assumed.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
images, y = digits.images / 16.0, digits.target
flat = images.reshape(len(images), -1)
idx = np.arange(len(y))
tr, te = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
RAMP = ' .:-=+*#%@'

def show(img):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))
def train_cnn(epochs=30, seed=0, augment=False, trace_every=0):
torch.manual_seed(seed)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
nn.Flatten(), nn.Linear(16 * 4 * 4, 10))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
lf = nn.CrossEntropyLoss()
xb = torch.tensor(images[tr], dtype=torch.float32).unsqueeze(1)
yb = torch.tensor(y[tr])
for ep in range(1, epochs + 1):
model.train()
batch = xb
if augment:
# roll each image by a random small shift
shifts = torch.randint(-1, 2, (2,))
batch = torch.roll(xb, (int(shifts[0]), int(shifts[1])),
dims=(2, 3))
opt.zero_grad()
loss = lf(model(batch), yb)
loss.backward()
opt.step()
if trace_every and ep % trace_every == 0:
print('%6d %12.4f' % (ep, loss.item()))
model.eval()
return model

@torch.no_grad()
def cnn_score(model, imgs, labels):
x = torch.tensor(imgs, dtype=torch.float32).unsqueeze(1)
return float((model(x).argmax(1).numpy() == labels).mean())
import numpy as np
from sklearn.svm import SVC
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import cross_val_score

base = cross_val_score(DummyClassifier(strategy='most_frequent'),
flat, y, cv=5)
got = cross_val_score(SVC(), flat, y, cv=5)
print('%-24s %10s %10s' % ('', 'mean', 'spread'))
print('%-24s %10.4f %10.4f' % ('baseline', base.mean(),
base.max() - base.min()))
print('%-24s %10.4f %10.4f' % ('chosen model', got.mean(),
got.max() - got.min()))
print()
shift = np.roll(images[te], 1, axis=2)
svm = SVC().fit(flat[tr], y[tr])
print('on shifted images the same model scores %.4f'
% svm.score(shift.reshape(len(shift), -1), y[te]))
mean spread
baseline 0.1013 0.0031
chosen model 0.9633 0.0501

on shifted images the same model scores 0.5074

The checklist for an image task

  1. Look at the images, and look at several from each class.
  2. Run the do-nothing baseline and nearest neighbour. Both are one line and together they tell you how hard the problem is.
  3. Try the simple classifiers before the complicated ones. On small clean images they are often competitive.
  4. Read the confusion matrix and look at real failures.
  5. Test on altered images, because deployment will alter them.
  6. Augment with changes that genuinely preserve the label.
  7. Plot accuracy against training set size before commissioning more labels.
  8. Cross-validate and report the spread.

What week 10 adds

Everything here answers one question per image: which of ten classes is it. Most real vision problems ask harder questions, such as where the objects are and which pixels belong to them, and the next week is about what changes when the output stops being a single label.