The Capstone

Week 18 of 18 · Capstone · 7 days

Full curriculum
Week 18 · Capstone

The Capstone

Week 18 · Day 1 of 7

The Brief, and Looking at the Data

A dataset the course has not used, and the splits that keep the answer honest

By 488 words

Seventeen weeks of techniques, each measured on a problem chosen to show it working. This week is one problem worked end to end, on a dataset none of the earlier weeks used, so nothing here has been tuned in advance to make the course look good.

Fashion-MNIST is ten classes of clothing, the same shape and size as the digits, and considerably harder. It is a fair test of whether the habits transferred.

Look at the data before writing a model

import os, time, copy
import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms

# A dataset no earlier week used, so nothing here is tuned to it.
tf_ = transforms.ToTensor()
train_all = datasets.FashionMNIST('data', train=True, download=True,
transform=tf_)
test_all = datasets.FashionMNIST('data', train=False, download=True,
transform=tf_)
CLASSES = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
'sandal', 'shirt', 'sneaker', 'bag', 'boot']

def stack(ds, n, start=0):
xs = torch.stack([ds[i][0] for i in range(start, start + n)])
ys = torch.tensor([ds[i][1] for i in range(start, start + n)])
return xs, ys

xtr, ytr = stack(train_all, 12000)
xva, yva = stack(train_all, 3000, start=12000)
xte, yte = stack(test_all, 3000)
print('train %s val %s test %s'
% (tuple(xtr.shape), tuple(xva.shape), tuple(xte.shape)))
counts = torch.bincount(ytr, minlength=10)
print()
print('%-10s %8s' % ('class', 'count'))
for i, c in enumerate(CLASSES):
print('%-10s %8d' % (c, counts[i]))
print()
print('most common class is %.4f of the data'
% (counts.max().item() / len(ytr)))
train (12000, 1, 28, 28) val (3000, 1, 28, 28) test (3000, 1, 28, 28)

class count
t-shirt 1122
trouser 1220
pullover 1201
dress 1212
coat 1181
sandal 1204
shirt 1244
sneaker 1192
bag 1195
boot 1229

most common class is 0.1037 of the data

Balanced, which means accuracy is a reasonable metric and the majority-class baseline will be about a tenth. If it had come back ninety percent one class, everything downstream would have needed to change, which is why this is the first thing to run rather than an afterthought.

Three splits, and the middle one is not optional

Training, validation and test. The validation set chooses the model and the epoch. The test set is looked at once, at the end. Every number in this week that says val was used to make a decision, and every number that says test was not.

Selecting on the test set is the most common way a reported result turns out to be optimistic, and it does not feel like cheating while you are doing it. It feels like iterating.

The brief

  • Classify a garment image into one of ten categories.
  • Twelve thousand labelled examples, which is few enough that generalisation is a real concern.
  • The result has to be servable: exportable, and fast enough that a request is not noticeably waiting on it.
  • Every claimed improvement has to survive the check that week 3 insisted on.
Week 18 · Day 2 of 7

Baselines First

The two numbers that make every later number mean something

By 619 words

Before any network, the two numbers that make every later number meaningful.

import os, time, copy
import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms

# A dataset no earlier week used, so nothing here is tuned to it.
tf_ = transforms.ToTensor()
train_all = datasets.FashionMNIST('data', train=True, download=True,
transform=tf_)
test_all = datasets.FashionMNIST('data', train=False, download=True,
transform=tf_)
CLASSES = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
'sandal', 'shirt', 'sneaker', 'bag', 'boot']

def stack(ds, n, start=0):
xs = torch.stack([ds[i][0] for i in range(start, start + n)])
ys = torch.tensor([ds[i][1] for i in range(start, start + n)])
return xs, ys

xtr, ytr = stack(train_all, 12000)
xva, yva = stack(train_all, 3000, start=12000)
xte, yte = stack(test_all, 3000)
def dense():
return nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(),
nn.Linear(128, 10))

def conv(width=32, norm=False, drop=0.0):
def block(i, o):
layers = [nn.Conv2d(i, o, 3, padding=1)]
if norm:
layers.append(nn.BatchNorm2d(o))
return layers + [nn.ReLU(), nn.MaxPool2d(2)]
return nn.Sequential(*(block(1, width) + block(width, width * 2)
+ [nn.Flatten(), nn.Dropout(drop),
nn.Linear(width * 2 * 49, 128), nn.ReLU(),
nn.Linear(128, 10)]))

def augment(x):
"""Horizontal flip and a small shift. A flipped shoe is still a
shoe, which is the check week 4 said to make before using one."""
n = len(x)
flip = torch.rand(n) < 0.5
x = torch.where(flip.reshape(n, 1, 1, 1), x.flip(3), x)
idx = torch.arange(28)
pad = nn.functional.pad(x, (2, 2, 2, 2))
rows = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
cols = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
out = pad[torch.arange(n).reshape(n, 1, 1), 0,
rows.reshape(n, 28, 1), cols.reshape(n, 1, 28)]
return out.unsqueeze(1)

def fit(model, epochs=8, lr=1e-3, seed=0, aug=False, wd=0.0):
torch.manual_seed(seed)
opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=wd)
lf = nn.CrossEntropyLoss()
loader = DataLoader(TensorDataset(xtr, ytr), batch_size=128,
shuffle=True)
best, best_state = 0.0, None
for _ in range(epochs):
model.train()
for xb, yb in loader:
if aug:
xb = augment(xb)
opt.zero_grad()
lf(model(xb), yb).backward()
opt.step()
acc = score(model, xva, yva)
if acc > best:
best = acc
best_state = copy.deepcopy(model.state_dict())
# keep the weights that were best, which week 2 was firm about
model.load_state_dict(best_state)
model.eval()
return model, best

@torch.no_grad()
def score(model, x, y):
model.eval()
out = torch.cat([model(x[i:i + 512]) for i in range(0, len(x), 512)])
return (out.argmax(1) == y).float().mean().item()
from sklearn.linear_model import LogisticRegression
print('%-38s %10s' % ('', 'test accuracy'))
counts = torch.bincount(ytr, minlength=10)
major = int(counts.argmax())
print('%-38s %10.4f' % ('always guess the commonest class',
(yte == major).float().mean().item()))
lr = LogisticRegression(max_iter=200)
lr.fit(xtr.flatten(1).numpy(), ytr.numpy())
print('%-38s %10.4f' % ('logistic regression on raw pixels',
lr.score(xte.flatten(1).numpy(), yte.numpy())))
test accuracy
always guess the commonest class 0.0993
logistic regression on raw pixels 0.8300

A linear model on raw pixels gets 0.83. That single number reframes the entire project. Anything that follows is competing for the remaining seventeen points, not for eighty-three, and a deep model that reaches 0.86 has added three points to a model that took two seconds to fit and needs no framework at all.

The baseline is the number people skip and then misreport

Without that row, 0.89 sounds like a strong result. With it, 0.89 is a six point improvement over logistic regression, and the interesting question becomes whether six points justifies the complexity, the training cost and the deployment weight. That is a real question with a real answer that depends on the application.

Week 18 · Day 3 of 7

Building It Up

Six changes, one at a time, each measured

By 680 words

Now the model, built up one change at a time, each measured. This is the ablation table the course has asked for since week 3, on a real problem.

import os, time, copy
import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms

# A dataset no earlier week used, so nothing here is tuned to it.
tf_ = transforms.ToTensor()
train_all = datasets.FashionMNIST('data', train=True, download=True,
transform=tf_)
test_all = datasets.FashionMNIST('data', train=False, download=True,
transform=tf_)
CLASSES = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
'sandal', 'shirt', 'sneaker', 'bag', 'boot']

def stack(ds, n, start=0):
xs = torch.stack([ds[i][0] for i in range(start, start + n)])
ys = torch.tensor([ds[i][1] for i in range(start, start + n)])
return xs, ys

xtr, ytr = stack(train_all, 12000)
xva, yva = stack(train_all, 3000, start=12000)
xte, yte = stack(test_all, 3000)
def dense():
return nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(),
nn.Linear(128, 10))

def conv(width=32, norm=False, drop=0.0):
def block(i, o):
layers = [nn.Conv2d(i, o, 3, padding=1)]
if norm:
layers.append(nn.BatchNorm2d(o))
return layers + [nn.ReLU(), nn.MaxPool2d(2)]
return nn.Sequential(*(block(1, width) + block(width, width * 2)
+ [nn.Flatten(), nn.Dropout(drop),
nn.Linear(width * 2 * 49, 128), nn.ReLU(),
nn.Linear(128, 10)]))

def augment(x):
"""Horizontal flip and a small shift. A flipped shoe is still a
shoe, which is the check week 4 said to make before using one."""
n = len(x)
flip = torch.rand(n) < 0.5
x = torch.where(flip.reshape(n, 1, 1, 1), x.flip(3), x)
idx = torch.arange(28)
pad = nn.functional.pad(x, (2, 2, 2, 2))
rows = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
cols = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
out = pad[torch.arange(n).reshape(n, 1, 1), 0,
rows.reshape(n, 28, 1), cols.reshape(n, 1, 28)]
return out.unsqueeze(1)

def fit(model, epochs=8, lr=1e-3, seed=0, aug=False, wd=0.0):
torch.manual_seed(seed)
opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=wd)
lf = nn.CrossEntropyLoss()
loader = DataLoader(TensorDataset(xtr, ytr), batch_size=128,
shuffle=True)
best, best_state = 0.0, None
for _ in range(epochs):
model.train()
for xb, yb in loader:
if aug:
xb = augment(xb)
opt.zero_grad()
lf(model(xb), yb).backward()
opt.step()
acc = score(model, xva, yva)
if acc > best:
best = acc
best_state = copy.deepcopy(model.state_dict())
# keep the weights that were best, which week 2 was firm about
model.load_state_dict(best_state)
model.eval()
return model, best

@torch.no_grad()
def score(model, x, y):
model.eval()
out = torch.cat([model(x[i:i + 512]) for i in range(0, len(x), 512)])
return (out.argmax(1) == y).float().mean().item()
runs = [('dense, 128 hidden', lambda: dense(), {}),
('convolutional', lambda: conv(), {}),
(' and batch norm', lambda: conv(norm=True), {}),
(' and augmentation', lambda: conv(norm=True), {'aug': True}),
(' and dropout 0.3', lambda: conv(norm=True, drop=0.3),
{'aug': True}),
(' and weight decay', lambda: conv(norm=True, drop=0.3),
{'aug': True, 'wd': 1e-2}),
(' and twice the width', lambda: conv(64, norm=True, drop=0.3),
{'aug': True, 'wd': 1e-2})]
print('%-24s %10s %10s %12s' % ('', 'val', 'test', 'parameters'))
for name, make, kw in runs:
model, val = fit(make(), **kw)
print('%-24s %10.4f %10.4f %12d'
% (name, val, score(model, xte, yte),
sum(p.numel() for p in model.parameters())))
val test parameters
dense, 128 hidden 0.8420 0.8493 101770
convolutional 0.8800 0.8787 421642
and batch norm 0.8940 0.8923 421834
and augmentation 0.8817 0.8767 421834
and dropout 0.3 0.8800 0.8760 421834
and weight decay 0.8760 0.8780 421834
and twice the width 0.8730 0.8763 879114

Read down the test column. Convolution over dense is worth about three points, which is the largest single gain in the table and the one that reflects a real fact about images. Batch normalisation adds another point and a bit. And then everything stops.

Augmentation made it worse. Dropout made it worse again. Weight decay recovered a fraction. Doubling the width, at twice the parameters, changed nothing. Four techniques that every tutorial recommends, applied in the order every tutorial recommends them, and the best model in the table is the third row.

Week 18 · Day 4 of 7

How Much of That Was Real

The seed spread, and the table it demolishes

By 762 words

Before drawing any conclusion from yesterday's table, the question week 3 made a rule: how much does the same recipe vary when nothing changes but the seed?

import os, time, copy
import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms

# A dataset no earlier week used, so nothing here is tuned to it.
tf_ = transforms.ToTensor()
train_all = datasets.FashionMNIST('data', train=True, download=True,
transform=tf_)
test_all = datasets.FashionMNIST('data', train=False, download=True,
transform=tf_)
CLASSES = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
'sandal', 'shirt', 'sneaker', 'bag', 'boot']

def stack(ds, n, start=0):
xs = torch.stack([ds[i][0] for i in range(start, start + n)])
ys = torch.tensor([ds[i][1] for i in range(start, start + n)])
return xs, ys

xtr, ytr = stack(train_all, 12000)
xva, yva = stack(train_all, 3000, start=12000)
xte, yte = stack(test_all, 3000)
def dense():
return nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(),
nn.Linear(128, 10))

def conv(width=32, norm=False, drop=0.0):
def block(i, o):
layers = [nn.Conv2d(i, o, 3, padding=1)]
if norm:
layers.append(nn.BatchNorm2d(o))
return layers + [nn.ReLU(), nn.MaxPool2d(2)]
return nn.Sequential(*(block(1, width) + block(width, width * 2)
+ [nn.Flatten(), nn.Dropout(drop),
nn.Linear(width * 2 * 49, 128), nn.ReLU(),
nn.Linear(128, 10)]))

def augment(x):
"""Horizontal flip and a small shift. A flipped shoe is still a
shoe, which is the check week 4 said to make before using one."""
n = len(x)
flip = torch.rand(n) < 0.5
x = torch.where(flip.reshape(n, 1, 1, 1), x.flip(3), x)
idx = torch.arange(28)
pad = nn.functional.pad(x, (2, 2, 2, 2))
rows = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
cols = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
out = pad[torch.arange(n).reshape(n, 1, 1), 0,
rows.reshape(n, 28, 1), cols.reshape(n, 1, 28)]
return out.unsqueeze(1)

def fit(model, epochs=8, lr=1e-3, seed=0, aug=False, wd=0.0):
torch.manual_seed(seed)
opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=wd)
lf = nn.CrossEntropyLoss()
loader = DataLoader(TensorDataset(xtr, ytr), batch_size=128,
shuffle=True)
best, best_state = 0.0, None
for _ in range(epochs):
model.train()
for xb, yb in loader:
if aug:
xb = augment(xb)
opt.zero_grad()
lf(model(xb), yb).backward()
opt.step()
acc = score(model, xva, yva)
if acc > best:
best = acc
best_state = copy.deepcopy(model.state_dict())
# keep the weights that were best, which week 2 was firm about
model.load_state_dict(best_state)
model.eval()
return model, best

@torch.no_grad()
def score(model, x, y):
model.eval()
out = torch.cat([model(x[i:i + 512]) for i in range(0, len(x), 512)])
return (out.argmax(1) == y).float().mean().item()
accs = []
for seed in [0, 1, 2]:
model, _ = fit(conv(norm=True), seed=seed, aug=True)
accs.append(score(model, xte, yte))
a = np.array(accs)
print('same recipe, three seeds: %s' % ' '.join('%.4f' % v for v in a))
print('spread %.4f' % (a.max() - a.min()))
print()
print('any improvement smaller than that is not an improvement')
same recipe, three seeds: 0.8700 0.8813 0.8847
spread 0.0147

any improvement smaller than that is not an improvement

The noise floor is larger than most of the table

Three runs of one recipe span 0.0147. Now go back to yesterday's ablation and compare each step against that.

Augmentation costing 0.0156, dropout costing 0.0007, weight decay gaining 0.0020, doubling the width costing 0.0017: not one of those four is distinguishable from running the same code again with a different seed. Even batch normalisation's 0.0136 gain sits inside the noise band, and it is the second largest effect in the table.

So the honest report of yesterday's work is much shorter than the table. Convolution beats a dense network by an amount that clearly exceeds the noise. Everything after that is unproven on this budget, in either direction. Saying so is the finding.

What to do about it

  • Measure the seed spread once, early, and treat it as the resolution of every comparison you make afterwards.
  • Run the promising configurations at three seeds, not one, and compare the ranges rather than the points.
  • When two options are within noise, choose on the other axes: fewer parameters, faster inference, less code.
  • Report the spread alongside the number. A result given as 0.88 invites a comparison a result given as 0.87 to 0.88 correctly refuses.

On that last criterion the choice here is easy. The third row is the best mean, uses half the parameters of the widest model, and involves no augmentation code at all. It wins on everything that is not a coin flip.

Week 18 · Day 5 of 7

Where the Errors Actually Are

Per-class recall, the confusion structure, and what it implies

By 811 words

An aggregate accuracy hides the shape of the problem. The confusion structure is where the remaining errors actually are, and it usually suggests what to do next far better than another hyperparameter.

import os, time, copy
import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms

# A dataset no earlier week used, so nothing here is tuned to it.
tf_ = transforms.ToTensor()
train_all = datasets.FashionMNIST('data', train=True, download=True,
transform=tf_)
test_all = datasets.FashionMNIST('data', train=False, download=True,
transform=tf_)
CLASSES = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
'sandal', 'shirt', 'sneaker', 'bag', 'boot']

def stack(ds, n, start=0):
xs = torch.stack([ds[i][0] for i in range(start, start + n)])
ys = torch.tensor([ds[i][1] for i in range(start, start + n)])
return xs, ys

xtr, ytr = stack(train_all, 12000)
xva, yva = stack(train_all, 3000, start=12000)
xte, yte = stack(test_all, 3000)
def dense():
return nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(),
nn.Linear(128, 10))

def conv(width=32, norm=False, drop=0.0):
def block(i, o):
layers = [nn.Conv2d(i, o, 3, padding=1)]
if norm:
layers.append(nn.BatchNorm2d(o))
return layers + [nn.ReLU(), nn.MaxPool2d(2)]
return nn.Sequential(*(block(1, width) + block(width, width * 2)
+ [nn.Flatten(), nn.Dropout(drop),
nn.Linear(width * 2 * 49, 128), nn.ReLU(),
nn.Linear(128, 10)]))

def augment(x):
"""Horizontal flip and a small shift. A flipped shoe is still a
shoe, which is the check week 4 said to make before using one."""
n = len(x)
flip = torch.rand(n) < 0.5
x = torch.where(flip.reshape(n, 1, 1, 1), x.flip(3), x)
idx = torch.arange(28)
pad = nn.functional.pad(x, (2, 2, 2, 2))
rows = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
cols = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
out = pad[torch.arange(n).reshape(n, 1, 1), 0,
rows.reshape(n, 28, 1), cols.reshape(n, 1, 28)]
return out.unsqueeze(1)

def fit(model, epochs=8, lr=1e-3, seed=0, aug=False, wd=0.0):
torch.manual_seed(seed)
opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=wd)
lf = nn.CrossEntropyLoss()
loader = DataLoader(TensorDataset(xtr, ytr), batch_size=128,
shuffle=True)
best, best_state = 0.0, None
for _ in range(epochs):
model.train()
for xb, yb in loader:
if aug:
xb = augment(xb)
opt.zero_grad()
lf(model(xb), yb).backward()
opt.step()
acc = score(model, xva, yva)
if acc > best:
best = acc
best_state = copy.deepcopy(model.state_dict())
# keep the weights that were best, which week 2 was firm about
model.load_state_dict(best_state)
model.eval()
return model, best

@torch.no_grad()
def score(model, x, y):
model.eval()
out = torch.cat([model(x[i:i + 512]) for i in range(0, len(x), 512)])
return (out.argmax(1) == y).float().mean().item()
model, _ = fit(conv(norm=True), aug=True)
with torch.no_grad():
pred = torch.cat([model(xte[i:i + 512]).argmax(1)
for i in range(0, len(xte), 512)])
cm = torch.zeros(10, 10, dtype=torch.long)
for t, p in zip(yte.tolist(), pred.tolist()):
cm[t][p] += 1
print('%-10s %7s %7s %s' % ('class', 'recall', 'errors', 'mistaken for'))
for i, name in enumerate(CLASSES):
row = cm[i].clone()
right = int(row[i])
total = int(row.sum())
row[i] = 0
worst = int(row.argmax())
print('%-10s %7.3f %7d %s (%d)'
% (name, right / total, total - right, CLASSES[worst],
int(row[worst])))
class recall errors mistaken for
t-shirt 0.781 66 shirt (52)
trouser 0.994 2 dress (2)
pullover 0.687 97 shirt (68)
dress 0.849 45 shirt (17)
coat 0.796 66 shirt (55)
sandal 0.968 9 sneaker (9)
shirt 0.805 58 t-shirt (29)
sneaker 0.966 10 boot (6)
bag 0.946 16 shirt (5)
boot 0.926 21 sneaker (19)

The model is not uniformly 0.88 accurate. Trousers are 0.994 and pullovers are 0.687. Almost every serious error is inside one group: t-shirt, pullover, coat and shirt, garments that are genuinely similar at 28 by 28 in greyscale. The footwear classes confuse only with each other, and bags and trousers are essentially solved.

What that tells you to do

  • More capacity will not fix this. The information that separates a coat from a pullover is largely absent at this resolution, which is an input problem rather than a model problem.
  • Higher resolution or colour would help more than any architecture change, and both are decisions about data collection.
  • If the application tolerates it, merging the four upper-body classes into one would take accuracy well above 0.95 immediately. Whether that is acceptable is a product question, and it is worth asking before spending a month on the model.
  • If it is not tolerable, the labelling effort should go where the errors are rather than being spread evenly.

This is the most transferable habit in the week

An accuracy number tells you how much room is left. A confusion breakdown tells you where it is and often what kind of problem it is. Most of the time the answer is not a better model, and you cannot discover that from the aggregate.

Week 18 · Day 6 of 7

Shipping It, and Writing It Up

Export, verify, measure, and report in a way somebody can act on

By 780 words

The chosen model, exported and checked, using week 16's procedure.

import os, time, copy
import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms

# A dataset no earlier week used, so nothing here is tuned to it.
tf_ = transforms.ToTensor()
train_all = datasets.FashionMNIST('data', train=True, download=True,
transform=tf_)
test_all = datasets.FashionMNIST('data', train=False, download=True,
transform=tf_)
CLASSES = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
'sandal', 'shirt', 'sneaker', 'bag', 'boot']

def stack(ds, n, start=0):
xs = torch.stack([ds[i][0] for i in range(start, start + n)])
ys = torch.tensor([ds[i][1] for i in range(start, start + n)])
return xs, ys

xtr, ytr = stack(train_all, 12000)
xva, yva = stack(train_all, 3000, start=12000)
xte, yte = stack(test_all, 3000)
def dense():
return nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(),
nn.Linear(128, 10))

def conv(width=32, norm=False, drop=0.0):
def block(i, o):
layers = [nn.Conv2d(i, o, 3, padding=1)]
if norm:
layers.append(nn.BatchNorm2d(o))
return layers + [nn.ReLU(), nn.MaxPool2d(2)]
return nn.Sequential(*(block(1, width) + block(width, width * 2)
+ [nn.Flatten(), nn.Dropout(drop),
nn.Linear(width * 2 * 49, 128), nn.ReLU(),
nn.Linear(128, 10)]))

def augment(x):
"""Horizontal flip and a small shift. A flipped shoe is still a
shoe, which is the check week 4 said to make before using one."""
n = len(x)
flip = torch.rand(n) < 0.5
x = torch.where(flip.reshape(n, 1, 1, 1), x.flip(3), x)
idx = torch.arange(28)
pad = nn.functional.pad(x, (2, 2, 2, 2))
rows = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
cols = torch.randint(0, 5, (n, 1)) + idx.reshape(1, 28)
out = pad[torch.arange(n).reshape(n, 1, 1), 0,
rows.reshape(n, 28, 1), cols.reshape(n, 1, 28)]
return out.unsqueeze(1)

def fit(model, epochs=8, lr=1e-3, seed=0, aug=False, wd=0.0):
torch.manual_seed(seed)
opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=wd)
lf = nn.CrossEntropyLoss()
loader = DataLoader(TensorDataset(xtr, ytr), batch_size=128,
shuffle=True)
best, best_state = 0.0, None
for _ in range(epochs):
model.train()
for xb, yb in loader:
if aug:
xb = augment(xb)
opt.zero_grad()
lf(model(xb), yb).backward()
opt.step()
acc = score(model, xva, yva)
if acc > best:
best = acc
best_state = copy.deepcopy(model.state_dict())
# keep the weights that were best, which week 2 was firm about
model.load_state_dict(best_state)
model.eval()
return model, best

@torch.no_grad()
def score(model, x, y):
model.eval()
out = torch.cat([model(x[i:i + 512]) for i in range(0, len(x), 512)])
return (out.argmax(1) == y).float().mean().item()
import onnxruntime as ort
model, _ = fit(conv(norm=True), aug=True)
torch.onnx.export(model, (xte[:1],), 'fashion.onnx', dynamo=False,
input_names=['image'], output_names=['logits'],
dynamic_axes={'image': {0: 'batch'},
'logits': {0: 'batch'}})
sess = ort.InferenceSession('fashion.onnx',
providers=['CPUExecutionProvider'])
out = sess.run(None, {'image': xte[:512].numpy()})[0]
with torch.no_grad():
ref = model(xte[:512]).numpy()
print('exported agrees to %.2e' % float(np.abs(out - ref).max()))
print('same predictions: %s' % bool((out.argmax(1) == ref.argmax(1)).all()))
one = xte[:1].numpy()
for _ in range(10):
sess.run(None, {'image': one})
times = []
for _ in range(200):
start = time.perf_counter()
sess.run(None, {'image': one})
times.append((time.perf_counter() - start) * 1000)
times.sort()
print('median latency %.3f ms' % times[len(times) // 2])
print('file size %.1f KB' % (os.path.getsize('fashion.onnx') / 1024))
exported agrees to 2.86e-06
same predictions: True
median latency 0.096 ms
file size 1648.5 KB

Identical predictions, agreement to about three parts in a million, and a tenth of a millisecond per image. The brief's performance requirement is met with several orders of magnitude to spare, which is worth noticing: none of week 16's compression techniques are needed here, and applying them anyway would be work with no beneficiary.

How to write this up

A report that a colleague can act on, in the order that respects their time:

  1. The problem, the data, and the class balance.
  2. The baselines, both of them, before any model.
  3. The chosen model in one sentence, with its parameter count.
  4. The test accuracy, given as a range across seeds.
  5. The ablation, stating plainly which rows are inside the noise.
  6. The confusion structure, and what it implies about where effort should go next.
  7. The serving numbers: export verified, latency, size.
  8. What you did not try, and why.

The last item is the one people leave out

A report that lists only what worked reads as though the space was searched thoroughly. Saying that you did not try transfer learning because there was no suitable pretrained greyscale model, or did not tune the learning rate because the range test showed a wide flat optimum, tells the reader where the remaining opportunity is. It is also what stops the next person repeating your dead ends.

Week 18 · Day 7 of 7

Eighteen Weeks, and What Comes After

What this covered, what it did not, and where to go

By 411 words

Eighteen weeks. Worth being explicit about what that did and did not cover.

What you can now do

  • Write and debug a training loop, and recognise the failure shapes from their symptoms.
  • Build convolutional, recurrent and transformer models, and explain why each suits the data it suits.
  • Fine-tune a pretrained model, and know when frozen features are the better choice.
  • Train generative models, and measure them rather than admiring the samples.
  • Train without labels, and check the result against the baseline that reveals whether it did anything.
  • Make a model cheap enough to ship, and know which techniques are worth the effort on your hardware.
  • Serve it, watch it, and notice when it has quietly stopped working.

What this course did not teach

  • Training at scale. Everything ran on one CPU on small subsets. Multiple GPUs, sharded data and distributed optimisers are a different discipline.
  • Large language models in depth. Week 10 built a small one and week 11 used a pretrained one. Instruction tuning, preference optimisation and serving models that do not fit in memory are all beyond this.
  • Reinforcement learning. Not touched at all.
  • The mathematics. This was a practitioner's course. The proofs behind why any of it works are worth study and are not here.
  • Data collection and labelling. Every dataset here arrived clean and labelled. In practice that is most of the work, and day 5 showed the model's remaining errors were an input problem rather than a modelling one.

Where to go next

  1. Do a project on data you collected yourself. Every hard part of this field that the course could not simulate lives in that step.
  2. Read papers with the code open. Weeks 8 to 11 give you enough to follow most architecture work.
  3. Reproduce a published result. It is the fastest way to find out how much of what you know is actually load-bearing.
  4. Pick a specialism: vision, language, speech, or the systems side of making any of it run economically.
  5. Learn the maths properly if you intend to do research rather than apply it.

The habit worth more than any technique here

This week's ablation looked like six improvements and turned out to contain one. The difference between those two readings was three training runs at different seeds, which cost minutes. Almost everything that separates careful work from confident work in this field is that cheap, and almost nobody does it. Measure the baseline, measure the noise, and only then believe the result.