Convolutional Networks

Week 5 of 18 · Vision · 7 days

Full curriculum
Week 05 · Vision

Convolutional Networks

Week 05 · Day 1 of 7

Convolution

Why a dense layer wastes weights on images, and what a kernel does

By 699 words

A dense layer treats a picture as a list of unrelated numbers. Convolution is the correction, and the argument for it is short: the meaning of a pixel is almost entirely determined by the pixels next to it, and a pattern is the same pattern wherever in the frame it appears.

Count the weights first

# A 32 by 32 colour image into a first hidden layer of 64 units.
pixels = 32 * 32 * 3
dense = pixels * 64 + 64

# The same first layer as convolution: 64 filters of 3 by 3 over 3 channels.
conv = 3 * 3 * 3 * 64 + 64

print('flattened input %8d values' % pixels)
print('dense layer %8d weights' % dense)
print('conv layer %8d weights' % conv)
print('ratio %8.0f to 1' % (dense / conv))

print('\nand at 224 by 224, which is a normal photograph:')
big = 224 * 224 * 3
print(' dense %d, conv %d, ratio %.0f to 1'
% (big * 64 + 64, conv, (big * 64 + 64) / conv))
flattened input 3072 values
dense layer 196672 weights
conv layer 1792 weights
ratio 110 to 1

and at 224 by 224, which is a normal photograph:
dense 9633856, conv 1792, ratio 5376 to 1

The gap widens with image size, because the convolution's parameter count does not depend on the image at all. That is the first advantage. The second matters more: a dense layer has to learn what an edge looks like separately for every position it could occupy, and a convolution learns it once.

Doing one by hand

import torch

def conv2d(image, kernel):
kh, kw = kernel.shape
h, w = image.shape
out = torch.zeros(h - kh + 1, w - kw + 1)
for i in range(out.shape[0]):
for j in range(out.shape[1]):
out[i, j] = (image[i:i + kh, j:j + kw] * kernel).sum()
return out

image = torch.zeros(8, 8)
image[:, 4:] = 1.0 # a vertical edge down the middle
vertical = torch.tensor([[-1.0, 0.0, 1.0],
[-2.0, 0.0, 2.0],
[-1.0, 0.0, 1.0]])

print('the image:')
for row in image:
print(' ', ''.join('#' if v > 0.5 else '.' for v in row.tolist()))
print('\nresponse to a vertical edge detector:')
for row in conv2d(image, vertical):
print(' ', ' '.join('%5.1f' % v for v in row.tolist()))
the image:
....####
....####
....####
....####
....####
....####
....####
....####

response to a vertical edge detector:
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0

The filter responds strongly exactly where the brightness changes from left to right, and gives zero everywhere the image is flat. Turn the same kernel on its side and it finds horizontal edges instead. A convolutional network is not given these kernels: it starts from random numbers and learns whichever nine weights reduce the loss.

The same thing in PyTorch

import torch
from torch import nn

image = torch.zeros(1, 1, 8, 8)
image[:, :, :, 4:] = 1.0

conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, bias=False)
with torch.no_grad():
conv.weight[:] = torch.tensor([[-1.0, 0.0, 1.0],
[-2.0, 0.0, 2.0],
[-1.0, 0.0, 1.0]])

out = conv(image)
print('input ', tuple(image.shape))
print('kernel', tuple(conv.weight.shape), '= (out, in, height, width)')
print('output', tuple(out.shape))
print('\nsame numbers as the hand-rolled version:')
for row in out[0, 0]:
print(' ', ' '.join('%5.1f' % v for v in row.tolist()))
input (1, 1, 8, 8)
kernel (1, 1, 3, 3) = (out, in, height, width)
output (1, 1, 6, 6)

same numbers as the hand-rolled version:
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0
0.0 0.0 4.0 4.0 0.0 0.0

Read the kernel shape

(out_channels, in_channels, height, width). A layer with 64 filters over a 3 channel input using 3 by 3 kernels has a weight tensor of shape (64, 3, 3, 3), which is 1,728 numbers plus 64 biases. One bias per filter, not one per position, which is the sharing again.

Day 1 takeaway

A convolution slides a small set of shared weights over the image, so it learns a pattern once instead of once per position and its parameter count does not depend on the image size. The kernel shape is (out, in, height, width), and one bias belongs to each filter.
Week 05 · Day 2 of 7

Padding, Pooling and Receptive Fields

The arithmetic that decides every shape downstream

By 676 words

Three settings decide the shape of everything downstream, and getting them wrong is the most common reason a network refuses to compile.

Padding, stride and the output size

def out_size(n, k, stride=1, pad=0, dilation=1):
effective = dilation * (k - 1) + 1
return (n + 2 * pad - effective) // stride + 1

print('%7s %7s %7s %5s %9s %9s %s'
% ('input', 'kernel', 'stride', 'pad', 'dilation', 'output', 'name'))
rows = [(32, 3, 1, 0, 1, 'valid'), (32, 3, 1, 1, 1, 'same'),
(32, 5, 1, 2, 1, 'same'), (32, 3, 2, 1, 1, 'halves it'),
(32, 1, 1, 0, 1, 'pointwise'), (32, 3, 1, 2, 2, 'dilated')]
for n, k, s, pad, d, name in rows:
print('%7d %7d %7d %5d %9d %9d %s'
% (n, k, s, pad, d, out_size(n, k, s, pad, d), name))
input kernel stride pad dilation output name
32 3 1 0 1 30 valid
32 3 1 1 1 32 same
32 5 1 2 1 32 same
32 3 2 1 1 16 halves it
32 1 1 0 1 32 pointwise
32 3 1 2 2 32 dilated

For a 3 by 3 kernel, padding=1 keeps the size. For 5 by 5 it is padding=2. The rule is padding = (kernel - 1) / 2 for odd kernels, which is most of the reason even-sized kernels are rare.

Pooling

import torch
from torch import nn

x = torch.arange(16.0).reshape(1, 1, 4, 4)
print('input')
print(x[0, 0])

print('\nmax pool 2x2')
print(nn.MaxPool2d(2)(x)[0, 0])
print('\naverage pool 2x2')
print(nn.AvgPool2d(2)(x)[0, 0])
print('\nglobal average pool')
print(nn.AdaptiveAvgPool2d(1)(x)[0, 0])

print('\nstrided convolution reaches the same size with weights:')
print(tuple(nn.Conv2d(1, 1, 3, stride=2, padding=1)(x).shape))
input
tensor([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.],
[12., 13., 14., 15.]])

max pool 2x2
tensor([[ 5., 7.],
[13., 15.]])

average pool 2x2
tensor([[ 2.5000, 4.5000],
[10.5000, 12.5000]])

global average pool
tensor([[7.5000]])

strided convolution reaches the same size with weights:
(1, 1, 2, 2)
DownsamplingParametersNotes
Max poolNoneThe traditional choice; keeps the strongest response
Average poolNoneSmoother; usually only at the very end
Strided convolutionYesLearns how to downsample; the modern default
Global average poolNoneCollapses each channel to one number before the classifier

Receptive field

Receptive field: The region of the input that a single unit deep in the network can see. It grows with depth, and a unit cannot possibly recognise something larger than its own receptive field.
stack = [('conv 3x3', 3, 1), ('conv 3x3', 3, 1), ('maxpool 2x2', 2, 2),
('conv 3x3', 3, 1), ('conv 3x3', 3, 1), ('maxpool 2x2', 2, 2),
('conv 3x3', 3, 1), ('conv 3x3', 3, 1)]

r, jump = 1, 1
print('%-16s %8s %14s' % ('after', 'stride', 'sees (pixels)'))
for name, k, s in stack:
r = r + (k - 1) * jump
jump = jump * s
print('%-16s %8d %14d' % (name, jump, r))
print('\non a 32 by 32 image, the last layer sees most of the picture.')
after stride sees (pixels)
conv 3x3 1 3
conv 3x3 1 5
maxpool 2x2 2 6
conv 3x3 2 10
conv 3x3 2 14
maxpool 2x2 4 16
conv 3x3 4 24
conv 3x3 4 32

on a 32 by 32 image, the last layer sees most of the picture.

Why everything is 3 by 3 now

Two stacked 3 by 3 convolutions see a 5 by 5 region using 18 weights per channel pair, where one 5 by 5 convolution uses 25 and has one nonlinearity instead of two. Three of them match a 7 by 7 for 27 weights against 49. Small kernels stacked deep win on parameters, on computation and on expressiveness at the same time, which is why architectures stopped using anything else after about 2014.

Day 2 takeaway

padding = (kernel - 1) / 2 preserves the size. Downsample with a stride or a pool; strided convolution is the modern default because it learns how. Track the receptive field, because a unit cannot recognise anything bigger than the region it can see.
Week 05 · Day 3 of 7

A Real Image Model

CIFAR-10, a dense baseline, and the convolutional network that beats it

By 1247 words

CIFAR-10 is 32 by 32 colour photographs in ten classes. It is small enough to train on a laptop and hard enough that the architecture genuinely matters, which is why it has been the standard teaching problem for a decade.

The data

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
print('training images', len(train_all))
print('classes:', CLASSES)
image, label = train_all[7]
print('\none example: shape %s, label %d (%s)'
% (tuple(image.shape), label, CLASSES[label]))

import collections
counts = collections.Counter(train_all.targets[:8000])
print('\nclass balance in the training subset:')
for i, name in enumerate(CLASSES):
print(' %-12s %d' % (name, counts[i]))
training images 50000
classes: ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

one example: shape (3, 32, 32), label 7 (horse)

class balance in the training subset:
airplane 796
automobile 765
bird 817
cat 809
deer 806
dog 765
frog 832
horse 809
ship 809
truck 792
import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
raw = datasets.CIFAR10('data', train=True, download=True)
image, label = raw[7]
grey = torch.tensor([[sum(image.getpixel((x, y))) / 765.0
for x in range(32)] for y in range(32)])
ramp = ' .:-=+*#%@'
print('%s, as brightness:' % CLASSES[label])
for row in grey:
print(''.join(ramp[min(9, int(v * 9.999))] for v in row.tolist()))
horse, as brightness:
...::..:::--:--::--::::.::....:.
....:.::::==:-=+++++===:----====
....-:.::=*+==+#***#************
...:-:...-=-=+*#######****######
--:.........:-+#%%%%##*++=**####
++:......:::---=*%%%#*+==--==*##
#+::...:::---==-=*#***+--::--*##
#+::. .-------===++++++=--::-*#*
%=.:. .------===++++====--:::+#*
%+.::..:--:===-==++==-==--::.=##
#=:::..:---=+=--=+===--=-:...-**
#=::..:.:-=++=--====-::--:...-**
#=:..=+::-====:-==---:::-:...-**
#+:.:*#*--=---::----:-::-:.. =##
#+::+#%%*-::::::---::::::... =#*
*--*%%###=.....---::::::. . =#*
#-+%%###%*....-==-:...:+:.=: -**
%##%%###%#-...--:.....=#- ++..*#
###%%#####*:..:-::..:=*#= =*: +#
####%######-..=#*+++*###+.:*= =#
###########= .+%%%######*.-#*.=#
###########= .*%########*:=##:=#
###########+ .*#######**+.=**-=#
*****####**+ .+####*****=.=**::*
**#**##****+. :*#***+++*-.=**:.*
*#####*****+.-.:=+***+++:.=*+::+
**####**#**+.==..=**+++=::=++--+
**#######**+.-+-.:=++++=-======+
****####***=:=++--=+++++++++=+=+
++***++***+==+++=+++++++++++++++
++*+++++***++++++**++++==+++++++
++++++++**+++++++++=++===+=++===

A dense baseline, so the comparison is honest

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
dense = nn.Sequential(nn.Flatten(),
nn.Linear(3 * 32 * 32, 512), nn.ReLU(),
nn.Linear(512, 256), nn.ReLU(),
nn.Linear(256, 10))
print('parameters %d' % sum(p.numel() for p in dense.parameters()))
print('accuracy %.4f' % fit(dense, epochs=8))
parameters 1707274
accuracy 0.4310

The convolutional version

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
cnn = nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(),
nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(),
nn.MaxPool2d(2), # 32 -> 16
nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
nn.Conv2d(64, 64, 3, padding=1), nn.ReLU(),
nn.MaxPool2d(2), # 16 -> 8
nn.Flatten(),
nn.Linear(64 * 8 * 8, 256), nn.ReLU(),
nn.Linear(256, 10))
print('parameters %d' % sum(p.numel() for p in cnn.parameters()))
print('accuracy %.4f' % fit(cnn, epochs=8))
parameters 1116970
accuracy 0.4975

Fewer parameters, better result

This is the demonstration the whole week exists for, and it is worth sitting with. The convolutional model has fewer weights than the dense one and is substantially more accurate, on identical data, with an identical training recipe. The difference is not capacity. It is that one architecture knows pixels have neighbours and the other does not.

Watching the shapes

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
layers = [nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(),
nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(64 * 8 * 8, 10)]

x = torch.zeros(1, 3, 32, 32)
print('%-22s %-18s %10s' % ('layer', 'output', 'params'))
print('%-22s %-18s %10s' % ('input', str(tuple(x.shape[1:])), ''))
for layer in layers:
x = layer(x)
n = sum(p.numel() for p in layer.parameters())
print('%-22s %-18s %10d'
% (type(layer).__name__, str(tuple(x.shape[1:])), n))
layer output params
input (3, 32, 32)
Conv2d (32, 32, 32) 896
ReLU (32, 32, 32) 0
Conv2d (32, 32, 32) 9248
ReLU (32, 32, 32) 0
MaxPool2d (32, 16, 16) 0
Conv2d (64, 16, 16) 18496
ReLU (64, 16, 16) 0
MaxPool2d (64, 8, 8) 0
Flatten (4096,) 0
Linear (10,) 40970

Read the parameter column. Almost everything is in the final linear layer, because flattening an 8 by 8 by 64 map produces 4,096 numbers and connecting those to anything is expensive. That single observation is what global average pooling exists to fix, and day 5 measures it.

Day 3 takeaway

On real images a convolutional network beats a dense one of similar size comfortably, using fewer parameters. Print the shape and parameter count at every layer, and expect the flatten-then-linear step to dominate the parameter count unless you do something about it.
Week 05 · Day 4 of 7

Equivariance and Augmentation

Measuring how little invariance you actually get for free

By 1068 words

Convolution is equivariant to translation: move the input and the output moves with it. That is not the same as being invariant, and the difference decides how much augmentation you need.

Equivariance, demonstrated

import torch
from torch import nn

torch.manual_seed(0)
conv = nn.Conv2d(1, 1, 3, padding=1, bias=False)

x = torch.zeros(1, 1, 8, 8)
x[0, 0, 2, 2] = 1.0
shifted = torch.roll(x, shifts=3, dims=3)

a = conv(x)[0, 0]
b = conv(shifted)[0, 0]
print('response moved with the input:',
bool(torch.allclose(torch.roll(a, 3, dims=1)[:, 3:], b[:, 3:],
atol=1e-6)))

pool = nn.AdaptiveMaxPool2d(1)
print('\nafter global max pooling:')
print(' original %.6f' % pool(conv(x)).item())
print(' shifted %.6f' % pool(conv(shifted)).item())
print(' identical, so pooling is what buys invariance')
response moved with the input: True

after global max pooling:
original 0.264297
shifted 0.264297
identical, so pooling is what buys invariance

How much invariance a real network actually has

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
cnn = nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(64 * 8 * 8, 10))
base = fit(cnn, epochs=4)
print('accuracy on the validation set as it is: %.4f' % base)

cnn.eval()
print('\n%14s %12s' % ('shift (pixels)', 'accuracy'))
for shift in [0, 1, 2, 4, 8]:
right = seen = 0
with torch.no_grad():
for xb, yb in DataLoader(val_set, batch_size=512):
moved = torch.roll(xb, shifts=shift, dims=3)
right += (cnn(moved).argmax(1) == yb).sum().item()
seen += yb.numel()
print('%14d %12.4f' % (shift, right / seen))
accuracy on the validation set as it is: 0.5375

shift (pixels) accuracy
0 0.5375
1 0.5285
2 0.5150
4 0.4740
8 0.4025

Convolution buys much less invariance than people assume

A four pixel shift on a 32 pixel image is enough to cost a convolutional network a serious amount of accuracy, despite every textbook diagram implying otherwise. Two pooling layers only discard position within a two by two block each; anything larger than that reaches the classifier as a genuinely different input.

This is exactly why augmentation matters so much on images, and why the standard CIFAR recipe includes random crops. You do not get invariance from the architecture. You train it in.

The standard CIFAR augmentation

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
aug_tf = transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(MEAN, STD),
])
aug_train = Subset(datasets.CIFAR10('data', train=True, download=True,
transform=aug_tf), range(4000))

def make():
torch.manual_seed(0)
return nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(),
nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
nn.Conv2d(64, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(64 * 8 * 8, 256), nn.ReLU(),
nn.Linear(256, 10))

print('%-22s %12s' % ('', 'accuracy'))
print('%-22s %12.4f' % ('no augmentation', fit(make(), epochs=5)))
print('%-22s %12.4f'
% ('crop and flip', fit(make(), epochs=5, train=aug_train)))
accuracy
no augmentation 0.4360
crop and flip 0.3525

Augmentation made it worse, and that is the expected result here

Read the table again: the augmented run is behind. This is not a contradiction of week 4, it is the budget. Augmentation works by making the training task harder so the model cannot memorise, and a model given five epochs on four thousand images has not begun to memorise anything. All the harder task does is slow it down.

Augmentation pays when training is long enough for overfitting to be the binding constraint. The standard CIFAR recipes that include random crops run for a hundred epochs or more on fifty thousand images, and there it is worth several points. If you add augmentation and your score drops, check whether you are training long enough to need it before concluding it does not work.

A horizontal flip is safe on CIFAR because a mirrored cat is a cat. It is unsafe on digits, on text, and on anything where left and right carry meaning. The random crop with four pixels of padding is doing exactly the job the shift experiment above showed was needed, and it will pay for itself on a longer run.

Day 4 takeaway

Convolution is equivariant to translation and pooling converts a little of that into invariance, but far less than people assume. Measure it by shifting your validation set. Then buy the rest with random crops and flips, which is why every CIFAR recipe ever published contains both.
Week 05 · Day 5 of 7

Normalisation, Pooling Heads and Depth

Four changes towards a modern architecture, measured one at a time

By 1224 words

Four changes that turn the model from day 3 into something resembling a modern architecture, added one at a time so each is measured.

Batch normalisation between convolution and activation

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
def make(bn):
torch.manual_seed(0)
def block(cin, cout):
layers = [nn.Conv2d(cin, cout, 3, padding=1, bias=not bn)]
if bn:
layers.append(nn.BatchNorm2d(cout))
layers.append(nn.ReLU())
return layers
return nn.Sequential(
*block(3, 32), *block(32, 32), nn.MaxPool2d(2),
*block(32, 64), *block(64, 64), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(64 * 8 * 8, 256), nn.ReLU(),
nn.Linear(256, 10))

print('%-24s %10s %12s' % ('', 'lr', 'accuracy'))
for bn in [False, True]:
for lr in [0.05, 0.3]:
acc = fit(make(bn), epochs=6, lr=lr)
print('%-24s %10.2f %12.4f'
% ('with batch norm' if bn else 'plain', lr, acc))
lr accuracy
plain 0.05 0.4585
plain 0.30 0.1080
with batch norm 0.05 0.5695
with batch norm 0.30 0.5390

The interesting column is the high learning rate. Without normalisation the model is unstable there; with it, the same rate is usable and often better. That is what batch normalisation was introduced for, and the regularising effect noted in week 4 was a side effect nobody planned.

Turn off the bias when a normalisation layer follows

Batch normalisation subtracts the mean, which removes any constant the convolution added. The bias is not merely useless, it is a parameter that can drift without affecting the loss. Pass bias=False, as the code above does. Every reference implementation does this and almost every first attempt forgets.

Global average pooling instead of flatten

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
def make(head):
torch.manual_seed(0)
body = [nn.Conv2d(3, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1, bias=False), nn.BatchNorm2d(64),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(64, 128, 3, padding=1, bias=False),
nn.BatchNorm2d(128), nn.ReLU()]
if head == 'flatten':
tail = [nn.Flatten(), nn.Linear(128 * 8 * 8, 10)]
else:
tail = [nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(128, 10)]
return nn.Sequential(*body, *tail)

print('%-16s %12s %12s' % ('head', 'params', 'accuracy'))
for head in ['flatten', 'global pool']:
model = make(head)
n = sum(p.numel() for p in model.parameters())
print('%-16s %12d %12.4f' % (head, n, fit(model, epochs=4)))
head params accuracy
flatten 175402 0.5035
global pool 94762 0.4670

Global pooling lost on this run, and it removed nearly half the parameters to do it. Both facts are the point: it is a trade, not an improvement, and at 32 by 32 with the subject filling the frame there is still enough positional information in an 8 by 8 map to be worth keeping.

The trade this makes

Global average pooling throws away where each feature was found and keeps only how strongly it was found. That removes most of the parameters, adds a lot of translation invariance, and costs you any information carried by position. On CIFAR, where the subject fills the frame, the trade is usually worth it. On a task where position is the signal it is not, and the Machine Learning course has a week 12 experiment where exactly this choice cost eleven points of accuracy.

Depth, and where it stops helping

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
def make(blocks):
torch.manual_seed(0)
layers, cin = [], 3
for i in range(blocks):
cout = min(32 * 2 ** (i // 2), 128)
layers += [nn.Conv2d(cin, cout, 3, padding=1, bias=False),
nn.BatchNorm2d(cout), nn.ReLU()]
if i % 2 == 1 and i < 6:
layers.append(nn.MaxPool2d(2))
cin = cout
layers += [nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(cin, 10)]
return nn.Sequential(*layers)

print('%8s %12s %12s' % ('conv layers', 'params', 'accuracy'))
for blocks in [2, 4, 8]:
model = make(blocks)
print('%8d %12d %12.4f'
% (blocks, sum(p.numel() for p in model.parameters()),
fit(model, epochs=4)))
conv layers params accuracy
2 10538 0.3470
4 66410 0.4555
8 584170 0.5540

Depth helps and then it stops helping, and beyond some point a deeper plain stack is actively worse than a shallower one, on the training set as well as the validation set. That is not overfitting, it is an optimisation failure, and fixing it is what week 6 is about.

Day 5 takeaway

Batch normalisation after each convolution, with the bias turned off, makes higher learning rates usable. Global average pooling removes most of the parameters and adds invariance, at the cost of position. Depth helps up to a point that arrives sooner than you expect.
Week 05 · Day 6 of 7

Looking Inside

Filters, feature maps, occlusion and the confusion matrix

By 1729 words

A trained network is not a black box if you are willing to look at four things: the filters, the feature maps, the errors, and what happens when you cover part of the input.

The filters it learned

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
model = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32),
nn.ReLU(), nn.MaxPool2d(2),
nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
fit(model, epochs=4)

w = model[0].weight.detach()
print('first layer kernels, averaged over the colour channels:')
for f in range(4):
k = w[f].mean(0)
print('\nfilter %d (sum %+.2f)' % (f, k.sum().item()))
for row in k:
print(' ' + ' '.join('%+.2f' % v for v in row.tolist()))
first layer kernels, averaged over the colour channels:

filter 0 (sum +0.06)
-0.06 -0.02 -0.08
-0.03 -0.07 -0.01
+0.10 +0.11 +0.11

filter 1 (sum -0.25)
-0.08 +0.08 +0.04
-0.04 +0.05 -0.05
-0.04 -0.10 -0.12

filter 2 (sum -0.04)
+0.14 +0.08 +0.09
-0.01 -0.03 -0.05
+0.01 -0.11 -0.16

filter 3 (sum +0.01)
-0.01 +0.21 +0.06
+0.01 -0.06 -0.12
+0.03 +0.03 -0.14

Read the signs rather than the sizes. A filter with negatives on one side and positives on the other detects an edge along that axis; one that is positive in the centre and negative around it detects a blob. Nobody specified these. They are what minimising the loss produced, and they are close to what image processing used by hand for forty years.

Feature maps

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
model = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32),
nn.ReLU(), nn.MaxPool2d(2),
nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
fit(model, epochs=4)
model.eval()

image, label = val_set[3]
with torch.no_grad():
maps = model[:3](image.unsqueeze(0))[0]
print('a %s, after the first convolution: %d maps of %d by %d'
% (CLASSES[label], maps.shape[0], maps.shape[1], maps.shape[2]))

ramp = ' .:-=+*#%@'
for f in [0, 1]:
m = maps[f]
m = (m - m.min()) / (m.max() - m.min() + 1e-9)
print('\nfeature map %d:' % f)
for row in m[::2]:
print(''.join(ramp[min(9, int(v * 9.999))] for v in row[::1].tolist()))
a airplane, after the first convolution: 16 maps of 32 by 32

feature map 0:
:=+++=--------------------=++*=:
::----: .. ........:---:::..
::---:.:... .......:::-:--..
::--- :::..
--------::......::.....:-=----:.
--==---==+=++++++=+=====--:::-:.
:::. .--------------------..
::==+**+. :--::::::::---::-..
::-----==: .:-:::-::----:.
:.::::::. :----:.
..::::: .-=- .=+-::: ..
..... .::-+#%#+-:=++++*+=:...


.--.
.:.

feature map 1:






.--:
=*=
+##+:
-#+++**##%*=
=#%+. .--=#%+.

==--------=====---=======--====.
+*#***#**##########%%##%%%%%#*-.
-+#***********************+::+%-
:+********+++++++===++*+-:+@@*=:

Occlusion: which pixels the decision depended on

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
model = nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1, bias=False), nn.BatchNorm2d(64),
nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(64 * 8 * 8, 10))
fit(model, epochs=5)
model.eval()

image, label = val_set[3]
with torch.no_grad():
base = model(image.unsqueeze(0)).softmax(1)[0, label].item()
print('%s, predicted with probability %.3f' % (CLASSES[label], base))

drop = torch.zeros(4, 4)
patch = 8
with torch.no_grad():
for i in range(4):
for j in range(4):
covered = image.clone()
covered[:, i * patch:(i + 1) * patch,
j * patch:(j + 1) * patch] = 0.0
p_ = model(covered.unsqueeze(0)).softmax(1)[0, label].item()
drop[i, j] = base - p_

print('\nfall in confidence when each 8x8 patch is covered:')
for row in drop:
print(' ' + ' '.join('%+6.3f' % v for v in row.tolist()))
worst = drop.flatten().argmax().item()
print('\nthe model depended most on the patch at row %d, column %d'
% (worst // 4, worst % 4))
airplane, predicted with probability 0.271

fall in confidence when each 8x8 patch is covered:
-0.015 +0.020 -0.085 +0.071
-0.002 -0.026 -0.237 -0.102
-0.105 -0.060 -0.314 -0.031
-0.011 -0.271 -0.251 -0.019

the model depended most on the patch at row 0, column 3

This is the cheapest explanation method there is

No gradients, no library, no theory: cover part of the input and see whether the answer changes. It costs one forward pass per patch and it answers the question people actually ask. It is also the first place to look when a model is suspiciously good, because a model that depends on the corner of the image is a model that has found something about your dataset rather than about the subject.

Where the errors are

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
torch.manual_seed(0)
model = nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1, bias=False), nn.BatchNorm2d(64),
nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(64 * 8 * 8, 10))
fit(model, epochs=5)
model.eval()

confusion = torch.zeros(10, 10, dtype=torch.long)
with torch.no_grad():
for xb, yb in DataLoader(val_set, batch_size=512):
for t, p_ in zip(yb, model(xb).argmax(1)):
confusion[t, p_] += 1

print('%-11s' % '' + ''.join('%5s' % c[:4] for c in CLASSES))
for i, row in enumerate(confusion):
print('%-11s' % CLASSES[i][:10] + ''.join('%5d' % v
for v in row.tolist()))

pairs = [(confusion[i, j].item(), CLASSES[i], CLASSES[j])
for i in range(10) for j in range(10) if i != j]
pairs.sort(reverse=True)
print('\nthe five worst confusions:')
for n, a, b in pairs[:5]:
print(' %-12s read as %-12s %d times' % (a, b, n))
airp auto bird cat deer dog frog hors ship truc
airplane 118 3 10 6 2 5 3 4 29 16
automobile 14 122 4 1 2 0 2 2 17 34
bird 29 2 84 12 23 16 11 9 6 3
cat 12 4 16 72 14 36 18 17 4 6
deer 13 3 45 18 61 15 16 21 6 0
dog 3 2 15 51 14 71 5 18 4 2
frog 3 1 10 19 17 10 143 2 3 8
horse 10 3 10 17 11 15 1 117 3 6
ship 25 9 0 8 1 2 1 1 157 13
truck 18 28 3 4 2 2 4 8 13 121

the five worst confusions:
dog read as cat 51 times
deer read as bird 45 times
cat read as dog 36 times
automobile read as truck 34 times
bird read as airplane 29 times

Day 6 takeaway

Look at the first layer kernels to check the model learned something sensible, at feature maps to see what a layer responds to, at occlusion to find which region the decision rested on, and at the confusion matrix to see whether the mistakes are the ones a person would make.
Week 05 · Day 7 of 7

A Complete Convolutional Network

The recipe, the comparison, and an honest note about the budget

By 911 words

Everything from the week, in one model, trained properly and measured against the baselines it has to beat.

The model

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
def build():
torch.manual_seed(0)
def block(cin, cout, pool=False):
layers = [nn.Conv2d(cin, cout, 3, padding=1, bias=False),
nn.BatchNorm2d(cout), nn.ReLU()]
if pool:
layers.append(nn.MaxPool2d(2))
return layers
return nn.Sequential(
*block(3, 64), *block(64, 64, pool=True), # 32 -> 16
*block(64, 128), *block(128, 128, pool=True), # 16 -> 8
*block(128, 256), *block(256, 256, pool=True), # 8 -> 4
nn.AdaptiveAvgPool2d(1), nn.Flatten(),
nn.Dropout(0.2), nn.Linear(256, 10))

model = build()
print('parameters %d' % sum(p.numel() for p in model.parameters()))
x = torch.zeros(1, 3, 32, 32)
for i, layer in enumerate(model):
x = layer(x)
if isinstance(layer, (nn.MaxPool2d, nn.AdaptiveAvgPool2d, nn.Linear)):
print(' after %-20s %s' % (type(layer).__name__,
tuple(x.shape[1:])))
parameters 1148874
after MaxPool2d (64, 16, 16)
after MaxPool2d (128, 8, 8)
after MaxPool2d (256, 4, 4)
after AdaptiveAvgPool2d (256, 1, 1)
after Linear (10,)

The comparison

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes

train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, log=False, seed=0):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
if log:
print(' epoch %d done' % (epoch + 1))
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in val:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
import time
aug_tf = transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(), transforms.Normalize(MEAN, STD)])
aug_train = Subset(datasets.CIFAR10('data', train=True, download=True,
transform=aug_tf), range(4000))

def block(cin, cout, pool=False):
layers = [nn.Conv2d(cin, cout, 3, padding=1, bias=False),
nn.BatchNorm2d(cout), nn.ReLU()]
if pool:
layers.append(nn.MaxPool2d(2))
return layers

def full():
torch.manual_seed(0)
return nn.Sequential(
*block(3, 64), *block(64, 64, pool=True),
*block(64, 128), *block(128, 128, pool=True),
*block(128, 256), *block(256, 256, pool=True),
nn.AdaptiveAvgPool2d(1), nn.Flatten(),
nn.Dropout(0.2), nn.Linear(256, 10))

def dense():
torch.manual_seed(0)
return nn.Sequential(nn.Flatten(), nn.Linear(3072, 512), nn.ReLU(),
nn.Linear(512, 256), nn.ReLU(), nn.Linear(256, 10))

print('%-34s %10s %10s %9s' % ('', 'params', 'accuracy', 'time'))
for name, maker, train in [
('dense baseline', dense, None),
('conv net, no augmentation', full, None),
('conv net, crop and flip', full, aug_train)]:
m = maker()
start = time.time()
acc = fit(m, epochs=6, train=train)
print('%-34s %10d %10.4f %8.0fs'
% (name, sum(p.numel() for p in m.parameters()), acc,
time.time() - start))
params accuracy time
dense baseline 1707274 0.4300 5s
conv net, no augmentation 1148874 0.5890 194s
conv net, crop and flip 1148874 0.5505 190s

These numbers are from 8,000 images and ten epochs

The full CIFAR-10 training set is 50,000 images, and published results train for a hundred epochs or more. A model of this shape reaches well over 0.90 with the full set and a longer schedule, and the gap between what is here and what is published is training budget, not architecture. Everything on this page was chosen so it finishes while you are reading it.

The pattern worth memorising

  1. Convolution, then normalisation, then activation. Bias off when normalisation follows.
  2. Two or three of those, then halve the resolution.
  3. Double the channel count whenever you halve the resolution.
  4. Repeat until the map is around 4 by 4.
  5. Global average pool, then one linear layer to the classes.
  6. Random crop and horizontal flip on the training set, if the domain allows a flip.
  7. SGD with momentum 0.9, weight decay 5e-4, one cycle schedule.
  8. Then check whether depth is still helping, because week 6 is about what to do when it stops.

Day 7 takeaway

A stack of small convolutions with normalisation, halving the resolution and doubling the channels, ending in global average pooling, is the shape of almost every image model. It beats a dense network of the same size comfortably, and the augmentation is not optional.