Residual Networks and Transfer Learning

Week 6 of 18 · Vision · 7 days

Full curriculum
Week 06 · Vision

Residual Networks and Transfer Learning

Week 06 · Day 1 of 7

Why Deep Stacks Fail

An optimisation problem mistaken for overfitting, and its one-line fix

By 947 words

Week 5 ended with a stack that stopped improving as it got deeper. That is worth taking seriously, because the obvious explanation is wrong and the right one led to the single most useful architectural idea of the last decade.

The degradation problem

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes
train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, seed=0, report_train=False):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
model.eval()
def score(dl):
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
if report_train:
return score(DataLoader(train if train is not None else train_set,
batch_size=512)), score(val)
return score(val)
def plain(depth, norm):
torch.manual_seed(0)
def block(cin, cout):
layers = [nn.Conv2d(cin, cout, 3, padding=1, bias=not norm)]
if norm:
layers.append(nn.BatchNorm2d(cout))
return layers + [nn.ReLU()]
layers = block(3, 32)
for _ in range(depth):
layers += block(32, 32)
layers += [nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10)]
return nn.Sequential(*layers)

print('%18s %8s %12s %12s' % ('', 'depth', 'train acc', 'val acc'))
for norm in [False, True]:
for depth in [2, 6, 12]:
tr, va = fit(plain(depth, norm), epochs=4, report_train=True)
print('%18s %8d %12.4f %12.4f'
% ('with batchnorm' if norm else 'no normalisation',
depth, tr, va))
depth train acc val acc
no normalisation 2 0.2120 0.2220
no normalisation 6 0.1050 0.0975
no normalisation 12 0.1055 0.0990
with batchnorm 2 0.3465 0.3395
with batchnorm 6 0.3740 0.3675
with batchnorm 12 0.3827 0.3665

Read the top three rows first. Without normalisation, adding depth does not merely fail to help: the six and twelve layer stacks collapse to 0.10, which on ten classes is chance. They learned nothing at all, and they are strictly more powerful than the two layer version that learned something.

Look at the training accuracy, not the validation accuracy

If the deeper model were overfitting, its training accuracy would be higher and its validation accuracy lower. It is not: the deeper network is worse at fitting the data it was trained on, despite having strictly more capacity. Any function the shallow network computes, the deep one could compute by making its extra layers do nothing at all.

So it is not failing for lack of capacity. It is failing because gradient descent cannot find the good solution, which is an optimisation problem rather than a statistical one.

The bottom three rows show batch normalisation rescuing it, and that is the historical order. Normalisation made stacks of this depth trainable, networks then got deeper, and the same wall reappeared. Residual connections are the general fix, and they are what allow hundreds of layers rather than tens.

The residual block

Residual connection: Add the input of a block to its output, so the block computes a change to its input rather than a replacement for it. Setting the block's weights to zero then leaves the input untouched, which means doing nothing is the easiest thing for it to learn rather than the hardest.
import torch
from torch import nn

class ResidualBlock(nn.Module):
def __init__(self, channels):
super().__init__()
self.body = nn.Sequential(
nn.Conv2d(channels, channels, 3, padding=1, bias=False),
nn.BatchNorm2d(channels), nn.ReLU(),
nn.Conv2d(channels, channels, 3, padding=1, bias=False),
nn.BatchNorm2d(channels))
self.act = nn.ReLU()

def forward(self, x):
return self.act(x + self.body(x)) # the whole idea

torch.manual_seed(0)
block = ResidualBlock(16)
x = torch.randn(2, 16, 8, 8)
print('in ', tuple(x.shape), ' out', tuple(block(x).shape))

# with the body zeroed, the block is the identity
with torch.no_grad():
for prm in block.body.parameters():
prm.zero_()
print('body zeroed, output equals relu(input):',
bool(torch.allclose(block(x), torch.relu(x))))
in (2, 16, 8, 8) out (2, 16, 8, 8)
body zeroed, output equals relu(input): True

What it does to the gradient

import torch
from torch import nn

torch.manual_seed(0)

def gradient_at_input(residual, depth=20):
x = torch.randn(4, 16, requires_grad=True)
h = x
layers = [nn.Sequential(nn.Linear(16, 16), nn.ReLU())
for _ in range(depth)]
for layer in layers:
h = h + layer(h) if residual else layer(h)
h.sum().backward()
return x.grad.norm().item()

torch.manual_seed(0)
print('plain stack of 20 gradient norm at the input %.3e'
% gradient_at_input(False))
torch.manual_seed(0)
print('residual stack of 20 gradient norm at the input %.3e'
% gradient_at_input(True))
plain stack of 20 gradient norm at the input 7.884e-08
residual stack of 20 gradient norm at the input 4.735e+02

The addition gives the gradient a route back to the early layers that does not pass through any weights at all. Differentiating x + f(x) gives 1 + f'(x), and that 1 is a path along which the gradient arrives undiminished no matter how many blocks it has travelled through.

Day 1 takeaway

A deep plain stack can be worse than a shallow one on the training set, which makes it an optimisation failure rather than overfitting. Adding the input to the output gives the gradient an unobstructed path backwards and makes the identity the easy default, which is what allows networks with hundreds of layers.
Week 06 · Day 2 of 7

Building Residual Blocks

Shortcuts that change shape, where the normalisation goes, bottlenecks

By 731 words

A working residual network needs two more details: what to do when the shape changes, and where exactly to put the normalisation.

Blocks that change shape

import torch
from torch import nn

class Block(nn.Module):
def __init__(self, cin, cout, stride=1):
super().__init__()
self.body = nn.Sequential(
nn.Conv2d(cin, cout, 3, stride=stride, padding=1, bias=False),
nn.BatchNorm2d(cout), nn.ReLU(),
nn.Conv2d(cout, cout, 3, padding=1, bias=False),
nn.BatchNorm2d(cout))
# the shortcut has to match the body's output shape
if stride != 1 or cin != cout:
self.shortcut = nn.Sequential(
nn.Conv2d(cin, cout, 1, stride=stride, bias=False),
nn.BatchNorm2d(cout))
else:
self.shortcut = nn.Identity()
self.act = nn.ReLU()

def forward(self, x):
return self.act(self.shortcut(x) + self.body(x))

torch.manual_seed(0)
x = torch.randn(2, 32, 16, 16)
print('same shape ', tuple(Block(32, 32)(x).shape))
print('more channels', tuple(Block(32, 64)(x).shape))
print('and halved ', tuple(Block(32, 64, stride=2)(x).shape))

same = Block(32, 32)
print('\nshortcut when nothing changes:', type(same.shortcut).__name__)
print('it is free, which is the point')
same shape (2, 32, 16, 16)
more channels (2, 64, 16, 16)
and halved (2, 64, 8, 8)

shortcut when nothing changes: Identity
it is free, which is the point

A 1 by 1 convolution is not a trick, it is a channel mixer

It looks strange the first time. A 1 by 1 kernel sees one pixel, so it cannot detect any spatial pattern at all. What it does is take the vector of channel values at each position and apply a learned linear map to it, which is exactly what you need to turn 32 channels into 64 without touching the geometry. It appears constantly: in shortcuts here, in bottleneck blocks below, and as the pointwise half of a depthwise separable convolution.

Where the normalisation goes

import torch
from torch import nn

torch.manual_seed(0)

class PostNorm(nn.Module):
"""The original: add, then activate."""
def __init__(self, c):
super().__init__()
self.body = nn.Sequential(nn.Conv2d(c, c, 3, padding=1, bias=False),
nn.BatchNorm2d(c))
def forward(self, x):
return torch.relu(x + self.body(x))

class PreNorm(nn.Module):
"""Normalise and activate first, so the shortcut is untouched."""
def __init__(self, c):
super().__init__()
self.body = nn.Sequential(nn.BatchNorm2d(c), nn.ReLU(),
nn.Conv2d(c, c, 3, padding=1, bias=False))
def forward(self, x):
return x + self.body(x)

x = torch.randn(4, 16, 8, 8, requires_grad=True)
for name, cls in [('post-activation', PostNorm), ('pre-activation', PreNorm)]:
torch.manual_seed(0)
h = x
for _ in range(12):
h = cls(16)(h)
print('%-18s output std %.4f' % (name, h.std().item()))
post-activation output std 2.2141
pre-activation output std 1.5875

Pre-activation keeps the shortcut path completely clear: nothing at all happens to x on its way through. Post-activation puts a ReLU on it, which removes the negative half of the signal at every single block. The difference is small at twelve blocks and decisive at a hundred, and pre-activation is what modern architectures use.

Bottleneck blocks

import torch
from torch import nn

def basic(c):
return nn.Sequential(nn.Conv2d(c, c, 3, padding=1, bias=False),
nn.BatchNorm2d(c), nn.ReLU(),
nn.Conv2d(c, c, 3, padding=1, bias=False),
nn.BatchNorm2d(c))

def bottleneck(c, squeeze=4):
mid = c // squeeze
return nn.Sequential(nn.Conv2d(c, mid, 1, bias=False),
nn.BatchNorm2d(mid), nn.ReLU(),
nn.Conv2d(mid, mid, 3, padding=1, bias=False),
nn.BatchNorm2d(mid), nn.ReLU(),
nn.Conv2d(mid, c, 1, bias=False),
nn.BatchNorm2d(c))

for c in [64, 256]:
b = sum(p.numel() for p in basic(c).parameters())
n = sum(p.numel() for p in bottleneck(c).parameters())
print('%4d channels: basic %8d, bottleneck %8d, ratio %.1f'
% (c, b, n, b / n))
64 channels: basic 73984, bottleneck 4544, ratio 16.3
256 channels: basic 1180672, bottleneck 70400, ratio 16.8

The expensive part of a convolution is the 3 by 3 kernel over many channels. A bottleneck squeezes the channels down with a 1 by 1, does the spatial work cheaply, then expands back. At 256 channels it is several times smaller for the same job, which is how ResNet-50 has more layers than ResNet-34 and fewer operations.

Day 2 takeaway

When the shape changes, the shortcut needs a 1 by 1 convolution to match it; when it does not, the shortcut should be free. Put the normalisation and activation inside the body so the shortcut path stays clear. Bottleneck blocks do the spatial work at reduced channel count and cost several times less.
Week 06 · Day 3 of 7

A ResNet, Assembled

The full architecture against a plain stack of the same depth

By 820 words

A complete residual network, and the comparison against the plain stack of the same depth that day 1 showed failing.

ResNet, assembled

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes
train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, seed=0, report_train=False):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
model.eval()
def score(dl):
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
if report_train:
return score(DataLoader(train if train is not None else train_set,
batch_size=512)), score(val)
return score(val)
class Block(nn.Module):
def __init__(self, cin, cout, stride=1):
super().__init__()
self.body = nn.Sequential(
nn.Conv2d(cin, cout, 3, stride=stride, padding=1, bias=False),
nn.BatchNorm2d(cout), nn.ReLU(),
nn.Conv2d(cout, cout, 3, padding=1, bias=False),
nn.BatchNorm2d(cout))
self.shortcut = (nn.Sequential(
nn.Conv2d(cin, cout, 1, stride=stride, bias=False),
nn.BatchNorm2d(cout))
if stride != 1 or cin != cout else nn.Identity())
self.act = nn.ReLU()

def forward(self, x):
return self.act(self.shortcut(x) + self.body(x))

class ResNet(nn.Module):
def __init__(self, blocks=(2, 2, 2), width=32, n_classes=10):
super().__init__()
self.stem = nn.Sequential(
nn.Conv2d(3, width, 3, padding=1, bias=False),
nn.BatchNorm2d(width), nn.ReLU())
stages, cin = [], width
for stage, count in enumerate(blocks):
cout = width * 2 ** stage
for i in range(count):
stages.append(Block(cin, cout,
stride=2 if (i == 0 and stage > 0) else 1))
cin = cout
self.stages = nn.Sequential(*stages)
self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
nn.Linear(cin, n_classes))

def forward(self, x):
return self.head(self.stages(self.stem(x)))

torch.manual_seed(0)
model = ResNet()
print('parameters %d' % sum(p.numel() for p in model.parameters()))
print('blocks %d' % len(model.stages))
print('output %s' % (tuple(model(torch.zeros(2, 3, 32, 32)).shape),))
parameters 696618
blocks 6
output (2, 10)

Residual against plain, at the same depth

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes
train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, seed=0, report_train=False):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
model.eval()
def score(dl):
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
if report_train:
return score(DataLoader(train if train is not None else train_set,
batch_size=512)), score(val)
return score(val)
class Block(nn.Module):
def __init__(self, c, residual):
super().__init__()
self.residual = residual
self.body = nn.Sequential(
nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c),
nn.ReLU(),
nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c))
self.act = nn.ReLU()

def forward(self, x):
h = self.body(x)
return self.act(x + h) if self.residual else self.act(h)

def stack(n_blocks, residual):
torch.manual_seed(0)
layers = [nn.Conv2d(3, 32, 3, padding=1, bias=False),
nn.BatchNorm2d(32), nn.ReLU()]
layers += [Block(32, residual) for _ in range(n_blocks)]
layers += [nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10)]
return nn.Sequential(*layers)

print('%8s %14s %12s %12s' % ('blocks', 'kind', 'train acc', 'val acc'))
for n_blocks in [2, 8]:
for residual in [False, True]:
tr, va = fit(stack(n_blocks, residual), epochs=4,
report_train=True)
print('%8d %14s %12.4f %12.4f'
% (n_blocks, 'residual' if residual else 'plain', tr, va))
blocks kind train acc val acc
2 plain 0.3745 0.3555
2 residual 0.3755 0.3740
8 plain 0.3620 0.3585
8 residual 0.4057 0.3930

Day 3 takeaway

The residual version of the same depth trains where the plain one stalls, and the difference grows with depth. A stem, a few stages that double the channels and halve the resolution, and a global pooling head is the shape of every image classifier you will meet.
Week 06 · Day 4 of 7

Transfer Learning

Reusing somebody else's features, and the preprocessing that comes with them

By 1375 words

Nobody trains an image model from random weights unless they have to. Somebody has already spent thousands of GPU hours learning what edges, textures and object parts look like, and those features are not specific to the labels they were learned with.

What torchvision gives you

import torch
from torchvision import models

net = models.resnet18(weights=None) # architecture only, no download
print('resnet18 parameters %d' % sum(p.numel() for p in net.parameters()))
print('\ntop level structure:')
for name, child in net.named_children():
n = sum(p.numel() for p in child.parameters())
print(' %-10s %-22s %d' % (name, type(child).__name__, n))
print('\nthe classifier at the end:', net.fc)
resnet18 parameters 11689512

top level structure:
conv1 Conv2d 9408
bn1 BatchNorm2d 128
relu ReLU 0
maxpool MaxPool2d 0
layer1 Sequential 147968
layer2 Sequential 525568
layer3 Sequential 2099712
layer4 Sequential 8393728
avgpool AdaptiveAvgPool2d 0
fc Linear 513000

the classifier at the end: Linear(in_features=512, out_features=1000, bias=True)
# With a network connection, this is one line and downloads the weights:
from torchvision.models import resnet18, ResNet18_Weights

net = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)
preprocess = ResNet18_Weights.IMAGENET1K_V1.transforms()

# The transforms object matters as much as the weights. It carries the resize,
# the crop and the exact normalisation the model was trained with, and using
# different ones quietly costs several points of accuracy.

The preprocessing is part of the model

A pretrained network expects images normalised with the statistics of the dataset it was trained on, at the resolution it was trained at. Feeding it your own normalisation is not a small mismatch: every filter in the first layer was tuned for a particular input scale. This is the most common reason a pretrained model performs worse than expected, and it produces no error at all.

Transfer learning, without a download

Since this page has no pretrained weights to hand, it demonstrates the mechanics on a source task it can build itself: train on five CIFAR classes, then transfer to the other five. Every line is what you would write against ImageNet weights.

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes
train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, seed=0, report_train=False):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
model.eval()
def score(dl):
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
if report_train:
return score(DataLoader(train if train is not None else train_set,
batch_size=512)), score(val)
return score(val)
import copy

def backbone():
torch.manual_seed(0)
return nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1, bias=False), nn.BatchNorm2d(64),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(64, 128, 3, padding=1, bias=False), nn.BatchNorm2d(128),
nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten())

targets = torch.tensor(train_all.targets)
source_idx = (targets < 5).nonzero().flatten()[:8000]
source = torch.utils.data.Subset(train_all, source_idx.tolist())

torch.manual_seed(0)
base = backbone()
src_model = nn.Sequential(base, nn.Linear(128, 5))
loader = DataLoader(source, batch_size=128, shuffle=True)
opt = torch.optim.SGD(src_model.parameters(), lr=0.05, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()
for _ in range(6):
src_model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(src_model(xb), yb).backward()
opt.step()
print('source task trained on classes 0 to 4')
torch.save(base.state_dict(), 'cifar_backbone.pt')
print('backbone saved, %d parameters'
% sum(p.numel() for p in base.parameters()))
source task trained on classes 0 to 4
backbone saved, 93472 parameters

Freeze, then fine-tune

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes
train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, seed=0, report_train=False):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
model.eval()
def score(dl):
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
if report_train:
return score(DataLoader(train if train is not None else train_set,
batch_size=512)), score(val)
return score(val)
import copy

def backbone():
torch.manual_seed(0)
return nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1, bias=False), nn.BatchNorm2d(64),
nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(64, 128, 3, padding=1, bias=False), nn.BatchNorm2d(128),
nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten())

targets = torch.tensor(train_all.targets)
src_idx = (targets < 5).nonzero().flatten()[:8000].tolist()
source = torch.utils.data.Subset(train_all, src_idx)

# pretrain
torch.manual_seed(0)
pre = backbone()
m = nn.Sequential(pre, nn.Linear(128, 5))
opt = torch.optim.SGD(m.parameters(), lr=0.05, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()
for _ in range(6):
m.train()
for xb, yb in DataLoader(source, batch_size=128, shuffle=True):
opt.zero_grad()
loss_fn(m(xb), yb).backward()
opt.step()

# target task: classes 5 to 9, relabelled, with few examples
class Relabelled(torch.utils.data.Dataset):
def __init__(self, base_ds, idx):
self.base_ds, self.idx = base_ds, idx
def __len__(self):
return len(self.idx)
def __getitem__(self, i):
x, y = self.base_ds[self.idx[i]]
return x, y - 5

tgt_idx = (targets >= 5).nonzero().flatten()[:1500].tolist()
target_train = Relabelled(train_all, tgt_idx)
test_targets = torch.tensor(test_all.targets)
val_idx = (test_targets >= 5).nonzero().flatten()[:1500].tolist()
target_val = Relabelled(test_all, val_idx)

def train_target(base, freeze, epochs=8, lr=0.05):
for prm in base.parameters():
prm.requires_grad = not freeze
model = nn.Sequential(base, nn.Linear(128, 5))
torch.manual_seed(1)
opt = torch.optim.SGD([p for p in model.parameters()
if p.requires_grad], lr=lr, momentum=0.9)
for _ in range(epochs):
model.train()
for xb, yb in DataLoader(target_train, batch_size=64, shuffle=True):
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in DataLoader(target_val, batch_size=512):
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen

print('%-30s %12s' % ('', 'accuracy'))
print('%-30s %12.4f' % ('from scratch',
train_target(backbone(), freeze=False)))
print('%-30s %12.4f' % ('pretrained, frozen',
train_target(copy.deepcopy(pre), freeze=True)))
tuned = copy.deepcopy(pre)
train_target(tuned, freeze=True, epochs=6)
print('%-30s %12.4f' % ('then fine-tuned at lr/20',
train_target(tuned, freeze=False, epochs=6,
lr=0.0025)))
accuracy
from scratch 0.7000
pretrained, frozen 0.7133
then fine-tuned at lr/20 0.6727

Two honest observations. The pretrained backbone is barely ahead of training from scratch, and fine-tuning made it worse. Both follow from the same cause: the source task here is five CIFAR classes, which is not enough world knowledge for its features to beat what the target's own data can learn. Transfer pays when the source is vastly larger than the target, and a backbone you pretrained yourself on a few thousand images is not that. Day 5 states the condition explicitly.

Freeze the head first, then unfreeze at a much lower rate

A randomly initialised head produces large, meaningless gradients for the first few hundred steps. Let those reach an unfrozen backbone and they will destroy the features you came for before the head has learned anything worth propagating. Train the head with the backbone frozen, then unfreeze and continue at a tenth to a fiftieth of the rate.

Day 4 takeaway

Start from pretrained weights whenever you have fewer than tens of thousands of images, and use the preprocessing that came with them. Train the new head with the backbone frozen, then fine-tune everything at a much lower rate.
Week 06 · Day 5 of 7

Fine-tuning Properly

The frozen layer that is not frozen, and per-layer learning rates

By 662 words

Batch normalisation makes fine-tuning subtler than it looks, and this catches almost everybody once.

Frozen weights are not a frozen model

import torch
from torch import nn

torch.manual_seed(0)
bn = nn.BatchNorm2d(4)
for _ in range(30): # 'pretraining'
bn(torch.randn(16, 4, 8, 8) * 2 + 1)

before = bn.running_mean.clone()
for prm in bn.parameters():
prm.requires_grad = False # freeze the learnable parts

bn.train()
for _ in range(30): # 'fine-tuning' on new data
bn(torch.randn(16, 4, 8, 8) * 5 - 3)

print('running mean before fine-tuning', before.round(decimals=3).tolist())
print('running mean after ',
bn.running_mean.round(decimals=3).tolist())
print('\nrequires_grad=False did not stop it moving, because the running')
print('statistics are buffers, not parameters, and train() mode updates them.')
running mean before fine-tuning [0.9449999928474426, 0.9779999852180481, 0.9710000157356262, 0.9340000152587891]
running mean after [-2.8450000286102295, -2.888000011444092, -2.822999954223633, -2.871000051498413]

requires_grad=False did not stop it moving, because the running
statistics are buffers, not parameters, and train() mode updates them.

Freezing a backbone takes two things, not one

requires_grad = False stops the optimiser updating the weights. It does nothing about the running mean and variance, which are buffers updated by the forward pass whenever the module is in train() mode. Fine-tune on a small dataset and those statistics drift towards it, which changes the frozen features you were trying to preserve.

The fix is to put the normalisation layers into eval() mode explicitly, after calling model.train(), every epoch.

import torch
from torch import nn

model = nn.Sequential(nn.Conv2d(3, 8, 3), nn.BatchNorm2d(8), nn.ReLU(),
nn.Flatten(), nn.Linear(8 * 30 * 30, 10))

def freeze_norm(module):
for m in module.modules():
if isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d)):
m.eval()
for prm in m.parameters():
prm.requires_grad = False

model.train()
freeze_norm(model) # order matters: after train(), not before
print('module modes after model.train() then freeze_norm:')
for name, m in model.named_children():
print(' %-6s %-16s training=%s' % (name, type(m).__name__, m.training))
module modes after model.train() then freeze_norm:
0 Conv2d training=True
1 BatchNorm2d training=False
2 ReLU training=True
3 Flatten training=True
4 Linear training=True

Discriminative learning rates

import torch
from torch import nn

backbone = nn.Sequential(nn.Conv2d(3, 16, 3), nn.ReLU(),
nn.Conv2d(16, 32, 3), nn.ReLU())
head = nn.Linear(32, 10)

opt = torch.optim.SGD([
{'params': backbone[0].parameters(), 'lr': 1e-4}, # earliest, slowest
{'params': backbone[2].parameters(), 'lr': 5e-4},
{'params': head.parameters(), 'lr': 5e-3}, # newest, fastest
], momentum=0.9)

for i, group in enumerate(opt.param_groups):
print('group %d: %d tensors at lr %.4f'
% (i, len(group['params']), group['lr']))
print('\nearly layers hold general features and should barely move.')
print('late layers hold task-specific ones and can move a lot.')
group 0: 2 tensors at lr 0.0001
group 1: 2 tensors at lr 0.0005
group 2: 2 tensors at lr 0.0050

early layers hold general features and should barely move.
late layers hold task-specific ones and can move a lot.

When transfer does not help

SituationExpect
Target images resemble the sourceA large gain, even frozen
Target is a different domain, such as medical scansA smaller gain; fine-tune more layers
Target has millions of labelled imagesLittle or nothing; training from scratch catches up
Source model is small or trained on little dataNothing. The features are not better than yours
Target images are a very different sizeReduced gain; the receptive fields no longer match

That fourth row is worth stating plainly, because it is the one people get wrong. Transfer learning works because ImageNet is 1.2 million images across a thousand classes. A backbone pretrained on a small dataset of your own carries no special knowledge, and the Machine Learning course has a week 12 experiment where exactly that setup fails to beat training from scratch at any label count.

Day 5 takeaway

Freezing a backbone means both requires_grad = False and putting its normalisation layers in eval() mode after every model.train(). Use lower learning rates for earlier layers. And expect transfer to buy nothing when the source model was not trained on far more data than you have.
Week 06 · Day 6 of 7

Architectures Since ResNet

Separable convolutions, channel attention, and what actually won

By 561 words

Three architectural ideas that came after ResNet and are worth recognising, because you will meet all three inside models you download.

Depthwise separable convolution

import torch
from torch import nn

cin, cout, k = 64, 128, 3

standard = nn.Conv2d(cin, cout, k, padding=1, bias=False)
separable = nn.Sequential(
nn.Conv2d(cin, cin, k, padding=1, groups=cin, bias=False), # depthwise
nn.Conv2d(cin, cout, 1, bias=False)) # pointwise

a = sum(p.numel() for p in standard.parameters())
b = sum(p.numel() for p in separable.parameters())
print('standard %8d weights' % a)
print('separable %8d weights' % b)
print('ratio %8.1f to 1' % (a / b))

x = torch.randn(1, cin, 16, 16)
print('\nsame output shape:', tuple(standard(x).shape),
tuple(separable(x).shape))
standard 73728 weights
separable 8768 weights
ratio 8.4 to 1

same output shape: (1, 128, 16, 16) (1, 128, 16, 16)

groups=cin means each input channel gets its own filter and no mixing happens, which handles the spatial part. The 1 by 1 that follows does all the channel mixing. Splitting the job in two costs roughly a tenth of the weights, and it is the core of MobileNet and every architecture designed to run on a phone.

Squeeze and excitation

import torch
from torch import nn

class SqueezeExcite(nn.Module):
"""Let the network reweight its own channels, per image."""
def __init__(self, channels, squeeze=8):
super().__init__()
self.gate = nn.Sequential(
nn.AdaptiveAvgPool2d(1), nn.Flatten(),
nn.Linear(channels, channels // squeeze), nn.ReLU(),
nn.Linear(channels // squeeze, channels), nn.Sigmoid())

def forward(self, x):
weights = self.gate(x) # (batch, channels)
return x * weights[:, :, None, None]

torch.manual_seed(0)
se = SqueezeExcite(32)
x = torch.randn(2, 32, 8, 8)
out = se(x)
print('shape unchanged:', tuple(out.shape))
print('parameters added: %d' % sum(p.numel() for p in se.parameters()))
w = se.gate(x)
print('\nper-image channel weights, first 6 of image 0:')
print(' ', w[0, :6].round(decimals=3).tolist())
print('a channel weighted near 0 is switched off for this image only.')
shape unchanged: (2, 32, 8, 8)
parameters added: 292

per-image channel weights, first 6 of image 0:
[0.4269999861717224, 0.5830000042915344, 0.5649999976158142, 0.5950000286102295, 0.5289999842643738, 0.4410000145435333]
a channel weighted near 0 is switched off for this image only.

Where the ideas ended up

ArchitectureIdea it introducedStill used?
VGGStacks of 3 by 3 convolutionsThe kernel size, yes; the architecture, no
ResNetResidual connectionsEverywhere, including transformers
InceptionSeveral kernel sizes in parallelRarely
MobileNetDepthwise separable convolutionsYes, wherever compute is limited
DenseNetConcatenate all earlier feature mapsOccasionally
SENetChannel attentionYes, folded into other models
ConvNeXtA ResNet retuned with transformer training recipesYes, and it is the honest baseline for a vision transformer

The last row is the interesting one

When vision transformers appeared and beat convolutional networks, part of the gap turned out to be the training recipe rather than the architecture: better augmentation, longer schedules, AdamW, layer normalisation. ConvNeXt applied those to a plain ResNet and closed most of it. The lesson is one that recurs on this course: when a new architecture wins, check what else changed at the same time before you conclude the architecture was responsible.

Day 6 takeaway

Depthwise separable convolutions split spatial and channel mixing and cost about a tenth as much. Squeeze and excitation lets a network reweight its channels per image for very few parameters. And a fair comparison between architectures holds the training recipe constant, which is rarer than it should be.
Week 06 · Day 7 of 7

The Recipe

Everything measured together, and the order to reach for it

By 690 words

The week assembled: a residual network with a modern head, trained with augmentation, against everything it should beat.

The comparison

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)
tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize(MEAN, STD)])
train_all = datasets.CIFAR10('data', train=True, download=True, transform=tf)
test_all = datasets.CIFAR10('data', train=False, download=True, transform=tf)
CLASSES = train_all.classes
train_set = Subset(train_all, range(4000))
val_set = Subset(test_all, range(2000))
def fit(model, epochs=4, lr=0.05, train=None, seed=0, report_train=False):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=128, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=lr, total_steps=epochs * len(loader))
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
sched.step()
model.eval()
def score(dl):
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen
if report_train:
return score(DataLoader(train if train is not None else train_set,
batch_size=512)), score(val)
return score(val)
import time

aug_tf = transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(), transforms.Normalize(MEAN, STD)])
aug_train = Subset(datasets.CIFAR10('data', train=True, download=True,
transform=aug_tf), range(4000))

class Block(nn.Module):
def __init__(self, cin, cout, stride=1, residual=True):
super().__init__()
self.residual = residual
self.body = nn.Sequential(
nn.Conv2d(cin, cout, 3, stride=stride, padding=1, bias=False),
nn.BatchNorm2d(cout), nn.ReLU(),
nn.Conv2d(cout, cout, 3, padding=1, bias=False),
nn.BatchNorm2d(cout))
self.shortcut = (nn.Sequential(
nn.Conv2d(cin, cout, 1, stride=stride, bias=False),
nn.BatchNorm2d(cout))
if stride != 1 or cin != cout else nn.Identity())
self.act = nn.ReLU()

def forward(self, x):
h = self.body(x)
return self.act(self.shortcut(x) + h if self.residual
else h)

def resnet(residual=True, width=32):
torch.manual_seed(0)
layers = [nn.Conv2d(3, width, 3, padding=1, bias=False),
nn.BatchNorm2d(width), nn.ReLU()]
cin = width
for stage in range(3):
cout = width * 2 ** stage
for i in range(2):
layers.append(Block(cin, cout,
stride=2 if (i == 0 and stage > 0) else 1,
residual=residual))
cin = cout
layers += [nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(cin, 10)]
return nn.Sequential(*layers)

print('%-38s %10s %10s %8s' % ('', 'params', 'accuracy', 'time'))
for name, maker, train in [
('plain stack, same depth', lambda: resnet(residual=False), None),
('residual', resnet, None),
('residual + crop and flip', resnet, aug_train)]:
model = maker()
start = time.time()
acc = fit(model, epochs=6, train=train)
print('%-38s %10d %10.4f %7.0fs'
% (name, sum(p.numel() for p in model.parameters()), acc,
time.time() - start))
params accuracy time
plain stack, same depth 696618 0.4870 133s
residual 696618 0.5525 144s
residual + crop and flip 696618 0.4955 144s

What to reach for, in order

  1. A pretrained model, unless your images are unlike anything in ImageNet or you have millions of your own.
  2. The preprocessing that came with it, exactly.
  3. A frozen backbone and a new head first, then fine-tune at a much lower rate with the normalisation layers in eval mode.
  4. Augmentation: random crop and horizontal flip as a minimum, if a flip is valid in your domain.
  5. Residual connections in anything you build yourself deeper than about eight layers.
  6. Global average pooling rather than flatten, unless position carries the signal.
  7. A smaller model, checked against the larger one, because most published architectures are sized for datasets far larger than yours.

The numbers on this page are deliberately small

Six thousand images and ten epochs on a CPU. A residual network of this shape on the full fifty thousand images with a hundred epoch schedule reaches well over 0.90, and the published figures are higher still. Everything here was sized so it runs while you read it, and the ordering of the results is what to take away, not their magnitudes.

Day 7 takeaway

Residual connections turn depth from a liability into an asset by giving the gradient a clear path backwards. Transfer learning beats architecture choices whenever a suitable pretrained model exists, which is nearly always. Build neither from scratch until you have checked whether somebody has already done it.