Regularisation and Generalisation

Week 4 of 18 · Foundations · 7 days

Full curriculum
Week 04 · Foundations

Regularisation and Generalisation

Week 04 · Day 1 of 7

Measuring Overfitting

The gap that matters, and the technique that beats all the others

By 754 words

A network with more parameters than examples can fit the training set exactly, including whatever is accidental about it. Every technique this week is a way of making that harder without making the useful fitting harder too.

The problem, measured

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
model = build(dropout=0.0)
print('parameters %d, training examples %d'
% (sum(p.numel() for p in model.parameters()), len(train_set)))
train_acc, val_acc = run(model)
print('\ntrain accuracy %.4f' % train_acc)
print('val accuracy %.4f' % val_acc)
print('gap %.4f' % (train_acc - val_acc))
parameters 669706, training examples 1000

train accuracy 1.0000
val accuracy 0.8920
gap 0.1080

The gap is the thing to watch, not the validation number

A validation accuracy that is lower than training accuracy is normal. A validation accuracy far lower, with training at or near 1.000, means the model has capacity it is spending on memorisation. That is a fixable situation and this week is the toolbox. A model where both numbers are low has the opposite problem and none of these techniques will help it.

More data is the technique that always works

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
print('%10s %12s %12s %10s' % ('examples', 'train acc', 'val acc', 'gap'))
for n in [250, 1000, 4000, 16000]:
subset = Subset(train_all, range(n))
tr, va = run(build(), epochs=12, train=subset)
print('%10d %12.4f %12.4f %10.4f' % (n, tr, va, tr - va))
examples train acc val acc gap
250 1.0000 0.7920 0.2080
1000 1.0000 0.8875 0.1125
4000 1.0000 0.9605 0.0395
16000 0.9996 0.9990 0.0006

Nothing else on this page competes with that column. Every regularisation technique is an attempt to buy some of the same effect when you cannot get more data, and they all buy less of it than the data itself would. It is worth remembering when a week of tuning is on the table and a day of labelling is also on the table.

Day 1 takeaway

Overfitting is a gap between training and validation performance with training near perfect. Measure the gap, not just the validation number. And know that more data beats every technique in this week, so check whether you can get some before you spend days on the alternatives.
Week 04 · Day 2 of 7

Weight Decay

Smaller weights, smoother functions, and why AdamW exists

By 1059 words

Weight decay is the oldest idea here and still one of the two that matter. It says: of all the models that fit the data, prefer the one with smaller weights.

What it does to the weights

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
print('%14s %12s %12s %14s' % ('weight decay', 'train acc', 'val acc',
'weight norm'))
for wd in [0.0, 1e-4, 1e-3, 1e-2, 1e-1]:
model = build()
tr, va = run(model, wd=wd)
norm = torch.cat([p.flatten() for n, p in model.named_parameters()
if 'weight' in n]).norm().item()
print('%14.4f %12.4f %12.4f %14.2f' % (wd, tr, va, norm))
weight decay train acc val acc weight norm
0.0000 1.0000 0.8920 20.49
0.0001 1.0000 0.8920 20.04
0.0010 1.0000 0.8880 16.48
0.0100 1.0000 0.8865 6.23
0.1000 0.8900 0.7965 3.78

The weight norm falls steadily, which is the layer doing exactly what it was asked. The validation accuracy does not improve at all, and at 0.1 the penalty is strong enough to stop the model fitting the training data in the first place. Weight decay is worth having and it is not, on this problem, where the improvement is going to come from. Day 7 puts all of them in one table for that reason.

Weight decay: Add a penalty proportional to the sum of the squared weights, which is the same thing as multiplying every weight by a number slightly below one at each step. Large weights mean a function that can change sharply between nearby inputs, and a function that changes sharply is one that can pass through every training point individually.

Why AdamW exists

import torch
from torch import nn

torch.manual_seed(0)

def final_weight(optimiser_cls, wd):
w = nn.Parameter(torch.tensor([1.0]))
opt = optimiser_cls([w], lr=0.1, weight_decay=wd)
for _ in range(50):
opt.zero_grad()
# a tiny, constant gradient, so the decay dominates
(w * 1e-4).sum().backward()
opt.step()
return w.item()

print('%-10s %14s %14s' % ('', 'no decay', 'decay 0.1'))
for name, cls in [('SGD', torch.optim.SGD), ('Adam', torch.optim.Adam),
('AdamW', torch.optim.AdamW)]:
print('%-10s %14.6f %14.6f'
% (name, final_weight(cls, 0.0), final_weight(cls, 0.1)))
no decay decay 0.1
SGD 0.999499 0.604611
Adam -3.999498 -0.005444
AdamW -3.999498 -3.344537

Adam and AdamW were given the same decay setting and ended in completely different places, differing by a factor of several hundred. They are not two implementations of one idea.

In Adam, weight_decay is not weight decay

Adam adds the penalty to the gradient and then divides by the running average of squared gradients, so a parameter with large gradients gets its decay divided away and a parameter with small ones gets it amplified. The amount of regularisation a weight receives ends up depending on its gradient history, which is not what anybody wanted. AdamW applies the decay directly to the weight, outside the adaptive step, which is what the W stands for. Use AdamW.

Not everything should be decayed

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
model = build()
decay, no_decay = [], []
for name, prm in model.named_parameters():
(no_decay if prm.dim() == 1 else decay).append(name)
print('decayed :', decay)
print('not decayed:', no_decay)

groups = [
{'params': [p for n, p in model.named_parameters() if p.dim() > 1],
'weight_decay': 1e-2},
{'params': [p for n, p in model.named_parameters() if p.dim() == 1],
'weight_decay': 0.0},
]
opt = torch.optim.AdamW(groups, lr=1e-3)
print('\nparameter groups:', len(opt.param_groups))
for g in opt.param_groups:
print(' %d tensors, weight decay %.3f'
% (len(g['params']), g['weight_decay']))
decayed : ['1.weight', '4.weight', '7.weight']
not decayed: ['1.bias', '4.bias', '7.bias']

parameter groups: 2
3 tensors, weight decay 0.010
3 tensors, weight decay 0.000

Biases and normalisation parameters are one number per channel rather than a whole matrix, they cannot overfit on their own, and shrinking them towards zero actively fights what the normalisation layer is for. Every serious training script separates them, and it is four lines.

Day 2 takeaway

Weight decay prefers smaller weights and therefore smoother functions. Use AdamW rather than Adam, because Adam's weight_decay is entangled with the adaptive step. Exclude biases and normalisation parameters from the decay.
Week 04 · Day 3 of 7

Dropout

Deleting half the network on purpose, and the mode that must be set

By 1036 words

Dropout is the other technique that matters, and it is stranger than it looks: during training it deletes a random half of the network, and at prediction time it uses all of it.

What it does

import torch
from torch import nn

torch.manual_seed(0)
layer = nn.Dropout(p=0.5)
x = torch.ones(2, 8)

layer.train()
print('training mode:')
print(layer(x))

layer.eval()
print('\neval mode:')
print(layer(x))
training mode:
tensor([[0., 0., 2., 0., 0., 0., 2., 2.],
[0., 2., 2., 2., 2., 0., 2., 2.]])

eval mode:
tensor([[1., 1., 1., 1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1., 1., 1., 1.]])

Notice the surviving units are 2.0, not 1.0

Dropping half the units halves the expected sum reaching the next layer, so PyTorch divides the survivors by the keep probability to put the expectation back. That is why nothing needs to change at evaluation time: the scaling has already been done during training. Frameworks that scale at test time instead are doing the same arithmetic in the other order, and mixing the two conventions is a classic bug when porting a model between libraries.

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
print('%10s %12s %12s %10s' % ('dropout', 'train acc', 'val acc', 'gap'))
for rate in [0.0, 0.2, 0.5, 0.8]:
tr, va = run(build(dropout=rate))
print('%10.1f %12.4f %12.4f %10.4f' % (rate, tr, va, tr - va))
dropout train acc val acc gap
0.0 1.0000 0.8920 0.1080
0.2 1.0000 0.8830 0.1170
0.5 0.9990 0.8895 0.1095
0.8 0.1170 0.1125 0.0045

And on this problem it does nothing. Validation accuracy is flat from 0.0 to 0.5 and the gap does not close, because a fully connected network on a thousand MNIST digits is not overfitting in the way dropout fixes. At 0.8 the model cannot learn at all, which is the same warning weight decay gave at 0.1: enough of any regulariser stops the fitting rather than the overfitting.

Dropout earns its reputation elsewhere

It was introduced for large fully connected layers on datasets where those layers held most of the parameters, and that is still where it helps most: the classifier head of a large network, and the feed forward blocks of a transformer in week 10. Between convolutional layers it is largely superseded, and on a small image model it is beaten comfortably by augmentation, as day 4 shows. Keep it in the toolbox and stop assuming it is doing something.

The mistake that costs a point of accuracy silently

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
torch.manual_seed(0)
model = build(dropout=0.5)
run(model, epochs=10)

xb, yb = next(iter(DataLoader(val_set, batch_size=512)))

model.train() # wrong mode for evaluation
with torch.no_grad():
a = (model(xb).argmax(1) == yb).float().mean().item()
b = (model(xb).argmax(1) == yb).float().mean().item()
model.eval()
with torch.no_grad():
c = (model(xb).argmax(1) == yb).float().mean().item()
d = (model(xb).argmax(1) == yb).float().mean().item()

print('train() mode, two runs: %.4f then %.4f' % (a, b))
print('eval() mode, two runs : %.4f then %.4f' % (c, d))
print('\nin the wrong mode the answer is random and worse.')
train() mode, two runs: 0.8379 then 0.8242
eval() mode, two runs : 0.8848 then 0.8848

in the wrong mode the answer is random and worse.

Where to put it

PositionVerdict
After a hidden activationThe standard place
On the inputRarely; augmentation is a better tool
Before the final classifierCommon and effective
Between convolutional layersUsually replaced by other methods; see Dropout2d in week 5
After the output layerNever

Day 3 takeaway

Dropout removes a random fraction of units during training and scales the survivors so the expectation is unchanged. It does nothing at evaluation time, which is why model.eval() is not optional. Start at 0.2 to 0.5 after hidden layers, and then check whether it did anything, because on this problem it did not.
Week 04 · Day 4 of 7

Augmentation

Manufacturing data, and the transformations that quietly lie

By 927 words

Augmentation makes more data out of the data you have, by applying changes that a human would say do not alter the answer. It is the technique with the largest effect on images and the one most easily got wrong.

Seeing it

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
torch.manual_seed(0)
aug = transforms.Compose([
transforms.RandomAffine(degrees=12, translate=(0.12, 0.12),
scale=(0.9, 1.1)),
transforms.ToTensor(),
])
raw = datasets.MNIST('data', train=True, download=True)
image, label = raw[0]

ramp = ' .:-=+*#%@'
def show(t):
for row in t[::2]:
print(''.join(ramp[min(9, int(v * 9.999))] for v in row[::1].tolist()))

print('original, label %d:' % label)
show(transforms.ToTensor()(image).squeeze(0))
for i in range(2):
print('\naugmented %d:' % (i + 1))
show(aug(image).squeeze(0))
original, label 5:



..-**@@@@@%*@@#:
%@@@@@##@@
*@-
#@:
-@@@=
-@@#
.+#@@%
=%@@@@#-
*%@@@@#-
+@@@%++


augmented 1:


..-**@@@@@%*@#:
-*=@@% .*
+@#
.@%*=
.#@@+.
@@@
.+%@@@@#
%@@@@#-
+@@%++




augmented 2:


=++*.*@@=
..-**@@@@@@---:.
%@@@@@@# .*
**@-
#@:
-@@@=
-@@@#
.++#@@%
.+@@@@#-
:%@@@@@
*%@@@@@@+
+@@

What it is worth

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
torch.manual_seed(0)
aug_tf = transforms.Compose([
transforms.RandomAffine(degrees=12, translate=(0.12, 0.12),
scale=(0.9, 1.1)),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
])
aug_train = Subset(datasets.MNIST('data', train=True, download=True,
transform=aug_tf), range(1000))

print('%-22s %12s %12s %10s' % ('', 'train acc', 'val acc', 'gap'))
tr, va = run(build(), epochs=30)
print('%-22s %12.4f %12.4f %10.4f' % ('no augmentation', tr, va, tr - va))
tr, va = run(build(), epochs=30, train=aug_train)
print('%-22s %12.4f %12.4f %10.4f' % ('with augmentation', tr, va, tr - va))
train acc val acc gap
no augmentation 1.0000 0.8920 0.1080
with augmentation 0.9250 0.9170 0.0080

Every augmentation is a claim, and some are false

Rotating a digit by twelve degrees preserves its identity. Rotating it by a hundred and eighty turns a 6 into a 9, and flipping it horizontally turns a 2 into something that is not a digit at all. The same applies everywhere: flipping a chest X-ray produces a patient with their heart on the wrong side, and adjusting the brightness of a medical scan changes the measurement the scan exists to record.

Choose augmentations from the domain, not from a list of what the library offers, and look at twenty augmented examples before you train on a million of them.

Augmentation belongs to the training set only

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
torch.manual_seed(0)
hard = transforms.Compose([
transforms.RandomAffine(degrees=25, translate=(0.2, 0.2)),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
])
aug_val = Subset(datasets.MNIST('data', train=True, download=True,
transform=hard), range(10000, 12000))

model = build()
run(model, epochs=20)
model.eval()

for name, ds in [('clean validation', val_set),
('augmented validation', aug_val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in DataLoader(ds, batch_size=512):
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
print('%-22s %.4f' % (name, right / seen))
print('\naugmenting the validation set measures a different, harder task.')
clean validation 0.8915
augmented validation 0.2245

augmenting the validation set measures a different, harder task.

Day 4 takeaway

Augmentation is the strongest regulariser available for images because it adds information about what should not change the answer. Apply it to training data only, choose the transformations from the domain, and look at the output before you trust it.
Week 04 · Day 5 of 7

Batch and Layer Normalisation

Two layers, one of which couples your prediction to its batch

By 700 words

Normalisation layers were introduced to make deep networks trainable, and they turned out to regularise as well. They are also the single most common source of a model that works in training and fails in production.

Batch normalisation

Batch normalisation: Normalise each feature to zero mean and unit variance across the batch, then apply a learned scale and shift. During training the statistics come from the current batch; at evaluation they come from a running average accumulated during training.
import torch
from torch import nn

torch.manual_seed(0)
bn = nn.BatchNorm1d(4)
x = torch.randn(16, 4) * 5 + 3

bn.train()
out = bn(x)
print('input mean %.3f std %.3f' % (x.mean(), x.std()))
print('output mean %.3f std %.3f' % (out.mean(), out.std()))
print('\nrunning estimates after one batch:')
print(' mean', bn.running_mean.round(decimals=3).tolist())
print(' var ', bn.running_var.round(decimals=3).tolist())

for _ in range(50):
bn(torch.randn(16, 4) * 5 + 3)
print('\nafter 50 batches:')
print(' mean', bn.running_mean.round(decimals=3).tolist())
print(' var ', bn.running_var.round(decimals=3).tolist())
input mean 3.463 std 5.252
output mean 0.000 std 1.008

running estimates after one batch:
mean [0.26600000262260437, 0.3149999976158142, 0.46299999952316284, 0.3400000035762787]
var [2.6700000762939453, 4.567999839782715, 2.753000020980835, 4.96999979019165]

after 50 batches:
mean [2.880000114440918, 2.7360000610351562, 2.437000036239624, 3.0239999294281006]
var [25.974000930786133, 25.28499984741211, 23.336999893188477, 21.656999588012695]
import torch
from torch import nn

torch.manual_seed(0)
bn = nn.BatchNorm1d(4)
for _ in range(50):
bn(torch.randn(16, 4) * 5 + 3)

one = torch.randn(1, 4) * 5 + 3
bn.eval()
print('one row in eval mode works fine:', tuple(bn(one).shape))

bn.train()
try:
bn(one)
except ValueError as e:
print('\none row in train mode:')
print(' ', str(e)[:88])
print('\na batch of one has zero variance, so there is nothing to')
print('normalise by. Small batches make the statistics noisy long')
print('before they make them impossible.')
one row in eval mode works fine: (1, 4)

one row in train mode:
Expected more than 1 value per channel when training, got input size torch.Size([1, 4])

a batch of one has zero variance, so there is nothing to
normalise by. Small batches make the statistics noisy long
before they make them impossible.

The layer that behaves differently in the two modes

Batch normalisation makes a prediction for one row depend on the other rows in its batch during training, and not during evaluation. Three consequences follow. Batch size becomes a hyperparameter that changes results. A model evaluated in train() mode gives different answers depending on what else is in the batch. And if your training distribution differs from your serving distribution, the running statistics are wrong in production in a way nothing in validation will reveal.

Layer normalisation, which has none of those problems

import torch
from torch import nn

torch.manual_seed(0)
x = torch.randn(8, 6) * 3 + 1

bn = nn.BatchNorm1d(6)
ln = nn.LayerNorm(6)

print('batch norm normalises down the columns:')
print(' column means after: ', bn(x).mean(dim=0).round(decimals=3).tolist())
print('layer norm normalises across each row:')
print(' row means after: ', ln(x).mean(dim=1).round(decimals=3).tolist())

ln.eval()
print('\nlayer norm on a single row in eval mode:',
tuple(ln(x[:1]).shape))
print('it has no running statistics and no train/eval difference at all,')
print('which is why every transformer in week 10 uses it.')
batch norm normalises down the columns:
column means after: [0.0, -0.0, -0.0, -0.0, 0.0, 0.0]
layer norm normalises across each row:
row means after: [0.0, 0.0, -0.0, -0.0, -0.0, -0.0, -0.0, -0.0]

layer norm on a single row in eval mode: (1, 6)
it has no running statistics and no train/eval difference at all,
which is why every transformer in week 10 uses it.
Batch normLayer norm
Normalises overThe batch, per featureThe features, per example
Depends on batch sizeYes, stronglyNo
Train and eval differYesNo
Works with batch size 1NoYes
Usual homeConvolutional networksTransformers and recurrent networks

Day 5 takeaway

Batch normalisation stabilises training and regularises, at the cost of coupling every prediction to its batch and behaving differently in the two modes. Layer normalisation avoids all of that and is what sequence models use. Either way, model.eval() is what makes the difference visible.
Week 04 · Day 6 of 7

Label Smoothing, Early Stopping and Mixup

Cheap techniques, and why loss turns before accuracy does

By 1318 words

Three smaller techniques that are cheap enough to be worth knowing, and one that is not a regulariser at all but is the most reliable way to stop at the right moment.

Label smoothing

Label smoothing: Instead of asking the model for probability 1.0 on the correct class, ask for 0.9 and spread the remaining 0.1 across the others. It removes the incentive to push logits to infinity, which is what a confidently overfitted model does.
import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
print('%16s %12s %12s' % ('label smoothing', 'val acc', 'mean max prob'))
for eps in [0.0, 0.05, 0.1, 0.2]:
model = build()
tr, va = run(model, label_smoothing=eps)
model.eval()
with torch.no_grad():
xb, yb = next(iter(DataLoader(val_set, batch_size=512)))
confidence = model(xb).softmax(1).max(1).values.mean().item()
print('%16.2f %12.4f %12.4f' % (eps, va, confidence))
label smoothing val acc mean max prob
0.00 0.8920 0.9704
0.05 0.9110 0.8155
0.10 0.9145 0.7521
0.20 0.9160 0.6575

The confidence falls sharply, which is the point of it, and on this problem the accuracy rises by more than two points as well. That second part is a bonus rather than the purpose. A model that is 97 percent sure of everything, including the answers it got wrong, is useless the moment you need to decide which predictions to trust or where to set a threshold.

Early stopping is still the best value

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
import copy
torch.manual_seed(0)
model = build()
loader = DataLoader(train_set, batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()

best = (float('inf'), 0, None)
print('%6s %12s %12s' % ('epoch', 'val loss', 'val acc'))
for epoch in range(1, 41):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
total = right = seen = 0
with torch.no_grad():
for xb, yb in val:
out = model(xb)
total += loss_fn(out, yb).item() * yb.numel()
right += (out.argmax(1) == yb).sum().item()
seen += yb.numel()
if total / seen < best[0]:
best = (total / seen, epoch, copy.deepcopy(model.state_dict()))
if epoch % 8 == 0:
print('%6d %12.4f %12.4f' % (epoch, total / seen, right / seen))

print('\nbest validation loss %.4f at epoch %d' % (best[0], best[1]))
print('validation loss at epoch 40 was worse, and accuracy was not')
epoch val loss val acc
8 0.5574 0.8920
16 0.6122 0.8905
24 0.6373 0.8915
32 0.6542 0.8910
40 0.6668 0.8915

best validation loss 0.4824 at epoch 3
validation loss at epoch 40 was worse, and accuracy was not

Validation loss turns before validation accuracy does

Watch the two columns. The loss reaches its minimum and starts climbing while accuracy is still flat or drifting up. That is the model becoming more confident about the answers it already had right and more confidently wrong about the rest. If you early stop on accuracy you will train for longer and ship a worse-calibrated model, so stop on loss unless accuracy is genuinely the only thing you care about.

Mixup

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
torch.manual_seed(0)
xb, yb = next(iter(DataLoader(train_set, batch_size=8, shuffle=True)))

lam = 0.7
perm = torch.randperm(xb.size(0))
mixed_x = lam * xb + (1 - lam) * xb[perm]

print('mixing example 0 (a %d) with example %d (a %d) at %.1f / %.1f'
% (yb[0], perm[0], yb[perm[0]], lam, 1 - lam))
print('the loss becomes a weighted sum of the two targets:')
print(' loss = %.1f * CE(out, y_a) + %.1f * CE(out, y_b)'
% (lam, 1 - lam))

ramp = ' .:-=+*#%@'
img = mixed_x[0, 0]
img = (img - img.min()) / (img.max() - img.min())
for row in img[::2]:
print(''.join(ramp[min(9, int(v * 9.999))] for v in row.tolist()))
mixing example 0 (a 9) with example 3 (a 1) at 0.7 / 0.3
the loss becomes a weighted sum of the two targets:
loss = 0.7 * CE(out, y_a) + 0.3 * CE(out, y_b)


:-.
.::.
=******%%@*
+*- .%@#.
+** :*@#.
-****@@%*+
.--:+*+
::: +*=
::: +*-
::: +*-
+*=
.+:

It looks absurd and it works, particularly on images. The usual explanation is that it forces the model to behave linearly between training examples rather than carving the space into confident regions with sharp boundaries. It costs three lines and is worth trying on any image problem where you are short of data.

Day 6 takeaway

Label smoothing costs nothing and fixes overconfidence. Early stopping on validation loss rather than accuracy is still the best value of anything here. Mixup is three lines and often worth a point on image tasks.
Week 04 · Day 7 of 7

Putting It Together

Every technique measured on its own, and how to tell the opposite problem

By 888 words

All of it, added one at a time to the same overfitting model, so the table says what each was actually worth.

The ablation

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
torch.manual_seed(0)
aug_tf = transforms.Compose([
transforms.RandomAffine(degrees=12, translate=(0.12, 0.12),
scale=(0.9, 1.1)),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
])
aug_train = Subset(datasets.MNIST('data', train=True, download=True,
transform=aug_tf), range(1000))

print('%-30s %11s %10s %8s' % ('', 'train acc', 'val acc', 'gap'))
rows = [
('nothing', dict()),
('+ weight decay 1e-3', dict(wd=1e-3)),
('+ dropout 0.2', dict(wd=1e-3, dropout=0.2)),
('+ label smoothing 0.1', dict(wd=1e-3, dropout=0.2,
label_smoothing=0.1)),
('+ augmentation', dict(wd=1e-3, dropout=0.2, label_smoothing=0.1,
aug=True)),
]
for label, cfg in rows:
model = build(dropout=cfg.get('dropout', 0.0))
tr, va = run(model, epochs=30, wd=cfg.get('wd', 0.0),
label_smoothing=cfg.get('label_smoothing', 0.0),
train=aug_train if cfg.get('aug') else None)
print('%-30s %11.4f %10.4f %8.4f' % (label, tr, va, tr - va))
train acc val acc gap
nothing 1.0000 0.8920 0.1080
+ weight decay 1e-3 1.0000 0.8880 0.1120
+ dropout 0.2 1.0000 0.8920 0.1080
+ label smoothing 0.1 1.0000 0.9215 0.0785
+ augmentation 0.9310 0.9415 -0.0105

Underfitting looks nothing like this

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

tf = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))])
train_all = datasets.MNIST('data', train=True, download=True, transform=tf)

# A small training set, so overfitting is easy to produce and easy to fix.
train_set = Subset(train_all, range(1000))
val_set = Subset(train_all, range(10000, 12000))
def build(dropout=0.0, hidden=512, seed=0):
torch.manual_seed(seed)
return nn.Sequential(
nn.Flatten(),
nn.Linear(784, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout),
nn.Linear(hidden, 10))

def run(model, epochs=30, wd=0.0, label_smoothing=0.0, train=None,
seed=0, lr=0.05):
torch.manual_seed(seed)
loader = DataLoader(train if train is not None else train_set,
batch_size=64, shuffle=True)
val = DataLoader(val_set, batch_size=512)
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9,
weight_decay=wd)
loss_fn = nn.CrossEntropyLoss(label_smoothing=label_smoothing)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()
model.eval()
plain = nn.CrossEntropyLoss()
out = []
for name, dl in [('train', DataLoader(train if train is not None
else train_set, batch_size=512)),
('val', val)]:
right = seen = 0
with torch.no_grad():
for xb, yb in dl:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
out.append(right / seen)
return out
print('%-26s %11s %10s' % ('', 'train acc', 'val acc'))
for label, hidden, wd, drop in [('too much regularisation', 512, 0.5, 0.8),
('model far too small', 4, 0.0, 0.0),
('sensible', 512, 1e-3, 0.2)]:
tr, va = run(build(dropout=drop, hidden=hidden), epochs=30, wd=wd)
print('%-26s %11.4f %10.4f' % (label, tr, va))
train acc val acc
too much regularisation 0.1170 0.1045
model far too small 0.7010 0.5480
sensible 1.0000 0.8920

If training accuracy is low, stop regularising

Every technique this week makes fitting the training data harder. Applied to a model that is already struggling to fit it, they make things worse and the symptom looks similar from a distance: a disappointing validation number. The distinguishing test takes one glance. Training accuracy near 1.0 with validation far below means regularise. Both numbers low means the opposite: more capacity, better features, longer training, less regularisation.

The order to reach for them

  1. More data, if there is any way to get it.
  2. Augmentation, if the domain admits any label preserving transformation.
  3. Early stopping on validation loss. Free, and it should already be in your loop.
  4. Weight decay, around 1e-4 to 1e-2, excluding biases and normalisation parameters.
  5. Dropout 0.2 to 0.5 on the hidden layers nearest the output.
  6. A smaller model, which is underrated and is often what the first four are compensating for.
  7. Label smoothing and mixup, cheap and usually worth a small amount.
  8. Ensembling, which always works and costs you a model for every point of it.

Day 7 takeaway

Diagnose before you treat: a large train-validation gap calls for this week's toolbox, and two low numbers call for the opposite. Augmentation and more data dominate everything else on images. Add one technique at a time and keep the table, because two of these usually do nothing on any given problem and you cannot guess which two.