Tensors and Automatic Differentiation

Week 1 of 18 · Foundations · 7 days

Full curriculum
Week 01 · Foundations

Tensors and Automatic Differentiation

Week 01 · Day 1 of 7

Tensors and Shapes

The array that remembers, and the broadcast that hides your bug

By 845 words

Deep learning has one idea in it. You write down a function with a very large number of adjustable numbers in it, you measure how wrong it is, and you work out which way to nudge every one of those numbers to make it less wrong. Everything else on this course is detail about how to do that quickly and without the whole thing falling over.

This week is about the machinery underneath. If you have used Keras or PyTorch before and found it mostly worked until it did not, this is the week that fixes that, because almost every confusing failure later on is a shape problem or a gradient problem, and both live here.

A tensor is an array that remembers what happened to it

Tensor: A multi-dimensional array of numbers, exactly like a NumPy array, with two additions: it can live on a GPU, and it can record the operations performed on it so that derivatives can be computed later.
import torch

scalar = torch.tensor(3.0)
vector = torch.tensor([1.0, 2.0, 3.0])
matrix = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
batch = torch.zeros(8, 3, 32, 32) # 8 images, 3 channels, 32 by 32

for name, t in [('scalar', scalar), ('vector', vector),
('matrix', matrix), ('batch', batch)]:
print('%-8s shape %-22s dim %d dtype %s'
% (name, str(tuple(t.shape)), t.dim(), t.dtype))
scalar shape () dim 0 dtype torch.float32
vector shape (3,) dim 1 dtype torch.float32
matrix shape (2, 2) dim 2 dtype torch.float32
batch shape (8, 3, 32, 32) dim 4 dtype torch.float32

Read the shape, always

The single most common error in this subject is a shape mismatch, and the single most useful debugging habit is printing tensor.shape before and after anything you are unsure about. By convention the first dimension is the batch, so a tensor of shape (8, 3, 32, 32) is eight images, each with three colour channels, each 32 pixels square. Nothing enforces that convention. It is a habit that every library and every paper follows.

The operations you will use constantly

import torch

a = torch.arange(6.0).reshape(2, 3)
b = torch.ones(3, 4)

print('a')
print(a)
print('a @ b shape', tuple((a @ b).shape))
print('a.T shape ', tuple(a.T.shape))
print('sum ', a.sum().item())
print('sum along rows ', a.sum(dim=0))
print('sum along columns', a.sum(dim=1))
print('mean of everything %.3f' % a.mean().item())
a
tensor([[0., 1., 2.],
[3., 4., 5.]])
a @ b shape (2, 4)
a.T shape (3, 2)
sum 15.0
sum along rows tensor([3., 5., 7.])
sum along columns tensor([ 3., 12.])
mean of everything 2.500

dim=0 means collapse the rows and keep the columns, which is the direction you want when you are summarising a batch. It is worth saying that out loud a few times, because getting it backwards produces code that runs, returns a number of the right type, and is silently wrong.

Broadcasting

Broadcasting: When two tensors have different shapes, the smaller one is stretched to match the larger, as long as each dimension is either equal or 1. No data is copied. It is what lets you add a bias vector to a whole batch in one expression.
import torch

batch = torch.ones(4, 3) # 4 rows, 3 features
bias = torch.tensor([10.0, 20.0, 30.0])
print('batch + bias')
print(batch + bias)

column = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
print('\nbatch * column')
print(batch * column)

try:
torch.ones(4, 3) + torch.ones(4)
except RuntimeError as e:
print('\nand when it cannot:')
print(' ', str(e)[:96])
batch + bias
tensor([[11., 21., 31.],
[11., 21., 31.],
[11., 21., 31.],
[11., 21., 31.]])

batch * column
tensor([[1., 1., 1.],
[2., 2., 2.],
[3., 3., 3.],
[4., 4., 4.]])

and when it cannot:
The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 1

The broadcast that does not fail is the one to fear

Adding a tensor of shape (4,) to one of shape (4, 1) does not raise. It broadcasts to (4, 4), and you now have sixteen numbers where you wanted four. This is the classic silent bug in a loss function: your targets come back as (batch,), your predictions as (batch, 1), and the loss you compute is the mean of every pairing rather than the mean of the matched pairs. It trains. It just trains on nonsense.

import torch

pred = torch.tensor([[0.9], [0.2], [0.7], [0.1]]) # (4, 1)
target = torch.tensor([1.0, 0.0, 1.0, 0.0]) # (4,)

wrong = ((pred - target) ** 2).mean()
right = ((pred.squeeze(1) - target) ** 2).mean()

print('difference shape when broadcast:', tuple((pred - target).shape))
print('loss computed carelessly %.4f' % wrong.item())
print('loss computed correctly %.4f' % right.item())
print('\nneither raised an error.')
difference shape when broadcast: (4, 4)
loss computed carelessly 0.3625
loss computed correctly 0.0375

neither raised an error.

Day 1 takeaway

A tensor is a NumPy array that can record its own history. Print shapes constantly, know that dim=0 collapses rows, and treat any mismatch between (n,) and (n, 1) as a bug waiting to happen, because broadcasting will hide it rather than report it.
Week 01 · Day 2 of 7

Automatic Differentiation

How one backward call fills in every gradient, and why it is cheap

By 579 words

Yesterday's tensors were just arrays. Today they start remembering, which is the one feature that separates a deep learning library from NumPy.

Derivatives, by hand, one last time

Take y = 3x² + 2x. Its derivative is 6x + 2, so at x = 2 the slope is 14. You can also get that number without doing any calculus, by nudging x a little and seeing how much y moves.

def f(x):
return 3 * x ** 2 + 2 * x

x = 2.0
print('%12s %18s' % ('step size', 'estimated slope'))
for h in [1.0, 0.1, 0.01, 0.0001, 1e-8, 1e-12]:
print('%12g %18.8f' % (h, (f(x + h) - f(x)) / h))
print('\nthe exact answer is 14')
step size estimated slope
1 17.00000000
0.1 14.30000000
0.01 14.03000000
0.0001 14.00030000
1e-08 13.99999974
1e-12 14.00124461

the exact answer is 14

Why nobody differentiates numerically

Watch the last two rows. As the step gets smaller the estimate gets better, and then it gets worse again, because subtracting two nearly equal floating point numbers destroys precision. Numerical differentiation is useful for checking a gradient you computed another way, which is exactly what day 5 uses it for, and useless for training a network with millions of parameters, since it would need one forward pass per parameter.

What autograd actually does

Automatic differentiation: Not numerical approximation and not symbolic algebra. Every operation you perform is recorded along with the rule for its own derivative, and the chain rule is applied backwards through that record. The cost of computing all the gradients is roughly the cost of computing the output once.
import torch

x = torch.tensor(2.0, requires_grad=True)
y = 3 * x ** 2 + 2 * x

print('y =', y.item())
print('y knows it was computed:', y.grad_fn)

y.backward()
print('dy/dx at x=2 is', x.grad.item())
y = 16.0
y knows it was computed: <AddBackward0 object at 0x000001B30968E590>
dy/dx at x=2 is 14.0
import torch

# A chain of operations, and the gradient through all of it.
x = torch.tensor(1.5, requires_grad=True)
w = torch.tensor(-0.8, requires_grad=True)
b = torch.tensor(0.3, requires_grad=True)

z = w * x + b
a = torch.sigmoid(z)
loss = (a - 1.0) ** 2

print('z %.4f' % z.item())
print('a %.4f' % a.item())
print('loss %.4f' % loss.item())

loss.backward()
print('\ndloss/dw %.6f' % w.grad.item())
print('dloss/db %.6f' % b.grad.item())
print('dloss/dx %.6f' % x.grad.item())
z -0.9000
a 0.2891
loss 0.5054

dloss/dw -0.438301
dloss/db -0.292201
dloss/dx 0.233761

Three gradients from one backward call, and none of them were derived by you. That is the whole trick. The forward pass builds a graph of operations; backward() walks it in reverse, multiplying local derivatives as it goes.

Checking it against the calculus

import torch

# The same thing worked out by hand:
# loss = (sigmoid(wx + b) - 1)^2
# dloss/da = 2(a - 1)
# da/dz = a(1 - a)
# dz/dw = x
x_v, w_v, b_v = 1.5, -0.8, 0.3
z_v = w_v * x_v + b_v
a_v = 1 / (1 + pow(2.718281828459045, -z_v))
by_hand = 2 * (a_v - 1) * a_v * (1 - a_v) * x_v

x = torch.tensor(x_v, requires_grad=True)
w = torch.tensor(w_v, requires_grad=True)
b = torch.tensor(b_v, requires_grad=True)
((torch.sigmoid(w * x + b) - 1.0) ** 2).backward()

print('by hand %.8f' % by_hand)
print('autograd %.8f' % w.grad.item())
print('agree:', abs(by_hand - w.grad.item()) < 1e-7)
by hand -0.43830102
autograd -0.43830103
agree: True

Day 2 takeaway

requires_grad=True makes a tensor record its history. A single backward() call fills in .grad on every tensor that contributed to the result, at roughly the cost of one forward pass. That efficiency is the reason networks with millions of parameters are trainable at all.
Week 01 · Day 3 of 7

How the Graph Behaves

Accumulating gradients, no_grad, detach, and the memory leak in your loop

By 611 words

The graph is built while the forward pass runs, which has consequences worth understanding before they bite: gradients accumulate rather than replace, some operations need to be kept out of the graph, and the graph is thrown away after you use it.

Gradients add up, and that is deliberate

import torch

w = torch.tensor(1.0, requires_grad=True)

for step in range(3):
loss = (w * 2) ** 2
loss.backward()
print('after backward %d, w.grad = %.1f' % (step + 1, w.grad.item()))

print('\nthe gradient is 8 every time, but it is being added on')
after backward 1, w.grad = 8.0
after backward 2, w.grad = 16.0
after backward 3, w.grad = 24.0

the gradient is 8 every time, but it is being added on

This is the bug everybody writes once

Leave optimizer.zero_grad() out of your training loop and the gradients from every previous batch stay in .grad, growing without limit. The loss usually explodes within a few dozen steps, though on a small learning rate it can instead just train badly and quietly. Call zero_grad() every step, before backward().

The behaviour is not a design mistake. It is what lets you accumulate gradients over several small batches and then step once, which is how people train large models on hardware that cannot hold a large batch.

import torch

w = torch.tensor(1.0, requires_grad=True)
for step in range(3):
if w.grad is not None:
w.grad.zero_()
loss = (w * 2) ** 2
loss.backward()
print('step %d, w.grad = %.1f' % (step + 1, w.grad.item()))
step 1, w.grad = 8.0
step 2, w.grad = 8.0
step 3, w.grad = 8.0

Keeping things out of the graph

import torch

w = torch.tensor(2.0, requires_grad=True)

tracked = w * 3
print('tracked requires grad:', tracked.requires_grad)

with torch.no_grad():
untracked = w * 3
print('inside no_grad :', untracked.requires_grad)

detached = (w * 3).detach()
print('after detach :', detached.requires_grad)

print('\nuse no_grad for evaluation, and for the parameter update')
print('itself, which must not become part of the next graph.')
tracked requires grad: True
inside no_grad : False
after detach : False

use no_grad for evaluation, and for the parameter update
itself, which must not become part of the next graph.
SituationUseBecause
Scoring a validation setwith torch.no_grad():No gradients are needed, and memory falls sharply
Updating parameters by handwith torch.no_grad():The update is bookkeeping, not part of the model
Using a value as a constant target.detach()Stops the gradient flowing back into whatever produced it
Reading a number for printing.item()Returns a plain Python float and holds no graph
Storing a running loss.item() or .detach()Keeping the tensor keeps the whole graph alive, which is the usual cause of a slow memory leak in a training loop

The graph is freed once you use it

import torch

w = torch.tensor(1.0, requires_grad=True)
loss = (w * 2) ** 2
loss.backward()

try:
loss.backward()
except RuntimeError as e:
print('second backward on the same graph:')
print(' ', str(e).split('.')[0])

w.grad.zero_()
loss = (w * 2) ** 2
loss.backward(retain_graph=True)
loss.backward()
print('\nwith retain_graph=True it works, and the gradient doubled:',
w.grad.item())
second backward on the same graph:
Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed)

with retain_graph=True it works, and the gradient doubled: 16.0

You will almost never need retain_graph=True. When you think you do, the usual cause is that you built the graph outside the loop and are trying to reuse it, and the fix is to build it inside.

Day 3 takeaway

Gradients accumulate, so zero them every step. Wrap evaluation and manual parameter updates in torch.no_grad(). Store .item() rather than tensors when you are logging, because a stored tensor keeps its entire graph alive.
Week 01 · Day 4 of 7

The Training Loop

Five steps written by hand, then the same five with the library

By 787 words

Enough machinery. Here is a complete training loop, written out in full with no library helpers at all, so that every later convenience is recognisable as a shortcut for something you have already done by hand.

Some data with a known answer

import torch

torch.manual_seed(0)
n = 200
X = torch.rand(n, 2) * 4 - 2 # two features in [-2, 2]
true_w = torch.tensor([2.0, -3.0])
true_b = 0.5
y = X @ true_w + true_b + torch.randn(n) * 0.3 # with noise

print('X', tuple(X.shape), ' y', tuple(y.shape))
print('the answer we are hoping to recover: w =', true_w.tolist(),
' b =', true_b)
X (200, 2) y (200,)
the answer we are hoping to recover: w = [2.0, -3.0] b = 0.5

The loop

import torch

torch.manual_seed(0)
n = 200
X = torch.rand(n, 2) * 4 - 2
true_w = torch.tensor([2.0, -3.0])
y = X @ true_w + 0.5 + torch.randn(n) * 0.3

w = torch.zeros(2, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
lr = 0.1

for epoch in range(101):
pred = X @ w + b # forward
loss = ((pred - y) ** 2).mean() # how wrong

if w.grad is not None:
w.grad.zero_()
b.grad.zero_()
loss.backward() # gradients

with torch.no_grad(): # the update
w -= lr * w.grad
b -= lr * b.grad

if epoch % 20 == 0:
print('epoch %3d loss %.4f w %s b %.3f'
% (epoch, loss.item(),
[round(v, 3) for v in w.tolist()], b.item()))
epoch 0 loss 15.7814 w [0.461, -0.731] b 0.015
epoch 20 loss 0.1056 w [2.011, -2.992] b 0.503
epoch 40 loss 0.1045 w [2.023, -3.004] b 0.523
epoch 60 loss 0.1045 w [2.023, -3.004] b 0.524
epoch 80 loss 0.1045 w [2.023, -3.004] b 0.524
epoch 100 loss 0.1045 w [2.023, -3.004] b 0.524

Five lines, and every one of them has a name

Forward pass, loss, zero the gradients, backward pass, update. Every training loop you will ever write is those five steps. A hundred-line loop from a research repository is those five steps plus logging, checkpointing, learning rate scheduling and mixed precision, and if you cannot find the five steps inside it, you are reading the wrong part of the file.

The same thing with the library doing the bookkeeping

import torch

torch.manual_seed(0)
n = 200
X = torch.rand(n, 2) * 4 - 2
y = X @ torch.tensor([2.0, -3.0]) + 0.5 + torch.randn(n) * 0.3

model = torch.nn.Linear(2, 1)
loss_fn = torch.nn.MSELoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)

for epoch in range(101):
pred = model(X).squeeze(1)
loss = loss_fn(pred, y)
opt.zero_grad()
loss.backward()
opt.step()
if epoch % 20 == 0:
print('epoch %3d loss %.4f' % (epoch, loss.item()))

print('\nlearned w', [round(v, 3) for v in
model.weight.detach().squeeze(0).tolist()])
print('learned b %.3f' % model.bias.item())
epoch 0 loss 15.2988
epoch 20 loss 0.1051
epoch 40 loss 0.1045
epoch 60 loss 0.1045
epoch 80 loss 0.1045
epoch 100 loss 0.1045

learned w [2.023, -3.004]
learned b 0.524

Identical arithmetic, less typing. nn.Linear holds the weight and bias for you, MSELoss is the mean of the squared differences, and opt.step() is the subtraction you wrote by hand. Note squeeze(1): nn.Linear(2, 1) returns shape (n, 1) and the targets are (n,), which is yesterday's silent broadcast waiting to happen.

What the learning rate does

import torch

def run(lr, steps=60):
torch.manual_seed(0)
X = torch.rand(200, 2) * 4 - 2
y = X @ torch.tensor([2.0, -3.0]) + 0.5 + torch.randn(200) * 0.3
w = torch.zeros(2, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
for _ in range(steps):
loss = ((X @ w + b - y) ** 2).mean()
if w.grad is not None:
w.grad.zero_(); b.grad.zero_()
loss.backward()
with torch.no_grad():
w -= lr * w.grad
b -= lr * b.grad
return loss.item()

print('%10s %16s' % ('rate', 'loss after 60'))
for lr in [0.001, 0.01, 0.1, 0.5, 1.0, 1.05]:
value = run(lr)
print('%10.3f %16s'
% (lr, 'diverged' if value != value or value > 1e6
else '%.4f' % value))
rate loss after 60
0.001 11.9438
0.010 1.0795
0.100 0.1045
0.500 0.1045
1.000 diverged
1.050 diverged

There is a cliff, and it is close to the best value

Too small and it crawls. Too large and it does not merely go slowly, it diverges to infinity or to nan. The best rate is usually within a factor of two or three of the one that breaks it, which is why the learning rate is the first hyperparameter to tune and why week 3 spends a day on choosing it. If your loss becomes nan, lower the learning rate before you change anything else.

Day 4 takeaway

Forward, loss, zero, backward, step. Write it by hand once and the library version stops being magic. Watch for the shape mismatch between a model's (n, 1) output and an (n,) target, and treat a nan loss as a learning rate that is too high until proven otherwise.
Week 01 · Day 5 of 7

Your First Network

Two layers on XOR, gradient checking, and why initialisation decides everything

By 722 words

A network is the same loop with more parameters and a nonlinearity between the layers. Building one from scratch takes about twenty lines, and it is worth doing once because it makes the failure modes in week 3 obvious rather than mysterious.

A two-layer network, entirely by hand

import torch

torch.manual_seed(0)

# The classic problem a single layer cannot solve.
X = torch.tensor([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
y = torch.tensor([[0.0], [1.0], [1.0], [0.0]]) # exclusive or

W1 = torch.randn(2, 8, requires_grad=True)
b1 = torch.zeros(8, requires_grad=True)
W2 = torch.randn(8, 1, requires_grad=True)
b2 = torch.zeros(1, requires_grad=True)
params = [W1, b1, W2, b2]

for step in range(3001):
hidden = torch.relu(X @ W1 + b1)
out = torch.sigmoid(hidden @ W2 + b2)
loss = -(y * out.log() + (1 - y) * (1 - out).log()).mean()

for prm in params:
if prm.grad is not None:
prm.grad.zero_()
loss.backward()
with torch.no_grad():
for prm in params:
prm -= 0.5 * prm.grad

if step % 750 == 0:
print('step %4d loss %.4f' % (step, loss.item()))

print('\npredictions', [round(v, 3) for v in out.detach().squeeze(1).tolist()])
print('targets ', [0.0, 1.0, 1.0, 0.0])
step 0 loss 0.7838
step 750 loss 0.0027
step 1500 loss 0.0011
step 2250 loss 0.0007
step 3000 loss 0.0005

predictions [0.001, 1.0, 0.999, 0.0]
targets [0.0, 1.0, 1.0, 0.0]

A single layer cannot solve that, because no straight line separates the two corners where the answer is 1 from the two where it is 0. The hidden layer builds a new representation in which the problem does become separable, and that is the entire justification for depth.

Proving the gradients are right

Gradient checking: Compare each analytic gradient against a numerical estimate from a tiny finite difference. If they agree to several decimal places, your backward pass is correct. It is slow, so you do it once on a small example, not during training.
import torch

torch.manual_seed(0)
X = torch.randn(5, 3)
y = torch.randn(5, 1)
W = torch.randn(3, 1, requires_grad=True)

def loss_of(weight):
return ((X @ weight - y) ** 2).mean()

loss_of(W).backward()
analytic = W.grad.clone().squeeze(1)

h = 1e-4
numeric = torch.zeros(3)
with torch.no_grad():
for i in range(3):
up, down = W.clone(), W.clone()
up[i] += h
down[i] -= h
numeric[i] = (loss_of(up) - loss_of(down)) / (2 * h)

print('%10s %14s %14s' % ('parameter', 'analytic', 'numerical'))
for i in range(3):
print('%10d %14.6f %14.6f' % (i, analytic[i], numeric[i]))
print('\nlargest disagreement %.2e'
% (analytic - numeric).abs().max().item())
parameter analytic numerical
0 -2.298965 -2.298355
1 2.040464 2.040863
2 2.095707 2.096891

largest disagreement 1.18e-03

Use the two-sided difference

Note (f(x+h) - f(x-h)) / 2h rather than the one-sided version from day 2. The two-sided form has an error proportional to h² instead of h, so it is far more accurate for the same step size. When you write a custom layer or a custom loss, this check is how you find out whether your derivation is right before you spend a day wondering why the model will not learn.

Why initialisation is not a detail

import torch

torch.manual_seed(0)
x = torch.randn(512, 100)

print('%18s %12s %12s' % ('initial scale', 'std after 1', 'std after 8'))
for scale, label in [(1.0, 'randn'), (0.1, 'randn * 0.1'),
((2.0 / 100) ** 0.5, 'He (sqrt(2/n))')]:
h = x
first = None
for layer in range(8):
W = torch.randn(100, 100) * scale
h = torch.relu(h @ W)
if layer == 0:
first = h.std().item()
print('%18s %12.4f %12.6f' % (label, first, h.std().item()))
initial scale std after 1 std after 8
randn 5.8124 7294867.000000
randn * 0.1 0.5768 0.051808
He (sqrt(2/n)) 0.8290 0.653480

With weights that are slightly too small the signal dies away to nothing after a few layers, and a signal of zero has a gradient of zero. Slightly too large and it grows without limit. The He initialisation in the last row is scaled specifically so that the variance of the activations survives a ReLU, which is why it is the default for every modern network and why PyTorch already applies something like it for you.

Day 5 takeaway

A network is the same five-step loop over more parameters, with a nonlinearity between layers to make depth worth having. Check any gradient you derive yourself against a two-sided finite difference. And remember that initialisation scale decides whether a deep stack has any signal left at the far end.
Week 01 · Day 6 of 7

Devices, Datasets and Loaders

Getting real data into batches without writing a loop

By 727 words

Two practical matters before the first real model: where tensors live, and how you get data into them without writing a loop over a list.

Devices

import torch

print('CUDA available:', torch.cuda.is_available())
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print('this course runs on:', device)

x = torch.ones(3)
x = x.to(device)
print('tensor device:', x.device)

model = torch.nn.Linear(3, 1).to(device)
print('model runs:', model(x).item() is not None)
CUDA available: False
this course runs on: cpu
tensor device: cpu
model runs: True

Everything has to be on the same device

A model on the GPU and a batch on the CPU produces Expected all tensors to be on the same device, which is the second most common error in PyTorch after shape mismatches. Write device once at the top, call .to(device) on the model once, and on each batch inside the loop. Everything on this course is written so it runs on a CPU, because the point is to understand the mechanics rather than to wait.

Datasets and loaders

Dataset and DataLoader: A Dataset answers two questions: how many examples are there, and what is example number i. A DataLoader wraps one and handles batching, shuffling and parallel loading. Separating them means the same loader code works for a tensor in memory and for a directory of ten million photographs.
import torch
from torch.utils.data import TensorDataset, DataLoader

torch.manual_seed(0)
X = torch.randn(10, 3)
y = torch.arange(10.0)

loader = DataLoader(TensorDataset(X, y), batch_size=4, shuffle=True)
print('10 examples, batch size 4, so %d batches' % len(loader))
for i, (xb, yb) in enumerate(loader):
print('batch %d x %s y %s'
% (i, tuple(xb.shape), [int(v) for v in yb.tolist()]))
10 examples, batch size 4, so 3 batches
batch 0 x (4, 3) y [7, 9, 2, 6]
batch 1 x (4, 3) y [1, 8, 4, 0]
batch 2 x (2, 3) y [3, 5]

The final batch has two rows rather than four. That is normal, and it is why you should average a metric over examples rather than over batches: the last batch is smaller and averaging batch means gives it too much weight.

Writing your own Dataset

import torch
from torch.utils.data import Dataset, DataLoader

class SquaresDataset(Dataset):
"""The two methods every Dataset must have."""

def __init__(self, n):
self.n = n

def __len__(self):
return self.n

def __getitem__(self, i):
x = torch.tensor([float(i)])
return x, x ** 2

loader = DataLoader(SquaresDataset(7), batch_size=3)
for xb, yb in loader:
print('x', [int(v) for v in xb.squeeze(1).tolist()],
' y', [int(v) for v in yb.squeeze(1).tolist()])
x [0, 1, 2] y [0, 1, 4]
x [3, 4, 5] y [9, 16, 25]
x [6] y [36]

__getitem__ runs once per example, so keep it cheap

It is the right place to read one image from disk and apply the transformations for it. It is the wrong place to load a CSV, open a database connection or compute statistics over the whole dataset, all of which belong in __init__. A slow __getitem__ is the usual reason a GPU sits idle while the loader struggles to keep up, and num_workers exists to run several copies of it in parallel.

Real data, for the first time

import torch
from torchvision import datasets, transforms

to_tensor = transforms.ToTensor()
train = datasets.MNIST('data', train=True, download=True,
transform=to_tensor)
test = datasets.MNIST('data', train=False, download=True,
transform=to_tensor)

print('training images', len(train))
print('test images ', len(test))
image, label = train[0]
print('one example: shape %s, label %d' % (tuple(image.shape), label))
print('pixel range %.1f to %.1f' % (image.min(), image.max()))
training images 60000
test images 10000
one example: shape (1, 28, 28), label 5
pixel range 0.0 to 1.0
import torch
from torchvision import datasets, transforms

train = datasets.MNIST('data', train=True, download=True,
transform=transforms.ToTensor())
image, label = train[0]

ramp = ' .:-=+*#%@'
print('label:', label)
pixels = image.squeeze(0)
for row in pixels[::1]:
print(''.join(ramp[min(9, int(v * 9.999))] for v in row.tolist()))
label: 5





=+*.*@@=
..-**@@@@@%*@@#:
.@@@@@@@@@@---:.
%@@@@@##@@
-*=@@% .*
*@-
+@#
#@:
.@%*=
-@@@=
.#@@+.
-@@#
@@@:
.+#@@%
.+%@@@@#
=%@@@@#-
:%@@@@#-
*%@@@@#-
:*%@@@@@+
+@@@%++


Day 6 takeaway

Put the model and every batch on the same device, decided once at the top of the script. A Dataset answers how many and which one; a DataLoader handles batching and shuffling. Keep __getitem__ cheap, because it runs once per example per epoch.
Week 01 · Day 7 of 7

MNIST End to End

A complete model, its confusion matrix, and the baseline it has to beat

By 1191 words

Everything from this week, applied to real digits, with a proper training loop and an honest evaluation.

The model

import torch

model = torch.nn.Sequential(
torch.nn.Flatten(), # (batch, 1, 28, 28) -> (batch, 784)
torch.nn.Linear(784, 128),
torch.nn.ReLU(),
torch.nn.Linear(128, 10), # ten digits, raw scores
)
print(model)
total = sum(prm.numel() for prm in model.parameters())
print('\nparameters: %d' % total)
for name, prm in model.named_parameters():
print(' %-10s %-16s %d' % (name, tuple(prm.shape), prm.numel()))
Sequential(
(0): Flatten(start_dim=1, end_dim=-1)
(1): Linear(in_features=784, out_features=128, bias=True)
(2): ReLU()
(3): Linear(in_features=128, out_features=10, bias=True)
)

parameters: 101770
1.weight (128, 784) 100352
1.bias (128,) 128
3.weight (10, 128) 1280
3.bias (10,) 10
Logits: the raw, unbounded scores a classifier produces before they are turned into probabilities. nn.CrossEntropyLoss expects logits, not probabilities, and applies the softmax internally for numerical stability. Adding a softmax yourself before the loss is a common mistake that trains a worse model rather than raising an error.

Training it

import time
import torch
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

torch.manual_seed(0)
tf = transforms.ToTensor()
train_full = datasets.MNIST('data', train=True, download=True, transform=tf)
test_full = datasets.MNIST('data', train=False, download=True, transform=tf)

# A subset, so every example on this page finishes while you watch it.
train = Subset(train_full, range(8000))
test = Subset(test_full, range(2000))
train_loader = DataLoader(train, batch_size=64, shuffle=True)
test_loader = DataLoader(test, batch_size=256)

model = torch.nn.Sequential(torch.nn.Flatten(),
torch.nn.Linear(784, 128), torch.nn.ReLU(),
torch.nn.Linear(128, 10))
loss_fn = torch.nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)

def accuracy(loader):
model.eval()
right = seen = 0
with torch.no_grad():
for xb, yb in loader:
right += (model(xb).argmax(1) == yb).sum().item()
seen += yb.numel()
return right / seen

start = time.time()
for epoch in range(1, 6):
model.train()
running = 0.0
for xb, yb in train_loader:
opt.zero_grad()
loss = loss_fn(model(xb), yb)
loss.backward()
opt.step()
running += loss.item() * yb.numel()
print('epoch %d train loss %.4f test accuracy %.4f'
% (epoch, running / len(train), accuracy(test_loader)))
print('\n%.1f seconds on the CPU' % (time.time() - start))
epoch 1 train loss 1.0575 test accuracy 0.8400
epoch 2 train loss 0.4176 test accuracy 0.8650
epoch 3 train loss 0.3342 test accuracy 0.8650
epoch 4 train loss 0.2965 test accuracy 0.8790
epoch 5 train loss 0.2691 test accuracy 0.8785

5.3 seconds on the CPU

What it gets wrong

import torch
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

torch.manual_seed(0)
tf = transforms.ToTensor()
train = Subset(datasets.MNIST('data', train=True, download=True,
transform=tf), range(8000))
test = Subset(datasets.MNIST('data', train=False, download=True,
transform=tf), range(2000))

model = torch.nn.Sequential(torch.nn.Flatten(),
torch.nn.Linear(784, 128), torch.nn.ReLU(),
torch.nn.Linear(128, 10))
opt = torch.optim.SGD(model.parameters(), lr=0.1)
loss_fn = torch.nn.CrossEntropyLoss()
for _ in range(5):
for xb, yb in DataLoader(train, batch_size=64, shuffle=True):
opt.zero_grad()
loss_fn(model(xb), yb).backward()
opt.step()

model.eval()
confusion = torch.zeros(10, 10, dtype=torch.long)
with torch.no_grad():
for xb, yb in DataLoader(test, batch_size=256):
for t, p in zip(yb, model(xb).argmax(1)):
confusion[t, p] += 1

print(' ' + ''.join('%5d' % i for i in range(10)))
for i, row in enumerate(confusion):
print('%2d ' % i + ''.join('%5d' % v for v in row.tolist()))

print('\nconfusions of three or more:')
for i in range(10):
for j in range(10):
if i != j and confusion[i, j] >= 3:
print(' a %d read as a %d, %d times' % (i, j, confusion[i, j]))
0 1 2 3 4 5 6 7 8 9
0 171 0 0 1 0 1 2 0 0 0
1 0 228 0 1 1 0 2 0 2 0
2 2 3 186 2 3 0 5 7 9 2
3 0 0 2 174 0 16 1 8 5 1
4 1 0 1 0 201 0 5 1 1 7
5 6 0 1 7 4 150 1 3 5 2
6 3 1 2 0 7 4 159 1 1 0
7 1 5 9 0 3 0 0 178 1 8
8 2 2 2 7 4 7 3 5 157 3
9 1 0 0 5 15 2 0 9 2 160

confusions of three or more:
a 2 read as a 1, 3 times
a 2 read as a 4, 3 times
a 2 read as a 6, 5 times
a 2 read as a 7, 7 times
a 2 read as a 8, 9 times
a 3 read as a 5, 16 times
a 3 read as a 7, 8 times
a 3 read as a 8, 5 times
a 4 read as a 6, 5 times
a 4 read as a 9, 7 times
a 5 read as a 0, 6 times
... (20 more lines)

The mistakes are the ones a person would make. Threes and fives are the worst pair by a distance, and they differ only in whether the top stroke closes to the left or the right. Nines and fours go both ways once the loop is closed, twos and sevens share a diagonal, and eights collect confusions from everything with two curves in it. If your confusion matrix looks random rather than sympathetic, suspect a bug in how labels are lining up with images before you suspect the model.

A baseline that is not a network

import torch
from torchvision import datasets, transforms
from sklearn.linear_model import LogisticRegression
import time

tf = transforms.ToTensor()
train = datasets.MNIST('data', train=True, download=True, transform=tf)
test = datasets.MNIST('data', train=False, download=True, transform=tf)

Xtr = train.data[:8000].reshape(8000, -1).numpy() / 255.0
ytr = train.targets[:8000].numpy()
Xte = test.data[:2000].reshape(2000, -1).numpy() / 255.0
yte = test.targets[:2000].numpy()

start = time.time()
lr = LogisticRegression(max_iter=200).fit(Xtr, ytr)
print('logistic regression %.4f in %.1fs'
% (lr.score(Xte, yte), time.time() - start))
logistic regression 0.8700 in 4.0s

Keep score honestly from the first week

A one-hidden-layer network on 8,000 digits beats plain logistic regression, but not by as much as the effort suggests it should. That gap is the thing the rest of this course is about: convolution in week 5, which knows that pixels have neighbours, and depth done properly in week 6, which is what turns a few points into a lot of points. Fit the cheap model every time, so you know what your complicated one is actually buying.

Before you move on

  1. Print the shape of anything you are unsure about, especially between a model's output and the target.
  2. Call zero_grad() every step.
  3. Put the model in train() mode for training and eval() mode for evaluation. It changes nothing today and changes a great deal from week 4.
  4. Wrap evaluation in torch.no_grad().
  5. Log loss.item(), never the loss tensor.
  6. Fit a non-network baseline and write the number down.

Day 7 takeaway

You have now written a training loop by hand, checked a gradient numerically, built a network out of parts, and trained one on real images against an honest baseline. Everything in the next seventeen weeks is a variation on this loop, and when something later behaves strangely, the answer is almost always in the five steps you wrote out on day 4.