The Transformer

Week 9 of 18 · Transformers · 7 days

Full curriculum
Week 09 · Transformers

The Transformer

Week 09 · Day 1 of 7

The Three Shapes

Encoder, decoder, and both, and why decoder only won

By 676 words

A transformer is a stack of the block week 8 finished with, plus an embedding at the front and a linear layer at the back. Everything else you have heard about them is a variation on where the blocks are and what they are allowed to look at.

The three shapes

ShapeBlocks seeTrained toExamples
Encoder onlyThe whole sequence, both directionsFill in masked positionsBERT, sentence embeddings, classifiers
Decoder onlyEarlier positions onlyPredict the next tokenGPT, Llama, almost every current model
Encoder and decoderEncoder sees all of the input; decoder sees the input and its own pastProduce one sequence from anotherThe original transformer, T5, translation

Decoder only won, and it is worth knowing why

Not because it is more expressive. An encoder can see both directions, which is strictly more information for understanding a sentence. Decoder only won because next-token prediction is a training task you can run on any text at all, with no labels, and because one model trained that way turns out to do translation, summarising and question answering without being built for any of them. The architecture is not the innovation. The training objective is.

An encoder, assembled

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
class Block(nn.Module):
"""Pre-normalisation transformer block, as week 8 day 7 built it."""
def __init__(self, width, heads, expansion=4, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(width, heads, dropout=dropout,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.ff = nn.Sequential(
nn.Linear(width, width * expansion), nn.GELU(),
nn.Linear(width * expansion, width), nn.Dropout(dropout))

def forward(self, x, mask=None):
h = self.norm1(x)
a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
return x + self.ff(self.norm2(x))
class Encoder(nn.Module):
def __init__(self, vocab=VOCAB, width=64, heads=4, blocks=2,
max_len=16):
super().__init__()
self.tok = nn.Embedding(vocab, width)
self.pos = nn.Embedding(max_len, width)
self.blocks = nn.ModuleList([Block(width, heads)
for _ in range(blocks)])
self.norm = nn.LayerNorm(width)

def forward(self, x):
h = self.tok(x) + self.pos(torch.arange(x.shape[1]))
for block in self.blocks:
h = block(h)
return self.norm(h)

torch.manual_seed(0)
enc = Encoder()
x = torch.randint(1, 10, (2, 6))
print('input ', tuple(x.shape))
print('output ', tuple(enc(x).shape), ' one vector per position')
print('parameters %d' % sum(p.numel() for p in enc.parameters()))
print('\nwhere the parameters are:')
for name, child in enc.named_children():
print(' %-8s %d' % (name, sum(p.numel() for p in child.parameters())))
input (2, 6)
output (2, 6, 64) one vector per position
parameters 101824

where the parameters are:
tok 704
pos 1024
blocks 99968
norm 128

Most of the weight is in the blocks, and inside a block most of it is in the feed forward network rather than the attention. That surprises people. Attention decides what to read; the feed forward layers are where most of the actual capacity lives, and current research on where a model stores facts keeps pointing at them.

Day 1 takeaway

A transformer is embeddings, a stack of identical blocks, and a final linear layer. Encoder only sees everything and is trained to fill gaps; decoder only sees the past and is trained to predict the next token, which is why it won. Most parameters sit in the feed forward networks, not the attention.
Week 09 · Day 2 of 7

Masking and Cross-Attention

The leak that makes a language model look perfect and generate nonsense

By 1119 words

Building the decoder means being careful about exactly what each position is allowed to see, because getting it wrong produces a model that scores beautifully and cannot generate anything.

The leak, demonstrated

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
class Block(nn.Module):
"""Pre-normalisation transformer block, as week 8 day 7 built it."""
def __init__(self, width, heads, expansion=4, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(width, heads, dropout=dropout,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.ff = nn.Sequential(
nn.Linear(width, width * expansion), nn.GELU(),
nn.Linear(width * expansion, width), nn.Dropout(dropout))

def forward(self, x, mask=None):
h = self.norm1(x)
a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
return x + self.ff(self.norm2(x))
class Decoder(nn.Module):
def __init__(self, vocab=VOCAB, width=64, heads=4, blocks=2,
max_len=16, causal=True):
super().__init__()
self.causal = causal
self.tok = nn.Embedding(vocab, width)
self.pos = nn.Embedding(max_len, width)
self.blocks = nn.ModuleList([Block(width, heads)
for _ in range(blocks)])
self.norm = nn.LayerNorm(width)
self.head = nn.Linear(width, vocab)

def forward(self, x):
T = x.shape[1]
mask = None
if self.causal:
mask = torch.triu(torch.ones(T, T, dtype=torch.bool),
diagonal=1)
h = self.tok(x) + self.pos(torch.arange(T))
for block in self.blocks:
h = block(h, mask)
return self.head(self.norm(h))

def train_lm(causal, epochs=4):
torch.manual_seed(0)
model = Decoder(causal=causal)
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
loss_fn = nn.CrossEntropyLoss()
data = torch.cat([torch.full((6000, 1), START),
make_copy(6000, seed=3)[0]], dim=1)
loader = DataLoader(TensorDataset(data), batch_size=64, shuffle=True)
for _ in range(epochs):
model.train()
for (batch,) in loader:
opt.zero_grad()
logits = model(batch[:, :-1])
loss = loss_fn(logits.reshape(-1, VOCAB),
batch[:, 1:].reshape(-1))
loss.backward()
opt.step()
return model, loss.item()

for causal in [True, False]:
model, loss = train_lm(causal)
print('%-22s final training loss %.4f'
% ('causal mask' if causal else 'no mask', loss))
print('\nthe digits are uniform random, so the best possible loss is')
print('log(9) = %.4f. A loss far below that means the model is reading'
% torch.tensor(9.0).log().item())
print('the answer, which is only possible without the mask.')
causal mask final training loss 2.2127
no mask final training loss 0.3644

the digits are uniform random, so the best possible loss is
log(9) = 2.1972. A loss far below that means the model is reading
the answer, which is only possible without the mask.

This is the most expensive bug in the subject

A loss well below what the task allows is not a triumph, it is a leak. Without the causal mask, position i can attend to position i+1, which is the token it is being asked to predict. The model learns to copy it, the loss collapses, every metric looks superb, and at generation time there is no next token to read so the output is nonsense.

Whenever a language model's training loss looks too good, check the mask before you check anything else. The arithmetic above is the check: work out the best loss the task permits and compare.

The encoder and decoder together

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
class Block(nn.Module):
"""Pre-normalisation transformer block, as week 8 day 7 built it."""
def __init__(self, width, heads, expansion=4, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(width, heads, dropout=dropout,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.ff = nn.Sequential(
nn.Linear(width, width * expansion), nn.GELU(),
nn.Linear(width * expansion, width), nn.Dropout(dropout))

def forward(self, x, mask=None):
h = self.norm1(x)
a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
return x + self.ff(self.norm2(x))
class CrossBlock(nn.Module):
"""A decoder block: masked self-attention, then cross-attention to
the encoder output, then the feed forward network."""

def __init__(self, width, heads, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.self_attn = nn.MultiheadAttention(width, heads,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.cross_attn = nn.MultiheadAttention(width, heads,
batch_first=True)
self.norm3 = nn.LayerNorm(width)
self.ff = nn.Sequential(nn.Linear(width, width * 4), nn.GELU(),
nn.Linear(width * 4, width))

def forward(self, x, memory, mask=None):
h = self.norm1(x)
a, _ = self.self_attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
h = self.norm2(x)
c, _ = self.cross_attn(h, memory, memory, need_weights=False)
x = x + c
return x + self.ff(self.norm3(x))

torch.manual_seed(0)
block = CrossBlock(64, 4)
target = torch.randn(2, 6, 64)
memory = torch.randn(2, 9, 64) # encoder output, different length
print('target ', tuple(target.shape))
print('memory ', tuple(memory.shape), ' a different length is fine')
print('output ', tuple(block(target, memory).shape))
print('\ncross-attention is where the two sequences meet, and it is the')
print('only place the decoder can see the input at all.')
target (2, 6, 64)
memory (2, 9, 64) a different length is fine
output (2, 6, 64)

cross-attention is where the two sequences meet, and it is the
only place the decoder can see the input at all.

Day 2 takeaway

The causal mask is what separates a language model from a lookup of the answer. Check your training loss against the best the task permits. A decoder block is masked self-attention, then cross-attention to the encoder, then a feed forward network, and the two sequences may be different lengths.
Week 09 · Day 3 of 7

A Full Encoder and Decoder

The copy task, whole-sequence accuracy, and teacher forcing

By 1191 words

A complete encoder and decoder trained on the copy task, which is the smallest problem that genuinely needs both halves.

The model

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
class Block(nn.Module):
"""Pre-normalisation transformer block, as week 8 day 7 built it."""
def __init__(self, width, heads, expansion=4, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(width, heads, dropout=dropout,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.ff = nn.Sequential(
nn.Linear(width, width * expansion), nn.GELU(),
nn.Linear(width * expansion, width), nn.Dropout(dropout))

def forward(self, x, mask=None):
h = self.norm1(x)
a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
return x + self.ff(self.norm2(x))
class Seq2Seq(nn.Module):
def __init__(self, vocab=VOCAB, width=64, heads=4, blocks=2,
max_len=16):
super().__init__()
self.tok = nn.Embedding(vocab, width)
self.pos = nn.Embedding(max_len, width)
self.encoder = nn.ModuleList([Block(width, heads)
for _ in range(blocks)])
self.self_attn = nn.ModuleList(
[nn.MultiheadAttention(width, heads, batch_first=True)
for _ in range(blocks)])
self.cross_attn = nn.ModuleList(
[nn.MultiheadAttention(width, heads, batch_first=True)
for _ in range(blocks)])
self.ff = nn.ModuleList(
[nn.Sequential(nn.Linear(width, width * 4), nn.GELU(),
nn.Linear(width * 4, width))
for _ in range(blocks)])
self.norms = nn.ModuleList([nn.LayerNorm(width)
for _ in range(blocks * 3)])
self.out_norm = nn.LayerNorm(width)
self.head = nn.Linear(width, vocab)

def embed(self, x):
return self.tok(x) + self.pos(torch.arange(x.shape[1]))

def encode(self, src):
h = self.embed(src)
for block in self.encoder:
h = block(h)
return h

def decode(self, tgt_in, memory):
T = tgt_in.shape[1]
mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
h = self.embed(tgt_in)
for i in range(len(self.self_attn)):
n1, n2, n3 = self.norms[i * 3:i * 3 + 3]
q = n1(h)
a, _ = self.self_attn[i](q, q, q, attn_mask=mask,
need_weights=False)
h = h + a
q = n2(h)
c, _ = self.cross_attn[i](q, memory, memory,
need_weights=False)
h = h + c
h = h + self.ff[i](n3(h))
return self.head(self.out_norm(h))

def forward(self, src, tgt_in):
return self.decode(tgt_in, self.encode(src))

torch.manual_seed(0)
model = Seq2Seq()
print('parameters %d' % sum(p.numel() for p in model.parameters()))
print('output', tuple(model(src_tr[:2], tin_tr[:2]).shape))
parameters 236043
output (2, 6, 11)

Training it

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
class Block(nn.Module):
"""Pre-normalisation transformer block, as week 8 day 7 built it."""
def __init__(self, width, heads, expansion=4, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(width, heads, dropout=dropout,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.ff = nn.Sequential(
nn.Linear(width, width * expansion), nn.GELU(),
nn.Linear(width * expansion, width), nn.Dropout(dropout))

def forward(self, x, mask=None):
h = self.norm1(x)
a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
return x + self.ff(self.norm2(x))
import time

class Seq2Seq(nn.Module):
def __init__(self, vocab=VOCAB, width=64, heads=4, blocks=2,
max_len=16):
super().__init__()
self.tok = nn.Embedding(vocab, width)
self.pos = nn.Embedding(max_len, width)
self.encoder = nn.ModuleList([Block(width, heads)
for _ in range(blocks)])
self.self_attn = nn.ModuleList(
[nn.MultiheadAttention(width, heads, batch_first=True)
for _ in range(blocks)])
self.cross_attn = nn.ModuleList(
[nn.MultiheadAttention(width, heads, batch_first=True)
for _ in range(blocks)])
self.ff = nn.ModuleList(
[nn.Sequential(nn.Linear(width, width * 4), nn.GELU(),
nn.Linear(width * 4, width))
for _ in range(blocks)])
self.norms = nn.ModuleList([nn.LayerNorm(width)
for _ in range(blocks * 3)])
self.out_norm = nn.LayerNorm(width)
self.head = nn.Linear(width, vocab)

def embed(self, x):
return self.tok(x) + self.pos(torch.arange(x.shape[1]))

def encode(self, src):
h = self.embed(src)
for block in self.encoder:
h = block(h)
return h

def decode(self, tgt_in, memory):
T = tgt_in.shape[1]
mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
h = self.embed(tgt_in)
for i in range(len(self.self_attn)):
n1, n2, n3 = self.norms[i * 3:i * 3 + 3]
q = n1(h)
a, _ = self.self_attn[i](q, q, q, attn_mask=mask,
need_weights=False)
h = h + a
q = n2(h)
c, _ = self.cross_attn[i](q, memory, memory,
need_weights=False)
h = h + c
h = h + self.ff[i](n3(h))
return self.head(self.out_norm(h))

def forward(self, src, tgt_in):
return self.decode(tgt_in, self.encode(src))

torch.manual_seed(0)
model = Seq2Seq()
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
loss_fn = nn.CrossEntropyLoss()
start = time.time()
for epoch in range(1, 9):
model.train()
for s, ti, to in train_loader:
opt.zero_grad()
logits = model(s, ti)
loss_fn(logits.reshape(-1, VOCAB), to.reshape(-1)).backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
if epoch % 2 == 0:
model.eval()
with torch.no_grad():
pred = model(src_va, tin_va).argmax(-1)
token_acc = (pred == tout_va).float().mean().item()
exact = (pred == tout_va).all(dim=1).float().mean().item()
print('epoch %d token accuracy %.4f whole sequence %.4f'
% (epoch, token_acc, exact))
print('\n%.0f seconds' % (time.time() - start))
epoch 2 token accuracy 1.0000 whole sequence 1.0000
epoch 4 token accuracy 1.0000 whole sequence 1.0000
epoch 6 token accuracy 1.0000 whole sequence 1.0000
epoch 8 token accuracy 1.0000 whole sequence 1.0000

21 seconds

Teacher forcing, and the gap it hides

During training the decoder is fed the correct previous tokens, not its own predictions. That is teacher forcing, and it is what makes training parallel: every output position is computed at once. At generation time the model must feed on its own output, so a single early mistake changes everything after it, and the model has never trained on that situation. The gap between training loss and generation quality is largely this, and it is why day 4 measures generation separately.

Day 3 takeaway

An encoder and decoder transformer is two stacks joined by cross-attention. Report whole-sequence accuracy as well as per-token accuracy, because getting 95 percent of tokens right can mean getting almost no sequences right. And remember that training uses teacher forcing, which generation does not.
Week 09 · Day 4 of 7

Generation

Greedy, temperature, top-k and nucleus, and why the settings matter

By 868 words

Generating from a trained model is a loop, and every decision inside it changes the output more than most architectural choices do.

Greedy decoding

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
class Block(nn.Module):
"""Pre-normalisation transformer block, as week 8 day 7 built it."""
def __init__(self, width, heads, expansion=4, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(width, heads, dropout=dropout,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.ff = nn.Sequential(
nn.Linear(width, width * expansion), nn.GELU(),
nn.Linear(width * expansion, width), nn.Dropout(dropout))

def forward(self, x, mask=None):
h = self.norm1(x)
a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
return x + self.ff(self.norm2(x))
class Decoder(nn.Module):
def __init__(self, vocab=VOCAB, width=64, heads=4, blocks=2,
max_len=16):
super().__init__()
self.tok = nn.Embedding(vocab, width)
self.pos = nn.Embedding(max_len, width)
self.blocks = nn.ModuleList([Block(width, heads)
for _ in range(blocks)])
self.norm = nn.LayerNorm(width)
self.head = nn.Linear(width, vocab)

def forward(self, x):
T = x.shape[1]
mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
h = self.tok(x) + self.pos(torch.arange(T))
for block in self.blocks:
h = block(h, mask)
return self.head(self.norm(h))

# A language model over a repeating pattern, so we can tell whether the
# generated output is right.
torch.manual_seed(0)
g = torch.Generator().manual_seed(0)
pattern = torch.tensor([1, 2, 3, 4, 5, 6])
rows = pattern.repeat(3000).view(-1, 6)
noise = torch.rand(rows.shape, generator=g) < 0.05
rows[noise] = torch.randint(1, 10, (int(noise.sum()),), generator=g)
data = torch.cat([torch.full((len(rows), 1), START), rows], dim=1)

model = Decoder()
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
loss_fn = nn.CrossEntropyLoss()
loader = DataLoader(TensorDataset(data), batch_size=64, shuffle=True)
for _ in range(6):
model.train()
for (batch,) in loader:
opt.zero_grad()
loss_fn(model(batch[:, :-1]).reshape(-1, VOCAB),
batch[:, 1:].reshape(-1)).backward()
opt.step()

model.eval()
seq = torch.tensor([[START]])
with torch.no_grad():
for _ in range(6):
nxt = model(seq)[:, -1, :].argmax(-1, keepdim=True)
seq = torch.cat([seq, nxt], dim=1)
print('greedy generation:', seq[0, 1:].tolist())
print('the pattern was :', pattern.tolist())
greedy generation: [1, 2, 3, 4, 5, 6]
the pattern was : [1, 2, 3, 4, 5, 6]

Temperature, top-k and nucleus sampling

import torch

torch.manual_seed(0)
logits = torch.tensor([3.2, 2.9, 1.4, 0.8, 0.4, 0.1, -0.5, -1.2])

def show(name, probs):
print('%-18s %s' % (name, ' '.join('%.3f' % v for v in probs.tolist())))

show('raw softmax', logits.softmax(-1))
show('temperature 0.5', (logits / 0.5).softmax(-1))
show('temperature 2.0', (logits / 2.0).softmax(-1))

# top-k: keep the k largest, renormalise
k = 3
kth = logits.topk(k).values[-1]
show('top-k, k=3', logits.masked_fill(logits < kth, float('-inf')).softmax(-1))

# nucleus: keep the smallest set whose probability exceeds p
probs = logits.softmax(-1)
order = probs.argsort(descending=True)
cumulative = probs[order].cumsum(0)
keep = order[cumulative - probs[order] < 0.9]
filtered = torch.full_like(logits, float('-inf'))
filtered[keep] = logits[keep]
show('nucleus, p=0.9', filtered.softmax(-1))
raw softmax 0.467 0.346 0.077 0.042 0.028 0.021 0.012 0.006
temperature 0.5 0.629 0.345 0.017 0.005 0.002 0.001 0.000 0.000
temperature 2.0 0.303 0.261 0.123 0.091 0.075 0.064 0.048 0.034
top-k, k=3 0.525 0.389 0.087 0.000 0.000 0.000 0.000 0.000
nucleus, p=0.9 0.501 0.371 0.083 0.045 0.000 0.000 0.000 0.000
StrategyGood forFailure mode
GreedyTasks with one right answer: translation, copyingRepetitive and bland on open-ended text
Temperature samplingVarietyIncoherence at high values
Top-kCutting off the nonsense tailA fixed k is too small when the model is unsure and too large when it is confident
Nucleus (top-p)The usual default nowStill needs a temperature alongside it
Beam searchTranslation and summarisingProduces bland text on open-ended generation, and costs beam width times as much

Sampling settings are not cosmetic

The same model at temperature 0.7 with nucleus sampling and at temperature 1.5 with none behaves like two different systems. When you compare a model against a published result and cannot reproduce it, the decoding settings are the first thing to check, and they are frequently not reported. They belong in the model contract from week 15 of the Machine Learning course, alongside the threshold.

Day 4 takeaway

Generation is a loop that feeds the model its own output. Greedy decoding is right when there is one correct answer, and sampling with a temperature and a nucleus cutoff is the default for open-ended text. Record the decoding settings, because they change the output as much as the weights do.
Week 09 · Day 5 of 7

Training a Transformer

Warmup, the failure table, and weight tying

By 1009 words

Transformers are harder to train than convolutional networks, and the difficulties are specific and well known.

Warmup, and when it earns its place

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
class Block(nn.Module):
"""Pre-normalisation transformer block, as week 8 day 7 built it."""
def __init__(self, width, heads, expansion=4, dropout=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(width, heads, dropout=dropout,
batch_first=True)
self.norm2 = nn.LayerNorm(width)
self.ff = nn.Sequential(
nn.Linear(width, width * expansion), nn.GELU(),
nn.Linear(width * expansion, width), nn.Dropout(dropout))

def forward(self, x, mask=None):
h = self.norm1(x)
a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
x = x + a
return x + self.ff(self.norm2(x))
class Small(nn.Module):
def __init__(self, width=64, heads=4, blocks=2):
super().__init__()
self.tok = nn.Embedding(VOCAB, width)
self.pos = nn.Embedding(16, width)
self.blocks = nn.ModuleList([Block(width, heads)
for _ in range(blocks)])
self.norm = nn.LayerNorm(width)
self.head = nn.Linear(width, VOCAB)

def forward(self, x):
T = x.shape[1]
mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
h = self.tok(x) + self.pos(torch.arange(T))
for block in self.blocks:
h = block(h, mask)
return self.head(self.norm(h))

# Learnable data, so the loss can actually fall and the warmup has
# something to affect. Random digits would already sit at their floor.
g = torch.Generator().manual_seed(5)
starts = torch.randint(1, 5, (4000, 1), generator=g)
data = torch.cat([torch.full((4000, 1), START),
starts + torch.arange(6)], dim=1)
loader = DataLoader(TensorDataset(data), batch_size=64, shuffle=True)
loss_fn = nn.CrossEntropyLoss()

def run(lr, warmup_steps):
torch.manual_seed(0)
model = Small()
opt = torch.optim.AdamW(model.parameters(), lr=lr)
step, losses = 0, []
for _ in range(3):
model.train()
for (batch,) in loader:
step += 1
if warmup_steps:
scale = min(1.0, step / warmup_steps)
for group in opt.param_groups:
group['lr'] = lr * scale
opt.zero_grad()
loss = loss_fn(model(batch[:, :-1]).reshape(-1, VOCAB),
batch[:, 1:].reshape(-1))
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
losses.append(loss.item())
return losses

print('%-28s %14s %14s' % ('', 'loss at step 20', 'final loss'))
for lr, warm in [(1e-2, 0), (1e-2, 100), (3e-3, 0)]:
losses = run(lr, warm)
print('%-28s %14.4f %14.4f'
% ('lr %g, warmup %d' % (lr, warm), losses[19], losses[-1]))
loss at step 20 final loss
lr 0.01, warmup 0 0.2426 0.2340
lr 0.01, warmup 100 0.4119 0.2297
lr 0.003, warmup 0 0.2583 0.2347

The best loss this task allows is about 0.23: the first token has four possibilities and every token after it is determined, so the average is log(4) divided by six. All three runs reach it, and the warmed-up run is behind at step 20 for the obvious reason that its learning rate is still climbing.

Warmup bought nothing here, and that is worth understanding

A three-block model on a six-token task is not where warmup matters. The problem it solves is specific: Adam's running estimates of gradient magnitude are unreliable for the first few dozen steps, so a full-size step taken on that estimate can move a large model somewhere it never recovers from. With a hundred and fifty thousand parameters there is not enough depth for that to happen.

It becomes close to mandatory as models get deep, as batches get large, and with post-normalisation blocks. Since it costs a few lines and cannot hurt, it stays in the recipe. But do not expect to measure its benefit on anything you can train while you watch.

Where the difficulties are

SymptomCauseFix
Loss spikes or becomes nan earlyLearning rate too high for the first stepsWarmup, then clip at norm 1.0
Loss falls to near zero immediatelyThe causal mask is missing or invertedCompare against the best loss the task permits
Trains but generates nonsenseTrained with teacher forcing only, or wrong mask at inferenceEvaluate by generating, not only by loss
Runs out of memory as sequences lengthenAttention is quadratic in lengthShorter context, gradient checkpointing, or an efficient attention implementation
Good on short inputs, poor on long onesPosition encodings do not extrapolateRotary encodings, or train at the length you will serve

Weight tying

import torch
from torch import nn

vocab, width = 5000, 256
embed = nn.Embedding(vocab, width)
head = nn.Linear(width, vocab, bias=False)
print('embedding %d, output head %d'
% (embed.weight.numel(), head.weight.numel()))

head.weight = embed.weight # one tensor, used twice
total = sum({id(p): p.numel()
for p in list(embed.parameters()) +
list(head.parameters())}.values())
print('after tying, distinct parameters %d' % total)
print('\nthe input embedding maps a token to a vector; the output head')
print('maps a vector to a score per token. Sharing them saves a lot and')
print('usually improves the model slightly.')
embedding 1280000, output head 1280000
after tying, distinct parameters 1280000

the input embedding maps a token to a vector; the output head
maps a vector to a score per token. Sharing them saves a lot and
usually improves the model slightly.

Day 5 takeaway

Warm the learning rate up over the first few hundred steps, which costs nothing and matters once the model is large. Clip the gradient norm. Check your loss against the best the task allows before believing it. Evaluate by generating as well as by loss. And tie the input embedding to the output head, which saves parameters and usually helps.
Week 09 · Day 6 of 7

The Library

Built-in layers, the two wrong defaults, and fused attention

By 519 words

PyTorch ships transformer layers, and they are worth using, provided you know which arguments matter.

The built-in layers

import torch
from torch import nn

torch.manual_seed(0)
layer = nn.TransformerEncoderLayer(d_model=64, nhead=4,
dim_feedforward=256, dropout=0.1,
batch_first=True, norm_first=True)
encoder = nn.TransformerEncoder(layer, num_layers=3)

x = torch.randn(2, 10, 64)
print('output', tuple(encoder(x).shape))
print('parameters %d' % sum(p.numel() for p in encoder.parameters()))

causal = torch.triu(torch.ones(10, 10, dtype=torch.bool), diagonal=1)
print('with a causal mask:', tuple(encoder(x, mask=causal).shape))
output (2, 10, 64)
parameters 149952
with a causal mask: (2, 10, 64)

norm_first defaults to False

Which gives you the 2017 post-normalisation block, not the pre-normalisation one that every model since has used and that day 3 explained. The default is there for backwards compatibility and it will make your model harder to train, needing a longer warmup and a lower rate. Pass norm_first=True. Pass batch_first=True as well, for the same reason as the recurrent layers.

Efficient attention

import torch
from torch import nn
import torch.nn.functional as F

torch.manual_seed(0)
B, H, T, D = 2, 4, 32, 16
q, k, v = (torch.randn(B, H, T, D) for _ in range(3))

# by hand
scores = q @ k.transpose(-2, -1) / D ** 0.5
manual = scores.softmax(-1) @ v

# the fused implementation
fused = F.scaled_dot_product_attention(q, k, v)

print('same answer:', bool(torch.allclose(manual, fused, atol=1e-5)))
print('\nand with a causal mask:')
causal_manual = scores.masked_fill(
torch.triu(torch.ones(T, T, dtype=torch.bool), 1),
float('-inf')).softmax(-1) @ v
causal_fused = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print('same answer:', bool(torch.allclose(causal_manual, causal_fused,
atol=1e-5)))
same answer: True

and with a causal mask:
same answer: True

scaled_dot_product_attention is the same arithmetic in a fused kernel that never builds the full score matrix in memory. On a GPU that turns the memory cost from quadratic in the sequence length to linear, which is what makes long contexts affordable. Use it rather than writing the four lines yourself, and pass is_causal=True instead of building a mask.

Counting the cost

def transformer_params(vocab, width, blocks, heads, expansion=4,
max_len=1024, tied=True):
embed = vocab * width + max_len * width
attn = 4 * width * width # q, k, v and output
ff = 2 * width * width * expansion
norms = 4 * width
per_block = attn + ff + norms
head = 0 if tied else vocab * width
return embed + blocks * per_block + head

print('%-22s %10s %14s' % ('model', 'blocks', 'parameters'))
for name, width, blocks, heads, vocab in [
('tiny', 128, 4, 4, 5000),
('small', 512, 8, 8, 32000),
('GPT-2 sized', 768, 12, 12, 50257),
('GPT-2 medium', 1024, 24, 16, 50257)]:
n = transformer_params(vocab, width, blocks, heads)
print('%-22s %10d %14s' % (name, blocks, '{:,}'.format(n)))
print('\nand at 4 bytes per parameter, GPT-2 sized is about %.0f MB'
% (transformer_params(50257, 768, 12, 12) * 4 / 1e6))
model blocks parameters
tiny 4 1,559,552
small 8 42,090,496
GPT-2 sized 12 124,355,328
GPT-2 medium 24 354,599,936

and at 4 bytes per parameter, GPT-2 sized is about 497 MB

Day 6 takeaway

Use nn.TransformerEncoderLayer with norm_first=True and batch_first=True, because both defaults are wrong for modern practice. Use F.scaled_dot_product_attention rather than writing attention by hand, and pass is_causal=True.
Week 09 · Day 7 of 7

A Small GPT

A decoder-only model that infers a rule nobody told it

By 1067 words

The week assembled: a decoder-only transformer trained as a language model, generating, and measured against everything before it.

The model, using the library

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
import time

class GPT(nn.Module):
def __init__(self, vocab=VOCAB, width=64, heads=4, blocks=3,
max_len=16, dropout=0.1):
super().__init__()
self.tok = nn.Embedding(vocab, width)
self.pos = nn.Embedding(max_len, width)
self.drop = nn.Dropout(dropout)
layer = nn.TransformerEncoderLayer(
d_model=width, nhead=heads, dim_feedforward=width * 4,
dropout=dropout, batch_first=True, norm_first=True,
activation='gelu')
self.blocks = nn.TransformerEncoder(layer, num_layers=blocks)
self.norm = nn.LayerNorm(width)
self.head = nn.Linear(width, vocab, bias=False)
self.head.weight = self.tok.weight # tied

def forward(self, x):
T = x.shape[1]
mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
h = self.drop(self.tok(x) + self.pos(torch.arange(T)))
return self.head(self.norm(self.blocks(h, mask=mask)))

@torch.no_grad()
def generate(self, prompt, steps, temperature=1.0, top_k=None):
self.eval()
seq = prompt
for _ in range(steps):
logits = self(seq)[:, -1, :] / temperature
if top_k:
kth = logits.topk(top_k).values[:, -1:]
logits = logits.masked_fill(logits < kth, float('-inf'))
nxt = torch.multinomial(logits.softmax(-1), 1)
seq = torch.cat([seq, nxt], dim=1)
return seq

torch.manual_seed(0)
model = GPT()
print('parameters %d' % sum(p.numel() for p in model.parameters()))
print('output', tuple(model(torch.randint(1, 10, (2, 6))).shape))
parameters 151808
output (2, 6, 11)

Trained, and generating

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

PAD, START = 0, 10
VOCAB = 11 # digits 1 to 9, plus pad and start

def make_copy(n, length=6, seed=0):
"""Source is a sequence of digits; target is the same sequence.

Trivial for a person and a real test for a model: it has to align
output position i with input position i, which is exactly what
cross-attention is for."""

g = torch.Generator().manual_seed(seed)
src = torch.randint(1, 10, (n, length), generator=g)
start = torch.full((n, 1), START)
tgt_in = torch.cat([start, src[:, :-1]], dim=1)
return src, tgt_in, src

src_tr, tin_tr, tout_tr = make_copy(6000, seed=0)
src_va, tin_va, tout_va = make_copy(1000, seed=1)
train_loader = DataLoader(TensorDataset(src_tr, tin_tr, tout_tr),
batch_size=64, shuffle=True)
import time

class GPT(nn.Module):
def __init__(self, vocab=VOCAB, width=64, heads=4, blocks=3,
max_len=16, dropout=0.1):
super().__init__()
self.tok = nn.Embedding(vocab, width)
self.pos = nn.Embedding(max_len, width)
self.drop = nn.Dropout(dropout)
layer = nn.TransformerEncoderLayer(
d_model=width, nhead=heads, dim_feedforward=width * 4,
dropout=dropout, batch_first=True, norm_first=True,
activation='gelu')
self.blocks = nn.TransformerEncoder(layer, num_layers=blocks)
self.norm = nn.LayerNorm(width)
self.head = nn.Linear(width, vocab, bias=False)
self.head.weight = self.tok.weight

def forward(self, x):
T = x.shape[1]
mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
h = self.drop(self.tok(x) + self.pos(torch.arange(T)))
return self.head(self.norm(self.blocks(h, mask=mask)))

@torch.no_grad()
def generate(self, prompt, steps, temperature=1.0):
self.eval()
seq = prompt
for _ in range(steps):
logits = self(seq)[:, -1, :] / temperature
seq = torch.cat([seq, torch.multinomial(logits.softmax(-1), 1)],
dim=1)
return seq

# a language with a rule: every sequence counts up from where it starts
torch.manual_seed(0)
g = torch.Generator().manual_seed(0)
starts = torch.randint(1, 5, (6000, 1), generator=g)
rows = starts + torch.arange(6)
data = torch.cat([torch.full((6000, 1), START), rows], dim=1)
loader = DataLoader(TensorDataset(data), batch_size=64, shuffle=True)

model = GPT()
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)
loss_fn = nn.CrossEntropyLoss()
start = time.time()
step = 0
for epoch in range(1, 7):
model.train()
for (batch,) in loader:
step += 1
for group in opt.param_groups:
group['lr'] = 3e-3 * min(1.0, step / 100) # warmup
opt.zero_grad()
loss = loss_fn(model(batch[:, :-1]).reshape(-1, VOCAB),
batch[:, 1:].reshape(-1))
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
if epoch % 2 == 0:
print('epoch %d loss %.4f' % (epoch, loss.item()))
print('trained in %.0f seconds\n' % (time.time() - start))

for temperature in [0.5, 1.0]:
torch.manual_seed(0)
out = model.generate(torch.full((3, 1), START), 6,
temperature=temperature)
print('temperature %.1f:' % temperature)
for row in out[:, 1:].tolist():
rule = all(row[i + 1] == row[i] + 1 for i in range(len(row) - 1))
print(' %s follows the rule: %s' % (row, rule))
epoch 2 loss 0.2484
epoch 4 loss 0.2523
epoch 6 loss 0.2607
trained in 14 seconds

temperature 0.5:
[1, 2, 3, 4, 5, 6] follows the rule: True
[1, 2, 3, 4, 5, 6] follows the rule: True
[1, 2, 3, 4, 5, 6] follows the rule: True
temperature 1.0:
[1, 2, 3, 4, 5, 6] follows the rule: True
[1, 2, 3, 4, 5, 6] follows the rule: True
[1, 2, 3, 4, 5, 6] follows the rule: True

It learned a rule it was never told

Nothing in the training loop mentions counting. The model saw six thousand sequences that happened to count upwards and inferred the pattern from next-token prediction alone. That is the whole idea behind language model pretraining, at a scale you can watch: predict the next thing, and structure you never specified falls out.

The checklist

  1. Pre-normalisation blocks: norm_first=True.
  2. A causal mask for anything that generates, and check the loss against the best the task permits.
  3. Warm the learning rate up over a few hundred steps.
  4. Clip the gradient norm at 1.0.
  5. AdamW, with weight decay excluded from normalisation parameters and biases.
  6. Tie the input embedding to the output head.
  7. Evaluate by generating, not only by loss.
  8. Record the decoding settings with the weights.

Day 7 takeaway

A decoder-only transformer is embeddings, a stack of pre-normalisation blocks with a causal mask, and a tied output head. Trained by next-token prediction it infers structure nobody specified, which is the entire basis of what week 11 does with real text.