Transformers and Large Language Models

Week 7 of 14 · Language · 7 days

Full curriculum
Week 07 · Language

Transformers and Large Language Models

Week 07 · Day 1 of 7

Vectors That Depend on the Sentence

Subword tokens, and the unknown word that no longer breaks anything

By 264 words

Week 6 left two problems. A static embedding gives one vector per word whatever it means in context, and averaging those vectors is blind to word order. Both are solved by the same change: compute the vector from the sentence rather than looking it up.

Contextual embedding: A representation of a word produced by a model that has read the whole sentence, so the same word receives different vectors in different contexts. This is what a transformer produces and it is the reason they replaced static vectors.

Subword tokenisation, which solves the unknown word problem

import os
os.environ['HF_HUB_OFFLINE'] = '0'
import torch
from transformers import AutoTokenizer, AutoModel

NAME = 'prajjwal1/bert-tiny'
tok = AutoTokenizer.from_pretrained(NAME)
model = AutoModel.from_pretrained(NAME)
model.eval()
for text in ['the trackpad is faulty', 'the zibbleflex is faulty']:
ids = tok(text)['input_ids']
print('%-30s %s' % (text, tok.convert_ids_to_tokens(ids)))
the trackpad is faulty ['[CLS]', 'the', 'track', '##pad', 'is', 'faulty', '[SEP]']
the zibbleflex is faulty ['[CLS]', 'the', 'z', '##ib', '##ble', '##fle', '##x', 'is', 'faulty', '[SEP]']

The invented word did not fail. It was broken into pieces that are in the vocabulary, so the model produces a representation for it assembled from familiar fragments. This is why a modern model never encounters a word it cannot process at all, and it is why token counts on an API bill do not match word counts.

The special tokens are not decoration

Notice the markers at each end. The first one accumulates a representation of the whole sequence during the forward pass, and it is what a classifier is usually attached to. They are part of how the model works rather than formatting.

Week 07 · Day 2 of 7

The Same Word, Different Meanings

Bank by the river against bank on the high street, measured

By 366 words

The promise was one vector per occurrence rather than one per word. Here it is, tested on the standard ambiguous example.

import os
os.environ['HF_HUB_OFFLINE'] = '0'
import torch
from transformers import AutoTokenizer, AutoModel

NAME = 'prajjwal1/bert-tiny'
tok = AutoTokenizer.from_pretrained(NAME)
model = AutoModel.from_pretrained(NAME)
model.eval()
import numpy as np

def word_vector(sentence, word):
"""The model's representation of one word in this sentence."""
enc = tok(sentence, return_tensors='pt')
with torch.no_grad():
out = model(**enc).last_hidden_state[0]
tokens = tok.convert_ids_to_tokens(enc['input_ids'][0])
picked = [i for i, t in enumerate(tokens) if t == word]
return out[picked].mean(0).numpy()

def cos(a, b):
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9))

river = word_vector('we sat on the bank of the river', 'bank')
money = word_vector('she paid it into the bank on friday', 'bank')
money2 = word_vector('the bank approved the loan last week', 'bank')
print('two financial senses %.3f' % cos(money, money2))
print('financial and riverside %.3f' % cos(money, river))
two financial senses 0.868
financial and riverside 0.880

The same word, three sentences, and the two financial uses are closer to each other than either is to the riverside one. A static embedding would have produced one identical vector for all three by construction, so this is the concrete difference between the two generations of method.

Word order, which averaging could not see

import os
os.environ['HF_HUB_OFFLINE'] = '0'
import torch
from transformers import AutoTokenizer, AutoModel

NAME = 'prajjwal1/bert-tiny'
tok = AutoTokenizer.from_pretrained(NAME)
model = AutoModel.from_pretrained(NAME)
model.eval()
import numpy as np

def sentence_vector(sentence):
enc = tok(sentence, return_tensors='pt')
with torch.no_grad():
out = model(**enc).last_hidden_state[0]
mask = enc['attention_mask'][0].unsqueeze(-1)
return ((out * mask).sum(0) / mask.sum()).numpy()

a = sentence_vector('the dog chased the cat')
b = sentence_vector('the cat chased the dog')
print('identical: %s' % bool(np.allclose(a, b, atol=1e-6)))
sim = float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print('similarity %.4f, which is high but not 1.0' % sim)
print()
print('week 6 averaged static vectors and got exactly identical')
identical: False
similarity 0.9996, which is high but not 1.0

week 6 averaged static vectors and got exactly identical

Not identical any more. The two sentences remain similar, which is correct since they share every word and a topic, and the model can now tell them apart at all, which averaging never could.

Week 07 · Day 3 of 7

Attention

Weighting every other word, worked through by hand

By 356 words

The mechanism behind this is called attention, and the idea is simpler than the reputation suggests: when building the representation of a word, look at the other words and weight them by how relevant they are.

Attention: For each word, score every other word in the sequence for relevance, turn the scores into weights that sum to one, and take the weighted average of their representations. The scores are computed from the words themselves, so the pattern differs for every sentence.
import numpy as np

# Attention by hand, on made up vectors, so the arithmetic is visible.
np.random.seed(0)
words = ['the', 'cat', 'sat', 'on', 'it']
vectors = np.random.randn(len(words), 4)

def attend(query_index):
q = vectors[query_index]
scores = vectors @ q / np.sqrt(vectors.shape[1])
weights = np.exp(scores - scores.max())
weights = weights / weights.sum()
return weights

w = attend(words.index('it'))
print('building the representation of "it", the model weights:')
for word, weight in zip(words, w):
print(' %-5s %.3f %s' % (word, weight, '#' * int(weight * 40)))
print()
print('weights sum to %.1f, and they are computed from the words,' % w.sum())
print('so a different sentence produces a different pattern')
building the representation of "it", the model weights:
the 0.115 ####
cat 0.396 ###############
sat 0.035 #
on 0.116 ####
it 0.338 #############

weights sum to 1.0, and they are computed from the words,
so a different sentence produces a different pattern

That is the whole operation. The scores here come from random vectors so the pattern means nothing, but the shape is right: every word gets a weight, the weights sum to one, and the result is a blend. In a trained model the weights on a pronoun concentrate on whatever it refers to, which is how the representation of it comes to carry information about cat.

Why this replaced everything else

Every word can look at every other word directly, in one step, regardless of how far apart they are. The methods that came before either ignored order entirely or passed information along the sentence one word at a time, losing it over distance. Attention has neither problem, and it parallelises, which is what made training on enormous corpora practical.

Week 07 · Day 4 of 7

Three Ways to Use a Pretrained Model

Frozen features against TF-IDF, and the expensive method losing

By 670 words

A pretrained model is useful in three quite different ways, and choosing between them is mostly a question of how many labels you have.

import os
os.environ['HF_HUB_OFFLINE'] = '0'
import torch
from transformers import AutoTokenizer, AutoModel

NAME = 'prajjwal1/bert-tiny'
tok = AutoTokenizer.from_pretrained(NAME)
model = AutoModel.from_pretrained(NAME)
model.eval()
import random

random.seed(3)

CATEGORY_WORDS = {
'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
'the {part} is {opinion}, {tail}',
'arrived quickly and the {part} was {opinion}, {tail}',
'i wanted to like it but the {part} is {opinion}, {tail}',
'after a month the {part} is still {opinion}, {tail}',
'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
'second one i have bought', 'delivery took a week',
'as described in the listing']

def make_review(category, positive):
part = random.choice(CATEGORY_WORDS[category])
# Half the positive reviews are written as a negated negative, so
# 'not flimsy' has to be read as praise. Bag of words cannot.
if positive:
if random.random() < 0.5:
opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
else:
opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
tail = random.choice(TAIL)
else:
if random.random() < 0.5:
opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
else:
opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
tail = random.choice(TAIL)
return random.choice(FRAME).format(part=part, opinion=opinion,
tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
for cat in CATEGORY_WORDS:
for pos in (True, False):
texts.append(make_review(cat, pos))
sentiment.append(1 if pos else 0)
category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
train_c, test_c) = train_test_split(texts, sentiment, category,
test_size=0.3, random_state=0,
stratify=sentiment)
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline

def embed_all(sentences, batch=32):
out = []
for i in range(0, len(sentences), batch):
enc = tok(sentences[i:i + batch], return_tensors='pt',
padding=True, truncation=True, max_length=48)
with torch.no_grad():
h = model(**enc).last_hidden_state
mask = enc['attention_mask'].unsqueeze(-1)
out.append(((h * mask).sum(1) / mask.sum(1)).numpy())
return np.vstack(out)

Etr, Ete = embed_all(train_x), embed_all(test_x)
frozen = LogisticRegression(max_iter=3000).fit(Etr, train_s)
tfidf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
LogisticRegression(max_iter=2000)).fit(train_x,
train_s)
print('%-38s %10s' % ('', 'sentiment'))
print('%-38s %10.4f' % ('tfidf bigrams from week 4',
tfidf.score(test_x, test_s)))
print('%-38s %10.4f' % ('frozen transformer features',
frozen.score(Ete, test_s)))
sentiment
tfidf bigrams from week 4 1.0000
frozen transformer features 0.6236

A result worth reading carefully rather than celebrating. The simple method from week 4 is not beaten here, because this corpus was built so that bigrams capture the negation completely, and a tiny transformer used as a frozen feature extractor has no advantage on a task a pair of words already solves.

The expensive method does not always win

This is a real result on real code, and it generalises further than it might look. Pretrained models earn their cost on messy, varied, ambiguous language. On narrow, templated text with a clear lexical signal, TF-IDF is frequently as good and it is thousands of times cheaper to run.

Measure both. The comparison takes an afternoon and it regularly saves a great deal of infrastructure.

The three ways to use a pretrained model

ApproachWhat you needWhen
Frozen featuresA few hundred labelsLabels are scarce and the domain is close to the model's
Fine-tuningThousands of labels, and a GPULabels are plentiful and the task is specific
Prompting a large modelNo labels, and an API billYou need something working today, or the task is open ended
Week 07 · Day 5 of 7

What a Large Language Model Is Doing

Next token prediction, and why hallucination is the objective working

By 341 words

The models everybody now means by AI are the same architecture, made very much larger and trained on a single objective: predict the next token. Everything they appear to do follows from that.

# Next token prediction, by counting, on a small corpus. A large
# language model does something far more sophisticated, and it is
# answering exactly this question.
import random
from collections import defaultdict, Counter

TEXT = ('the cat sat on the mat . the cat sat on the rug . '
'the dog sat on the mat . the dog barked at the cat . '
'the cat ran from the dog . the dog ran to the mat . ') * 6
words = TEXT.split()
nxt = defaultdict(Counter)
for a, b in zip(words, words[1:]):
nxt[a][b] += 1

print('after "the", the model expects:')
total = sum(nxt['the'].values())
for word, n in nxt['the'].most_common(4):
print(' %-6s %.2f' % (word, n / total))
print()
random.seed(0)
current, out = 'the', ['the']
for _ in range(14):
choices = list(nxt[current].items())
current = random.choices([c for c, _ in choices],
[n for _, n in choices])[0]
out.append(current)
print('generated: %s' % ' '.join(out))
after "the", the model expects:
cat 0.33
dog 0.33
mat 0.25
rug 0.08

generated: the dog ran from the mat . the cat sat on the mat . the

Predict a distribution over the next word, sample from it, append, repeat. That is the entire generation loop, and a model with hundreds of billions of parameters runs the same loop. The difference is entirely in how good the distribution is.

This is why they make things up

Nothing in that loop consults a source or checks a fact. It produces the token that fits, and a fluent falsehood fits as well as a fluent truth. Hallucination is not a bug that will be patched out, it is the objective working exactly as specified.

The practical consequence is that anything requiring factual accuracy needs either retrieval, so the facts are in front of the model, or verification after the fact. Trusting the output because it reads well is the single most common mistake being made with these systems.

Week 07 · Day 6 of 7

Using Them Well

Prompting, retrieval, and the row that costs money

By 280 words

Using a large language model well is mostly about knowing which problems it suits and how to check what comes back.

Prompting, briefly

  • Say what you want precisely. Most disappointing output is a faithful response to an ambiguous instruction.
  • Give examples. Two or three worked examples in the prompt reliably outperform a paragraph of description.
  • Ask for a structure. Requesting JSON with named fields makes the output checkable by a program instead of by reading.
  • Supply the facts. If the answer depends on your documents, put them in the prompt. This is retrieval augmented generation and it is the standard remedy for hallucination.

Where they genuinely earn their cost

Suits themDoes not
Open ended generation with no single right answerAnything needing a guaranteed correct answer
Tasks with no labelled data yetHigh volume classification where labels exist
Extremely varied input a rule could never coverNarrow templated text a regular expression handles
Prototyping, to find out whether a task is possible at allArithmetic, sorting, or anything a program does exactly

The second row is the one that costs money. Sending millions of documents through a large model to assign one of four labels is expensive, slow and usually beaten by a TF-IDF classifier trained on a few thousand examples, as day 4 measured on a much smaller scale.

A reasonable default order

Prototype with a large model to find out whether the task is solvable and to generate a first batch of labels. Then, if the volume justifies it, train something small on those labels and keep the large model for the cases the small one is unsure about. You get the capability without the running cost.

Week 07 · Day 7 of 7

Three Generations, One Task

The whole language half compared, with the clock running

By 732 words

The language half of this course, from counting words to contextual representations, in one comparison on the same task.

import os
os.environ['HF_HUB_OFFLINE'] = '0'
import torch
from transformers import AutoTokenizer, AutoModel

NAME = 'prajjwal1/bert-tiny'
tok = AutoTokenizer.from_pretrained(NAME)
model = AutoModel.from_pretrained(NAME)
model.eval()
import random

random.seed(3)

CATEGORY_WORDS = {
'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
'the {part} is {opinion}, {tail}',
'arrived quickly and the {part} was {opinion}, {tail}',
'i wanted to like it but the {part} is {opinion}, {tail}',
'after a month the {part} is still {opinion}, {tail}',
'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
'second one i have bought', 'delivery took a week',
'as described in the listing']

def make_review(category, positive):
part = random.choice(CATEGORY_WORDS[category])
# Half the positive reviews are written as a negated negative, so
# 'not flimsy' has to be read as praise. Bag of words cannot.
if positive:
if random.random() < 0.5:
opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
else:
opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
tail = random.choice(TAIL)
else:
if random.random() < 0.5:
opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
else:
opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
tail = random.choice(TAIL)
return random.choice(FRAME).format(part=part, opinion=opinion,
tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
for cat in CATEGORY_WORDS:
for pos in (True, False):
texts.append(make_review(cat, pos))
sentiment.append(1 if pos else 0)
category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
train_c, test_c) = train_test_split(texts, sentiment, category,
test_size=0.3, random_state=0,
stratify=sentiment)
import time
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

def timed(fn):
start = time.perf_counter()
score = fn()
return score, time.perf_counter() - start

def unigrams():
m = make_pipeline(CountVectorizer(),
LogisticRegression(max_iter=2000))
m.fit(train_x, train_s)
return m.score(test_x, test_s)

def bigrams():
m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
LogisticRegression(max_iter=2000))
m.fit(train_x, train_s)
return m.score(test_x, test_s)

def transformer():
def embed(sentences, batch=32):
out = []
for i in range(0, len(sentences), batch):
enc = tok(sentences[i:i + batch], return_tensors='pt',
padding=True, truncation=True, max_length=48)
with torch.no_grad():
h = model(**enc).last_hidden_state
mask = enc['attention_mask'].unsqueeze(-1)
out.append(((h * mask).sum(1) / mask.sum(1)).numpy())
return np.vstack(out)
clf = LogisticRegression(max_iter=3000).fit(embed(train_x), train_s)
return clf.score(embed(test_x), test_s)

print('%-30s %10s %10s' % ('', 'accuracy', 'seconds'))
for name, fn in [('bag of words, week 4', unigrams),
('tfidf bigrams, week 4', bigrams),
('frozen transformer, week 7', transformer)]:
score, took = timed(fn)
print('%-30s %10.4f %10.2f' % (name, score, took))
accuracy seconds
bag of words, week 4 0.4931 0.03
tfidf bigrams, week 4 1.0000 0.03
frozen transformer, week 7 0.6236 0.62

Three generations of technique on one task, with the time each took. The middle row is the best value by a wide margin here, and that ordering is specific to this corpus rather than general. What is general is that the comparison is cheap to run and almost nobody runs it.

What the language half established

  • Text becomes numbers before anything else happens, and that step decides more than the model does.
  • Word order is worth everything on some tasks and nothing on others. Measure which you have.
  • Meaning can be learned from context alone, and compressed into vectors where related words sit near each other.
  • Contextual models solve ambiguity and order together, which is why they replaced static vectors.
  • Generation is next token prediction, which is why fluency and accuracy are unrelated in these systems.
  • The expensive method is not automatically better, and the comparison takes an afternoon.

Going into the vision weeks

Weeks 8 to 10 do the same journey for images: raw pixels, then features somebody designed, then features learned from data. The arc is deliberately the same, because the lesson is the same one, and the habits from this half transfer intact.