Pretrained Transformers

Week 11 of 18 · Transformers · 7 days

Week 11 · Transformers

Pretrained Transformers

Week 11 · Day 1 of 7

Why Not Train Your Own

The task, the baseline, and the case for starting from somebody else's weights

Week 10 trained a language model from nothing and produced something that had learned an invented language and nothing else. The alternative is to start from a model somebody has already trained on a great deal of real text, which is what almost everybody actually does.

The data

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
print('training reviews %d, balance %.3f'
      % (len(train_texts), train_y.float().mean()))
print()
for i in range(6):
    print('[%s] %s' % ('positive' if train_y[i] else 'negative',
                       train_texts[i]))

training reviews 1200, balance 0.518

[negative] the delivery is faulty.
[positive] the battery is sturdy.
[positive] Honestly, the motor is comfortable.
[negative] the screen is broken.
[negative] Honestly, the fabric is faulty in daily use.
[positive] On the whole the software is reliable overall.

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
print('reviews using the training vocabulary:')
for i in range(2):
    print('  [%s] %s' % ('positive' if seen_y[i] else 'negative',
                         seen_texts[i]))
print('\nreviews using words the training set never contained:')
for i in range(2):
    print('  [%s] %s' % ('positive' if new_y[i] else 'negative',
                         new_texts[i]))
print('\ntraining adjectives:', TRAIN_GOOD + TRAIN_BAD)
print('new adjectives     :', NEW_GOOD + NEW_BAD)

reviews using the training vocabulary:
  [positive] On the whole the delivery is sturdy.
  [positive] Honestly, the motor is sturdy so far.

reviews using words the training set never contained:
  [negative] Honestly, the delivery is defective.
  [positive] I found that the delivery is pleasant for the price.

training adjectives: ['excellent', 'reliable', 'comfortable', 'sturdy', 'broken', 'flimsy', 'noisy', 'faulty']
new adjectives     : ['superb', 'dependable', 'pleasant', 'robust', 'damaged', 'fragile', 'loud', 'defective']

A bag of words baseline

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

print('%-16s %14s %14s' % ('', 'same words', 'new words'))
for label, ngrams in [('unigrams', (1, 1)), ('1 to 2 grams', (1, 2))]:
    model = make_pipeline(TfidfVectorizer(ngram_range=ngrams),
                          LogisticRegression(max_iter=2000))
    model.fit(train_texts, train_y.numpy())
    print('%-16s %14.4f %14.4f'
          % (label, model.score(seen_texts, seen_y.numpy()),
             model.score(new_texts, new_y.numpy())))

                     same words      new words
unigrams                 1.0000         0.5000
1 to 2 grams             1.0000         0.4925

The second column is the whole point of this week

A bag of words does well on reviews written with the words it was trained on, and falls to chance on reviews written with synonyms. It has not learned that a product can be good or bad. It has learned that the string reliable predicts one label, and nothing in the method could have done otherwise.

A model pretrained on a great deal of English already knows that dependable and reliable are used in similar places. That knowledge is what you are borrowing, and the rest of this week is about how to borrow it.

Day 1 takeaway

A bag of words can only recognise the vocabulary it was trained on. Evaluate on words your training set never contained and the gap between memorising strings and understanding language becomes a number. That gap is what a pretrained model closes.

Week 11 · Day 2 of 7

Loading a Pretrained Model

Weights, tokeniser and config, and what tokenisation does to your text

By Pawan Rama Mali627 words

A pretrained model arrives as three things: weights, a tokeniser, and a configuration. All three have to match, and the library keeps them together for exactly that reason.

Loading one

from transformers import AutoTokenizer, AutoModel
import torch

NAME = 'prajjwal1/bert-mini'
tok = AutoTokenizer.from_pretrained(NAME)
model = AutoModel.from_pretrained(NAME)

print('model      %s' % NAME)
print('parameters %s' % '{:,}'.format(sum(p.numel()
                                          for p in model.parameters())))
print('vocabulary %s' % '{:,}'.format(tok.vocab_size))
print('hidden size %d, layers %d, heads %d'
      % (model.config.hidden_size, model.config.num_hidden_layers,
         model.config.num_attention_heads))

model      prajjwal1/bert-mini
parameters 11,170,560
vocabulary 30,522
hidden size 256, layers 4, heads 4

The tokeniser is part of the model

The weights were trained against one specific vocabulary, one specific way of splitting words, and specific special tokens. Load weights from one model and a tokeniser from another and every token id means something different. Nothing raises. You get a model that runs and produces noise. Always load both from the same name, which is what AutoTokenizer and AutoModel exist to make easy.

What tokenisation actually does to your text

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('prajjwal1/bert-mini')

for text in ['the battery is not reliable',
             'the packaging is disappointing',
             'unbelievably discombobulated']:
    ids = tok(text)['input_ids']
    print('%-34s -> %d tokens' % (text, len(ids)))
    print('   %s' % tok.convert_ids_to_tokens(ids))

the battery is not reliable        -> 7 tokens
   ['[CLS]', 'the', 'battery', 'is', 'not', 'reliable', '[SEP]']
the packaging is disappointing     -> 6 tokens
   ['[CLS]', 'the', 'packaging', 'is', 'disappointing', '[SEP]']
unbelievably discombobulated       -> 11 tokens
   ['[CLS]', 'un', '##bel', '##ie', '##va', '##bly', 'disco', '##mbo', '##bula', '##ted', '[SEP]']

Note the [CLS] and [SEP] tokens the tokeniser adds, and the ## prefixes marking word continuations. A common word is one token; an unusual one is several pieces. Nothing is ever unknown, which is the property day 5 of week 10 built by hand.

Running it

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained('prajjwal1/bert-mini')
model = AutoModel.from_pretrained('prajjwal1/bert-mini')
model.eval()

batch = tok(train_texts[:4], padding=True, truncation=True,
            max_length=32, return_tensors='pt')
print('input_ids     ', tuple(batch['input_ids'].shape))
print('attention_mask', tuple(batch['attention_mask'].shape))
print('\nthe mask marks real tokens, so padding is ignored:')
print(batch['attention_mask'])

with torch.no_grad():
    out = model(**batch)
print('\nlast_hidden_state', tuple(out.last_hidden_state.shape))
print('one vector per token per example.')

input_ids      (4, 9)
attention_mask (4, 9)

the mask marks real tokens, so padding is ignored:
tensor([[1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 0, 0]])

last_hidden_state (4, 9, 256)
one vector per token per example.

Day 2 takeaway

Load the weights, the tokeniser and the configuration together from the same name. The tokeniser adds special tokens and splits unusual words into pieces. The model returns one vector per token, and the attention mask is what stops padding being read.

Week 11 · Day 3 of 7

Frozen Features

One forward pass, mean pooling under the mask, and a classifier on top

By Pawan Rama Mali794 words

The cheapest way to use a pretrained model is to freeze it entirely and treat its output as features for something else. It takes one forward pass over your data and no training of the transformer at all.

Sentence vectors

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained('prajjwal1/bert-mini')
bert = AutoModel.from_pretrained('prajjwal1/bert-mini')
bert.eval()

def embed(texts, how='mean', batch_size=64):
    out = []
    for i in range(0, len(texts), batch_size):
        batch = tok(texts[i:i + batch_size], padding=True,
                    truncation=True, max_length=32, return_tensors='pt')
        with torch.no_grad():
            hidden = bert(**batch).last_hidden_state
        if how == 'cls':
            out.append(hidden[:, 0, :])
        else:
            mask = batch['attention_mask'].unsqueeze(-1)
            out.append((hidden * mask).sum(1) / mask.sum(1))
    return torch.cat(out)

vectors = embed(train_texts[:200])
print('embeddings', tuple(vectors.shape))
print('one vector per review, of width %d' % vectors.shape[1])

embeddings (200, 256)
one vector per review, of width 256

Mean pooling beats the CLS token unless the model was trained for it

The [CLS] vector is only a sentence representation if the model was trained to make it one, which BERT was, for its next-sentence task, and most models since are not. Averaging the token vectors under the attention mask is the more reliable default, and it is what sentence embedding models do. Note the mask in that average: without it you are averaging in the padding.

Frozen features against the baseline

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
from transformers import AutoTokenizer, AutoModel
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline
import time

tok = AutoTokenizer.from_pretrained('prajjwal1/bert-mini')
bert = AutoModel.from_pretrained('prajjwal1/bert-mini')
bert.eval()

def embed(texts, batch_size=64):
    out = []
    for i in range(0, len(texts), batch_size):
        batch = tok(texts[i:i + batch_size], padding=True,
                    truncation=True, max_length=32, return_tensors='pt')
        with torch.no_grad():
            hidden = bert(**batch).last_hidden_state
        mask = batch['attention_mask'].unsqueeze(-1)
        out.append((hidden * mask).sum(1) / mask.sum(1))
    return torch.cat(out).numpy()

start = time.time()
Xtr, Xva = embed(train_texts), embed(new_texts)
embed_time = time.time() - start

print('evaluated on the vocabulary never seen in training')
print('%-30s %10s' % ('', 'accuracy'))
tfidf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                      LogisticRegression(max_iter=2000))
tfidf.fit(train_texts, train_y.numpy())
print('%-30s %10.4f' % ('tf-idf 1 to 2 grams',
                        tfidf.score(new_texts, new_y.numpy())))

clf = LogisticRegression(max_iter=2000).fit(Xtr, train_y.numpy())
print('%-30s %10.4f' % ('frozen transformer features',
                        clf.score(Xva, new_y.numpy())))
print('\nembedding 1600 reviews took %.0f seconds' % embed_time)

evaluated on the vocabulary never seen in training
                                 accuracy
tf-idf 1 to 2 grams                0.4925
frozen transformer features        0.8825

embedding 1600 reviews took 1 seconds

Day 3 takeaway

Freezing the model and using its output as features costs one forward pass and no training. Average the token vectors under the attention mask rather than taking the CLS token. It is the first thing to try, and on a task with a strong local signal a bigram model can still beat it.

Week 11 · Day 4 of 7

Fine-tuning

The new head, and the learning rate that is a hundred times smaller

By Pawan Rama Mali614 words

Fine-tuning updates the pretrained weights on your task. It costs considerably more than freezing and is usually worth it.

The classification head

from transformers import AutoModelForSequenceClassification
import torch

model = AutoModelForSequenceClassification.from_pretrained(
    'prajjwal1/bert-mini', num_labels=2)
print('the classifier that was added:')
print(' ', model.classifier)
print('\nit is randomly initialised, which is why the library warns you.')
print('everything below it carries the pretrained weights.')
total = sum(p.numel() for p in model.parameters())
head = sum(p.numel() for p in model.classifier.parameters())
print('\nparameters: %s total, %s in the new head'
      % ('{:,}'.format(total), '{:,}'.format(head)))

the classifier that was added:
  Linear(in_features=256, out_features=2, bias=True)

it is randomly initialised, which is why the library warns you.
everything below it carries the pretrained weights.

parameters: 11,171,074 total, 514 in the new head

Fine-tuning it

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from torch.utils.data import DataLoader, TensorDataset
from torch import nn
import time

tok = AutoTokenizer.from_pretrained('prajjwal1/bert-mini')

def encode(texts):
    b = tok(texts, padding='max_length', truncation=True, max_length=24,
            return_tensors='pt')
    return b['input_ids'], b['attention_mask']

ids_tr, mask_tr = encode(train_texts)
ids_va, mask_va = encode(new_texts)
loader = DataLoader(TensorDataset(ids_tr, mask_tr, train_y),
                    batch_size=32, shuffle=True)

torch.manual_seed(0)
model = AutoModelForSequenceClassification.from_pretrained(
    'prajjwal1/bert-mini', num_labels=2)
opt = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.01)
steps = 3 * len(loader)
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=5e-5,
                                            total_steps=steps,
                                            pct_start=0.1)

def score():
    model.eval()
    with torch.no_grad():
        logits = model(input_ids=ids_va, attention_mask=mask_va).logits
    return (logits.argmax(1) == new_y).float().mean().item()

start = time.time()
for epoch in range(1, 4):
    model.train()
    for ids, mask, y in loader:
        opt.zero_grad()
        out = model(input_ids=ids, attention_mask=mask, labels=y)
        out.loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        sched.step()
    print('epoch %d  accuracy %.4f' % (epoch, score()))
print('\n%.0f seconds on the CPU' % (time.time() - start))

epoch 1  accuracy 0.7525
epoch 2  accuracy 0.6975
epoch 3  accuracy 0.7200

25 seconds on the CPU

The learning rate is a hundred times smaller than you are used to

Fine-tuning uses rates around 1e-5 to 5e-5, not the 1e-3 that trains a model from scratch. The weights are already good, and the job is to adjust them slightly rather than to find them. A rate of 1e-3 on a pretrained transformer destroys the pretraining in the first few hundred steps, which shows up as a model that trains to roughly the accuracy you would get from random initialisation.

Warmup matters here too, for the reason week 9 gave: the classification head is random, and its early gradients are large.

Day 4 takeaway

AutoModelForSequenceClassification adds a randomly initialised head to the pretrained body. Fine-tune the whole thing at 1e-5 to 5e-5 with a short warmup and gradient clipping. The learning rate is the single most important difference from training from scratch.

Week 11 · Day 5 of 7

The Three Approaches Compared

Accuracy and cost side by side on the same data

By Pawan Rama Mali703 words

The three approaches compared on the same data, with the costs, so the choice is a decision rather than a habit.

The comparison

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
from transformers import (AutoTokenizer, AutoModel,
                          AutoModelForSequenceClassification)
from torch.utils.data import DataLoader, TensorDataset
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline
import time

NAME = 'prajjwal1/bert-mini'
tok = AutoTokenizer.from_pretrained(NAME)
results = []

# 1. bag of words
start = time.time()
m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                  LogisticRegression(max_iter=2000))
m.fit(train_texts, train_y.numpy())
results.append(('tf-idf bigrams', m.score(new_texts, new_y.numpy()),
                time.time() - start))

# 2. frozen features
bert = AutoModel.from_pretrained(NAME).eval()
def embed(texts):
    out = []
    for i in range(0, len(texts), 64):
        b = tok(texts[i:i + 64], padding=True, truncation=True,
                max_length=24, return_tensors='pt')
        with torch.no_grad():
            h = bert(**b).last_hidden_state
        mask = b['attention_mask'].unsqueeze(-1)
        out.append((h * mask).sum(1) / mask.sum(1))
    return torch.cat(out).numpy()
start = time.time()
clf = LogisticRegression(max_iter=2000).fit(embed(train_texts),
                                            train_y.numpy())
results.append(('frozen features', clf.score(embed(new_texts),
                                             new_y.numpy()),
                time.time() - start))

# 3. fine-tuned
def encode(texts):
    b = tok(texts, padding='max_length', truncation=True, max_length=24,
            return_tensors='pt')
    return b['input_ids'], b['attention_mask']
ids_tr, mask_tr = encode(train_texts)
ids_va, mask_va = encode(new_texts)
torch.manual_seed(0)
model = AutoModelForSequenceClassification.from_pretrained(NAME,
                                                           num_labels=2)
opt = torch.optim.AdamW(model.parameters(), lr=5e-5)
loader = DataLoader(TensorDataset(ids_tr, mask_tr, train_y),
                    batch_size=32, shuffle=True)
start = time.time()
for _ in range(3):
    model.train()
    for ids, mask, y in loader:
        opt.zero_grad()
        model(input_ids=ids, attention_mask=mask, labels=y).loss.backward()
        opt.step()
model.eval()
with torch.no_grad():
    acc = (model(input_ids=ids_va,
                 attention_mask=mask_va).logits.argmax(1)
           == new_y).float().mean().item()
results.append(('fine-tuned', acc, time.time() - start))

print('all three evaluated on the new vocabulary')
print('%-22s %10s %10s' % ('', 'accuracy', 'seconds'))
for name, acc, secs in results:
    print('%-22s %10.4f %10.0f' % (name, acc, secs))

all three evaluated on the new vocabulary
                         accuracy    seconds
tf-idf bigrams             0.4925          0
frozen features            0.8825          2
fine-tuned                 0.8100         24

Fine-tuning came out behind the frozen features

That is not the textbook ordering, and the reason is worth understanding. Fine-tuning adjusts the pretrained weights towards the training vocabulary, and the evaluation deliberately uses different words. Some of what it learns is reliable means positive, which is exactly the knowledge that does not transfer, and it partially overwrites the general knowledge that would have.

Freezing cannot do that, because the representation never moves. So when your evaluation differs from your training data in a way that matters, frozen features are more robust and fine-tuning is a way of specialising towards data you may not have. Run both. The ordering is not a law.

Approach	Cost	Use when
Bag of words	Seconds	Always, as the baseline. Often enough on its own
Frozen features	One forward pass	Few labels, many tasks over the same text, or no GPU
Fine-tune the head only	Minutes	Rarely worth it; either freeze everything or tune everything
Fine-tune everything	Minutes to hours	A few thousand labels and accuracy that justifies the cost
A hosted large model	An API bill	Very few labels, or a genuinely open-ended task

Day 5 takeaway

Bag of words first, then frozen features, then fine-tuning, and stop as soon as one is good enough. On a small task with a strong local pattern the cheapest option frequently wins, and the only way to know is to run all three.

Week 11 · Day 6 of 7

Parameter Efficient Fine-tuning

What training memory really costs, and how LoRA avoids most of it

By Pawan Rama Mali605 words

Fine-tuning every weight of a large model is expensive in memory as well as time. Two techniques make it affordable, and one of them has become the default.

What full fine-tuning costs

def memory_gb(params, optimiser='adamw'):
    weights = params * 4
    grads = params * 4
    state = params * 8 if optimiser == 'adamw' else 0
    return (weights + grads + state) / 1e9

print('%-16s %14s %14s %14s'
      % ('model', 'parameters', 'inference GB', 'training GB'))
for name, params in [('bert-tiny', 4.4e6), ('bert-base', 110e6),
                     ('7 billion', 7e9), ('70 billion', 70e9)]:
    print('%-16s %14s %14.2f %14.1f'
          % (name, '{:,.0f}'.format(params), params * 4 / 1e9,
             memory_gb(params)))
print('\ntraining needs about four times what inference needs, because')
print('the gradients and the optimiser state are each the size of the')
print('weights. That is what puts full fine-tuning out of reach.')

model                parameters   inference GB    training GB
bert-tiny             4,400,000           0.02            0.1
bert-base           110,000,000           0.44            1.8
7 billion         7,000,000,000          28.00          112.0
70 billion       70,000,000,000         280.00         1120.0

training needs about four times what inference needs, because
the gradients and the optimiser state are each the size of the
weights. That is what puts full fine-tuning out of reach.

Low rank adaptation

LoRA: Freeze the pretrained weights entirely and add a small trainable update to chosen layers, expressed as the product of two thin matrices. A weight of shape 768 by 768 gets an update built from 768 by 8 and 8 by 768, which is fifty times fewer numbers to train and to store.

import torch
from torch import nn

class LoRALinear(nn.Module):
    def __init__(self, base, rank=8, alpha=16):
        super().__init__()
        self.base = base
        for prm in self.base.parameters():
            prm.requires_grad = False              # frozen
        d_in, d_out = base.in_features, base.out_features
        self.a = nn.Parameter(torch.randn(rank, d_in) * 0.01)
        self.b = nn.Parameter(torch.zeros(d_out, rank))
        self.scale = alpha / rank

    def forward(self, x):
        return self.base(x) + (x @ self.a.T @ self.b.T) * self.scale

torch.manual_seed(0)
base = nn.Linear(768, 768)
wrapped = LoRALinear(base, rank=8)

trainable = sum(p.numel() for p in wrapped.parameters()
                if p.requires_grad)
frozen = sum(p.numel() for p in wrapped.parameters()
             if not p.requires_grad)
print('frozen    %s' % '{:,}'.format(frozen))
print('trainable %s' % '{:,}'.format(trainable))
print('ratio     %.1f to 1' % (frozen / trainable))

x = torch.randn(2, 768)
print('\nb starts at zero, so at step 0 the wrapper is the identity:',
      bool(torch.allclose(wrapped(x), base(x))))

frozen    590,592
trainable 12,288
ratio     48.1 to 1

b starts at zero, so at step 0 the wrapper is the identity: True

Starting from zero is the detail that makes it work

One of the two matrices is initialised to zero, so the adapter contributes nothing at the first step and the model behaves exactly as the pretrained one did. Training then moves away from that point gradually. Initialise both randomly and you have corrupted the pretrained model before the first gradient arrives, which is the same mistake as unfreezing a backbone under a random head in week 6.

Adapters in practice

Method	Trainable share	Notes
Full fine-tuning	100%	Best results, largest cost
LoRA	Around 0.1 to 1%	The default now; adapters can be merged into the weights afterwards
QLoRA	Around 0.1%	LoRA on a 4-bit quantised base; fits large models on one card
Prompt tuning	A few thousand	Learns input vectors only; weaker but very cheap
Head only	Under 1%	Rarely competitive; usually beaten by frozen features plus a stronger classifier

Day 6 takeaway

Training memory is roughly four times inference memory because of gradients and optimiser state. LoRA freezes the base and trains a thin low-rank update, cutting trainable parameters by two or three orders of magnitude. Initialise one of its matrices to zero so the adapter starts as the identity.

Week 11 · Day 7 of 7

Choosing a Model

The families, the checklist, and how the answer moves with label count

By Pawan Rama Mali793 words

Choosing a pretrained model, and the questions worth asking before you build anything on top of one.

What is actually on offer

Family	Shape	Good for
BERT and descendants	Encoder only	Classification, retrieval, token labelling
Sentence transformers	Encoder, trained for similarity	Search, clustering, deduplication
GPT family	Decoder only	Generation, and everything else now
T5 and BART	Encoder and decoder	Summarising, translation, structured rewriting
CLIP	Two encoders, images and text	Zero-shot image classification, search across both
Whisper	Encoder and decoder, audio in	Transcription

A checklist before you commit to one

Licence. Some weights forbid commercial use, and some are trained on data with its own restrictions. Check before you build, not after.
Size against your hardware. Four bytes per parameter for inference, roughly sixteen for full fine-tuning.
The language and domain it was trained on. A model trained on English web text is a poor starting point for Hindi clinical notes.
Context length, if your documents are long.
Whether a smaller one is enough. Measure it. A tiny model that is good enough is better than a large one that is marginally better.
What it does when it is wrong. A generative model produces confident text either way, which is a different failure shape from a classifier returning a probability.

Everything, on one task

import torch

# Two vocabularies. The model trains on one and is evaluated on both,
# which is what separates a model that learned words from a model that
# learned what the words mean.
TRAIN_GOOD = ['excellent', 'reliable', 'comfortable', 'sturdy']
TRAIN_BAD = ['broken', 'flimsy', 'noisy', 'faulty']
NEW_GOOD = ['superb', 'dependable', 'pleasant', 'robust']
NEW_BAD = ['damaged', 'fragile', 'loud', 'defective']

THING = ['the delivery', 'the battery', 'the screen', 'the handle',
         'the packaging', 'the software', 'the fabric', 'the motor']
OPENER = ['I found that ', 'Honestly, ', 'After a week, ', '',
          'On the whole ']
CLOSER = [' overall.', ' for the price.', ' so far.', '.',
          ' in daily use.']

def make_reviews(n, good, bad, seed):
    """Short product reviews, balanced, with the sentiment carried by
    one adjective."""
    g = torch.Generator().manual_seed(seed)
    def pick(xs):
        return xs[int(torch.randint(len(xs), (1,), generator=g))]
    texts, labels = [], []
    for _ in range(n):
        positive = int(torch.randint(2, (1,), generator=g))
        word = pick(good if positive else bad)
        texts.append(pick(OPENER) + pick(THING) + ' is ' + word
                     + pick(CLOSER))
        labels.append(positive)
    return texts, torch.tensor(labels)

train_texts, train_y = make_reviews(1200, TRAIN_GOOD, TRAIN_BAD, 0)
seen_texts, seen_y = make_reviews(400, TRAIN_GOOD, TRAIN_BAD, 1)
new_texts, new_y = make_reviews(400, NEW_GOOD, NEW_BAD, 2)
from transformers import (AutoTokenizer, AutoModel,
                          AutoModelForSequenceClassification)
from torch.utils.data import DataLoader, TensorDataset
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline

NAME = 'prajjwal1/bert-mini'
tok = AutoTokenizer.from_pretrained(NAME)

# how the three approaches behave as labels get scarce
def tfidf_score(n):
    m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                      LogisticRegression(max_iter=2000))
    m.fit(train_texts[:n], train_y[:n].numpy())
    return m.score(new_texts, new_y.numpy())

bert = AutoModel.from_pretrained(NAME).eval()
def embed(texts):
    out = []
    for i in range(0, len(texts), 64):
        b = tok(texts[i:i + 64], padding=True, truncation=True,
                max_length=24, return_tensors='pt')
        with torch.no_grad():
            h = bert(**b).last_hidden_state
        mask = b['attention_mask'].unsqueeze(-1)
        out.append((h * mask).sum(1) / mask.sum(1))
    return torch.cat(out).numpy()
Etr, Eva = embed(train_texts), embed(new_texts)

def frozen_score(n):
    clf = LogisticRegression(max_iter=2000).fit(Etr[:n],
                                                train_y[:n].numpy())
    return clf.score(Eva, new_y.numpy())

def encode(texts):
    b = tok(texts, padding='max_length', truncation=True, max_length=24,
            return_tensors='pt')
    return b['input_ids'], b['attention_mask']
ids_va, mask_va = encode(new_texts)

def finetune_score(n, epochs=3):
    ids, mask = encode(train_texts[:n])
    torch.manual_seed(0)
    model = AutoModelForSequenceClassification.from_pretrained(
        NAME, num_labels=2)
    opt = torch.optim.AdamW(model.parameters(), lr=5e-5)
    loader = DataLoader(TensorDataset(ids, mask, train_y[:n]),
                        batch_size=16, shuffle=True)
    for _ in range(epochs):
        model.train()
        for a, b_, y in loader:
            opt.zero_grad()
            model(input_ids=a, attention_mask=b_,
                  labels=y).loss.backward()
            opt.step()
    model.eval()
    with torch.no_grad():
        return (model(input_ids=ids_va,
                      attention_mask=mask_va).logits.argmax(1)
                == new_y).float().mean().item()

print('%10s %14s %16s %14s'
      % ('labels', 'tf-idf', 'frozen bert', 'fine-tuned'))
for n in [50, 200, 1200]:
    print('%10d %14.4f %16.4f %14.4f'
          % (n, tfidf_score(n), frozen_score(n), finetune_score(n)))

    labels         tf-idf      frozen bert     fine-tuned
        50         0.4725           0.8300         0.5325
       200         0.5175           0.8925         0.8625
      1200         0.4925           0.8825         0.7475

The label count is what decides this

That table is the whole argument for pretraining in one place. With very few labels the pretrained model has an advantage, because it already knows what words mean and only has to learn the task. With plenty of labels a bag of words catches up, because the task is simple enough to learn from the data alone.

Run this experiment on your own problem before deciding. The answer moves with the number of labels, the difficulty of the task and how far your text is from what the model was pretrained on.

Day 7 takeaway

Check the licence, the size against your hardware, and the domain it was pretrained on. Then measure bag of words, frozen features and fine-tuning across a range of label counts, because which one wins depends on how many labels you have and nothing else predicts it reliably.