Text, Embeddings and Sequence Models

Week 13 of 16 · Deep learning · 7 days

Week 13 · Deep learning

Text, Embeddings and Sequence Models

Week 13 · Day 1 of 7

Turning Text Into Numbers

Tokens, vocabularies, sparse matrices and the information you throw away

Every model so far has been handed numbers. Text is not numbers, and the whole of this week is about the step in between, because that step, not the choice of classifier, is what decides whether the model works.

The corpus

The broadband company whose customers you have been predicting for twelve weeks has a support inbox. This is it: 4,200 tickets, each with a free-text body, a category, and a flag for whether the customer is describing a problem that is now over. Generate it the same way you generated customers.csv in week 1.

import io
import os

import numpy as np
import pandas as pd

SEED = 13
N = 4200

THING = ["router", "connection", "line", "broadband", "wifi", "signal",
         "hub", "internet"]

# The two halves of the construction. A positive state word with no negation
# means the problem is over; the same word negated means it is not.
GOOD_STATE = ["working", "stable", "fixed", "fine", "sorted", "reliable",
              "back to normal"]
BAD_STATE = ["slow", "down", "unstable", "broken", "patchy", "unreliable",
             "dropping out"]

# Every one of these appears equally often on both sides of the label.
MODIFIER = ["", "still ", "not ", "no longer "]
NEGATING = {"not ", "no longer "}          # these flip the state word

FRAME = [
    "the {thing} is {mod}{state}",
    "my {thing} is {mod}{state}",
    "it is {mod}{state}",
    "the {thing} has been {mod}{state} {when}",
    "as of this morning the {thing} is {mod}{state}",
    "{when} the {thing} is {mod}{state}",
]

WHEN = ["every evening", "since tuesday", "all weekend", "for three weeks",
        "in the mornings", "since the storm", "most nights"]

# Context sentences carry the category and nothing else -- no resolution
# information at all, so the two labels stay genuinely different tasks.
CONTEXT = {
    "technical": [
        "the engineer visited on tuesday",
        "this is the third time i have written about the hub",
        "i have restarted the equipment twice",
        "the speed test reads well under what i pay for",
        "there is a flashing red light on the box",
    ],
    "billing": [
        "this relates to invoice 40182",
        "the account is on the fibre tariff",
        "i pay by direct debit on the eighth",
        "there is a charge on the statement i do not recognise",
        "the promotional discount was supposed to run for a year",
    ],
    "cancellation": [
        "i have been a customer for six years",
        "i am considering my options",
        "my contract ends next month",
        "a competitor has offered me a better package",
        "i would like to know the termination fee",
    ],
}

GREETING = ["", "", "", "hi team, ", "hello, ", "hi, ", "morning, ", "hi there "]
SIGNOFF = ["", "", "", "", " thanks", " please advise", " regards", " ta",
           " please call me back"]

TYPOS = {"the": "teh", "and": "adn", "connection": "conection",
         "internet": "internt", "please": "plese", "invoice": "invoce"}


def _core(rng, resolved):
    """One clause whose meaning comes from the modifier-plus-state pairing.

    Pick the modifier first and independently of the label, then choose the
    state word that makes the clause mean what the label says. That ordering
    is what keeps every modifier balanced across both classes -- generate it
    the other way round and the modifier leaks the answer to a unigram model.
    """
    mod = rng.choice(MODIFIER)
    good_state = resolved if mod not in NEGATING else not resolved
    state = rng.choice(GOOD_STATE if good_state else BAD_STATE)
    return rng.choice(FRAME).format(thing=rng.choice(THING), mod=mod,
                                    state=state, when=rng.choice(WHEN))


def _rough(rng, text):
    """Put the mess back in: casing, punctuation, typos."""
    if rng.random() < 0.10:
        words = text.split()
        for i, w in enumerate(words):
            if w in TYPOS and rng.random() < 0.5:
                words[i] = TYPOS[w]
        text = " ".join(words)
    if rng.random() < 0.12:
        text = text.upper()
    elif rng.random() < 0.25:
        text = text.capitalize()
    text = rng.choice(GREETING) + text + rng.choice(SIGNOFF)
    text += rng.choice([".", ".", "", "!", "!!", "...", " ."])
    if rng.random() < 0.08:
        text = "  " + text + "  "
    return text


def build(seed=SEED, n=N):
    rng = np.random.default_rng(seed)
    cats = rng.choice(["technical", "billing", "cancellation"], size=n,
                      p=[0.52, 0.33, 0.15])
    rows = []
    for i, cat in enumerate(cats):
        resolved = int(rng.random() < 0.42)
        parts = [_core(rng, resolved)]
        # A second clause of the same polarity. Mixed polarity would leave the
        # ticket with no correct answer, so it never happens.
        if rng.random() < 0.30:
            parts.append(_core(rng, resolved))
        # The category sentence, which says nothing about resolution.
        if rng.random() < 0.75:
            where = rng.integers(0, len(parts) + 1)
            parts.insert(int(where), rng.choice(CONTEXT[cat]))
        text = _rough(rng, ". ".join(parts))
        rows.append({"ticket_id": "T%05d" % (i + 1), "text": text,
                     "category": cat, "resolved": resolved})

    df = pd.DataFrame(rows)

    # ~4% label noise, so a perfect score is impossible and anybody who gets
    # one goes looking for the leak instead of celebrating.
    flip = rng.random(len(df)) < 0.04
    df.loc[flip, "resolved"] = 1 - df.loc[flip, "resolved"]

    # a few empty and near-empty tickets
    blank = rng.choice(len(df), size=18, replace=False)
    df.loc[blank[:9], "text"] = ""
    df.loc[blank[9:], "text"] = rng.choice(["?", "...", "help", "hi", "!!"],
                                           size=len(blank) - 9)

    # double submissions
    dupes = df.iloc[rng.choice(len(df), size=90, replace=False)]
    df = pd.concat([df, dupes], ignore_index=True)
    df = df.iloc[rng.permutation(len(df))].reset_index(drop=True)
    return df

build().to_csv('tickets.csv', index=False, encoding='utf-8')

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
print('tickets %d' % len(tickets))
print(tickets[['ticket_id', 'category', 'resolved']].head())
print('\nresolved rate %.3f' % tickets['resolved'].mean())
print('categories:')
print(tickets['category'].value_counts())

tickets 4200
  ticket_id      category  resolved
0    T02210       billing         0
1    T00107       billing         1
2    T00680       billing         1
3    T03436  cancellation         0
4    T02257       billing         1

resolved rate 0.425
categories:
category
technical       2144
billing         1429
cancellation     627
Name: count, dtype: int64

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
for _, r in tickets.head(6).iterrows():
    print('[%-12s resolved=%d]  %s'
          % (r['category'], r['resolved'], r['text'][:78]))

[billing      resolved=0]  AS OF THIS MORNING THE HUB IS PATCHY. MOST NIGHTS THE HUB IS NO LONGER WORKING
[billing      resolved=1]  hello, my hub is not unstable. my wifi is still fixed.
[billing      resolved=1]  hello, There is a charge on the statement i do not recognise. it is working. t
[cancellation resolved=0]  the hub is down please advise
[billing      resolved=1]  hi, the account is on the fibre tariff. as of this morning the router is no lo
[billing      resolved=0]  in the mornings the wifi is unstable please advise .

Look at the raw strings before you clean anything

Shouting in capitals, doubled exclamation marks, leading spaces, typos, and nine tickets that are empty. Every one of those is in the file on purpose, because every one of them is in real inboxes. The instinct to normalise it all away immediately is the right instinct and the wrong first move, some of that mess is signal. Angry customers do write in capitals.

Tokenising

Token: The unit a model actually sees. Usually a word, sometimes a word-piece, occasionally a character. Deciding what counts as a token is the first modelling decision you make about text, and it is made before any model exists.

text = '  THE ROUTER is Not working!! since tuesday.  '

print('raw        %r' % text)
print('stripped   %r' % text.strip())
print('lowered    %r' % text.strip().lower())
print('naive split %s' % text.strip().lower().split())

import re
tokens = re.findall(r"[a-z0-9']+", text.lower())
print('regex      %s' % tokens)

raw        '  THE ROUTER is Not working!! since tuesday.  '
stripped   'THE ROUTER is Not working!! since tuesday.'
lowered    'the router is not working!! since tuesday.'
naive split ['the', 'router', 'is', 'not', 'working!!', 'since', 'tuesday.']
regex      ['the', 'router', 'is', 'not', 'working', 'since', 'tuesday']

split() leaves working!! and tuesday. as tokens distinct from working and tuesday, which quietly doubles parts of your vocabulary and halves the evidence for each. The regex keeps letters, digits and apostrophes and drops the rest.

Building a bag of words by hand

import re
from collections import Counter

docs = ['the router is not working',
        'the router is working',
        'my line is not slow']

tokenised = [re.findall(r"[a-z0-9']+", d.lower()) for d in docs]
vocab = sorted({t for doc in tokenised for t in doc})
index = {t: i for i, t in enumerate(vocab)}
print('vocabulary (%d): %s' % (len(vocab), vocab))

import numpy as np
M = np.zeros((len(docs), len(vocab)), dtype=int)
for r, doc in enumerate(tokenised):
    for token, n in Counter(doc).items():
        M[r, index[token]] = n

print('\ndocument-term matrix:')
print('      ' + ' '.join('%-8s' % v[:7] for v in vocab))
for r, row in enumerate(M):
    print('doc %d ' % r + ' '.join('%-8d' % v for v in row))

vocabulary (8): ['is', 'line', 'my', 'not', 'router', 'slow', 'the', 'working']

document-term matrix:
      is       line     my       not      router   slow     the      working 
doc 0 1        0        0        1        1        0        1        1       
doc 1 1        0        0        0        1        0        1        1       
doc 2 1        1        1        1        0        1        0        0       

Read rows 0 and 1 again

They differ by one word and mean opposite things. In this representation they differ by one column out of eight, and nothing records that not was sitting immediately before working. Every limitation in the next two days follows from that single fact.

The same thing from scikit-learn

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer()
M = vec.fit_transform(tickets['text'])
print('matrix %s, %.4f%% non-zero'
      % (M.shape, 100 * M.nnz / (M.shape[0] * M.shape[1])))
print('stored as %s' % type(M).__name__)

counts = np.asarray(M.sum(axis=0)).ravel()
names = vec.get_feature_names_out()
print('\nmost common tokens:')
for i in counts.argsort()[:-1][:12]:
    print('  %-14s %5d' % (names[i], counts[i]))

matrix (4200, 130), 10.9416% non-zero
stored as csr_matrix

most common tokens:
  help               3
  invoce             6
  internt           21
  conection         26
  competitor        83
  better            83
  package           83
  offered           83
  considering       85
  options           85
  am                85
  customer          92

Sparse matrix: almost every entry is zero, so storing only the non-zero positions costs a fraction of the memory. A vocabulary of 20,000 words over 100,000 documents is two billion cells dense and perhaps twenty million stored sparse. Every scikit-learn text tool returns one, and calling .toarray() on a large one is the quickest way to exhaust your memory.

Day 1 takeaway

Text becomes a matrix by choosing a tokeniser and a vocabulary, and both choices are yours. A bag of words records which tokens occurred and discards the order they occurred in, keep that sentence in mind, because tomorrow it costs a linear model forty points of accuracy.

Week 13 · Day 2 of 7

TF-IDF, n-grams and What Order Costs

Two tasks in one corpus: one a bag of words solves, one it cannot

By Pawan Rama Mali1882 words

Two tasks live in this corpus. One is to say which category a ticket belongs to; the other is to say whether the customer's problem is resolved. They look equally easy. They are not, and the difference is the most useful thing in this week.

TF-IDF

TF-IDF: Term frequency times inverse document frequency. A token scores highly in a document when it appears often there and rarely everywhere else. It is a reweighting of the bag of words, not a different representation. The order is still gone.

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.feature_extraction.text import TfidfVectorizer

vec = TfidfVectorizer()
M = vec.fit_transform(tickets['text'])
names = vec.get_feature_names_out()

row = M[0].toarray().ravel()
print('ticket: %s' % tickets['text'][0][:70])
print('\nhighest weighted terms in it:')
for i in row.argsort()[:-1][:6]:
    if row[i] > 0:
        print('  %-14s %.3f  (document frequency %.3f)'
              % (names[i], row[i],
                 (M[:, i] > 0).sum() / M.shape[0]))

ticket: AS OF THIS MORNING THE HUB IS PATCHY. MOST NIGHTS THE HUB IS NO LONGER

highest weighted terms in it:

Task one: which category?

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

Xc_tr, Xc_te, yc_tr, yc_te = train_test_split(
    tickets['text'], tickets['category'], test_size=0.25,
    stratify=tickets['category'], random_state=42)

model = make_pipeline(TfidfVectorizer(),
                      LogisticRegression(max_iter=2000)).fit(Xc_tr, yc_tr)
print('category accuracy %.4f' % model.score(Xc_te, yc_te))
print('always guessing the commonest: %.4f'
      % (yc_te == yc_tr.mode()[0]).mean())

category accuracy 0.8400
always guessing the commonest: 0.5105

A bag of words handles this comfortably. Words like invoice, tariff and termination belong to one category and nowhere else, so their mere presence settles the question and their position in the sentence is irrelevant.

Task two: is the problem resolved?

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score

model = make_pipeline(TfidfVectorizer(),
                      LogisticRegression(max_iter=2000)).fit(X_tr, y_tr)
print('accuracy   %.4f' % model.score(X_te, y_te))
print('ROC AUC    %.4f'
      % roc_auc_score(y_te, model.predict_proba(X_te)[:, 1]))
print('\nalways predicting unresolved: %.4f' % (1 - y_te.mean()))

accuracy   0.5597
ROC AUC    0.5297

always predicting unresolved: 0.5750

It is worse than a constant

The model that reads the ticket does worse than the model that ignores it and always says unresolved. This is not a bug, a bad hyperparameter or too little data. Something about the combination of this representation and this model cannot express the answer, and working out which of the two is at fault is the whole of the rest of this day.

Why, in one table

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
import re

def has(word):
    return tickets['text'].str.lower().str.contains(
        r'\b%s\b' % word, regex=True)

print('%-14s %10s %10s %12s' % ('token', 'appears', 'resolved', 'unresolved'))
for word in ['not', 'still', 'no', 'longer', 'working', 'slow',
             'fixed', 'broken']:
    m = has(word)
    print('%-14s %10d %10d %12d'
          % (word, m.sum(), (m & (tickets['resolved'] == 1)).sum(),
             (m & (tickets['resolved'] == 0)).sum()))

token             appears   resolved   unresolved
not                  1346        568          778
still                1331        562          769
no                   1273        555          718
longer               1273        555          718
working               382        168          214
slow                  366        154          212
fixed                 379        165          214
broken                357        154          203

Every one of those tokens is split roughly evenly between the two classes. Alone, each is worth nothing. A linear model assigns one weight per token and adds them up, so it has to commit to a single number for not, and no single number can be right when not working means one thing and not slow means the opposite.

Interaction: A pattern where the effect of one feature depends on the value of another. Week 8 met these in tabular data; this is the same phenomenon in text. A sum of per-feature weights cannot represent one, which is exactly why the linear model is stuck at chance.

So whose fault is it, the features or the model?

There are two ways out of an interaction, and they are worth separating because they lead to different code. Either hand the model the interaction as a feature, or use a model that finds interactions by itself. Try the second one first, keeping the unigrams exactly as they are.

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer

# HistGradientBoosting needs a dense array, and this vocabulary is
# small enough that densifying it is safe. On a real corpus it is not.
densify = FunctionTransformer(lambda M: np.asarray(M.todense()),
                              accept_sparse=True)

print('unigram counts only -- no bigrams anywhere')
print('%-32s %10s' % ('model', 'accuracy'))
for name, clf, dense in [
        ('logistic regression (linear)',
         LogisticRegression(max_iter=2000), False),
        ('gradient boosting',
         HistGradientBoostingClassifier(random_state=42), True),
        ('neural net (32, 16)',
         MLPClassifier((32, 16), max_iter=800, random_state=42), False)]:
    steps = [CountVectorizer()] + ([densify] if dense else []) + [clf]
    m = make_pipeline(*steps).fit(X_tr, y_tr)
    print('%-32s %10.4f' % (name, m.score(X_te, y_te)))
print('%-32s %10.4f' % ('always predicting unresolved', 1 - y_te.mean()))

unigram counts only -- no bigrams anywhere
model                              accuracy
logistic regression (linear)         0.5664
gradient boosting                    0.9026
neural net (32, 16)                  0.8672
always predicting unresolved         0.5750

The information was there the whole time

Nothing about the features changed. The same unigram counts that left logistic regression below a constant prediction take gradient boosting most of the way to the ceiling, because a tree can split on not and then split again on slow inside that branch, which is precisely what representing an interaction means.

So the bag of words did not throw away the answer. It threw away the convenience of the answer, and a linear model is a model that can only use convenient answers. Keep the two ideas separate when you diagnose your own models: the features do not contain it and my model cannot reach it look identical from the accuracy score and call for opposite fixes.

The other way out: n-grams

n-gram: A contiguous run of n tokens treated as a single feature. ngram_range=(1, 2) keeps every word and every adjacent pair, so not working becomes a feature in its own right, entirely separate from not and working.

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score

print('%-16s %10s %10s %10s' % ('features', 'columns', 'accuracy', 'auc'))
for label, rng in [('unigrams', (1, 1)), ('1-2 grams', (1, 2)),
                   ('1-3 grams', (1, 3)), ('2-grams only', (2, 2))]:
    m = make_pipeline(TfidfVectorizer(ngram_range=rng),
                      LogisticRegression(max_iter=2000)).fit(X_tr, y_tr)
    print('%-16s %10d %10.4f %10.4f'
          % (label, len(m[0].vocabulary_), m.score(X_te, y_te),
             roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])))

features            columns   accuracy        auc
unigrams                130     0.5597     0.5297
1-2 grams              1079     0.9542     0.9591
1-3 grams              4139     0.9532     0.9589
2-grams only            949     0.9599     0.9605

What the bigram model learned

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                  LogisticRegression(max_iter=2000)).fit(X_tr, y_tr)
names = m[0].get_feature_names_out()
coef = m[1].coef_.ravel()

print('strongest evidence the problem IS resolved:')
for i in coef.argsort()[:-1][:8]:
    print('  %-24s %+.3f' % (names[i], coef[i]))
print('\nstrongest evidence it is NOT:')
for i in coef.argsort()[:8]:
    print('  %-24s %+.3f' % (names[i], coef[i]))

strongest evidence the problem IS resolved:
  not stable               -3.788
  longer fine              -3.734
  longer fixed             -3.733
  still dropping           -3.729
  not sorted               -3.661
  longer back              -3.597
  still unreliable         -3.586
  not back                 -3.507

strongest evidence it is NOT:
  not stable               -3.788
  longer fine              -3.734
  longer fixed             -3.733
  still dropping           -3.729
  not sorted               -3.661
  longer back              -3.597
  still unreliable         -3.586
  not back                 -3.507

This is what a good feature looks like

The model has recovered the construction the corpus was built from, and it did so without being told. Bigrams sit at the top of both lists and single words sit near zero. If you ever want to know whether a text representation is working, print the coefficients and see whether they read like something a person would say.

Notice too that bigrams plus a linear model beat gradient boosting on unigrams. Giving the model the right interaction directly beats asking it to search for interactions in general, which is week 8's argument, arriving here in a different costume. Feature engineering is not what you do when you cannot afford a better model; it is how you tell the model what you already know.

Day 2 takeaway

TF-IDF reweights a bag of words; it does not restore order. When the label lives in an interaction between tokens, a linear model on unigrams can score below a constant prediction, while the very same features carry a tree ensemble most of the way home. Separate "the features lack it" from "my model cannot reach it", because the fixes are different and the symptom is the same.

Week 13 · Day 3 of 7

Embeddings

Dense vectors with a geometry, and what they do and do not fix

By Pawan Rama Mali1165 words

Bigrams worked, and they scale badly: every pair you keep is a new column, trigrams are worse, and none of it generalises. A model that has learned not working knows nothing about not functioning. Embeddings are the alternative.

The problem with one column per word

import numpy as np

vocab = ['router', 'hub', 'invoice', 'working', 'broken']
eye = np.eye(len(vocab), dtype=int)
print('one-hot:')
for w, row in zip(vocab, eye):
    print('  %-9s %s' % (w, row))

print('\ncosine similarity between every pair: ', end='')
sims = {round(float(a @ b), 3) for i, a in enumerate(eye)
        for j, b in enumerate(eye) if i != j}
print(sims)
print('router and hub are exactly as similar as router and invoice.')

one-hot:
  router    [1 0 0 0 0]
  hub       [0 1 0 0 0]
  invoice   [0 0 1 0 0]
  working   [0 0 0 1 0]
  broken    [0 0 0 0 1]

cosine similarity between every pair: {0.0}
router and hub are exactly as similar as router and invoice.

Embedding: A dense vector of fixed width, learned for each token, positioned so that tokens used in similar contexts end up near each other. It replaces a 20,000-column one-hot with perhaps 64 numbers, and unlike one-hot it has a geometry.

Learning one

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
vec = tf.keras.layers.TextVectorization(max_tokens=400,
                                        output_sequence_length=24)
vec.adapt(X_tr.to_numpy())

print('vocabulary size %d' % vec.vocabulary_size())
print('first 12 tokens: %s' % vec.get_vocabulary()[:12])

example = tf.constant(['the router is not working',
                       'the router is working'])
print('\nencoded:')
print(vec(example).numpy()[:, :10])

vocabulary size 134
first 12 tokens: ['', '[UNK]', np.str_('the'), np.str_('is'), np.str_('i'), np.str_('hi'), np.str_('not'), np.str_('morning'), np.str_('this'), np.str_('still'), np.str_('no'), np.str_('longer')]

encoded:
[[ 2 30  3  6 52  0  0  0  0  0]
 [ 2 30  3 52  0  0  0  0  0  0]]

Index 0 is padding and index 1 is the unknown token

TextVectorization reserves both. Every sequence is padded with zeros to a fixed length so it can sit in a rectangular array, and any word not in the vocabulary becomes [UNK]. Forgetting that index 0 is not a real word is how people end up with an embedding for padding that quietly dominates their averages.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
vec = tf.keras.layers.TextVectorization(max_tokens=400,
                                        output_sequence_length=24)
vec.adapt(X_tr.to_numpy())

tf.random.set_seed(42)
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(1,), dtype=tf.string),
    vec,
    tf.keras.layers.Embedding(vec.vocabulary_size(), 16, mask_zero=True),
    tf.keras.layers.GlobalAveragePooling1D(),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
model.compile(optimizer='adam', loss='binary_crossentropy',
              metrics=['accuracy'])
model.fit(X_tr.to_numpy(), y_tr.to_numpy(), epochs=25, batch_size=64,
          verbose=0)
print('accuracy %.4f'
      % model.evaluate(X_te.to_numpy(), y_te.to_numpy(), verbose=0)[1])

accuracy 0.9188

Averaging the embeddings discards the order just as thoroughly as CountVectorizer did, two tickets with the same words in a different arrangement produce an identical average. What rescues the score is the Dense layer sitting after the average, which is nonlinear and can therefore represent the interaction, exactly as gradient boosting did on day 2. It lands in the same region as that model, and behind bigrams, for the same reason: it has to discover the pairing rather than being handed it.

The geometry it learned

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
vec = tf.keras.layers.TextVectorization(max_tokens=400,
                                        output_sequence_length=24)
vec.adapt(tickets['text'].to_numpy())

tf.random.set_seed(42)
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(1,), dtype=tf.string),
    vec,
    tf.keras.layers.Embedding(vec.vocabulary_size(), 16),
    tf.keras.layers.GlobalAveragePooling1D(),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax'),
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
codes = {'billing': 0, 'cancellation': 1, 'technical': 2}
model.fit(tickets['text'].to_numpy(),
          tickets['category'].map(codes).to_numpy(),
          epochs=30, batch_size=64, verbose=0)

E = model.layers[1].get_weights()[0]
words = vec.get_vocabulary()
idx = {w: i for i, w in enumerate(words)}
norm = E / (np.linalg.norm(E, axis=1, keepdims=True) + 1e-9)

def nearest(word, k=4):
    if word not in idx:
        return '(not in vocabulary)'
    sims = norm @ norm[idx[word]]
    order = [i for i in sims.argsort()[:-1] if i != idx[word]][:k]
    return ', '.join('%s %.2f' % (words[i], sims[i]) for i in order)

for w in ['invoice', 'router', 'termination', 'engineer']:
    print('%-13s -> %s' % (w, nearest(w)))

invoice       -> regards -0.86, have -0.81, conection -0.81, internt -0.72
router        -> considering -0.84, six -0.84, years -0.83, options -0.83
termination   -> unreliable -0.84, on -0.79, router -0.77, pay -0.76
engineer      -> fine -0.91, fibre -0.74, account -0.74, to -0.73

The neighbours come from the task, not from English

These vectors were trained to predict the ticket category, so words land near each other when they push the category the same way. That is not the same as meaning similarity, and it is why the embedding you train on 3,000 tickets is far less useful than one trained on billions of words. The latter has seen enough context to learn something closer to actual usage. Day 6 returns to this.

Representation	Columns	Order?	Generalises to unseen words?
One-hot / bag of words	One per vocabulary word	No	No
TF-IDF	Same, reweighted	No	No
n-grams	Explodes with n	Locally	No
Learned embedding	Chosen, e.g. 64	Not by itself	Only via subwords
Pretrained embedding	Chosen, e.g. 300	Not by itself	Yes, if the word was in the source corpus

Day 3 takeaway

An embedding gives every token a dense vector with a usable geometry, learned from the task. On its own it does not restore order, averaging embeddings is a bag of words with fewer columns, though, as day 2 established, a nonlinear head on top of that average can still find the interaction.

Week 13 · Day 4 of 7

Recurrent Networks

State across a sequence, vanishing gradients, LSTM, padding and masking

By Pawan Rama Mali1529 words

A recurrent layer walks the sequence one token at a time, carrying a state that it updates at every step. That state is what lets it know not came before working.

A recurrent cell, in numpy

import numpy as np

rng = np.random.default_rng(0)
d_in, d_hidden = 4, 3
Wx = rng.normal(0, 0.5, (d_in, d_hidden))
Wh = rng.normal(0, 0.5, (d_hidden, d_hidden))
b = np.zeros(d_hidden)

def run(sequence):
    h = np.zeros(d_hidden)
    for x in sequence:
        h = np.tanh(x @ Wx + h @ Wh + b)   # the whole recurrence
    return h

a = rng.normal(size=(1, d_in))
c = rng.normal(size=(1, d_in))
print('state after [a, b] %s' % run(np.vstack([a, c])).round(3))
print('state after [b, a] %s' % run(np.vstack([c, a])).round(3))
print('\nsame two inputs, different order, different state.')
print('that is the entire reason this layer exists.')

state after [a, b] [0.038 0.046 0.361]
state after [b, a] [0.172 0.144 0.398]

same two inputs, different order, different state.
that is the entire reason this layer exists.

Why plain recurrence fails on long sequences

import numpy as np

print('%8s %16s %16s' % ('steps', 'weight 0.5', 'weight 1.5'))
for steps in [5, 10, 25, 50, 100]:
    print('%8d %16.6g %16.6g'
          % (steps, 0.5 ** steps, 1.5 ** steps))
print('\nthe gradient is a product of one factor per step.')

   steps       weight 0.5       weight 1.5
       5          0.03125          7.59375
      10      0.000976562           57.665
      25      2.98023e-08          25251.2
      50      8.88178e-16      6.37622e+08
     100      7.88861e-31      4.06561e+17

the gradient is a product of one factor per step.

Vanishing and exploding gradients: Backpropagating through t steps multiplies t derivatives together. If they are consistently below one, the gradient reaching the earliest tokens is effectively zero and the network cannot learn long-range dependencies; if consistently above one, it overflows. Both failures get worse the longer the sequence.

What LSTM and GRU actually changed

Both add a path along which the state is added to rather than repeatedly multiplied, plus learned gates that decide what to keep, what to forget and what to output. The additive path is the important part: a sum does not shrink geometrically, so gradients survive far more steps. GRU is the same idea with two gates instead of three, fewer parameters, usually indistinguishable results.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
for name, layer in [('SimpleRNN', tf.keras.layers.SimpleRNN(32)),
                    ('GRU', tf.keras.layers.GRU(32)),
                    ('LSTM', tf.keras.layers.LSTM(32))]:
    layer.build((None, 20, 16))
    print('%-11s %6d parameters'
          % (name, sum(int(np.prod(w.shape)) for w in layer.weights)))
print('\nfor the same 16-wide input and 32-wide state.')
print('LSTM has four internal transforms, GRU three, SimpleRNN one.')

SimpleRNN     1568 parameters
GRU           4800 parameters
LSTM          6272 parameters

for the same 16-wide input and 32-wide state.
LSTM has four internal transforms, GRU three, SimpleRNN one.

Padding, masking and why they matter

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
vec = tf.keras.layers.TextVectorization(max_tokens=400,
                                        output_sequence_length=20)
vec.adapt(X_tr.to_numpy())

sample = tf.constant(['the hub is broken', X_tr.iloc[0]])
encoded = vec(sample).numpy()
print('short ticket encoded:')
print(encoded[0])
print('%d real tokens, %d padding zeros'
      % ((encoded[0] > 0).sum(), (encoded[0] == 0).sum()))

emb = tf.keras.layers.Embedding(vec.vocabulary_size(), 8, mask_zero=True)
out = emb(encoded)
mask = emb.compute_mask(encoded)
print('\nmask for the short ticket:')
print(mask.numpy()[0])
print('False means "ignore this step" for every layer downstream.')

short ticket encoded:
[ 2 24  3 59  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0]
4 real tokens, 16 padding zeros

mask for the short ticket:
[ True  True  True  True False False False False False False False False
 False False False False False False False False]
False means "ignore this step" for every layer downstream.

`mask_zero=True` is not the default

Without it, a recurrent layer processes twelve padding tokens after your eight real ones and the final state is whatever those zeros left behind. Average pooling is worse: it divides by the padded length, so a four-word ticket has its signal diluted by sixteen zeros. Short documents suffer most, which is precisely the population you are usually worst at already.

The clearest way to see what masking buys you is to change nothing but the padded length. The words are the same, so the pooled vector representing them should be the same.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
text = tf.constant(['the hub is broken'])

# Seed the initialiser itself. tf.random.set_seed does not control the
# seed generator Keras uses for layer weights, so without this the two
# lengths get different embeddings and the comparison measures nothing.
init = tf.keras.initializers.RandomUniform(-0.05, 0.05, seed=0)

for mask_zero in [False, True]:
    pooled = {}
    for length in [8, 40]:
        vec = tf.keras.layers.TextVectorization(
            max_tokens=400, output_sequence_length=length)
        vec.adapt(X_tr.to_numpy())
        e = vec(text)
        emb = tf.keras.layers.Embedding(vec.vocabulary_size(), 8,
                                        embeddings_initializer=init,
                                        mask_zero=mask_zero)
        pooled[length] = tf.keras.layers.GlobalAveragePooling1D()(
            emb(e), mask=emb.compute_mask(e) if mask_zero else None)
    same = float(tf.reduce_max(tf.abs(pooled[8] - pooled[40])))
    print('mask_zero=%-6s  padded to 8 vs to 40: largest difference %.4f'
          % (mask_zero, same))

mask_zero=False   padded to 8 vs to 40: largest difference 0.0163
mask_zero=True    padded to 8 vs to 40: largest difference 0.0000

With masking, the padded length makes no difference at all: the layer averages four vectors either way. Without it, the answer changes when you change a number that is supposed to be an implementation detail. Note what is not happening in the unmasked case: index 0 has its own embedding vector like any other token, so the padding does not dilute the average towards zero, it drags it towards whatever the padding vector happens to be. On a corpus of mostly-short documents, that vector ends up carrying a large share of your representation.

The bug masking gives you for free

Masking has one failure mode, and this corpus contains it. A document with no tokens encodes to nothing but padding, so the mask is False at every position and average pooling divides a sum of nothing by a count of nothing.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
vec = tf.keras.layers.TextVectorization(max_tokens=50,
                                        output_sequence_length=8)
vec.adapt(tf.constant(['the router is broken', 'the invoice is wrong']))

batch = vec(tf.constant(['the router is broken', '', '...']))
emb = tf.keras.layers.Embedding(vec.vocabulary_size(), 4, mask_zero=True)
e = emb(batch)
mask = emb.compute_mask(batch)
pooled = tf.keras.layers.GlobalAveragePooling1D()(e, mask=mask)

print('encoded:')
for text, row in zip(['the router is broken', '(empty)', '...'],
                     batch.numpy()):
    print('  %-22s %s' % (text, row))
print('\npooled vectors:')
print(pooled.numpy().round(3))
print('\nany NaN?', bool(np.isnan(pooled.numpy()).any()))

encoded:
  the router is broken   [2 5 3 7 0 0 0 0]
  (empty)                [0 0 0 0 0 0 0 0]
  ...                    [0 0 0 0 0 0 0 0]

pooled vectors:
[[ 0.006 -0.     0.015 -0.007]
 [   nan    nan    nan    nan]
 [   nan    nan    nan    nan]]

any NaN? True

Non-empty is not the same as non-empty

Row three is the important one. '...' is a perfectly good string. It passes any len(text) > 0 check you write, but TextVectorization strips punctuation by default, so it tokenises to nothing and produces the same NaN as the genuinely empty row. The guard has to be on the token count, not on the string length.

And the NaN does not raise. It propagates through the dense layers, turns the loss into nan, and makes every gradient after it nan too, so one such document destroys an entire training run and the error, if it ever surfaces, points somewhere else. This is why the split used from day 2 onwards filters on [A-Za-z0-9] rather than on length.

Day 4 takeaway

A recurrent layer carries state across a sequence, which is what lets order matter. Plain recurrence cannot learn long-range dependencies because the gradient is a product of per-step factors; LSTM and GRU add an additive path to fix it. Always set mask_zero=True, or padding becomes part of your input.

Week 13 · Day 5 of 7

The Honest Comparison

LSTM against bigrams, measured on accuracy, time and everything else

By Pawan Rama Mali1030 words

Everything is now on the table: a bag of words, bigrams, averaged embeddings and a recurrent layer that genuinely reads the sequence. Time to run them against each other on the task that needs order, and to be honest about the result.

The contenders

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
import time
from sklearn.metrics import roc_auc_score

def build(kind, vec):
    layers = [tf.keras.layers.Input(shape=(1,), dtype=tf.string), vec,
              tf.keras.layers.Embedding(vec.vocabulary_size(), 24,
                                        mask_zero=True)]
    if kind == 'average':
        layers.append(tf.keras.layers.GlobalAveragePooling1D())
    elif kind == 'lstm':
        layers.append(tf.keras.layers.LSTM(32))
    elif kind == 'bilstm':
        layers.append(tf.keras.layers.Bidirectional(
            tf.keras.layers.LSTM(32)))
    elif kind == 'gru':
        layers.append(tf.keras.layers.GRU(32))
    layers += [tf.keras.layers.Dense(16, activation='relu'),
               tf.keras.layers.Dense(1, activation='sigmoid')]
    return tf.keras.Sequential(layers)

xs, ys = X_tr.to_numpy(), y_tr.to_numpy()
xv, yv = X_te.to_numpy(), y_te.to_numpy()

print('%-12s %10s %10s %10s' % ('model', 'accuracy', 'auc', 'fit time'))
for kind in ['average', 'gru', 'lstm', 'bilstm']:
    tf.random.set_seed(42)
    vec = tf.keras.layers.TextVectorization(max_tokens=400,
                                            output_sequence_length=24)
    vec.adapt(xs)
    m = build(kind, vec)
    m.compile(optimizer='adam', loss='binary_crossentropy',
              metrics=['accuracy'])
    t = time.time()
    m.fit(xs, ys, epochs=25, batch_size=64, verbose=0)
    took = time.time() - t
    proba = m.predict(xv, verbose=0).ravel()
    print('%-12s %10.4f %10.4f %9.1fs'
          % (kind, ((proba > 0.5).astype(int) == yv).mean(),
             roc_auc_score(yv, proba), took))

model          accuracy        auc   fit time
average          0.9179     0.9049       3.0s
gru              0.9608     0.9610      10.1s
lstm             0.9570     0.9601      10.4s
bilstm           0.9561     0.9614      16.0s

Against the model from day 2

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
import time
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score

t = time.time()
m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                  LogisticRegression(max_iter=2000)).fit(X_tr, y_tr)
took = time.time() - t
proba = m.predict_proba(X_te)[:, 1]
print('tf-idf bigrams + logistic regression')
print('  accuracy %.4f' % m.score(X_te, y_te))
print('  auc      %.4f' % roc_auc_score(y_te, proba))
print('  fit time %.2fs' % took)
print('  parameters: %d coefficients' % m[1].coef_.size)

tf-idf bigrams + logistic regression
  accuracy 0.9542
  auc      0.9591
  fit time 0.07s
  parameters: 1079 coefficients

Read this before you reach for an LSTM

Both approaches solve the task. One of them took a fraction of a second, fits in a line, produces coefficients you can read out loud, and has no seed, no epoch count and no learning rate to get wrong. The other needed an embedding width, a state width, a sequence length, a batch size and twenty-five epochs, and its parameters mean nothing to anybody.

On four thousand short tickets that is not a close call. Sequence models start earning their keep when documents are long, when the dependency spans more than two or three words, or when you are starting from a model pretrained on far more text than you have, which is the subject of day 6.

Where each one actually fails

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.model_selection import train_test_split

# Drop tickets with no word characters at all -- the empty ones, and the
# ones that are only punctuation. Both tokenise to nothing, and day 4
# shows what an all-padding sequence does to masked pooling.
tickets = tickets[tickets['text'].str.contains(r'[A-Za-z0-9]',
                                               regex=True)].reset_index(drop=True)

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['resolved'], test_size=0.25,
    stratify=tickets['resolved'], random_state=42)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                  LogisticRegression(max_iter=2000)).fit(X_tr, y_tr)
proba = m.predict_proba(X_te)[:, 1]
pred = (proba > 0.5).astype(int)
wrong = np.where(pred != y_te.to_numpy())[0]

print('%d wrong out of %d' % (len(wrong), len(y_te)))
print('\nthe six it was most confident about, and wrong:')
conf = sorted(wrong, key=lambda i: abs(proba[i] - 0.5), reverse=True)[:6]
for i in conf:
    print('  truth=%d p=%.2f  %s'
          % (y_te.iloc[i], proba[i], X_te.iloc[i][:64]))

48 wrong out of 1047

the six it was most confident about, and wrong:
  truth=1 p=0.11  the engineer visited on tuesday. my connection is still patchy. 
  truth=1 p=0.12  i am considering my options. the router is not reliable. my sign
  truth=1 p=0.14  hello, THE ENGINEER VISITED ON TUESDAY. IT IS NOT RELIABLE. THE 
  truth=1 p=0.15  My connection is still unreliable. there is a charge on the stat
  truth=1 p=0.15  as of this morning the connection is still slow. the connection 
  truth=1 p=0.15  my hub is broken. the broadband is still unstable. this relates 

Some of those are the four percent of labels the generator deliberately flipped, and no model can get them right, which is the point of putting them there. Reading your confident errors is the fastest way to tell an irreducible label problem from a fixable modelling one.

Day 5 takeaway

Sequence models read order, and so do bigrams. On short documents with local dependencies, TF-IDF with bigrams and a linear model is faster, simpler, interpretable and just as accurate. Fit it first, every time, and make the recurrent model beat it before you ship one.

Week 13 · Day 6 of 7

Attention and the Modern Landscape

Self-attention from scratch, subword tokens, and what actually wins

By Pawan Rama Mali971 words

Recurrence has a structural problem that no amount of gating fixes: it is sequential. Token 500 cannot be processed until token 499 has been, which means a recurrent model cannot use a GPU's parallelism over the length of the sequence. Attention removes that constraint, and removing it is what made models trained on the whole internet possible.

Self-attention: Every token computes a compatibility score against every other token, turns those scores into weights, and takes a weighted average of their values. Distance in the sequence costs nothing, token 1 and token 500 are one operation apart, and every token is computed at the same time.

Attention, in about fifteen lines

import numpy as np

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V):
    scores = Q @ K.T / np.sqrt(K.shape[-1])   # who is relevant to whom
    weights = softmax(scores)                 # rows sum to 1
    return weights @ V, weights               # weighted average of values

rng = np.random.default_rng(0)
tokens = ['the', 'router', 'is', 'not', 'working']
X = rng.normal(size=(len(tokens), 8))

# with random projections the pattern is random; the point here is shape
Wq, Wk, Wv = (rng.normal(0, 0.5, (8, 8)) for _ in range(3))
out, weights = attention(X @ Wq, X @ Wk, X @ Wv)

print('output shape %s, one vector per token' % (out.shape,))
print('\nattention weights (row = token doing the looking):')
print('%-9s %s' % ('', ' '.join('%7s' % t for t in tokens)))
for t, row in zip(tokens, weights):
    print('%-9s %s' % (t, ' '.join('%7.3f' % v for v in row)))
print('\nevery row sums to 1: %s' % np.allclose(weights.sum(axis=1), 1))

output shape (5, 8), one vector per token

attention weights (row = token doing the looking):
              the  router      is     not working
the         0.122   0.088   0.434   0.248   0.108
router      0.614   0.012   0.027   0.010   0.338
is          0.302   0.145   0.151   0.188   0.214
not         0.225   0.159   0.237   0.103   0.276
working     0.446   0.012   0.224   0.044   0.274

every row sums to 1: True

Those weights are meaningless, and that is deliberate

The projections here are random, so the pattern shows nothing about language, only the mechanism and the shapes. In a trained model the row for not would put weight on working, which is how attention captures exactly the dependency that defeated the unigram model on day 2. Published attention heat-maps are trained weights, and reading them as explanations is a well-documented trap: attention shows what the model looked at, not why it decided.

Why the position has to be added back

import numpy as np

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V):
    return softmax(Q @ K.T / np.sqrt(K.shape[-1])) @ V

rng = np.random.default_rng(0)
X = rng.normal(size=(4, 6))
Wq, Wk, Wv = (rng.normal(0, 0.5, (6, 6)) for _ in range(3))

a = attention(X @ Wq, X @ Wk, X @ Wv)
shuffled = X[[2, 0, 3, 1]]
b = attention(shuffled @ Wq, shuffled @ Wk, shuffled @ Wv)

print('shuffle the tokens and the outputs are the same rows, reordered:')
print(np.allclose(a[[2, 0, 3, 1]], b))
print('\nso attention alone is order-blind. positional encodings are')
print('added to the embeddings precisely to fix this.')

shuffle the tokens and the outputs are the same rows, reordered:
True

so attention alone is order-blind. positional encodings are
added to the embeddings precisely to fix this.

The landscape, honestly

Approach	Needs	Use when
TF-IDF + linear	Minutes, a laptop	Short texts, a clear vocabulary signal. Always try first.
Embedding + pooling	A GPU is optional	Large corpus, many classes, vocabulary too large for n-grams
LSTM / GRU	Patience	Long-range order matters and you cannot use a pretrained model
Pretrained transformer, frozen	A download	Small labelled set; use the embeddings as features
Pretrained transformer, fine-tuned	A GPU, hours	A few thousand labels and accuracy that justifies the cost
A hosted large language model	An API budget	Few or no labels, or the task is genuinely open-ended

Where week 12's lesson repeats itself

The reason transformers dominate text is the same reason convolutional networks dominate images: the architecture matches the structure of the data, and there is enough data to fill it. A transformer trained from scratch on your four thousand tickets will lose to logistic regression, comfortably. A transformer pretrained on a trillion tokens and fine-tuned on your four thousand tickets will win. The architecture is not the thing that wins, the pretraining is.

Subword tokenisation, and why it exists

# Word-level vocabularies break on anything they have not seen.
vocab = {'the', 'router', 'is', 'not', 'working', 'slow'}
for sentence in ['the router is not working',
                 'the routers are misconfigured']:
    tokens = sentence.split()
    marked = [t if t in vocab else '[UNK]' for t in tokens]
    print('%-32s -> %s' % (sentence, ' '.join(marked)))

print('\nsubword tokenisers split unknown words into known pieces:')
print('  routers        -> router + s')
print('  misconfigured  -> mis + config + ured')
print('so the vocabulary is finite and nothing is ever fully unknown.')

the router is not working        -> the router is not working
the routers are misconfigured    -> the [UNK] [UNK] [UNK]

subword tokenisers split unknown words into known pieces:
  routers        -> router + s
  misconfigured  -> mis + config + ured
so the vocabulary is finite and nothing is ever fully unknown.

This is why modern models report vocabularies of around 30,000 to 100,000 pieces rather than millions of words, and why they cope with typos, product codes and languages with heavy compounding. It is also why token counts on an API bill do not match word counts. You are charged per piece.

Day 6 takeaway

Self-attention lets every token look at every other in one parallel operation, which is what made training on internet-scale text practical. It is order-blind by itself, so position is added to the embeddings. And the winning ingredient in modern NLP is pretraining, not the architecture diagram.

Week 13 · Day 7 of 7

A Complete Text Workflow

Grid search inside a pipeline, error analysis, and the text-specific traps

By Pawan Rama Mali1231 words

A complete text pipeline on the category task, plus the failure modes that are specific to text and that no amount of cross-validation will catch on its own.

The pipeline

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, train_test_split

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['category'], test_size=0.25,
    stratify=tickets['category'], random_state=42)

pipe = Pipeline([('tfidf', TfidfVectorizer()),
                 ('clf', LogisticRegression(max_iter=2000))])
grid = {'tfidf__ngram_range': [(1, 1), (1, 2)],
        'tfidf__min_df': [1, 3],
        'tfidf__sublinear_tf': [False, True],
        'clf__C': [0.5, 2.0, 8.0]}

search = GridSearchCV(pipe, grid, cv=5, scoring='f1_macro', n_jobs=1)
search.fit(X_tr, y_tr)
print('best macro F1 in cross-validation %.4f' % search.best_score_)
for k, v in sorted(search.best_params_.items()):
    print('  %-24s %s' % (k, v))
print('\nheld-out accuracy %.4f' % search.score(X_te, y_te))

best macro F1 in cross-validation 0.8732
  clf__C                   0.5
  tfidf__min_df            1
  tfidf__ngram_range       (1, 2)
  tfidf__sublinear_tf      True

held-out accuracy 0.8456

Vectorise inside the pipeline, never before the split

TfidfVectorizer learns a vocabulary and a document frequency for every term. Fit it on all your text and those statistics have seen the test set, which is week 4's leakage wearing a different hat. Inside a Pipeline, every cross-validation fold refits the vectoriser on that fold's training rows only. This is the single most common leak in text projects, and it flatters your score by a few points, just enough to be believable.

Reading the errors

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

X_tr, X_te, y_tr, y_te = train_test_split(
    tickets['text'], tickets['category'], test_size=0.25,
    stratify=tickets['category'], random_state=42)
m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                  LogisticRegression(max_iter=2000, C=2.0)).fit(X_tr, y_tr)
pred = m.predict(X_te)
labels = sorted(y_te.unique())

print(classification_report(y_te, pred, digits=3))
print('confusion matrix (rows = truth):')
print('%-14s %s' % ('', ' '.join('%13s' % l for l in labels)))
for l, row in zip(labels, confusion_matrix(y_te, pred, labels=labels)):
    print('%-14s %s' % (l, ' '.join('%13d' % v for v in row)))

              precision    recall  f1-score   support

     billing      0.842     0.804     0.822       357
cancellation      0.950     0.732     0.827       157
   technical      0.840     0.922     0.879       536

    accuracy                          0.853      1050
   macro avg      0.877     0.819     0.843      1050
weighted avg      0.857     0.853     0.852      1050

confusion matrix (rows = truth):
                     billing  cancellation     technical
billing                  287             3            67
cancellation              15           115            27
technical                 39             3           494

The pitfalls that are specific to text

Duplicate documents across the split. This corpus contains 90 double-submitted tickets on purpose. Split without deduplicating and the same text is in training and test, so you are measuring memory. Deduplicate on the text, not just the id.
Fitting the vectoriser before the split. Covered above, and worth repeating because it is so easy to do accidentally in a notebook.
Time. Ticket language drifts, new products, new outages, new slang. A random split tells you how you do on the past; a split by date tells you how you will do next month, and it is always the lower number.
Very short and empty documents. Nine tickets here are empty strings. They produce an all-zero row, which every linear model will classify identically. Decide deliberately whether to drop them or route them to a human.
Class imbalance. Cancellation is the smallest and usually the most valuable category. Use macro-averaged F1 rather than accuracy, as the grid above does.
Personal data. Free text contains names, addresses, account and card numbers. Redact before you store features, and certainly before anything leaves your infrastructure for an API.

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

raw = pd.read_csv('tickets.csv')            # with the duplicates left in
raw['text'] = raw['text'].fillna('').str.strip()

A, B, ya, yb = train_test_split(raw['text'], raw['resolved'],
                                test_size=0.25, random_state=0)

# Split the test set by whether its text was also in training. Comparing
# those two subsets isolates memorisation; comparing a deduplicated run
# against this one would not, because deduplicating changes the task.
seen = B.isin(set(A))
print('test rows whose text also appears in training: %d of %d\n'
      % (seen.sum(), len(B)))

print('%-22s %8s %8s %8s' % ('', 'seen', 'unseen', 'overall'))
for name, clf in [('1-nearest neighbour', KNeighborsClassifier(1)),
                  ('logistic regression',
                   LogisticRegression(max_iter=2000))]:
    m = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), clf).fit(A, ya)
    print('%-22s %8.4f %8.4f %8.4f'
          % (name, m.score(B[seen], yb[seen]),
             m.score(B[~seen], yb[~seen]), m.score(B, yb)))

test rows whose text also appears in training: 38 of 1073

                           seen   unseen  overall
1-nearest neighbour      0.9474   0.6580   0.6682
logistic regression      0.8947   0.9449   0.9432

The gap is the leak, and how big it is depends entirely on how much the model can memorise. One-nearest-neighbour finds the identical training ticket and copies its label, so it looks strong on the duplicated rows and collapses on everything else. The regularised linear model, which cannot store individual documents, shows no gap at all.

Do not conclude that duplicates are harmless

They are harmless to this model, on a corpus where 38 rows out of a thousand overlap. Scale either of those up, a scraped corpus where the same article appears on forty sites, or any model with the capacity to memorise, which includes every fine-tuned transformer, and the same leak moves your headline number by a lot. Deduplicate anyway: the check is two lines and the failure is invisible.

Saving the whole thing

import numpy as np
import pandas as pd

tickets = pd.read_csv('tickets.csv')
tickets['text'] = tickets['text'].fillna('').str.strip()
tickets = tickets.drop_duplicates(subset='ticket_id').reset_index(drop=True)
import joblib
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)),
                      LogisticRegression(max_iter=2000, C=2.0))
model.fit(tickets['text'], tickets['category'])
joblib.dump(model, 'ticket_router.joblib')

loaded = joblib.load('ticket_router.joblib')
new = ['the invoice is wrong again and nobody has called back',
       'my router is no longer dropping out, thanks',
       'i want to leave, what is the termination fee']
for text, label in zip(new, loaded.predict(new)):
    proba = loaded.predict_proba([text]).max()
    print('%-13s p=%.2f  %s' % (label, proba, text[:52]))

billing       p=0.72  the invoice is wrong again and nobody has called bac
technical     p=0.47  my router is no longer dropping out, thanks
cancellation  p=0.80  i want to leave, what is the termination fee

One object holds the tokeniser, the vocabulary, the IDF weights and the coefficients, and it takes raw strings as input. That is the property that matters in production: the serving code should never have to reimplement your preprocessing, because the day it drifts from the training version is the day your model silently degrades. Week 15 builds the service around exactly this file.

Day 7 takeaway

Put the vectoriser inside the pipeline, deduplicate before you split, split by time when time exists, use macro F1 when classes are imbalanced, and redact personal data. Then save one object that goes from raw string to prediction.