Text as Data

Week 4 of 14 · Language · 7 days

Week 04 · Language

Text as Data

Week 04 · Day 1 of 7

Turning Sentences Into Tokens

Splitting, normalising, and the stop word that reverses your meaning

A computer cannot do arithmetic on a sentence. Everything in natural language processing begins with turning text into numbers, and the choices made in that step decide more about the final accuracy than the choice of model does.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
print('%d reviews, two labels each' % len(texts))
for t, s, c in list(zip(texts, sentiment, category))[:5]:
    print('%-9s %-8s %s' % (c, 'positive' if s else 'negative', t))

2400 reviews, two labels each
laptop    positive after a month the keyboard is still quite reliable, as described in the listing
laptop    negative for the price the charger is not sturdy, second one i have bought
kitchen   positive for the price the lid is hardly scratched, as described in the listing
kitchen   negative for the price the blade is not solid, the packaging was fine
clothing  positive the sleeve is very solid, as described in the listing

Step one: splitting into tokens

Tokenisation: Cutting text into the units the model will treat as atomic. Splitting on spaces is the obvious approach and it is wrong in ways that matter.

text = "The kettle's lid isn't sturdy -- I've returned it. Cost 24.99!"
print('naive split on spaces')
print(text.split())
print()

import re
print('splitting on word boundaries')
print(re.findall(r"[a-z0-9']+", text.lower()))

naive split on spaces
['The', "kettle's", 'lid', "isn't", 'sturdy', '--', "I've", 'returned', 'it.', 'Cost', '24.99!']

splitting on word boundaries
['the', "kettle's", 'lid', "isn't", 'sturdy', "i've", 'returned', 'it', 'cost', '24', '99']

The naive split leaves punctuation glued to words, so it. and it become different things. The second version lowercases and strips punctuation, which fixes that and quietly creates new problems: isn't survives as one token, and the price has become two.

Normalisation, and what it destroys

import re

STOP = {'the', 'is', 'a', 'and', 'it', 'to', 'of', 'i', 'was'}

def normalise(text, drop_stopwords):
    words = re.findall(r"[a-z']+", text.lower())
    if drop_stopwords:
        words = [w for w in words if w not in STOP]
    return words

for sentence in ['the pan is not sturdy', 'the pan is sturdy']:
    print('%-24s -> %s' % (sentence, normalise(sentence, False)))
print()
for sentence in ['the pan is not sturdy', 'the pan is sturdy']:
    print('%-24s -> %s' % (sentence, normalise(sentence, True)))

the pan is not sturdy    -> ['the', 'pan', 'is', 'not', 'sturdy']
the pan is sturdy        -> ['the', 'pan', 'is', 'sturdy']

the pan is not sturdy    -> ['pan', 'not', 'sturdy']
the pan is sturdy        -> ['pan', 'sturdy']

Removing stop words is not free

The usual advice is to drop common words because they carry no meaning. Look at what it did above: two sentences that mean opposite things became identical, because not is on almost every stop word list.

For topic classification this rarely matters. For anything involving negation, opinion or instruction it is fatal, and it is applied by default in a great deal of tutorial code.

Week 04 · Day 2 of 7

Counting Words

The document term matrix, sparsity, and TF-IDF

By Pawan Rama Mali904 words

With tokens in hand, the standard representation counts them. Every document becomes a row of numbers, one column per word in the vocabulary.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer()
X = vec.fit_transform(train_x)
print('%d documents, %d columns' % X.shape)
print('%.4f%% of the matrix is non-zero'
      % (100 * X.nnz / (X.shape[0] * X.shape[1])))
print()
print('first 12 words of the vocabulary:')
print(list(vec.get_feature_names_out())[:12])

1680 documents, 65 columns
17.0302% of the matrix is non-zero

first 12 words of the vocabulary:
['after', 'and', 'arrived', 'as', 'battery', 'blade', 'bought', 'but', 'charger', 'collar', 'delivery', 'described']

Note the sparsity. Almost every entry is zero, because a short review contains a handful of the vocabulary's words. This is why text matrices are stored in a sparse format and why they can have tens of thousands of columns without exhausting memory.

Not every word deserves equal weight

TF-IDF: Term frequency times inverse document frequency. A word counts for more when it appears often in this document and rarely across the collection. Words that appear everywhere, which is most of the common ones, are scaled towards nothing without anybody writing a stop word list.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

vec = TfidfVectorizer()
X = vec.fit_transform(train_x)
words = vec.get_feature_names_out()
idf = vec.idf_
order = np.argsort(idf)
print('%-16s %8s' % ('most common, so worth least', 'idf'))
for i in order[:5]:
    print('%-16s %8.3f' % (words[i], idf[i]))
print()
print('%-16s %8s' % ('rarest, so worth most', 'idf'))
for i in order[-5:]:
    print('%-16s %8.3f' % (words[i], idf[i]))

most common, so worth least      idf
the                 1.000
is                  1.227
was                 2.082
for                 2.484
price               2.484

rarest, so worth most      idf
stitching           3.754
blade               3.764
fabric              3.764
kettle              3.802
screen              3.842

Week 04 · Day 3 of 7

What Order Is Worth

The same text, two tasks, and a jump from chance to perfect

By Pawan Rama Mali1044 words

The representations so far throw away word order completely. Here is what that costs, on two tasks over exactly the same text.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

print('%-20s %12s %12s' % ('', 'sentiment', 'category'))
for name, make in [('unigrams', lambda: CountVectorizer()),
                   ('unigrams + bigrams',
                    lambda: CountVectorizer(ngram_range=(1, 2))),
                   ('tfidf unigrams', lambda: TfidfVectorizer()),
                   ('tfidf + bigrams',
                    lambda: TfidfVectorizer(ngram_range=(1, 2)))]:
    s = make_pipeline(make(),
                      LogisticRegression(max_iter=2000)).fit(train_x,
                                                            train_s)
    c = make_pipeline(make(),
                      LogisticRegression(max_iter=2000)).fit(train_x,
                                                            train_c)
    print('%-20s %12.4f %12.4f'
          % (name, s.score(test_x, test_s), c.score(test_x, test_c)))

                        sentiment     category
unigrams                   0.4931       1.0000
unigrams + bigrams         1.0000       1.0000
tfidf unigrams             0.4958       1.0000
tfidf + bigrams            1.0000       1.0000

The category column is 1.000 everywhere. Deciding whether a review is about a laptop or a pan needs only the nouns, and word order adds nothing at all.

The sentiment column is the interesting one. Unigrams score 0.493, which on a balanced two-class problem is exactly chance. The model has learned nothing whatsoever, and adding bigrams takes it to 1.000.

Why unigrams score exactly nothing here

Half the positive reviews in this corpus are written as a negated negative, not flimsy, and half the negative ones as a negated positive, not sturdy. So the word flimsy appears equally often on both sides, and so does sturdy. Every individual word is uninformative by construction, and a representation that only sees individual words cannot do better than guessing.

Pairs of adjacent words separate them completely, because not flimsy and very flimsy are different columns. The jump from 0.493 to 1.000 is that stark because this corpus was built to isolate the effect. Real text is messier and the gap is smaller, but the direction is the same and the cause is identical.

N-grams: Sequences of n adjacent tokens. Bigrams are pairs, trigrams are triples. They recover local word order at the cost of a much larger vocabulary, since the number of possible pairs is far greater than the number of words.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import CountVectorizer

for n in [(1, 1), (1, 2), (1, 3)]:
    vec = CountVectorizer(ngram_range=n).fit(train_x)
    print('ngram_range=%-8s %6d columns' % (str(n),
                                            len(vec.vocabulary_)))

ngram_range=(1, 1)       65 columns
ngram_range=(1, 2)      280 columns
ngram_range=(1, 3)     1203 columns

Three times the columns for bigrams, and more again for trigrams. On this corpus that is affordable. On a large collection the vocabulary grows faster than the useful signal, most of the extra columns appear once and help nothing, and you end up limiting the vocabulary by frequency.

Week 04 · Day 4 of 7

Reading What the Model Learned

Weights per word, and using them to debug

By Pawan Rama Mali594 words

A model built this way is not a black box. The weights are one number per column and you can read them, which is worth doing on every text model you build.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
import numpy as np

vec = CountVectorizer(ngram_range=(1, 2))
X = vec.fit_transform(train_x)
clf = LogisticRegression(max_iter=2000).fit(X, train_s)
words = vec.get_feature_names_out()
order = np.argsort(clf.coef_[0])
print('%-24s %8s' % ('most positive', 'weight'))
for i in order[::-1][:6]:
    print('%-24s %8.3f' % (words[i], clf.coef_[0][i]))
print()
print('%-24s %8s' % ('most negative', 'weight'))
for i in order[:6]:
    print('%-24s %8.3f' % (words[i], clf.coef_[0][i]))

most positive              weight
hardly loose                2.760
from faulty                 2.706
from scratched              2.688
hardly scratched            2.605
not faulty                  2.591
not scratched               2.576

most negative              weight
not solid                  -2.823
from solid                 -2.680
hardly sturdy              -2.626
hardly reliable            -2.615
hardly solid               -2.607
not sharp                  -2.564

The model found the negation pairs on its own. Nobody wrote a rule about not, and nobody listed which adjectives are complimentary. It learned both from four hundred labelled examples, and the table above is a complete account of what it believes.

Reading these tables is a debugging tool

If a weight makes no sense, ask why that word is in your data. It is often a leak: a phrase that only appears in one class for reasons unrelated to the task.
If the top weights are all rare words appearing once or twice, the model is memorising and needs a minimum document frequency.
If the top weights look reasonable and the accuracy is poor, the representation is the problem rather than the model, which is exactly what yesterday's unigram row was telling you.

Week 04 · Day 5 of 7

How Far Bag of Words Gets You

Four classifiers on identical features, and when to stop

By Pawan Rama Mali576 words

A representation that ignores order is crude, and crude is not the same as useless. Here is what the bag of words approach still does well, measured against the alternatives.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
import time
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline

print('%-24s %10s %10s' % ('', 'accuracy', 'seconds'))
for name, clf in [('naive bayes', MultinomialNB()),
                  ('logistic regression',
                   LogisticRegression(max_iter=2000)),
                  ('linear svm', LinearSVC()),
                  ('random forest',
                   RandomForestClassifier(n_estimators=150,
                                          random_state=0))]:
    pipe = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), clf)
    start = time.perf_counter()
    pipe.fit(train_x, train_s)
    took = time.perf_counter() - start
    print('%-24s %10.4f %10.2f'
          % (name, pipe.score(test_x, test_s), took))

                           accuracy    seconds
naive bayes                  1.0000       0.04
logistic regression          1.0000       0.03
linear svm                   1.0000       0.03
random forest                1.0000       0.78

Four quite different algorithms, all fed the same features, and the differences between them are far smaller than the difference between unigrams and bigrams was yesterday. That is the general finding in text work and it is worth internalising early: the representation matters more than the classifier.

When to stop here

A TF-IDF vectoriser and a linear model trains in under a second, runs on a laptop, needs no accelerator, and can be explained line by line to somebody who has to sign it off. For topic classification, routing, tagging and filtering it is frequently as good as anything else.

Week 7 introduces the models that beat it. They cost several orders of magnitude more to run, and it is worth knowing that this is the thing they have to beat.

Week 04 · Day 6 of 7

Text From the Real World

Unknown words, uneven lengths, and a vocabulary that will not stop growing

By Pawan Rama Mali949 words

Three practical problems that appear the moment text comes from the real world rather than a tidy corpus.

Words the model has never seen

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer().fit(train_x)
novel = 'the flange is thoroughly cromulent'
row = vec.transform([novel])
known = vec.inverse_transform(row)[0]
print('sentence: %s' % novel)
print('words the model recognises: %s' % list(known))
print('everything else was silently discarded')

sentence: the flange is thoroughly cromulent
words the model recognises: [np.str_('is'), np.str_('the')]
everything else was silently discarded

A word absent from the training vocabulary does not raise anything. It is dropped, and the model classifies what remains. A document made entirely of unfamiliar words becomes an empty row, and the model still returns a confident answer based on nothing at all.

Documents of very different lengths

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
import numpy as np

docs = ['sturdy pan', 'sturdy pan ' * 20]
for name, vec in [('counts', CountVectorizer()),
                  ('tfidf', TfidfVectorizer())]:
    X = vec.fit_transform(docs).toarray()
    print('%-8s row sums: %s' % (name, np.round(X.sum(axis=1), 3)))
print()
print('raw counts make a long document twenty times louder')
print('tfidf normalises each row to the same length')

counts   row sums: [ 2 40]
tfidf    row sums: [1.414 1.414]

raw counts make a long document twenty times louder
tfidf normalises each row to the same length

The vocabulary that keeps growing

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import CountVectorizer

print('%-28s %8s' % ('setting', 'columns'))
for name, kw in [('everything', {}),
                 ('seen at least 3 times', {'min_df': 3}),
                 ('in under 80% of docs', {'max_df': 0.8}),
                 ('top 200 by frequency', {'max_features': 200})]:
    vec = CountVectorizer(ngram_range=(1, 2), **kw).fit(train_x)
    print('%-28s %8d' % (name, len(vec.vocabulary_)))

setting                       columns
everything                        280
seen at least 3 times             280
in under 80% of docs              279
top 200 by frequency              200

min_df is the setting worth reaching for first. A term appearing once cannot generalise, it can only be memorised, and dropping those terms usually shrinks the model substantially without costing accuracy.

Week 04 · Day 7 of 7

A Text Pipeline End to End

Assembled, cross-validated, and reported per class

By Pawan Rama Mali621 words

The whole week as one pipeline, with the choices that were measured rather than assumed.

import random

random.seed(3)

CATEGORY_WORDS = {
    'laptop': ['screen', 'keyboard', 'battery', 'trackpad', 'charger'],
    'kitchen': ['pan', 'lid', 'handle', 'blade', 'kettle'],
    'clothing': ['stitching', 'zip', 'collar', 'sleeve', 'fabric'],
}
GOOD = ['excellent', 'solid', 'reliable', 'sturdy', 'sharp']
BAD = ['flimsy', 'loose', 'faulty', 'scratched', 'noisy']
# The same intensifiers appear on both sides, so a model that keys on
# 'very' alone learns nothing at all.
DEGREE = ['very', 'quite', 'surprisingly', 'really']
NEGATE = ['not', 'far from', 'hardly']
FRAME = [
    'the {part} is {opinion}, {tail}',
    'arrived quickly and the {part} was {opinion}, {tail}',
    'i wanted to like it but the {part} is {opinion}, {tail}',
    'after a month the {part} is still {opinion}, {tail}',
    'for the price the {part} is {opinion}, {tail}',
]
# One pool of closing phrases for both sentiments. An earlier version of
# this corpus gave each sentiment its own closers, which handed the
# label to anything that could read the last three words, and every
# model scored a perfect 1.000.
TAIL = ['ordered on tuesday', 'the packaging was fine',
        'second one i have bought', 'delivery took a week',
        'as described in the listing']

def make_review(category, positive):
    part = random.choice(CATEGORY_WORDS[category])
    # Half the positive reviews are written as a negated negative, so
    # 'not flimsy' has to be read as praise. Bag of words cannot.
    if positive:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(BAD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(GOOD))
        tail = random.choice(TAIL)
    else:
        if random.random() < 0.5:
            opinion = '%s %s' % (random.choice(NEGATE), random.choice(GOOD))
        else:
            opinion = '%s %s' % (random.choice(DEGREE), random.choice(BAD))
        tail = random.choice(TAIL)
    return random.choice(FRAME).format(part=part, opinion=opinion,
                                       tail=tail)

texts, sentiment, category = [], [], []
for _ in range(400):
    for cat in CATEGORY_WORDS:
        for pos in (True, False):
            texts.append(make_review(cat, pos))
            sentiment.append(1 if pos else 0)
            category.append(cat)

from sklearn.model_selection import train_test_split
(train_x, test_x, train_s, test_s,
 train_c, test_c) = train_test_split(texts, sentiment, category,
                                     test_size=0.3, random_state=0,
                                     stratify=sentiment)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report

pipe = make_pipeline(
    TfidfVectorizer(ngram_range=(1, 2), min_df=2),
    LogisticRegression(max_iter=2000))

base = cross_val_score(make_pipeline(TfidfVectorizer(),
                                     DummyClassifier(strategy='prior')),
                       train_x, train_s, cv=5)
got = cross_val_score(pipe, train_x, train_s, cv=5)
print('%-22s %10s %10s' % ('', 'mean', 'spread'))
print('%-22s %10.4f %10.4f' % ('baseline', base.mean(),
                               base.max() - base.min()))
print('%-22s %10.4f %10.4f' % ('the pipeline', got.mean(),
                               got.max() - got.min()))
print()
pipe.fit(train_x, train_s)
print(classification_report(test_s, pipe.predict(test_x),
                            target_names=['negative', 'positive']))

                             mean     spread
baseline                   0.5000     0.0000
the pipeline               1.0000     0.0000

              precision    recall  f1-score   support

    negative       1.00      1.00      1.00       360
    positive       1.00      1.00      1.00       360

    accuracy                           1.00       720
   macro avg       1.00      1.00      1.00       720
weighted avg       1.00      1.00      1.00       720

The checklist for any text task

Print some documents. Every decision below depends on what the text actually looks like.
Decide whether order matters for your task. Topic rarely needs it; opinion, negation and instruction always do.
Do not remove stop words before checking that not is not one of them.
Start with TF-IDF and a linear model. It is the baseline everything else has to beat.
Add bigrams and measure. If they help a lot, order matters and you should keep going in that direction.
Use min_df to drop terms that appear once.
Read the learned weights. They are the cheapest debugging tool in this field.

The finding to carry forward

On this data the choice between four different classifiers changed almost nothing, and the choice between unigrams and bigrams changed everything, from chance to perfect. Time spent on how text becomes numbers is worth more than time spent choosing the model that consumes them.