Meaning and Embeddings

Week 6 of 14 · Language · 7 days

Full curriculum
Week 06 · Language

Meaning and Embeddings

Week 06 · Day 1 of 7

You Shall Know a Word by the Company It Keeps

The distributional hypothesis, and counting context

By 829 words

Week 5 ended on a failure: two sentences meaning the same thing scored lower for similarity than two meaning the opposite. The cause is that every method so far treats words as arbitrary distinct symbols. Film and movie are simply two different columns, no more related than film and hydraulic.

The distributional hypothesis: Words that appear in similar contexts tend to mean similar things. It is the idea the whole of modern language processing rests on, it requires no dictionary and no linguist, and it is testable.
import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
print('%d sentences, %d distinct words' % (len(SENTENCES), len(VOCAB)))
for s in SENTENCES[:4]:
print(' ' + s)
20 sentences, 90 distinct words
the dog barked at the postman in the street
the cat slept on the warm windowsill all day
the dog slept on the rug by the fire
the cat chased a mouse across the kitchen floor

Counting company

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
for word in ['dog', 'car']:
row = M[INDEX[word]]
top = np.argsort(row)[::-1][:6]
print('%-6s appears near: %s'
% (word, ', '.join('%s(%d)' % (VOCAB[i], row[i])
for i in top if row[i] > 0)))
dog appears near: the(2), slept(1), on(1), barked(1), meat(1), at(1)
car appears near: the(3), needs(1), down(1), drove(1), fuel(1)

That row of counts is already a representation of the word, and it is one in which meaning is present. Nothing in the corpus states that a dog is an animal or a car is a vehicle. The information is in the company each word keeps.

Week 06 · Day 2 of 7

Similar Words Have Similar Rows

Neighbours from twenty sentences, and why counting will not scale

By 975 words

If similar words keep similar company, then similar words have similar rows, and similarity between rows is something we can compute.

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
for word in ['dog', 'car', 'bread', 'london']:
pairs = similar(M, word, 4)
print('%-8s -> %s' % (word,
', '.join('%s %.2f' % p for p in pairs)))
dog -> cat 0.73, slept 0.62, rug 0.62, postman 0.62
car -> van 1.00, i 0.78, in 0.74, slept 0.71
bread -> strong 0.52, for 0.49, ate 0.49, from 0.49
london -> manchester 0.79, to 0.32, large 0.32, passengers 0.32

Cat comes out near dog, and van near car, from twenty sentences and a counting loop. No dictionary was consulted and nobody defined any of these words.

This is the whole idea, at the smallest possible scale

Word vectors trained on billions of sentences do the same thing with more sophistication: they compress the counts, weight them better and use wider context. The mechanism above is genuinely what is happening underneath, which is worth knowing because it explains both the strengths and the failures in the rest of this week.

The problem with raw counts

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
print('the co-occurrence matrix is %d by %d' % M.shape)
print('%.1f%% of it is zero' % (100 * (M == 0).mean()))
print()
print('and it grows with the square of the vocabulary:')
for v in [100, 10_000, 1_000_000]:
print(' %9d words -> %15d cells' % (v, v * v))
the co-occurrence matrix is 90 by 90
94.3% of it is zero

and it grows with the square of the vocabulary:
100 words -> 10000 cells
10000 words -> 100000000 cells
1000000 words -> 1000000000000 cells

A million word vocabulary would need a matrix with a trillion cells, almost all of them zero. Raw counting does not scale, and the fix is to compress each row into a few hundred numbers that preserve the distances.

Week 06 · Day 3 of 7

Compressing Meaning Into a Few Numbers

Embeddings, neighbourhoods that survive, and the oversold analogy

By 1113 words

Compressing the matrix while keeping what matters is a standard operation, and it produces the thing everybody means by an embedding.

An embedding: A dense vector of a few hundred numbers representing a word, arranged so that words used similarly land near each other. Dense means almost every number is non-zero, which is the opposite of the sparse count rows and is why it is compact.
import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
from sklearn.decomposition import TruncatedSVD

# log scaling stops very common words dominating every direction
weighted = np.log1p(M)
E = TruncatedSVD(n_components=8, random_state=0).fit_transform(weighted)
print('each word is now %d numbers instead of %d' % (E.shape[1],
M.shape[1]))
print()
for word in ['dog', 'car', 'bread', 'london']:
pairs = similar(E, word, 4)
print('%-8s -> %s' % (word,
', '.join('%s %.2f' % p for p in pairs)))
each word is now 8 numbers instead of 90

dog -> on 0.95, cat 0.93, slept 0.89, warm 0.87
car -> van 1.00, market 0.98, i 0.97, kitchen 0.96
bread -> cheese 0.97, was 0.92, jam 0.91, came 0.90
london -> manchester 0.93, passengers 0.80, hour 0.73, is 0.70

Eight numbers per word instead of one column per vocabulary entry, and the neighbourhoods survive. That compression is what makes embeddings practical: a real one is three hundred numbers for a vocabulary of millions.

Directions in the space mean something

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
from sklearn.decomposition import TruncatedSVD

E = TruncatedSVD(n_components=8,
random_state=0).fit_transform(np.log1p(M))

def analogy(a, b, c, n=3):
"""a is to b as c is to what."""
target = E[INDEX[b]] - E[INDEX[a]] + E[INDEX[c]]
norms = np.linalg.norm(E, axis=1) * np.linalg.norm(target) + 1e-9
sims = (E @ target) / norms
out = []
for i in np.argsort(sims)[::-1]:
if VOCAB[i] not in (a, b, c):
out.append((VOCAB[i], float(sims[i])))
if len(out) == n:
return out

print('dog is to puppy as cat is to ...')
print(' %s' % analogy('dog', 'puppy', 'cat'))
print()
print('car is to motorway as train is to ...')
print(' %s' % analogy('car', 'motorway', 'train'))
dog is to puppy as cat is to ...
[('chased', 0.911852059028957), ('across', 0.8425314884061901), ('mouse', 0.8273696571298044)]

car is to motorway as train is to ...
[('to', 0.92566759284489), ('passengers', 0.8777832552791965), ('carried', 0.8417030748678817)]

Analogies are the most oversold result in this field

On twenty sentences the answers are wobbly, and even on large embeddings the famous examples are curated. The published results typically exclude the three input words from the answer, which is doing a substantial amount of the work, and most analogies you try yourself will not come out.

The genuinely reliable property is the first one: similar words are near each other. That is what embeddings are used for in practice, and it is enough.

Week 06 · Day 4 of 7

From Words to Documents

Averaging vectors, and the order it still cannot see

By 955 words

A document is more than a bag of its words' meanings, but averaging the vectors is a surprisingly serviceable start and it fixes week 5's paraphrase problem.

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
from sklearn.decomposition import TruncatedSVD

E = TruncatedSVD(n_components=8,
random_state=0).fit_transform(np.log1p(M))

def embed(sentence):
vecs = [E[INDEX[w]] for w in sentence.split() if w in INDEX]
return np.mean(vecs, axis=0) if vecs else np.zeros(E.shape[1])

def sim(a, b):
va, vb = embed(a), embed(b)
return float(va @ vb / (np.linalg.norm(va) * np.linalg.norm(vb)
+ 1e-9))

pairs = [('the dog slept', 'the cat slept'),
('the dog slept', 'the van needs fuel'),
('i drove the car', 'i drove the van')]
for a, b in pairs:
print('%.3f %-22s %s' % (sim(a, b), a, b))
0.997 the dog slept the cat slept
0.944 the dog slept the van needs fuel
1.000 i drove the car i drove the van

What averaging throws away

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
from sklearn.decomposition import TruncatedSVD

E = TruncatedSVD(n_components=8,
random_state=0).fit_transform(np.log1p(M))

def embed(sentence):
vecs = [E[INDEX[w]] for w in sentence.split() if w in INDEX]
return np.mean(vecs, axis=0) if vecs else np.zeros(E.shape[1])

a = embed('the dog chased the cat')
b = embed('the cat chased the dog')
print('identical vectors: %s' % bool(np.allclose(a, b)))
print('averaging is order blind, exactly like bag of words')
identical vectors: True
averaging is order blind, exactly like bag of words

Two sentences with opposite meanings and the same average. Embeddings solved the vocabulary problem, they did not solve the order problem, and that is precisely the gap the transformer architecture in week 7 was built to close.

Week 06 · Day 5 of 7

Embeddings Learn the Assumptions Too

An association nobody wrote down, recorded faithfully

By 390 words

Embeddings learn from the text they are given, which includes everything that text assumes. This is not a defect that better training removes, it is what the method does.

import numpy as np

# A corpus with a pattern nobody put there deliberately, of exactly
# the kind found in real text scraped from the web.
SENTENCES = [
'the nurse checked her patients before the ward round',
'the nurse collected her notes from the desk',
'the surgeon washed his hands before the operation',
'the surgeon reviewed his notes before the operation',
'the secretary typed her letters in the morning',
'the engineer checked his drawings in the workshop',
'she trained as a nurse at the local hospital',
'he trained as a surgeon at the teaching hospital',
'she worked as a secretary for the practice',
'he worked as an engineer for the firm',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
M = np.zeros((len(VOCAB), len(VOCAB)))
for s in TOKENS:
for i, w in enumerate(s):
for j in range(max(0, i - 3), min(len(s), i + 4)):
if i != j:
M[INDEX[w], INDEX[s[j]]] += 1

def closeness(a, b):
va, vb = M[INDEX[a]], M[INDEX[b]]
return float(va @ vb / (np.linalg.norm(va) * np.linalg.norm(vb)
+ 1e-9))

print('%-12s %8s %8s' % ('job', 'she/her', 'he/his'))
for job in ['nurse', 'surgeon', 'secretary', 'engineer']:
fem = max(closeness(job, w) for w in ['she', 'her'])
masc = max(closeness(job, w) for w in ['he', 'his'])
print('%-12s %8.3f %8.3f' % (job, fem, masc))
job she/her he/his
nurse 0.663 0.591
surgeon 0.573 0.650
secretary 0.572 0.480
engineer 0.531 0.560

The corpus contains no statement about who can do which job. It contains a statistical association, and the representation records it faithfully, because recording associations is the entire mechanism.

Why this matters more than it first appears

An embedding is rarely the final product. It is an input to a screening model, a search ranker or a recommender, and the association travels into all of them silently. A CV screening system built on embeddings can rank candidates by association with a job title without any field for gender ever being present.

Removing the obvious words does not fix it, because the association is distributed across the whole vocabulary. Week 11 returns to this properly with measurements and remedies.

Week 06 · Day 6 of 7

Where Static Vectors Run Out

One vector per word, and nothing for words never seen

By 983 words

Two limits of static embeddings, both of which motivate what came next.

One vector per word, whatever it means

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
for word in ['london', 'street']:
print('%-8s -> %s' % (word, [w for w, _ in similar(M, word, 4)]))
print()
print('the word "bank" would get one vector averaging')
print('the riverside sense and the financial sense together,')
print('which is a compromise that suits neither')
london -> ['manchester', 'to', 'large', 'passengers']
street -> ['garden', 'postman', 'in', 'ball']

the word "bank" would get one vector averaging
the riverside sense and the financial sense together,
which is a compromise that suits neither

A static embedding assigns one vector per word type. Every sense of the word is blended into it, weighted by how common each sense is in the training text. For genuinely ambiguous words the resulting vector represents nothing that anybody ever means.

Nothing outside the vocabulary

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
for word in ['dog', 'aardvark', 'refactoring']:
print('%-14s in vocabulary: %s' % (word, word in INDEX))
print()
print('a word not in the training text has no vector at all,')
print('so misspellings, product codes and new terms are invisible')
dog in vocabulary: True
aardvark in vocabulary: False
refactoring in vocabulary: False

a word not in the training text has no vector at all,
so misspellings, product codes and new terms are invisible

What fixed both

  • Contextual embeddings. Compute the vector from the sentence rather than looking it up, so bank gets a different vector in each sense. This is what a transformer produces and it is the main reason they replaced static vectors.
  • Subword tokenisation. Break rare words into pieces that are in the vocabulary, so an unseen word still receives a representation assembled from familiar parts.
  • Sentence embeddings. Models trained to place whole sentences rather than words, which is what modern semantic search and retrieval actually use.
Week 06 · Day 7 of 7

Words Against Meaning

The same question answered both ways, and what comes next

By 697 words

The week in one comparison: the same similarity question, answered by word matching and by embeddings.

import numpy as np

# A small corpus written so that related words share company. Nothing
# in it states that a dog is like a cat; the sentences merely use them
# in the same way, which turns out to be enough.
SENTENCES = [
'the dog barked at the postman in the street',
'the cat slept on the warm windowsill all day',
'the dog slept on the rug by the fire',
'the cat chased a mouse across the kitchen floor',
'my dog eats meat and biscuits every morning',
'my cat eats fish and biscuits every morning',
'the puppy barked and chased the ball in the garden',
'the kitten slept and chased a ball of wool',
'i drove the car down the motorway to london',
'i drove the van down the motorway to manchester',
'the car needs fuel and a service before winter',
'the van needs fuel and new tyres before winter',
'the lorry carried freight along the motorway at night',
'the train carried passengers to london every hour',
'she ate bread and cheese for lunch at her desk',
'he ate bread and jam for breakfast before work',
'the bread was fresh from the bakery this morning',
'the cheese was strong and came from the market',
'london is a large city with an old river',
'manchester is a large city with a busy centre',
]
TOKENS = [s.split() for s in SENTENCES]
VOCAB = sorted({w for s in TOKENS for w in s})
INDEX = {w: i for i, w in enumerate(VOCAB)}
def cooccurrence(window=2):
"""Count how often each pair of words appears near each other."""
M = np.zeros((len(VOCAB), len(VOCAB)))
for sentence in TOKENS:
for i, word in enumerate(sentence):
lo = max(0, i - window)
hi = min(len(sentence), i + window + 1)
for j in range(lo, hi):
if j != i:
M[INDEX[word], INDEX[sentence[j]]] += 1
return M

M = cooccurrence()

def similar(vectors, word, n=5):
v = vectors[INDEX[word]]
norms = np.linalg.norm(vectors, axis=1) * np.linalg.norm(v) + 1e-9
sims = (vectors @ v) / norms
order = np.argsort(sims)[::-1]
return [(VOCAB[i], float(sims[i])) for i in order if VOCAB[i] != word][:n]
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

E = TruncatedSVD(n_components=8,
random_state=0).fit_transform(np.log1p(M))

def embed(sentence):
vecs = [E[INDEX[w]] for w in sentence.split() if w in INDEX]
return np.mean(vecs, axis=0) if vecs else np.zeros(E.shape[1])

def emb_sim(a, b):
va, vb = embed(a), embed(b)
return float(va @ vb / (np.linalg.norm(va) * np.linalg.norm(vb)
+ 1e-9))

tfidf = TfidfVectorizer().fit(SENTENCES)
def word_sim(a, b):
return float(cosine_similarity(tfidf.transform([a]),
tfidf.transform([b]))[0][0])

pairs = [('the dog slept', 'the cat slept'),
('i drove the car', 'i drove the van'),
('the dog barked', 'the puppy barked')]
print('%-40s %10s %10s' % ('', 'words', 'embedding'))
for a, b in pairs:
label = '%s / %s' % (a, b)
print('%-40s %10.3f %10.3f' % (label, word_sim(a, b),
emb_sim(a, b)))
words embedding
the dog slept / the cat slept 0.556 0.997
i drove the car / i drove the van 0.547 1.000
the dog barked / the puppy barked 0.537 0.990

On every pair the embedding scores the relatedness higher, because it knows that dog and cat keep similar company while TF-IDF only knows they are different columns.

What the week established

  • Meaning can be derived from context alone, with no dictionary, because words used similarly are used in similar company.
  • Compressing the co-occurrence counts gives dense vectors that keep the neighbourhoods and fit in memory.
  • Nearest neighbours are the reliable property. Analogies are largely a demonstration.
  • Averaging word vectors gives a usable document vector and remains blind to word order.
  • Whatever associations exist in the training text end up in the vectors, and travel silently into anything built on them.
  • One vector per word cannot represent an ambiguous word, and nothing outside the vocabulary gets a vector at all.

The two problems week 7 solves

Static embeddings fixed vocabulary and left order and ambiguity untouched. Producing a different vector for a word depending on the sentence around it fixes both at once, and that single idea is what the transformer does and why everything since has been built on it.