The Standard Language Tasks

Week 4 built one classifier. Most language work in industry is one of a handful of standard tasks, and knowing the list tells you what to search for and what a reasonable result looks like.

The first six are solvable to a useful standard with the methods in this week. The last two produce new text, which is a different kind of problem and is where week 7's models are genuinely required rather than merely fashionable.

Finding the people, places, dates and amounts in a document is one of the most commercially useful of these tasks, because it turns prose into rows in a table.

Rules first, as always

Dates, emails and phone numbers are genuinely well suited to rules: they have a defined format, the format rarely changes, and a regular expression is exact. This is week 3's lesson arriving in a language context.

Where rules stop

Apple is a company or a fruit depending on the sentence. Washington is a person, a city or a state. No list of suffixes settles these, because the evidence is the surrounding words, and using surrounding words as evidence is exactly what a learned model does.

Deciding what a document is about, without being given any categories, is a different shape of problem. Nobody supplies labels, so nothing can be scored against an answer key, and that changes how you work.

Keywords by weight

TF-IDF gives keyword extraction almost for free: the highest weighted terms in a document are, by construction, the ones that are frequent here and rare elsewhere, which is a workable definition of what something is about.

Themes across a whole collection

Nobody told it there were three product categories. It found groups of words that tend to occur together, and those groups correspond to the categories because that is the strongest structure in the text.

Similarity underpins search, deduplication, recommendation and retrieval. With documents already represented as vectors, it is one operation.

What word matching cannot do

That result is the clearest possible statement of the limit. Film and movie are different columns, so a paraphrase shares almost nothing. Excellent and terrible are also different columns, but the surrounding words are identical, so the opposite pair scores higher.

Sentiment deserves its own day because it is the task most often deployed carelessly, and because its failure modes are instructive.

The first two are handled, which is what week 4's bigrams bought. The rest are the standard hard cases and the model has no way to reach them: sarcasm inverts the meaning of positive words, faint praise sits between the classes, and a sentence with one positive and one negative clause has no single answer at all.

The problem with one label per document

• Mixed opinions. A review praising the blade and criticising the handle is not positive or negative, and forcing it into one is discarding the useful information.
• Aspect based sentiment is the version of the task that predicts an opinion per feature rather than per document, and it is usually what the business actually wanted.
• Sarcasm and understatement require knowing what is normal, which is context no sentence contains.
• Domain shift. A model trained on product reviews reads clinical notes badly, because positive means the opposite there.

Every task in this week needs a different measurement, and using the wrong one is how projects report success and deliver nothing.

Measuring entity extraction

Accuracy is meaningless here, because there is no fixed number of decisions being made. The measurement has to be over spans: how many of the ones you found were right, and how many of the real ones did you find. Note also that a span found with the wrong label counts as both a miss and an invention.

Measuring the unsupervised tasks

• Topics have no ground truth. Judge them by whether a person can name each theme, and by whether documents assigned to a theme belong together.
• Search and ranking are measured by where the good results land, not by whether they appear at all. A relevant document at position fifty is a failure.
• Summarisation is measured badly by every automatic metric available, all of which compare word overlap with a reference summary and none of which detect a fluent summary that says something untrue.

One document, run through several of the week's tasks, which is what a real pipeline looks like.

Four different techniques, each suited to its part of the job: a learned classifier for category, another for sentiment, a regular expression for the date, and vector similarity for retrieval. Nothing here needed a large model, and each piece can be measured on its own.

The checklist

1. Name the task before choosing a method. Most language work is one of eight standard shapes.
2. Use rules for anything with a fixed format, and learning for anything that depends on context.
3. For unsupervised tasks, accept that there is no score and design human review into the process.
4. Measure spans with precision and recall, not accuracy.
5. Check the hard cases by hand: negation, mixed opinion, sarcasm. The aggregate number hides all of them.
6. Before deploying anywhere new, test on text from that domain. Vocabulary does not transfer as well as it appears to.