4200 support tickets · the GRU won accuracy by 0.66 points
What We'll Cover
Introduction
Everyone agrees you should start with a baseline. Almost nobody publishes what happened when they did, because the interesting write-up is the one where the new method won.
Here are four comparisons from our own course material where it did not. In each case the simple method is within a point of the sophisticated one, or beats it outright, while costing between ten and a hundred times less to fit.
None of this is an argument against neural networks. It is an argument for knowing the size of what you are buying before you pay for it.
TF-IDF bigrams against three recurrent networks
Four thousand two hundred support tickets, classified by category. We fit a GRU, an LSTM, a bidirectional LSTM and a mean-pooled embedding, then the same problem with TF-IDF bigrams and logistic regression.
| Pos | model | accuracy | auc | fit time | Behind leader |
|---|---|---|---|---|---|
| 1 | gru | 0.9608 | 0.9610 | 10.1s | leader |
| 2 | lstm | 0.9570 | 0.9601 | 10.4s | -0.0038 |
| 3 | bilstm | 0.9561 | 0.9614 | 16.0s | -0.0047 |
| 4 | average | 0.9179 | 0.9049 | 3.0s | -0.0429 |
The GRU wins on accuracy by 0.66 points and on AUC by 0.19. It also takes 144 times longer to fit, and the bidirectional LSTM takes 228 times longer to end up slightly behind the linear model on accuracy.
The thousand and seventy-nine coefficients of the linear model can be read directly: you can sort them and see which phrases push a ticket into which category. Explaining the GRU's decision is a separate project.
0.66 points of accuracy is sometimes worth 144 times the fit time and an opaque model. Often it is not. The mistake is not picking one, it is not knowing the exchange rate.
A frozen transformer that lost to bag of words
Sentiment on a small review corpus, comparing three representations: raw word counts, TF-IDF with bigrams, and frozen features pulled out of a pretrained transformer.
| Pos | accuracy | seconds | Behind leader | |
|---|---|---|---|---|
| 1 | tfidf bigrams, week 4 | 1.0000 | 0.03 | leader |
| 2 | frozen transformer, week 7 | 0.6236 | 0.62 | -0.3764 |
| 3 | bag of words, week 4 | 0.4931 | 0.03 | -0.5069 |
| A perfect 1.0000 means the corpus is small and separable, not that TF-IDF is magnificent. All three saw identical data. | ||||
The transformer scored 0.6236 where TF-IDF bigrams scored 1.0000, and took twenty times longer to produce the features. Bag of words alone was at chance, which is the control that makes the bigram result meaningful: word order is doing the work here, and bigrams capture it at a cost of nothing.
The honest caveat, which we put in the lesson too: a perfect 1.0000 means the corpus is small and separable, not that TF-IDF is magnificent. The comparison is still fair, because all three methods saw exactly the same data.
What this shows is narrower and more useful than "transformers are overrated". Frozen features from a general-purpose model are not tuned to your task. Fine-tuning would very likely win. Frozen features are the cheap option, and the cheap option lost to an even cheaper one.
A support vector machine that beat the convolutional network
Handwritten digits, six classifiers, same held-out set.
| Pos | accuracy | Behind leader | |
|---|---|---|---|
| 1 | support vector machine | 0.9870 | leader |
| 2 | nearest neighbour | 0.9852 | -0.0018 |
| 3 | random forest | 0.9778 | -0.0092 |
| 4 | logistic regression | 0.9704 | -0.0166 |
| 5 | convolutional network | 0.8630 | -0.1240 |
| 6 | guess at random | 0.1111 | -0.8759 |
| 7 | guess the commonest digit | 0.1019 | -0.8851 |
The support vector machine came first at 0.9870. The convolutional network, the method the problem is famous for, came last of the trained models at 0.8630.
The reason is not that convolutions are bad at images. It is that this dataset has 1,257 training images at 8 by 8 pixels, and a convolutional network is a machine for exploiting large amounts of high-resolution data. Given a small, low-resolution, clean dataset, the classical methods are simply better suited.
Architecture families have operating ranges. A method being state of the art on a million images says nothing about it being right for your twelve hundred.
Keyword rules with no training data at all
The capstone of the AI course routes support tickets and predicts urgency. Before any model, the brief requires a rule-based baseline: a list of keywords, written by hand, in an afternoon.
| Pos | routing | urgency | Behind leader | |
|---|---|---|---|---|
| 1 | hand written keyword rules | 0.8689 | 0.6433 | leader |
| 2 | always the commonest answer | 0.3333 | 0.4878 | -0.5356 |
Nearly 87 percent of tickets routed correctly with no training data, no model, and no serving infrastructure. The trained models did beat it, and comfortably. The point is what the rules cost, and what they tell you: if your model cannot clear 0.8689, it is not learning anything a person could not have written down.
A rule baseline also gives you something a model cannot: it works on day one, it fails in ways you can read, and it is the fallback when the model is down.
What this does not mean
It does not mean simple methods usually win. In three of these four comparisons the sophisticated method did come out ahead. It means the margin is frequently much smaller than expected, and the cost difference much larger.
It also does not mean these results transfer to your problem. They will not. That is the whole argument: every one of these numbers came from measuring rather than assuming, and the only way to know your own exchange rate is to measure yours.
Run the baseline. Write down what it scored. Then decide whether the improvement is worth what it costs, with both numbers in front of you rather than one.
Where these numbers come from. They come from the Machine Learning and Artificial Intelligence programmes, both of which measure every method against a simpler one before accepting it. Every figure on this page was printed by code that was run, not written from what the result should have been. The course is free to read and the workbooks are free to download.