Measuring a Model Honestly

Week 2 of 14 · Foundations · 7 days

Full curriculum
Week 02 · Foundations

Measuring a Model Honestly

Week 02 · Day 1 of 7

The Model That Catches Nothing

Why accuracy is the wrong metric for anything rare

By 658 words

Week 1 ended with a warning: an accuracy figure with no baseline beside it means nothing. This week takes that apart properly, because on real data accuracy is not merely unhelpful, it is actively misleading.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
2800 training rows, 1200 held out
fraud is 4.7% of the data

The number that looks like success

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression

lazy = DummyClassifier(strategy='most_frequent').fit(Xtr, ytr)
model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
print('%-34s %10s' % ('', 'accuracy'))
print('%-34s %10.4f' % ('predict "not fraud" every time',
lazy.score(Xte, yte)))
print('%-34s %10.4f' % ('an actual model', model.score(Xte, yte)))
2800 training rows, 1200 held out
fraud is 4.7% of the data
accuracy
predict "not fraud" every time 0.9525
an actual model 0.9650

A model that has never once said the word fraud is 96 percent accurate. Put that figure in a slide and it reads as a triumph. It catches nothing, it would save nobody any money, and it is a single line of code that ignores its input entirely.

This is the most common dishonest number in the field

It is rarely dishonest on purpose. Accuracy is the default metric in every library, it is the one non-specialists ask for, and on imbalanced data it is dominated entirely by the majority class. Any problem where the interesting event is rare, and most valuable problems are, has this property.

What to look at instead

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
tn, fp, fn, tp = confusion_matrix(yte, model.predict(Xte)).ravel()
print('%-28s %6d' % ('fraud caught', tp))
print('%-28s %6d' % ('fraud missed', fn))
print('%-28s %6d' % ('honest flagged as fraud', fp))
print('%-28s %6d' % ('honest left alone', tn))
print()
print('of the %d real fraud cases, it found %d' % (tp + fn, tp))
2800 training rows, 1200 held out
fraud is 4.7% of the data
fraud caught 18
fraud missed 39
honest flagged as fraud 3
honest left alone 1140

of the 57 real fraud cases, it found 18
The confusion matrix: The four counts above. Every classification metric is some ratio of them, and looking at the four directly is almost always more informative than any single number derived from them, because the four are what a decision maker actually cares about.
Week 02 · Day 2 of 7

Precision, Recall and the Threshold

One model, five different systems, and who decides which

By 869 words

Two ratios of those four counts come up constantly, they are easy to confuse, and the difference between them is usually the whole conversation with whoever is paying for the system.

Precision: Of the cases the model flagged, what share were really fraud. Low precision means you are wasting people's time investigating honest customers.
Recall: Of the cases that really were fraud, what share did the model flag. Low recall means fraud is getting through.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score, f1_score

model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
pred = model.predict(Xte)
print('%-14s %8.4f' % ('accuracy', model.score(Xte, yte)))
print('%-14s %8.4f' % ('precision', precision_score(yte, pred)))
print('%-14s %8.4f' % ('recall', recall_score(yte, pred)))
print('%-14s %8.4f' % ('f1', f1_score(yte, pred)))
2800 training rows, 1200 held out
fraud is 4.7% of the data
accuracy 0.9650
precision 0.8571
recall 0.3158
f1 0.4615

You get to choose the trade

A classifier does not really output a class. It outputs a score, and somebody chose a threshold of 0.5 to turn that into a decision. That choice is yours, it is not a technical detail, and moving it moves precision and recall in opposite directions.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score

model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
scores = model.predict_proba(Xte)[:, 1]
print('%10s %10s %10s %12s' % ('threshold', 'precision', 'recall',
'flagged'))
for t in [0.05, 0.10, 0.25, 0.50, 0.75]:
pred = (scores >= t).astype(int)
if pred.sum() == 0:
continue
print('%10.2f %10.4f %10.4f %12d'
% (t, precision_score(yte, pred), recall_score(yte, pred),
pred.sum()))
2800 training rows, 1200 held out
fraud is 4.7% of the data
threshold precision recall flagged
0.05 0.1674 0.7018 239
0.10 0.3274 0.6491 113
0.25 0.5750 0.4035 40
0.50 0.8571 0.3158 21
0.75 1.0000 0.1228 7

One model, five different systems. At a low threshold it catches far more fraud and sends many more honest customers to be investigated. At a high threshold it is almost always right when it speaks and it stays quiet through most of the fraud.

Neither column is the right answer on its own

The right threshold depends on what each error costs. If investigating a flagged case costs five pounds and a missed fraud costs five hundred, the arithmetic is not close and you want the low threshold. If flagging a customer means freezing their account, the calculation changes completely.

That is a business decision informed by a table like this one. It is not a hyperparameter, and it should not be left at whatever the library defaulted to.

The metric that does not need a threshold

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.metrics import roc_auc_score, average_precision_score

model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
scores = model.predict_proba(Xte)[:, 1]
coin = DummyClassifier(strategy='stratified',
random_state=0).fit(Xtr, ytr)
print('%-34s %10s %12s' % ('', 'roc auc', 'avg precision'))
print('%-34s %10.4f %12.4f'
% ('the model', roc_auc_score(yte, scores),
average_precision_score(yte, scores)))
print('%-34s %10.4f %12.4f'
% ('random guessing',
roc_auc_score(yte, coin.predict_proba(Xte)[:, 1]),
yte.mean()))
2800 training rows, 1200 held out
fraud is 4.7% of the data
roc auc avg precision
the model 0.8281 0.5114
random guessing 0.4891 0.0475

ROC AUC asks whether the model ranks a real fraud above an honest case more often than not, across every possible threshold. Random guessing scores 0.5 by construction, which makes it easy to read. Average precision does something similar but weights the top of the ranking, and its random baseline is the fraud rate itself, which is why the second column's two numbers are so far apart.

Week 02 · Day 3 of 7

One Number Is Not a Measurement

Split luck, cross-validation, and reporting the spread

By 558 words

A single held out set gives you one number, measured once, on one particular slice of the data. Whether you can trust it depends on how big that slice is and how lucky the split was.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

print('%8s %12s' % ('split seed', 'roc auc'))
aucs = []
for seed in range(6):
a, b, ya, yb = train_test_split(X, y, test_size=0.3, stratify=y,
random_state=seed)
m = LogisticRegression(max_iter=2000).fit(a, ya)
auc = roc_auc_score(yb, m.predict_proba(b)[:, 1])
aucs.append(auc)
print('%8d %12.4f' % (seed, auc))
print()
print('spread across splits %.4f' % (max(aucs) - min(aucs)))
2800 training rows, 1200 held out
fraud is 4.7% of the data
split seed roc auc
0 0.8281
1 0.8270
2 0.8429
3 0.8613
4 0.8357
5 0.8045

spread across splits 0.0569

Same data, same model, same code. Only which rows landed in which half differs, and the answer moves. That spread is the resolution of your measurement: any improvement smaller than it has not been demonstrated.

Cross-validation: Split the data into k parts, train on k minus 1 and test on the remaining one, k times over, so every row is tested on exactly once. You get k numbers instead of one, and their spread tells you how much to trust the average.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
print('%-24s %10s %10s' % ('', 'mean auc', 'spread'))
for name, m in [('logistic regression',
LogisticRegression(max_iter=2000)),
('random forest',
RandomForestClassifier(n_estimators=120,
random_state=0))]:
s = cross_val_score(m, X, y, cv=cv, scoring='roc_auc')
print('%-24s %10.4f %10.4f' % (name, s.mean(), s.max() - s.min()))
2800 training rows, 1200 held out
fraud is 4.7% of the data
mean auc spread
logistic regression 0.8322 0.0951
random forest 0.8585 0.0711

Stratify, or the folds will not be comparable

With fraud at four percent, an ordinary five-way split can easily leave one fold with noticeably fewer fraud cases than another, and then the folds are measuring different problems. StratifiedKFold keeps the class balance the same in every fold. On imbalanced data it is not an optimisation, it is the difference between a meaningful average and a noisy one.

Week 02 · Day 4 of 7

Leakage

How to get an excellent score on a model that cannot work

By 520 words

There is a way to get an excellent held out score on a model that will fail completely in production, and it is common enough to have a name.

Leakage: When information that would not be available at prediction time leaks into training. The model learns to use it, the held out score is excellent because the held out data has the same leak, and the system collapses the moment it meets real input.

The most common version, demonstrated

Scaling the data before splitting it. It looks harmless: the scaler only computes a mean and a standard deviation. But it computes them over the test rows too, so information about the held out set is baked into the training data.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

# wrong: the scaler sees every row, including the fold being tested on
leaked = StandardScaler().fit_transform(X)
bad = cross_val_score(LogisticRegression(max_iter=2000), leaked, y,
cv=cv, scoring='roc_auc').mean()

# right: the scaler is part of the pipeline, refitted inside each fold
good = cross_val_score(make_pipeline(StandardScaler(),
LogisticRegression(max_iter=2000)),
X, y, cv=cv, scoring='roc_auc').mean()
print('%-38s %10.4f' % ('scaled before splitting', bad))
print('%-38s %10.4f' % ('scaled inside the pipeline', good))
print('%-38s %10.4f' % ('difference', bad - good))
2800 training rows, 1200 held out
fraud is 4.7% of the data
scaled before splitting 0.8326
scaled inside the pipeline 0.8325
difference 0.0001

On this data the gap is small, and that is exactly why leakage is dangerous rather than reassuring. A small unearned advantage does not look like a bug. It looks like your model being slightly better than the alternative, which is the conclusion you were hoping for.

The versions that are not small

  • A feature computed from the future. Predicting whether a customer will churn, using their total spend, which includes spending after they churned.
  • An identifier. A customer number that happens to correlate with the outcome because of how the data was collected.
  • Duplicate rows across the split. The same case in both halves, so the model is tested on something it memorised.
  • Choosing features by looking at all the data. Picking the twenty columns most correlated with the target, over the whole set, before splitting.

The rule that prevents all of them

Every step that learns anything from the data, and that includes scalers, encoders, imputers and feature selectors, belongs inside the pipeline so it is refitted on each training fold. If you find yourself calling fit on anything before train_test_split, stop.

Week 02 · Day 5 of 7

Working With Imbalance

Class weights measured, and what they actually change

By 421 words

Imbalanced data does not only break the metric. It also affects training, because a loss averaged over every row is dominated by the majority class. There are standard remedies and they are worth measuring rather than assuming.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (roc_auc_score, average_precision_score,
recall_score, precision_score)

def report(name, model):
model.fit(Xtr, ytr)
s = model.predict_proba(Xte)[:, 1]
pred = model.predict(Xte)
print('%-32s %8.4f %10.4f %10.4f %8.4f'
% (name, roc_auc_score(yte, s),
average_precision_score(yte, s),
precision_score(yte, pred, zero_division=0),
recall_score(yte, pred)))

print('%-32s %8s %10s %10s %8s'
% ('', 'auc', 'avg prec', 'precision', 'recall'))
report('logistic regression',
LogisticRegression(max_iter=2000))
report(' with balanced class weights',
LogisticRegression(max_iter=2000, class_weight='balanced'))
report('random forest',
RandomForestClassifier(n_estimators=200, random_state=0))
report(' with balanced class weights',
RandomForestClassifier(n_estimators=200, random_state=0,
class_weight='balanced'))
2800 training rows, 1200 held out
fraud is 4.7% of the data
auc avg prec precision recall
logistic regression 0.8281 0.5114 0.8571 0.3158
with balanced class weights 0.8223 0.5057 0.1639 0.6842
random forest 0.8736 0.5400 0.8889 0.2807
with balanced class weights 0.8724 0.5606 0.8462 0.1930

Read the ranking columns and the decision columns separately, because they answer different questions. Class weighting mostly does not change how well the model ranks cases, which is what AUC and average precision measure. What it changes is where the default threshold falls, which moves precision and recall a long way.

Class weighting is often just threshold moving in disguise

That is worth knowing because it is usually presented as a way to make the model better at finding the rare class. If the ranking metric barely moves, the model has not got better at the problem, it has become more willing to say yes, and you could have achieved the same thing by choosing a threshold from yesterday's table.

The advantage of doing it explicitly with a threshold is that you can see the trade you are making and pick the point you actually want.

Week 02 · Day 6 of 7

Two More Ways to Fool Yourself

Tuning on the test set, and not knowing the human ceiling

By 486 words

Two more things that quietly invalidate a measurement, both of which look like good practice while you are doing them.

Tuning on the test set

You try eight models, pick the one with the best held out score, and report that score. The number is now optimistic, because you used the held out set to make a choice, which makes it a training set for that choice.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

# a proper three way split: train, choose on validation, report on test
Xa, Xrest, ya, yrest = train_test_split(X, y, test_size=0.4,
stratify=y, random_state=0)
Xv, Xt, yv, yt = train_test_split(Xrest, yrest, test_size=0.5,
stratify=yrest, random_state=0)

best, best_c = -1, None
for c in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
m = LogisticRegression(C=c, max_iter=2000).fit(Xa, ya)
v = roc_auc_score(yv, m.predict_proba(Xv)[:, 1])
if v > best:
best, best_c = v, c

final = LogisticRegression(C=best_c, max_iter=2000).fit(Xa, ya)
test = roc_auc_score(yt, final.predict_proba(Xt)[:, 1])
print('chose C=%g on validation, scoring %.4f' % (best_c, best))
print('the same model on the untouched test set %.4f' % test)
print('the validation figure was optimistic by %.4f' % (best - test))
2800 training rows, 1200 held out
fraud is 4.7% of the data
chose C=10 on validation, scoring 0.8285
the same model on the untouched test set 0.8152
the validation figure was optimistic by 0.0133

The validation score is the one you used to choose, so it is the highest of six numbers and is biased upward by that selection alone. The test score is the honest one. With six candidates the gap is modest; with two hundred it is not.

Not knowing what a person would score

A model at 0.85 sounds mediocre until you learn that two trained annotators only agree with each other 0.87 of the time. Above that ceiling the remaining errors are largely disagreement about what the right answer even is, and no model will fix them.

  • Have two people label the same two hundred cases and measure how often they agree. That is your ceiling.
  • If they agree rarely, the task definition is the problem, not the model. Fix the definition first.
  • Report the ceiling next to your score. A model at 0.85 against a human ceiling of 0.87 is a very different result from 0.85 against 0.99.
Week 02 · Day 7 of 7

The Honest Procedure

Everything assembled, and the checklist

By 472 words

The whole week as one procedure, in the order that keeps it honest.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# A fraud-shaped problem: one case in twenty five is real, which is
# roughly what payment fraud looks like and nothing like a textbook set.
X, y = make_classification(n_samples=4000, n_features=12, n_informative=5,
n_redundant=2, weights=[0.96, 0.04],
flip_y=0.02, class_sep=0.9, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
print('%d training rows, %d held out' % (len(Xtr), len(Xte)))
print('fraud is %.1f%% of the data' % (100 * y.mean()))
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.metrics import (confusion_matrix, roc_auc_score,
precision_score, recall_score)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))

# 1. baseline, 2. model, both cross validated, both on a ranking metric
base = cross_val_score(DummyClassifier(strategy='stratified',
random_state=0),
X, y, cv=cv, scoring='roc_auc')
got = cross_val_score(pipe, X, y, cv=cv, scoring='roc_auc')
print('%-22s %10s %10s' % ('', 'mean auc', 'spread'))
print('%-22s %10.4f %10.4f' % ('baseline', base.mean(),
base.max() - base.min()))
print('%-22s %10.4f %10.4f' % ('model', got.mean(),
got.max() - got.min()))
print()

# 3. choose a threshold deliberately, on validation not on test
pipe.fit(Xtr, ytr)
s = pipe.predict_proba(Xte)[:, 1]
for t in [0.10, 0.50]:
pred = (s >= t).astype(int)
tn, fp, fn, tp = confusion_matrix(yte, pred).ravel()
print('threshold %.2f caught %3d missed %3d false alarms %3d'
% (t, tp, fn, fp))
2800 training rows, 1200 held out
fraud is 4.7% of the data
mean auc spread
baseline 0.4939 0.0553
model 0.8325 0.0950

threshold 0.10 caught 37 missed 20 false alarms 77
threshold 0.50 caught 17 missed 40 false alarms 3

The checklist

  1. State the class balance before anything else. It decides which metrics are meaningful.
  2. Run a do-nothing baseline and keep it beside every later number.
  3. Put every step that learns from data inside a pipeline.
  4. Cross-validate, and report the spread as well as the average.
  5. Choose the metric from the decision being made, not from the library default. On imbalanced problems that is rarely accuracy.
  6. Choose the threshold deliberately, from the cost of each kind of error.
  7. Keep a test set that has influenced no decision at all, and look at it once.
  8. Find out what a person scores on the same task.

What this week is really teaching

Almost every inflated result in applied machine learning comes from one of four things: the wrong metric for an imbalanced problem, information leaking across the split, a number that was selected for being the highest, or no baseline. None of them is a modelling error and none is caught by better algorithms. They are caught by the procedure above.