Model Evaluation and Validation

Week 7 of 16 · Evaluation and features · 7 days

Week 07 · Evaluation and features

Model Evaluation and Validation

Week 07 · Day 1 of 7

Cross-Validation Done Properly

Stratified, grouped and time-ordered folds, and the nested version

You have used cross_val_score since week 4 without asking how the folds are made. That choice is where most silently broken evaluations come from.

Why one split is not enough

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import numpy as np

X, y = df[NUM + CAT], df[TARGET]
scores = []
for seed in range(12):
    a, b, c, d = train_test_split(X, y, test_size=0.25, stratify=y,
                                  random_state=seed)
    m = Pipeline([('prep', preprocessor()),
                  ('clf', LogisticRegression(max_iter=1000))]).fit(a, c)
    scores.append(roc_auc_score(d, m.predict_proba(b)[:, 1]))

scores = np.array(scores)
print('twelve different random splits of the same data:')
print('  min %.4f   max %.4f   spread %.4f'
      % (scores.min(), scores.max(), scores.max() - scores.min()))
print('  mean %.4f  std %.4f' % (scores.mean(), scores.std()))

twelve different random splits of the same data:
  min 0.8030   max 0.8364   spread 0.0334
  mean 0.8207  std 0.0108

Two points of AUC, from nothing but the seed

Same data, same model, same code, only the split changed. If you compare two models on one split each and they differ by less than this spread, you have measured the seed, not the models. Reporting a single split's score to four decimal places implies a precision that does not exist.

The splitters, and when each is required

Splitter	Use when	Failure if you use KFold instead
`KFold`	Rows are independent and balanced	-
`StratifiedKFold`	Classification, any imbalance	Folds get different class rates; scores wobble
`GroupKFold`	Rows cluster by entity	The same entity lands in train and test, leakage
`TimeSeriesSplit`	Rows are ordered in time	The model trains on the future
`RepeatedStratifiedKFold`	You need a tighter estimate	-

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.model_selection import KFold, StratifiedKFold
import numpy as np

y = df[TARGET].to_numpy()
print('overall churn rate %.4f\n' % y.mean())

for name, splitter in [('KFold          ', KFold(5, shuffle=True, random_state=0)),
                       ('StratifiedKFold', StratifiedKFold(5, shuffle=True,
                                                           random_state=0))]:
    rates = [y[test].mean() for _, test in splitter.split(np.zeros(len(y)), y)]
    print('%s fold churn rates %s  spread %.4f'
          % (name, np.round(rates, 4), max(rates) - min(rates)))

overall churn rate 0.2680

KFold           fold churn rates [0.265  0.2733 0.25   0.2933 0.2583]  spread 0.0433
StratifiedKFold fold churn rates [0.2683 0.2683 0.2683 0.2683 0.2667]  spread 0.0017

Grouped data, where the trap is worst

import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score, KFold, GroupKFold
from sklearn.ensemble import RandomForestClassifier

rng = np.random.default_rng(0)
n_customers, per_customer = 200, 5

# Five readings per customer. The label belongs to the customer, not the
# row, and each reading is that customer's traits plus a little noise --
# which is exactly what repeated measurement of one entity looks like.
customer = np.repeat(np.arange(n_customers), per_customer)
label = np.repeat(rng.integers(0, 2, n_customers), per_customer)
traits = np.repeat(rng.normal(size=(n_customers, 4)), per_customer, axis=0)
X = traits + rng.normal(0, 0.02, traits.shape)

clf = RandomForestClassifier(n_estimators=100, random_state=0)
naive = cross_val_score(clf, X, label, cv=KFold(5, shuffle=True, random_state=0))
grouped = cross_val_score(clf, X, label, cv=GroupKFold(5), groups=customer)

print('KFold      accuracy %.4f   <- the same customer is in both halves'
      % naive.mean())
print('GroupKFold accuracy %.4f   <- honest' % grouped.mean())

KFold      accuracy 0.9970   <- the same customer is in both halves
GroupKFold accuracy 0.5880   <- honest

Ninety-nine percent, and the model has learned nothing

Forty-one points of pure illusion. Repeated measurements of one patient, several sessions from one user, multiple photographs of one object, split at random and near-duplicates of every test row sit in the training set. The model does not learn the pattern; it memorises which customer each reading came from and looks up their label. GroupKFold keeps a customer's rows together and reveals the truth: 0.59, barely above chance.

This is the most expensive mistake in this course, because the model looks superb right up until it meets someone new. Whenever rows cluster by an entity, pass groups.

Nested cross-validation

If you tune hyperparameters by cross-validation and then report that same cross-validated score, the score is optimistic: you chose the settings that happened to suit those folds.

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import (GridSearchCV, cross_val_score,
                                     StratifiedKFold)
import numpy as np

X, y = df[NUM + CAT], df[TARGET]
grid = {'clf__min_samples_leaf': [1, 5, 20, 50]}
pipe = Pipeline([('prep', preprocessor()),
                 ('clf', RandomForestClassifier(n_estimators=150,
                                                random_state=42))])

inner = StratifiedKFold(4, shuffle=True, random_state=1)
outer = StratifiedKFold(4, shuffle=True, random_state=2)

search = GridSearchCV(pipe, grid, cv=inner, scoring='roc_auc')
search.fit(X, y)
print('best inner score (optimistic) %.4f' % search.best_score_)

nested = cross_val_score(search, X, y, cv=outer, scoring='roc_auc')
print('nested score     (honest)     %.4f  +/- %.4f'
      % (nested.mean(), nested.std()))

best inner score (optimistic) 0.8216
nested score     (honest)     0.8176  +/- 0.0122

The nested figure re-runs the entire search inside each outer fold, so the settings are never chosen using the rows they are scored on. It costs inner times outer fits, which is why people skip it, but when you need to report a number you will be held to, this is the number.

Day 1 takeaway

One split's score varies by a couple of points on the seed alone. Stratify for classification, group whenever rows cluster by an entity, and split by time when order matters. A hyperparameter score from the same folds you tuned on is optimistic; nested cross-validation is the honest version.

Week 07 · Day 2 of 7

The Leaks Cross-Validation Cannot See

Scaling, selection and duplicates outside the pipeline, and noise scoring 0.7

By Pawan Rama Mali865 words

Cross-validation protects you from one thing only: scoring on rows you trained on. It cannot see a leak that happened before the folds were made.

Scaling before the split

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np

X, y = df[NUM].to_numpy(), df[TARGET].to_numpy()
X = SimpleImputer(strategy='median').fit_transform(X)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

# Wrong: the scaler sees every row, including the ones it will be tested on.
X_leaked = StandardScaler().fit_transform(X)
leaked = cross_val_score(LogisticRegression(max_iter=1000), X_leaked, y,
                         cv=cv, scoring='roc_auc')

# Right: the scaler is refitted inside every fold.
clean = cross_val_score(make_pipeline(StandardScaler(),
                                      LogisticRegression(max_iter=1000)),
                        X, y, cv=cv, scoring='roc_auc')

print('scaled before splitting %.4f' % leaked.mean())
print('scaled inside the fold  %.4f' % clean.mean())
print('difference              %+.4f' % (leaked.mean() - clean.mean()))

scaled before splitting 0.7721
scaled inside the fold  0.7721
difference              +0.0000

Zero difference is what makes this dangerous

The two numbers are identical to four decimal places. With three thousand rows, a mean and a standard deviation computed on 80 percent of them are indistinguishable from the same figures computed on all of them, so this particular leak costs nothing measurable.

That is precisely the problem. The same mistake with target encoding, an aggregate feature, or feature selection produces gaps of ten points or more, and the next snippet turns pure noise into an apparent 0.80. Nothing in the score tells you which case you are in, so the rule cannot be “check whether it matters”. It has to be absolute: every step that learns from data goes inside the pipeline.

Feature selection before the split, which is catastrophic

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold

rng = np.random.default_rng(0)
n, p = 200, 5000
X = rng.normal(size=(n, p))       # pure noise
y = rng.integers(0, 2, n)         # unrelated labels
cv = StratifiedKFold(5, shuffle=True, random_state=0)

# Wrong: pick the 20 best columns using ALL the labels, then cross-validate.
best = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaked = cross_val_score(LogisticRegression(max_iter=1000), best, y, cv=cv)

# Right: selection happens inside each fold.
clean = cross_val_score(make_pipeline(SelectKBest(f_classif, k=20),
                                      LogisticRegression(max_iter=1000)),
                        X, y, cv=cv)

print('there is no signal in this data at all -- 0.5 is the truth')
print('  selection outside the folds %.4f' % leaked.mean())
print('  selection inside the folds  %.4f' % clean.mean())

there is no signal in this data at all -- 0.5 is the truth
  selection outside the folds 0.7950
  selection inside the folds  0.5450

Random noise, random labels, no relationship whatsoever, and selecting features before cross-validating reports accuracy far above chance. Out of five thousand noise columns, twenty will correlate with the labels by luck, and choosing them using every label is how the luck gets in. This is a documented cause of retracted findings in published research.

Duplicate rows

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
import pandas as pd

raw = pd.read_csv('customers.csv')            # 138 duplicated rows
deduped = raw.drop_duplicates()
cv = StratifiedKFold(5, shuffle=True, random_state=0)
pipe = make_pipeline(SimpleImputer(strategy='median'),
                     RandomForestClassifier(n_estimators=200, random_state=42))

for name, d in [('with duplicates   ', raw), ('deduplicated      ', deduped)]:
    s = cross_val_score(pipe, d[NUM], d[TARGET], cv=cv, scoring='roc_auc')
    print('%s AUC %.4f  (%d rows)' % (name, s.mean(), len(d)))

with duplicates    AUC 0.7503  (3138 rows)
deduplicated       AUC 0.7212  (3000 rows)

Only 138 duplicates in three thousand rows, so the inflation is modest. Scale that to an export that ran twice and the effect is severe. De-duplicate first, always.

A checklist you can actually run

Is every fitted transformation inside the Pipeline?
Were duplicates removed before splitting?
Do any rows share an entity that could straddle the split?
Are the rows time-ordered, and does the split respect that?
Was any feature computed using an aggregate over the whole dataset?
For every feature: would I have this value at prediction time?
Is the score suspiciously good?

Suspicion is a diagnostic

The last item is not a joke. Experienced practitioners treat an unexpectedly excellent score as a bug report. If a model jumps from 0.82 to 0.97 because you added one feature, the overwhelmingly likely explanation is that the feature leaks.

Day 2 takeaway

Cross-validation only protects against training on the rows you score. Anything fitted before the split, scalers, imputers, encoders, and above all feature selection, leaks. Selecting features on all the labels can make pure noise look strongly predictive. Put every fitted step in the pipeline, and treat a surprisingly good score as a defect until proven otherwise.

Week 07 · Day 3 of 7

Imbalanced Classes

Why accuracy rises as the problem gets harder, and what actually helps

By Pawan Rama Mali942 words

Churn is 27 percent, which is mild. Fraud is 0.1 percent, and at that level everything you have learned about accuracy stops working.

The problem, at increasing severity

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, roc_auc_score,
                             average_precision_score, recall_score)

print('%10s %10s %10s %10s %10s'
      % ('positives', 'accuracy', 'recall', 'ROC AUC', 'avg prec'))
for weight in [0.5, 0.1, 0.02, 0.005]:
    X, y = make_classification(n_samples=20000, n_features=12, n_informative=5,
                               weights=[1 - weight], flip_y=0.01, random_state=0)
    a, b, c, d = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
    m = LogisticRegression(max_iter=2000).fit(a, c)
    pred, proba = m.predict(b), m.predict_proba(b)[:, 1]
    print('%10.3f %10.4f %10.4f %10.4f %10.4f'
          % (d.mean(), accuracy_score(d, pred), recall_score(d, pred),
             roc_auc_score(d, proba), average_precision_score(d, proba)))

 positives   accuracy     recall    ROC AUC   avg prec
500     0.9365     0.9494     0.9782     0.9724
104     0.9623     0.7388     0.9520     0.8674
025     0.9835     0.3791     0.8491     0.6144
011     0.9915     0.2031     0.6895     0.3925

Accuracy climbs toward 1.0 as the problem gets harder, because guessing the majority class gets easier. Recall collapses. Accuracy is not a weak metric here. It is an actively misleading one.

Class weights, which cost nothing

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (recall_score, precision_score,
                             average_precision_score, roc_auc_score)

for weight in [None, 'balanced']:
    m = Pipeline([('prep', preprocessor()),
                  ('clf', LogisticRegression(max_iter=1000, class_weight=weight,
                                             random_state=42))]).fit(X_tr, y_tr)
    pred, proba = m.predict(X_te), m.predict_proba(X_te)[:, 1]
    print('class_weight=%-9s recall %.4f  precision %.4f  AUC %.4f  AP %.4f'
          % (str(weight), recall_score(y_te, pred),
             precision_score(y_te, pred), roc_auc_score(y_te, proba),
             average_precision_score(y_te, proba)))

class_weight=None      recall 0.4328  precision 0.6304  AUC 0.8174  AP 0.5993
class_weight=balanced  recall 0.7960  precision 0.4923  AUC 0.8172  AP 0.5973

class_weight='balanced' is mostly a threshold move

Recall jumps and precision falls, but ROC AUC and average precision barely change, because the ranking of customers by risk is nearly the same. Weighting mostly shifts where the 0.5 cut lands. That is worth knowing: if you were going to tune the threshold anyway, as week 5 did, class weights add little. They matter more when the algorithm's loss is genuinely dominated by the majority class.

Resampling

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT)])

samplers = {'none         ': None,
            'oversample   ': RandomOverSampler(random_state=0),
            'undersample  ': RandomUnderSampler(random_state=0),
            'SMOTE        ': SMOTE(random_state=0)}

for name, sampler in samplers.items():
    steps = [('prep', prep)]
    if sampler is not None:
        steps.append(('sample', sampler))
    steps.append(('clf', LogisticRegression(max_iter=1000, random_state=42)))
    m = ImbPipeline(steps).fit(X_tr, y_tr)
    proba = m.predict_proba(X_te)[:, 1]
    print('%s AUC %.4f   avg precision %.4f'
          % (name, roc_auc_score(y_te, proba),
             average_precision_score(y_te, proba)))

none          AUC 0.8174   avg precision 0.5993
oversample    AUC 0.8143   avg precision 0.5923
undersample   AUC 0.8110   avg precision 0.5856
SMOTE         AUC 0.8160   avg precision 0.5951

SMOTE: Synthetic Minority Over-sampling. Rather than copying minority rows, it creates new ones along the lines between a minority row and its nearest minority neighbours. That gives the model variation instead of exact duplicates.

Resample inside the pipeline, and never the test set

imblearn's Pipeline exists precisely because scikit-learn's will not resample. It applies the sampler during fit and skips it during predict, which is the only correct behaviour. Oversampling before the split copies minority rows into both halves. The model is then tested on rows it trained on, and the score is fiction.

Notice how little any of it achieved on this problem. Resampling is oversold; it changes the ranking barely at all, and at 27 percent minority there is not much to fix. Reach for it when the minority class is under a few percent, and measure rather than assume.

What to actually do about imbalance

Use the right metric: average precision, not accuracy. This alone fixes most of the confusion.
Tune the threshold against the cost of each error, as in week 5. Free, and usually the largest gain.
Try class weights, which cost one argument.
Then consider resampling, measuring whether it helped.
Collect more minority examples if you possibly can. Nothing else comes close.

Day 3 takeaway

Accuracy rises as imbalance worsens, which makes it useless. Report average precision. Class weights and threshold tuning mostly do the same job, and threshold tuning is more direct. Resampling must happen inside the pipeline and on training folds only, and it helps less often than its reputation suggests.

Week 07 · Day 4 of 7

Calibration

Ranking well and still being wrong about the numbers

By Pawan Rama Mali952 words

A model that ranks customers perfectly can still be badly wrong about the numbers. If anyone will multiply your probability by a pound value, ranking is not enough.

Calibration: A model is calibrated if, among all the cases it gives a 30 percent chance, close to 30 percent actually happen. It is independent of discrimination: a model can rank well and be calibrated badly, or the reverse.

Measure it

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, roc_auc_score
import numpy as np

models = {
    'logistic': LogisticRegression(max_iter=1000, random_state=42),
    'forest  ': RandomForestClassifier(n_estimators=300, random_state=42),
    'naive-B ': GaussianNB(),
}
for name, clf in models.items():
    m = Pipeline([('prep', preprocessor()), ('clf', clf)]).fit(X_tr, y_tr)
    proba = m.predict_proba(X_te)[:, 1]
    print('%s  AUC %.4f   Brier %.4f'
          % (name, roc_auc_score(y_te, proba),
             brier_score_loss(y_te, proba)))

logistic  AUC 0.8174   Brier 0.1475
forest    AUC 0.7772   Brier 0.1674
naive-B   AUC 0.8044   Brier 0.2438

The Brier score is the mean squared error of the probabilities. Lower is better, and unlike AUC it punishes a model for being confidently wrong about the level rather than just the order.

The reliability table

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.calibration import calibration_curve
import numpy as np

for name, clf in [('logistic', LogisticRegression(max_iter=1000, random_state=42)),
                  ('naive-B ', GaussianNB())]:
    m = Pipeline([('prep', preprocessor()), ('clf', clf)]).fit(X_tr, y_tr)
    proba = m.predict_proba(X_te)[:, 1]
    true_rate, predicted = calibration_curve(y_te, proba, n_bins=6,
                                             strategy='quantile')
    print('%s   predicted -> actual' % name)
    for pr, ac in zip(predicted, true_rate):
        print('           %.3f    -> %.3f' % (pr, ac))
    print()

logistic   predicted -> actual
           0.010    -> 0.008
           0.058    -> 0.072
           0.162    -> 0.152
           0.285    -> 0.304
           0.433    -> 0.456
           0.628    -> 0.616

naive-B    predicted -> actual
           0.000    -> 0.024
           0.003    -> 0.064
           0.123    -> 0.160
           0.613    -> 0.312
           0.955    -> 0.448
           0.989    -> 0.600

Logistic regression tracks the diagonal closely, which is expected: it is fitted by minimising log loss, and log loss is minimised by telling the truth about probabilities. Naive Bayes, which multiplies correlated probabilities as though independent, is pushed toward the extremes.

Repairing it

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.naive_bayes import GaussianNB
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss, roc_auc_score

base = Pipeline([('prep', preprocessor()), ('clf', GaussianNB())])

fitted = base.fit(X_tr, y_tr)
raw = fitted.predict_proba(X_te)[:, 1]
print('uncalibrated   Brier %.4f   AUC %.4f'
      % (brier_score_loss(y_te, raw), roc_auc_score(y_te, raw)))

for method in ['sigmoid', 'isotonic']:
    cal = CalibratedClassifierCV(base, method=method, cv=5).fit(X_tr, y_tr)
    proba = cal.predict_proba(X_te)[:, 1]
    print('%-14s Brier %.4f   AUC %.4f'
          % (method, brier_score_loss(y_te, proba), roc_auc_score(y_te, proba)))

uncalibrated   Brier 0.2438   AUC 0.8044
sigmoid        Brier 0.1559   AUC 0.8060
isotonic       Brier 0.1525   AUC 0.8043

Method	Fits	Use when
`sigmoid` (Platt)	A logistic curve, two parameters	Small data; the distortion is a smooth S-shape
`isotonic`	Any non-decreasing step function	Plenty of data; more flexible, can overfit

Calibration barely changes AUC

Both methods apply a monotone transformation, and a monotone transformation cannot change the ordering. So AUC stays put while the Brier score improves substantially. That is the clearest possible demonstration that ranking and calibration are separate properties.

When it matters

Expected value calculations: probability times customer value only means something if the probability is real.
Combining models: averaging miscalibrated probabilities is meaningless.
Anything shown to a human: a clinician told 80 percent will act on 80 percent.
Thresholds set from cost: week 5's calculation assumed the probabilities meant something.

If you only ever sort customers and contact the top thousand, calibration is irrelevant. Know which situation you are in.

Day 4 takeaway

Ranking and calibration are different. Check with a reliability table and the Brier score. Logistic regression is calibrated more or less for free; naive Bayes and boosted trees are not. CalibratedClassifierCV repairs it without changing the ranking, and you need it whenever a number, not an order, leaves the model.

Week 07 · Day 5 of 7

Learning and Validation Curves

Diagnosing whether you need more data, more features or less model

By Pawan Rama Mali1016 words

Your model underperforms. Do you collect more data, add features, or pick a different algorithm? Guessing wastes weeks. Two curves answer it.

Learning curves: is more data the answer?

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import learning_curve, StratifiedKFold
import numpy as np

X, y = df[NUM + CAT], df[TARGET]
cv = StratifiedKFold(5, shuffle=True, random_state=0)

for name, clf in [('logistic (well matched)',
                   LogisticRegression(max_iter=1000)),
                  ('deep tree (high variance)',
                   DecisionTreeClassifier(random_state=0))]:
    sizes, tr, va = learning_curve(
        Pipeline([('prep', preprocessor()), ('clf', clf)]), X, y, cv=cv,
        scoring='roc_auc', train_sizes=np.linspace(0.1, 1.0, 5))
    print(name)
    print('%10s %10s %10s %8s' % ('rows', 'train', 'val', 'gap'))
    for n, a, b in zip(sizes, tr.mean(axis=1), va.mean(axis=1)):
        print('%10d %10.4f %10.4f %8.4f' % (n, a, b, a - b))
    print()

logistic (well matched)
      rows      train        val      gap
       240     0.8044     0.8056  -0.0012
       780     0.8141     0.8180  -0.0039
      1320     0.8245     0.8180   0.0065
      1860     0.8236     0.8198   0.0038
      2400     0.8267     0.8220   0.0047

deep tree (high variance)
      rows      train        val      gap
       240     1.0000     0.5973   0.4027
       780     1.0000     0.6119   0.3881
      1320     1.0000     0.6207   0.3793
      1860     1.0000     0.6270   0.3730
      2400     1.0000     0.6344   0.3656

Pattern	Diagnosis	What to do
Both curves low, converged	High bias, underfitting	Better features, a more flexible model
Wide gap, validation still rising	High variance	More data, more regularisation, fewer features
Both high, converged	You are done	Ship it
Validation falling as data grows	Something is wrong	Check for leakage or a distribution shift

The logistic curves converge with almost no gap: it has extracted what this feature set contains, and more rows will not help. The deep tree keeps a large gap at every size, which is the signature of variance. There, more data genuinely would.

Validation curves: is this parameter set right?

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import validation_curve, StratifiedKFold
import numpy as np

X, y = df[NUM + CAT], df[TARGET]
values = [1, 2, 5, 10, 25, 50, 100, 200]
tr, va = validation_curve(
    Pipeline([('prep', preprocessor()),
              ('clf', RandomForestClassifier(n_estimators=150, random_state=0))]),
    X, y, param_name='clf__min_samples_leaf', param_range=values,
    cv=StratifiedKFold(4, shuffle=True, random_state=0), scoring='roc_auc')

print('%18s %10s %10s %8s' % ('min_samples_leaf', 'train', 'val', 'gap'))
for v, a, b in zip(values, tr.mean(axis=1), va.mean(axis=1)):
    print('%18d %10.4f %10.4f %8.4f' % (v, a, b, a - b))

  min_samples_leaf      train        val      gap
                 1     1.0000     0.7837   0.2163
                 2     0.9834     0.7984   0.1851
                 5     0.9198     0.8117   0.1081
                10     0.8810     0.8151   0.0659
                25     0.8492     0.8187   0.0305
                50     0.8357     0.8179   0.0178
               100     0.8262     0.8158   0.0104
               200     0.8214     0.8151   0.0063

Read it left to right: at leaf size 1 the training score is near perfect and the gap enormous. As the leaves grow the gap closes and validation improves, up to a point, then both fall as the model becomes too constrained. The best value is where validation peaks, and the shape tells you whether you are on the overfitting or underfitting side of it.

A peak at the edge of your range means the range is wrong

If the best value is the largest or smallest you tried, you have not found the optimum. You have found the edge of your search. Extend the range and look again. The same rule applied to ridge's alpha in week 4, and it applies to every hyperparameter search in week 14.

Putting both together

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate, StratifiedKFold

cv = StratifiedKFold(5, shuffle=True, random_state=0)
for name, clf in [('logistic', LogisticRegression(max_iter=1000, random_state=0)),
                  ('boosting', HistGradientBoostingClassifier(random_state=0))]:
    r = cross_validate(Pipeline([('prep', preprocessor()), ('clf', clf)]),
                       X_tr, y_tr, cv=cv, scoring='roc_auc',
                       return_train_score=True)
    print('%-9s train %.4f   val %.4f   gap %+.4f'
          % (name, r['train_score'].mean(), r['test_score'].mean(),
             r['train_score'].mean() - r['test_score'].mean()))

logistic  train 0.8289   val 0.8238   gap +0.0052
boosting  train 0.9845   val 0.7966   gap +0.1879

return_train_score=True is the cheapest diagnostic in scikit-learn and almost nobody uses it. One extra argument turns a score into a diagnosis: boosting's gap is many times logistic regression's, which says it is fitting noise this dataset does not reward.

Day 5 takeaway

A learning curve tells you whether more data will help; a validation curve tells you whether a parameter is set right and which side of the optimum you are on. Pass return_train_score=True so every cross-validation reports the gap, not just the score. A best value at the edge of your range means the range was too narrow.

Week 07 · Day 6 of 7

Error Analysis and Fairness

Who the model fails, and why the headline score hides it

By Pawan Rama Mali1245 words

An aggregate score tells you how the model does on average. It never tells you who it fails, and that is what determines whether you can deploy it.

Score by segment

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score
import pandas as pd

m = Pipeline([('prep', preprocessor()),
              ('clf', LogisticRegression(max_iter=1000,
                                         random_state=42))]).fit(X_tr, y_tr)
proba = m.predict_proba(X_te)[:, 1]

seg = df.loc[X_te.index].copy()
seg['proba'] = proba
seg['truth'] = y_te.to_numpy()

rows = []
for name, group in seg.groupby('contract'):
    if group['truth'].nunique() < 2:
        continue
    rows.append({'segment': name, 'n': len(group),
                 'churn_rate': group['truth'].mean(),
                 'auc': roc_auc_score(group['truth'], group['proba']),
                 'avg_prec': average_precision_score(group['truth'],
                                                     group['proba'])})
print(pd.DataFrame(rows).round(4).to_string(index=False))
print('\noverall AUC %.4f' % roc_auc_score(y_te, proba))

       segment   n  churn_rate    auc  avg_prec
Month-To-Month 416      0.4135 0.7072    0.6279
      One Year 184      0.1141 0.8113    0.4052
      Two Year 150      0.0533 0.8688    0.3955

overall AUC 0.8174

Segment AUCs can all be lower than the overall AUC

This looks impossible and is not. Much of the overall ranking skill comes from separating month-to-month customers from two-year ones. Within a contract type that advantage is gone, and only the weaker signals remain. If your product treats each segment separately, the within-segment number is the one that matters, and it is worse than the headline.

Look at what it got most wrong

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
import pandas as pd
import numpy as np

m = Pipeline([('prep', preprocessor()),
              ('clf', LogisticRegression(max_iter=1000,
                                         random_state=42))]).fit(X_tr, y_tr)
seg = df.loc[X_te.index].copy()
seg['proba'] = m.predict_proba(X_te)[:, 1]
seg['error'] = (seg[TARGET] - seg['proba']).abs()

cols = ['tenure_months', 'contract', 'support_calls', 'monthly_charges',
        TARGET, 'proba']
print('confidently wrong -- said stay, they left:')
worst = seg[seg[TARGET] == 1].nsmallest(4, 'proba')[cols]
print(worst.round(3).to_string(index=False))
print('\nconfidently wrong -- said leave, they stayed:')
worst = seg[seg[TARGET] == 0].nlargest(4, 'proba')[cols]
print(worst.round(3).to_string(index=False))

confidently wrong -- said stay, they left:
 tenure_months contract  support_calls  monthly_charges  churned  proba
            23 Two Year              0            26.42        1  0.009
            41 One Year              0            53.26        1  0.031
            12 Two Year              1              NaN        1  0.042
            19 One Year              0            23.08        1  0.047

confidently wrong -- said leave, they stayed:
 tenure_months       contract  support_calls  monthly_charges  churned  proba
             5 Month-To-Month              3            73.19        0  0.782
             6 Month-To-Month              1            79.59        0  0.751
            17 Month-To-Month              4            86.67        0  0.729
             9 Month-To-Month              1            79.64        0  0.719

Long-tenure two-year customers who left anyway, and short-tenure month-to-month customers who stayed. Both groups contradict the strongest pattern in the data. In a real project this is where you go and ask someone: is there a reason these people behaved differently, and is it something we could record?

Where the errors concentrate

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
import pandas as pd

m = Pipeline([('prep', preprocessor()),
              ('clf', LogisticRegression(max_iter=1000,
                                         random_state=42))]).fit(X_tr, y_tr)
seg = df.loc[X_te.index].copy()
seg['proba'] = m.predict_proba(X_te)[:, 1]
seg['error'] = (seg[TARGET] - seg['proba']).abs()
seg['tenure_band'] = pd.cut(seg['tenure_months'], [0, 6, 18, 36, 72],
                            labels=['0-6', '7-18', '19-36', '37-72'])

print(seg.groupby('tenure_band', observed=True)
      .agg(n=('error', 'size'), mean_error=('error', 'mean'),
           churn_rate=(TARGET, 'mean'), mean_proba=('proba', 'mean'))
      .round(4).to_string())

               n  mean_error  churn_rate  mean_proba
tenure_band                                         
0-6           72      0.4086      0.4167      0.4470
7-18         314      0.3982      0.3854      0.3716
19-36        218      0.2457      0.1927      0.1886
37-72        146      0.0791      0.0548      0.0490

Compare the last two columns: where the mean predicted probability sits below the actual churn rate, the model systematically under-predicts that band. That is a bias you can act on, and it is invisible in any single aggregate number.

Fairness is the same technique with higher stakes

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score, precision_score
import pandas as pd

m = Pipeline([('prep', preprocessor()),
              ('clf', LogisticRegression(max_iter=1000,
                                         random_state=42))]).fit(X_tr, y_tr)
seg = df.loc[X_te.index].copy()
seg['pred'] = m.predict(X_te)
seg['truth'] = y_te.to_numpy()

rows = []
for name, g in seg.groupby('has_dependents', dropna=False):
    rows.append({'group': str(name), 'n': len(g),
                 'selection_rate': g['pred'].mean(),
                 'recall': recall_score(g['truth'], g['pred'], zero_division=0),
                 'precision': precision_score(g['truth'], g['pred'],
                                              zero_division=0)})
print(pd.DataFrame(rows).round(4).to_string(index=False))

group   n  selection_rate  recall  precision
   No 498          0.1988  0.4532     0.6364
  Yes 205          0.1561  0.4286     0.6562
  nan  47          0.1489  0.2308     0.4286

Equal treatment has several incompatible definitions

Equal selection rate, equal recall, and equal precision are three different fairness criteria, and it is mathematically impossible to satisfy all of them at once when the groups have different base rates. You have to choose which one your situation demands and say so out loud. For a retention offer, unequal selection rates may be fine. For a loan decision they are not.

Day 6 takeaway

Break the score down by segment, and expect within-segment performance to be worse than the headline. Read the rows the model got most confidently wrong. They point at missing features. Compare predicted rates against actual rates by band to find systematic bias. The same technique, applied to groups of people, is fairness auditing.

Week 07 · Day 7 of 7

A Reusable Evaluation Report

One function covering stability, discrimination, calibration and segments

By Pawan Rama Mali965 words

Everything this week, as one function you can run on any classifier.

The report

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.metrics import (roc_auc_score, average_precision_score,
                             brier_score_loss, confusion_matrix,
                             precision_score, recall_score)
from sklearn.model_selection import cross_validate, StratifiedKFold
import numpy as np
import pandas as pd

def evaluate(model, X_tr, y_tr, X_te, y_te, frame, costs=None):
    """A full evaluation: stability, discrimination, calibration,
    the operating point, and where it fails."""
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    r = cross_validate(model, X_tr, y_tr, cv=cv, scoring='roc_auc',
                       return_train_score=True)
    print('CROSS-VALIDATION')
    print('  train AUC %.4f   val AUC %.4f +/- %.4f   gap %+.4f'
          % (r['train_score'].mean(), r['test_score'].mean(),
             r['test_score'].std(),
             r['train_score'].mean() - r['test_score'].mean()))

    model.fit(X_tr, y_tr)
    proba = model.predict_proba(X_te)[:, 1]
    print('\nHELD-OUT TEST')
    print('  ROC AUC           %.4f' % roc_auc_score(y_te, proba))
    print('  average precision %.4f  (baseline %.4f)'
          % (average_precision_score(y_te, proba), y_te.mean()))
    print('  Brier score       %.4f' % brier_score_loss(y_te, proba))

    if costs:
        offer, value, save = costs
        grid = np.arange(0.05, 0.95, 0.01)
        net = []
        for t in grid:
            tn, fp, fn, tp = confusion_matrix(y_te, (proba >= t).astype(int)).ravel()
            net.append(tp * save * value - (tp + fp) * offer)
        best_t = grid[int(np.argmax(net))]
        pred = (proba >= best_t).astype(int)
        print('\nOPERATING POINT')
        print('  threshold %.2f (net value %.0f, against %.0f at 0.5)'
              % (best_t, max(net), net[int(np.argmin(np.abs(grid - 0.5)))]))
        print('  precision %.4f   recall %.4f'
              % (precision_score(y_te, pred, zero_division=0),
                 recall_score(y_te, pred)))
    else:
        pred = (proba >= 0.5).astype(int)

    print('\nBY SEGMENT')
    seg = frame.loc[X_te.index].copy()
    seg['proba'], seg['truth'] = proba, y_te.to_numpy()
    for name, g in seg.groupby('contract'):
        if g['truth'].nunique() < 2:
            continue
        print('  %-16s n=%4d  churn %.3f  AUC %.4f'
              % (name, len(g), g['truth'].mean(),
                 roc_auc_score(g['truth'], g['proba'])))
    return proba

from sklearn.linear_model import LogisticRegression
model = Pipeline([('prep', preprocessor()),
                  ('clf', LogisticRegression(max_iter=1000, random_state=42))])
_ = evaluate(model, X_tr, y_tr, X_te, y_te, df, costs=(30, 250, 0.35))

CROSS-VALIDATION
  train AUC 0.8289   val AUC 0.8238 +/- 0.0305   gap +0.0052

HELD-OUT TEST
  ROC AUC           0.8174
  average precision 0.5993  (baseline 0.2680)
  Brier score       0.1475

OPERATING POINT
  threshold 0.34 (net value 4537, against 3472 at 0.5)
  precision 0.5423   recall 0.7015

BY SEGMENT
  Month-To-Month   n= 416  churn 0.413  AUC 0.7072
  One Year         n= 184  churn 0.114  AUC 0.8113
  Two Year         n= 150  churn 0.053  AUC 0.8688

Run it on a competitor

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), NUM),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.metrics import (roc_auc_score, average_precision_score,
                             brier_score_loss)

cv = StratifiedKFold(5, shuffle=True, random_state=0)
for name, clf in [('logistic', __import__('sklearn.linear_model',
                    fromlist=['LogisticRegression']).LogisticRegression(
                        max_iter=1000, random_state=42)),
                  ('boosting', HistGradientBoostingClassifier(random_state=42))]:
    m = Pipeline([('prep', preprocessor()), ('clf', clf)])
    r = cross_validate(m, X_tr, y_tr, cv=cv, scoring='roc_auc',
                       return_train_score=True)
    m.fit(X_tr, y_tr)
    proba = m.predict_proba(X_te)[:, 1]
    print('%-9s val %.4f +/-%.4f  gap %+.4f  test AUC %.4f  AP %.4f  Brier %.4f'
          % (name, r['test_score'].mean(), r['test_score'].std(),
             r['train_score'].mean() - r['test_score'].mean(),
             roc_auc_score(y_te, proba), average_precision_score(y_te, proba),
             brier_score_loss(y_te, proba)))

logistic  val 0.8238 +/-0.0305  gap +0.0052  test AUC 0.8174  AP 0.5993  Brier 0.1475
boosting  val 0.7966 +/-0.0167  gap +0.1879  test AUC 0.7797  AP 0.5262  Brier 0.1690

Three numbers decide it: the validation score, the train-validation gap, and the Brier score. Boosting fits harder and generalises no better here, which is the same conclusion week 6 reached from a different direction.

What to write down

The metric you optimised and why that one.
The cross-validated score with its spread, never a bare number.
The train-validation gap.
The held-out test score, computed once.
The operating threshold and the cost assumptions behind it.
Performance broken down by every segment that matters.
The known failure modes, from the confidently-wrong rows.
The date, the data version, and the random seeds.

The last one is not bureaucracy

In six months someone will ask why the model behaves differently. Without the data version and the seeds you cannot reproduce what you did, so you cannot answer. Week 15 turns this into an artefact the pipeline writes automatically.

Your assignment

Run evaluate on the random forest and on naive Bayes. One will have a much larger train-validation gap; the other will have a much worse Brier score. Write one paragraph recommending a model for a retention programme that multiplies each probability by customer value, and note which of the two failings actually disqualifies a model for that use.

Day 7 takeaway

A complete evaluation reports cross-validated performance with its spread, the train-validation gap, held-out discrimination and calibration, the chosen operating point with its cost assumptions, and a per-segment breakdown. Make it a function, run it on every candidate, and record the versions and seeds.