Classification Fundamentals

Week 5 of 16 · Supervised learning · 7 days

Full curriculum
Week 05 · Supervised learning

Classification Fundamentals

Week 05 · Day 1 of 7

Logistic Regression

The sigmoid, log-odds, odds ratios, and regularisation you did not ask for

By 1119 words

Linear regression predicts a number on an unbounded scale. A probability lives between 0 and 1. Logistic regression is what you get when you insist on that constraint, and it remains the model to beat on tabular data.

The sigmoid

import numpy as np

def sigmoid(z):
return 1 / (1 + np.exp(-z))

for z in [-6, -2, -1, 0, 1, 2, 6]:
print('z = %3d -> probability %.4f' % (z, sigmoid(z)))
z = -6 -> probability 0.0025
z = -2 -> probability 0.1192
z = -1 -> probability 0.2689
z = 0 -> probability 0.5000
z = 1 -> probability 0.7311
z = 2 -> probability 0.8808
z = 6 -> probability 0.9975

Any real number in, a probability out. Zero maps to 0.5, and the curve saturates at both ends, which is why a very confident model needs an enormous change in z to become slightly more confident.

Logistic regression: Computes z = X @ w + b exactly as linear regression does, then passes z through the sigmoid to get a probability. It is fitted by minimising log loss, not squared error. Despite the name it is a classifier.

Log-odds, which is the scale the coefficients live on

import numpy as np

def sigmoid(z):
return 1 / (1 + np.exp(-z))

print('%10s %10s %12s' % ('probability', 'odds', 'log-odds (z)'))
for prob in [0.1, 0.25, 0.5, 0.75, 0.9]:
odds = prob / (1 - prob)
print('%10.2f %10.3f %12.4f' % (prob, odds, np.log(odds)))
probability odds log-odds (z)
0.10 0.111 -2.1972
0.25 0.333 -1.0986
0.50 1.000 0.0000
0.75 3.000 1.0986
0.90 9.000 2.1972

A coefficient is a change in log-odds

That is why they are hard to read directly. Exponentiate one and you get an odds ratio: a coefficient of 0.7 means exp(0.7) = 2.01, so a one-unit increase roughly doubles the odds. Doubling the odds is not doubling the probability, from 0.1 it goes to 0.18, from 0.5 it goes to 0.67.

Fit it on the churn problem

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, log_loss, accuracy_score

model = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]).fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]

print('accuracy %.4f' % accuracy_score(y_te, model.predict(X_te)))
print('ROC AUC %.4f' % roc_auc_score(y_te, proba))
print('log loss %.4f' % log_loss(y_te, proba))
print('\npredicted probabilities: min %.3f, median %.3f, max %.3f'
% (proba.min(), np.median(proba), proba.max()))
accuracy 0.7800
ROC AUC 0.8174
log loss 0.4451

predicted probabilities: min 0.000, median 0.232, max 0.829

Read the coefficients as odds ratios

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
import numpy as np
import pandas as pd

model = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]).fit(X_tr, y_tr)
names = model.named_steps['prep'].get_feature_names_out()
coef = model.named_steps['clf'].coef_[0]

table = pd.DataFrame({'coefficient': coef, 'odds_ratio': np.exp(coef)},
index=names).sort_values('coefficient')
print(table.round(3).to_string())
coefficient odds_ratio
cat__contract_Two Year -1.454 0.234
num__tenure_months -0.978 0.376
cat__payment_method_Credit card -0.205 0.815
cat__payment_method_Bank transfer -0.174 0.841
cat__internet_service_No internet -0.163 0.850
cat__has_dependents_Yes -0.102 0.903
cat__payment_method_Mailed check -0.088 0.916
cat__internet_service_DSL -0.060 0.942
cat__has_dependents_No 0.087 1.091
cat__contract_One Year 0.128 1.137
cat__internet_service_Fibre optic 0.207 1.230
num__support_calls 0.279 1.322
num__monthly_charges 0.436 1.547
cat__payment_method_Electronic check 0.451 1.570
cat__contract_Month-To-Month 1.310 3.706

An odds ratio above 1 pushes toward churn, below 1 pushes away. Month-to-month roughly triples the odds against the average; a two-year contract cuts them to about a quarter. Compare that with the generator in week 1: it used +1.55 and −0.85 on exactly those two.

Odds ratios on scaled features are per standard deviation

The numeric columns went through StandardScaler, so their coefficients describe a one standard deviation change, not one month or one pound. To talk to a business audience in real units, divide the coefficient by the scaler's scale_ for that column, or fit an unscaled model purely for explanation.

Regularisation is on by default

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
import numpy as np

print('%10s %12s %14s' % ('C', 'test AUC', 'sum |coef|'))
for C_val in [0.001, 0.01, 0.1, 1.0, 100.0]:
m = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(C=C_val, max_iter=2000,
random_state=42))]).fit(X_tr, y_tr)
auc = roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])
print('%10s %12.4f %14.3f'
% (C_val, auc, np.abs(m.named_steps['clf'].coef_).sum()))
C test AUC sum |coef|
0.001 0.8205 0.885
0.01 0.8230 3.114
0.1 0.8198 5.147
1.0 0.8174 6.122
100.0 0.8169 8.108

C is the inverse of alpha

Ridge takes alpha, where larger means more penalty. LogisticRegression takes C, where smaller means more penalty. The default of C=1.0 means you are already regularising whether you meant to or not, which surprises people comparing against an unpenalised implementation elsewhere.

Day 1 takeaway

Logistic regression is linear regression pushed through a sigmoid and fitted on log loss. Coefficients are changes in log-odds; exponentiate them for odds ratios, and remember they are per standard deviation when the features were scaled. Regularisation is on by default, controlled by C, which runs the opposite way to alpha.
Week 05 · Day 2 of 7

The Decision Threshold

Why 0.5 is almost always wrong, and how to choose the number that is right

By 1312 words

Week 1 ended with a model that found fewer than half the churners, and the promise that the model was fine and the threshold was wrong. Today you collect on that.

predict() is predict_proba() plus an arbitrary cut

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
import numpy as np

model = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]).fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]

print('predict() agrees with proba > 0.5 on %d of %d rows'
% ((model.predict(X_te) == (proba > 0.5).astype(int)).sum(), len(proba)))
print('\nrows with probability above:')
for t in [0.2, 0.3, 0.4, 0.5, 0.6]:
print(' %.1f %4d (%.1f%% of the test set)'
% (t, (proba > t).sum(), 100 * (proba > t).mean()))
predict() agrees with proba > 0.5 on 750 of 750 rows

rows with probability above:
0.2 400 (53.3% of the test set)
0.3 294 (39.2% of the test set)
0.4 209 (27.9% of the test set)
0.5 138 (18.4% of the test set)
0.6 70 (9.3% of the test set)

0.5 is a default, not a decision

It is only optimal when the classes are balanced and the two kinds of mistake cost the same. Churn is 27 percent, and missing a leaver who was worth keeping costs far more than a wasted retention call. Neither condition holds, so 0.5 is simply the wrong number.

The four outcomes

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, precision_score, recall_score
import numpy as np

model = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]).fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]

print('%10s %6s %6s %6s %6s %10s %8s'
% ('threshold', 'TN', 'FP', 'FN', 'TP', 'precision', 'recall'))
for t in [0.15, 0.2, 0.27, 0.35, 0.5, 0.7]:
pred = (proba >= t).astype(int)
tn, fp, fn, tp = confusion_matrix(y_te, pred).ravel()
print('%10.2f %6d %6d %6d %6d %10.3f %8.3f'
% (t, tn, fp, fn, tp,
precision_score(y_te, pred, zero_division=0),
recall_score(y_te, pred)))
threshold TN FP FN TP precision recall
0.15 287 262 18 183 0.411 0.910
0.20 328 221 22 179 0.448 0.891
0.27 384 165 42 159 0.491 0.791
0.35 433 116 66 135 0.538 0.672
0.50 498 51 114 87 0.630 0.433
0.70 544 5 180 21 0.808 0.104
TermMeansCost in a churn programme
True positivePredicted churn, did churnA save, the point of the exercise
False positivePredicted churn, stayedA wasted retention offer
False negativePredicted stay, churnedA customer lost silently
True negativePredicted stay, stayedNothing spent
Precision and recall: Precision is the share of your positive predictions that were right, how much of your retention budget was well spent. Recall is the share of actual churners you caught, how much of the problem you addressed. Lowering the threshold always raises recall and lowers precision.

Choose the threshold from the cost of being wrong

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
import numpy as np

model = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]).fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]

OFFER_COST = 30 # what a retention offer costs us
CUSTOMER_VALUE = 250 # what keeping a customer is worth
SAVE_RATE = 0.35 # share of contacted churners we actually keep

best = None
for t in np.arange(0.05, 0.95, 0.01):
pred = (proba >= t).astype(int)
tn, fp, fn, tp = confusion_matrix(y_te, pred).ravel()
value = tp * SAVE_RATE * CUSTOMER_VALUE - (tp + fp) * OFFER_COST
if best is None or value > best[1]:
best = (t, value, tp, fp)

print('best threshold %.2f -> net value %.0f (contacted %d, saved about %.0f)'
% (best[0], best[1], best[2] + best[3], best[2] * SAVE_RATE))

pred_default = (proba >= 0.5).astype(int)
tn, fp, fn, tp = confusion_matrix(y_te, pred_default).ravel()
print('at the default 0.5 -> net value %.0f'
% (tp * SAVE_RATE * CUSTOMER_VALUE - (tp + fp) * OFFER_COST))
best threshold 0.34 -> net value 4537 (contacted 260, saved about 49)
at the default 0.5 -> net value 3472

Same model, same data, no retraining. Moving the threshold is free, and it is the single highest-return change available on most classification projects.

ROC and precision-recall curves

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (roc_curve, precision_recall_curve,
roc_auc_score, average_precision_score)
import numpy as np

model = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]).fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]

fpr, tpr, _ = roc_curve(y_te, proba)
prec, rec, _ = precision_recall_curve(y_te, proba)
print('ROC AUC %.4f' % roc_auc_score(y_te, proba))
print('average precision %.4f' % average_precision_score(y_te, proba))
print('baseline for AP %.4f <- the positive class rate' % y_te.mean())
print('baseline for ROC AUC 0.5000')
ROC AUC 0.8174
average precision 0.5993
baseline for AP 0.2680 <- the positive class rate
baseline for ROC AUC 0.5000

ROC AUC flatters models on imbalanced data

The false positive rate has the large negative class in its denominator, so a flood of false positives barely moves it. A model can look strong on ROC and be useless in practice. The precision-recall curve uses no true negatives at all, so it stays honest, and its baseline is the positive rate, 0.27 here, not 0.5. Report both, and lead with average precision when positives are rare.

Day 2 takeaway

predict() is predict_proba() thresholded at 0.5, and 0.5 is only right when classes are balanced and errors cost the same. Pick the threshold from the actual cost of each mistake. Report average precision alongside ROC AUC, because ROC hides false positives when the negative class is large.
Week 05 · Day 3 of 7

k-Nearest Neighbours

Voting among similar rows, and why distance stops working in high dimensions

By 976 words

The simplest possible classifier: find the most similar customers you have seen and copy their answer. It has no training phase at all.

How it works

k-Nearest Neighbours: To classify a new row, find the k closest rows in the training set and take a vote. There is no model, fit just stores the data. All the work happens at prediction time, which is the reverse of every other algorithm here.
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import roc_auc_score, accuracy_score

for k in [1, 5, 25, 101]:
m = Pipeline([('prep', preprocessor()),
('clf', KNeighborsClassifier(n_neighbors=k))]).fit(X_tr, y_tr)
print('k=%3d train acc %.4f test acc %.4f test AUC %.4f'
% (k, accuracy_score(y_tr, m.predict(X_tr)),
accuracy_score(y_te, m.predict(X_te)),
roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])))
k= 1 train acc 0.9991 test acc 0.7333 test AUC 0.6507
k= 5 train acc 0.8258 test acc 0.7453 test AUC 0.7517
k= 25 train acc 0.7924 test acc 0.7640 test AUC 0.8023
k=101 train acc 0.7831 test acc 0.7653 test AUC 0.8122

k=1 scores perfectly on training data, by definition

The nearest neighbour of a training row is itself, at distance zero. That perfect training accuracy is the purest example of why training scores are meaningless. Larger k averages over more neighbours, smoothing the decision boundary and trading variance for bias.

Scaling is not optional here

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score

# Two columns on wildly different scales: months against pounds.
WIDE = ['tenure_months', 'total_charges']
Xn_tr, Xn_te = df.loc[X_tr.index, WIDE], df.loc[X_te.index, WIDE]

unscaled = make_pipeline(SimpleImputer(strategy='median'),
KNeighborsClassifier(25)).fit(Xn_tr, y_tr)
scaled = make_pipeline(SimpleImputer(strategy='median'), StandardScaler(),
KNeighborsClassifier(25)).fit(Xn_tr, y_tr)

print('unscaled AUC %.4f' % roc_auc_score(y_te, unscaled.predict_proba(Xn_te)[:, 1]))
print('scaled AUC %.4f' % roc_auc_score(y_te, scaled.predict_proba(Xn_te)[:, 1]))
print('\ncolumn ranges:')
print((Xn_tr.max() - Xn_tr.min()).round(1).to_string())
unscaled AUC 0.6316
scaled AUC 0.7411

column ranges:
tenure_months 71.0
total_charges 7591.6

Eleven points of AUC, from one line. total_charges spans 7,592 while tenure spans 71, so unscaled the distance calculation is effectively only about total charges, tenure contributes about one percent of the typical gap between two rows and is ignored. Scaling gives each column an equal say. No other algorithm in this course is punished this hard for skipping it.

The curse of dimensionality

import numpy as np

rng = np.random.default_rng(0)
print('%6s %14s %14s %10s' % ('dims', 'nearest', 'furthest', 'ratio'))
for d in [2, 5, 20, 100, 500]:
pts = rng.uniform(size=(1000, d))
query = rng.uniform(size=d)
dist = np.linalg.norm(pts - query, axis=1)
print('%6d %14.4f %14.4f %10.3f'
% (d, dist.min(), dist.max(), dist.max() / dist.min()))
dims nearest furthest ratio
2 0.0098 1.3341 135.618
5 0.1443 1.6328 11.317
20 1.0474 2.2671 2.164
100 3.0126 4.8993 1.626
500 8.1975 9.7549 1.190

In high dimensions, everything is equally far away

At two dimensions the furthest point is several times further than the nearest. At 500, the ratio approaches 1: every point is roughly the same distance from every other. “Nearest neighbour” stops meaning anything, and kNN degrades toward guessing. This is why kNN is a poor choice after one-hot encoding produces a wide sparse matrix, and why week 10's dimensionality reduction exists.

Distance weighting

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import roc_auc_score

for weights in ['uniform', 'distance']:
m = Pipeline([('prep', preprocessor()),
('clf', KNeighborsClassifier(25, weights=weights))]).fit(X_tr, y_tr)
print('%-9s AUC %.4f'
% (weights, roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])))
uniform AUC 0.8023
distance AUC 0.8009

weights='distance' gives closer neighbours more say, which usually helps when k is large.

kNNLogistic regression
Training costNoneOne optimisation
Prediction costScans the training setOne dot product
MemoryStores all training dataStores k+1 numbers
BoundaryArbitrarily shapedA straight hyperplane
InterpretableNoYes, as odds ratios
Needs scalingAbsolutelyFor regularisation to be fair

Day 3 takeaway

kNN stores the training set and votes among the closest rows. It needs scaling more than any other algorithm, k trades variance for bias, and it degrades badly in high dimensions because distances converge. Cheap to train, expensive to predict, the opposite trade-off from everything else.
Week 05 · Day 4 of 7

Naive Bayes

A false assumption that ranks well and calibrates badly

By 899 words

Naive Bayes applies the theorem from week 3 directly, under an assumption that is almost always false. It works anyway, and understanding why is worth more than the algorithm itself.

The assumption

Naive Bayes: Applies Bayes' theorem assuming every feature is independent of every other, given the class. That lets it multiply individual probabilities instead of estimating a joint distribution, which is what makes it fast and what makes it naive.
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'

# Are contract and tenure independent given the class? Week 3 said no.
sub = df[df['churned'] == 1]
a = (sub['contract'] == 'Month-To-Month')
b = (sub['tenure_months'] > 24)
print('among churners:')
print(' P(month-to-month) %.4f' % a.mean())
print(' P(tenure > 24) %.4f' % b.mean())
print(' product (if independent) %.4f' % (a.mean() * b.mean()))
print(' actual joint probability %.4f' % (a & b).mean())
among churners:
P(month-to-month) 0.8545
P(tenure > 24) 0.1182
product (if independent) 0.1010
actual joint probability 0.0945

Not independent, not even close. Naive Bayes will assume they are. The surprise is that this often costs less than it should.

Fit it anyway

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, accuracy_score, log_loss

for name, clf in [('naive bayes', GaussianNB()),
('logistic ', LogisticRegression(max_iter=1000,
random_state=42))]:
m = Pipeline([('prep', preprocessor()), ('clf', clf)]).fit(X_tr, y_tr)
proba = m.predict_proba(X_te)[:, 1]
print('%s acc %.4f AUC %.4f log loss %.4f'
% (name, accuracy_score(y_te, m.predict(X_te)),
roc_auc_score(y_te, proba), log_loss(y_te, proba)))
naive bayes acc 0.7173 AUC 0.8044 log loss 1.1953
logistic acc 0.7800 AUC 0.8174 log loss 0.4451

Good ranking, bad probabilities

Look at the gap between AUC and log loss. Naive Bayes ranks customers by risk reasonably well, but its actual probability estimates are poor, because multiplying correlated probabilities as though they were independent pushes the result toward 0 or 1. If you only need an ordering it is fine. If you need a number a business can act on, “this customer has a 30 percent chance”. It is not.

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.naive_bayes import GaussianNB
import numpy as np

m = Pipeline([('prep', preprocessor()), ('clf', GaussianNB())]).fit(X_tr, y_tr)
proba = m.predict_proba(X_te)[:, 1]
print('share of predictions below 0.01: %.3f' % (proba < 0.01).mean())
print('share of predictions above 0.99: %.3f' % (proba > 0.99).mean())
print('share between 0.2 and 0.8: %.3f'
% ((proba > 0.2) & (proba < 0.8)).mean())
share of predictions below 0.01: 0.320
share of predictions above 0.99: 0.068
share between 0.2 and 0.8: 0.131

Most predictions are pinned at the extremes. A well-calibrated model on a 27 percent problem should produce a spread of intermediate probabilities. Week 7 shows how to measure and repair calibration.

Which variant for which data

VariantFeaturesTypical use
GaussianNBContinuousNumeric tabular data
MultinomialNBCountsWord counts in text classification
BernoulliNBBinaryPresence or absence flags
CategoricalNBCategorical codesSurvey responses
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

notes = ['billing error on my invoice', 'charged twice this month',
'internet connection keeps dropping', 'speed is very slow',
'wrong amount on my bill', 'internet drops every evening']
labels = ['billing', 'billing', 'technical', 'technical',
'billing', 'technical']

m = make_pipeline(CountVectorizer(), MultinomialNB()).fit(notes, labels)
for text in ['my bill is wrong again', 'internet connection drops',
'charged the wrong amount']:
print('%-32s -> %s' % (text, m.predict([text])[0]))
my bill is wrong again -> billing
internet connection drops -> technical
charged the wrong amount -> billing

This is what naive Bayes is genuinely good at. Text has thousands of sparse features, the independence assumption is less damaging when each word contributes a little, and training takes milliseconds. It remains a sensible baseline for text classification.

Day 4 takeaway

Naive Bayes assumes conditional independence, which is nearly always false, and still ranks well because ranking survives the assumption better than probability estimation does. Its probabilities are pushed to the extremes and should not be quoted. It is fast, needs little data, and remains a strong baseline for text.
Week 05 · Day 5 of 7

Support Vector Machines

Margins, kernels, and the two parameters that must be tuned together

By 791 words

Logistic regression draws a boundary that separates the classes. A support vector machine asks a better question: of all the boundaries that separate them, which one leaves the most room on either side?

The margin

Support vector machine: Finds the separating boundary with the widest margin, the largest distance to the nearest point of either class. Only those nearest points, the support vectors, affect the answer. Move any other point and the boundary does not shift.
import numpy as np
from sklearn.svm import SVC
from sklearn.datasets import make_blobs

X, y = make_blobs(n_samples=200, centers=2, cluster_std=1.2, random_state=7)
m = SVC(kernel='linear', C=1.0).fit(X, y)

print('training points %d' % len(X))
print('support vectors %d' % len(m.support_))
print('so the boundary depends on %.1f%% of the data'
% (100 * len(m.support_) / len(X)))
training points 200
support vectors 3
so the boundary depends on 1.5% of the data

The kernel trick

Some classes cannot be separated by a straight line at all. A kernel maps the data into a higher-dimensional space where they can be, without ever computing the coordinates in that space.

import numpy as np
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import cross_val_score

X, y = make_circles(n_samples=400, factor=0.4, noise=0.12, random_state=0)
print('one class forms a ring around the other -- no line can split it')

for kernel in ['linear', 'poly', 'rbf']:
acc = cross_val_score(SVC(kernel=kernel, random_state=0), X, y, cv=5).mean()
print(' %-8s accuracy %.4f' % (kernel, acc))
one class forms a ring around the other -- no line can split it
linear accuracy 0.5450
poly accuracy 0.6250
rbf accuracy 0.9975

The linear kernel manages barely better than guessing, because no straight line can separate a ring from its centre. The RBF kernel handles it almost perfectly.

C and gamma

ParameterLow valueHigh value
CWide margin, tolerates mistakes (more bias)Narrow margin, fits training data hard (more variance)
gammaEach point influences a wide area (smoother)Influence is local, boundary wraps individual points (overfits)
import numpy as np
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_circles(n_samples=400, factor=0.4, noise=0.12, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)

print('%8s %8s %10s %10s' % ('C', 'gamma', 'train', 'test'))
for C_val, g in [(1, 0.1), (1, 1), (1, 100), (1000, 100)]:
m = SVC(C=C_val, gamma=g, random_state=0).fit(X_tr, y_tr)
print('%8s %8s %10.4f %10.4f'
% (C_val, g, accuracy_score(y_tr, m.predict(X_tr)),
accuracy_score(y_te, m.predict(X_te))))
C gamma train test
1 0.1 0.9750 0.9583
1 1 1.0000 1.0000
1 100 1.0000 0.9917
1000 100 1.0000 0.9917

At gamma 100 the model scores near-perfectly on training data and worse on test: each support vector now influences only its immediate neighbourhood, so the boundary wraps individual points. Classic overfitting, and visible only because we looked at both columns.

On the churn problem

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
import time

for name, clf in [('svm rbf ', SVC(probability=True, random_state=42)),
('logistic ', LogisticRegression(max_iter=1000,
random_state=42))]:
t = time.perf_counter()
m = Pipeline([('prep', preprocessor()), ('clf', clf)]).fit(X_tr, y_tr)
elapsed = time.perf_counter() - t
auc = roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])
print('%s AUC %.4f fitted in %.2fs' % (name, auc, elapsed))
svm rbf AUC 0.7804 fitted in 0.50s
logistic AUC 0.8174 fitted in 0.02s

SVMs scale badly

Training cost grows between quadratically and cubically with the number of rows. At three thousand rows that is a second; at a million it is impractical. probability=True makes it worse still, because it fits an extra calibration model by internal cross-validation. For large tabular datasets, reach for the gradient boosting of week 6 instead.

Day 5 takeaway

An SVM maximises the margin and depends only on the support vectors near the boundary. Kernels let it draw curved boundaries without computing high-dimensional coordinates. C and gamma both trade bias for variance and must be tuned together. It scales poorly, which limits it to small and medium data.
Week 05 · Day 6 of 7

More Than Two Classes

One-vs-rest, confusion matrices, averaging traps and softmax

By 686 words

Everything so far has had two classes. Most real problems have more: which of nine defect types, which of five support queues, which of three risk bands.

Two strategies

StrategyModels fittedEach model learnsCost
One-vs-restOne per classThis class against all othersn models
One-vs-oneOne per pairThis class against that onen(n−1)/2 models
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

X, y = load_wine(return_X_y=True)
print('classes %d, rows %d, features %d' % (len(set(y)), *X.shape))

base = LogisticRegression(max_iter=5000, random_state=0)
for name, clf in [('built-in ', base),
('one-vs-rest ', OneVsRestClassifier(base)),
('one-vs-one ', OneVsOneClassifier(base))]:
acc = cross_val_score(make_pipeline(StandardScaler(), clf), X, y, cv=5).mean()
print('%s accuracy %.4f' % (name, acc))
classes 3, rows 178, features 13
built-in accuracy 0.9832
one-vs-rest accuracy 0.9889
one-vs-one accuracy 0.9833

scikit-learn handles multi-class automatically for most estimators, so you rarely wrap anything by hand. Knowing the strategies matters when you need to explain why a model produced an inconsistent set of scores.

The confusion matrix earns its name here

from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report

X, y = load_wine(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.35,
stratify=y, random_state=0)
m = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=5000, random_state=0)).fit(X_tr, y_tr)

print('rows = truth, columns = prediction')
print(confusion_matrix(y_te, m.predict(X_te)))
print()
print(classification_report(y_te, m.predict(X_te), digits=3))
rows = truth, columns = prediction
[[21 0 0]
[ 1 24 0]
[ 0 0 17]]

precision recall f1-score support

0 0.955 1.000 0.977 21
1 1.000 0.960 0.980 25
2 1.000 1.000 1.000 17

accuracy 0.984 63
macro avg 0.985 0.987 0.985 63
weighted avg 0.985 0.984 0.984 63

The off-diagonal cells tell you which classes the model confuses, and that is usually more actionable than the overall accuracy. Two classes that get mixed up may need a feature that distinguishes them specifically.

Averaging, and why the choice matters

import numpy as np
from sklearn.metrics import f1_score, precision_score, recall_score

# 100 of class 0, 100 of class 1, 10 of class 2 (the rare, important one)
truth = np.array([0] * 100 + [1] * 100 + [2] * 10)
pred = truth.copy()
pred[200:] = 0 # every single class-2 row misclassified

for avg in ['micro', 'macro', 'weighted']:
print('%-9s F1 %.4f' % (avg, f1_score(truth, pred, average=avg)))
micro F1 0.9524
macro F1 0.6508
weighted F1 0.9297

micro and weighted hide a total failure on a rare class

The model gets every one of the rare class wrong, and micro-averaged F1 still reads 0.95 because the rare class is 5 percent of the rows. Macro-averaging treats every class equally regardless of size and drops to 0.65, which is the honest figure. When the rare class is the one you care about, fraud, defects, disease, use macro, or report per-class figures.

Probabilities across several classes

import numpy as np

def softmax(z):
z = z - z.max() # subtract the max for numerical stability
e = np.exp(z)
return e / e.sum()

scores = np.array([2.0, 1.0, 0.1])
probs = softmax(scores)
print('raw scores ', scores)
print('probabilities', probs.round(4))
print('they sum to %.4f' % probs.sum())

big = np.array([1000.0, 999.0, 998.0])
print('\nwithout the max subtraction exp(1000) overflows;')
print('with it:', softmax(big).round(4))
raw scores [2. 1. 0.1]
probabilities [0.659 0.2424 0.0986]
they sum to 1.0000

without the max subtraction exp(1000) overflows;
with it: [0.6652 0.2447 0.09 ]

Softmax is the multi-class sigmoid, and it appears again as the output layer of every classification network in week 12. Subtracting the maximum changes nothing mathematically and prevents overflow, which is why every real implementation does it.

Day 6 takeaway

Multi-class works by fitting one model per class or per pair, and scikit-learn does it for you. Read the confusion matrix to see which classes get mixed up. Choose macro averaging when a rare class matters, because micro and weighted will hide a complete failure on it. Softmax generalises the sigmoid.
Week 05 · Day 7 of 7

Comparing Classifiers Honestly

Five algorithms, cross-validation, and whether the differences are real

By 1016 words

Five algorithms, one problem, one honest comparison, including how long each takes and whether the differences mean anything.

Compare by cross-validation

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_validate, StratifiedKFold
import numpy as np

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
candidates = {
'baseline ': DummyClassifier(strategy='most_frequent'),
'logistic ': LogisticRegression(max_iter=1000, random_state=42),
'kNN k=25 ': KNeighborsClassifier(25),
'naive bayes': GaussianNB(),
'svm rbf ': SVC(probability=True, random_state=42),
'tree d=5 ': DecisionTreeClassifier(max_depth=5, random_state=42),
}

print('%-12s %14s %16s %10s' % ('model', 'ROC AUC', 'avg precision', 'fit secs'))
for name, clf in candidates.items():
pipe = Pipeline([('prep', preprocessor()), ('clf', clf)])
r = cross_validate(pipe, X_tr, y_tr, cv=cv,
scoring=['roc_auc', 'average_precision'])
print('%-12s %.4f +/-%.3f %10.4f +/-%.3f %10.2f'
% (name, r['test_roc_auc'].mean(), r['test_roc_auc'].std(),
r['test_average_precision'].mean(),
r['test_average_precision'].std(), r['fit_time'].mean()))
model ROC AUC avg precision fit secs
baseline 0.5000 +/-0.000 0.2680 +/-0.001 0.01
logistic 0.8208 +/-0.014 0.5974 +/-0.029 0.03
kNN k=25 0.8001 +/-0.010 0.5659 +/-0.022 0.01
naive bayes 0.8098 +/-0.006 0.5774 +/-0.027 0.01
svm rbf 0.7944 +/-0.020 0.6017 +/-0.034 0.34
tree d=5 0.7886 +/-0.014 0.5161 +/-0.025 0.01

Read the standard deviations before declaring a winner

If two models differ by less than the spread across folds, you have not shown one is better. Week 3's bootstrap made the same point about a single test set. The honest summary is usually “these three are equivalent, and one of them is ten times faster”.

Is the difference real?

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score, StratifiedKFold
from scipy import stats
import numpy as np

cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)
a = cross_val_score(Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]),
X_tr, y_tr, cv=cv, scoring='roc_auc')
b = cross_val_score(Pipeline([('prep', preprocessor()),
('clf', SVC(probability=True, random_state=42))]),
X_tr, y_tr, cv=cv, scoring='roc_auc')

print('logistic %.4f +/- %.4f' % (a.mean(), a.std()))
print('svm %.4f +/- %.4f' % (b.mean(), b.std()))
t, pval = stats.ttest_rel(a, b)
print('\npaired t-test on the same folds: p = %.4f' % pval)
logistic 0.8228 +/- 0.0175
svm 0.7926 +/- 0.0247

paired t-test on the same folds: p = 0.0002

A paired test compares the two models fold by fold, which removes the variation caused by some folds simply being harder. It is the right test here precisely because both models saw identical splits.

Choosing

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor():
return ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (roc_auc_score, average_precision_score,
confusion_matrix, classification_report)
import numpy as np

final = Pipeline([('prep', preprocessor()),
('clf', LogisticRegression(max_iter=1000,
random_state=42))]).fit(X_tr, y_tr)
proba = final.predict_proba(X_te)[:, 1]

# The threshold chosen on day 2, applied once to held-out data.
OFFER_COST, VALUE, SAVE_RATE = 30, 250, 0.35
best_t = max(np.arange(0.05, 0.95, 0.01),
key=lambda t: (lambda c: c[3] * SAVE_RATE * VALUE
- (c[3] + c[1]) * OFFER_COST)(
confusion_matrix(y_te, (proba >= t).astype(int)).ravel()))

print('TEST SET')
print(' ROC AUC %.4f' % roc_auc_score(y_te, proba))
print(' average precision %.4f' % average_precision_score(y_te, proba))
print(' chosen threshold %.2f' % best_t)
print()
print(classification_report(y_te, (proba >= best_t).astype(int),
target_names=['stayed', 'churned'], digits=3))
TEST SET
ROC AUC 0.8174
average precision 0.5993
chosen threshold 0.34

precision recall f1-score support

stayed 0.878 0.783 0.828 549
churned 0.542 0.701 0.612 201

accuracy 0.761 750
macro avg 0.710 0.742 0.720 750
weighted avg 0.788 0.761 0.770 750

Recall on the churned class is now far above what the default threshold gave in week 1, at the cost of precision, which is exactly the trade the business case asked for.

Your assignment

Take the two best models from the comparison and average their predicted probabilities. Score the average. You will often find it beats both. That is ensembling, and week 6 explains why it works. Then check whether the improvement is larger than the fold-to-fold spread before you believe it.

Day 7 takeaway

Compare classifiers on cross-validated ROC AUC and average precision, with fit time alongside, and treat differences smaller than the fold spread as noise. Use a paired test when both models saw the same folds. Then pick the operating threshold from the business case, not from the default.