Everything so far, assembled. Today you build a working churn model, measure it honestly, and see why the obvious metric misleads.
Step 1: load and clean
import pandas as pd
df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')
print(df.shape, '| churn rate %.3f' % df['churned'].mean())
(3000, 11) | churn rate 0.268
Step 2: split before anything else
The split comes first. Any decision you make after looking at the test set, which columns to keep, how to fill gaps, which model to use, contaminates it.
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')
num_cols = ['tenure_months', 'support_calls', 'monthly_charges', 'total_charges']
cat_cols = ['contract', 'internet_service', 'payment_method', 'has_dependents']
X = df[num_cols + cat_cols]
y = df['churned']
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
print('train', X_train.shape, 'churn %.3f' % y_train.mean())
print('test ', X_test.shape, 'churn %.3f' % y_test.mean())
train (2250, 8) churn 0.268
test (750, 8) churn 0.268
stratify=y
Without it, a random split can hand you a test set with a noticeably different churn rate than the training set, and the score you get reflects that accident rather than the model. stratify keeps the class proportions identical in both halves. Always pass it for classification.
Step 3: one object that preprocesses and predicts
Numeric columns need their gaps filled and their scales evened out. Categorical columns need turning into numbers. A ColumnTransformer applies different treatment to different columns; a Pipeline chains that to the model so the whole thing behaves like a single estimator.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')
num_cols = ['tenure_months', 'support_calls', 'monthly_charges', 'total_charges']
cat_cols = ['contract', 'internet_service', 'payment_method', 'has_dependents']
X, y = df[num_cols + cat_cols], df['churned']
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
numeric = Pipeline([
('impute', SimpleImputer(strategy='median')),
('scale', StandardScaler()),
])
categorical = Pipeline([
('impute', SimpleImputer(strategy='most_frequent')),
('encode', OneHotEncoder(handle_unknown='ignore')),
])
preprocess = ColumnTransformer([
('num', numeric, num_cols),
('cat', categorical, cat_cols),
])
model = Pipeline([
('prep', preprocess),
('clf', LogisticRegression(max_iter=1000, random_state=42)),
])
model.fit(X_train, y_train)
print('fitted: %.3f' % model.score(X_test, y_test))
fitted: 0.780
Twelve lines of setup and one fit. Everything inside learns from the training data only, because the pipeline passes the split through in the right order for you.
Step 4: measure it against the floor
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, roc_auc_score, confusion_matrix
df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')
num_cols = ['tenure_months', 'support_calls', 'monthly_charges', 'total_charges']
cat_cols = ['contract', 'internet_service', 'payment_method', 'has_dependents']
X, y = df[num_cols + cat_cols], df['churned']
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
preprocess = ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), num_cols),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('e', OneHotEncoder(handle_unknown='ignore'))]), cat_cols),
])
model = Pipeline([('prep', preprocess),
('clf', LogisticRegression(max_iter=1000, random_state=42))])
model.fit(X_train, y_train)
baseline = DummyClassifier(strategy='most_frequent').fit(X_train, y_train)
pred = model.predict(X_test)
proba = model.predict_proba(X_test)[:, 1]
print('baseline accuracy %.3f' % accuracy_score(y_test, baseline.predict(X_test)))
print('model accuracy %.3f' % accuracy_score(y_test, pred))
print('model ROC AUC %.3f' % roc_auc_score(y_test, proba))
print('\nconfusion matrix (rows = truth, cols = prediction)')
print(confusion_matrix(y_test, pred))
baseline accuracy 0.732
model accuracy 0.780
model ROC AUC 0.818
confusion matrix (rows = truth, cols = prediction)
[[499 50]
[115 86]]
78 percent sounds good until you read the matrix
Predicting “nobody churns” scores 73.2 percent. Our model scores 78.0, just under five points of real improvement, not the triumph the raw number suggests. And the matrix shows where it goes wrong: of 201 customers who actually churned, it caught 86 and missed 115. It finds fewer than half of the people you most wanted to find.
This is not a broken model. An AUC of 0.818 means it ranks customers by risk genuinely well. It is a badly chosen threshold. The model outputs a probability, and something has to decide where to cut. The default cut is 0.5, which is almost never the right choice when one class is rarer than the other. Week 7 is about fixing exactly this.
Step 5: what did it learn
import pandas as pd, numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')
num_cols = ['tenure_months', 'support_calls', 'monthly_charges', 'total_charges']
cat_cols = ['contract', 'internet_service', 'payment_method', 'has_dependents']
X, y = df[num_cols + cat_cols], df['churned']
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
preprocess = ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), num_cols),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('e', OneHotEncoder(handle_unknown='ignore'))]), cat_cols),
])
model = Pipeline([('prep', preprocess),
('clf', LogisticRegression(max_iter=1000, random_state=42))])
model.fit(X_train, y_train)
names = model.named_steps['prep'].get_feature_names_out()
coefs = model.named_steps['clf'].coef_[0]
effect = pd.Series(coefs, index=names).sort_values()
print('pushes towards staying:')
print(effect.head(3).round(2))
print('\npushes towards churning:')
print(effect.tail(3).round(2))
pushes towards staying:
cat__contract_Two Year -1.44
num__tenure_months -0.81
cat__payment_method_Credit card -0.20
dtype: float64
pushes towards churning:
cat__payment_method_Electronic check 0.46
num__monthly_charges 0.48
cat__contract_Month-To-Month 1.31
dtype: float64
Compare that against the generator you wrote on day 3: month-to-month contracts push churn up, two-year contracts and long tenure push it down, electronic-check payment pushes it up. The model recovered the rule that created the data. That is what a good fit looks like.
Your assignment
Swap LogisticRegression for RandomForestClassifier(n_estimators=300, random_state=42), a one-line change, because both obey the same contract. Record accuracy and AUC. You will find the forest scores worse on AUC. Write down why you think that is; week 6 gives the answer, and it is not that you did something wrong.
Day 7 takeaway
The end-to-end shape is: clean, split, build a pipeline that preprocesses and predicts, fit on train, score on test, compare against a baseline. You have a churn model beating the majority-class floor by just under five points with an AUC of 0.818, and a confusion matrix showing it misses most churners, because the default 0.5 threshold is wrong for imbalanced data.