Feature Engineering

Week 8 of 16 · Evaluation and features · 7 days

Week 08 · Evaluation and features

Feature Engineering

Week 08 · Day 1 of 7

Why Features Beat Models

The discipline, the baseline, and a first attempt that barely moves

Week 6 ended with a week of ensembles drawing against a straight line. That is not a limit of the algorithms; it is a limit of the features. Better inputs beat better models, reliably, and this week is where the gains actually live.

Feature engineering: Turning raw columns into inputs that express what the model needs to know. Sometimes that is a transformation, sometimes an interaction, sometimes an aggregate from another table. It is the part of the job that requires knowing what the data means.

The discipline that makes it safe

Every feature you build learns something, a mean, a category list, a bin edge. Learn it from all your data and you have leaked. So every engineered feature belongs inside the pipeline, fitted on training folds only. That is not a style preference; week 7 showed it turning noise into an apparent 0.80.

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(5, shuffle=True, random_state=0)
baseline = Pipeline([('prep', preprocessor()),
                     ('clf', LogisticRegression(max_iter=1000,
                                                random_state=42))])
s = cross_val_score(baseline, X_tr, y_tr, cv=cv, scoring='roc_auc')
print('the number to beat: %.4f  +/- %.4f' % (s.mean(), s.std()))

the number to beat: 0.8238 +/- 0.0305

Write the baseline down before you start

Feature engineering without a recorded baseline is how people spend a fortnight building features that made things worse. Every idea this week gets measured against 0.8238, on the same folds, and kept only if it beats it by more than the fold spread.

Where features come from

Source	Example	Watch out for
Transformation	`log(charges)`	Undefined at zero, use `log1p`
Ratio	charges per month of tenure	Division by zero; and it may reconstruct the target
Interaction	month-to-month and high support calls	Column count explodes quickly
Aggregate	average spend for this contract type	Must be computed on training folds only
Time	months since signup	Never derive from today's date
Domain knowledge	is this customer out of contract	The most valuable and the least automatable

A first attempt, measured

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold, train_test_split
import numpy as np

eng = df.copy()
eng['charge_per_tenure'] = eng['monthly_charges'] / eng['tenure_months']
eng['calls_per_year'] = eng['support_calls'] / (eng['tenure_months'] / 12)
eng['log_calls'] = np.log1p(eng['support_calls'])

NEW = NUM + ['charge_per_tenure', 'calls_per_year', 'log_calls']
Xe, ye = eng[NEW + CAT], eng[TARGET]
Xe_tr, Xe_te, ye_tr, ye_te = train_test_split(Xe, ye, test_size=0.25,
                                              stratify=ye, random_state=42)

cv = StratifiedKFold(5, shuffle=True, random_state=0)
m = Pipeline([('prep', preprocessor(num=NEW)),
              ('clf', LogisticRegression(max_iter=1000, random_state=42))])
s = cross_val_score(m, Xe_tr, ye_tr, cv=cv, scoring='roc_auc')
print('with three new features: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('baseline was            0.8238')

with three new features: 0.8278  +/- 0.0220
baseline was            0.8238

Barely moved, and that is the normal outcome of a first attempt. Ratios invented without a reason rarely help. The features that work come from knowing something about the problem, which is the rest of this week.

Day 1 takeaway

Better features beat better models, but only when they encode something real. Record the baseline before you start, measure every idea on the same folds, and keep only what beats the fold spread. Every feature that learns from data must be fitted inside the pipeline.

Week 08 · Day 2 of 7

Numeric Transformations

Choosing a scaler, and buying curvature with bins or splines

By Pawan Rama Mali876 words

Scaling is not one choice. Which scaler you pick changes what the model sees, and the default is wrong whenever outliers are present.

The three scalers

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.preprocessing import (StandardScaler, MinMaxScaler,
                                   RobustScaler, QuantileTransformer)
import numpy as np

x = df[['support_calls']].to_numpy(dtype=float)   # contains the 97 outlier

print('%-22s %8s %8s %8s %8s' % ('', 'median', 'IQR', 'max', 'p99'))
print('%-22s %8.3f %8.3f %8.3f %8.3f'
      % ('raw', np.median(x), np.subtract(*np.percentile(x, [75, 25])),
         x.max(), np.percentile(x, 99)))
for name, sc in [('StandardScaler', StandardScaler()),
                 ('MinMaxScaler', MinMaxScaler()),
                 ('RobustScaler', RobustScaler()),
                 ('QuantileTransformer', QuantileTransformer(
                     output_distribution='normal', random_state=0))]:
    z = sc.fit_transform(x)
    print('%-22s %8.3f %8.3f %8.3f %8.3f'
          % (name, np.median(z), np.subtract(*np.percentile(z, [75, 25])),
             z.max(), np.percentile(z, 99)))

                         median      IQR      max      p99
raw                       1.000    2.000   97.000    5.000
StandardScaler           -0.127    0.952   45.590    1.778
MinMaxScaler              0.010    0.021    1.000    0.052
RobustScaler              0.000    1.000   48.000    2.000
QuantileTransformer      -0.045    5.890    5.199    2.483

Scaler	Centres on	Divides by	Use when
`StandardScaler`	Mean	Standard deviation	Roughly symmetric, no extreme values
`MinMaxScaler`	Minimum	Range	You need a bounded 0 to 1 range
`RobustScaler`	Median	Interquartile range	Outliers are present, usually the safer default
`QuantileTransformer`	-	-	You want a specific output shape regardless of input

MinMaxScaler is destroyed by a single outlier

One customer with 97 calls sets the maximum, so every other customer. The ones with 0 to 5 calls, is compressed into the bottom five percent of the range. The scaler has technically done its job and practically destroyed the column's resolution. RobustScaler uses quartiles, which one row cannot move.

Binning

Sometimes the relationship is not smooth. A customer at month 11 and one at month 13 may behave very differently if the contract renews at 12.

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.preprocessing import KBinsDiscretizer
from sklearn.linear_model import LogisticRegression
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import cross_val_score, StratifiedKFold, train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

binned = ColumnTransformer([
    ('bin', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('b', KBinsDiscretizer(n_bins=8, encode='onehot-dense',
                                             strategy='quantile'))]),
     ['tenure_months']),
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]),
     ['support_calls', 'monthly_charges']),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
m = Pipeline([('prep', binned),
              ('clf', LogisticRegression(max_iter=1000, random_state=42))])
s = cross_val_score(m, X_tr, y_tr, cv=cv, scoring='roc_auc')
print('tenure binned into 8: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('baseline:             0.8238')

tenure binned into 8: 0.8221  +/- 0.0295
baseline:             0.8238

Binning throws information away

Every customer inside a bin becomes identical. You gain the ability to express a non-linear step and you lose all resolution within the bin. It helps when the true relationship really does jump; it hurts when the relationship was smooth, which is the case here. Measure, do not assume.

Splines, which give curvature without losing resolution

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.preprocessing import SplineTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.model_selection import cross_val_score, StratifiedKFold, train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

spl = ColumnTransformer([
    ('spline', Pipeline([('i', SimpleImputer(strategy='median')),
                         ('s', SplineTransformer(n_knots=5, degree=3))]),
     ['tenure_months']),
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]),
     ['support_calls', 'monthly_charges']),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
m = Pipeline([('prep', spl),
              ('clf', LogisticRegression(max_iter=1000, random_state=42))])
s = cross_val_score(m, X_tr, y_tr, cv=cv, scoring='roc_auc')
print('tenure as a spline: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('baseline:           0.8238')

tenure as a spline: 0.8235  +/- 0.0303
baseline:           0.8238

A spline is a smooth curve built from overlapping local basis functions. It lets a linear model bend without the staircase that binning imposes, and it is usually the better tool when you suspect curvature.

Day 2 takeaway

RobustScaler is the safer default when outliers exist, because StandardScaler and especially MinMaxScaler are moved by single extreme rows. Binning buys a step change at the cost of all within-bin resolution; splines buy curvature without that cost. Measure both against the baseline.

Week 08 · Day 3 of 7

Encoding High-Cardinality Categories

Target encoding, how it leaks, and what makes it safe

By Pawan Rama Mali1016 words

One-hot encoding works until a column has a thousand levels. Then you need target encoding, which is powerful, and which leaks unless you are very careful.

The problem

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 5000
postcodes = ['PC%04d' % i for i in rng.integers(0, 900, n)]
s = pd.Series(postcodes)
print('rows            %d' % n)
print('distinct values %d' % s.nunique())
print('one-hot columns %d' % s.nunique())
print('\nlevels seen fewer than 5 times: %d'
      % (s.value_counts() < 5).sum())

rows            5000
distinct values 896
one-hot columns 896

levels seen fewer than 5 times: 307

Target encoding: Replace each category with the mean of the target for that category. One column instead of hundreds, and it carries exactly the information the model wants. It is also the most reliable way to leak your target if computed carelessly.

The naive version, and why it fails

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

rng = np.random.default_rng(0)
n = 2000
# A category with no relationship to the target whatsoever.
cat = pd.Series(['C%03d' % i for i in rng.integers(0, 400, n)])
y = pd.Series(rng.integers(0, 2, n))

# Naive: compute the mean target per category using every row.
means = y.groupby(cat).mean()
encoded = cat.map(means).to_frame('te')

cv = StratifiedKFold(5, shuffle=True, random_state=0)
s = cross_val_score(LogisticRegression(), encoded, y, cv=cv, scoring='roc_auc')
print('random category, random target -- truth is 0.5')
print('naive target encoding scores:  %.4f' % s.mean())

random category, random target -- truth is 0.5
naive target encoding scores:  0.7587

It encoded the answer into the feature

With 400 categories over 2000 rows, each category has about five rows. Its mean target is therefore mostly determined by those five labels, including the one you are about to predict. The feature is a leaky copy of the target, and cross-validation cannot save you because the leak happened before the folds existed.

The safe version

import numpy as np
import pandas as pd
from sklearn.preprocessing import TargetEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold

rng = np.random.default_rng(0)
n = 2000
cat = np.array([['C%03d' % i] for i in rng.integers(0, 400, n)])
y = rng.integers(0, 2, n)

cv = StratifiedKFold(5, shuffle=True, random_state=0)
m = make_pipeline(TargetEncoder(random_state=0), LogisticRegression())
s = cross_val_score(m, cat, y, cv=cv, scoring='roc_auc')
print('sklearn TargetEncoder, inside a pipeline: %.4f' % s.mean())
print('the truth is 0.5 -- and now it says so')

sklearn TargetEncoder, inside a pipeline: 0.5055
the truth is 0.5 -- and now it says so

scikit-learn's TargetEncoder does two things that make it safe. It fits an internal cross-fitting scheme, so a row's encoding never uses its own label. And it shrinks each category mean toward the global mean in proportion to how few rows that category has, so a category seen twice barely moves away from the overall rate.

import numpy as np
from sklearn.preprocessing import TargetEncoder

cat = np.array([['common'] * 500 + ['rare'] * 3]).reshape(-1, 1)
y = np.array([1] * 250 + [0] * 250 + [1, 1, 1])   # rare is 100% positive

print('global mean       %.4f' % y.mean())
print('raw mean for rare 1.0000  (3 rows, all positive)\n')
print('%10s %12s %10s' % ('smooth', 'common', 'rare'))
for smooth in ['auto', 1.0, 5.0, 20.0]:
    enc = TargetEncoder(random_state=0, cv=5, smooth=smooth).fit(cat, y)
    print('%10s %12.4f %10.4f'
          % (smooth, enc.transform([['common']])[0, 0],
             enc.transform([['rare']])[0, 0]))

global mean       0.5030
raw mean for rare 1.0000  (3 rows, all positive)

    smooth       common       rare
      auto       0.5000     1.0000
       1.0       0.5000     0.8757
       5.0       0.5000     0.6894
      20.0       0.5001     0.5678

smooth='auto' did not shrink it at all

The default estimates how much to shrink from the variance within each category. Our rare group is three rows that are all 1, so its within-group variance is zero, the estimator concludes the mean is perfectly reliable, and it leaves 1.0 untouched. Three rows, and the encoder is certain.

An explicit smooth is a fixed number of pseudo-observations at the global mean, so it pulls regardless: 0.88 at 1, 0.69 at 5, 0.57 at 20. When your rare categories are small and internally consistent, which is exactly when they are most dangerous, set smooth yourself rather than trusting the automatic estimate.

On our data

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.preprocessing import TargetEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(5, shuffle=True, random_state=0)
te = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('t', TargetEncoder(random_state=0))]), CAT),
])
m = Pipeline([('prep', te),
              ('clf', LogisticRegression(max_iter=1000, random_state=42))])
s = cross_val_score(m, X_tr, y_tr, cv=cv, scoring='roc_auc')
print('target encoded: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('one-hot was:    0.8238')

target encoded: 0.8197  +/- 0.0271
one-hot was:    0.8238

Level with one-hot, which is expected: our categories have three or four levels each, so one-hot costs nothing and target encoding has nothing to compress. Reach for it at hundreds of levels, not at four.

Day 3 takeaway

Target encoding replaces a high-cardinality column with the mean target per category, and computed naively it leaks the answer, scoring 0.76 on data with no signal at all. Use scikit-learn's TargetEncoder inside a pipeline, which cross-fits and shrinks rare categories toward the global mean. It earns its place only when cardinality is genuinely high.

Week 08 · Day 4 of 7

Interactions and Domain Features

The thing a linear model cannot do, and how to hand it over

By Pawan Rama Mali878 words

A linear model cannot express “month-to-month customers who also call support a lot”. It can only add the two effects. Handing it the combination is often where the real gain is.

Why a linear model needs help

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
n = 4000
a = rng.integers(0, 2, n)
b = rng.integers(0, 2, n)
y = (a ^ b)                                   # exclusive or
y = np.where(rng.uniform(size=n) < 0.05, 1 - y, y)

X = np.column_stack([a, b])
print('without the interaction %.4f'
      % cross_val_score(make_pipeline(StandardScaler(), LogisticRegression()),
                        X, y, cv=5, scoring='roc_auc').mean())

X2 = np.column_stack([a, b, a * b])
print('with a*b as a column    %.4f'
      % cross_val_score(make_pipeline(StandardScaler(), LogisticRegression()),
                        X2, y, cv=5, scoring='roc_auc').mean())

without the interaction 0.5073
with a*b as a column    0.9558

Chance, then near-perfect, from one extra column. Neither feature alone says anything about the target; only their combination does. A tree finds this by splitting twice; a linear model finds it only if you supply it.

Automatic interactions, and their cost

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.preprocessing import PolynomialFeatures, StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(5, shuffle=True, random_state=0)
inter = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
])
m = Pipeline([('prep', inter),
              ('poly', PolynomialFeatures(degree=2, interaction_only=True,
                                          include_bias=False)),
              ('clf', LogisticRegression(max_iter=3000, random_state=42))])
m.fit(X_tr, y_tr)
print('columns after encoding      %d'
      % m.named_steps['prep'].transform(X_tr).shape[1])
print('columns after interactions  %d'
      % m.named_steps['poly'].transform(
          m.named_steps['prep'].transform(X_tr)).shape[1])
s = cross_val_score(m, X_tr, y_tr, cv=cv, scoring='roc_auc')
print('\nall pairwise interactions: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('baseline:                  0.8238')

columns after encoding      15
columns after interactions  120

all pairwise interactions: 0.8140  +/- 0.0240
baseline:                  0.8238

Every pair, including the meaningless ones

interaction_only=True at degree 2 gives you every pairwise product, and most of them mean nothing, fibre optic times mailed check is not a concept. You have multiplied the column count and handed the model far more opportunity to overfit for one or two useful combinations. Pair it with regularisation, or engineer the interactions you can justify.

Deliberate features instead

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold, train_test_split
import numpy as np

eng = df.copy()

# 1. On a rolling contract AND unhappy: the combination we can justify.
eng['rolling_and_calling'] = (
    (eng['contract'] == 'Month-To-Month') & (eng['support_calls'] >= 2)
).astype(int)

# 2. Paying a lot relative to others on the same service.
eng['price_vs_service'] = (
    eng['monthly_charges']
    - eng.groupby('internet_service')['monthly_charges'].transform('median'))

# 3. Still new. Churn risk is concentrated in the first months.
eng['is_new'] = (eng['tenure_months'] <= 6).astype(int)

# 4. Calls per year of tenure, not raw calls.
eng['call_rate'] = eng['support_calls'] / np.maximum(eng['tenure_months'], 1) * 12

NEW = NUM + ['rolling_and_calling', 'price_vs_service', 'is_new', 'call_rate']
Xe, ye = eng[NEW + CAT], eng[TARGET]
Xe_tr, _, ye_tr, _ = train_test_split(Xe, ye, test_size=0.25,
                                      stratify=ye, random_state=42)

cv = StratifiedKFold(5, shuffle=True, random_state=0)
m = Pipeline([('prep', preprocessor(num=NEW)),
              ('clf', LogisticRegression(max_iter=1000, random_state=42))])
s = cross_val_score(m, Xe_tr, ye_tr, cv=cv, scoring='roc_auc')
print('four justified features: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('baseline:                0.8238')

four justified features: 0.8267  +/- 0.0246
baseline:                0.8238

price_vs_service is computed over the whole dataset

That group median uses every row, including the ones in the validation fold. It is a small leak and it is still a leak: the score above is very slightly optimistic. Day 6 rebuilds it as a proper transformer that learns its medians on training folds only, which is the correct way to ship a feature like this.

Day 4 takeaway

Linear models add effects; they cannot multiply them. Supplying an interaction turns an impossible problem into a trivial one. Generating every pairwise product mostly generates noise, so prefer combinations you can justify, and note that any feature built from a group statistic leaks unless it is fitted inside the pipeline.

Week 08 · Day 5 of 7

Feature Selection

Filters, wrappers and embedded methods, and what filters cannot see

By Pawan Rama Mali811 words

More features is not better. Each irrelevant column adds noise, slows fitting, and gives the model another chance to find a coincidence.

Watch noise columns do damage

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np
import pandas as pd

cv = StratifiedKFold(5, shuffle=True, random_state=0)
rng = np.random.default_rng(0)

for extra in [0, 10, 50, 200]:
    Xn = X_tr.copy()
    for j in range(extra):
        Xn['noise_%d' % j] = rng.normal(size=len(Xn))
    num = NUM + ['noise_%d' % j for j in range(extra)]
    m = Pipeline([('prep', preprocessor(num=num)),
                  ('clf', LogisticRegression(max_iter=2000, random_state=42))])
    s = cross_val_score(m, Xn, y_tr, cv=cv, scoring='roc_auc')
    print('%4d noise columns: %.4f  +/- %.4f' % (extra, s.mean(), s.std()))

noise columns: 0.8238  +/- 0.0305
noise columns: 0.8194  +/- 0.0318
noise columns: 0.8053  +/- 0.0342
noise columns: 0.7910  +/- 0.0275

Two thousand rows and two hundred pure-noise columns costs real performance. The model spends its capacity fitting coincidences, and regularisation only partly protects it.

Three families of method

Family	How	Cost	Weakness
Filter	Score each feature against the target alone	Very cheap	Blind to combinations
Wrapper	Repeatedly fit the model with subsets	Expensive	Can overfit the selection itself
Embedded	The model selects while fitting (lasso, trees)	Free	Tied to that model's assumptions

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.feature_selection import (SelectKBest, mutual_info_classif,
                                       RFE, SelectFromModel)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(5, shuffle=True, random_state=0)
prep = preprocessor()

selectors = {
    'none          ': None,
    'filter, k=8   ': SelectKBest(mutual_info_classif, k=8),
    'wrapper, RFE 8': RFE(LogisticRegression(max_iter=1000), n_features_to_select=8),
    'embedded, L1  ': SelectFromModel(
        LogisticRegression(penalty='l1', solver='liblinear', C=0.1)),
}
for name, sel in selectors.items():
    steps = [('prep', prep)]
    if sel is not None:
        steps.append(('sel', sel))
    steps.append(('clf', LogisticRegression(max_iter=1000, random_state=42)))
    s = cross_val_score(Pipeline(steps), X_tr, y_tr, cv=cv, scoring='roc_auc')
    print('%s %.4f  +/- %.4f' % (name, s.mean(), s.std()))

none           0.8238  +/- 0.0305
filter, k=8    0.8184  +/- 0.0250
wrapper, RFE 8 0.8229  +/- 0.0313
embedded, L1   0.8234  +/- 0.0299

Selection lives inside the pipeline

Every selector above is a pipeline step, so it is refitted on each training fold and never sees the fold it is scored on. Week 7 showed what happens otherwise: selection over all the labels made pure noise score 0.80. There is no version of feature selection that is safe to do before splitting.

What filter methods miss

import numpy as np
from sklearn.feature_selection import mutual_info_classif, f_classif

rng = np.random.default_rng(0)
n = 4000
a = rng.integers(0, 2, n)
b = rng.integers(0, 2, n)
y = a ^ b
X = np.column_stack([a, b, rng.normal(size=n)])

print('mutual information with the target, feature by feature:')
for name, score in zip(['a', 'b', 'noise'], mutual_info_classif(X, y, random_state=0)):
    print('  %-6s %.5f' % (name, score))
print('\na and b together determine y exactly. Individually both')
print('score near zero -- a is indistinguishable from pure noise.')

mutual information with the target, feature by feature:
  a      0.00000
  b      0.01169
  noise  0.00000

a and b together determine y exactly. Individually both
score near zero -- a is indistinguishable from pure noise.

Any filter method evaluates one feature at a time, so it cannot see that two useless-looking columns are jointly decisive. If interactions matter in your problem, a filter will discard exactly the features you need.

Day 5 takeaway

Irrelevant columns cost real performance. Filters are cheap and blind to combinations; wrappers are expensive and can overfit the selection; embedded methods are free but inherit the model's assumptions. Whichever you use, it is a pipeline step, never a preprocessing pass.

Week 08 · Day 6 of 7

Writing Your Own Transformer

Turning a leaky groupby into a pipeline step that fits per fold

By Pawan Rama Mali839 words

Day 4 built a feature from a group median computed over the whole dataset, and flagged it as a leak. Today you fix it properly, by writing a transformer that obeys the scikit-learn contract.

The contract, from week 1

A transformer needs fit, which learns and returns self, and transform, which applies what was learned. Inherit two base classes and you get fit_transform, get_params and pipeline compatibility for nothing.

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin

class GroupRelativeValue(BaseEstimator, TransformerMixin):
    """How far a customer's value sits from the median of their group.

    The medians are learned in fit, from training rows only, so this is
    safe to use inside cross-validation -- which the day 4 version, built
    with a groupby over the whole frame, was not.
    """

    def __init__(self, group_col, value_col):
        self.group_col = group_col
        self.value_col = value_col

    def fit(self, X, y=None):
        self.medians_ = X.groupby(self.group_col)[self.value_col].median()
        self.global_median_ = X[self.value_col].median()
        return self

    def transform(self, X):
        # An unseen group falls back to the global median rather than NaN.
        base = X[self.group_col].map(self.medians_).fillna(self.global_median_)
        return (X[self.value_col] - base).to_frame('relative_value')

    def get_feature_names_out(self, input_features=None):
        return np.array(['relative_value'])


demo = pd.DataFrame({'internet_service': ['DSL', 'DSL', 'Fibre optic', 'Fibre optic'],
                     'monthly_charges': [50.0, 60.0, 75.0, 85.0]})
t = GroupRelativeValue('internet_service', 'monthly_charges').fit(demo)
print('learned medians:')
print(t.medians_.to_string())
print('\ntransformed:')
print(t.transform(demo).to_string(index=False))

unseen = pd.DataFrame({'internet_service': ['Satellite'], 'monthly_charges': [90.0]})
print('\nunseen group falls back to the global median:')
print(t.transform(unseen).to_string(index=False))

learned medians:
internet_service
DSL            55.0
Fibre optic    80.0

transformed:
 relative_value
           -5.0
            5.0
           -5.0
            5.0

unseen group falls back to the global median:
 relative_value
           22.5

Put it in a pipeline

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.model_selection import train_test_split

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np

class GroupRelativeValue(BaseEstimator, TransformerMixin):
    def __init__(self, group_col, value_col):
        self.group_col = group_col
        self.value_col = value_col

    def fit(self, X, y=None):
        self.medians_ = X.groupby(self.group_col)[self.value_col].median()
        self.global_median_ = X[self.value_col].median()
        return self

    def transform(self, X):
        base = X[self.group_col].map(self.medians_).fillna(self.global_median_)
        return (X[self.value_col] - base).fillna(0).to_frame('relative_value')

    def get_feature_names_out(self, input_features=None):
        return np.array(['relative_value'])


prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ('rel', Pipeline([('g', GroupRelativeValue('internet_service',
                                               'monthly_charges')),
                      ('s', StandardScaler())]),
     ['internet_service', 'monthly_charges']),
])

cv = StratifiedKFold(5, shuffle=True, random_state=0)
m = Pipeline([('prep', prep),
              ('clf', LogisticRegression(max_iter=1000, random_state=42))])
s = cross_val_score(m, X_tr, y_tr, cv=cv, scoring='roc_auc')
print('leak-free relative price: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('baseline:                 0.8238')

leak-free relative price: 0.8239  +/- 0.0308
baseline:                 0.8238

The medians are refitted on every fold

Five folds, five different sets of medians, each learned from four fifths of the data and applied to the fifth. No validation row contributed to the statistic used to encode it. That is the whole point of writing the transformer rather than doing a groupby in your notebook.

FunctionTransformer, for anything stateless

import numpy as np
import pandas as pd
from sklearn.preprocessing import FunctionTransformer

def add_ratios(X):
    out = X.copy()
    out['calls_per_year'] = (out['support_calls']
                             / np.maximum(out['tenure_months'], 1) * 12)
    out['log_calls'] = np.log1p(out['support_calls'])
    return out

# No feature_names_out='one-to-one' here: that promises the same columns
# out as in, and this function adds two. sklearn checks, and raises.
step = FunctionTransformer(add_ratios)
demo = pd.DataFrame({'support_calls': [0, 3, 12], 'tenure_months': [2, 24, 6]})
print(step.fit_transform(demo).round(3).to_string(index=False))

 support_calls  tenure_months  calls_per_year  log_calls
             0              2             0.0      0.000
             3             24             1.5      1.386
            12              6            24.0      2.565

Only when the function learns nothing

FunctionTransformer has no fit worth the name, so it is safe exactly when your function uses no statistic from the data. A ratio of two columns in the same row is fine. Anything involving a mean, a median, a category list or a bin edge needs a real transformer with a fit.

Day 6 takeaway

Any feature built from a statistic must learn that statistic in fit and apply it in transform. Inherit BaseEstimator and TransformerMixin, store learned values with a trailing underscore, and handle unseen categories. Use FunctionTransformer only for row-wise arithmetic that learns nothing.

Week 08 · Day 7 of 7

A Complete Feature Engineering Project

Every idea measured against the baseline, failures included

By Pawan Rama Mali1222 words

Everything this week, applied properly, measured honestly, against the 0.8238 that a week of ensembles could not beat.

The candidate features

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import (StandardScaler, OneHotEncoder,
                                   SplineTransformer, FunctionTransformer)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (cross_val_score, StratifiedKFold,
                                     train_test_split)
import numpy as np

def enrich(X):
    out = X.copy()
    out['rolling_and_calling'] = (
        (out['contract'] == 'Month-To-Month') & (out['support_calls'] >= 2)
    ).astype(int)
    out['is_new'] = (out['tenure_months'] <= 6).astype(int)
    out['call_rate'] = (out['support_calls']
                        / np.maximum(out['tenure_months'], 1) * 12)
    return out

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)

sample = enrich(X_tr.head(4))
print(sample[['contract', 'support_calls', 'tenure_months',
              'rolling_and_calling', 'is_new', 'call_rate']]
      .round(2).to_string(index=False))

      contract  support_calls  tenure_months  rolling_and_calling  is_new  call_rate
Month-To-Month              0              9                    0       0        0.0
      Two Year              0             21                    0       0        0.0
Month-To-Month              0              2                    0       1        0.0
Month-To-Month              0             12                    0       0        0.0

Assemble and measure

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def preprocessor(num=NUM, cat=CAT):
    return ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), cat),
    ])
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import (StandardScaler, OneHotEncoder,
                                   SplineTransformer, FunctionTransformer)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (cross_val_score, StratifiedKFold,
                                     train_test_split)
import numpy as np

class GroupRelativeValue(BaseEstimator, TransformerMixin):
    def __init__(self, group_col, value_col):
        self.group_col = group_col
        self.value_col = value_col

    def fit(self, X, y=None):
        self.medians_ = X.groupby(self.group_col)[self.value_col].median()
        self.global_median_ = X[self.value_col].median()
        return self

    def transform(self, X):
        base = X[self.group_col].map(self.medians_).fillna(self.global_median_)
        return (X[self.value_col] - base).fillna(0).to_frame('rel')

    def get_feature_names_out(self, input_features=None):
        return np.array(['rel'])


def enrich(X):
    out = X.copy()
    out['rolling_and_calling'] = (
        (out['contract'] == 'Month-To-Month') & (out['support_calls'] >= 2)
    ).astype(int)
    out['is_new'] = (out['tenure_months'] <= 6).astype(int)
    out['call_rate'] = (out['support_calls']
                        / np.maximum(out['tenure_months'], 1) * 12)
    return out


DERIVED = ['rolling_and_calling', 'is_new', 'call_rate']

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('derived', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), DERIVED),
    ('spline', Pipeline([('i', SimpleImputer(strategy='median')),
                         ('s', SplineTransformer(n_knots=5, degree=3))]),
     ['tenure_months']),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT),
    ('rel', Pipeline([('g', GroupRelativeValue('internet_service',
                                               'monthly_charges')),
                      ('s', StandardScaler())]),
     ['internet_service', 'monthly_charges']),
])

X, y = df[NUM + CAT], df[TARGET]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

full = Pipeline([
    ('enrich', FunctionTransformer(enrich)),
    ('prep', prep),
    ('clf', LogisticRegression(max_iter=3000, random_state=42)),
])
s = cross_val_score(full, X_tr, y_tr, cv=cv, scoring='roc_auc')
print('engineered: %.4f  +/- %.4f' % (s.mean(), s.std()))
print('baseline:   0.8238')
print('gain:       %+.4f  (fold spread %.4f)' % (s.mean() - 0.8238, s.std()))

engineered: 0.8259  +/- 0.0252
baseline:   0.8238
gain:       +0.0021  (fold spread 0.0252)

Does it help the ensembles too?

import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv', parse_dates=['signup_date'])
df = df.drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
TARGET = 'churned'
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, FunctionTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import (cross_val_score, StratifiedKFold,
                                     train_test_split)
import numpy as np

def enrich(X):
    out = X.copy()
    out['rolling_and_calling'] = (
        (out['contract'] == 'Month-To-Month') & (out['support_calls'] >= 2)
    ).astype(int)
    out['is_new'] = (out['tenure_months'] <= 6).astype(int)
    out['call_rate'] = (out['support_calls']
                        / np.maximum(out['tenure_months'], 1) * 12)
    return out

DERIVED = ['rolling_and_calling', 'is_new', 'call_rate']
X, y = df[NUM + CAT], df[TARGET]
X_tr, _, y_tr, _ = train_test_split(X, y, test_size=0.25, stratify=y,
                                    random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

def build(num, clf):
    prep = ColumnTransformer([
        ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                          ('s', StandardScaler())]), num),
        ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                          ('o', OneHotEncoder(handle_unknown='ignore'))]), CAT)])
    return Pipeline([('enrich', FunctionTransformer(enrich)),
                     ('prep', prep), ('clf', clf)])

for name, clf in [('logistic', LogisticRegression(max_iter=3000, random_state=42)),
                  ('boosting', HistGradientBoostingClassifier(random_state=42))]:
    plain = cross_val_score(build(NUM, clf), X_tr, y_tr, cv=cv,
                            scoring='roc_auc').mean()
    rich = cross_val_score(build(NUM + DERIVED, clf), X_tr, y_tr, cv=cv,
                           scoring='roc_auc').mean()
    print('%-9s plain %.4f   engineered %.4f   change %+.4f'
          % (name, plain, rich, rich - plain))

logistic  plain 0.8238   engineered 0.8268   change +0.0030
boosting  plain 0.7966   engineered 0.7958   change -0.0009

The features help the linear model more than the trees

That is exactly what you should expect. is_new and rolling_and_calling are thresholds and interactions, and a tree can already discover those by splitting. Handing them to a linear model gives it something it genuinely could not express. Feature engineering is most valuable for the models with the least flexibility.

The honest verdict

The gains this week are small. That is worth saying plainly, because most material on feature engineering implies otherwise. On this dataset the generator used a straightforward weighted sum, so there is not much structure left for a clever feature to expose, and the honest result of measuring carefully is often that an idea did not work.

Report the failures, and be strict about what counts as a win

The engineered pipeline gains about half a point of AUC, and the fold-to-fold spread is five times that. By the standard set on day 1, keep it only if it beats the baseline by more than the spread. This week has not produced a result you could defend. Say so.

You measured roughly a dozen ideas and most changed nothing. That is a normal, healthy ratio. The discipline is not in producing features that work; it is in measuring against a fixed baseline on fixed folds so you can tell the difference, and in discarding what does not earn its place rather than shipping it because you built it.

Your assignment

The dataset generator used payment_method == 'Electronic check' with a coefficient of +0.42. Build a feature that captures an interaction between payment method and something else, and measure it. Then try a feature the generator definitely did not use, a ratio of charges to support calls, say. Record both results, including the one that fails.

Day 7 takeaway

Engineer against a fixed baseline on fixed folds, with every fitted step inside the pipeline. Justified features beat generated ones. Thresholds and interactions help linear models far more than trees, because trees can already find them. And most ideas will not work, measuring is what makes that useful information rather than a wasted fortnight.