Neural Networks From Scratch, Then Keras

Week 11 of 16 · Deep learning · 7 days

Week 11 · Deep learning

Neural Networks From Scratch, Then Keras

Week 11 · Day 1 of 7

From a Neuron to a Network

The perceptron, why XOR broke it, and what a hidden layer actually does

A neural network is not a new idea bolted onto machine learning. It is week 3's X @ w + b, stacked, with a nonlinearity between the layers. Everything else is engineering.

The perceptron

import numpy as np

def step(z):
    return (z > 0).astype(float)

# AND: fire only when both inputs are 1.
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y_and = np.array([0, 0, 0, 1], dtype=float)

w = np.zeros(2)
b = 0.0
for epoch in range(20):
    for xi, target in zip(X, y_and):
        error = target - step(xi @ w + b)
        w += 0.1 * error * xi
        b += 0.1 * error

print('weights %s, bias %.2f' % (w.round(2), b))
print('predictions %s' % step(X @ w + b))
print('targets     %s' % y_and)

weights [0.2 0.1], bias -0.20
predictions [0. 0. 0. 1.]
targets     [0. 0. 0. 1.]

A single neuron, trained by nudging the weights whenever it is wrong. It learned AND in a few passes. It will learn OR just as easily.

And then it fails

import numpy as np

def step(z):
    return (z > 0).astype(float)

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y_xor = np.array([0, 1, 1, 0], dtype=float)   # exclusive or

w, b = np.zeros(2), 0.0
for epoch in range(500):
    for xi, target in zip(X, y_xor):
        error = target - step(xi @ w + b)
        w += 0.1 * error * xi
        b += 0.1 * error

print('after 500 epochs: predictions %s' % step(X @ w + b))
print('targets                       %s' % y_xor)
print('\nit never converges, and it never will')

after 500 epochs: predictions [1. 1. 0. 0.]
targets                       [0. 1. 1. 0.]

it never converges, and it never will

One neuron draws one straight line

XOR needs the output to be 1 in two opposite corners of the square and 0 in the other two. No single straight line separates those. This limitation, published in 1969, stopped neural network research for roughly fifteen years, and the fix, a second layer, was already known. What was missing was an efficient way to train it.

Two layers solve it

import numpy as np

def relu(z):
    return np.maximum(0, z)

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)

# Hand-chosen weights, to show that a solution exists.
W1 = np.array([[1.0, 1.0], [1.0, 1.0]])
b1 = np.array([0.0, -1.0])
W2 = np.array([[1.0], [-2.0]])
b2 = np.array([0.0])

hidden = relu(X @ W1 + b1)
out = hidden @ W2 + b2

print('%12s %18s %10s' % ('input', 'hidden', 'output'))
for xi, h, o in zip(X, hidden, out.ravel()):
    print('%12s %18s %10.1f' % (xi, h, o))
print('\ntarget                                  [0, 1, 1, 0]')

       input             hidden     output
     [0. 0.]            [0. 0.]        0.0
     [0. 1.]            [1. 0.]        1.0
     [1. 0.]            [1. 0.]        1.0
     [1. 1.]            [2. 1.]        0.0

target                                  [0, 1, 1, 0]

The first hidden unit computes “at least one input is on”, the second computes “both are on”, and the output subtracts twice the second from the first. The hidden layer built a representation in which the problem became linearly separable. That is what hidden layers are for.

Why the nonlinearity is essential

import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(5, 4))
W1 = rng.normal(size=(4, 8))
W2 = rng.normal(size=(8, 3))

two_layers = (X @ W1) @ W2          # no activation between them
one_layer = X @ (W1 @ W2)           # collapsed into a single matrix

print('identical:', np.allclose(two_layers, one_layer))
print('so without an activation, depth buys you nothing at all.')

identical: True
so without an activation, depth buys you nothing at all.

Stacked linear layers are one linear layer

Matrix multiplication is associative, so a hundred layers with no activation collapse to a single matrix. The activation function is not a detail or a tuning choice. It is the entire reason depth means anything.

The activation functions

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def tanh(z):
    return np.tanh(z)

def relu(z):
    return np.maximum(0, z)

def leaky_relu(z, a=0.01):
    return np.where(z > 0, z, a * z)

zs = np.array([-3.0, -0.5, 0.0, 0.5, 3.0])
print('%8s %10s %10s %8s %12s' % ('z', 'sigmoid', 'tanh', 'relu', 'leaky relu'))
for z in zs:
    print('%8.1f %10.4f %10.4f %8.1f %12.4f'
          % (z, sigmoid(z), tanh(z), relu(z), leaky_relu(z)))

       z    sigmoid       tanh     relu   leaky relu
    -3.0     0.0474    -0.9951      0.0      -0.0300
    -0.5     0.3775    -0.4621      0.0      -0.0050
     0.0     0.5000     0.0000      0.0       0.0000
     0.5     0.6225     0.4621      0.5       0.5000
     3.0     0.9526     0.9951      3.0       3.0000

Activation	Range	Gradient issue	Use for
Sigmoid	0 to 1	Saturates; derivative caps at 0.25	Binary output layer only
Tanh	−1 to 1	Saturates, but centred on zero	Rarely; recurrent layers sometimes
ReLU	0 upward	Dead units when input stays negative	The default for hidden layers
Leaky ReLU	All reals	Fixes dead units	When ReLU units are dying

Week 3 showed the sigmoid's derivative never exceeding 0.25. Multiply several of those together through a deep network and the gradient reaching the first layer effectively vanishes. ReLU's derivative is exactly 1 wherever the input is positive, which is why replacing sigmoid with ReLU in hidden layers made deep networks trainable.

Day 1 takeaway

A neuron is a dot product and an activation. One neuron draws one straight line and cannot solve XOR; a hidden layer builds a representation in which the problem becomes separable. Without a nonlinearity, stacked layers collapse into one. ReLU is the default for hidden layers because its gradient does not shrink.

Week 11 · Day 2 of 7

Backpropagation, Implemented

The chain rule backwards, gradient checking, and why initialisation scale matters

By Pawan Rama Mali867 words

Yesterday's two-layer network had weights chosen by hand. Today you derive how to learn them, which is the chain rule from week 3, applied layer by layer.

Forward, then backward

Forward pass: push the input through each layer and compute the loss.
Backward pass: work out how much each weight contributed to that loss, from the output backwards.
Update: step every weight against its gradient.

Backpropagation: Applying the chain rule from the loss backwards through the network, reusing each layer's result to compute the layer before it. The reuse is the whole trick: it makes the cost of all the gradients roughly the same as one forward pass, rather than one forward pass per weight.

A complete network, in numpy

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -60, 60)))

rng = np.random.default_rng(0)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0.0], [1.0], [1.0], [0.0]])          # XOR again

# He initialisation: scale by sqrt(2/fan_in), which keeps ReLU signals alive.
W1 = rng.normal(0, np.sqrt(2 / 2), (2, 4))
b1 = np.zeros(4)
W2 = rng.normal(0, np.sqrt(2 / 4), (4, 1))
b2 = np.zeros(1)
lr = 0.5

for epoch in range(1, 4001):
    # ---- forward
    z1 = X @ W1 + b1
    a1 = np.maximum(0, z1)               # relu
    z2 = a1 @ W2 + b2
    out = sigmoid(z2)
    loss = -np.mean(y * np.log(out + 1e-9) + (1 - y) * np.log(1 - out + 1e-9))

    # ---- backward
    # For sigmoid output with log loss, the gradient simplifies to (out - y).
    d2 = (out - y) / len(X)
    dW2 = a1.T @ d2
    db2 = d2.sum(axis=0)
    d1 = (d2 @ W2.T) * (z1 > 0)          # chain rule through the relu
    dW1 = X.T @ d1
    db1 = d1.sum(axis=0)

    # ---- update
    W2 -= lr * dW2; b2 -= lr * db2
    W1 -= lr * dW1; b1 -= lr * db1

    if epoch in (1, 100, 1000, 4000):
        print('epoch %5d  loss %.6f' % (epoch, loss))

print('\npredictions %s' % out.ravel().round(3))
print('targets     %s' % y.ravel())

epoch     1  loss 0.774224
epoch   100  loss 0.210313
epoch  1000  loss 0.004712
epoch  4000  loss 0.000931

predictions [0.003 1.    1.    0.   ]
targets     [0. 1. 1. 0.]

Forty lines, no framework, and it learned XOR from random initial weights. Every deep learning library is this loop, made fast and general.

Why (out - y) and not something messier

The derivative of log loss with respect to the output, multiplied by the derivative of the sigmoid, simplifies to exactly out − y. The two awkward terms cancel. This is not a coincidence. It is why log loss is paired with sigmoid, and cross-entropy with softmax. Pair them differently and you get a messier gradient that trains worse.

Check your gradients numerically

The most useful debugging technique for anyone implementing this: compare your analytical gradient against a numerical one.

import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(20, 3))
y = (X[:, 0] + X[:, 1] > 0).astype(float).reshape(-1, 1)
W = rng.normal(size=(3, 1)) * 0.1

def loss_of(W):
    out = 1 / (1 + np.exp(-(X @ W)))
    return -np.mean(y * np.log(out + 1e-9) + (1 - y) * np.log(1 - out + 1e-9))

out = 1 / (1 + np.exp(-(X @ W)))
analytic = X.T @ (out - y) / len(X)

numeric = np.zeros_like(W)
h = 1e-6
for i in range(W.shape[0]):
    up, down = W.copy(), W.copy()
    up[i, 0] += h
    down[i, 0] -= h
    numeric[i, 0] = (loss_of(up) - loss_of(down)) / (2 * h)

print('analytic %s' % analytic.ravel().round(8))
print('numeric  %s' % numeric.ravel().round(8))
print('max difference %.2e' % np.abs(analytic - numeric).max())

analytic [-0.44631288 -0.20424764 -0.04042924]
numeric  [-0.44631288 -0.20424763 -0.04042924]
max difference 9.48e-10

Anything above about 1e-6 means a bug

A gradient that is subtly wrong still trains, just badly, so this class of bug is invisible without the check. Run it once on a tiny network with random data before you trust any hand-written backpropagation. Frameworks compute gradients automatically, which is the main reason to use one.

Initialisation matters more than it looks

import numpy as np

rng = np.random.default_rng(0)

def signal_through(scale, depth=12, width=100):
    a = rng.normal(size=(200, width))
    for _ in range(depth):
        W = rng.normal(0, scale, (width, width))
        a = np.maximum(0, a @ W)
    return a.std()

width = 100
print('%-26s %16s' % ('initialisation', 'std after 12 layers'))
for name, scale in [('too small (0.01)', 0.01),
                    ('Xavier  sqrt(1/n)', np.sqrt(1 / width)),
                    ('He      sqrt(2/n)', np.sqrt(2 / width)),
                    ('too large (0.3)', 0.3)]:
    print('%-26s %16.6g' % (name, signal_through(scale)))

initialisation             std after 12 layers
too small (0.01)                1.20237e-14
Xavier  sqrt(1/n)                 0.0231427
He      sqrt(2/n)                  0.720329
too large (0.3)                     7450.56

Too small and the signal decays to nothing by the twelfth layer; too large and it explodes. He initialisation, which scales by the square root of two over the number of inputs, keeps the variance roughly constant through ReLU layers, the factor of two compensates for ReLU discarding half the values. Keras uses it by default, which is why you rarely think about it.

Day 2 takeaway

Backpropagation is the chain rule applied backwards with reuse, so all the gradients cost about one forward pass. Log loss with a sigmoid gives the clean gradient out − y. Check hand-written gradients numerically. And initialisation scale decides whether a signal survives depth at all.

Week 11 · Day 3 of 7

Your First Keras Model

The same network in eight lines, and how to read a model summary

By Pawan Rama Mali841 words

You have written a network. Now use a framework, and see what it gives you in exchange for the loop you just wrote.

The same XOR, in Keras

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype='float32')
y = np.array([0, 1, 1, 0], dtype='float32')

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(2,)),
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
model.compile(optimizer=tf.keras.optimizers.Adam(0.05),
              loss='binary_crossentropy', metrics=['accuracy'])

hist = model.fit(X, y, epochs=500, verbose=0)
print('final loss %.6f' % hist.history['loss'][-1])
print('predictions %s' % model.predict(X, verbose=0).ravel().round(3))
print('targets     %s' % y)

final loss 0.477525
predictions [0.334 0.999 0.334 0.334]
targets     [0. 1. 1. 0.]

Eight lines against forty, automatic gradients, and a better optimiser. That is the trade: you give up seeing the arithmetic and you gain everything else.

Reading a model summary

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(15,)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
model.summary()

Model: "sequential"
┌─────────────────────────────────┬────────────────────────┬───────────────┐
│ Layer (type)                    │ Output Shape           │       Param # │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 32)             │           512 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 1)              │            17 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 1,057 (4.13 KB)
 Trainable params: 1,057 (4.13 KB)
 Non-trainable params: 0 (0.00 B)

print('first layer:  15 inputs x 32 units + 32 biases = %d' % (15 * 32 + 32))
print('second layer: 32 inputs x 16 units + 16 biases = %d' % (32 * 16 + 16))
print('output layer: 16 inputs x  1 unit  +  1 bias   = %d' % (16 * 1 + 1))
print('total                                          = %d'
      % (15 * 32 + 32 + 32 * 16 + 16 + 17))

first layer:  15 inputs x 32 units + 32 biases = 512
second layer: 32 inputs x 16 units + 16 biases = 528
output layer: 16 inputs x  1 unit  +  1 bias   = 17
total                                          = 1057

Count the parameters against your row count

1,073 parameters and 2,250 training rows is about two rows per parameter, which is thin. A rough sanity rule for tabular data is that you want at least ten rows per parameter before you start trusting the network, and far more if the signal is weak. This calculation takes ten seconds and saves days.

The three arguments to compile

Argument	Choose	For
`loss`	`binary_crossentropy`	Two classes, sigmoid output
	`categorical_crossentropy`	Several classes, one-hot labels, softmax
	`sparse_categorical_crossentropy`	Several classes, integer labels
	`mse` or `mae`	Regression
`optimizer`	`adam`	The default that almost always works
`metrics`	`AUC`, `accuracy`	Reported but never optimised

A metric is not a loss

Metrics are computed and printed; the loss is what gradients come from. Adding metrics=['AUC'] does not make the network optimise AUC. If you need a different objective you have to change the loss, and most ranking metrics are not differentiable, which is why everyone optimises cross-entropy and then tunes the threshold afterwards, exactly as week 5 did.

On the churn data

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
model.compile(optimizer='adam', loss='binary_crossentropy')
hist = model.fit(Xt, yt, epochs=40, batch_size=64,
                 validation_split=0.2, verbose=0)

proba = model.predict(Xv, verbose=0).ravel()
print('training loss   %.4f' % hist.history['loss'][-1])
print('validation loss %.4f' % hist.history['val_loss'][-1])
print('test ROC AUC    %.4f' % roc_auc_score(yv, proba))
print('\nlogistic regression got 0.8174 on this split.')

training loss   0.4037
validation loss 0.4526
test ROC AUC    0.8050

logistic regression got 0.8174 on this split.

A neural network, several hundred times the parameters, landing in the same place. Hold that thought until week 12 day 7, which takes it seriously.

Day 3 takeaway

Keras gives you automatic gradients, good optimisers and a readable summary. Check the parameter count against your row count before you trust anything. Choose the loss to match the output layer, and remember that metrics are only reported. The loss is what is optimised.

Week 11 · Day 4 of 7

Optimisers and Learning Rates

Momentum, Adam, and what adaptive really means

By Pawan Rama Mali1156 words

Plain gradient descent works. Every optimiser since exists because it works slowly, and because the learning rate is painful to choose.

Momentum

import numpy as np

# A ravine: steep across, shallow along. Gradient descent zig-zags.
def loss(w):
    return 0.5 * (w[0] ** 2 * 20 + w[1] ** 2)

def grad(w):
    return np.array([20 * w[0], w[1]])

for name, momentum in [('plain     ', 0.0), ('momentum  ', 0.9)]:
    w = np.array([1.0, 1.0])
    v = np.zeros(2)
    for _ in range(60):
        v = momentum * v - 0.05 * grad(w)
        w = w + v
    print('%s after 60 steps: loss %.6f  position %s'
          % (name, loss(w), w.round(4)))

plain      after 60 steps: loss 0.001061  position [0.     0.0461]
momentum   after 60 steps: loss 0.009800  position [-0.0305  0.0312]

Momentum accumulates a running average of the gradient. Across the ravine the gradient keeps flipping sign and cancels; along it the gradient is consistent and builds up. The result is less zig-zag and faster progress.

Adam

Adam: Adaptive Moment Estimation. Keeps a running average of the gradient (like momentum) and of its square, then divides the step by the square root of the second. Parameters with consistently large gradients get smaller steps, and rarely-updated ones get larger steps. This per-parameter adaptation is why it works without careful tuning.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

def build():
    return tf.keras.Sequential([
        tf.keras.layers.Input(shape=(Xt.shape[1],)),
        tf.keras.layers.Dense(32, activation='relu'),
        tf.keras.layers.Dense(16, activation='relu'),
        tf.keras.layers.Dense(1, activation='sigmoid'),
    ])

optimisers = {
    'SGD          ': tf.keras.optimizers.SGD(0.01),
    'SGD+momentum ': tf.keras.optimizers.SGD(0.01, momentum=0.9),
    'RMSprop      ': tf.keras.optimizers.RMSprop(0.001),
    'Adam         ': tf.keras.optimizers.Adam(0.001),
}
for name, opt in optimisers.items():
    tf.random.set_seed(42)
    m = build()
    m.compile(optimizer=opt, loss='binary_crossentropy')
    h = m.fit(Xt, yt, epochs=25, batch_size=64, verbose=0)
    print('%s final loss %.4f   test AUC %.4f'
          % (name, h.history['loss'][-1],
             roc_auc_score(yv, m.predict(Xv, verbose=0).ravel())))

SGD           final loss 0.4492   test AUC 0.8100
SGD+momentum  final loss 0.4210   test AUC 0.8157
RMSprop       final loss 0.4193   test AUC 0.8154
Adam          final loss 0.4180   test AUC 0.8137

The learning rate still matters

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

print('%10s %14s %12s' % ('lr', 'final loss', 'test AUC'))
for lr in [0.0001, 0.001, 0.01, 0.1, 1.0]:
    tf.random.set_seed(42)
    m = tf.keras.Sequential([
        tf.keras.layers.Input(shape=(Xt.shape[1],)),
        tf.keras.layers.Dense(32, activation='relu'),
        tf.keras.layers.Dense(1, activation='sigmoid'),
    ])
    m.compile(optimizer=tf.keras.optimizers.Adam(lr),
              loss='binary_crossentropy')
    h = m.fit(Xt, yt, epochs=25, batch_size=64, verbose=0)
    print('%10s %14.4f %12.4f'
          % (lr, h.history['loss'][-1],
             roc_auc_score(yv, m.predict(Xv, verbose=0).ravel())))

        lr     final loss     test AUC
    0.0001         0.5349       0.7365
     0.001         0.4218       0.8121
      0.01         0.4128       0.8099
       0.1         0.4235       0.8041
       1.0         0.5750       0.5519

Adam adapts per parameter, not the global scale

It is often described as removing the need to tune the learning rate. It does not. It removes the need to tune it per parameter. Set it to 1.0 and the model still falls apart. Adam's default of 0.001 is a good starting point and the value most worth trying next is one order of magnitude either side.

Schedules

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score
import numpy as np

tf.random.set_seed(42)
schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.01, decay_steps=100, decay_rate=0.9)

m = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
m.compile(optimizer=tf.keras.optimizers.Adam(schedule),
          loss='binary_crossentropy')
h = m.fit(Xt, yt, epochs=40, batch_size=64, verbose=0)

print('decaying schedule: final loss %.4f   test AUC %.4f'
      % (h.history['loss'][-1],
         roc_auc_score(yv, m.predict(Xv, verbose=0).ravel())))
print('\nlearning rate at step 0, 200, 800:')
for step in [0, 200, 800]:
    print('  %4d  %.6f' % (step, float(schedule(step))))

decaying schedule: final loss 0.4039   test AUC 0.8148

learning rate at step 0, 200, 800:
     0  0.010000
   200  0.008100
   800  0.004305

Start large to cover ground, finish small to settle. A decaying rate is the standard recipe for long training runs, and ReduceLROnPlateau, which day 6 covers, does the same thing reactively, cutting the rate whenever progress stalls.

Day 4 takeaway

Momentum smooths oscillation by averaging gradients; Adam adds a per-parameter scale and is the sensible default at 0.001. It does not remove the need to choose a sensible global learning rate. Decaying the rate over training helps a model settle.

Week 11 · Day 5 of 7

Stopping a Network Memorising

Dropout, weight decay and the early stopping argument everyone forgets

By Pawan Rama Mali1371 words

A network with enough parameters will memorise its training set. Today, the four things that stop it.

Watch it overfit first

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

tf.random.set_seed(42)
big = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
big.compile(optimizer='adam', loss='binary_crossentropy')
h = big.fit(Xt, yt, epochs=60, batch_size=32, validation_split=0.2, verbose=0)

print('parameters: %d for %d training rows' % (big.count_params(), len(Xt)))
print('%8s %12s %12s' % ('epoch', 'train loss', 'val loss'))
for e in [0, 9, 29, 59]:
    print('%8d %12.4f %12.4f'
          % (e + 1, h.history['loss'][e], h.history['val_loss'][e]))
print('\ntest AUC %.4f' % roc_auc_score(yv, big.predict(Xv, verbose=0).ravel()))

parameters: 135937 for 2250 training rows
   epoch   train loss     val loss
       1       0.4699       0.4433
      10       0.3929       0.5115
      30       0.2770       0.8956
      60       0.1921       1.4788

test AUC 0.7818

Training loss falls steadily; validation loss bottoms out early and then climbs. More parameters than training rows, and the network is recording the data rather than learning from it.

Dropout

Dropout: During training, randomly set a fraction of a layer's outputs to zero on every batch. No unit can rely on any particular other unit being present, so the network cannot build fragile co-adapted paths. At prediction time nothing is dropped.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

print('%10s %12s %12s %10s' % ('dropout', 'train loss', 'val loss', 'test AUC'))
for rate in [0.0, 0.2, 0.5]:
    tf.random.set_seed(42)
    m = tf.keras.Sequential([
        tf.keras.layers.Input(shape=(Xt.shape[1],)),
        tf.keras.layers.Dense(256, activation='relu'),
        tf.keras.layers.Dropout(rate),
        tf.keras.layers.Dense(256, activation='relu'),
        tf.keras.layers.Dropout(rate),
        tf.keras.layers.Dense(1, activation='sigmoid'),
    ])
    m.compile(optimizer='adam', loss='binary_crossentropy')
    h = m.fit(Xt, yt, epochs=60, batch_size=32, validation_split=0.2, verbose=0)
    print('%10.1f %12.4f %12.4f %10.4f'
          % (rate, h.history['loss'][-1], h.history['val_loss'][-1],
             roc_auc_score(yv, m.predict(Xv, verbose=0).ravel())))

   dropout   train loss     val loss   test AUC
       0.0       0.2726       1.1482     0.7817
       0.2       0.3219       0.6458     0.7842
       0.5       0.3861       0.5365     0.8009

Dropout is active in training and off in prediction

Keras handles the switch for you. It is worth knowing because it explains something that confuses people: your training loss can be worse than your validation loss with heavy dropout, because the training figure is measured on a crippled network and the validation figure on the whole one.

Weight decay

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

print('%12s %12s %12s %10s' % ('l2', 'train loss', 'val loss', 'test AUC'))
for l2 in [0.0, 0.001, 0.01]:
    tf.random.set_seed(42)
    reg = tf.keras.regularizers.l2(l2) if l2 else None
    m = tf.keras.Sequential([
        tf.keras.layers.Input(shape=(Xt.shape[1],)),
        tf.keras.layers.Dense(256, activation='relu', kernel_regularizer=reg),
        tf.keras.layers.Dense(256, activation='relu', kernel_regularizer=reg),
        tf.keras.layers.Dense(1, activation='sigmoid'),
    ])
    m.compile(optimizer='adam', loss='binary_crossentropy')
    h = m.fit(Xt, yt, epochs=60, batch_size=32, validation_split=0.2, verbose=0)
    print('%12s %12.4f %12.4f %10.4f'
          % (l2, h.history['loss'][-1], h.history['val_loss'][-1],
             roc_auc_score(yv, m.predict(Xv, verbose=0).ravel())))

          l2   train loss     val loss   test AUC
         0.0       0.2686       0.7303     0.7802
       0.001       0.4138       0.5613     0.8106
        0.01       0.4426       0.4747     0.8161

The same L2 penalty as ridge regression in week 4, applied to every weight matrix. The mechanism is identical: a weight has to reduce the loss by more than it adds to the penalty.

Early stopping, which is the one you will always use

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

tf.random.set_seed(42)
m = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
m.compile(optimizer='adam', loss='binary_crossentropy')
stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=8,
                                        restore_best_weights=True)
h = m.fit(Xt, yt, epochs=300, batch_size=32, validation_split=0.2,
          callbacks=[stop], verbose=0)

print('allowed 300 epochs, ran %d' % len(h.history['loss']))
print('best epoch was %d' % (int(min(range(len(h.history['val_loss'])),
                                     key=lambda i: h.history['val_loss'][i])) + 1))
print('test AUC %.4f' % roc_auc_score(yv, m.predict(Xv, verbose=0).ravel()))

allowed 300 epochs, ran 9
best epoch was 1
test AUC 0.8174

restore_best_weights is not the default

Without it, training stops after the patience runs out and you keep the weights from that last epoch, which is several epochs past the best one, and therefore worse. Setting it to True rewinds to the best validation loss. It is a one-word change that most tutorials omit.

Day 5 takeaway

Dropout randomly removes units during training so no path becomes fragile. Weight decay is ridge regression applied to every layer. Early stopping with restore_best_weights=True is the cheapest and most reliable of the three, and the one to reach for first.

Week 11 · Day 6 of 7

Batch Normalisation and Callbacks

Stable layer inputs, reacting to plateaus, and reading a training curve

By Pawan Rama Mali1165 words

Two more techniques that made deep networks practical, and the tools for watching training rather than guessing at it.

Batch normalisation

Batch normalisation: Normalise each layer's outputs to zero mean and unit variance across the batch, then let the network learn a scale and shift. It keeps the distribution entering each layer stable while the weights below it change, which allows higher learning rates and much deeper networks.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score

def build(batchnorm):
    layers = [tf.keras.layers.Input(shape=(Xt.shape[1],))]
    for _ in range(6):
        layers.append(tf.keras.layers.Dense(64, use_bias=not batchnorm))
        if batchnorm:
            layers.append(tf.keras.layers.BatchNormalization())
        layers.append(tf.keras.layers.Activation('relu'))
    layers.append(tf.keras.layers.Dense(1, activation='sigmoid'))
    return tf.keras.Sequential(layers)

print('%14s %10s %14s %10s' % ('', 'lr', 'final loss', 'test AUC'))
for bn in [False, True]:
    for lr in [0.001, 0.02]:
        tf.random.set_seed(42)
        m = build(bn)
        m.compile(optimizer=tf.keras.optimizers.Adam(lr),
                  loss='binary_crossentropy')
        h = m.fit(Xt, yt, epochs=30, batch_size=64, verbose=0)
        print('%14s %10s %14.4f %10.4f'
              % ('batchnorm' if bn else 'plain', lr,
                 h.history['loss'][-1],
                 roc_auc_score(yv, m.predict(Xv, verbose=0).ravel())))

                       lr     final loss   test AUC
         plain      0.001         0.3024     0.7707
         plain       0.02         0.3947     0.8026
     batchnorm      0.001         0.1113     0.7355
     batchnorm       0.02         0.3341     0.7530

Batch normalisation behaves differently in training and prediction

In training it uses the current batch's statistics; at prediction time it uses a running average accumulated during training. That is why predicting on a single row works at all. It also means a very small batch size makes the statistics noisy and the layer unreliable, below about 16 rows per batch, prefer LayerNormalization.

Callbacks: watching, saving and reacting

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import roc_auc_score
import os

tf.random.set_seed(42)
m = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
m.compile(optimizer=tf.keras.optimizers.Adam(0.01),
          loss='binary_crossentropy')

callbacks = [
    tf.keras.callbacks.EarlyStopping(patience=12, restore_best_weights=True),
    tf.keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5, min_lr=1e-5),
    tf.keras.callbacks.ModelCheckpoint('best.keras', save_best_only=True),
]
h = m.fit(Xt, yt, epochs=200, batch_size=64, validation_split=0.2,
          callbacks=callbacks, verbose=0)

print('epochs run: %d of 200' % len(h.history['loss']))
print('learning rate went from %.5f to %.5f'
      % (h.history['learning_rate'][0], h.history['learning_rate'][-1]))
print('checkpoint written:', os.path.exists('best.keras'))
print('test AUC %.4f' % roc_auc_score(yv, m.predict(Xv, verbose=0).ravel()))

epochs run: 15 of 200
learning rate went from 0.01000 to 0.00250
checkpoint written: True
test AUC 0.8134

Callback	Does
`EarlyStopping`	Stops when validation stops improving
`ReduceLROnPlateau`	Cuts the learning rate when progress stalls
`ModelCheckpoint`	Writes the best model to disk as it goes
`TensorBoard`	Logs everything for the live dashboard
`CSVLogger`	Appends per-epoch metrics to a file

Reading the training curve

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
import numpy as np

tf.random.set_seed(42)
m = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
m.compile(optimizer='adam', loss='binary_crossentropy')
h = m.fit(Xt, yt, epochs=50, batch_size=32, validation_split=0.2, verbose=0)

tr = np.array(h.history['loss'])
va = np.array(h.history['val_loss'])
print('%8s %12s %12s %10s' % ('epoch', 'train', 'val', 'gap'))
for e in [0, 4, 9, 19, 34, 49]:
    print('%8d %12.4f %12.4f %10.4f' % (e + 1, tr[e], va[e], va[e] - tr[e]))
print('\nbest validation loss at epoch %d' % (int(va.argmin()) + 1))
print('everything after that is overfitting')

   epoch        train          val        gap
       1       0.5349       0.4457    -0.0892
       5       0.4210       0.4559     0.0349
      10       0.4122       0.4628     0.0505
      20       0.3964       0.4713     0.0749
      35       0.3687       0.5238     0.1551
      50       0.3420       0.5999     0.2579

best validation loss at epoch 2
everything after that is overfitting

Pattern	Means	Do
Both falling, gap small	Still learning	Train longer
Train falls, validation rises	Overfitting	Early stopping, dropout, fewer units
Both flat and high	Underfitting or the rate is too low	Bigger network, higher rate
Loss becomes `nan`	Exploded	Lower the rate; check for infinities in your inputs
Validation wildly jumpy	Batch or validation split too small	Increase both

Day 6 takeaway

Batch normalisation stabilises the distribution entering each layer and lets you use higher learning rates, at the cost of behaving differently in training and prediction. Use callbacks rather than guessing an epoch count: early stopping, a plateau-triggered learning-rate cut, and a checkpoint. Then read the curve: the gap tells you which problem you have.

Week 11 · Day 7 of 7

A Complete Keras Workflow

End to end on churn, and an honest comparison against a straight line

By Pawan Rama Mali1448 words

A complete Keras workflow on the churn problem, and an honest comparison against everything you built in weeks 5 and 6.

Preprocessing belongs to scikit-learn

Keras has preprocessing layers, but a ColumnTransformer is clearer for tabular data and it is what the rest of your pipeline already speaks. Fit it on training data, and keep it. You will need it at prediction time.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
print('features after encoding: %d' % Xt.shape[1])
print('train %s   test %s' % (Xt.shape, Xv.shape))
print('positive rate: train %.3f, test %.3f' % (yt.mean(), yv.mean()))

features after encoding: 15
train (2250, 15)   test (750, 15)
positive rate: train 0.268, test 0.268

Build, train, evaluate

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.metrics import (roc_auc_score, average_precision_score,
                             brier_score_loss)

tf.random.set_seed(42)
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
model.compile(optimizer=tf.keras.optimizers.Adam(0.003),
              loss='binary_crossentropy',
              metrics=[tf.keras.metrics.AUC(name='auc')])

callbacks = [
    tf.keras.callbacks.EarlyStopping(monitor='val_auc', mode='max',
                                     patience=20, restore_best_weights=True),
    tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5,
                                         patience=8, min_lr=1e-5),
]
h = model.fit(Xt, yt, epochs=300, batch_size=64, validation_split=0.2,
              callbacks=callbacks, verbose=0)

proba = model.predict(Xv, verbose=0).ravel()
print('epochs run        %d' % len(h.history['loss']))
print('parameters        %d' % model.count_params())
print('test ROC AUC      %.4f' % roc_auc_score(yv, proba))
print('average precision %.4f' % average_precision_score(yv, proba))
print('Brier score       %.4f' % brier_score_loss(yv, proba))

epochs run        22
parameters        3137
test ROC AUC      0.8152
average precision 0.5824
Brier score       0.1490

Against the alternatives

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import (roc_auc_score, average_precision_score,
                             brier_score_loss)
import time

print('%-16s %10s %14s %10s %10s'
      % ('model', 'ROC AUC', 'avg precision', 'Brier', 'seconds'))

for name, clf in [('logistic', LogisticRegression(max_iter=1000, random_state=42)),
                  ('boosting', HistGradientBoostingClassifier(random_state=42))]:
    t = time.perf_counter()
    clf.fit(Xt, yt)
    el = time.perf_counter() - t
    pr = clf.predict_proba(Xv)[:, 1]
    print('%-16s %10.4f %14.4f %10.4f %10.2f'
          % (name, roc_auc_score(yv, pr), average_precision_score(yv, pr),
             brier_score_loss(yv, pr), el))

tf.random.set_seed(42)
net = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
net.compile(optimizer=tf.keras.optimizers.Adam(0.003), loss='binary_crossentropy')
t = time.perf_counter()
net.fit(Xt, yt, epochs=100, batch_size=64, validation_split=0.2,
        callbacks=[tf.keras.callbacks.EarlyStopping(patience=15,
                                                    restore_best_weights=True)],
        verbose=0)
el = time.perf_counter() - t
pr = net.predict(Xv, verbose=0).ravel()
print('%-16s %10.4f %14.4f %10.4f %10.2f'
      % ('neural net', roc_auc_score(yv, pr), average_precision_score(yv, pr),
         brier_score_loss(yv, pr), el))

model               ROC AUC  avg precision      Brier    seconds
logistic             0.8174         0.5993     0.1475       0.02
boosting             0.7796         0.5259     0.1691       2.44
neural net           0.8135         0.5949     0.1491       3.26

Read the last column before you celebrate

Logistic regression fits in a fraction of a second and matches everything else on this data. The network takes orders of magnitude longer, has thousands of parameters instead of fifteen, cannot be explained to a stakeholder, and needs a GPU to train at any serious scale.

None of that means networks are bad. It means tabular data with three thousand rows is not where they win. Weeks 12 and 13 show what they are actually for.

Saving a model properly

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

prep = ColumnTransformer([
    ('num', Pipeline([('i', SimpleImputer(strategy='median')),
                      ('s', StandardScaler())]), NUM),
    ('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
                      ('o', OneHotEncoder(handle_unknown='ignore',
                                          sparse_output=False))]), CAT),
])

X, y = df[NUM + CAT], df['churned']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)
Xt = prep.fit_transform(X_tr).astype('float32')
Xv = prep.transform(X_te).astype('float32')
yt = y_tr.to_numpy().astype('float32')
yv = y_te.to_numpy().astype('float32')
import joblib
import os
import numpy as np

tf.random.set_seed(42)
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(Xt.shape[1],)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(Xt, yt, epochs=15, batch_size=64, verbose=0)

# BOTH artefacts, or the model is useless: raw columns must be encoded
# exactly as they were in training.
model.save('churn_net.keras')
joblib.dump(prep, 'churn_prep.joblib')

loaded = tf.keras.models.load_model('churn_net.keras')
loaded_prep = joblib.load('churn_prep.joblib')

fresh = loaded_prep.transform(X_te.head(5)).astype('float32')
print('reloaded predictions %s' % loaded.predict(fresh, verbose=0).ravel().round(4))
print('original predictions %s' % model.predict(Xv[:5], verbose=0).ravel().round(4))
print('\nfiles: %s' % sorted(f for f in os.listdir('.')
                             if f.startswith('churn_')))

reloaded predictions [0.6116 0.1354 0.328  0.1717 0.5191]
original predictions [0.6116 0.1354 0.328  0.1717 0.5191]

files: ['churn_by_contract.png', 'churn_net.keras', 'churn_prep.joblib']

The preprocessor is half the model

Save the network alone and you have a function that takes fifteen mystery numbers. The ColumnTransformer holds the imputation medians, the scaling parameters and the category lists, all learned from training data, all required to turn a customer record into those fifteen numbers. Losing it means retraining. Week 15 makes this a single versioned artefact.

Your assignment

Take the network above and make it deliberately too small, one hidden layer of 4 units, then deliberately too large, 512 units in three layers. Record the training loss, validation loss and test AUC for each. You should see underfitting and overfitting from the same code with one number changed, and the middle should not beat logistic regression either.

Day 7 takeaway

The full workflow is: scikit-learn preprocessing, a modest network, Adam, dropout, early stopping on the metric you care about, then evaluate on discrimination and calibration. Save the preprocessor with the model. And compare against logistic regression every time, on tabular data of this size it will hold its own.