Convolutional Networks and Representation Learning

Week 12 of 16 · Deep learning · 7 days

Full curriculum
Week 12 · Deep learning

Convolutional Networks and Representation Learning

Week 12 · Day 1 of 7

Convolution, By Hand

Why dense layers waste weights on images, and what a kernel really does

By 1153 words

Week 11 built networks out of dense layers, where every input touches every unit. That is the right structure when your columns are tenure_months and contract, because those two have no natural neighbours. It is the wrong structure for a picture, where the value of a pixel is almost entirely explained by the pixels immediately around it.

Count the weights before you write the model

# A 200 x 200 colour photograph into one hidden layer of 128 units.
pixels = 200 * 200 * 3
dense_weights = pixels * 128 + 128

# The same first layer done as convolution: 128 filters, each 3x3,
# looking at all 3 colour channels.
conv_weights = 3 * 3 * 3 * 128 + 128

print('flattened input %9d values' % pixels)
print('dense first layer %9d weights' % dense_weights)
print('conv first layer %9d weights' % conv_weights)
print('difference %9.0fx' % (dense_weights / conv_weights))
flattened input 120000 values
dense first layer 15360128 weights
conv first layer 3584 weights
difference 4286x

Four thousand times the weights, and the dense version is the worse model of the two, not merely the more expensive one. A dense layer has to learn what an edge looks like separately in every position it might appear. A convolutional layer learns it once.

Convolution: Slide a small grid of weights (a kernel) across the image, and at each position record the sum of the kernel multiplied by the patch beneath it. The same weights are used at every position, which is what makes the layer small and what makes it care about shape rather than location.

Doing it by hand

import numpy as np
from sklearn.datasets import load_digits

def conv2d(img, kernel):
kh, kw = kernel.shape
out = np.zeros((img.shape[0] - kh + 1, img.shape[1] - kw + 1))
for i in range(out.shape[0]):
for j in range(out.shape[1]):
out[i, j] = float((img[i:i + kh, j:j + kw] * kernel).sum())
return out

def show(a):
chars = ' .:-=+*#%@'
lo, hi = a.min(), a.max()
a = (a - lo) / (hi - lo + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

digits = load_digits()
img = digits.images[0]
print('digit %d, as 8 x 8 greyscale:' % digits.target[0])
show(img)
digit 0, as 8 x 8 greyscale:
-%+
%@*@-
.@. #+
:# ++
-+ ++
:# #=
.@-*#
-%*

Eight by eight is small enough to print, which is the reason this week uses it. Every operation below can be checked with your eyes rather than taken on trust.

import numpy as np
from sklearn.datasets import load_digits

def conv2d(img, kernel):
kh, kw = kernel.shape
out = np.zeros((img.shape[0] - kh + 1, img.shape[1] - kw + 1))
for i in range(out.shape[0]):
for j in range(out.shape[1]):
out[i, j] = float((img[i:i + kh, j:j + kw] * kernel).sum())
return out

def show(a):
chars = ' .:-=+*#%@'
lo, hi = a.min(), a.max()
a = (a - lo) / (hi - lo + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

digits = load_digits()
img = digits.images[0]
vertical = np.array([[-1., 0., 1.],
[-2., 0., 2.],
[-1., 0., 1.]])

response = conv2d(img, vertical)
print('input 8 x 8, kernel 3 x 3, output %d x %d'
% response.shape)
print('\nstrength of vertical edges:')
show(np.abs(response))
input 8 x 8, kernel 3 x 3, output 6 x 6

strength of vertical edges:
%#: .#
@.%=-%
%:%++*
*:**+*
#.+#.%
%::.=*

That kernel is the Sobel operator: negative on the left, positive on the right, so it produces a large number wherever brightness climbs from left to right and roughly zero across a flat region. Its transpose finds horizontal edges instead.

import numpy as np
from sklearn.datasets import load_digits

def conv2d(img, kernel):
kh, kw = kernel.shape
out = np.zeros((img.shape[0] - kh + 1, img.shape[1] - kw + 1))
for i in range(out.shape[0]):
for j in range(out.shape[1]):
out[i, j] = float((img[i:i + kh, j:j + kw] * kernel).sum())
return out

def show(a):
chars = ' .:-=+*#%@'
lo, hi = a.min(), a.max()
a = (a - lo) / (hi - lo + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

digits = load_digits()
img = digits.images[0]
vertical = np.array([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])
kernels = [('vertical edges', vertical),
('horizontal edges', vertical.T),
('blur', np.ones((3, 3)) / 9.0),
('sharpen', np.array([[0., -1., 0.],
[-1., 5., -1.],
[0., -1., 0.]]))]

for name, k in kernels:
print('%s:' % name)
show(np.abs(conv2d(img, k)))
print('')
vertical edges:
%#: .#
@.%=-%
%:%++*
*:**+*
#.+#.%
%::.=*

horizontal edges:
-:===*
.-@@=
-:
..
-**.-
- #=+*

blur:
.*@%+.
-*#+*=
--. ==
:: ==
:---+-
.=#*=

sharpen:
-*#:@
@===-
*:.:-
.:.:--
*--*-
.@:=#-

These four are not learned

Sobel, blur and sharpen are hand-designed kernels that image processing has used since the 1960s. The only thing a convolutional network changes is that it does not accept your kernels. It starts from random numbers and learns the nine weights that reduce the loss. Frequently the first layer of a trained network rediscovers something very close to Sobel, which is a good sign rather than a coincidence.

Why sharing the weights is the whole point

import numpy as np
from sklearn.datasets import load_digits

def conv2d(img, kernel):
kh, kw = kernel.shape
out = np.zeros((img.shape[0] - kh + 1, img.shape[1] - kw + 1))
for i in range(out.shape[0]):
for j in range(out.shape[1]):
out[i, j] = float((img[i:i + kh, j:j + kw] * kernel).sum())
return out

def show(a):
chars = ' .:-=+*#%@'
lo, hi = a.min(), a.max()
a = (a - lo) / (hi - lo + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

digits = load_digits()
img = digits.images[0]
vertical = np.array([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])

shifted = np.roll(img, 2, axis=1) # move the digit two columns right
a = conv2d(img, vertical)
b = conv2d(shifted, vertical)

print('row 3 of the response, original image:')
print(a[3].round(1))
print('row 3 of the response, shifted image:')
print(b[3].round(1))
print('\nshifted response is the original moved by 2:',
np.allclose(b[:, 2:], a[:, :-2]))
row 3 of the response, original image:
[ 39. -18. -38. 38. 30. -38.]
row 3 of the response, shifted image:
[-31. 18. 39. -18. -38. 38.]

shifted response is the original moved by 2: True
Equivariance: Move the input, and the output moves with it. Convolution is equivariant to translation by construction, because the same kernel is applied everywhere. A dense layer has no such property: shift the image two pixels and every weight is now multiplying a different pixel, so as far as the layer is concerned it is looking at an unrelated picture.

Equivariance is not invariance

The response moved, it did not stay the same. Invariance, getting the same answer regardless of position, comes from pooling and from the global averaging at the end of the network, which day 2 covers. Day 4 measures how much of it you actually get, and it is less than most people assume.

Day 1 takeaway

A convolution is a small grid of weights applied at every position. Sharing those weights across positions is what makes the layer thousands of times smaller than a dense one and what lets it recognise a shape it has only ever seen elsewhere in the frame. Everything in a convolutional network is this operation, repeated.
Week 12 · Day 2 of 7

Padding, Pooling and Receptive Fields

The three settings that decide what your network can see

By 1101 words

One convolution is a filter. A network needs three more ideas before it is a network: what happens at the edges, how to shrink the picture, and how far a unit deep in the stack can actually see.

Padding and stride decide the output size

def out_size(n, k, stride=1, pad=0):
return (n + 2 * pad - k) // stride + 1

print('%7s %7s %7s %5s %8s %s'
% ('input', 'kernel', 'stride', 'pad', 'output', 'called'))
rows = [(28, 3, 1, 0, 'valid'), (28, 3, 1, 1, 'same'),
(28, 5, 1, 0, 'valid'), (28, 5, 1, 2, 'same'),
(28, 3, 2, 1, 'same, stride 2'), (8, 3, 1, 0, 'valid'),
(8, 3, 1, 1, 'same')]
for n, k, s, pad, name in rows:
print('%7d %7d %7d %5d %8d %s'
% (n, k, s, pad, out_size(n, k, s, pad), name))
input kernel stride pad output called
28 3 1 0 26 valid
28 3 1 1 28 same
28 5 1 0 24 valid
28 5 1 2 28 same
28 3 2 1 14 same, stride 2
8 3 1 0 6 valid
8 3 1 1 8 same
SettingKerasEffect
No paddingpadding='valid'Output shrinks by kernel − 1 each layer
Pad to keep sizepadding='same'Output matches the input, so you can stack many layers
Stride 2strides=2Halves the resolution; an alternative to pooling

Use same unless you have a reason not to

With valid padding a 3×3 kernel costs you two pixels per layer. Six layers into a 28×28 image and you have a 16×16 map, having thrown away the border for no benefit. same keeps the arithmetic simple and the border intact.

Pooling

Pooling: Divide the feature map into small blocks and replace each block with a single number, usually its maximum. It reduces resolution, which cuts computation, and it discards exactly where within the block the response occurred, which is where a little translation invariance comes from.
import numpy as np
from sklearn.datasets import load_digits

def conv2d(img, kernel):
kh, kw = kernel.shape
out = np.zeros((img.shape[0] - kh + 1, img.shape[1] - kw + 1))
for i in range(out.shape[0]):
for j in range(out.shape[1]):
out[i, j] = float((img[i:i + kh, j:j + kw] * kernel).sum())
return out

def show(a):
chars = ' .:-=+*#%@'
lo, hi = a.min(), a.max()
a = (a - lo) / (hi - lo + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

digits = load_digits()
img = digits.images[0]
def pool(a, k=2, how='max'):
h = a.shape[0] // k * k
w = a.shape[1] // k * k
b = a[:h, :w].reshape(h // k, k, w // k, k)
return b.max(axis=(1, 3)) if how == 'max' else b.mean(axis=(1, 3))

print('original 8 x 8:')
show(img)
print('\nmax pooled to %d x %d:' % pool(img).shape)
show(pool(img))
print('\naverage pooled:')
show(pool(img, how='mean'))
original 8 x 8:
-%+
%@*@-
.@. #+
:# ++
-+ ++
:# #=
.@-*#
-%*

max pooled to 4 x 4:
@@-
:@#+
-##+
.@#

average pooled:
@#.
.*=-
.==-
%*

Max pooling keeps the strongest response and is the usual choice, because a feature map answers "is this pattern here" and the strongest evidence is the informative part. Average pooling blurs, which is occasionally what you want at the very end of a network.

Receptive field: what a deep unit can see

r, jump = 1, 1
print('%-24s %8s %8s' % ('after', 'stride', 'sees'))
stack = [('conv 3x3', 3, 1), ('conv 3x3', 3, 1), ('maxpool 2x2', 2, 2),
('conv 3x3', 3, 1), ('conv 3x3', 3, 1), ('maxpool 2x2', 2, 2),
('conv 3x3', 3, 1)]
for name, k, s in stack:
r = r + (k - 1) * jump
jump = jump * s
print('%-24s %8d %6d px' % (name, jump, r))
after stride sees
conv 3x3 1 3 px
conv 3x3 1 5 px
maxpool 2x2 2 6 px
conv 3x3 2 10 px
conv 3x3 2 14 px
maxpool 2x2 4 16 px
conv 3x3 4 24 px

A single unit in the last layer above responds to a 24×24 region of the input, despite every kernel in the stack being 3×3. This is why deep stacks of small kernels replaced the large kernels of early architectures: two 3×3 layers see as much as one 5×5 layer, with fewer weights and an extra nonlinearity in between.

The same thing in Keras

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(10, activation='softmax'),
])
model.summary(line_length=64)
Model: "sequential"
┌───────────────────────────┬─────────────────────┬────────────┐
│ Layer (type) │ Output Shape │ Param # │
├───────────────────────────┼─────────────────────┼────────────┤
│ conv2d (Conv2D) │ (None, 8, 8, 16) │ 160 │
├───────────────────────────┼─────────────────────┼────────────┤
│ max_pooling2d │ (None, 4, 4, 16) │ 0 │
│ (MaxPooling2D) │ │ │
├───────────────────────────┼─────────────────────┼────────────┤
│ conv2d_1 (Conv2D) │ (None, 4, 4, 32) │ 4,640 │
├───────────────────────────┼─────────────────────┼────────────┤
│ global_average_pooling2d │ (None, 32) │ 0 │
│ (GlobalAveragePooling2D) │ │ │
├───────────────────────────┼─────────────────────┼────────────┤
│ dense (Dense) │ (None, 10) │ 330 │
└───────────────────────────┴─────────────────────┴────────────┘
Total params: 5,130 (20.04 KB)
Trainable params: 5,130 (20.04 KB)
Non-trainable params: 0 (0.00 B)

Read the shape column, not the layer names

The output shape tells you what the network is doing to the picture: 8×8 with 16 channels, then 4×4 after pooling, then 4×4 with 32 channels, then a single vector of 32 once global pooling has averaged each channel over all positions. Resolution falls, channel count rises. That trade is the shape of nearly every convolutional architecture ever published.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
conv = tf.keras.layers.Conv2D(32, 3, padding='same')
conv.build((None, 8, 8, 16))
kernel, bias = conv.get_weights()

print('kernel shape', kernel.shape, ' = (height, width, in, out)')
print('weights %d = 3 * 3 * 16 * 32' % kernel.size)
print('biases %d' % bias.size)
print('total %d' % (kernel.size + bias.size))

flat = tf.keras.layers.Dense(32)
flat.build((None, 8 * 8 * 16))
print('\na dense layer over the same input would need %d'
% sum(w.size for w in flat.get_weights()))
kernel shape (3, 3, 16, 32) = (height, width, in, out)
weights 4608 = 3 * 3 * 16 * 32
biases 32
total 4640

a dense layer over the same input would need 32800

Every filter has one bias, not one per position, another consequence of sharing. The count is independent of image size, which is why the same architecture handles a larger picture without a single new parameter.

Day 2 takeaway

padding='same' keeps the map size so you can stack layers; pooling or a stride of 2 shrinks it deliberately. Resolution falls as channel count rises. Stacking small kernels buys a large receptive field cheaply, and a convolutional layer's parameter count depends on kernel size and channels only, never on the size of the image.
Week 12 · Day 3 of 7

Training a Convolutional Network

Digits, a dense baseline, a matched-budget CNN and an honest floor

By 1849 words

Time to train one. The digits bundled with scikit-learn are 1,797 handwritten numerals at 8×8, which is small enough that everything on this page finishes in seconds and large enough that overfitting is a real risk.

The data

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
print('images ', X.shape, X.dtype)
print('labels ', y.shape, 'values %d to %d' % (y.min(), y.max()))
print('training', X_tr.shape[0], ' test', X_te.shape[0])
print('\nclass balance:')
print(np.bincount(y_tr))
images (1797, 8, 8, 1) float32
labels (1797,) values 0 to 9
training 1347 test 450

class balance:
[133 136 133 137 136 136 136 134 131 135]

Scale the pixels

The raw values run 0 to 16. Dividing by 16 puts them in [0, 1], which matters for the same reason it mattered in week 11: the first layer's gradients are proportional to its inputs, and inputs an order of magnitude larger than the initialised weights make the first few steps wild. Images are the one case where you can usually get away with a constant divisor instead of StandardScaler, because every feature is already on the same scale as every other.

Your numbers will be close, not identical

Every figure on this page came from actually running the code beside it, with tf.random.set_seed fixed. Run it yourself and you should land within a few tenths of a point, but not on the same digits. TensorFlow parallelises across CPU threads, and floating-point addition is not associative, so the order the sums complete in changes the last decimal places and those differences compound over epochs. Seeds control the random draws; they do not control the thread scheduler. This is why the comparisons below average over several runs wherever the margin is small.

A dense baseline first

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
tf.random.set_seed(42)
dense = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax'),
])
dense.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
dense.fit(X_tr, y_tr, epochs=40, batch_size=64, verbose=0)

print('parameters %d' % dense.count_params())
print('test accuracy %.4f' % dense.evaluate(X_te, y_te, verbose=0)[1])
parameters 4810
test accuracy 0.9778
sparse_categorical_crossentropy: use it when your labels are integers (3). Use categorical_crossentropy when they are one-hot rows ([0,0,0,1,0,...]). They compute the same loss; the only difference is the label format, and mixing them up produces a shape error rather than a silently wrong model, which is a mercy.

The convolutional version, at a matched parameter budget

Day 2's architecture: two convolutions, a pooling layer, and global average pooling to collapse each channel to a single number before the classifier. It is the standard modern shape.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
tf.random.set_seed(42)
cnn = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(10, activation='softmax'),
])
cnn.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
cnn.fit(X_tr, y_tr, epochs=40, batch_size=64, verbose=0)

print('parameters %d' % cnn.count_params())
print('test accuracy %.4f' % cnn.evaluate(X_te, y_te, verbose=0)[1])
parameters 5130
test accuracy 0.8578

The convolutional network is worse. Not marginally, by more than eleven points, with more parameters than the dense model it lost to. Before reaching for more epochs or a bigger network, work out what the architecture is throwing away.

Diagnosing it

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
tf.random.set_seed(42)
features = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
])
maps = features.predict(X_te[:1], verbose=0)

print('feature maps reaching the classifier:', maps.shape[1:])
print(' that is %d numbers' % int(np.prod(maps.shape[1:])))
print('flatten passes on %d' % int(np.prod(maps.shape[1:])))
print('global average pooling passes on %d' % maps.shape[-1])
print('\naveraging each 4 x 4 map to one number answers'
' "is this feature present"')
print('and discards "where". At 8 x 8, where is most of the signal.')
feature maps reaching the classifier: (4, 4, 32)
that is 512 numbers
flatten passes on 512
global average pooling passes on 32

averaging each 4 x 4 map to one number answers "is this feature present"
and discards "where". At 8 x 8, where is most of the signal.

Global average pooling is not a free win

It is the right default for a 224×224 photograph, where a cat is a cat wherever it sits in the frame and the final feature map is still 7×7 after many layers of downsampling. On an 8×8 digit that has been pooled once, averaging over the remaining 4×4 grid throws away the spatial arrangement that distinguishes a 6 from a 9. Day 4 shows the same layer being exactly the right choice on a different problem. The layer is not wrong, the match between layer and problem was.

The same network, keeping position

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
tf.random.set_seed(42)
cnn = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.Flatten(), # was GlobalAveragePooling2D
tf.keras.layers.Dense(10, activation='softmax'),
])
cnn.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
cnn.fit(X_tr, y_tr, epochs=40, batch_size=64, verbose=0)

print('parameters %d' % cnn.count_params())
print('test accuracy %.4f' % cnn.evaluate(X_te, y_te, verbose=0)[1])
parameters 9930
test accuracy 0.9733

One layer changed, and the eleven points come back. Note also what the network has not bought: it is now level with the dense baseline rather than ahead of it. At this resolution there is very little spatial structure left for convolution to exploit. Day 4 constructs the case where there is.

And a model with no network in it at all

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
import time

flat_tr = X_tr.reshape(len(X_tr), -1)
flat_te = X_te.reshape(len(X_te), -1)

for name, model in [('logistic regression', LogisticRegression(max_iter=2000)),
('gradient boosting', HistGradientBoostingClassifier(random_state=42))]:
t = time.time()
model.fit(flat_tr, y_tr)
print('%-20s %.4f fitted in %.1fs'
% (name, model.score(flat_te, y_te), time.time() - t))
logistic regression 0.9622 fitted in 0.1s
gradient boosting 0.9600 fitted in 4.5s

Where the mistakes are

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.metrics import confusion_matrix

tf.random.set_seed(42)
cnn = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(10, activation='softmax'),
])
cnn.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
cnn.fit(X_tr, y_tr, epochs=40, batch_size=64, verbose=0)

pred = cnn.predict(X_te, verbose=0).argmax(axis=1)
cm = confusion_matrix(y_te, pred)
print(' ' + ''.join('%4d' % i for i in range(10)))
for i, row in enumerate(cm):
print('%2d ' % i + ''.join('%4d' % v for v in row))

print('\nthe confusions that happened more than once:')
for i in range(10):
for j in range(10):
if i != j and cm[i, j] > 1:
print(' a %d read as a %d, %d times' % (i, j, cm[i, j]))
0 1 2 3 4 5 6 7 8 9
0 44 0 0 0 1 0 0 0 0 0
1 0 46 0 0 0 0 0 0 0 0
2 0 0 44 0 0 0 0 0 0 0
3 0 0 0 45 0 1 0 0 0 0
4 0 0 0 0 45 0 0 0 0 0
5 0 0 0 0 0 45 0 0 0 1
6 0 0 0 0 0 0 44 0 1 0
7 0 0 0 0 0 0 0 45 0 0
8 0 2 0 0 0 0 0 1 40 0
9 0 0 0 0 0 0 0 0 0 45

the confusions that happened more than once:
a 8 read as a 1, 2 times

A confusion matrix is the first thing to read, always

An accuracy figure tells you how often the model is wrong. The confusion matrix tells you what it is wrong about, and the pattern is almost always interpretable, digits that share a shape at this resolution get muddled, and digits that do not, do not. If the errors look random, suspect a bug in the label alignment before you suspect the model.

Day 3 takeaway

Use sparse_categorical_crossentropy with integer labels and a softmax output sized to your class count. When a convolutional network underperforms a dense one, suspect the layer that reduces the feature maps before you suspect the convolutions. And always fit a non-network baseline on the flattened pixels, at small resolutions it is competitive, and knowing that is worth more than another epoch of tuning.
Week 12 · Day 4 of 7

Augmentation and Invariance

Where a dense network collapses, and how to buy back what it lost

By 1864 words

Day 3 was a fair fight on a dataset where every digit is already centred in the frame. Real images are not centred, and that is where the difference between the two architectures stops being academic.

A dataset built to make the point

import numpy as np

def draw(kind, cx, cy, r, size=20):
yy, xx = np.mgrid[0:size, 0:size]
dx, dy = xx - cx, yy - cy
if kind == 0: # ring
m = np.abs(np.hypot(dx, dy) - r) < 1.0
elif kind == 1: # square outline
m = np.abs(np.maximum(np.abs(dx), np.abs(dy)) - r) < 1.0
else: # cross
m = (((np.abs(dx) < 1.0) & (np.abs(dy) <= r))
| ((np.abs(dy) < 1.0) & (np.abs(dx) <= r)))
return m.astype('float32')

def make(n, seed, jitter):
rng = np.random.default_rng(seed)
X = np.zeros((n, 20, 20), dtype='float32')
y = rng.integers(0, 3, size=n)
for i, kind in enumerate(y):
r = int(rng.integers(4, 7))
if jitter:
cx = int(rng.integers(r + 1, 20 - r))
cy = int(rng.integers(r + 1, 20 - r))
else:
cx = cy = 10
X[i] = draw(kind, cx, cy, r)
X += rng.normal(0, 0.05, X.shape).astype('float32')
return X[..., np.newaxis], y.astype('int32')
X_centre, y_centre = make(1500, seed=0, jitter=False)
X_jitter, y_jitter = make(500, seed=1, jitter=True)

print('centred training set', X_centre.shape)
print('jittered test set ', X_jitter.shape)
print('classes: 0 ring, 1 square, 2 cross')

chars = ' .:-=+*#%@'
def show(a):
a = (a - a.min()) / (a.max() - a.min() + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

print('\na centred training example, class %d:' % y_centre[0])
show(X_centre[0, :, :, 0])
print('\na jittered test example, class %d:' % y_jitter[0])
show(X_jitter[0, :, :, 0])
centred training set (1500, 20, 20, 1)
jittered test set (500, 20, 20, 1)
classes: 0 ring, 1 square, 2 cross

a centred training example, class 2:
....... ........ ...
.. . .. ... . .....
.... ... ... ..:...
.... .......... ....
.. . ...%...... ..
.... ....@..: ... .
... . ....@ .... :
.: .: .. :%.. ...
....... ..@. . ..:..
.... ..%. ... ..
:. %@@@@%%@%@%@%...
.. .. .@...... .
.... ... @..... ..
.. ..... @....... .
... .... @. :.. .
. .. .. .@.......
. .....%. ..:.. .
. ........... .....
. .. :... ....... .
... . .. ... ...

a jittered test example, class 1:
..... :...... . .
.... . .. . ...
:.... .... .....
. . ... . .. ...
. ..... ... .. ..
. .. . ... .. .. .
..... .%%@@@%@%%...
.... ..@.. ....@.. .
. .....@. .. .%.
. ... @.. . .@ ...
: . . .@ ....@ .
. . ..%. .. %.. .
..... @.:. . @. .
......% . ...@....
.. ...@%@@@@%@%....
. . ... . ... ....
....... . .... .....
... ...... . ...
.. ... . . .. .. ..
.:. .. .: .....

Train on centred, test on moved

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np

def draw(kind, cx, cy, r, size=20):
yy, xx = np.mgrid[0:size, 0:size]
dx, dy = xx - cx, yy - cy
if kind == 0: # ring
m = np.abs(np.hypot(dx, dy) - r) < 1.0
elif kind == 1: # square outline
m = np.abs(np.maximum(np.abs(dx), np.abs(dy)) - r) < 1.0
else: # cross
m = (((np.abs(dx) < 1.0) & (np.abs(dy) <= r))
| ((np.abs(dy) < 1.0) & (np.abs(dx) <= r)))
return m.astype('float32')

def make(n, seed, jitter):
rng = np.random.default_rng(seed)
X = np.zeros((n, 20, 20), dtype='float32')
y = rng.integers(0, 3, size=n)
for i, kind in enumerate(y):
r = int(rng.integers(4, 7))
if jitter:
cx = int(rng.integers(r + 1, 20 - r))
cy = int(rng.integers(r + 1, 20 - r))
else:
cx = cy = 10
X[i] = draw(kind, cx, cy, r)
X += rng.normal(0, 0.05, X.shape).astype('float32')
return X[..., np.newaxis], y.astype('int32')
X_tr, y_tr = make(1500, seed=0, jitter=False)
X_same, y_same = make(500, seed=2, jitter=False)
X_moved, y_moved = make(500, seed=1, jitter=True)

def dense_net():
return tf.keras.Sequential([
tf.keras.layers.Input(shape=(20, 20, 1)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(3, activation='softmax')])

def conv_net():
return tf.keras.Sequential([
tf.keras.layers.Input(shape=(20, 20, 1)),
tf.keras.layers.Conv2D(8, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(3, activation='softmax')])

print('%-12s %8s %14s %14s' % ('', 'params', 'centred test', 'moved test'))
for name, build in [('dense', dense_net), ('conv', conv_net)]:
tf.random.set_seed(42)
m = build()
m.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
m.fit(X_tr, y_tr, epochs=30, batch_size=64, verbose=0)
print('%-12s %8d %14.4f %14.4f'
% (name, m.count_params(),
m.evaluate(X_same, y_same, verbose=0)[1],
m.evaluate(X_moved, y_moved, verbose=0)[1]))
params centred test moved test
dense 25859 1.0000 0.3880
conv 1299 1.0000 0.9840

Both networks are perfect on centred shapes. Move the shape and the dense network falls to barely above the one-in-three you would get by guessing, while the convolutional network, with twenty times fewer parameters, barely notices.

The layer day 3 blamed is the hero here

This network ends in GlobalAveragePooling2D, the same layer that cost eleven points on the digits. Here it is exactly right: "is there a ring in this picture" is a question whose answer must not depend on where the ring is, and averaging over positions is precisely how you enforce that. Discarding position is a feature when position is noise and a bug when position is signal. Nothing about the layer changed, the problem did.

Augmentation: move the training data instead

Data augmentation: Apply label-preserving transformations to the training images, shift, rotate, flip, zoom, adjust brightness, so the model sees many versions of each example and cannot rely on any one of them. It is regularisation, and unlike dropout it encodes a specific belief about what should not change the answer.
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np

def draw(kind, cx, cy, r, size=20):
yy, xx = np.mgrid[0:size, 0:size]
dx, dy = xx - cx, yy - cy
if kind == 0: # ring
m = np.abs(np.hypot(dx, dy) - r) < 1.0
elif kind == 1: # square outline
m = np.abs(np.maximum(np.abs(dx), np.abs(dy)) - r) < 1.0
else: # cross
m = (((np.abs(dx) < 1.0) & (np.abs(dy) <= r))
| ((np.abs(dy) < 1.0) & (np.abs(dx) <= r)))
return m.astype('float32')

def make(n, seed, jitter):
rng = np.random.default_rng(seed)
X = np.zeros((n, 20, 20), dtype='float32')
y = rng.integers(0, 3, size=n)
for i, kind in enumerate(y):
r = int(rng.integers(4, 7))
if jitter:
cx = int(rng.integers(r + 1, 20 - r))
cy = int(rng.integers(r + 1, 20 - r))
else:
cx = cy = 10
X[i] = draw(kind, cx, cy, r)
X += rng.normal(0, 0.05, X.shape).astype('float32')
return X[..., np.newaxis], y.astype('int32')
X_tr, y_tr = make(1500, seed=0, jitter=False)

augment = tf.keras.Sequential([
tf.keras.layers.RandomTranslation(0.2, 0.2, fill_mode='constant'),
tf.keras.layers.RandomRotation(0.05, fill_mode='constant'),
])

batch = tf.convert_to_tensor(X_tr[:1])
chars = ' .:-=+*#%@'
def show(a):
a = (a - a.min()) / (a.max() - a.min() + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

print('original, class %d:' % y_tr[0])
show(X_tr[0, :, :, 0])
for i in range(2):
print('\naugmented copy %d:' % (i + 1))
show(augment(batch, training=True).numpy()[0, :, :, 0])
original, class 2:
....... ........ ...
.. . .. ... . .....
.... ... ... ..:...
.... .......... ....
.. . ...%...... ..
.... ....@..: ... .
... . ....@ .... :
.: .: .. :%.. ...
....... ..@. . ..:..
.... ..%. ... ..
:. %@@@@%%@%@%@%...
.. .. .@...... .
.... ... @..... ..
.. ..... @....... .
... .... @. :.. .
. .. .. .@.......
. .....%. ..:.. .
. ........... .....
. .. :... ....... .
... . .. ... ...

augmented copy 1:
.
... .
. ++...
. .+*..
.. *+.
.. .. .**.
**
:-:::-#*:::::
=%@@@%@@%%%%%
.**:::--
**.
. *+ .
. ** ..
.*+ ..
.. .+= ..
. .. ... .
...
.



augmented copy 2:



..
. .
.. . .. .
... .-=... .
... :**...
. . :#= .
. -@: .
.-:. .=@
-#*+==*%. .
-=*%%@%+=--:..
-%##%##+:
. :@:..:=+.
....-@... .
.+#...
.. .*+ ..
. ... :*- ...
.. .:. ...

training=True is not optional

Augmentation layers are deliberately inert at prediction time. You do not want your test images randomly shifted. Inside model.fit Keras sets the flag for you. Calling the layer yourself without it returns the input untouched, which looks exactly like a broken augmentation pipeline and is the single most common confusion here.

Does it actually help?

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np

def draw(kind, cx, cy, r, size=20):
yy, xx = np.mgrid[0:size, 0:size]
dx, dy = xx - cx, yy - cy
if kind == 0: # ring
m = np.abs(np.hypot(dx, dy) - r) < 1.0
elif kind == 1: # square outline
m = np.abs(np.maximum(np.abs(dx), np.abs(dy)) - r) < 1.0
else: # cross
m = (((np.abs(dx) < 1.0) & (np.abs(dy) <= r))
| ((np.abs(dy) < 1.0) & (np.abs(dx) <= r)))
return m.astype('float32')

def make(n, seed, jitter):
rng = np.random.default_rng(seed)
X = np.zeros((n, 20, 20), dtype='float32')
y = rng.integers(0, 3, size=n)
for i, kind in enumerate(y):
r = int(rng.integers(4, 7))
if jitter:
cx = int(rng.integers(r + 1, 20 - r))
cy = int(rng.integers(r + 1, 20 - r))
else:
cx = cy = 10
X[i] = draw(kind, cx, cy, r)
X += rng.normal(0, 0.05, X.shape).astype('float32')
return X[..., np.newaxis], y.astype('int32')
X_tr, y_tr = make(1500, seed=0, jitter=False)
X_moved, y_moved = make(500, seed=1, jitter=True)

def conv_net(with_augmentation):
layers = [tf.keras.layers.Input(shape=(20, 20, 1))]
if with_augmentation:
layers += [tf.keras.layers.RandomTranslation(0.25, 0.25,
fill_mode='constant')]
layers += [
tf.keras.layers.Conv2D(8, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(3, activation='softmax')]
return tf.keras.Sequential(layers)

# One run proves nothing at this margin, so average over three seeds.
print('%-22s %10s %18s' % ('', 'mean', 'runs'))
for label, aug in [('no augmentation', False), ('with translation', True)]:
scores = []
for seed in range(3):
tf.random.set_seed(seed)
m = conv_net(aug)
m.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
m.fit(X_tr, y_tr, epochs=30, batch_size=64, verbose=0)
scores.append(m.evaluate(X_moved, y_moved, verbose=0)[1])
print('%-22s %10.4f %18s'
% (label, float(np.mean(scores)),
' '.join('%.3f' % s for s in scores)))
mean runs
no augmentation 0.9713 0.978 0.970 0.966
with translation 0.9820 0.994 0.960 0.992

About a point on average, and now read the individual runs, because they are the more instructive column. Augmentation won twice and lost once. A single run of this experiment could have shown you a two-point gain, no gain, or a loss, depending entirely on which seed you happened to use.

One run is not a measurement

Five hundred test images means one point of accuracy is five images. Any comparison at that margin needs several runs and a look at the spread, or you are reporting the seed rather than the technique. This is the same discipline week 7 applied with cross-validation, and it does not stop being necessary because the model is a neural network. It becomes more necessary, because networks have a second source of randomness in their initialisation.

The effect here is real but modest, which is what augmentation looks like on a problem whose architecture already handles the transformation. The large gains come from augmenting a model with no built-in defence. The dense network in the previous table, which had none at all.

AugmentationSafe forWrong for
Horizontal flipPhotographs of objects, animalsText, digits, anything where b and d differ
Vertical flipSatellite and microscope imageryAlmost every photograph of the world
Rotation ±10°Most natural imagesDigits at large angles, 6 becomes 9
Brightness, contrastAnything photographed under varying lightMedical scans where intensity is the measurement
Random cropLarge images with a marginSmall images where the subject fills the frame

Augmentation encodes an assumption, and it can be false

Flipping a chest X-ray horizontally produces an image of a patient with situs inversus, a rare condition. Train on flipped X-rays and you have taught the model that heart position carries no information. Every augmentation is a claim that a transformation does not change the label. Check the claim against the domain, not against the accuracy number.

Day 4 takeaway

A dense network that scores well on centred images can collapse when the subject moves; convolution plus pooling degrades far more gracefully. Augmentation is the cheapest way to buy invariance you do not otherwise have, but only apply transformations that genuinely preserve the label in your domain.
Week 12 · Day 5 of 7

Transfer Learning

Reusing learned features, freezing, fine-tuning, and reading the filters

By 2114 words

Nobody trains an image model from random weights any more, unless they have millions of labelled examples. They take a network trained on something large and reuse its early layers, because the features those layers learned, edges, corners, textures, are not specific to the task it was trained on.

Transfer learning: Take a model trained on a large dataset, keep its learned feature extractor, and train only a new output layer on your own smaller dataset. Optionally unfreeze the top of the extractor afterwards and continue at a much lower learning rate.

Why this page pretrains its own base

The usual demonstration downloads ImageNet weights for MobileNetV2 or ResNet50, and in your own work that is exactly what you should do, one line, tf.keras.applications.MobileNetV2(weights='imagenet', include_top=False). It needs a network connection, so instead this page pretrains a small base on digits 0 to 4 and transfers it to digits 5 to 9. The mechanics are identical, the numbers are real, and they reproduce offline.

Step 1: train the base on the source task

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
src = y_tr < 5 # digits 0-4, the source task

tf.random.set_seed(42)
base = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
], name='features')

source = tf.keras.Sequential([base,
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(5, activation='softmax')])
source.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
source.fit(X_tr[src], y_tr[src], epochs=40, batch_size=64, verbose=0)

mask = y_te < 5
print('source task (digits 0-4) test accuracy %.4f'
% source.evaluate(X_te[mask], y_te[mask], verbose=0)[1])
base.save_weights('digit_base.weights.h5')
print('feature extractor saved, %d parameters' % base.count_params())
source task (digits 0-4) test accuracy 0.9956
feature extractor saved, 4800 parameters

Step 2: transfer it, with very few labels

The target task is digits 5 to 9, relabelled 0 to 4, with fifteen training examples of each. Two models: one starting from random weights, one reusing the frozen base. Three seeds each, because a seventy-five-row training set produces enough run-to-run variation to show whatever you were hoping to see.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
def make_base():
return tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
], name='features')

def head(base, trainable):
base.trainable = trainable
m = tf.keras.Sequential([base, tf.keras.layers.Flatten(),
tf.keras.layers.Dense(5, activation='softmax')])
m.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return m

src = y_tr < 5
tgt = np.where(y_tr >= 5)[0]
mask = y_te >= 5
Xv, yv = X_te[mask], y_te[mask] - 5

scratch_runs, frozen_runs = [], []
for seed in range(3):
rng = np.random.default_rng(seed)
few = np.concatenate([rng.permutation(tgt[y_tr[tgt] == c])[:15]
for c in range(5, 10)])
Xf, yf = X_tr[few], y_tr[few] - 5

tf.random.set_seed(100 + seed) # pretrain on 0-4
pre = make_base()
head(pre, True).fit(X_tr[src], y_tr[src], epochs=40, batch_size=64,
verbose=0)

tf.random.set_seed(seed)
a = head(make_base(), True)
a.fit(Xf, yf, epochs=60, batch_size=16, verbose=0)
scratch_runs.append(a.evaluate(Xv, yv, verbose=0)[1])

tf.random.set_seed(seed)
b = head(pre, False)
b.fit(Xf, yf, epochs=60, batch_size=16, verbose=0)
frozen_runs.append(b.evaluate(Xv, yv, verbose=0)[1])

print('target training examples: 75')
print('%-28s %9s %18s' % ('', 'mean', 'runs'))
for name, runs in [('from scratch', scratch_runs),
('frozen pretrained features', frozen_runs)]:
print('%-28s %9.4f %18s'
% (name, float(np.mean(runs)),
' '.join('%.3f' % r for r in runs)))
print('\ntrainable parameters: scratch %d, frozen %d'
% (sum(int(np.prod(w.shape)) for w in a.trainable_weights),
sum(int(np.prod(w.shape)) for w in b.trainable_weights)))
target training examples: 75
mean runs
from scratch 0.9509 0.955 0.960 0.938
frozen pretrained features 0.9435 0.960 0.946 0.924

trainable parameters: scratch 7365, frozen 2565

Transfer learning lost, and the honest thing is to say so

Every textbook example of this has the pretrained model winning comfortably. Here it is behind on average, and it wins one run of three, which is to say the pretrained features bought nothing detectable. That is not a bug in the code above: the mechanics are exactly what you would write against ImageNet weights. It is a fact about the source task, and it is worth more than a rigged win.

Why it lost

Transfer only helps when the features the source task learned are better than the features the target's own data can learn. The source here is 675 images of five digit classes. That is not a lot of world knowledge to bring. Growing it should narrow the gap, and it does.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
def make_base():
return tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
], name='features')

def head(base, trainable):
base.trainable = trainable
m = tf.keras.Sequential([base, tf.keras.layers.Flatten(),
tf.keras.layers.Dense(5, activation='softmax')])
m.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return m

src = np.where(y_tr < 5)[0]
tgt = np.where(y_tr >= 5)[0]
mask = y_te >= 5
Xv, yv = X_te[mask], y_te[mask] - 5

print('%15s %12s' % ('source images', 'frozen acc'))
for n_src in [50, 150, 400, len(src)]:
runs = []
for seed in range(3):
rng = np.random.default_rng(seed)
few = np.concatenate([rng.permutation(tgt[y_tr[tgt] == c])[:15]
for c in range(5, 10)])
tf.random.set_seed(100 + seed)
pre = make_base()
head(pre, True).fit(X_tr[rng.permutation(src)[:n_src]],
y_tr[rng.permutation(src)[:n_src]],
epochs=40, batch_size=64, verbose=0)
tf.random.set_seed(seed)
m = head(pre, False)
m.fit(X_tr[few], y_tr[few] - 5, epochs=60, batch_size=16, verbose=0)
runs.append(m.evaluate(Xv, yv, verbose=0)[1])
print('%15d %12.4f' % (n_src, float(np.mean(runs))))
source images frozen acc
50 0.9196
150 0.9271
400 0.9360
675 0.9390

This is the precondition, stated as a number

The frozen features get steadily better as the source task grows, which is the mechanism working exactly as advertised. It simply has not got far enough by 675 images to beat a network trained directly on the target. Now extrapolate: ImageNet is 1.2 million images across a thousand classes. That is why transfer learning is the default in real work, and why this page could not demonstrate it honestly without a network connection.

The practical rule: transfer when your source is far larger and richer than your target. When it is not, you are just borrowing somebody else's constraints.

Step 3: fine-tuning

Once the new head has stopped being random, you can unfreeze the base and continue at a learning rate one to two orders of magnitude smaller. The order matters: unfreeze while the head is still random and the large gradients coming back from it will destroy the pretrained weights before they are ever used.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
def make_base():
return tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
], name='features')

tf.random.set_seed(42)
base = make_base()
src = y_tr < 5
warm = tf.keras.Sequential([base, tf.keras.layers.Flatten(),
tf.keras.layers.Dense(5, activation='softmax')])
warm.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
warm.fit(X_tr[src], y_tr[src], epochs=40, batch_size=64, verbose=0)

rng = np.random.default_rng(0)
tgt = np.where(y_tr >= 5)[0]
few = np.concatenate([rng.permutation(tgt[y_tr[tgt] == c])[:15]
for c in range(5, 10)])
Xf, yf = X_tr[few], y_tr[few] - 5
mask = y_te >= 5
Xv, yv = X_te[mask], y_te[mask] - 5

tf.random.set_seed(7)
base.trainable = False
model = tf.keras.Sequential([base, tf.keras.layers.Flatten(),
tf.keras.layers.Dense(5, activation='softmax')])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(Xf, yf, epochs=60, batch_size=16, verbose=0)
print('after training the head only %.4f'
% model.evaluate(Xv, yv, verbose=0)[1])
before = model.predict(Xv, verbose=0).argmax(1)
kernel_before = base.get_weights()[0].copy()

base.trainable = True
model.compile(optimizer=tf.keras.optimizers.Adam(1e-4),
loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(Xf, yf, epochs=40, batch_size=16, verbose=0)
print('after fine-tuning at 1e-4 %.4f'
% model.evaluate(Xv, yv, verbose=0)[1])

after = model.predict(Xv, verbose=0).argmax(1)
print('\npredictions that changed: %d of %d' % ((before != after).sum(),
len(yv)))
print('largest weight change in the base: %.6f'
% np.abs(base.get_weights()[0] - kernel_before).max())
after training the head only 0.9509
after fine-tuning at 1e-4 0.9554

predictions that changed: 1 of 224
largest weight change in the base: 0.016407

Fine-tuning at 1e-4 for forty epochs on seventy-five images is a nudge, not a retraining. The largest weight in the base moved by about a hundredth, and exactly one test prediction changed. That is the intended behaviour: if unfreezing moves your accuracy dramatically, the rate is too high and you are erasing the thing you came for.

Recompile after changing trainable

Keras decides which weights the optimiser will touch when you compile. Flip base.trainable and skip the recompile and nothing changes, the model trains exactly as it did before, and you spend an afternoon wondering why fine-tuning does nothing.

What the first layer actually learned

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
tf.random.set_seed(42)
m = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(8, 3, padding='same', activation='relu',
name='first'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(10, activation='softmax'),
])
m.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
m.fit(X_tr, y_tr, epochs=40, batch_size=64, verbose=0)

k = m.get_layer('first').get_weights()[0] # (3, 3, 1, 8)
print('four of the eight learned 3x3 kernels:')
for f in range(4):
w = k[:, :, 0, f]
print('\nfilter %d (sum %+.2f)' % (f, w.sum()))
for row in w:
print(' ' + ' '.join('%+.2f' % v for v in row))
four of the eight learned 3x3 kernels:

filter 0 (sum +0.11)
+0.62 +0.57 +0.47
+0.28 -0.01 +0.18
-0.53 -0.82 -0.65

filter 1 (sum -0.09)
+0.43 -0.57 +0.52
-0.60 -0.69 +0.23
-0.12 +0.60 +0.13

filter 2 (sum +0.72)
+0.62 -0.48 -0.84
+0.42 +0.14 -0.24
+0.09 +0.76 +0.25

filter 3 (sum +1.59)
+0.26 +0.55 +0.35
+0.53 +0.47 +0.32
+0.22 -0.44 -0.67

Read the signs rather than the magnitudes. A filter with negatives on one side and positives on the other is an edge detector oriented along that axis; one that is positive in the middle and negative around it is a blob detector. Nobody told the network to learn these. They are what minimising the loss produced, and they are the reason the features transfer to a task the base never saw.

Day 5 takeaway

Reuse a pretrained feature extractor whenever you have fewer than tens of thousands of labelled images and a genuinely large source model to borrow from. That second condition is the one this page could not meet, and the one most tutorials never mention. Train the new head with the base frozen, then optionally unfreeze at a much lower rate, recompiling after every change to trainable.
Week 12 · Day 6 of 7

Autoencoders and Learned Representations

Compression without labels, denoising, and anomaly scores from error

By 1699 words

Week 10 ended by finding unusual customers with PCA: rebuild each row from two components and flag whichever rows rebuild worst. An autoencoder is that idea with a network in place of the linear projection, and it is the bridge between supervised deep learning and everything in weeks 9 and 10.

Autoencoder: A network trained to reproduce its own input through a bottleneck narrower than the input. Because the middle layer cannot hold everything, training forces it to keep whatever structure is most common and discard the rest. No labels are involved.

An autoencoder on the churn data

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

prep = ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore',
sparse_output=False))]), CAT),
])
Z = prep.fit_transform(df).astype('float32')
print('encoded width %d columns' % Z.shape[1])

tf.random.set_seed(42)
auto = tf.keras.Sequential([
tf.keras.layers.Input(shape=(Z.shape[1],)),
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(2, activation='relu', name='bottleneck'),
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(Z.shape[1]),
])
auto.compile(optimizer='adam', loss='mse')
auto.fit(Z, Z, epochs=120, batch_size=64, verbose=0)

error = ((auto.predict(Z, verbose=0) - Z) ** 2).mean(axis=1)
print('reconstruction error: median %.4f, 99th %.4f, max %.4f'
% (np.median(error), np.percentile(error, 99), error.max()))

worst = np.argsort(error)[:-1][:5]
print('\nthe five customers the network cannot rebuild:')
print(df.iloc[worst][['customer_id'] + NUM].to_string(index=False))
encoded width 15 columns
reconstruction error: median 0.1285, 99th 0.2909, max 29.5497

the five customers the network cannot rebuild:
customer_id tenure_months support_calls monthly_charges
C01643 1 0 50.61
C00487 6 0 48.84
C00988 9 0 49.83
C00451 3 0 57.29
C02559 8 0 48.30

Same idea as PCA, one important difference

PCA can only rebuild a row as a weighted sum of fixed directions. An autoencoder can bend, so in principle it captures structure that is not a flat subspace, for instance that high charges are normal on a fibre contract and unusual otherwise, which no single linear direction expresses. The cost is that you now have a network to train, a seed to fix, and no equivalent of explained variance to report.

Note that the customer at the top of that list is the one with 97 support calls, the same row week 10 found with PCA. Two quite different methods agreeing on the most unusual customer is the reassuring outcome.

Is the flexibility earning anything?

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
import pandas as pd

df = pd.read_csv('customers.csv').drop_duplicates().copy()
df['contract'] = df['contract'].str.strip().str.title()
df['total_charges'] = pd.to_numeric(df['total_charges'], errors='coerce')

NUM = ['tenure_months', 'support_calls', 'monthly_charges']
CAT = ['contract', 'internet_service', 'payment_method', 'has_dependents']
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.decomposition import PCA

prep = ColumnTransformer([
('num', Pipeline([('i', SimpleImputer(strategy='median')),
('s', StandardScaler())]), NUM),
('cat', Pipeline([('i', SimpleImputer(strategy='most_frequent')),
('o', OneHotEncoder(handle_unknown='ignore',
sparse_output=False))]), CAT),
])
Z = prep.fit_transform(df).astype('float32')

pca = PCA(n_components=2, random_state=42).fit(Z)
pca_err = ((pca.inverse_transform(pca.transform(Z)) - Z) ** 2).mean()

tf.random.set_seed(42)
auto = tf.keras.Sequential([
tf.keras.layers.Input(shape=(Z.shape[1],)),
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(2, activation='relu'),
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(Z.shape[1]),
])
auto.compile(optimizer='adam', loss='mse')
auto.fit(Z, Z, epochs=120, batch_size=64, verbose=0)
auto_err = float(auto.evaluate(Z, Z, verbose=0))

print('mean squared reconstruction error, 2 dimensions')
print(' PCA %.4f' % pca_err)
print(' autoencoder %.4f' % auto_err)
mean squared reconstruction error, 2 dimensions
PCA 0.1930
autoencoder 0.2305

PCA wins. Given the same two-dimensional budget, the method with a closed form solution and no hyperparameters rebuilds these rows better than the network does, and it does so in milliseconds, deterministically, with an explained-variance figure you can put in a report.

Fifteen columns is not where an autoencoder shines

The flexibility that makes autoencoders powerful needs something to be flexible about. After one-hot encoding, this data is fifteen columns of largely linear structure, and a linear method models linear structure optimally by definition. Autoencoders start to win on inputs with hundreds or thousands of dimensions and strong nonlinear structure, images, audio, embeddings. Run the comparison before you assume which side you are on.

A convolutional autoencoder on the digits

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
tf.random.set_seed(42)
auto = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2), # 4 x 4 x 16
tf.keras.layers.Conv2D(8, 3, padding='same', activation='relu'),
tf.keras.layers.UpSampling2D(2), # back to 8 x 8
tf.keras.layers.Conv2D(1, 3, padding='same', activation='sigmoid'),
])
auto.compile(optimizer='adam', loss='mse')
auto.fit(X_tr, X_tr, epochs=60, batch_size=64, verbose=0)

rebuilt = auto.predict(X_te[:1], verbose=0)
chars = ' .:-=+*#%@'
def show(a):
a = (a - a.min()) / (a.max() - a.min() + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

print('test error %.4f' % auto.evaluate(X_te, X_te, verbose=0))
print('\noriginal, a %d:' % y_te[0])
show(X_te[0, :, :, 0])
print('\nrebuilt from a 4 x 4 x 8 code:')
show(rebuilt[0, :, :, 0])
test error 0.0088

original, a 1:
=@@=
@@@.
.@@*
.@@#
-@@%
@@*
@@*
=@@

rebuilt from a 4 x 4 x 8 code:
=@@=
@@@:
@@#
.@@%
.@@%
@@*
%@%
=@#

Denoising, which is the version people actually use

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
rng = np.random.default_rng(0)
noisy_tr = np.clip(X_tr + rng.normal(0, 0.3, X_tr.shape), 0, 1).astype('float32')
noisy_te = np.clip(X_te + rng.normal(0, 0.3, X_te.shape), 0, 1).astype('float32')

tf.random.set_seed(42)
auto = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.Conv2D(1, 3, padding='same', activation='sigmoid'),
])
auto.compile(optimizer='adam', loss='mse')
# input: the corrupted image. target: the clean one.
auto.fit(noisy_tr, X_tr, epochs=60, batch_size=64, verbose=0)

cleaned = auto.predict(noisy_te[:1], verbose=0)
chars = ' .:-=+*#%@'
def show(a):
a = (a - a.min()) / (a.max() - a.min() + 1e-9)
for row in a:
print(''.join(chars[min(9, int(v * 9.999))] for v in row))

print('noisy input:')
show(noisy_te[0, :, :, 0])
print('\ndenoised output:')
show(cleaned[0, :, :, 0])
print('\nclean target:')
show(X_te[0, :, :, 0])
noisy input:
.@*+= -
+@-%:.#
=@@= -
#@*
=@@% :#
@*# :
@%@ .
-*@

denoised output:
%#*:
=@#%.
.%@=
*@+
%@*
%%*
#%%
-%%

clean target:
=@@=
@@@.
.@@*
.@@#
-@@%
@@*
@@*
=@@

Nothing about the architecture changed. The only difference is what was put on each side of fit: corrupted input, clean target. That small substitution is the trick behind a surprising amount of modern self-supervised learning, hide part of the input, train the model to reconstruct it, and the representation you get for free turns out to be useful for tasks you have not thought of yet.

Using the bottleneck as features

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression

tf.random.set_seed(42)
encoder = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(12, activation='relu', name='code'),
])
decoder = tf.keras.Sequential([
tf.keras.layers.Input(shape=(12,)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(64, activation='sigmoid'),
tf.keras.layers.Reshape((8, 8, 1)),
])
auto = tf.keras.Sequential([encoder, decoder])
auto.compile(optimizer='adam', loss='mse')
auto.fit(X_tr, X_tr, epochs=80, batch_size=64, verbose=0)

# the labels were never shown to the autoencoder
code_tr = encoder.predict(X_tr, verbose=0)
code_te = encoder.predict(X_te, verbose=0)

flat = LogisticRegression(max_iter=2000).fit(
X_tr.reshape(len(X_tr), -1), y_tr)
coded = LogisticRegression(max_iter=2000).fit(code_tr, y_tr)
print('logistic on 64 raw pixels %.4f'
% flat.score(X_te.reshape(len(X_te), -1), y_te))
print('logistic on 12 learned codes %.4f'
% coded.score(code_te, y_te))
logistic on 64 raw pixels 0.9622
logistic on 12 learned codes 0.9000

Twelve numbers, learned without ever seeing a label, carry most, not all, of what sixty-four pixels carry. That is the honest shape of the result: a five-fold compression costing a few points of accuracy. The reason anybody accepts that trade is the case where labels are the scarce thing: you can train the encoder on a million unlabelled images and then fit the classifier on the two hundred you could afford to label.

Day 6 takeaway

An autoencoder learns a compressed representation with no labels, and the reconstruction error doubles as an anomaly score. The denoising variant, corrupt the input, keep the clean target, costs one line and is the more useful of the two. Compare against PCA before assuming the network earns its complexity.
Week 12 · Day 7 of 7

Does Deep Learning Earn Its Place?

A complete image workflow, and the reckoning week 11 promised

By 1070 words

Week 11 day 3 promised this reckoning. A neural network matched logistic regression on the churn data while carrying several hundred times the parameters. This week it has been winning. The difference between those two situations is the most useful thing in either week.

The complete workflow, end to end

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import tensorflow as tf
tf.random.set_seed(42)
tf.get_logger().setLevel('ERROR')
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
import json
import time

X_fit, X_val = X_tr[:1000], X_tr[1000:]
y_fit, y_val = y_tr[:1000], y_tr[1000:]

tf.random.set_seed(42)
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8, 8, 1)),
tf.keras.layers.RandomTranslation(0.1, 0.1, fill_mode='constant'),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Flatten(), # day 3: position matters at 8 x 8
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Dense(10, activation='softmax'),
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

start = time.time()
hist = model.fit(X_fit, y_fit, validation_data=(X_val, y_val),
epochs=200, batch_size=64, verbose=0,
callbacks=[tf.keras.callbacks.EarlyStopping(
patience=20, restore_best_weights=True)])

print('epochs run %d of 200 in %.1fs' % (len(hist.history['loss']),
time.time() - start))
print('test accuracy %.4f' % model.evaluate(X_te, y_te, verbose=0)[1])

model.save('digits_cnn.keras')
json.dump({'input': [8, 8, 1], 'scaling': 'divide by 16',
'classes': list(range(10))},
open('digits_cnn.json', 'w'))

reloaded = tf.keras.models.load_model('digits_cnn.keras')
print('reloaded and agrees:',
bool((reloaded.predict(X_te[:50], verbose=0).argmax(1)
== model.predict(X_te[:50], verbose=0).argmax(1)).all()))
epochs run 67 of 200 in 10.0s
test accuracy 0.9867
reloaded and agrees: True

Save the preprocessing, not just the weights

The .keras file contains the architecture and the weights. It does not contain the fact that you divided by 16, or which index means which digit. Serve the model without those and it will happily return confident nonsense for inputs on the wrong scale. The small JSON file beside it is not optional, week 15 turns it into a proper contract.

What it cost against what it bought

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

d = load_digits()
X = (d.images / 16.0).astype('float32')[..., np.newaxis]
y = d.target.astype('int32')
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=42)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.svm import SVC
import time

flat_tr = X_tr.reshape(len(X_tr), -1)
flat_te = X_te.reshape(len(X_te), -1)

print('%-24s %10s %10s' % ('', 'accuracy', 'fit time'))
models = [('logistic regression', LogisticRegression(max_iter=2000)),
('random forest', RandomForestClassifier(random_state=42)),
('gradient boosting', HistGradientBoostingClassifier(random_state=42)),
('support vector machine', SVC())]
for name, m in models:
t = time.time()
m.fit(flat_tr, y_tr)
print('%-24s %10.4f %9.1fs'
% (name, m.score(flat_te, y_te), time.time() - t))
accuracy fit time
logistic regression 0.9622 0.1s
random forest 0.9600 0.3s
gradient boosting 0.9600 3.7s
support vector machine 0.9911 0.1s

A support vector machine on the raw flattened pixels, fitted in about a tenth of a second with default settings and no tuning whatsoever, beats the convolutional network that took roughly a hundred times longer to fit and needed batch normalisation, dropout, augmentation and early stopping to get there.

This is the result, not a rhetorical flourish

It would have been easy to leave this comparison out. The reason it is here is that the same comparison, run on 32×32 colour photographs or 224×224 medical scans, comes out the opposite way and it is not close, and the only way to know which situation you are in is to fit the cheap model and look. Eight by eight greyscale digits are, in the end, a 64-dimensional tabular problem wearing a picture's clothing.

When deep learning earns its place

SituationReach forBecause
Tabular data, thousands of rowsGradient boostingWins on accuracy, trains in seconds, needs no scaling
Tabular data, millions of rows and rich categoricalsBoosting first, network secondNetworks become competitive, but rarely by much
Images, audio, videoConvolutional network, pretrainedThe structure of the data is spatial; nothing else exploits that
TextPretrained transformerWeek 13
Fewer than a few hundred labelsTransfer learning, or classicalThere is not enough signal to fit a network from scratch
The decision must be explainedLinear or tree model"The network said so" fails an audit
Latency budget in single millisecondsLinear modelA dot product is hard to beat

The honest summary of two weeks

Deep learning is not a better version of machine learning. It is the answer to a specific question: what do I do when the raw input has structure that no feature I can write down captures? A picture has that problem. A spectrogram has it. A sentence has it. A spreadsheet of customer attributes, where somebody has already done the hard work of deciding that tenure_months is a column, generally does not. The feature engineering is finished before you arrive.

Before you ship an image model

  1. Check the class balance and look at a sample from each class by eye. Mislabelled images are far more common than mislabelled spreadsheet rows.
  2. Split before augmenting. Augmenting first and splitting afterwards puts shifted copies of the same picture on both sides, the image version of the leakage in week 4.
  3. Verify that your augmentations preserve the label in your domain.
  4. Start from pretrained weights unless you have a good reason not to.
  5. Train the head with the base frozen, then fine-tune at a low rate, recompiling in between.
  6. Read the confusion matrix and look at the actual images the model gets wrong.
  7. Fit gradient boosting on the flattened pixels as a floor. If it wins, your images are probably too small or too few for a network.
  8. Save the preprocessing contract alongside the weights.

Day 7 takeaway

Convolution wins where the input is spatial and the useful features cannot be written down by hand. On a spreadsheet it does not, and week 11's result was not a failure of that network. It was the correct answer to the wrong question. Know which kind of problem you have before you choose the tool.