Images as Data

Week 8 of 14 · Vision · 7 days

Full curriculum
Week 08 · Vision

Images as Data

Week 08 · Day 1 of 7

A Grid of Numbers

What an image is, and why a pixel is a poor feature

By 407 words

An image is a grid of numbers. Everything in computer vision follows from that one fact, and most of the difficulty follows from a second: the numbers describe light, not objects, and nothing in them says where one thing ends and another begins.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
print('%d images, each %d by %d' % X.shape)
print('values run from %.0f to %.0f' % (X.min(), X.max()))
print()
print('this one is labelled %d:' % y[0])
show(X[0])
print()
print('the top left corner as numbers:')
print(X[0][:4, :4])
1797 images, each 8 by 8
values run from 0 to 16

this one is labelled 0:
-%+
%@*@-
.@. *=
:# ==
-= +=
:* #=
.%-*#
-%*

the top left corner as numbers:
[[ 0. 0. 5. 13.]
[ 0. 0. 13. 15.]
[ 0. 3. 15. 2.]
[ 0. 4. 12. 0.]]

Eight by eight, one number per pixel for brightness. A colour photograph is the same idea with three numbers per pixel, and a modern camera image is a few million pixels rather than sixty-four, but the structure does not change.

What makes this hard

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np

a = X[0]
shifted = np.roll(a, 1, axis=1)
brighter = np.clip(a * 1.4, 0, 16)
print('%-28s %10s' % ('', 'pixels changed'))
for name, img in [('shifted one pixel right', shifted),
('40% brighter', brighter)]:
print('%-28s %10d' % (name, int((img != a).sum())))
print()
print('to a human both are obviously the same digit')
print('to a model comparing pixels they are largely different images')
pixels changed
shifted one pixel right 46
40% brighter 35

to a human both are obviously the same digit
to a model comparing pixels they are largely different images

Why a pixel is a poor feature

Moving an image one pixel changes most of its numbers while changing nothing about what it depicts. Brightness, contrast, rotation and scale all do the same. A model treating each pixel position as an independent feature has to learn every appearance of every object separately, which is why the whole field is about finding representations that survive these changes.

Week 08 · Day 2 of 7

Convolution

Nine numbers that find every edge in an image

By 454 words

The classical answer, and still the foundation, is to look at small neighbourhoods rather than individual pixels. Slide a small grid of weights across the image and record what it responds to.

Convolution: Slide a small matrix, called a kernel or filter, across the image. At each position multiply the overlapping numbers together and add them up. The result is a new image showing where that pattern occurs.
import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np

def convolve(image, kernel):
kh, kw = kernel.shape
h, w = image.shape[0] - kh + 1, image.shape[1] - kw + 1
out = np.zeros((h, w))
for r in range(h):
for c in range(w):
patch = image[r:r + kh, c:c + kw]
out[r, c] = float((patch * kernel).sum())
return out

VERTICAL = np.array([[-1., 0., 1.],
[-2., 0., 2.],
[-1., 0., 1.]])
HORIZONTAL = VERTICAL.T

img = X[0]
print('the digit:')
show(img)
print()
print('where the vertical edges are:')
show(np.abs(convolve(img, VERTICAL)), scale=40.0)
print()
print('where the horizontal edges are:')
show(np.abs(convolve(img, HORIZONTAL)), scale=40.0)
the digit:
-%+
%@*@-
.@. *=
:# ==
-= +=
:* #=
.%-*#
-%*

where the vertical edges are:
@@= :@
@:@*=@
@-@%#%
@=@@#@
@:#@:@
@---+%

where the horizontal edges are:
-:+==*
.-@@+
-:
..
-**.-
... (1 more lines)

Two kernels, nine numbers each, and the image has been turned into a map of where its edges run and in which direction. Nothing was learned; those weights were chosen by a person decades ago and they work on any image.

Other kernels do other things

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np

def convolve(image, kernel):
kh, kw = kernel.shape
h, w = image.shape[0] - kh + 1, image.shape[1] - kw + 1
out = np.zeros((h, w))
for r in range(h):
for c in range(w):
out[r, c] = float((image[r:r + kh, c:c + kw] * kernel).sum())
return out

BLUR = np.ones((3, 3)) / 9.0
SHARPEN = np.array([[0., -1., 0.], [-1., 5., -1.], [0., -1., 0.]])
img = X[0]
for name, k, scale in [('blurred', BLUR, 16.0),
('sharpened', SHARPEN, 16.0)]:
print('%s:' % name)
show(np.clip(convolve(img, k), 0, None), scale=scale)
print()
blurred:
:=++=:
-====-
--::--
--::--
----=-
:-==-:

sharpened:
@@*@.
@ @@
@ #@
+# @@
.@ @@
@ @@

This is the operation a neural network learns

Week 9 builds a model that does exactly this and works out the nine numbers for itself, from the data, instead of using the ones a person chose. The mechanism is identical. What changes is where the kernel comes from, and that single change is most of what modern computer vision is.

Week 08 · Day 3 of 7

Building a Description

From edges to features, and what invariance means

By 448 words

Edges alone are not a description of an object. The classical approach built features on top of them, and one simple version is worth constructing because it makes clear what a feature is for.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np

def convolve(image, kernel):
kh, kw = kernel.shape
h, w = image.shape[0] - kh + 1, image.shape[1] - kw + 1
out = np.zeros((h, w))
for r in range(h):
for c in range(w):
out[r, c] = float((image[r:r + kh, c:c + kw] * kernel).sum())
return out

VERT = np.array([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])
HORIZ = VERT.T

def edge_features(image):
"""Split the image into quarters and record how much edge energy
of each direction is in each. Sixteen numbers instead of 64."""
gx, gy = convolve(image, VERT), convolve(image, HORIZ)
feats = []
for rows in [slice(0, 3), slice(3, 6)]:
for cols in [slice(0, 3), slice(3, 6)]:
feats.extend([np.abs(gx[rows, cols]).mean(),
np.abs(gy[rows, cols]).mean()])
return np.array(feats)

for i in [0, 1]:
print('digit %d -> %s' % (y[i],
np.round(edge_features(X[i]), 1)))
digit 0 -> [35.8 15.3 27.6 15.3 28.3 9.7 30.4 14.7]
digit 1 -> [37.4 13.9 39.8 5.6 41.3 10.7 42.1 4.6]

Eight numbers summarising where the edges are and which way they run. That is a feature: a description that keeps what distinguishes things and discards what does not. Whether these particular eight are good ones is a question with an experimental answer, which is tomorrow.

The point of a feature

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np

a, b = X[0], np.roll(X[0], 1, axis=1)
raw_change = np.abs(a - b).mean()
print('shifting by one pixel changes the raw image by %.2f on average'
% raw_change)
print()
print('a good feature would barely move, because the digit is')
print('the same digit. that is the property being engineered for.')
shifting by one pixel changes the raw image by 4.78 on average

a good feature would barely move, because the digit is
the same digit. that is the property being engineered for.
Invariance: A representation is invariant to a change if the change does not alter it. Useful vision features are invariant to the things that do not matter, such as position, brightness and small rotations, and sensitive to the things that do.
Week 08 · Day 4 of 7

Do Designed Features Help

Three representations measured, and the one that lost

By 416 words

Time to find out whether any of this helps, by measuring three representations against each other on the same classifier.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import train_test_split

def convolve(image, kernel):
kh, kw = kernel.shape
h, w = image.shape[0] - kh + 1, image.shape[1] - kw + 1
out = np.zeros((h, w))
for r in range(h):
for c in range(w):
out[r, c] = float((image[r:r + kh, c:c + kw] * kernel).sum())
return out

VERT = np.array([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])
HORIZ = VERT.T

def edge_features(image):
gx, gy = convolve(image, VERT), convolve(image, HORIZ)
feats = []
for rows in [slice(0, 3), slice(3, 6)]:
for cols in [slice(0, 3), slice(3, 6)]:
feats.extend([np.abs(gx[rows, cols]).mean(),
np.abs(gy[rows, cols]).mean()])
return np.array(feats)

flat = X.reshape(len(X), -1)
edges = np.array([edge_features(img) for img in X])
both = np.hstack([flat, edges])

print('%-30s %8s %10s' % ('representation', 'columns', 'accuracy'))
for name, data in [('raw pixels', flat),
('edge features only', edges),
('both together', both)]:
a, b, ya, yb = train_test_split(data, y, test_size=0.3,
random_state=0, stratify=y)
m = LogisticRegression(max_iter=5000).fit(a, ya)
print('%-30s %8d %10.4f' % (name, data.shape[1], m.score(b, yb)))
a, b, ya, yb = train_test_split(flat, y, test_size=0.3, random_state=0,
stratify=y)
print('%-30s %8d %10.4f'
% ('guessing the commonest digit', 0,
DummyClassifier(strategy='most_frequent').fit(a, ya).score(b, yb)))
representation columns accuracy
raw pixels 64 0.9611
edge features only 8 0.7759
both together 72 0.9815
guessing the commonest digit 0 0.1019

Read that table honestly. Eight hand-designed numbers do considerably better than guessing and considerably worse than simply handing over all sixty-four pixels, and combining them adds little to the pixels alone.

Hand-designed features lost, on this data

That is the honest result and it is worth sitting with, because it is a miniature of what happened to the whole field. Decades of work went into designing better image features, and they were overtaken by letting a model learn its own from data.

The caveat is that these digits are small, centred and clean, which flatters raw pixels enormously. On photographs raw pixels collapse and designed features held on much longer. The direction of the result generalises; its size does not.

Week 08 · Day 5 of 7

Testing What Survives

Shifting, dimming and flipping the held out images

By 394 words

Since a feature is meant to survive irrelevant changes, that claim can be tested directly.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

flat = X.reshape(len(X), -1)
a, b, ya, yb = train_test_split(flat, y, test_size=0.3, random_state=0,
stratify=y)
model = LogisticRegression(max_iter=5000).fit(a, ya)

test_images = X[len(a):] if False else None
idx = np.arange(len(X))
_, idx_test = train_test_split(idx, test_size=0.3, random_state=0,
stratify=y)
originals = X[idx_test]
labels = y[idx_test]

def score(images):
return float((model.predict(images.reshape(len(images), -1))
== labels).mean())

print('%-34s %10s' % ('the held out images, altered', 'accuracy'))
print('%-34s %10.4f' % ('unchanged', score(originals)))
print('%-34s %10.4f' % ('shifted one pixel right',
score(np.roll(originals, 1, axis=2))))
print('%-34s %10.4f' % ('shifted two pixels down',
score(np.roll(originals, 2, axis=1))))
print('%-34s %10.4f' % ('30% dimmer', score(originals * 0.7)))
print('%-34s %10.4f' % ('flipped horizontally',
score(originals[:, :, ::-1])))
the held out images, altered accuracy
unchanged 0.9611
shifted one pixel right 0.4630
shifted two pixels down 0.0852
30% dimmer 0.9630
flipped horizontally 0.3852

A one pixel shift costs a great deal of accuracy. A model built on raw pixels has no notion that an image can move; it learned which positions tend to be bright for each digit, and moving everything invalidates that.

The flip result is not a failure

A horizontally flipped digit genuinely is a different thing, and often not a digit at all. The model should do badly there and it does. This is the point week 4 made about stop words arriving in a new form: an alteration is only irrelevant if it does not change the right answer, and deciding which is which is a judgement about the problem rather than about the data.

The two standard remedies

  • Augmentation. Add shifted, dimmed and slightly rotated copies to the training set, so the model sees the variation and learns to ignore it. Cheap, and it works.
  • An architecture that is built for it. Convolution applies the same kernel at every position, so a pattern is recognised wherever it appears. That is the structural answer and it is what week 9 uses.
Week 08 · Day 6 of 7

Images From the Real World

Size, lighting, backgrounds and the cost of labels

By 357 words

Real images bring problems that a clean dataset hides, and they are worth naming before week 9 makes everything look easy.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np

print('%-34s %14s %12s' % ('', 'numbers/image', 'for 10k images'))
for name, shape in [('these digits, 8x8 grey', (8, 8, 1)),
('a thumbnail, 64x64 colour', (64, 64, 3)),
('a web image, 512x512 colour', (512, 512, 3)),
('a phone photo, 4032x3024 colour', (4032, 3024, 3))]:
n = int(np.prod(shape))
print('%-34s %14d %11.1fGB'
% (name, n, n * 10_000 * 4 / 1e9))
numbers/image for 10k images
these digits, 8x8 grey 64 0.0GB
a thumbnail, 64x64 colour 12288 0.5GB
a web image, 512x512 colour 786432 31.5GB
a phone photo, 4032x3024 colour 36578304 1463.1GB

This is why images are resized before anything else happens, and why the resizing decision matters: too small and the thing you care about disappears, too large and training becomes unaffordable.

The rest of the list

  • Colour is not reliable. The same object photographed under different lighting gives quite different numbers, which is why many pipelines convert to greyscale or normalise aggressively.
  • Backgrounds are learned by accident. If every picture of a cow in your data has grass in it, the model learns grass. This is one of the best documented failures in the field.
  • Labels are expensive and inconsistent. Drawing a box around every object in ten thousand photographs is weeks of work, and two people will not draw the same boxes.
  • The camera is part of the model. A system trained on one device's images often degrades on another's, which is week 13's problem arriving early.

The background problem is worth taking seriously

A model can reach excellent accuracy by learning something entirely beside the point, and nothing in the accuracy figure reveals it. The only way to catch it is to look at what the model is responding to, which is week 12's subject, or to test on images where the incidental correlation is broken.

Week 08 · Day 7 of 7

Where Images Stand

The week assembled, and the change that comes next

By 342 words

The week's methods on one figure, and a summary of where images stand before any learning is applied to them.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.images, digits.target
RAMP = ' .:-=+*#%@'

def show(img, scale=16.0):
for row in img:
print(''.join(RAMP[min(9, max(0, int(v / scale * 9.999)))]
for v in row))
import numpy as np

def convolve(image, kernel):
kh, kw = kernel.shape
h, w = image.shape[0] - kh + 1, image.shape[1] - kw + 1
out = np.zeros((h, w))
for r in range(h):
for c in range(w):
out[r, c] = float((image[r:r + kh, c:c + kw] * kernel).sum())
return out

VERT = np.array([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])
img = X[13]
print('a %d, as pixels:' % y[13])
show(img)
print()
print('as vertical edge response:')
show(np.abs(convolve(img, VERT)), scale=40.0)
print()
print('64 numbers became 36, keeping the strokes and')
print('discarding the flat areas that carry no information')
a 3, as pixels:
.+@%+.
:%=+@=
-%@.
*%.
.@*
.@:
--%@-
.##%*

as vertical edge response:
%*:=+@
-*@=@@
#@ @#
-@+@@
..*@=@
+==%*@

64 numbers became 36, keeping the strokes and
discarding the flat areas that carry no information

What the week established

  • An image is a grid of numbers describing light, and nothing in it marks where objects are.
  • A raw pixel is a poor feature because irrelevant changes alter almost all of them.
  • Convolution looks at neighbourhoods rather than points, and a nine-number kernel turns an image into a map of its edges.
  • Hand-designed features lost to raw pixels on this clean data, which is a miniature of what happened to the field.
  • A model built on raw pixels loses accuracy from a one pixel shift, and the two remedies are augmentation and convolution.
  • Real images bring size, lighting, background and labelling problems that clean datasets hide entirely.

What week 9 changes

Every kernel in this week was chosen by a person. Next week a model works out its own kernels from the data, which is the single change that separates classical computer vision from the modern kind, and the week measures what it is worth.