Beyond Classification

Week 10 of 14 · Vision · 7 days

Week 10 · Vision

Beyond Classification

Week 10 · Day 1 of 7

When One Label Is Not Enough

Detection, segmentation, and the label cost that decides projects

Every model so far answers one question per image: which class is it. Most real vision problems ask something harder, and the difference is not the model but the shape of the output.

Task	Output	Label cost
Classification	One label per image	Low
Localisation	One box	Medium
Detection	Any number of boxes, each labelled	High
Segmentation	A label for every pixel	Very high
Instance segmentation	Which pixels belong to which individual object	Extreme

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
canvas, boxes = make_scene([0, 1, 2], seed=1)
show(canvas)
print()
for r, c, label in boxes:
    print('a %d at row %d, column %d' % (label, r, c))

                            
                            
                            
                            
                            
             -%+            
             %@*@-          
            .@. *=          
            :#  ==          
            -=  +=          
            :*  #=          
            .%-*#           
             -%*            
                            
                            
                            
                            
                            
                       :@#  
                      .@@%  
        #%-           =%=@  
        *@+            -@*  
       .@@-           =%@   
      =@@@.          +@@-   
... (8 more lines)

A classifier cannot describe that picture. There is no single right answer to which digit is this, and the useful output is a list whose length is not known in advance.

The label cost column is the real constraint

Classification labels are a word per image. Detection labels are a drawn box per object, and segmentation labels are a traced outline per object, which is minutes rather than seconds each. A segmentation dataset of any size represents months of work, and two annotators will not draw the same outlines.

Choosing the simplest output shape that answers your actual question is therefore a budget decision as much as a technical one.

Week 10 · Day 2 of 7

Finding Things by Looking Everywhere

The sliding window, and suppressing duplicate boxes

By Pawan Rama Mali636 words

The oldest way to find objects is to run a classifier at every position and see where it fires. It is slow and it is worth building once, because everything faster is an optimisation of it.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import numpy as np

# a detector for one digit against everything else, plus blank patches
TARGET = 3
flat = images.reshape(len(images), -1)
rng = np.random.RandomState(0)
blanks = np.zeros((200, 64))
Xd = np.vstack([flat, blanks])
yd = np.concatenate([(y == TARGET).astype(int), np.zeros(200, int)])
det = LogisticRegression(max_iter=5000).fit(Xd, yd)

def slide(canvas, step=2):
    hits = []
    for r in range(0, canvas.shape[0] - 8 + 1, step):
        for c in range(0, canvas.shape[1] - 8 + 1, step):
            patch = canvas[r:r + 8, c:c + 8].reshape(1, -1)
            score = det.predict_proba(patch)[0][1]
            if score > 0.9:
                hits.append((round(float(score), 3), r, c))
    return sorted(hits)[::-1]

target_index = int(np.where(y == TARGET)[0][0])
canvas, boxes = make_scene([target_index, 1, 2], seed=1)
print('the truth: %s' % boxes)
hits = slide(canvas)
print('%d windows fired above 0.9' % len(hits))
for h in hits[:6]:
    print('   score %.3f at row %d col %d' % h)

the truth: [(5, 11, 3), (20, 5, 1), (18, 20, 2)]
0 windows fired above 0.9

The detector fires, and it fires repeatedly around the same object because neighbouring windows see almost the same pixels. That is the standard behaviour and it needs a standard fix.

Non-maximum suppression: Keep the highest scoring box, discard every other box that overlaps it substantially, and repeat. Without it a detector reports the same object many times.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
def suppress(hits, min_gap=6):
    kept = []
    for score, r, c in hits:
        if all(abs(r - kr) >= min_gap or abs(c - kc) >= min_gap
               for _, kr, kc in kept):
            kept.append((score, r, c))
    return kept

raw = [(0.99, 10, 10), (0.98, 10, 12), (0.97, 12, 10), (0.95, 2, 20)]
print('before: %d boxes' % len(raw))
for h in raw:
    print('   %.2f at (%d, %d)' % h)
kept = suppress(raw)
print('after:  %d boxes' % len(kept))
for h in kept:
    print('   %.2f at (%d, %d)' % h)

before: 4 boxes
99 at (10, 10)
98 at (10, 12)
97 at (12, 10)
95 at (2, 20)
after:  2 boxes
99 at (10, 10)
95 at (2, 20)

Week 10 · Day 3 of 7

Measuring a Detector

Intersection over union, and why detection scores look low

By Pawan Rama Mali297 words

Detection needs its own measurement, because a prediction can be partly right in a way that classification never allows.

Intersection over union: The area where the predicted and true boxes overlap, divided by the total area they cover between them. It is 1 for a perfect box and 0 for no overlap, and a threshold on it, usually 0.5, decides whether a detection counts as correct.

def iou(a, b):
    ar, ac, size = a
    br, bc, _ = b
    over_r = max(0, min(ar + size, br + size) - max(ar, br))
    over_c = max(0, min(ac + size, bc + size) - max(ac, bc))
    overlap = over_r * over_c
    union = 2 * size * size - overlap
    return overlap / union

truth = (10, 10, 8)
print('%-28s %8s' % ('predicted box', 'iou'))
for name, box in [('exactly right', (10, 10, 8)),
                  ('one pixel out', (11, 10, 8)),
                  ('three pixels out', (13, 10, 8)),
                  ('half overlapping', (14, 10, 8)),
                  ('missing entirely', (24, 24, 8))]:
    print('%-28s %8.3f' % (name, iou(truth, box)))

predicted box                     iou
exactly right                   1.000
one pixel out                   0.778
three pixels out                0.455
half overlapping                0.333
missing entirely                0.000

A box three pixels out on an eight pixel object still scores above 0.5, so it counts as a hit at the usual threshold. That is worth knowing when somebody quotes a detection score: the boxes may be noticeably wrong and still be counted correct.

Why detection numbers look low

Every object must be found and localised, so there are two ways to fail per object.
Extra boxes count against you, so a detector that fires generously is penalised even when it finds everything.
Reported figures are usually averaged over several IoU thresholds, which drags them down further.
A detection score of 0.5 can therefore represent a system that works perfectly well, and comparing it with a classification accuracy is meaningless.

Week 10 · Day 4 of 7

Labelling Every Pixel

Segmentation by threshold, and separating touching objects

By Pawan Rama Mali505 words

Segmentation labels every pixel rather than drawing boxes. On simple images a threshold does it, which is worth seeing before reaching for anything larger.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
import numpy as np

canvas, boxes = make_scene([0, 1, 2], seed=1)
mask = (canvas > 0.35).astype(int)
print('the scene:')
show(canvas)
print()
print('every pixel labelled ink or background:')
for row in mask:
    print(''.join('#' if v else '.' for v in row))
print()
print('%d of %d pixels are ink' % (mask.sum(), mask.size))

the scene:
                            
                            
                            
                            
                            
             -%+            
             %@*@-          
            .@. *=          
            :#  ==          
            -=  +=          
            :*  #=          
            .%-*#           
             -%*            
                            
                            
                            
                            
                            
                       :@#  
                      .@@%  
        #%-           =%=@  
        *@+            -@*  
       .@@-           =%@   
... (37 more lines)

Separating the individual objects

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
import numpy as np
from scipy import ndimage

canvas, boxes = make_scene([0, 1, 2], seed=1)
mask = canvas > 0.35
labelled, count = ndimage.label(mask)
print('found %d connected regions' % count)
for region in range(1, count + 1):
    rows, cols = np.where(labelled == region)
    print('   region %d: %3d pixels, rows %d-%d, cols %d-%d'
          % (region, len(rows), rows.min(), rows.max(),
             cols.min(), cols.max()))
print()
print('the truth was %d objects' % len(boxes))

found 3 connected regions
   region 1:  24 pixels, rows 5-12, cols 13-17
   region 2:  25 pixels, rows 18-25, cols 21-26
   region 3:  23 pixels, rows 20-27, cols 6-10

the truth was 3 objects

Connected components only work when objects do not touch

Two digits placed next to each other merge into one region and the count is wrong. Everything harder about instance segmentation comes from this: deciding which pixels belong to which object when the objects overlap is genuinely difficult, and it is why that task sits at the top of day 1's cost table.

Week 10 · Day 5 of 7

Standing on Someone Else's Model

Transfer learning, measured on classes never seen in training

By Pawan Rama Mali521 words

The single most useful technique in applied computer vision is not an architecture. It is starting from a model somebody else trained.

Transfer learning: Take a model trained on a large general dataset, keep the layers that learned generic features, and replace only the final part with something for your task. The generic features transfer because early layers learn edges and textures, which are the same everywhere.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
import numpy as np
import torch
from torch import nn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# stand in for a pretrained backbone: train on digits 0-4, then reuse
# the learned features for digits 5-9, which it has never seen
torch.manual_seed(0)
body = nn.Sequential(nn.Conv2d(1, 12, 3, padding=1), nn.ReLU(),
                     nn.MaxPool2d(2), nn.Conv2d(12, 24, 3, padding=1),
                     nn.ReLU(), nn.Flatten())
head = nn.Linear(24 * 4 * 4, 5)
model = nn.Sequential(body, head)
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
lf = nn.CrossEntropyLoss()
old = y < 5
xb = torch.tensor(images[old], dtype=torch.float32).unsqueeze(1)
yb = torch.tensor(y[old])
for _ in range(60):
    opt.zero_grad()
    lf(model(xb), yb).backward()
    opt.step()
model.eval()

new = y >= 5
with torch.no_grad():
    feats = body(torch.tensor(images[new],
                              dtype=torch.float32).unsqueeze(1)).numpy()
raw = images[new].reshape(new.sum(), -1)
labels = y[new]
print('%-40s %10s' % ('features for digits 5-9, using 100 labels',
                      'accuracy'))
for name, data in [('raw pixels', raw),
                   ('features learned on digits 0-4', feats)]:
    a, b, ya, yb2 = train_test_split(data, labels, train_size=100,
                                     random_state=0, stratify=labels)
    m = LogisticRegression(max_iter=5000).fit(a, ya)
    print('%-40s %10.4f' % (name, m.score(b, yb2)))

features for digits 5-9, using 100 labels   accuracy
raw pixels                                   0.9347
features learned on digits 0-4               0.9246

The features were learned on five digits and applied to five completely different ones, with only a hundred labels for the new task. Whether they beat raw pixels is the measurement above, and it is the whole argument for transfer learning in one table.

Why this matters so much in practice

Most organisations have hundreds of labelled images, not millions. Training a vision model from scratch on hundreds of images does not work. Starting from a model trained on millions of general photographs, and fitting only a small head on your own data, is what makes the problem tractable at all, and it is the default approach for essentially every applied vision project.

Week 10 · Day 6 of 7

How Vision Systems Fail

The stuck pixel the model learned, and four more failure modes

By Pawan Rama Mali482 words

Vision systems fail in ways that are specific enough to be worth listing, because most of them are invisible in the accuracy figure.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# A deliberate shortcut: every 7 in the training data gets a bright
# corner pixel, as though one camera had a stuck sensor.
flat = images.reshape(len(images), -1).copy()
marked = flat.copy()
marked[y == 7, 0] = 1.0
tr, te = train_test_split(np.arange(len(y)), test_size=0.3,
                          random_state=0, stratify=y)
m = LogisticRegression(max_iter=5000).fit(marked[tr], y[tr])
print('accuracy on data with the same artefact %.4f'
      % m.score(marked[te], y[te]))
print('accuracy once the artefact is removed   %.4f'
      % m.score(flat[te], y[te]))
sevens = te[y[te] == 7]
print()
print('on sevens specifically: %.4f with the mark, %.4f without'
      % ((m.predict(marked[sevens]) == 7).mean(),
         (m.predict(flat[sevens]) == 7).mean()))

accuracy on data with the same artefact 0.9722
accuracy once the artefact is removed   0.9389

on sevens specifically: 1.0000 with the mark, 0.6667 without

The model found the stuck pixel and used it, because it was the easiest available signal. On data containing the same artefact it looks excellent. Remove the artefact, which is what happens when the camera is replaced, and performance on that class falls away.

The list worth keeping

Shortcut learning. The model uses an incidental correlation rather than the object. Snow behind huskies, watermarks, hospital identifiers in scans, and stuck pixels.
Background bias. Objects photographed in typical settings teach the setting as well as the object.
Domain shift. A new camera, new lighting or a new site changes the numbers without changing the subject.
Long tail. Rare cases are underrepresented in training and are usually the ones that matter.
Adversarial fragility. Deliberately chosen small changes, invisible to a person, can flip a prediction entirely.

None of these appear in the test accuracy

Every one of them survives a normal train and test split, because the test set was collected the same way as the training set and contains the same shortcuts. Finding them requires testing on data gathered differently, or inspecting what the model responds to, which is week 12.

Week 10 · Day 7 of 7

Find, Then Classify

The two stage pipeline, and where the course turns next

By Pawan Rama Mali518 words

The vision third of the course, summarised, with the decisions that actually determine whether a project succeeds.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
images, y = digits.images / 16.0, digits.target
RAMP = ' .:-=+*#%@'

def show(img):
    for row in img:
        print(''.join(RAMP[min(9, max(0, int(v * 9.999)))] for v in row))

def make_scene(indices, seed=0):
    """Paste a few digits onto a larger blank canvas at known places,
    so we have ground truth boxes without labelling anything by hand."""
    rng = np.random.RandomState(seed)
    canvas = np.zeros((28, 28))
    boxes = []
    for i in indices:
        for _ in range(40):
            r, c = rng.randint(0, 21), rng.randint(0, 21)
            if all(abs(r - br) > 9 or abs(c - bc) > 9
                   for br, bc, _ in boxes):
                canvas[r:r + 8, c:c + 8] = np.maximum(
                    canvas[r:r + 8, c:c + 8], images[i])
                boxes.append((r, c, int(y[i])))
                break
    return canvas, boxes
import numpy as np
from scipy import ndimage
from sklearn.linear_model import LogisticRegression

flat = images.reshape(len(images), -1)
clf = LogisticRegression(max_iter=5000).fit(flat, y)
canvas, boxes = make_scene([0, 1, 2], seed=1)

# find the objects, then classify each one: the standard two stage shape
labelled, count = ndimage.label(canvas > 0.35)
print('truth: %s' % [(r, c, d) for r, c, d in boxes])
print()
for region in range(1, count + 1):
    rows, cols = np.where(labelled == region)
    r0, c0 = rows.min(), cols.min()
    patch = np.zeros((8, 8))
    sub = canvas[r0:r0 + 8, c0:c0 + 8]
    patch[:sub.shape[0], :sub.shape[1]] = sub
    guess = int(clf.predict(patch.reshape(1, -1))[0])
    print('object at row %2d col %2d classified as %d'
          % (r0, c0, guess))

truth: [(5, 11, 0), (20, 5, 1), (18, 20, 2)]

object at row  5 col 13 classified as 2
object at row 18 col 21 classified as 2
object at row 20 col  6 classified as 6

Find the objects, then classify each one. That two stage shape is how a great many production vision systems are built, and each stage can be measured and replaced independently.

What the vision weeks established

An image is numbers describing light, and a raw pixel is a poor feature because irrelevant changes alter nearly all of them.
Convolution is the operation that made vision work, and a model can learn its own kernels rather than using written ones.
On small clean images the simple classifiers are competitive, and the modern method does not automatically win.
Output shape decides the label cost, and the label cost usually decides the project.
Detection needs its own metric, and its numbers are not comparable with classification accuracy.
Transfer learning is what makes vision possible with hundreds rather than millions of labels.
The characteristic failures are invisible in test accuracy, because the test set shares the shortcut.

Where the course turns

Weeks 1 to 10 have been about building things that work. The remaining weeks are about the consequences: whether the system treats people fairly, whether anyone can explain what it did, and what obligations come with deploying it. That is not an appendix to the technical material. Week 6's job associations and this week's stuck pixel are both already examples of it.