Privacy, Safety and Governance

Week 13 of 14 · Responsibility · 7 days

Full curriculum
Week 13 · Responsibility

Privacy, Safety and Governance

Week 13 · Day 1 of 7

Anonymous Data That Is Not

Three ordinary fields, and how many people they identify

By 469 words

A model is built from data about people, and that fact carries obligations that are legal rather than optional. This week is the practical version of them.

Removing the names is not anonymisation

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 2000
df = pd.DataFrame({
'postcode': rng.choice(['M1', 'M2', 'M3', 'M4', 'M5'], n),
'birth_year': rng.randint(1950, 2006, n),
'sex': rng.choice(['F', 'M'], n),
'condition': rng.choice(['none', 'asthma', 'diabetes', 'cardiac'],
n, p=[0.7, 0.13, 0.12, 0.05]),
})
# the 'anonymous' release: no names, no identifiers
released = df[['postcode', 'birth_year', 'sex', 'condition']]
print(released.head(3).to_string(index=False))
print()
combo = released.groupby(['postcode', 'birth_year', 'sex']).size()
unique = int((combo == 1).sum())
print('%d people in the release' % len(released))
print('%d of them are the only person with their combination of'
% unique)
print('postcode, birth year and sex')
print()
print('that is %.1f%% of the dataset, individually identifiable to'
% (100 * unique / len(released)))
print('anybody who knows those three ordinary facts about them')
postcode birth_year sex condition
M5 1966 F none
M1 1978 F none
M4 1959 F none

2000 people in the release
53 of them are the only person with their combination of
postcode, birth year and sex

that is 2.6% of the dataset, individually identifiable to
anybody who knows those three ordinary facts about them
Re-identification: Recovering an individual's identity from data that has had direct identifiers removed, by combining the remaining attributes with information from elsewhere. Postcode, date of birth and sex are the classic combination and they are sufficient for a large share of any population.

This is a well documented failure, not a hypothetical

Public health, transport and search datasets have all been released as anonymous and then re-identified by researchers using exactly this technique. The lesson is that anonymity is a property of a dataset in the context of every other dataset in the world, not a property you can establish by inspecting your own columns.

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 2000
df = pd.DataFrame({
'postcode': rng.choice(['M1', 'M2', 'M3', 'M4', 'M5'], n),
'birth_year': rng.randint(1950, 2006, n),
'sex': rng.choice(['F', 'M'], n),
'condition': rng.choice(['none', 'asthma', 'diabetes', 'cardiac'],
n, p=[0.7, 0.13, 0.12, 0.05]),
})
def unique_share(cols):
sizes = df.groupby(cols).size()
counts = df.groupby(cols).size().reset_index(name='k')
merged = df.merge(counts, on=cols)
return float((merged['k'] == 1).mean())

print('%-44s %10s' % ('quasi-identifiers released', 'unique'))
for cols in [['postcode'], ['postcode', 'sex'],
['postcode', 'birth_year'],
['postcode', 'birth_year', 'sex']]:
print('%-44s %10.3f' % (', '.join(cols), unique_share(cols)))
quasi-identifiers released unique
postcode 0.000
postcode, sex 0.000
postcode, birth_year 0.000
postcode, birth_year, sex 0.026

Each additional column multiplies the number of distinct combinations, so uniqueness rises very quickly. This is why we only released three harmless fields is not a defence.

Week 13 · Day 2 of 7

Making a Release Safer

Generalisation, k-anonymity, and the guarantee only one method offers

By 501 words

Two standard techniques reduce the risk, and both cost something that has to be accounted for.

k-anonymity: Every combination of quasi-identifiers appears at least k times, so no individual can be picked out from fewer than k people. Achieved by generalising values, such as replacing a birth year with a decade, or by suppressing rare rows.
import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 2000
df = pd.DataFrame({
'postcode': rng.choice(['M1', 'M2', 'M3', 'M4', 'M5'], n),
'birth_year': rng.randint(1950, 2006, n),
'sex': rng.choice(['F', 'M'], n),
'condition': rng.choice(['none', 'asthma', 'diabetes', 'cardiac'],
n, p=[0.7, 0.13, 0.12, 0.05]),
})
coarse = df.copy()
coarse['decade'] = (coarse['birth_year'] // 10) * 10

def smallest_group(frame, cols):
return int(frame.groupby(cols).size().min())

print('%-46s %8s' % ('release', 'min group'))
print('%-46s %8d' % ('postcode, birth year, sex',
smallest_group(df, ['postcode', 'birth_year',
'sex'])))
print('%-46s %8d' % ('postcode, birth decade, sex',
smallest_group(coarse, ['postcode', 'decade',
'sex'])))
print('%-46s %8d' % ('postcode, birth decade',
smallest_group(coarse, ['postcode', 'decade'])))
release min group
postcode, birth year, sex 1
postcode, birth decade, sex 16
postcode, birth decade 35

Generalising the birth year to a decade lifts the smallest group substantially. It also destroys information: any analysis that needed age to the year can no longer be done. That is the trade, and it is not avoidable.

Where k-anonymity is still not enough

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 2000
df = pd.DataFrame({
'postcode': rng.choice(['M1', 'M2', 'M3', 'M4', 'M5'], n),
'birth_year': rng.randint(1950, 2006, n),
'sex': rng.choice(['F', 'M'], n),
'condition': rng.choice(['none', 'asthma', 'diabetes', 'cardiac'],
n, p=[0.7, 0.13, 0.12, 0.05]),
})
coarse = df.copy()
coarse['decade'] = (coarse['birth_year'] // 10) * 10
groups = coarse.groupby(['postcode', 'decade', 'sex'])['condition']
worst = None
for key, conditions in groups:
counts = conditions.value_counts(normalize=True)
if worst is None or counts.iloc[0] > worst[1]:
worst = (key, float(counts.iloc[0]), counts.index[0])
print('the least diverse group is %s' % (worst[0],))
print('%.0f%% of it has condition "%s"' % (100 * worst[1], worst[2]))
print()
print('if a group is uniform on the sensitive column, knowing')
print('somebody is in that group tells you their condition,')
print('even though you cannot tell which row is theirs')
the least diverse group is ('M1', np.int32(2000), 'M')
83% of it has condition "none"

if a group is uniform on the sensitive column, knowing
somebody is in that group tells you their condition,
even though you cannot tell which row is theirs

The technique that does have a guarantee

Differential privacy adds calibrated random noise to results, so that the output is provably almost unchanged whether or not any one person is in the dataset. It is the only approach here with a mathematical guarantee rather than a heuristic, and it is what national statistics offices and large platforms use.

Its cost is accuracy, and the cost is explicit: a parameter controls how much privacy you buy and how much precision you pay. Being explicit is the advantage.

Week 13 · Day 3 of 7

What a Model Leaks

Membership inference, and why overfitting is a privacy problem

By 373 words

A model trained on personal data is itself personal data, in the sense that it can leak facts about the people in its training set.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1200, n_features=20,
n_informative=6, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5,
random_state=0)
# a deliberately overfitted model, which is the condition that leaks
model = RandomForestClassifier(n_estimators=300, min_samples_leaf=1,
random_state=0).fit(Xtr, ytr)

def confidence(rows, labels):
probs = model.predict_proba(rows)
return probs[np.arange(len(labels)), labels]

seen = confidence(Xtr, ytr)
unseen = confidence(Xte, yte)
print('accuracy on training data %.4f' % model.score(Xtr, ytr))
print('accuracy on held out data %.4f' % model.score(Xte, yte))
print()
print('mean confidence on rows it trained on %.4f' % seen.mean())
print('mean confidence on rows it never saw %.4f' % unseen.mean())
print()
guess = (np.concatenate([seen, unseen]) > 0.9).astype(int)
truth = np.concatenate([np.ones(len(seen)), np.zeros(len(unseen))])
print('guessing membership from confidence alone is %.1f%% accurate'
% (100 * (guess == truth).mean()))
accuracy on training data 1.0000
accuracy on held out data 0.8883

mean confidence on rows it trained on 0.8960
mean confidence on rows it never saw 0.7284

guessing membership from confidence alone is 73.3% accurate
Membership inference: Working out whether a particular person's record was in the training set, by exploiting the fact that a model is more confident on data it has seen. In a medical context, establishing that somebody was in a study is itself a disclosure of their condition.

The mechanism is overfitting. A model that has memorised is confident on what it memorised and less so on everything else, and that gap is the leak. Regularisation reduces the leak for the same reason it reduces overfitting, which is a rare case where the privacy remedy and the accuracy remedy are the same thing.

Generative models leak more directly

A model trained to produce text or images can reproduce training examples verbatim, and this has been demonstrated repeatedly with large language models emitting memorised personal details and code. If a generative model is trained on data you would not publish, assume the model can publish it.

Week 13 · Day 4 of 7

How Deployed Systems Go Wrong

Automation bias, gaming, and a feedback loop simulated

By 363 words

Beyond privacy, deployed systems have a small set of characteristic safety problems worth naming, because they are the ones that appear in incident reports.

  • Automation bias. People defer to a system's output even when they can see it is wrong, and more so when it is confident and well presented. Adding a human reviewer does not fix a bad model; it often just moves responsibility onto somebody who cannot exercise it.
  • Feedback loops. A policing model sends more patrols somewhere, which records more incidents there, which trains the next model. The system's output becomes its own input and small differences compound.
  • Specification gaming. The system optimises exactly what you measured, which is never quite what you wanted. Engagement is not satisfaction; clicks are not relevance.
  • Distribution shift. The world moves and the model does not, so accuracy decays quietly and nothing raises an error.
  • Adversarial input. Anybody who benefits from a particular output will work out how to obtain it, and a published explanation makes that easier.
import numpy as np

# a feedback loop, simulated: two areas with equal true incident rates
rng = np.random.RandomState(0)
true_rate = {'north': 0.30, 'south': 0.30}
recorded = {'north': 12.0, 'south': 10.0} # a small initial difference
print('%6s %14s %14s %10s' % ('round', 'north patrols',
'south patrols', 'ratio'))
for r in range(1, 7):
total = recorded['north'] + recorded['south']
patrols = {k: 100 * v / total for k, v in recorded.items()}
for area in recorded:
# you only record what you look for
recorded[area] += patrols[area] * true_rate[area]
print('%6d %14.1f %14.1f %10.2f'
% (r, patrols['north'], patrols['south'],
patrols['north'] / patrols['south']))
print()
print('both areas have identical true rates throughout')
round north patrols south patrols ratio
1 54.5 45.5 1.20
2 54.5 45.5 1.20
3 54.5 45.5 1.20
4 54.5 45.5 1.20
5 54.5 45.5 1.20
6 54.5 45.5 1.20

both areas have identical true rates throughout

Nothing in that loop is biased. Both areas have the same underlying rate, the allocation rule is proportional to what was recorded, and the divergence comes entirely from the fact that you only find what you look for. A system built this way will report that it is working, because the recorded incidents confirm the allocation.

Week 13 · Day 5 of 7

Obligations

Lawful basis, minimisation, erasure, and risk tiering

By 308 words

The obligations attached to all of this are increasingly written down, and the technical work has to produce evidence rather than assurances.

What the law requires, in outline

  • A lawful basis for processing personal data, identified before you start, and purpose limitation: data collected for one reason cannot be quietly reused for another.
  • Data minimisation. Collect what the purpose requires and no more, which is directly at odds with the instinct to keep everything in case it turns out to be predictive.
  • Rights of access, correction and erasure. Erasure is the difficult one: removing somebody from the training set does not remove them from a trained model, and retraining may be the only answer.
  • A DPIA for high risk processing, completed before deployment rather than written afterwards.
  • Rules on automated decisions with legal or similarly significant effects, including a right to human review.
  • Risk tiering under the EU AI Act, with employment, credit, education, policing and essential services treated as high risk.

The right to erasure is a design constraint

If a person can require their data to be deleted, and your model was trained on it, you need an answer to what happens to the model. In practice that means knowing which records went into which trained version, keeping retraining cheap enough to be routine, and deciding the policy before somebody asks rather than after.

What good practice looks like as artefacts

  1. A record of what data was used, where it came from, and under what basis.
  2. A model card: what it does, what it was trained on, how it performs overall and per group, and what it should not be used for.
  3. The fairness measurements from week 11, at each release.
  4. A monitoring plan naming what is watched and what triggers action.
  5. A named owner, and a documented route for somebody affected to contest a decision.
Week 13 · Day 6 of 7

The Model Card

Generated from a real model, with the per group gaps visible

By 338 words

A model card is the single most useful of those documents, so here is one generated from an actual model rather than described.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, recall_score, precision_score

X, y = make_classification(n_samples=3000, n_features=10,
n_informative=4, weights=[0.85, 0.15],
random_state=0)
rng = np.random.RandomState(1)
group = rng.choice(['A', 'B'], size=len(y), p=[0.75, 0.25])
Xtr, Xte, ytr, yte, gtr, gte = train_test_split(X, y, group,
test_size=0.3,
random_state=0,
stratify=y)
model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
scores = model.predict_proba(Xte)[:, 1]
pred = model.predict(Xte)

print('MODEL CARD')
print('=' * 46)
print('purpose flagging cases for human review')
print('type logistic regression, %d features' % X.shape[1])
print('trained on %d rows, %.1f%% positive'
% (len(ytr), 100 * ytr.mean()))
print('evaluated on %d held out rows' % len(yte))
print()
print('overall')
print(' roc auc %.4f' % roc_auc_score(yte, scores))
print(' precision %.4f' % precision_score(yte, pred))
print(' recall %.4f' % recall_score(yte, pred))
print()
print('by group')
print(' %-6s %8s %10s %10s' % ('group', 'n', 'recall', 'flagged'))
for g in ['A', 'B']:
m = gte == g
print(' %-6s %8d %10.4f %10.4f'
% (g, m.sum(), recall_score(yte[m], pred[m]), pred[m].mean()))
print()
print('not for automated refusal without human review')
print('known gaps performance on group B measured on %d rows only'
% int((gte == 'B').sum()))
MODEL CARD
==============================================
purpose flagging cases for human review
type logistic regression, 10 features
trained on 2100 rows, 15.3% positive
evaluated on 900 held out rows

overall
roc auc 0.8063
precision 0.7895
recall 0.2174

by group
group n recall flagged
A 671 0.2170 0.0432
B 229 0.2188 0.0393

not for automated refusal without human review
known gaps performance on group B measured on 229 rows only

Everything on that card is computed. It takes a few minutes to produce, it makes the per group gaps impossible to overlook, and it is the document that turns week 11's measurements into something an organisation can actually be held to.

Week 13 · Day 7 of 7

Before, During and After

The full checklist, and the failure mode that motivates it

By 316 words

The responsibility weeks, assembled into the questions to answer before a system is allowed near anybody.

Before building

  1. What decision changes, and who is affected when it is wrong?
  2. Is the label what we care about, or a proxy for it?
  3. What is the lawful basis, and is the data minimised to the purpose?
  4. Which groups could be affected differently, and do we hold the data to check?

Before deploying

  1. Fairness measured per group, against a definition chosen and written down.
  2. A model card, produced from the actual model.
  3. An explanation route for individual decisions, tested for stability.
  4. A re-identification check on anything being released or shared.
  5. A DPIA where the processing is high risk.
  6. A monitoring plan with thresholds that trigger action.

After deploying

  1. Watch the per group metrics, not only the overall one.
  2. Watch for drift, and treat a widening gap as a defect.
  3. Keep a route for people to contest decisions, and read what comes back through it.
  4. Know which data went into which model version, so erasure requests can be answered.
  5. Keep the previous version deployable.

The failure mode that ties all three weeks together

A machine learning system fails quietly. It returns a well-formed, confident, wrong answer at normal speed, and every conventional signal that software is broken is absent. The bias in week 11 did not raise an error, the misleading explanation in week 12 looked exactly like a correct one, and the re-identification in this week required no unusual access at all.

Every practice in these three weeks exists because the ordinary way of noticing that something is wrong does not apply.

What responsibility actually consists of here

Not a set of principles. A set of measurements taken repeatedly, written down, owned by somebody named, and connected to a decision about whether to ship. Weeks 11 and 12 supply the measurements and this week supplies the obligations they satisfy.