Bias and Fairness

Week 11 of 14 · Responsibility · 7 days

Full curriculum
Week 11 · Responsibility

Bias and Fairness

Week 11 · Day 1 of 7

Where Bias Comes From

A lending dataset whose truth we know by construction

By 448 words

The remaining weeks are about the consequences of deploying the things the first ten built. This is not an appendix. Week 6's embedding learned who does which job, and week 10's model learned a stuck pixel, and both were doing exactly what they were built to do.

Algorithmic bias: A system producing systematically different outcomes for different groups of people, in a way that is not justified by the decision being made. It usually arises from the data rather than the algorithm, which is why it survives changing the model.

A lending dataset with a known history

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
print(df.groupby('group')[FEATURES + ['repays']].mean().round(2))
print()
print('group sizes: %s' % df['group'].value_counts().to_dict())
print('the two groups repay at almost the same rate,')
print('and differ substantially on history and balance')
history income balance missed repays
group
A 7.03 33.03 2087.57 0.89 0.76
B 3.04 32.93 897.57 0.90 0.75

group sizes: {'A': 4217, 'B': 1783}
the two groups repay at almost the same rate,
and differ substantially on history and balance

This data was constructed so that repayment is generated identically for both groups. Any difference in outcome the model produces is therefore not about who repays. It comes from the features, which record a history of unequal access rather than unequal reliability.

Why a synthetic dataset here

Because we get to know the truth. On real data you cannot separate the model is unfair from the groups genuinely differ, and that ambiguity is where most arguments about fairness stall. Here the answer is known by construction, so the measurements can be interpreted without argument.

Week 11 · Day 2 of 7

Fairness Through Unawareness

Removing the attribute, and recovering it from what is left

By 882 words

The obvious first move is to leave the sensitive attribute out. It is also, on its own, close to useless.

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

def fit(features):
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
m.fit(train[features], train['repays'])
return m

def rates(model, features, threshold=0.5):
"""Approval rate and error rates, per group."""
out = {}
scores = model.predict_proba(test[features])[:, 1]
pred = (scores >= threshold).astype(int)
for g in ['A', 'B']:
m = (test['group'] == g).values
truth = test['repays'].values[m]
p = pred[m]
approved = p.mean()
tpr = p[truth == 1].mean()
fpr = p[truth == 0].mean()
out[g] = (approved, tpr, fpr, (p == truth).mean())
return out
model = fit(FEATURES)
print('the model never sees the group column')
print('overall accuracy %.4f' % model.score(test[FEATURES],
test['repays']))
print()
print('%-8s %10s %10s %10s %10s'
% ('group', 'approved', 'tpr', 'fpr', 'accuracy'))
for g, (ap, tpr, fpr, acc) in rates(model, FEATURES).items():
print('%-8s %10.4f %10.4f %10.4f %10.4f' % (g, ap, tpr, fpr, acc))
the model never sees the group column
overall accuracy 0.8883

group approved tpr fpr accuracy
A 0.7869 0.9538 0.3043 0.8875
B 0.8172 0.9532 0.3333 0.8903

Group B is approved at a substantially lower rate, from a model that has no idea group B exists. The information travelled in through credit history and balance, which are correlated with the group and which the model used because they are genuinely predictive of something.

Proxy variable: A feature that carries information about a protected attribute without being it. Postcode for ethnicity, credit history length for age or immigration status, employment gaps for parenthood. Removing the protected attribute leaves every proxy in place.
import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

def fit(features):
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
m.fit(train[features], train['repays'])
return m

def rates(model, features, threshold=0.5):
"""Approval rate and error rates, per group."""
out = {}
scores = model.predict_proba(test[features])[:, 1]
pred = (scores >= threshold).astype(int)
for g in ['A', 'B']:
m = (test['group'] == g).values
truth = test['repays'].values[m]
p = pred[m]
approved = p.mean()
tpr = p[truth == 1].mean()
fpr = p[truth == 0].mean()
out[g] = (approved, tpr, fpr, (p == truth).mean())
return out
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# how well can the group be predicted from the 'neutral' features?
guess = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
guess.fit(train[FEATURES], (train['group'] == 'B').astype(int))
print('predicting group membership from the features alone: %.4f'
% guess.score(test[FEATURES], (test['group'] == 'B').astype(int)))
print('a coin would score about %.4f'
% max((test['group'] == 'B').mean(),
(test['group'] == 'A').mean()))
predicting group membership from the features alone: 0.9389
a coin would score about 0.6961

Fairness through unawareness does not work

The group can be recovered from the remaining columns with high accuracy, so the model has effectively been told. This is the single most common misunderstanding in this area, and it is often written into policy: we do not collect that attribute, therefore we cannot discriminate on it. The opposite is closer to true, because without the attribute you cannot even measure whether you are.

Week 11 · Day 3 of 7

Four Definitions That Disagree

Measured on one model, and why no method will reconcile them

By 561 words

There are several reasonable definitions of a fair decision, they can be measured, and the uncomfortable part is that they disagree.

DefinitionRequiresArgument for it
Demographic parityEqual approval rates across groupsOutcomes should not depend on group
Equal opportunityEqual true positive rates: of those who would repay, equal shares are approvedNobody who deserves it should be refused more often
Equalised oddsEqual true and false positive ratesBoth kinds of error should be shared equally
Predictive parityEqual precision: an approval means the same thing in each groupThe score should mean one thing
import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

def fit(features):
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
m.fit(train[features], train['repays'])
return m

def rates(model, features, threshold=0.5):
"""Approval rate and error rates, per group."""
out = {}
scores = model.predict_proba(test[features])[:, 1]
pred = (scores >= threshold).astype(int)
for g in ['A', 'B']:
m = (test['group'] == g).values
truth = test['repays'].values[m]
p = pred[m]
approved = p.mean()
tpr = p[truth == 1].mean()
fpr = p[truth == 0].mean()
out[g] = (approved, tpr, fpr, (p == truth).mean())
return out
model = fit(FEATURES)
r = rates(model, FEATURES)
a, b = r['A'], r['B']
print('%-26s %10s %10s %10s' % ('', 'group A', 'group B', 'gap'))
for i, name in [(0, 'approval rate'), (1, 'true positive rate'),
(2, 'false positive rate'), (3, 'accuracy')]:
print('%-26s %10.4f %10.4f %10.4f'
% (name, a[i], b[i], a[i] - b[i]))
group A group B gap
approval rate 0.7869 0.8172 -0.0303
true positive rate 0.9538 0.9532 0.0007
false positive rate 0.3043 0.3333 -0.0290
accuracy 0.8875 0.8903 -0.0028

Every one of those gaps is a fairness problem under some definition and acceptable under another. There is no measurement that settles which matters; that is a decision about what the system is for and who bears the cost of each error.

The definitions are mathematically incompatible

It is proved that when the base rates genuinely differ between groups, you cannot have equal precision and equal error rates at the same time, except in degenerate cases. This is not a limitation of current techniques and no future method will resolve it.

So a project cannot be fair in general. It can satisfy a chosen definition, and the choice has to be made explicitly, written down, and defended.

Week 11 · Day 4 of 7

What Can Actually Be Done

Group thresholds, dropping proxies, and the cost of each

By 878 words

Three remedies, applied to this model, measured rather than described.

One: move the threshold per group

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

def fit(features):
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
m.fit(train[features], train['repays'])
return m

def rates(model, features, threshold=0.5):
"""Approval rate and error rates, per group."""
out = {}
scores = model.predict_proba(test[features])[:, 1]
pred = (scores >= threshold).astype(int)
for g in ['A', 'B']:
m = (test['group'] == g).values
truth = test['repays'].values[m]
p = pred[m]
approved = p.mean()
tpr = p[truth == 1].mean()
fpr = p[truth == 0].mean()
out[g] = (approved, tpr, fpr, (p == truth).mean())
return out
import numpy as np

model = fit(FEATURES)
scores = model.predict_proba(test[FEATURES])[:, 1]
is_b = (test['group'] == 'B').values
target = (scores[~is_b] >= 0.5).mean()
# find the threshold for B that gives the same approval rate
best, gap = 0.5, 9.0
for t in np.arange(0.05, 0.95, 0.01):
rate = (scores[is_b] >= t).mean()
if abs(rate - target) < gap:
best, gap = t, abs(rate - target)
print('group A threshold 0.50 approves %.4f' % target)
print('group B threshold %.2f approves %.4f'
% (best, (scores[is_b] >= best).mean()))
pred = np.where(is_b, scores >= best, scores >= 0.5).astype(int)
truth = test['repays'].values
print()
print('overall accuracy before %.4f, after %.4f'
% (((scores >= 0.5).astype(int) == truth).mean(),
(pred == truth).mean()))
group A threshold 0.50 approves 0.7869
group B threshold 0.66 approves 0.7843

overall accuracy before 0.8883, after 0.8850

Demographic parity achieved, at a small cost in accuracy. Note what this required: applying a different rule depending on the group, which means using the protected attribute explicitly. In several jurisdictions that is itself unlawful, so the remedy that most directly equalises outcomes can be the one you are least able to deploy.

Two: drop the features that carry the proxy

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

def fit(features):
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
m.fit(train[features], train['repays'])
return m

def rates(model, features, threshold=0.5):
"""Approval rate and error rates, per group."""
out = {}
scores = model.predict_proba(test[features])[:, 1]
pred = (scores >= threshold).astype(int)
for g in ['A', 'B']:
m = (test['group'] == g).values
truth = test['repays'].values[m]
p = pred[m]
approved = p.mean()
tpr = p[truth == 1].mean()
fpr = p[truth == 0].mean()
out[g] = (approved, tpr, fpr, (p == truth).mean())
return out
model = fit(FEATURES)
reduced = ['income', 'missed']
model2 = fit(reduced)
print('%-34s %10s %10s %10s'
% ('', 'accuracy', 'approve A', 'approve B'))
for name, m, feats in [('all features', model, FEATURES),
('without history and balance', model2, reduced)]:
r = rates(m, feats)
print('%-34s %10.4f %10.4f %10.4f'
% (name, m.score(test[feats], test['repays']),
r['A'][0], r['B'][0]))
accuracy approve A approve B
all features 0.8883 0.7869 0.8172
without history and balance 0.8889 0.7829 0.8172

Three: measure, and keep measuring

The remedy that always applies is the one that sounds least like a solution. Collect the protected attribute, report the gaps for every group at every release, and treat a widening gap as a defect. Without the attribute none of this is possible, which is the argument against the unawareness approach in its most practical form.

Week 11 · Day 5 of 7

The Five Places Bias Enters

From historical data to feedback loops, and the worst one

By 542 words

Bias does not enter at one point. It enters at several, and each has a different remedy.

  1. The historical data. If past decisions were biased, a model trained to reproduce them reproduces the bias faithfully. This is the hardest source because the data looks correct.
  2. What was measured. Credit history length is a fact about access, not reliability. Choosing it as a feature encodes that history into every future decision.
  3. Who is in the sample. A group that is a small share of the training data gets less of the model's capacity, and its error rate is usually higher.
  4. The label itself. Arrested is not committed a crime. Promoted is not performed well. When the label is a proxy for what you care about, its biases become the model's objective.
  5. Deployment. A system used more heavily on one group produces more data about that group, which changes the next model. Feedback loops amplify small initial differences.
import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

# sample size effect: train with group B badly underrepresented
b_rows = df[df['group'] == 'B']
a_rows = df[df['group'] == 'A']
print('%-28s %12s %12s' % ('group B share of training', 'acc A', 'acc B'))
for share in [0.02, 0.10, 0.30]:
k = int(len(a_rows) * share / (1 - share))
sub = pd.concat([a_rows, b_rows.iloc[:k]])
tr, te = train_test_split(sub, test_size=0.3, random_state=0)
m = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=2000))
m.fit(tr[FEATURES], tr['repays'])
accs = []
for g in ['A', 'B']:
rows = te[te['group'] == g]
accs.append(m.score(rows[FEATURES], rows['repays'])
if len(rows) > 20 else float('nan'))
print('%-28.2f %12.4f %12.4f' % (share, accs[0], accs[1]))
group B share of training acc A acc B
0.02 0.8858 0.8571
0.10 0.8861 0.8947
0.30 0.8862 0.8654

The label problem is the one to worry about most

Every other source can be measured and partly corrected. A label that is a biased proxy for the thing you actually care about cannot be, because the model is optimising exactly what you asked for and the evaluation uses the same proxy. Both training and testing agree, and both are wrong in the same direction.

Week 11 · Day 6 of 7

Obligations

Protected characteristics, indirect discrimination, and what has to exist on paper

By 286 words

The obligations attached to this are increasingly legal rather than ethical, and the technical work has to be organised to meet them.

  • Protected characteristics are defined in law and differ by jurisdiction. In the UK the Equality Act names nine.
  • Direct discrimination is treating someone worse because of a characteristic. Indirect discrimination is a neutral rule that disadvantages a group without justification, and it is the one an algorithm produces by default.
  • Automated decision making with legal or similarly significant effects carries specific obligations under UK and EU data protection law, including a right to meaningful information about the logic involved.
  • The EU AI Act classifies systems by risk, with employment, credit, education and essential services treated as high risk and carrying documentation, monitoring and human oversight requirements.

What this means for how you work

It means the fairness measurements have to exist as artefacts, not as something a data scientist checked once. Written down: which definition you chose and why, the gaps at release, who reviewed them, and what the appeal route is for somebody the system refused. If those documents do not exist, the honest position is that the system is not ready.

Questions to answer before deployment

  1. Which groups could this system affect differently, and do we hold the data to check?
  2. Which fairness definition are we committing to, and what did we give up by choosing it?
  3. What is the gap on each metric now, and what size of gap would trigger a change?
  4. Is the label what we actually care about, or a proxy for it?
  5. Can a person who is refused find out why, and contest it?
  6. Who is accountable when it is wrong, and how would we find out?
Week 11 · Day 7 of 7

A Fairness Report

Every measurement in one place, and what it does and does not settle

By 605 words

The week's measurements in one place, on one model, which is what a fairness report should look like.

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
n = 6000

# Two groups, equally creditworthy on average. Group B has historically
# had less access to credit, so its members have shorter credit files
# and lower recorded balances. Nothing here is about ability to repay.
group = rng.choice(['A', 'B'], size=n, p=[0.7, 0.3])
is_b = group == 'B'

# the thing we would like to predict, generated the same way for both
repays = rng.rand(n) < 0.75

# observable features. credit history is systematically shorter for B
history = np.where(is_b, rng.normal(3, 1.6, n), rng.normal(7, 2.2, n))
history = np.clip(history, 0, None)
income = rng.normal(30, 8, n) + repays * 4
balance = np.where(is_b, rng.normal(900, 400, n),
rng.normal(2100, 700, n))
# the genuinely predictive signal, identical in both groups
missed = rng.poisson(np.where(repays, 0.4, 2.4))

df = pd.DataFrame({'history': history, 'income': income,
'balance': balance, 'missed': missed,
'group': group, 'repays': repays.astype(int)})
FEATURES = ['history', 'income', 'balance', 'missed']

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3, random_state=0,
stratify=df['repays'])
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

def fit(features):
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
m.fit(train[features], train['repays'])
return m

def rates(model, features, threshold=0.5):
"""Approval rate and error rates, per group."""
out = {}
scores = model.predict_proba(test[features])[:, 1]
pred = (scores >= threshold).astype(int)
for g in ['A', 'B']:
m = (test['group'] == g).values
truth = test['repays'].values[m]
p = pred[m]
approved = p.mean()
tpr = p[truth == 1].mean()
fpr = p[truth == 0].mean()
out[g] = (approved, tpr, fpr, (p == truth).mean())
return out
model = fit(FEATURES)
r = rates(model, FEATURES)
print('model: logistic regression on %s' % ', '.join(FEATURES))
print('the group attribute is not among them')
print()
print('%-24s %10s %10s %10s' % ('', 'A', 'B', 'gap'))
labels = ['approval rate', 'true positive rate',
'false positive rate', 'accuracy']
for i, name in enumerate(labels):
print('%-24s %10.4f %10.4f %+10.4f'
% (name, r['A'][i], r['B'][i], r['A'][i] - r['B'][i]))
print()
print('known by construction: both groups repay at the same rate,')
print('so every gap above is produced by the features, not by risk')
model: logistic regression on history, income, balance, missed
the group attribute is not among them

A B gap
approval rate 0.7869 0.8172 -0.0303
true positive rate 0.9538 0.9532 +0.0007
false positive rate 0.3043 0.3333 -0.0290
accuracy 0.8875 0.8903 -0.0028

known by construction: both groups repay at the same rate,
so every gap above is produced by the features, not by risk

What the week established

  • Bias usually comes from the data and the labels, so changing the model does not remove it.
  • Removing the protected attribute does not work, because proxies carry it and because you can no longer measure anything.
  • There are several reasonable definitions of fairness and they are mathematically incompatible when base rates differ.
  • The remedies are real but each costs something: accuracy, predictive features, or the legal ability to use the attribute at all.
  • Bias enters at five distinct points, and a biased label is the hardest of them because evaluation shares the bias.
  • The obligations are increasingly statutory, and they require documents rather than good intentions.

The sentence to carry into week 12

You cannot fix what you cannot see, and you cannot see it without collecting the attribute and reporting per group. The next week is about the related problem of seeing inside the model itself, which is what turns the system refused you into something a person can actually contest.