Explaining What a Model Did

Week 12 of 14 · Responsibility · 7 days

Week 12 · Responsibility

Explaining What a Model Did

Week 12 · Day 1 of 7

Readable Models and Stories About Models

Interpretability against explainability, and the accuracy gap between them

Week 11 ended on a system refusing somebody. The obvious next question is why, and for most modern models that question is much harder to answer than it sounds.

Interpretability and explainability: An interpretable model is one a person can read directly, such as a short decision tree or a linear equation. An explanation is an account of a model that is not itself readable, produced after the fact. The distinction matters: the first is the model, the second is a story about it that may be wrong.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
print('%-24s %10s' % ('', 'accuracy'))
print('%-24s %10.4f' % ('logistic regression',
                        linear.score(test[FEATURES], test['repays'])))
print('%-24s %10.4f' % ('random forest',
                        forest.score(test[FEATURES], test['repays'])))
print()
print('the forest contains %d trees' % len(forest.estimators_))
print('the first has %d nodes'
      % forest.estimators_[0].tree_.node_count)

                           accuracy
logistic regression          0.8442
random forest                0.8442

the forest contains 250 trees
the first has 489 nodes

Two models at similar accuracy. One is five numbers you can print. The other is two hundred and fifty trees with thousands of nodes each, and no person will ever read it. When the accuracies are this close, the readable one is often the better engineering choice for reasons that have nothing to do with accuracy.

The readable model, read

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
import numpy as np

clf = linear.named_steps['logisticregression']
for name, w in sorted(zip(FEATURES, clf.coef_[0]),
                      key=lambda t: -abs(t[1])):
    print('%-10s %+8.3f' % (name, w))
print()
print('these are the whole model: multiply, add, done')

missed       -3.386
income       +1.107
history      +0.531
salary       +0.114
noise        +0.021

these are the whole model: multiply, add, done

Week 12 · Day 2 of 7

What Matters Overall

Built-in importance, its bias, and permutation instead

By Pawan Rama Mali597 words

For models that cannot be read, the standard first tool is to ask which features matter overall. There are two common ways to answer, and one of them is misleading often enough to be worth avoiding.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
import numpy as np

print('%-10s %14s' % ('feature', 'built-in importance'))
for name, imp in sorted(zip(FEATURES, forest.feature_importances_),
                        key=lambda t: -t[1]):
    print('%-10s %14.4f' % (name, imp))

feature    built-in importance
missed             0.5774
income             0.1248
salary             0.1216
history            0.1018
noise              0.0744

The built-in number is biased towards certain features

Tree importance counts how often a feature was split on and how much it reduced impurity. Features with many distinct values offer more places to split, so continuous and high cardinality features score higher than they deserve, and a pure noise column with thousands of distinct values can outrank a genuinely useful binary one.

Permutation importance: Shuffle one column and measure how much the score falls. If performance drops, the model was relying on that column. It measures the model as used rather than as built, and it works for any model.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
from sklearn.inspection import permutation_importance
import numpy as np

r = permutation_importance(forest, test[FEATURES], test['repays'],
                           n_repeats=10, random_state=0)
print('%-10s %14s %10s' % ('feature', 'permutation', 'std'))
order = np.argsort(r.importances_mean)[::-1]
for i in order:
    print('%-10s %14.4f %10.4f'
          % (FEATURES[i], r.importances_mean[i], r.importances_std[i]))

feature       permutation        std
missed             0.2979     0.0140
income             0.0277     0.0062
history            0.0216     0.0033
salary             0.0151     0.0048
noise              0.0036     0.0032

The noise column, which is unrelated to the outcome by construction, should sit at approximately zero. Anything meaningfully above it is carrying real signal, and that gives you a reference line the raw importance numbers do not provide.

Week 12 · Day 3 of 7

The Trap of Correlated Features

Two columns that hide each other, and how to catch it

By Pawan Rama Mali621 words

There is a trap in importance measures that catches people constantly, and this dataset contains it deliberately.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
from sklearn.inspection import permutation_importance
import numpy as np

print('income and salary are nearly the same column: correlation %.3f'
      % np.corrcoef(df['income'], df['salary'])[0, 1])
print()
r = permutation_importance(forest, test[FEATURES], test['repays'],
                           n_repeats=10, random_state=0)
for name, m in zip(FEATURES, r.importances_mean):
    print('%-10s %10.4f' % (name, m))
print()
print('shuffling either one alone barely hurts, because the model')
print('can read the same information from the other')

income and salary are nearly the same column: correlation 0.994

history        0.0216
income         0.0277
salary         0.0151
missed         0.2979
noise          0.0036

shuffling either one alone barely hurts, because the model
can read the same information from the other

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
import numpy as np
from sklearn.ensemble import RandomForestClassifier

base = forest.score(test[FEATURES], test['repays'])
shuffled = test.copy()
rng = np.random.RandomState(1)
for col in ['income', 'salary']:
    shuffled[col] = rng.permutation(shuffled[col].values)
print('accuracy with both income columns shuffled together %.4f'
      % forest.score(shuffled[FEATURES], shuffled['repays']))
print('accuracy untouched                                  %.4f' % base)
print()
print('so income does matter a great deal, and testing each')
print('column on its own said otherwise')

accuracy with both income columns shuffled together 0.7933
accuracy untouched                                  0.8442

so income does matter a great deal, and testing each
column on its own said otherwise

Correlated features hide each other

Any importance method that removes one feature at a time will understate every feature that has a substitute. In real data, duplicated and near-duplicated columns are extremely common: salary and annual pay, age and date of birth, several encodings of a postcode.

The remedy is to group correlated features and permute the group, as above. A ranked list of individually permuted features is one of the most commonly misread outputs in this field.

Week 12 · Day 4 of 7

Explaining One Decision

Local probing, and the counterfactual people actually want

By Pawan Rama Mali647 words

Global importance says what matters on average. Somebody refused a loan wants to know about their own case, which is a different question with different tools.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
import numpy as np

row = test.iloc[[0]]
prob = forest.predict_proba(row[FEATURES])[0][1]
print('this applicant scores %.3f' % prob)
print(row[FEATURES].round(2).to_string(index=False))
print()
print('%-10s %10s %12s' % ('change', 'to', 'new score'))
for feat in FEATURES:
    for direction in [-1, 1]:
        probe = row.copy()
        step = train[feat].std() * direction
        probe[feat] = probe[feat] + step
        new = forest.predict_proba(probe[FEATURES])[0][1]
        if abs(new - prob) > 0.02:
            print('%-10s %10.2f %12.3f'
                  % (feat, float(probe[feat].iloc[0]), new))

this applicant scores 0.814
 history  income  salary  missed  noise
    5.67   39.12   39.92       1   0.81

change             to    new score
history          3.27        0.745
income          30.24        0.514
income          48.00        0.715
salary          30.98        0.573
salary          48.86        0.693
missed          -0.04        0.994
missed           2.04        0.030
noise           -0.18        0.793
noise            1.80        0.722

Changing one input at a time and watching the score is the simplest possible local explanation, and it is often enough. It answers the question people actually ask, which is not why but what would have to be different.

Counterfactual explanation: The smallest change to the inputs that would change the decision. It is usually more useful than an attribution, because it is actionable: with one fewer missed payment this would have been approved tells somebody what to do, and a list of weights does not.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
import numpy as np

refused = test[forest.predict(test[FEATURES]) == 0].iloc[[0]]
print('refused, score %.3f'
      % forest.predict_proba(refused[FEATURES])[0][1])
print()
found = False
for feat in ['missed', 'income', 'history']:
    for step in np.arange(0.5, 6.1, 0.5):
        probe = refused.copy()
        delta = -step if feat == 'missed' else step
        probe[feat] = probe[feat] + delta
        if forest.predict(probe[FEATURES])[0] == 1:
            print('would be approved with %s changed by %+.1f'
                  % (feat, delta))
            found = True
            break
if not found:
    print('no single change within the range flips this decision')

refused, score 0.315

would be approved with missed changed by -0.5

Week 12 · Day 5 of 7

What Shape Is the Relationship

Partial dependence, and the combinations that never happen

By Pawan Rama Mali372 words

A third question: not which features matter, but what the model believes the relationship to be.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
from sklearn.inspection import partial_dependence
import numpy as np

for feat in ['missed', 'income']:
    pd_result = partial_dependence(forest, test[FEATURES], [feat],
                                   grid_resolution=6,
                                   response_method='predict_proba')
    values = pd_result['grid_values'][0]
    average = pd_result['average'][0]
    print('%s:' % feat)
    for v, a in zip(values, average):
        bar = '#' * int(a * 40)
        print('   %8.2f  %.3f %s' % (v, a, bar))
    print()

missed:
       0.00  0.910 ####################################
       0.60  0.462 ##################
       1.20  0.462 ##################
       1.80  0.079 ###
       2.40  0.079 ###
       3.00  0.045 #

income:
      17.73  0.401 ################
      23.59  0.471 ##################
      29.46  0.491 ###################
      35.33  0.505 ####################
      41.19  0.544 #####################
      47.06  0.587 #######################

Missed payments push the score down and income pushes it up, which is how the data was generated. Seeing the shape is the point: a relationship that should be smooth and is not, or that reverses direction somewhere implausible, is a sign of a problem the accuracy will not show you.

Partial dependence assumes the features are independent

It computes the average prediction as one feature is swept, holding the others as they are. When features are correlated this evaluates combinations that do not occur in reality, such as a very low income with a very high salary in this dataset. The resulting curve is a statement about the model in regions it was never trained on.

Week 12 · Day 6 of 7

Explanations Can Be Wrong

Approximate, unstable, gameable, and what to do instead

By Pawan Rama Mali481 words

Explanations are themselves models, and they can be wrong. Three things worth knowing before presenting one to anybody.

They are approximations. A local explanation fits a simple model near one point. If the real model is not simple there, the explanation is a poor fit and nothing reports that.
They can be unstable. Two nearly identical applicants can receive quite different explanations, which is difficult to defend when both are shown to the same regulator.
They can be gamed. If a system publishes which features drive its decisions, people optimise for those features, which is rational and which breaks the model.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
import numpy as np

# how much does the local explanation move for a tiny input change?
row = test.iloc[[3]]
rng = np.random.RandomState(0)
print('%-10s %12s %12s' % ('feature', 'effect A', 'effect B'))
effects = []
for jitter in [0.0, 0.02]:
    probe = row.copy()
    for f in FEATURES:
        probe[f] = probe[f] * (1 + jitter)
    base = forest.predict_proba(probe[FEATURES])[0][1]
    row_effects = []
    for feat in FEATURES:
        p2 = probe.copy()
        p2[feat] = p2[feat] + train[feat].std()
        row_effects.append(forest.predict_proba(p2[FEATURES])[0][1] - base)
    effects.append(row_effects)
for i, feat in enumerate(FEATURES):
    print('%-10s %12.3f %12.3f' % (feat, effects[0][i], effects[1][i]))
print()
print('the inputs differ by 2 percent; the attributions differ by more')

feature        effect A     effect B
history           0.025        0.010
income            0.248        0.242
salary            0.367        0.355
missed           -0.010       -0.018
noise            -0.016       -0.023

the inputs differ by 2 percent; the attributions differ by more

The most reliable route to an explainable system

Use a model that does not need explaining. A logistic regression or a shallow decision tree is its own explanation, it cannot disagree with itself, and on tabular data of this kind it is frequently within a point or two of the complicated alternative.

Day 1 measured exactly that. When the gap is small, choosing the readable model buys you contestability, stability and a much shorter conversation with whoever has to sign it off.

Week 12 · Day 7 of 7

Four Questions, Four Tools

Assembled, with the checklist and the legal reason it matters

By Pawan Rama Mali517 words

The four questions and the tool for each, applied to one model.

Question	Tool	Watch out for
What matters overall	Permutation importance	Correlated features hiding each other
Why this case	One at a time probing, or a local attribution method	Instability between similar cases
What would change it	Counterfactual search	Suggesting changes nobody can actually make
What is the relationship	Partial dependence	Regions the model never saw

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.RandomState(0)
n = 4000
history = np.clip(rng.normal(6, 2.5, n), 0, None)
income = rng.normal(32, 9, n)
missed = rng.poisson(1.1, n)
noise = rng.normal(0, 1, n)
# a copy of income, to show what correlated features do to explanations
salary = income + rng.normal(0, 1.0, n)

risk = (-1.4 * missed + 0.06 * income + 0.10 * history
        + rng.normal(0, 0.8, n))
repays = (risk > np.median(risk)).astype(int)

df = pd.DataFrame({'history': history, 'income': income,
                   'salary': salary, 'missed': missed,
                   'noise': noise, 'repays': repays})
FEATURES = ['history', 'income', 'salary', 'missed', 'noise']
train, test = train_test_split(df, test_size=0.3, random_state=0,
                               stratify=df['repays'])
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

forest = RandomForestClassifier(n_estimators=250, random_state=0,
                                min_samples_leaf=3)
forest.fit(train[FEATURES], train['repays'])
linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
linear.fit(train[FEATURES], train['repays'])
from sklearn.inspection import permutation_importance
import numpy as np

print('accuracy: linear %.4f, forest %.4f'
      % (linear.score(test[FEATURES], test['repays']),
         forest.score(test[FEATURES], test['repays'])))
print()
r = permutation_importance(forest, test[FEATURES], test['repays'],
                           n_repeats=10, random_state=0)
print('what the forest relies on:')
for i in np.argsort(r.importances_mean)[::-1]:
    print('   %-10s %8.4f' % (FEATURES[i], r.importances_mean[i]))
print()
clf = linear.named_steps['logisticregression']
print('what the linear model says, which is the model itself:')
for name, w in sorted(zip(FEATURES, clf.coef_[0]),
                      key=lambda t: -abs(t[1])):
    print('   %-10s %+8.3f' % (name, w))

accuracy: linear 0.8442, forest 0.8442

what the forest relies on:
   missed       0.2979
   income       0.0277
   history      0.0216
   salary       0.0151
   noise        0.0036

what the linear model says, which is the model itself:
   missed       -3.386
   income       +1.107
   history      +0.531
   salary       +0.114
   noise        +0.021

The checklist

Try the interpretable model first and record the accuracy gap. It is often small enough to settle the question.
Use permutation importance rather than the built-in numbers, and include a pure noise column as a reference line.
Group correlated features before permuting anything.
For individual decisions, prefer counterfactuals: people want to know what to change.
Check whether the explanation is stable across similar cases before you show it to anybody.
Never present an explanation as what the model did. It is a model of what the model did.

Why this is a practical subject and not a philosophical one

Somebody refused a loan has a legal right to meaningful information about the logic involved. A system that cannot produce it is not merely opaque, it may be undeployable. The cheapest way to satisfy that is usually to choose a model that explains itself, and the measurement on day 1 is what tells you whether you can afford to.