Why 97% Accuracy Can Mean Your Model Learned Nothing

Reading a confusion matrix when one class barely exists

By | 11 min read | Level: Beginner to Intermediate | Category: Machine Learning

What We'll Cover

Introduction

Every batch has the same moment. A student trains their first classifier on a fraud dataset, or a churn dataset, or a medical screening dataset. The notebook prints an accuracy of 0.97 and they are delighted.

Then we ask them to print the confusion matrix, and the delight goes away. The model has learned to say "no" to everything, and because "no" is right 97% of the time, accuracy rewards it handsomely.

Why This Matters

This is not a beginner mistake you grow out of. Production models get shipped on the strength of an accuracy number every week. The problems worth solving with classification are almost all imbalanced, because the interesting event is rare. Fraud is rare. Equipment failure is rare. Disease in a screening population is rare. If your positive class were common, you would not need a model to find it.

1

The 97% model that predicts one thing

Here is the whole problem in eight lines. A dataset of 1,000 transactions where 30 are fraudulent, and a "model" that never predicts fraud at all.

# A model that has learned absolutely nothing import numpy as np from sklearn.metrics import accuracy_score, confusion_matrix y_true = np.array([1]*30 + [0]*970) # 30 frauds in 1000 y_pred = np.zeros(1000, dtype=int) # predict "not fraud" every time print(accuracy_score(y_true, y_pred)) # 0.97 print(confusion_matrix(y_true, y_pred))

The accuracy is 0.97. The confusion matrix tells the truth:

[[970 0] # 970 genuine correctly passed, 0 flagged wrongly [ 30 0]] # 30 frauds, all of them missed

Every fraud in the dataset went through. The model caught nothing. And it scored 97%.

Rule of thumb

Before you look at any metric, work out what accuracy a model would get by always predicting the majority class. If your model is not comfortably beating that, it has not learned anything useful. That baseline takes ten seconds to compute and it will save you from a lot of embarrassment.

2

The four boxes, and what each one costs

A confusion matrix has four cells and they are not equally important. Naming them abstractly is what makes them confusing. Name them in terms of your actual problem and they become obvious.

For a fraud detector:

  • True negative: a genuine transaction, correctly allowed through. Costs nothing. There are thousands of these and they dominate accuracy.
  • True positive: a fraud, correctly caught. This is the whole point of the system.
  • False positive: a genuine transaction, wrongly blocked. A customer is annoyed, someone in operations reviews it, the cost is real but bounded.
  • False negative: a fraud, missed. The money is gone.

Now the important question, and it is not a statistics question: how many false positives would you accept to avoid one false negative? For fraud on large transactions, the answer might be fifty. For a spam filter, where a false positive means a customer's invoice ends up in junk, the answer might be less than one.

That ratio is a business decision. It is the input to everything that follows, and no metric can supply it for you.

3

Precision and recall, without the formulas first

Students memorise these formulas and then cannot say which one they need. Learn the sentences instead, and the formulas will follow.

Precision answers: when the model raises an alarm, how often is it right? It is the question the person handling the alerts asks.

Recall answers: of all the real cases out there, how many did the model catch? It is the question the person accountable for the losses asks.

# precision = TP / (TP + FP) "of what I flagged, how much was real" # recall = TP / (TP + FN) "of what was real, how much did I flag" from sklearn.metrics import classification_report print(classification_report(y_true, y_pred, target_names=['genuine', 'fraud']))

Notice that neither formula contains true negatives. That is exactly why they survive imbalance: the enormous, uninformative majority class cannot inflate them.

Which one to optimise

Optimise recall when missing a case is dangerous or expensive and a false alarm is merely annoying: disease screening, safety monitoring, fraud on high-value accounts. Optimise precision when acting on the prediction is costly or intrusive and missing one is survivable: sending a field engineer, blocking an account, cold-calling a customer.

4

F1, and when it is the wrong compromise

F1 is the harmonic mean of precision and recall. The harmonic mean, rather than the ordinary average, because it punishes imbalance between the two: a model with precision 1.0 and recall 0.0 gets an F1 of 0, where a plain average would give it 0.5.

That makes F1 a reasonable single number when you genuinely have no view on which error matters more. Reporting it in a competition or a course exercise is fine.

Using it as your production objective usually is not, because it hard-codes an assumption you probably disagree with: that a false positive and a false negative cost the same. If you decided in section 2 that you would accept fifty false alarms to catch one more fraud, F1 does not encode that at all.

When you have a real cost ratio, use it. Weighting recall more heavily is what the F-beta score does, and it is one argument:

from sklearn.metrics import fbeta_score # beta > 1 weights recall higher; beta = 2 is the common choice fbeta_score(y_true, y_pred, beta=2)
5

The threshold is your decision, not the model's

This is the part most courses skip, and it is the part that matters most in a real job.

A classifier does not output a class. It outputs a probability. Calling .predict() applies a threshold of 0.5 and hands you a class, and 0.5 is a default, not an answer. Nobody chose it for your problem.

On imbalanced data 0.5 is almost always wrong, because the model rarely gets confident enough about the rare class to cross it.

from sklearn.metrics import precision_recall_curve probs = model.predict_proba(X_test)[:, 1] precision, recall, thresholds = precision_recall_curve(y_test, probs) # the lowest threshold that still keeps precision acceptable for p, r, t in zip(precision, recall, thresholds): if p >= 0.30: print(f"threshold {t:.3f} -> precision {p:.2f}, recall {r:.2f}") break

Read that loop carefully, because it is how the decision actually gets made. You state what you can tolerate, then find the threshold that delivers it. You do not accept a threshold and then discover what you got.

In an interview

If you are asked how you would improve a poorly performing classifier and your first answer is "tune the threshold and look at the precision-recall curve", you will sound like someone who has shipped a model. Most candidates say "try a different algorithm".

6

ROC AUC looks fine when things are not

ROC AUC is the metric people reach for once they have learned that accuracy is unreliable. It is better, but it has its own blind spot on heavily imbalanced data.

The reason is in the false positive rate, which divides by the number of genuine negatives. When that number is enormous, adding hundreds of false positives barely moves it. So the curve stays high and the AUC stays impressive while the alert queue fills with noise.

The precision-recall curve does not have this problem, because precision divides by the number of things you flagged. Add false positives and precision drops immediately and visibly.

Report both if you like, but on a rare-event problem make the precision-recall curve the one you make decisions from. Its baseline is worth knowing too: a random model sits at the positive class rate, so 0.03 in our example, not at 0.5.

7

What to do about the imbalance itself

Three approaches, in the order we would try them.

Class weights first. Most estimators accept class_weight='balanced', which tells the loss function to care more about the rare class. It changes nothing about your data, adds no artefacts, and takes one argument. Start here.

from sklearn.ensemble import RandomForestClassifier model = RandomForestClassifier(class_weight='balanced', random_state=42)

Then resampling, carefully. Oversampling the minority or undersampling the majority both work. SMOTE, which generates synthetic minority examples by interpolating between neighbours, often works well.

The critical rule: resample inside the cross-validation loop, on the training fold only. Resampling the whole dataset before splitting copies minority examples into both training and validation, and your scores become fiction. Use a pipeline so this cannot happen by accident.

from imblearn.pipeline import Pipeline from imblearn.over_sampling import SMOTE # SMOTE runs per fold, on training data only pipe = Pipeline([ ('smote', SMOTE(random_state=42)), ('model', RandomForestClassifier(random_state=42)), ])

Then better features. Honestly, this is where the largest gains usually are. A model that cannot separate the classes is often a model that has not been given the column that separates them. Time since last transaction, deviation from the customer's own average, hour of day: these beat any amount of resampling on a fraud problem.

The short version

  • Compute the majority-class baseline before celebrating any accuracy
  • Print the confusion matrix every time, not just the score
  • Decide what a false positive and a false negative each cost, in your terms
  • Choose precision or recall from that, not from habit
  • Tune the threshold deliberately; 0.5 is a default, not a decision
  • Prefer the precision-recall curve to ROC on rare-event problems
  • Try class weights before resampling, and resample inside the fold

Learn model evaluation properly with EDUSHARK TRAINING

Model evaluation is where most self-taught learners have gaps, and it is what interviewers probe first. Our Data Analytics and machine learning programmes cover it with real datasets rather than clean toy examples.

View Our Data Analytics Program
Back to All Blogs Next: Train, Validation, Test