Train, Validation, Test: The Splits That Quietly Break Your Model

Why 0.94 in the notebook becomes 0.61 in production

By | 12 min read | Level: Beginner to Intermediate | Category: Machine Learning

What We'll Cover

Introduction

The single most common message we get from past students goes something like this: the model scored 0.94 in the notebook, it has been live for two weeks, and it is performing about as well as a coin toss.

Nine times out of ten, nothing is wrong with the model. Something was wrong with how the data was split, and the 0.94 was never real.

Why This Matters

Leakage is dangerous precisely because it is silent. It has no error message and no warning. It makes your numbers better, which is the direction you were hoping for, so nobody investigates. You find out when the model reaches production, which is the most expensive possible moment.

1

Three sets, three different jobs

Most people can recite the three names. Fewer can say why there are three rather than two, and that is the part that matters.

  • Training set. The model fits its parameters here. It is allowed to see this as many times as it likes.
  • Validation set. You compare options here: which algorithm, which hyperparameters, which features. You look at this repeatedly.
  • Test set. You touch it once, at the end, to estimate how the chosen model will perform on data it has never influenced.

The third set exists because of what you do with the second. Every time you try a configuration and keep the one that scores best on validation, you fit a little bit of that validation set. Do it fifty times and the validation score is optimistic, because you selected for it.

That is not cheating, it is just what selection does. The test set is the untouched sample that lets you find out how much optimism accumulated.

Practical version

With plenty of data, split once into three: roughly 60/20/20. With less data, split into train and test, then use cross-validation inside the training portion instead of a fixed validation set. Either way, the test set is set aside first and not looked at again.

2

Scaling before splitting, the classic leak

This appears in more tutorials than we would like, and it is wrong every time.

# WRONG: the scaler sees the test data scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # fitted on everything X_train, X_test, y_train, y_test = train_test_split( X_scaled, y, test_size=0.2, random_state=42)

The scaler computed a mean and a standard deviation using every row, including the ones about to become the test set. Those statistics then went into the training features. Information from the test set is now in the training data.

The right order is split first, fit the scaler on the training data only, and apply the same fitted scaler to the test data.

# RIGHT: split first, fit only on training data X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42) scaler = StandardScaler() X_train = scaler.fit_transform(X_train) # fit AND transform X_test = scaler.transform(X_test) # transform only

The same rule applies to everything that learns from data before the model does: imputers filling missing values with a mean, encoders building a category list, feature selectors ranking columns, PCA finding components. If the step has a fit, it has to be fitted on training data only.

The effect is usually small on a large clean dataset and large on a small messy one, which is the wrong way round for beginners, because beginners work with small messy datasets.

3

Pipelines make leakage hard to write

Remembering the rule works until you have six preprocessing steps and are cross-validating. Then it stops working, because the rule has to hold inside every fold and nobody tracks that by hand.

A pipeline solves it structurally. Everything inside is fitted on whatever the pipeline is currently being fitted on, which during cross-validation is the training fold.

from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.impute import SimpleImputer from sklearn.linear_model import LogisticRegression from sklearn.model_selection import cross_val_score pipe = Pipeline([ ('impute', SimpleImputer(strategy='median')), ('scale', StandardScaler()), ('model', LogisticRegression(max_iter=1000)), ]) # each fold: impute and scale fitted on that fold's training part only scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring='f1')

Habit worth forming

Put every preprocessing step in a pipeline from the first line of the project, even when it feels like overkill for one scaler. It costs nothing and it removes a whole category of mistake permanently. It also means the thing you deploy is the same object you validated.

4

If the data has time in it, do not shuffle

A random split assumes rows are interchangeable. For anything with a time dimension, they are not.

Shuffling sales data from 2023 to 2026 puts December 2025 in the training set and June 2025 in the test set. The model is being asked to predict the past having seen the future, which is not the task it will face, and it will do unrealistically well at it because the surrounding weeks are in training.

Split by time instead, and validate by moving the boundary forward.

from sklearn.model_selection import TimeSeriesSplit df = df.sort_values('date') # each split trains on the past, validates on the period straight after tscv = TimeSeriesSplit(n_splits=5) for train_idx, val_idx in tscv.split(df): train, val = df.iloc[train_idx], df.iloc[val_idx] # fit on train, score on val

This applies well beyond forecasting. Churn, fraud, credit risk and demand prediction are all time-ordered even when the target does not look like a time series, because behaviour and conditions drift. If your model will be used on tomorrow's data, validate it on data that came after the training data.

5

Grouped data needs GroupKFold

The subtlest of the common leaks. It happens whenever one real-world entity contributes several rows.

One patient with twelve visits. One customer with two hundred transactions. One machine with a year of daily readings. Split those rows randomly and the same patient appears in training and in test. The model does not have to learn anything general; it can learn that patient.

The score looks excellent. Then the model meets a patient it has never seen, which is the only situation that will ever occur in production, and it fails.

from sklearn.model_selection import GroupKFold # a patient's rows all land on the same side of every split gkf = GroupKFold(n_splits=5) for train_idx, val_idx in gkf.split(X, y, groups=df['patient_id']): ...

Before you split anything, ask one question: does a single real-world thing appear more than once in these rows? If yes, that identifier is your group, and the split has to respect it.

6

Features that know the future

This one is not about splitting at all, and no splitting strategy protects against it. A feature contains information that would not have existed at the moment of prediction.

Predicting whether a customer will churn, using a column called account_closure_reason. Predicting equipment failure, using a maintenance ticket that gets raised after the failure. Predicting loan default, using a field the collections team fills in.

These produce spectacular scores. An AUC of 0.99 on a genuinely hard problem is not a triumph, it is a symptom.

Two checks catch most of them. First, for every feature, ask when its value is recorded and whether that is before or after the thing you are predicting. If it is after, or if you are unsure, drop it. Second, if a single feature carries an implausible share of the model's importance, go and read its definition before you celebrate.

7

Using the test set more than once

The last one is a discipline problem rather than a code problem, and it is the hardest to avoid.

You evaluate on the test set. The score is disappointing. You adjust something and evaluate again. That second evaluation is no longer an unbiased estimate, because you used the first one to make a decision. The test set has become a validation set, and you no longer have a test set.

Keep it in a separate file. Write the evaluation as a script you run once, at the end, when the model is chosen. If you genuinely have to go back and change something, that is fine, but be honest that the number is now optimistic and say so when you report it.

Sanity check before you ship

Take your best model and train it on shuffled labels. The score should collapse to the baseline. If it does not, something in your setup is leaking, and you have just found it before production did.

The short version

  • Split first, then fit anything that learns from data
  • Put preprocessing in a pipeline so folds cannot leak
  • Time-ordered data gets a time-ordered split, never a shuffle
  • Repeated entities get GroupKFold on their identifier
  • Audit every feature for whether it existed at prediction time
  • Touch the test set once
  • A suspiciously good score is a bug report, not a result

Build models that survive contact with production

Our programmes work with messy, grouped, time-ordered data rather than clean textbook datasets, because that is what you will be handed in a real job. Validation discipline is taught alongside the modelling, not after it.

View Our Data Analytics Program
Back to All Blogs Next: Why 97% Accuracy Can Mislead