Data Leakage: The Silent Killer of Model Performance

the sneakiest bug in ML: too good to be true

Data leakage happens when information that wouldn't actually be available at prediction time sneaks into training — through a feature that's determined by the outcome, or through evaluation data influencing the training process. The symptom is always the same shape: training and validation scores look excellent, sometimes suspiciously so, and then the model performs noticeably worse once it's making real predictions in production, on data that never had the leaked information to lean on in the first place. Leakage doesn't throw an error — it makes a bad model look like a good one, right up until it's making real decisions.

two flavors of leakage

Type	What happens
Target leakage	a feature itself encodes information that wouldn't exist yet at the moment a prediction actually needs to be made
Train-test contamination	information from validation/test data leaks into how the training data — or the preprocessing applied to it — gets prepared

target leakage: features that don't exist yet

Say someone adds a new feature to the CI failure-prediction model: manual_intervention_flag — whether an engineer stepped in to investigate the run. It would almost certainly turn out to be a spectacular predictor of "fail." It's also completely useless, because it's only ever set after a human has already noticed the run failing — by definition, the model needs to predict failure before that happens, and at that moment, the feature hasn't been set for anything yet. Training on it produces a model that looks nearly perfect on historical data (where the flag was always set correctly, in hindsight) and is worthless the moment it has to predict a brand-new run in real time.

The test that catches this isn't "does this feature correlate with the target" — it will, often strongly, which is exactly what makes target leakage tempting. The test is about timing: would this exact piece of information genuinely be available, in this exact form, at the moment the prediction actually has to be made? Any feature that's created or updated as a consequence of the outcome — rather than as a cause or a concurrent fact — is target leakage, no matter how useful it looks in a training run.

train-test contamination: what this track already flagged

This half of leakage has actually come up four separate times already in this track, each time as a specific, concrete warning rather than under this name. Naming the pattern now makes the common thread explicit:

Where this track already covered it	The contamination
Feature Engineering — scaling	fitting a scaler's mean/std on the full dataset before splitting, instead of on the training split alone
Feature Engineering — target encoding	computing a category's average target value from rows that include the very examples being predicted
Train/Test Splits & Cross-Validation — near-duplicates	rows from the same PR (near-identical, same outcome) split across both training and test
Train/Test Splits & Cross-Validation — time order	a random shuffle mixing "future" and "past" rows across a genuinely time-ordered split

Every one of these is the same underlying mistake wearing a different disguise: information that should have been strictly off-limits to the training process (because it either comes from validation/test rows, or reflects the future) leaks in anyway.

the giveaway: suspiciously good validation scores

Two practical red flags are worth actively watching for, not just reacting to after the fact. First, a validation score that's dramatically better than what's reasonable for the problem — near-perfect precision and recall on a genuinely hard, noisy prediction task should prompt suspicion before celebration. Second, checking model.feature_importances_ from Ensemble Methods and finding one feature towering over every other by a wide margin is a strong signal to go investigate that specific feature's timing before trusting the result.

the general test: could this feature exist at prediction time?

Every specific case on this page collapses into one question, worth asking of every single feature before it goes into a model: if I had to make this exact prediction right now, for a brand-new run that just started, would I actually have this piece of information, in this exact form, available to me? lines_changed and touches_slow_suite pass — they're known the instant the run is triggered. manual_intervention_flag fails — it doesn't exist until well after the outcome is already known. Running every feature through this question before training is cheap insurance against the entire category of bug this page covers.

practical notes

from sklearn.pipeline import Pipeline| https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html | bundles preprocessing (scaling, encoding) and the model into one object -- cross_val_score run on a Pipeline correctly re-fits preprocessing on only each fold's training portion, closing the contamination gap structurally instead of by discipline alone |'dl_sk1'

This is the structural fix for train-test contamination specifically: wrap the entire preprocessing chain — scalers, encoders, everything from Feature Engineering — inside a Pipeline before handing it to cross_val_score or train_test_split-based validation, rather than manually calling .fit_transform() on the whole dataset up front and hoping to remember the right order every time.

where to go from here

Next in this track: Model Explainability — feature importance, partial dependence, and SHAP.
Train/Test Splits & Cross-Validation and Feature Engineering — the four specific contamination cases this page ties together.
Ensemble Methods — feature_importances_, the diagnostic tool this page's red-flag check depends on.

reference

scikit-learn — Common Pitfalls: Data Leakage
Kaggle — Data Leakage