Train/Test Splits and Cross-Validation: Why Random Splits Can Lie to You

why held-out data at all

Decision Trees closed on a warning: a model can reach perfect training accuracy by memorizing the training examples instead of learning a pattern that generalizes. The only way to catch that is to measure performance on data the model never got to train on. This whole approach rests on one assumption worth naming explicitly: that training data, held-out data, and the real-world data the model will eventually see are all drawn from the same underlying distribution — often called the i.i.d. assumption (independent and identically distributed). Every technique on this page is really just different ways of trying to honor that assumption, and every failure mode below is a way it quietly gets violated.

the three-way split: train, validation, test

Split	Used for	Touched how often
Training set	fitting the model's parameters (the `w`'s and `b` from earlier pages)	every training run
Validation set	comparing models and hyperparameters — decision tree depth, polynomial degree, which algorithm to use at all	repeatedly, once per candidate you try
Test set	one final, honest read of how the chosen model generalizes	once, at the very end

The validation set is the one people skip without meaning to. If you try five different max_depth values on the CI failure-prediction tree and pick whichever one scores best on your "test" set, that set has stopped being a test set — you've just used it to make a modeling decision, so its score is now optimistic, for the same reason training accuracy was optimistic. That's precisely why a separate validation set exists: to absorb all that repeated poking, leaving the test set clean for one final, honest check. Typical split sizes scale with how much data you have — 60/20/20 is reasonable under a few tens of thousands of examples; with millions of examples, something like 98/1/1 already gives a validation and test set large enough to be statistically meaningful.

why a random split can quietly leak the future

shuffle, then split is the default move, and it's the right one when rows are genuinely independent. CI run data usually isn't, in two specific ways worth naming:

Near-duplicate rows from the same source. A single pull request is often pushed several times — a flaky test triggers a retry, a reviewer asks for one more commit. Each retry logs as its own row, but they share the same author, nearly the same diff, and often the same outcome. A random shuffle can easily put three retries of the same PR into training and a fourth into the test set. The model doesn't need to have learned anything general about CI failures to get that fourth row right — it's effectively seen the answer already. The fix is to split by group (all rows from one PR go entirely into training or entirely into test, never split across both), not by individual row.

Order matters when the world changes over time. If a new, faster CI runner gets rolled out in month eight of a year of data, run durations before and after that change follow different patterns. A plain random split scatters "before" and "after" rows into both training and test, so the model gets to train on some post-upgrade examples and is tested on others — flattering a model that would actually perform worse in production, where it only ever sees the future, never a mix of past and future. When order genuinely matters, splitting chronologically (train on earlier runs, test on later ones) gives an honest read. This is the default assumption, not the exception, for genuinely time-ordered data — covered in full in the Time Series & Quant Trading track.

k-fold cross-validation

A single validation split has a downside: which rows happened to land in it is partly luck, so the score it produces has some noise — a model might look good (or bad) on this particular slice of data by chance. k-fold cross-validation averages that luck away:

Step	What happens
1	split the training data into k equal-sized folds (5 and 10 are common choices)
2	train on k−1 of the folds, validate on the one left out
3	repeat k times, holding out a different fold each time
4	average the k validation scores into one number

Every row gets used for validation exactly once and for training k−1 times, so the final averaged score is far less sensitive to which specific rows happened to land where. The cost is k separate training runs instead of one — worth it for a decision tree that trains in milliseconds, potentially not worth it for a model that takes hours. As a concrete use, this is exactly how you'd honestly compare polynomial degrees for the CI duration regressor from Linear Regression: cross-validation error drops sharply as degree increases from 1, flattens out, then climbs again past some point as the higher-degree fits start overfitting each fold's training portion — that turn from falling to rising is the signal for which degree actually generalizes best.

matching your split to the real distribution

One more way the i.i.d. assumption breaks: training data and real usage don't always come from the same place. Say a year of CI history is almost entirely from one monorepo, but the model is meant to also serve two newer microservice repos with a different testing setup. A validation set drawn purely from the monorepo will never surface how the model performs on the microservice repos — because it never sees any. The fix is to make sure validation and test data actually reflect the mix the model will face in production, even if that means deliberately setting aside examples from the newer repos for validation rather than letting a random split (correctly) mostly ignore them for being a small minority of the data.

practical notes

from sklearn.model_selection import train_test_split| https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html | the basic random split -- stratify=y keeps the same pass/fail ratio in both splits, important given the class imbalance from the ML pipeline page |'tts_sk1'

train_test_split(X, y, shuffle=False)| | turns off the random shuffle -- use this whenever row order encodes real information (time-ordered CI runs, any sequential data) |'tts_sk2'

from sklearn.model_selection import GroupKFold| https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html | folds respect a group id (e.g. pull_request_id) -- no group's rows are ever split across train and validation |'tts_sk3'

from sklearn.model_selection import cross_val_score| https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html | runs the full k-fold loop above in one call, returning all k scores |'tts_sk4'

where to go from here

This closes out the beginner section of ML Foundations. Next: Gradient Descent — how a model's parameters actually get adjusted during that "training" step every page so far has assumed.
The ML Pipeline — where splitting and cross-validation fit into the bigger seven-stage picture.
Data Leakage (advanced, later in this track) — a deeper dive into subtler leakage patterns than the ones covered here.

reference

scikit-learn — Cross-Validation
Google ML Crash Course — Dividing the Dataset