Overfitting vs Underfitting: Diagnosing Bias and Variance
Low training error was never the goal. It was always a proxy for something you can't measure directly — and proxies can be gamed.
Intermediate
TopNotchNote
Notes that matter
signal vs. noise: what training loss doesn't tell you
Every training example carries two kinds of information. Signal is the
part that generalizes — the genuine relationship between diff size and CI failure risk.
Noise is everything specific to that particular batch of historical runs and
nothing more — a flaky test that happened to fail three times in a row for unrelated reasons,
a runner having a bad day. Gradient descent, as covered in the previous page, has no way to
tell these apart; it just minimizes training loss, and training loss goes down whether the
model is learning signal or memorizing noise. Validation loss is what tells them apart,
because noise learned from the training set, by definition, doesn't show up again in
validation data.
That gives a precise vocabulary for the two ways a model can fail to generalize:
underfitting (formally, high bias) is a model that hasn't
learned enough signal — both training and validation error are high. Overfitting
(high variance) is a model that learned the training set's noise on top of
its signal — training error is low, but validation error stays high, and the gap between
them is a direct readout of how much noise got memorized.
underfitting: too simple to see the pattern
Fit a plain straight line to the CI-duration data from Linear
Regression when the true relationship curves upward for very large diffs, and the line
can't represent that curve no matter how well it's trained — it's the wrong shape of function
for the job. Both training and validation error stay stubbornly high, because the model isn't
complex enough to capture even the pattern that's sitting right there in the training data.
More training, more iterations of gradient descent, more data — none of it helps, because the
bottleneck isn't optimization or data volume, it's the model's own capacity.
overfitting: too much freedom to memorize
The opposite failure: a decision tree from Decision Trees
grown without any depth limit, or a degree-15 polynomial fit to the duration data. Given
enough flexibility, either can thread a curve through every single training point exactly —
training error near zero. But some of what it threaded through was noise: a run that happened
to be unusually slow that day for reasons nothing in the features captures. The fitted curve
now wiggles to accommodate that one point, and those wiggles actively hurt predictions on new
data that doesn't share the same noise. Training error near zero with validation error still
high is the unambiguous signature.
diagnosing with a baseline
"High" training error is meaningless without something to compare it to — some problems
are just hard, and even a well-fit model won't reach zero error. A practical
baseline gives that comparison a number: the majority-class baseline from
The ML Pipeline (accuracy from always predicting "pass") is one
concrete floor; a human engineer's own accuracy at eyeballing a diff and guessing pass/fail is
another. Compare all three numbers together:
| Pattern | Diagnosis |
|---|---|
| baseline and training error are both high, close together | underfitting (high bias) — the model isn't even matching a simple baseline |
| training error is close to baseline, but validation error is much higher | overfitting (high variance) — the model matches the data it trained on but not new data |
| training and validation error are both close to the baseline | a well-fit model — the "just right" middle ground both failure modes are defined against |
learning curves: does more data even help?
A different, very practical diagnostic: retrain the same model on progressively larger
slices of the training set (10%, 25%, 50%, 100%) and plot both training and validation error
against training-set size. The shape of that curve answers a question that matters directly
for deciding what to do next — is it worth the effort of collecting more CI run history?
| Curve shape | Meaning |
|---|---|
| Both curves plateau early, at a high error, close to each other | high bias — more data won't help, the model itself needs to change |
| Training error stays low; validation error slowly converges toward it as data grows | high variance — more data is directly useful and will keep closing the gap |
This is often the single most useful chart to check before spending real effort on data
collection: a high-bias learning curve tells you upfront that gathering another year of CI
history won't move the needle, saving that effort for a model or feature change instead.
fixing each one
| Underfitting (high bias) | Overfitting (high variance) |
|---|---|
| add more/better features | collect more training data (often the single most effective fix, when available) |
| add polynomial features, or use a more expressive model | use fewer features, or a simpler model |
| use a bigger neural network (covered in the Deep Learning track) | increase regularization — a dedicated page, next in this track |
| decrease regularization strength | early stopping — stop training once validation error starts climbing while training error keeps falling |
Notice the two columns pull in opposite directions on shared knobs — more model capacity
fixes bias but risks variance, more regularization fixes variance but risks bias. There's no
universal setting; the diagnostics above exist precisely to tell you which direction to push
for the model and data in front of you, rather than guessing.
practical notes
from sklearn.model_selection import learning_curve| https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.learning_curve.html | generates the train-size-vs-error data needed to plot a learning curve directly, using cross-validation under the hood |'of_sk1'
One caveat worth flagging early: everything above assumes training and validation data
come from the same distribution. If they don't — the mismatched-distribution
scenario from Train/Test Splits & Cross-Validation — a
large train/validation gap can look exactly like high variance while actually being a data
mismatch problem. Fix the split before trusting the diagnosis.
where to go from here
Next in this track: Regularization — the mechanics of
L1/L2 penalties, early stopping, and dropout, the main tool for fixing overfitting.
Decision Trees and Linear Regression — the two models used as running examples on this page.
Train/Test Splits & Cross-Validation — how the validation data this whole page depends on is produced in the first place.
Decision Trees and Linear Regression — the two models used as running examples on this page.
Train/Test Splits & Cross-Validation — how the validation data this whole page depends on is produced in the first place.