supervised learning only knows what it's seen

The logistic regression and decision tree classifiers built earlier in this track were trained on a few dozen historical failures, mostly explained by recognizable patterns like touches_slow_suite. Now imagine a genuinely new failure mode shows up — a poisoned dependency that silently corrupts build artifacts, something that has never occurred in the training data and doesn't resemble any failure the model has seen. A supervised classifier has no mechanism to flag it; it was only ever taught to recognize the specific patterns present in its training examples, and this one isn't among them. Anomaly detection reframes the question entirely: instead of "does this match a known failure pattern," it asks "does this run look statistically normal at all" — a question that doesn't require ever having seen the specific way it's about to go wrong.

density estimation: modeling what "normal" looks like

The core idea is density estimation: build a probability model from the vast majority of runs that behave normally, then score every new run by how probable it is under that model. A run that lands in a high-probability region looks like the runs the model has already seen; a run whose features are far out in the tails — a duration ten times the typical spread, an unusual combination of hour and diff size — gets a low probability and is flagged as anomalous, whether or not anything like it ever appeared during training.

the Gaussian model

The standard starting point models each feature as an independent Gaussian (normal, bell curve) distribution, fit directly from the normal (non-anomalous) training examples: 

mean = (1/m) * sum(x_i for i in 1..m)
variance = (1/m) * sum((x_i - mean)^2 for i in 1..m)

# for a new value x, its probability density under this Gaussian:
p(x) = (1 / sqrt(2*pi*variance)) * exp(-(x - mean)^2 / (2*variance))
A Gaussian is a convenient, cheap-to-fit default — just two numbers per feature — and a reasonable first assumption when there's no specific reason to expect a different shape. It's not the only option, and the practical-notes section below covers modern alternatives that drop this assumption entirely.

combining features: the independence assumption

A CI run has many features, not one. The simplest way to get one overall probability for the whole run is to multiply the individual per-feature probabilities together: 

p(run) = p(duration) * p(lines_changed) * p(files_changed) * ...
Multiplying probabilities like this is only mathematically correct if the features are statistically independent — and in practice, they rarely are (a large diff usually also touches more files). The approximation is used anyway because it's cheap and tends to work well enough, but it's worth naming honestly rather than treating as exact. When two features are strongly correlated, one practical fix is exactly the tool from the previous page: PCA can decorrelate the feature set first, so the independence assumption is closer to true by the time the Gaussian model is fit.

picking the threshold, epsilon

A run is flagged as anomalous when p(run) < epsilon, for some threshold epsilon chosen the same disciplined way every other hyperparameter in this track has been: never guessed, always tuned against held-out data. Say the CI history has 10,000 normal runs and only 20 confirmed anomalies — split similarly to Train/Test Splits & Cross-Validation, fit the Gaussian parameters (mean, variance) on the training split of normal runs only, then use the validation split — normal runs plus most of the known anomalies — to sweep candidate values of epsilon and pick whichever one gives the best F1 score from Evaluation Metrics. With only 20 positive examples total, this is about as extreme a class-imbalance problem as this track has covered — precision and recall matter far more here than accuracy ever could.

anomaly detection vs. supervised learning: when to use which

Use anomaly detection whenUse supervised learning when
very few labeled positive examples existplenty of labeled positive examples exist
those few examples don't cover every way things can go wrongthe labeled examples reasonably cover the full range of failure types expected in production
future failures may look nothing like any failure seen so farfuture failures are expected to resemble past ones
The pass/fail classifier from Logistic Regression and Decision Trees is the right tool when failures cluster into a handful of recognizable, recurring types — exactly the "flaky test / timeout / compile error" categories that showed up throughout this track. Anomaly detection is the right tool for the tail case those classifiers structurally can't cover: the failure mode nobody has catalogued yet.

feature choice matters more here

In supervised learning, a weak or irrelevant feature mostly just gets a small weight and fades into the background — regularization actively encourages this. In anomaly detection, every included feature directly shapes the density estimate; there's no training signal quietly down-weighting the unhelpful ones. Feature selection deserves real care here, and features that are heavily skewed rather than roughly bell-shaped (a heavy-tailed lines_changed, for instance) usually benefit from a transform — a log transform is a common first choice — to make them look closer to Gaussian before fitting, since the whole model is built on that assumption.

practical notes

from scipy.stats import norm; norm.pdf(x, mean, std)| https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html | the direct implementation of the per-feature Gaussian density formula above |'ad_sk1'
from sklearn.covariance import EllipticEnvelope| https://scikit-learn.org/stable/modules/generated/sklearn.covariance.EllipticEnvelope.html | fits a single multivariate Gaussian (not per-feature) accounting for correlations directly, instead of relying on the independence assumption above |'ad_sk2'
from sklearn.ensemble import IsolationForest| https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html | a tree-based alternative that drops the Gaussian assumption entirely -- isolates anomalies by how few random splits it takes to separate a point from the rest, generally a stronger default than the hand-fit Gaussian model above |'ad_sk3'

where to go from here

Next in this track: Data Leakage — the silent killer of model performance.
Unsupervised Learning — the PCA decorrelation technique referenced above.
Evaluation Metrics — precision, recall, and F1, the tools this page's threshold-tuning step depends on.

reference

scikit-learn — Novelty and Outlier Detection
Google — Clustering: Anomaly Detection