Evaluation Metrics: Precision, Recall, F1, and Why Accuracy Lies
The last piece of the intermediate toolkit — because a model is only as good as the number you chose to judge it by.
Intermediate
TopNotchNote
Notes that matter
why accuracy lies
Back in The ML Pipeline, the CI dataset was 95% "pass," 5%
"fail." A model that predicts "pass" for every single run — one that has learned nothing
about failures at all — scores 95% accuracy. It's a completely useless model wearing an
impressive-looking number. Accuracy — correct predictions divided by total
predictions — treats every mistake as equally uninformative, but on imbalanced data like this,
accuracy is dominated by how well a model handles the majority class, which is usually the
class nobody cares about predicting. Whenever one class outnumbers another by a wide margin,
accuracy stops being a meaningful number on its own.
the confusion matrix: four outcomes, not one
Treat "fail" as the class actually worth detecting (the positive class).
Every prediction the model makes falls into exactly one of four buckets:
| Predicted fail | Predicted pass | |
|---|---|---|
| Actually failed | True Positive (TP) — caught it | False Negative (FN) — missed it |
| Actually passed | False Positive (FP) — false alarm | True Negative (TN) — correctly ignored |
Accuracy collapses all four numbers into one ratio:
(TP + TN) / everything.
With 950 true negatives and only 50 real failures, a model can get every single failure wrong
(FN = 50, TP = 0) and still post 95% accuracy, because TN alone carries the score. Precision
and recall exist specifically to look at TP, FP, and FN individually instead of blending them
all together.
precision: of the failures I flagged, how many were real?
precision = TP / (TP + FP)
Precision answers: when the model says "this run will fail," how often is it actually
right? Low precision means a lot of false alarms — engineers get paged, extra checks run, and
the model's warnings start getting ignored because they're too often wrong. Precision is the
metric that matters most when a false positive is expensive: flagging a perfectly fine run as
risky wastes real engineering time and erodes trust in the system.
recall: of the real failures, how many did I catch?
recall = TP / (TP + FN)
Recall answers the opposite question: of every run that actually failed, what fraction
did the model catch? Low recall means real failures slip through undetected — exactly the
outcome the model was built to prevent. Recall matters most when a false negative is the
expensive mistake: missing an actual failure means the full slow test suite runs anyway,
achieving nothing the model was supposed to save.
the precision/recall tradeoff and the threshold
These two numbers pull in opposite directions, and Logistic
Regression already introduced the knob that controls the trade: the classification
threshold. Lowering the threshold from 0.5 to, say, 0.3 makes the model flag "fail" more
readily — it catches more of the real failures (recall goes up), but also flags more runs
that were actually fine (precision goes down). Raising the threshold does the reverse: fewer
false alarms, but more real failures slip through uncaught. Neither direction is
free — there is no threshold that improves both at once for a fixed model. Which direction to
push depends entirely on which mistake costs more in the actual system, not on a rule of
thumb.
F1: one number balancing both
When neither precision nor recall alone tells the full story, and no clear business reason
favors one over the other, F1 combines them into a single score — the
harmonic mean, not the plain average:
F1 = 2 * (precision * recall) / (precision + recall)
The harmonic mean matters here, not just the arithmetic one. Take a model with
precision = 1.0 but recall = 0.01 — it's right every time it dares to flag a failure, but it
almost never dares to. The plain average of those two numbers is a deceptively respectable
0.505. F1 gives
2 * (1.0 * 0.01) / (1.0 + 0.01) ≈ 0.02 — correctly reflecting
that a model catching essentially none of the real failures is nearly useless, no matter how
clean its rare guesses are. The harmonic mean is dragged down hard by whichever of the two
numbers is smallest, which is exactly the property that makes it a fair single-number summary
of a genuine trade-off.
multiclass metrics: macro, micro, and weighted
If the problem grows from binary pass/fail into classifying which kind of
failure occurred (flaky test, compile error, timeout, infra issue), precision and recall have
to be computed per class, then combined into one number three different ways:
| Averaging | How it combines per-class scores | Use when |
|---|---|---|
| Macro | plain average across classes, every class weighted equally | the rare failure category matters just as much as the common one, regardless of how often it occurs |
| Micro | pool every class's TP/FP/FN together first, then compute one precision/recall — mathematically equivalent to overall accuracy | class sizes vary a lot and the overall, volume-weighted rate is what matters |
| Weighted | average across classes, weighted by how many true examples each class has | a middle ground: reflects the real class distribution without letting one dominant class fully define the score, the way micro-averaging does |
practical notes
from sklearn.metrics import classification_report| https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html | prints precision/recall/F1 per class plus macro and weighted averages in one call -- the standard first check after fitting any classifier |'em_sk1'
from sklearn.metrics import confusion_matrix| https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html | the raw TP/FP/TN/FN counts behind every metric on this page |'em_sk2'
from sklearn.metrics import precision_recall_curve| https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html | traces precision and recall across every possible threshold -- the concrete tool for picking a threshold deliberately instead of leaving it at the 0.5 default |'em_sk3'
Decide which metric matters most before looking at results, not after. Picking
whichever metric happens to make a given model look best, after the fact, is a subtle form of
fooling yourself — the choice should follow from the real cost of a false positive vs. a false
negative in the system, settled in advance, the same discipline the ML Pipeline page raised
when introducing the majority-class baseline.
where to go from here
This closes the intermediate section of ML Foundations.
Next: Unsupervised Learning — K-means clustering
and PCA, the first advanced-tier page.
Logistic Regression — the threshold mechanic this page's precision/recall trade-off depends on.
The ML Pipeline — where the majority-class baseline this page keeps referring back to was first introduced.
Logistic Regression — the threshold mechanic this page's precision/recall trade-off depends on.
The ML Pipeline — where the majority-class baseline this page keeps referring back to was first introduced.