Bayesian Learning and Naive Bayes
No gradient descent, no margin, no splits — just Bayes' theorem, a base rate, and a deliberately naive assumption.
Advanced
TopNotchNote
Notes that matter
a different way to think about probability
Every classifier so far in this track answers "what's
P(fail | features)?"
directly — logistic regression fits weights so that a
squashed weighted sum approximates it. Naive Bayes answers the exact same
question a completely different way: instead of fitting that probability directly, it uses
Bayes' theorem to flip the question around, compute the easier, flipped
version from simple counts, and flip it back:
P(fail | features) = P(features | fail) * P(fail) / P(features)
Each term has a name and a plain-language meaning.
P(fail) is the
prior — the base rate of failure before looking at any features at all
(the same 5%-fail majority-class baseline introduced in
The ML Pipeline). P(features | fail) is the
likelihood — given that a run truly did fail, how likely were these
particular feature values? P(features) is the evidence — how
common these feature values are overall, regardless of outcome; since it's the same number
for every class being compared, it can be dropped entirely when the only goal is to rank
classes against each other. What's left, P(fail | features), is the
posterior — the updated belief about failure after actually seeing the
features.
naive bayes: the "naive" independence assumption
P(features | fail) is still hard to estimate directly the moment there's more
than one feature — it would require enough historical data to have seen every specific
combination of touches_slow_suite, lines_changed, and every other
feature, conditioned on failure. Naive Bayes sidesteps this with the same independence
assumption already used once before in this track, in
Anomaly Detection's Gaussian model: treat every feature
as conditionally independent given the class, so the joint likelihood becomes a simple
product of per-feature likelihoods:
P(features | fail) = P(touches_slow_suite | fail) * P(large_diff | fail) * ...
This is the "naive" part, and it's honestly false —
touches_slow_suite and
large_diff are probably correlated in real CI data, not independent. The
assumption is made anyway because it turns an intractable estimation problem into counting,
and in practice the resulting classifier tends to work well even when the independence
assumption clearly doesn't hold, in much the same way the anomaly detection model tolerated
the same approximation.
computing it: a worked example
Say historical data shows 10 failed runs and 90 passing runs out of 100 (prior:
P(fail) = 0.1, P(pass) = 0.9), and two binary features with these
observed rates within each class:
P(touches_slow_suite=yes | fail) = 0.8 P(touches_slow_suite=yes | pass) = 0.2
P(large_diff=yes | fail) = 0.7 P(large_diff=yes | pass) = 0.3
# a new run comes in with touches_slow_suite=yes, large_diff=yes:
score(fail) = P(fail) * P(touches_slow_suite=yes|fail) * P(large_diff=yes|fail)
= 0.1 * 0.8 * 0.7 = 0.056
score(pass) = P(pass) * P(touches_slow_suite=yes|pass) * P(large_diff=yes|pass)
= 0.9 * 0.2 * 0.3 = 0.054
# normalize the two scores so they sum to 1 (this recovers the P(features) denominator):
P(fail | features) = 0.056 / (0.056 + 0.054) = 0.509
P(pass | features) = 0.054 / (0.056 + 0.054) = 0.491
Despite a base rate of only 10% failures, two moderately fail-associated feature values
together were just enough to tip the posterior narrowly past 50% — a concrete illustration of
the prior getting updated by evidence, rather than a rule being triggered.
types of naive bayes for different feature types
| Variant | Assumes each feature is | Typical use |
|---|---|---|
| Gaussian NB | normally distributed within each class | continuous features, e.g. lines_changed or duration — the same per-class Gaussian fitting from Anomaly Detection, applied per-class instead of just to "normal" |
| Multinomial NB | a count | word counts in text classification (spam filtering, topic classification) |
| Bernoulli NB | a binary presence/absence flag | binary features like touches_slow_suite, or word-present/absent text features |
why naive bayes despite the naive assumption
Training a naive Bayes classifier requires no gradient
descent at all — the priors and per-feature likelihoods are just counts and averages read
directly from the training data, computed once in closed form. That makes it extremely fast to
train, even on very high-dimensional data (thousands of vocabulary words in a text
classifier), and a reasonable choice when the training set is small, since it only ever needs
to estimate simple per-feature statistics rather than a model with many interacting
parameters. It's rarely the most accurate option on this track's CI example specifically, but
it remains a fast, well-understood baseline worth trying before reaching for something more
expensive.
practical notes
from sklearn.naive_bayes import GaussianNB| https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html | fits one Gaussian per feature per class from the training data -- no iterative training, fit() just computes means and variances |'nb_sk1'
from sklearn.naive_bayes import BernoulliNB| https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html | the right choice for binary features like touches_slow_suite |'nb_sk2'
model.predict_proba(X_test)| | returns the normalized posterior probabilities directly, the same P(fail|features) computed by hand above |'nb_sk3'
where to go from here
Anomaly Detection — the same Gaussian and
independence assumptions, applied to a different problem.
Next in this track: VC Dimension & Computational Learning Theory.
The ML Pipeline — the majority-class baseline that became this page's prior.
Next in this track: VC Dimension & Computational Learning Theory.
The ML Pipeline — the majority-class baseline that became this page's prior.