Ensemble Methods: Random Forests, Bagging, and Gradient Boosting

the idea: many weak-ish models beat one strong one

Decision Trees flagged a specific weakness: a small change in the CI training data — a handful of extra runs, a slightly different random split — can flip an early split and cascade into a noticeably different tree. That's high variance in exactly the sense covered in Overfitting vs Underfitting. An ensemble doesn't try to build one perfectly stable tree; it builds many imperfect, differently-wrong trees and combines their predictions, on the premise that their individual mistakes are at least partly uncorrelated and will average out. The two dominant strategies for building that combination — bagging and boosting — differ in one fundamental choice: whether the trees are built independently or in sequence, each one aware of the last.

bagging: bootstrap aggregating

Bagging (bootstrap aggregating) builds many trees independently and averages them:

Step	What happens
1	draw a bootstrap sample: randomly sample `m` rows from the `m`-row training set, with replacement, so some rows appear multiple times and others not at all
2	train one full decision tree on that bootstrap sample
3	repeat steps 1-2 independently, B times, building B different trees on B different bootstrap samples
4	to predict a new CI run: average the B trees' predictions (regression), or take a majority vote (classification)

The statistical reason this works: each individual tree is still a high-variance, somewhat-overfit estimator, but its errors are largely independent of the errors made by trees trained on different bootstrap samples. Averaging B roughly-independent, unbiased estimators shrinks the variance of the average by a factor of roughly 1/B — the individual mistakes cancel rather than compound. Because every tree trains on its own bootstrap sample with no dependency on any other tree, all B trees can be trained in parallel — a practical advantage bagging has over the sequential approach below.

random forests: bagging plus one more source of randomness

Plain bagging has a subtle limitation: if one feature — say touches_slow_suite from Decision Trees' entropy example — is a genuinely dominant predictor, nearly every bootstrap sample will still pick it as the best root split, because it's the best split for almost any subset of the data. The trees end up correlated with each other despite training on different data, which weakens the variance-cancellation argument above — correlated errors don't average away as well as independent ones do.

A random forest adds a second layer of randomness to force more diversity: at every split in every tree, instead of considering all available features, it restricts the choice to a random subset of them (a common default is roughly the square root of the total feature count). A tree that can't consider touches_slow_suite at a given split has to find the next-best available feature instead, producing meaningfully different trees even when they're trained on similar bootstrap samples. This one change — feature subsampling at each split, on top of bagging's row subsampling — is the entire difference between "bagged trees" and a "random forest."

boosting: fixing what the ensemble gets wrong

Boosting takes the opposite approach: trees are built one at a time, each new tree deliberately targeting the mistakes of the ensemble built so far, rather than a fresh independent random sample. Gradient boosting is the dominant version of this idea:

Step	What happens
1	start with a naive first model — even something as crude as always predicting the average CI duration
2	use the current ensemble to predict every training example, and compute the loss (e.g. MSE from Linear Regression)
3	fit a new tree whose job is specifically to reduce that loss — in practice, to predict the current ensemble's errors (residuals)
4	add the new tree to the ensemble, scaled down by a small learning rate so no single tree dominates
5	repeat from step 2, with the ensemble now including the new tree

The "gradient" in gradient boosting is literal: fitting each new tree to the current residuals is mathematically equivalent to taking a gradient descent step on the loss function, in the space of possible trees rather than the space of numeric weights from Gradient Descent. Because each tree explicitly depends on the output of every tree before it, boosting is inherently sequential — it can't be parallelized across trees the way bagging can. XGBoost ("extreme gradient boosting") is the implementation most commonly reached for in practice: the same core algorithm as above, with substantial engineering for speed and built-in regularization on top.

bagging vs. boosting

	Bagging / Random Forest	Boosting (Gradient Boosting / XGBoost)
Trees built	independently, in parallel	sequentially, each depending on the last
Primarily fixes	variance (instability) — individual deep trees can stay complex	bias — individual trees are often deliberately shallow ("weak learners")
Overfitting behavior	adding more trees rarely hurts; variance keeps shrinking	adding too many rounds, or trees too deep, can genuinely overfit — needs the regularization and early-stopping ideas from earlier in this track
Training cost	parallelizable across trees	inherently sequential, though each tree is often shallower and cheaper
Typical accuracy ceiling	strong, reliable default	often the higher ceiling on tabular data, at the cost of more careful tuning

practical notes

from sklearn.ensemble import RandomForestClassifier| https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html | n_estimators is B (number of trees); max_features controls the per-split feature subsampling that defines a random forest |'ens_sk1'

from sklearn.ensemble import GradientBoostingClassifier| https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html | learning_rate scales each new tree's contribution; smaller values need more n_estimators but generalize better |'ens_sk2'

import xgboost; xgboost.XGBClassifier()| https://xgboost.readthedocs.io/en/stable/python/python_api.html | the scikit-learn-compatible API for XGBoost -- drop-in for GradientBoostingClassifier in most pipelines, generally faster and more tunable |'ens_sk3'

model.feature_importances_| | available on both random forests and gradient-boosted trees -- a quick first pass at which features (touches_slow_suite, lines_changed, ...) the ensemble actually leaned on |'ens_sk4'

where to go from here

Decision Trees — the single-tree instability problem this whole page exists to fix.
Next in this track: Evaluation Metrics — precision, recall, and F1, rounding out how to honestly score any model built so far.
Overfitting vs Underfitting and Regularization — the bias/variance vocabulary this page relies on throughout.

reference

scikit-learn — Ensemble Methods
XGBoost — Introduction to Boosted Trees