the idea: penalize complexity, not just error

Overfitting vs Underfitting named the failure mode: a model with enough capacity will fit training noise if nothing stops it. Regularization stops it by changing what "minimize the cost" even means — instead of only rewarding a low error on the training data, the cost function also penalizes the model for having large parameter weights. A model that leans hard on one feature's weight to explain a few noisy training points now pays a price for that weight being large, not just for being wrong. The model is nudged toward simpler explanations, even at the cost of a slightly higher training error, in exchange for a lower validation error.

L2 regularization (ridge): shrinking weights smoothly

The most common approach adds the sum of squared weights to the cost function, scaled by a hyperparameter lambda: 

J(w, b) = MSE(w, b) + (lambda / 2m) * sum(w_j^2 for each feature j)

# the bias term b is conventionally left out of the penalty --
# only the feature weights are pushed toward zero
This is L2 regularization (also called ridge). Because every weight contributes to the penalty in proportion to its square, a weight of 10 costs 100 times more than a weight of 1 — the penalty falls disproportionately on the largest weights, which is exactly where overfitting tends to concentrate: a model chasing one noisy training point often does it by inflating a single weight to an extreme value. The same idea applies unchanged to logistic regression's log loss, and to every weight in a neural network — only the cost function being penalized changes.

L1 regularization (lasso): shrinking weights to zero

L1 regularization penalizes the sum of absolute weight values instead of squared ones: 

J(w, b) = MSE(w, b) + (lambda / m) * sum(abs(w_j) for each feature j)
The practical difference matters: L2's penalty gets gentler as a weight approaches zero (the derivative of w² shrinks with w), so it shrinks weights toward zero without much pressure to reach it exactly — most features keep a small, nonzero weight. L1's penalty stays constant regardless of how small the weight already is, so it keeps pushing until some weights land on exactly zero. Concretely: if day_of_week turns out to carry no real signal about CI failures, L1 regularization can drive its weight to precisely 0 — the feature is effectively removed from the model — while L2 would leave it at some small nonzero value that still technically contributes to every prediction. That makes L1 a built-in feature-selection mechanism, useful when you suspect many features are irrelevant and want a sparser, more interpretable model; L2 is the more common default otherwise.

picking lambda

lambda directly controls the bias/variance tradeoff from the previous page. lambda = 0 is no regularization at all — back to the original overfitting risk. A very large lambda crushes every weight toward zero so hard that the model can no longer fit even the training data — a regularization-induced underfitting. The right value sits between the two, and the honest way to find it is the same cross-validation sweep from Train/Test Splits & Cross-Validation: train the model once per candidate lambda, score each on the validation set, and pick whichever minimizes validation error — not training error, which decreasing lambda will always improve on its own.

regularized gradient descent, and why L2 is called "weight decay"

Adding the L2 penalty to the cost function changes its gradient by one extra term, and that term has a clean interpretation once it's plugged into the update rule from Gradient Descent: 

w_j = w_j - alpha * (dJ/dw_j + (lambda/m) * w_j)
    = w_j * (1 - alpha*lambda/m) - alpha * (dJ/dw_j)
Rearranged that way, every single gradient descent step first shrinks w_j by a small multiplicative factor (1 - alpha*lambda/m, slightly less than 1) before applying the usual gradient-based update — the parameter literally decays toward zero a little every step, independent of whatever the data says. That's the origin of the term weight decay, used interchangeably with L2 regularization in most of the literature. One nuance worth flagging for later: this clean equivalence between L2 and weight decay holds for plain gradient descent, but breaks down for adaptive optimizers like Adam — exactly the gap AdamW was built to close, covered in the Deep Learning & PyTorch Engineering track.

early stopping: a regularizer in disguise

The previous page introduced early stopping as a fix for overfitting — halt training once validation error starts climbing while training error keeps falling. It's worth naming explicitly as a form of regularization: weights generally grow larger the longer training continues, as the model works harder to shrink an already-small training error further. Stopping early caps how large the weights are ever allowed to get, without touching the cost function at all — a regularization effect achieved through the training schedule instead of an explicit penalty term.

dropout: regularization for neural networks

None of the models covered so far in this track (linear/logistic regression, decision trees) have a natural notion of "dropout" — it's specific to neural networks, briefly worth previewing here since the title of this page promises it. During training, each forward pass randomly zeroes out a fraction of a layer's neurons (a common default is 20-50%), forcing the rest of the network to produce useful output without relying on any single neuron being present. This discourages neurons from co-adapting into fragile, overly specific combinations, pushing the network toward more redundant, generalizable features — conceptually similar to what happens when weight decay discourages any one weight from growing too large. No neurons are dropped at inference time; the full network is used, which is why architectural consistency between training and evaluation modes matters. The runnable PyTorch implementation belongs to the Deep Learning & PyTorch Engineering track.

practical notes

from sklearn.linear_model import Ridge| https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html | linear regression with L2 regularization built in -- the alpha parameter here is this page's lambda |'reg_sk1'
from sklearn.linear_model import Lasso| https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html | linear regression with L1 -- inspect coef_ after fitting to see which weights actually landed on exactly zero |'reg_sk2'
LogisticRegression(C=1.0)| https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html | sharp edge: scikit-learn's C is the INVERSE of lambda (C = 1/lambda) -- a SMALLER C means STRONGER regularization, the opposite of what the name suggests at a glance |'reg_sk3'

where to go from here

Overfitting vs Underfitting — the diagnosis this page's toolbox is a treatment for.
Next in this track: Feature Engineering — scaling, normalization, and encoding, the data-side counterpart to this page's model-side fixes.
Deep Learning & PyTorch Engineering — dropout, batch normalization, and AdamW's decoupled weight decay in code.

reference

scikit-learn — Ridge Regression
Google ML Crash Course — Regularization