Gradient Descent: Batch, Stochastic, and Mini-Batch

the goal: walking downhill on the cost function

Linear Regression defined MSE as a cost function and said "minimize it," without saying how. Picture the cost as a landscape: one axis for each parameter (w, b), height equal to the cost at that combination. Because MSE is convex, that landscape is a single smooth bowl — one lowest point, no false valleys to get trapped in. Gradient descent is the strategy of starting somewhere on that bowl and repeatedly taking a small step in whichever direction goes downhill fastest, until further steps stop changing the height meaningfully.

the update rule

The gradient of the cost function with respect to a parameter is the slope at the current point — how much the cost would change if that parameter nudged up slightly. Gradient descent moves each parameter a small step in the opposite direction of its gradient, since the gradient points toward increasing cost and the goal is to decrease it:


w = w - alpha * (dJ/dw)
b = b - alpha * (dJ/db)

# alpha (the learning rate) controls step size, covered below
# both updates use the OLD w and b -- computing the new b with the
# already-updated w would mix two different points on the cost surface

For the CI-duration linear regression model, the gradient of MSE with respect to w works out to a clean, computable expression — the average, over every training example, of the prediction error times that example's input:


dJ/dw = (1/m) * sum((predicted_i - actual_i) * x_i  for i in 1..m)
dJ/db = (1/m) * sum((predicted_i - actual_i)        for i in 1..m)

Notice the intuition baked into that formula: examples where the model is very wrong contribute a large term and pull the parameters harder; examples it already gets right contribute almost nothing. This exact derivative — figured out by hand here for plain linear regression — is what backpropagation computes automatically and efficiently for far more complex models later in this site's Deep Learning track; the underlying idea (the chain rule, applied layer by layer) is the same.

batch, stochastic, and mini-batch: how much data per step

The formulas above sum over m training examples to take a single step. How many examples that sum actually covers is a choice with real consequences:

Variant	Examples per step	Behavior
Batch gradient descent	all `m`	one very stable, low-noise step per full pass over the data — but each step is expensive, and the whole dataset must fit in memory at once
Stochastic gradient descent (SGD)	1	a step after every single example — fast and cheap per step, but each step is noisy since one example is a poor estimate of the overall trend; the path toward the minimum zig-zags
Mini-batch gradient descent	a small subset (commonly 32-1024)	the practical middle ground — smoother than SGD, far cheaper per step than full batch, and batch sizes as powers of two map efficiently onto GPU memory

An epoch is one full pass through the training data. With batch gradient descent, one epoch means exactly one parameter update; with mini-batches of size 64 on 6,400 training runs, one epoch means 100 separate updates. That's the real reason mini-batches often train faster in wall-clock time even though each individual step is less precise — far more update steps happen per pass over the data. For a training set small enough to fit comfortably in memory (a rough rule of thumb: under a couple thousand examples), plain batch gradient descent is simple and works fine; mini-batches earn their complexity once the dataset — or the model — gets big enough that a full-batch step becomes slow or memory-bound.

picking a learning rate

alpha is a hyperparameter, chosen before training, not learned during it. Too small, and training crawls toward the minimum, technically correct but wasting enormous amounts of compute to get there. Too large, and a single step can overshoot the minimum entirely, landing higher up the opposite wall of the bowl than where it started — repeated often enough, the cost visibly increases instead of decreases, and training diverges.

The standard diagnostic is to plot cost against training step (a learning curve). A healthy run shows cost decreasing every step, flattening out as it approaches the minimum. A curve that oscillates or trends upward means the learning rate is too high; a curve that decreases only very slightly per step, still clearly falling after many iterations, means it's too low. One detail worth knowing: gradient descent can use a single fixed learning rate and still land precisely on the minimum, because the gradient itself naturally shrinks to zero as the surface flattens out near the bottom of the bowl — the step size effectively tapers off on its own even though alpha never changes.

when the surface isn't a clean bowl

Everything above relies on convexity — a single minimum with no distracting dips. Linear and logistic regression's cost functions have that property by construction, which is exactly why Logistic Regression insisted on log loss instead of squared error. Neural network cost surfaces generally don't have this guarantee: they can have local minima (valleys that aren't the deepest one) and, more commonly in practice, plateaus and saddle points — large flat-ish regions where the gradient is close to zero in most directions, so plain gradient descent slows to a crawl without actually being near a good solution. This is exactly the gap that momentum-based optimizers and Adam/AdamW are built to close — covered in depth, with PyTorch code, in the Deep Learning & PyTorch Engineering track, along with learning-rate scheduling (gradually shrinking alpha over training) as a complementary fix.

practical notes

Seeing gradient descent as plain code once, before a library hides it, is worth the five minutes. This is mini-batch gradient descent for the single-feature CI-duration linear regression from earlier in this track, no framework involved:


w, b = 0.0, 0.0
alpha = 0.01

for epoch in range(num_epochs):
    for batch in mini_batches(X_train, y_train, batch_size=64):
        x_batch, y_batch = batch
        predicted = w * x_batch + b
        error = predicted - y_batch

        dw = (error * x_batch).mean()
        db = error.mean()

        w = w - alpha * dw
        b = b - alpha * db

from sklearn.linear_model import SGDRegressor| https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.html | scikit-learn's mini-batch/stochastic gradient descent linear model -- LinearRegression from earlier in this track uses the closed-form solution instead, not gradient descent at all |'gd_sk1'

That last point is worth calling out explicitly: plain linear regression rarely uses gradient descent in practice, precisely because the closed-form normal equation from the Linear Regression page solves it directly. Gradient descent earns its place once a model gets complex enough (logistic regression, and every neural network from here on) that no closed-form solution exists at all.

where to go from here

Linear Regression and Logistic Regression — the cost functions this page has been minimizing.
Next in this track: Overfitting vs Underfitting — what a low training cost from gradient descent does and doesn't guarantee.
Deep Learning & PyTorch Engineering — momentum, Adam/AdamW, and learning-rate scheduling, with runnable PyTorch code.

reference

Google ML Crash Course — Gradient Descent
scikit-learn — Stochastic Gradient Descent