Feature Engineering: Scaling, Normalization, and Encoding Categorical Data
Every model in this track assumes clean, numeric, comparably-scaled input. Real data almost never arrives that way — this is the work that closes the gap.
Intermediate
missing values: first, don't guess blindly
Before filling in a missing value, ask why it's missing — the answer changes what "filling
it in" even means. If author_seniority_years is blank because the run was
triggered by an automated dependency-bot rather than a human, there's no real number to
recover; the value is missing because the underlying quantity genuinely doesn't apply, and
inventing one would be fabricating information. If lines_changed is blank because
of a one-off logging bug on a specific date, the true value existed and simply wasn't captured
— that's a case worth trying to recover or estimate.
| Approach | Trade-off |
|---|
| Drop the column | simplest, but throws away every other value in that column, even the useful ones — only reasonable when the vast majority of a column is missing |
| Impute (fill with the mean, median, or most common value) | keeps the column and the row; the filled value is never exactly right, but this usually beats dropping the column entirely |
| Impute + add a "was missing" indicator column | keeps the imputed guess honest by giving the model a way to tell "this was a real measurement" from "this was filled in" — sometimes meaningfully better, sometimes no different, worth testing |
scaling vs. normalization: not the same operation
The two terms get used interchangeably in casual conversation, but they do different
things. Scaling changes the range of a feature — squeezing
lines_changed (which might span 1 to 5,000) and files_changed
(which might span 1 to 40) onto a comparable range, without changing the shape of their
distributions. Normalization changes the shape of a feature's
distribution — reshaping it to look more like a standard bell curve, for algorithms that
assume roughly normally-distributed input.
The practical motivation for scaling connects directly back to
Gradient Descent: features on wildly different scales
distort the cost surface into a long, narrow valley instead of a round bowl, and gradient
descent zig-zags slowly down a narrow valley instead of heading straight for the minimum.
Distance-based algorithms have their own, separate reason to care — k-nearest neighbors and
SVMs (both covered later in this track) measure literal distance between points, so an
unscaled feature with a huge numeric range would dominate that distance calculation regardless
of how relevant it actually is.
two ways to scale: mean normalization and z-score
# mean normalization -- squeezes values into roughly [-1, 1]
x_scaled = (x - mean(x)) / (max(x) - min(x))
# z-score normalization (standardization) -- centers on 0 with unit spread,
# not bounded to a fixed range the way mean normalization is
x_scaled = (x - mean(x)) / std(x)
Both center the data around zero; the difference is what they divide by. Mean
normalization uses the full range (max minus min), so it's sensitive to a single extreme
outlier — one 50,000-line diff stretches the denominator for every other value.
Z-score normalization divides by the standard deviation instead, which is far less distorted
by one extreme point, and is the more common default for exactly that reason.
the leakage trap: fit scalers on training data only
The mean and standard deviation used to scale a feature must be computed from the
training split only, then applied unchanged to the validation and test splits.
Computing them from the full dataset before splitting leaks information from validation/test
rows into the numbers used to transform the training data — a subtle version of exactly the
leakage problem covered in Train/Test Splits &
Cross-Validation. In code, this means calling .fit() only on the training
split, then .transform() (never .fit() again) on validation and
test — a distinction the scikit-learn API is deliberately built around.
encoding categorical features
Linear Regression already flagged the core trap:
assigning arbitrary integers to categories (0, 1, 2...) implies an ordering and a distance
between categories that usually isn't real. The right encoding depends on whether that
ordering genuinely exists.
| Encoding | Use when | Example |
|---|
| Ordinal / label encoding | categories have a real, meaningful order | test_severity: low=0, medium=1, high=2 |
| One-hot encoding | categories have no order, and there aren't too many of them | branch_type: feature/release/hotfix → three binary columns |
| Target encoding | too many distinct categories for one-hot to be practical (high cardinality) | author, with hundreds of distinct contributors |
One-hot encoding turns a k-category feature into k binary columns — clean and
unambiguous, but a author column with 400 distinct contributors would explode
into 400 mostly-empty columns. Target encoding solves the high-cardinality
case differently: replace each category with a number derived from the target itself — each
author's historical failure rate, for instance, instead of an arbitrary index or a wall of
one-hot columns.
Target encoding has a sharp edge worth naming explicitly: computing "this author's failure
rate" directly from the same rows being used to train the model leaks the label into the
feature — a new, subtler instance of the leakage problem raised twice already on this page.
The standard fix is smoothing, blending each category's own average with the
overall average, weighted by how much data that category actually has:
weight = n / (n + m) # n = how many times this author appears in training data
# m = a smoothing hyperparameter, larger m trusts the overall average more
encoded_value = weight * this_authors_failure_rate + (1 - weight) * overall_failure_rate
A prolific author with hundreds of runs gets a weight close to 1 — their own
history is trusted. A first-time contributor with one run gets a weight close to
0 — their single data point is too noisy to trust alone, so the encoding falls back toward the
overall average instead of overreacting to one observation.
creating new features, briefly
Not all feature engineering is about reshaping existing columns — sometimes the most
useful feature is a new one, built by combining what's already there. lines_changed /
files_changed (average lines touched per file) can carry signal that neither input
carries alone; a count feature like "number of risk factors present" (touches core code AND
skips tests AND is a large diff, summed as 0/1 flags) aggregates several weak signals into one
stronger one. This kind of feature creation is where domain knowledge about CI systems — or
whatever the actual problem is — tends to matter more than any general-purpose technique.
from sklearn.preprocessing import StandardScaler| https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html | z-score normalization -- scaler.fit(X_train) then scaler.transform(X_train) and scaler.transform(X_test), never fit() on test data |'fe_sk1'
from sklearn.preprocessing import OneHotEncoder| https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html | handle_unknown='ignore' avoids a crash when the test split contains a category never seen during training |'fe_sk2'
from sklearn.compose import ColumnTransformer| https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html | applies different preprocessing to different columns (scale the numeric ones, one-hot the categorical ones) in a single fit/transform step |'fe_sk3'
df['lines_changed'].isnull().sum()| | pandas' standard way to audit missing values per column before deciding drop vs. impute |'fe_sk4'