Unsupervised Learning: K-Means Clustering and PCA

no labels, just structure

What Is Machine Learning, Actually? previewed this exact scenario: cluster the CI runs that failed into a handful of recurring failure "shapes" — nobody has pre-labeled them as "flaky test," "timeout," or "compile error"; the goal is to discover that such groups exist at all, purely from how similar the failed runs look to each other across features like duration, lines changed, and time of day. Every algorithm earlier in this track needed a labeled y to learn from. Unsupervised learning works with X alone.

k-means: two alternating steps

K-means groups points by straight-line (Euclidean) distance to one of k centroids — points in the feature space that don't have to coincide with any actual data point. The algorithm alternates between two simple steps until nothing changes:


1. initialize k centroids (e.g. randomly chosen from the data points)
2. repeat until convergence:
     a. ASSIGN: put every point into the cluster of its nearest centroid (centroids held fixed)
     b. UPDATE: move each centroid to the mean position of the points now assigned to it (assignments held fixed)

Each step only ever improves (or leaves unchanged) the same underlying cost: the total squared distance from every point to its assigned centroid, sometimes called distortion. Reassigning a point always moves it to a strictly closer centroid or leaves it where it was, and recomputing a centroid as the mean of its cluster always minimizes total squared distance for that fixed assignment — so the cost can only go down or plateau, which is why the algorithm is guaranteed to converge. This is a genuinely different kind of optimization than gradient descent — no learning rate, no derivatives, just alternating exact minimizations — but the same underlying goal of driving a cost function down step by step.

picking k: the elbow method

Nothing about the algorithm above says what k should be — that's chosen beforehand, and there's no single correct answer. One practical approach: run k-means for a range of k values, plot the final distortion against k, and look for the elbow — the point where adding another cluster stops meaningfully reducing the cost. (Distortion decreases monotonically as k grows, hitting zero when k equals the number of data points — every point its own cluster, which is obviously not a useful clustering.) If the failure data genuinely separates into around four recurring shapes, the elbow plot should show a sharp drop from k=1 through k=4, then a much flatter slope afterward, as additional clusters start splitting real groups apart rather than separating genuinely different ones.

centroid initialization: getting stuck in a local minimum

K-means always converges — but not necessarily to the best clustering. A different random starting position for the centroids can converge to a different, worse local minimum of the distortion cost, the same local-minimum concern raised for non-convex surfaces in Gradient Descent. The standard mitigation is simply to run k-means several times from different random initializations and keep whichever run produced the lowest final distortion — cheap insurance against a single unlucky starting position.

k-means is sensitive to scale

Because clustering is entirely built on straight-line distance, the scaling discipline from Feature Engineering isn't optional here — it's load-bearing. Left unscaled, lines_changed (ranging into the thousands) would dominate every distance calculation over time_of_day (ranging 0-23), and the resulting clusters would essentially just be "small diffs" vs. "large diffs," ignoring whatever timing signal actually exists in the data. Standardizing every feature (z-score normalization, from that same page) before clustering is close to mandatory.

PCA: describing the data by its axes of variation

K-means partitions the data points into groups. PCA (Principal Component Analysis) does something orthogonal: it partitions the variation in the data. Instead of describing each failed run using the original features (duration, lines_changed, hour_of_day, ...), PCA finds new features — principal components — that are weighted combinations of the originals, chosen so the first component captures as much of the data's total variation as possible, the second component captures the most variation left over that's uncorrelated with the first, and so on. The weights defining each component are called loadings. Like k-means, PCA is scale-sensitive and is standard practice to run only on already-standardized data.

what PCA is actually useful for

Use case	Why it works
Dimensionality reduction	when features are highly correlated (redundant), most of the real variation collapses into just the first few components — the rest can often be dropped with little information lost
Visualization	projecting many features down to the first 2-3 components makes it possible to actually plot and eyeball the failure clusters, something impossible to do directly with a dozen raw features
Noise reduction	shared background noise across correlated sensor-like features often concentrates in the low-variance components, which can be discarded to boost signal-to-noise
Decorrelation	algorithms that struggle with correlated inputs get uncorrelated components instead — a different fix for the same multicollinearity that motivates dropping redundant features elsewhere

Each component comes with an explained variance ratio — what fraction of the data's total variation that one component accounts for. That's a measure of statistical significance, not predictive usefulness: a low-variance component can still turn out to be the one that actually correlates with CI failures, and a high-variance one can be irrelevant noise as far as the target is concerned. What "important" means still depends on what's being predicted.

practical notes

from sklearn.cluster import KMeans| https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html | n_init runs the whole algorithm that many times from different random starts and keeps the best -- scikit-learn's default already implements the local-minimum mitigation above |'ul_sk1'

from sklearn.metrics import silhouette_score| https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html | a second, complementary way to pick k -- scores how well-separated the resulting clusters are, rather than just how low the distortion is |'ul_sk2'

from sklearn.decomposition import PCA; PCA(n_components=2)| https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html | fit_transform() returns the projected components; explained_variance_ratio_ gives the per-component variance breakdown from above |'ul_sk3'

Pipeline order matters: standardize first, then run PCA or k-means on the standardized data — never the other way around, and never re-fit the scaler after the fact on a subset of the data.

where to go from here

Next in this track: Anomaly Detection — catching the outliers supervised learning misses, a close relative of the unsupervised ideas on this page.
Feature Engineering — the scaling discipline this page depends on throughout.
What Is Machine Learning, Actually? — where this exact clustering example was first previewed.

reference

scikit-learn — K-Means
scikit-learn — PCA