Support Vector Machines and the Kernel Trick

the idea: the widest possible margin

Logistic Regression finds a linear decision boundary by minimizing log loss over every training point. A support vector machine (SVM) finds a linear boundary too, but with a completely different objective: instead of best fitting every point's probability, it looks for the boundary that leaves the widest possible gap — the margin — between the two classes. Picture the CI pass/fail data plotted on lines_changed and touches_slow_suite: many different straight lines could separate passes from failures, but an SVM specifically picks the one positioned exactly in the middle of the gap between the closest pass and the closest failure, as far from both as the data allows.

support vectors: only the closest points matter

The boundary is fully determined by the handful of points sitting right on the edge of the margin — the support vectors that give the algorithm its name. Every other training point, however far it sits from the boundary, could be moved anywhere further into its own territory without changing the decision boundary at all. That's a genuinely different relationship to the data than logistic regression, where every single training point contributes to the loss and therefore to where the boundary ends up. An SVM effectively throws away most of the training set once fitted and keeps only the few points that were actually decisive.

soft margins: allowing some violations

Real CI data is rarely perfectly separable by a straight line — some passing runs will sit deep in "failure territory" on the feature plane and vice versa. A hard margin (no violations allowed at all) would either fail outright or contort itself around a handful of outliers. A soft margin allows some points to sit inside the margin, or even on the wrong side of the boundary entirely, at a cost — and the hyperparameter C controls how much that cost is weighed against margin width. A large C penalizes violations heavily, producing a narrower margin that hugs the training data closely (more variance, echoing Overfitting vs Underfitting); a small C tolerates more violations in exchange for a wider, smoother margin (more bias). As with regularization's lambda, the right value is found by sweeping candidates against a validation set, never guessed.

the kernel trick: separating what isn't linearly separable at all

Some patterns have no linear boundary whatsoever — imagine failures clustered in a blob surrounded by passing runs on every side; no straight line separates a ring from its center. The SVM's optimization and decision rule, it turns out, never actually need the raw feature vectors — they only ever need the dot product between pairs of points. A kernel function K(x, x') computes exactly what that dot product would be if both points were first transformed into some higher-dimensional space — without ever constructing that transformation explicitly. In the transformed space, a pattern that was a ring in the original two features can become linearly separable by a flat plane; the kernel delivers the benefit of that transformation at the computational cost of working in the original, low-dimensional space.

common kernels

Kernel	What it captures
Linear	no transformation at all — equivalent to the plain margin-maximizing classifier described above
Polynomial	curved boundaries and feature interactions up to a chosen degree, similar in spirit to the polynomial regression from Linear Regression
RBF (Gaussian)	an implicit infinite-dimensional transformation — the default choice when there's no strong prior belief about the boundary's shape, and the most commonly used kernel in practice

gamma and C: the two knobs of an RBF SVM

The RBF kernel adds a second hyperparameter, gamma, controlling how far a single training point's influence reaches. A small gamma means each point influences a wide region, producing a smooth, simple boundary (more bias); a large gamma means influence drops off sharply, letting the boundary bend tightly around individual points — including noisy ones (more variance). Between C and gamma, an RBF SVM has two separate dials for the exact same bias/variance tradeoff covered in Overfitting vs Underfitting, and both are tuned together via the same cross-validation sweep from Train/Test Splits & Cross-Validation, not independently.

why SVMs need scaled features

Like k-means in Unsupervised Learning, an SVM's margin is a geometric, distance-based notion — an unscaled lines_changed (ranging into the thousands) would swamp a 0/1 feature like touches_slow_suite in that geometry, regardless of which one actually separates the classes better. The z-score standardization from Feature Engineering, fit on the training split only, is close to mandatory before fitting any SVM.

practical notes

from sklearn.svm import SVC; SVC(kernel='rbf', C=1.0, gamma='scale')| https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html | gamma='scale' (the default) sets gamma automatically based on the feature variance -- a reasonable starting point before sweeping manually |'svm_sk1'

from sklearn.svm import LinearSVC| https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html | a specialized, much faster solver for the linear-kernel case only -- prefer this over SVC(kernel='linear') on larger datasets |'svm_sk2'

SVMs scale poorly to very large datasets — training cost grows faster than linear in the number of examples for the kernelized case — which is a real reason gradient-boosted trees have displaced them as the default choice for large tabular problems in recent years. They remain a strong, well-understood option on small-to-medium datasets, especially when the number of features is large relative to the number of examples.

where to go from here

Logistic Regression — the margin-free linear classifier this page's whole approach is defined against.
Next in this track: Bayesian Learning and Naive Bayes.
Overfitting vs Underfitting — the bias/variance framing behind both C and gamma.

reference

scikit-learn — Support Vector Machines
scikit-learn — RBF SVM Parameters