Model Explainability: Feature Importance, Partial Dependence, and SHAP

three different questions about "why"

"Explain the model" collapses three genuinely different questions into one phrase, and conflating them is the most common way explainability discussions go in circles:

Question	Tool
Which features did the model rely on most, overall?	Feature importance
How does one specific feature affect predictions, across its whole range?	Partial dependence plots
For this one prediction, right here, how much did each feature contribute?	SHAP values

All three are computed after a model is already trained — they explain a fixed, already-fitted model's behavior, rather than changing how it trains.

feature importance via permutation

Ensemble Methods already introduced one flavor of feature importance — model.feature_importances_, built into tree-based models from how much each feature reduced impurity across all their splits. Permutation importance answers the same question a different, more general way, and works for any model, not just trees:


1. take the trained model and the validation set (never the training set)
2. record its baseline performance (accuracy, F1, whatever metric matters)
3. for each feature, one at a time:
     a. randomly shuffle just that column's values across the validation rows
        (everything else, including the target, stays in place)
     b. re-score the model on this corrupted data
     c. importance = how much performance dropped from the baseline
4. repeat the shuffles a few times per feature and average, to smooth out randomness

Shuffling touches_slow_suite destroys any real relationship between that column and the actual outcome, without touching the model itself — if the model leaned on it heavily, performance craters; if it barely mattered, performance barely moves. Occasionally a feature shows a slightly negative importance — the shuffled version scored better by chance — which just means that feature's true importance was close to zero and the difference is noise, more common on small validation sets where there's more room for a lucky shuffle.

partial dependence plots: how one feature affects predictions

Feature importance says lines_changed matters a lot; it says nothing about the shape of that relationship. A partial dependence plot (PDP) fills that in: hold every other feature fixed, sweep lines_changed across its full range of observed values, and at each value, ask the model for its predicted failure probability. Repeat across many rows and average, then plot predicted probability against lines_changed.

The resulting curve reveals structure a single number never could: maybe risk barely moves below 500 lines changed, then climbs sharply, then flattens out again past 3,000 — a shape no linear regression coefficient can express, but exactly the kind of pattern a decision tree or gradient-boosted ensemble can learn and a PDP can reveal after the fact. A 2D partial dependence plot extends the same idea to two features at once — sweeping both lines_changed and hour_of_day together — to surface interaction effects that neither feature's individual PDP would show.

SHAP: explaining one single prediction

Neither tool above answers the question an on-call engineer actually asks when a specific run gets flagged: why did the model predict an 85% failure probability for THIS run, specifically? SHAP (SHapley Additive exPlanations) decomposes one single prediction into a per-feature contribution, with a guarantee that makes it more than a rough heuristic:


sum(SHAP value for every feature) = prediction_for_this_run - baseline_prediction

The baseline is roughly the model's average prediction across the whole dataset. For one flagged run, SHAP might attribute +0.30 to touches_slow_suite, +0.20 to an unusually large lines_changed, and -0.05 to a normally low-risk hour_of_day — numbers that add up exactly to the gap between this run's 85% prediction and the baseline. That's a genuinely different, more actionable answer than "this feature matters on average" — it's "this feature is why this prediction, specifically, came out the way it did," the same kind of per-decision accountability a bank needs to explain a loan rejection or a healthcare model needs to justify a risk score.

practical notes

from sklearn.inspection import permutation_importance| https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html | n_repeats controls how many shuffles get averaged per feature -- the importances_std attribute on the result is the noise measure referenced above |'me_sk1'

from sklearn.inspection import PartialDependenceDisplay| https://scikit-learn.org/stable/modules/generated/sklearn.inspection.PartialDependenceDisplay.html | plots one or more PDPs directly; pass a pair of feature names for a 2D interaction plot |'me_sk2'

import shap; explainer = shap.TreeExplainer(model)| https://shap.readthedocs.io/en/latest/ | TreeExplainer is the fast, exact path for tree-based models from the Ensemble Methods page; explainer.shap_values(X) returns the per-row, per-feature decomposition |'me_sk3'

where to go from here

Ensemble Methods — the built-in tree feature_importances_ this page's permutation-importance section builds on and generalizes beyond.
Next in this track: Support Vector Machines — the kernel trick.
Data Leakage — a suspiciously dominant feature in an importance ranking is one of that page's red flags for leakage.

reference

scikit-learn — Permutation Importance
SHAP documentation