Explainability

Chapter 1: Policy Visualization

The simplest form of explainability is just looking at what the policy does. If your state space is low-dimensional (2D or 3D), you can plot the policy as a map: for every point in state space, color it by what action the policy takes.

Consider an autonomous car navigating a 2D plane. The state is (x-position, y-position). The policy outputs one of four actions: accelerate, brake, turn left, turn right. A policy heatmap colors each grid cell by the chosen action. At a glance, you see "the car brakes when it's near an obstacle, accelerates on clear road, and turns when it's drifting toward a wall."

For higher-dimensional states, you fix all dimensions except two and plot a slice. If the car's full state is (x, y, speed, heading, distance_to_obstacle), you might fix speed = 30, heading = north, and plot the action as a function of (x, distance_to_obstacle). Multiple slices reveal how the policy changes with each variable.

Trajectory aggregation takes a different approach. Instead of plotting the action at every state, you overlay many simulated trajectories on the same map. Color them by outcome (green = safe, red = failure). Clusters of red trajectories reveal failure regions in state space — areas where the policy consistently makes bad decisions.

Visualization has clear limits. It works for 2D state spaces, or 2D slices of higher-dimensional ones. For a 50-dimensional observation space (like a flattened image), you need the methods in the following chapters. But as a first pass, always start here.

Chapter 2: Sensitivity Analysis

You have a policy π that maps an observation x = (x₁, x₂, …, x_n) to an action. You want to know: which features matter most? The most direct approach is to wiggle each one and see what happens.

Sensitivity analysis perturbs one input feature at a time, holding all others fixed, and measures how much the output changes. If changing x₃ by ±10% causes the action to swing wildly, then x₃ is a high-sensitivity feature. If changing x₇ barely affects the output, x₇ is low-sensitivity.

This is just the magnitude of the finite-difference derivative with respect to feature i. A large value means the policy is sensitive to that feature at this operating point.

To get a global picture, evaluate sensitivity at many operating points (sampled from your test distribution) and average. This gives mean absolute sensitivity — a rough ranking of feature importance across the whole input space.

Notice something critical: the sensitivity of x₁ depends on the current value of x₁, because the derivative of 3x₁² is 6x₁. At x₁ = 0, the sensitivity to x₁ is zero! This is a fundamental property of nonlinear functions — sensitivity is a function of where you are, not a fixed property of the feature.

Chapter 3: Saliency & Gradients

Sensitivity analysis uses finite differences — perturb by δ, measure the change, divide. But if your policy is a differentiable neural network, you can compute the exact derivative using backpropagation. This is saliency mapping.

The saliency of feature i at input x is simply the partial derivative of the policy's output with respect to that input:

Backpropagation computes all n partial derivatives in a single backward pass — O(1) cost per feature, compared to O(n) forward passes for finite differences. For a policy that takes a 224×224 image as input (50,176 features), saliency is fast; finite-difference sensitivity would require 50,176 separate evaluations.

The gradient tells you the direction of steepest ascent. The sign matters: a positive gradient on x_i means increasing x_i increases the output; negative means the opposite. The magnitude tells you how steeply.

Two common refinements:

Both help, but neither solves the saturation problem fundamentally. For that, we need the path-based method in the next chapter.

Chapter 4: Integrated Gradients

Gradients at a point are like reading the slope of a mountain at your current location. Useful, but it misses the full picture. If you walked from sea level (a baseline) to your current position, the total altitude gain is what matters — not just the local slope.

Integrated gradients formalize this idea. Instead of computing the gradient at one point, they integrate the gradient along a straight-line path from a baseline input x' (typically all zeros or a black image) to the actual input x:

Let's unpack this. The integral walks along the path from x' to x, parameterized by α ∈ [0, 1]. At each point on the path, it samples the gradient with respect to feature i. Then it multiplies by (x_i − x'_i) — the total change in that feature from baseline to actual input.

In practice, the integral is approximated by summing over m equally-spaced points along the path:

python
def integrated_gradients(model, x, baseline, m=50):
    # x, baseline: input tensors of same shape
    alphas = torch.linspace(0, 1, m+1)[1:]  # skip 0
    grads = []
    for alpha in alphas:
        interp = baseline + alpha * (x - baseline)
        interp.requires_grad_(True)
        out = model(interp)
        out.backward()
        grads.append(interp.grad.clone())
    avg_grad = torch.stack(grads).mean(dim=0)
    return (x - baseline) * avg_grad

Integrated gradients satisfy two important axioms:

Completeness is remarkable: it means the attributions are a complete accounting of why the output is what it is, relative to the baseline. Nothing is left unaccounted for.

Chapter 5: Shapley Values

Imagine three engineers — Alice, Bob, and Carol — working on a project. The project earns $100. How much credit does each person deserve?

If Alice alone produces $30, Bob alone $40, and Carol alone $10 — but Alice and Bob together produce $90 (they have synergy!) — a simple "divide by contribution" approach breaks down. Alice's value depends on whether Bob is also on the team.

Lloyd Shapley solved this in 1953 with an elegant idea from cooperative game theory. For each player, consider every possible subset of the other players. Add the player to each subset. The marginal contribution is how much the function value changes. The Shapley value is the average marginal contribution across all subsets.

Let's decode this. N is the set of all n features. S is a subset that does not include feature i. The weighting factor |S|!(n − |S| − 1)!/n! ensures we average uniformly over all possible orderings in which feature i could join the coalition. f(S) is the function value when only features in S are "present" (others set to baseline).

Let's work through a concrete example with n = 3 features. The function is:

The interaction term 3x₁x₂ means features 1 and 2 have synergy. Their combined contribution exceeds the sum of their individual contributions.

SHAP (SHapley Additive exPlanations) approximates Shapley values by sampling random subsets instead of enumerating all of them. Kernel SHAP fits a weighted linear model around the prediction. Tree SHAP exploits the structure of tree-based models for exact computation in polynomial time. These approximations make Shapley values practical for production use.

Chapter 6: Surrogate Models

What if, instead of explaining the neural network directly, we train a simpler model to mimic it — and explain that?

A surrogate model is a simple, interpretable model (typically linear regression) fitted to the neural network's predictions. The surrogate doesn't need to be perfect. It just needs to capture the important input-output relationships well enough that its coefficients are meaningful.

The weights w_i are the surrogate's "explanation." A large positive w₃ means feature 3 is strongly positively associated with the output. The sign tells direction, the magnitude tells strength.

To ensure the surrogate highlights only the most important features, use LASSO regularization (L1 penalty). LASSO drives small weights to exactly zero, producing a sparse model where only the features that truly matter have nonzero coefficients. This is a form of automatic feature selection.

The parameter λ controls the sparsity-fidelity tradeoff. Large λ produces a very simple model (few nonzero weights) that may not fit well. Small λ produces an accurate but complex model that is harder to interpret. The best practice is to report the R² score alongside the surrogate, so the reader knows how well the simple model actually approximates the original policy.

Chapter 7: Decision Trees

Linear surrogates give weights. Decision trees give rules. "If speed > 40 and distance_to_obstacle < 10, then brake. Else if heading_error > 15°, then turn. Else accelerate." A validation engineer can read this, check it against domain knowledge, and spot problems.

A decision tree surrogate is fitted to the neural network's predictions (not the true labels). The tree is grown by recursively splitting on the feature and threshold that best separates the data, then pruned to a desired depth.

The canvas shows two things: the tree's decision regions (colored areas) and the neural network's decision boundary (dashed line). At depth 1, the tree makes a single vertical or horizontal cut — a crude approximation. At depth 3-4, the axis-aligned rectangles start to match the curved boundary reasonably well. At depth 6, the match is near-perfect, but the tree has up to 64 leaf nodes — no longer a simple explanation.

Chapter 8: Counterfactuals

A patient is denied a loan. They don't care about Shapley values or saliency maps. They want one thing: "What would I need to change to get approved?"

A counterfactual explanation answers exactly this question. It finds the smallest change to the input that flips the model's decision. "If your income were $5,000 higher, or your debt were $2,000 lower, you would be approved."

Formally, given input x with undesired outcome y, we seek x' such that the model predicts the desired outcome y', while minimizing a combination of four objectives:

Counterfactuals are fundamentally different from the other methods in this chapter. Saliency, gradients, and Shapley values are attributive — they decompose the current decision. Counterfactuals are contrastive — they show what could be different. Both are valuable, but they answer different questions.

Chapter 9: Failure Mode Clustering

The methods so far explain individual decisions. But a validation engineer often needs to understand failure patterns: "Why does the system fail, and are there distinct failure types?"

Suppose you run 10,000 simulations and 200 result in failure. Looking at each one individually is tedious. Instead, you can cluster the failure trajectories to discover groups of similar failures. Each cluster represents a distinct failure mode.

The process has three steps:

The key question is: what features do you extract? Simple statistics (max speed, min distance to obstacle, duration) work for basic analysis. But for temporal patterns ("the car swerved, then overcorrected, then went off-road"), you need features that capture sequence structure. This is where temporal logic comes in (next chapter).

Choosing k (the number of clusters) matters. Too few clusters merge distinct failure modes. Too many create artificial distinctions. The elbow method plots total within-cluster variance against k and looks for the "elbow" where adding more clusters gives diminishing returns. Alternatively, silhouette scores measure how well-separated the clusters are.

Chapter 10: Temporal Logic Features

A failure trajectory is not just a point — it's a sequence of states over time. Simple features like "max speed" throw away the temporal structure. The car might have been going fast at the beginning and slow at the end, or vice versa. These are different failure patterns that "max speed" merges together.

Signal Temporal Logic (STL) provides a language for describing temporal properties of trajectories. A few examples:

Here G means "globally" (always true in the time interval) and F means "finally" (true at some point in the interval). These are the same operators from Chapter 3's property specification.

Parametric STL (PSTL) extends this by making the thresholds into parameters. Instead of "speed < 30", you write "speed < c" where c is a free parameter. Given a trajectory, you can compute the tightest parameter that makes the formula true. For a trajectory where max speed is 42, the tightest c for G(speed < c) is c = 42.

The robustness of an STL formula measures how much margin there is. A robustness of +5 for "speed < 30" means the speed stayed 5 units below 30. A robustness of −3 means it exceeded 30 by 3 units. Robustness values can also serve as continuous features for clustering — more informative than binary "satisfied/violated."