Intro to Validation

Chapter 1: The V-Model

Building a safety-critical system is not "write code, then test." There is a structured development lifecycle, and it has a name: the V-model. Imagine the letter V. The left side goes down (decomposing requirements into design into code). The right side goes up (testing each level against its corresponding requirement).

At the top-left, you start with stakeholder needs: "the car must not hit pedestrians." This decomposes into system requirements ("detect pedestrians within 50m"), then architecture ("use LiDAR + camera fusion"), then detailed design ("run YOLO at 30fps"), then implementation (actual code). That is the left side — the descent from abstract to concrete.

The right side mirrors it. Unit tests check the code. Integration tests check the architecture. System tests check the full requirements. And acceptance tests check the original stakeholder needs. Each level on the right validates the corresponding level on the left.

Traditional V-model validation uses testing: run the system, see what happens. But for autonomous systems, testing alone is not enough. A self-driving car would need to drive billions of miles to statistically prove it is safer than humans. We cannot wait that long. This book's algorithms — falsification, reachability, importance sampling — are designed to replace brute-force testing with smarter, targeted analysis.

Chapter 2: When Things Break

Validation is not abstract. Real systems have failed, and real people have been harmed. Understanding these failures is the best way to internalize why every algorithm in this book exists.

Notice the pattern. In every case, the system did what it was designed to do. Therac-25 ran the software correctly (the race condition was a design flaw). MCAS followed its algorithm (the single-sensor architecture was the flaw). Amazon's model optimized its loss function (the training data encoded bias). Validation failures are not about bugs in the usual sense — they are about the gap between what you built and what you should have built.

Chapter 3: System = Agent + World

To validate a system, we first need a formal language for describing it. The book models autonomous systems as an agent interacting with a world. The agent receives observations, takes actions, and the world transitions to new states. This is the POMDP framework — a partially observable Markov decision process.

Think of a self-driving car. The state s is everything about the world: positions and velocities of every vehicle, the road layout, weather, traffic signals. The agent does not see this directly — it receives observations o through noisy sensors (cameras, LiDAR, radar). Based on observations, it takes actions a (steer, accelerate, brake). The world then transitions to a new state s' according to its dynamics.

Here, S is the state space (all possible world configurations), A is the action space (everything the agent can do), and O is the observation space (everything the agent can see). The dynamics are governed by two functions:

Chapter 4: Observations & Policies

The agent needs a strategy for choosing actions. This strategy is called a policy, written π. A policy maps from what the agent has seen (observations) to what it does (actions).

Policies can be deterministic — given an observation, the agent always picks the same action: a = π(o). Or they can be stochastic — the agent samples actions from a probability distribution: a ~ π(a|o). A stochastic policy adds randomness, which can help with exploration or robustness.

Why does partial observability matter for validation? Because the agent's behavior depends on what it sees, not what is true. If a pedestrian is behind an occlusion, the true state includes the pedestrian, but the observation does not. A policy that is safe given full state information may be dangerous given partial observations.

In the POMDP framework, the agent often maintains a belief state b — a probability distribution over possible states given all past observations. The policy then maps beliefs to actions: π(b) or π(a|b). This is how the agent deals with uncertainty: it tracks what it thinks the world looks like, and acts accordingly.

Chapter 5: Trajectories

When we run a system, we get a sequence of states, observations, and actions unfolding over time. This sequence is called a trajectory, denoted τ:

A trajectory is one complete "story" of the system — from initial conditions to termination. If the system is stochastic (random transitions, noisy sensors, stochastic policy), then each run produces a different trajectory. The trajectory distribution p(τ) captures the probability of every possible story.

Let's derive the trajectory density. Each trajectory is a chain of transitions, observations, and actions. By the chain rule of probability:

Reading left to right: we start in state s₁ (with probability p(s₁)), observe o₁ (with probability O(o₁|s₁)), take action a₁ (with probability π(a₁|o₁)), transition to s₂ (with probability T(s₂|s₁, a₁)), and so on.

Generating trajectories is called rollout. You sample s₁ from the initial distribution, observe, act, transition, repeat. Each rollout is one Monte Carlo sample from p(τ). Simple, but exponentially many trajectories exist — most of the interesting (dangerous) ones are vanishingly rare.

pseudocode
# Rollout: sample one trajectory from p(tau)
def rollout(T, pi, transition, observe, p_init):
    s = sample(p_init)        # initial state
    tau = [s]
    for t in range(T):
        o = sample(observe(s))  # O(o|s)
        a = sample(pi(o))       # pi(a|o)
        s = sample(transition(s, a)) # T(s'|s,a)
        tau.append((o, a, s))
    return tau

Chapter 6: Specifications

A trajectory tells us what happened. A specification tells us whether what happened was acceptable. Formally, a specification is a function that takes a trajectory and returns true (pass) or false (fail):

Think of it as a referee. The trajectory is the game tape. The specification watches the tape and blows the whistle if anything went wrong.

Some specifications are simple: "the car never exceeds 65 mph." Others are complex: "if a pedestrian enters the crosswalk, the car must stop within 3 seconds." The complexity of the specification determines how hard it is to validate — a speed limit check is trivial, a temporal safety constraint requires reasoning about sequences of events.

This book uses specifications of increasing complexity:

The simplest specification is falsification: find any trajectory where ψ(τ) = false. Even one counterexample proves the system is flawed. The hardest is formal verification: prove that ψ(τ) = true for all possible trajectories. Most validation lives between these extremes — estimating the probability that ψ fails, or bounding the set of reachable states.

Chapter 7: Algorithm Outputs

Different validation algorithms answer different questions. The book organizes them by what they produce:

These are not competing approaches — they are complementary. Falsification finds bugs quickly. Probability estimation quantifies residual risk. Formal methods provide guarantees on simplified models. Explanations help engineers fix the root cause. Runtime monitoring is the last line of defense.

Chapter 8: Swiss Cheese

No single validation method catches everything. James Reason's Swiss Cheese Model of accident causation provides the right mental model: think of each validation layer as a slice of Swiss cheese. Each slice has holes — weaknesses, blind spots, things it misses. A failure occurs only when the holes in every layer happen to line up, letting a hazard pass through all defenses.

The more layers you have, the less likely it is that all holes align. Formal verification has holes (it works on simplified models, not the real system). Testing has holes (it only covers scenarios you thought of). Runtime monitoring has holes (it can only detect problems it was designed to recognize). But together, the uncovered region shrinks dramatically.

The lesson is profound: validation is not a single activity. It is a portfolio. The book's algorithms — falsification (Chapters 4-5), failure analysis (6-7), reachability (8-10), explainability (11), runtime monitoring (12) — are not alternatives. They are layers of cheese. A responsible validation effort uses as many as the budget allows.

Chapter 9: Summary

This chapter introduced the validation problem. An autonomous system is an agent interacting with a world through noisy sensors, producing trajectories. Validation asks: do these trajectories satisfy our specifications? And do we have enough evidence to trust the answer?