System Modeling

Chapter 1: Probability 101

Models are built from probability distributions. Before we estimate anything, we need a quick refresher on the building blocks: PMFs, PDFs, and the Gaussian.

A probability mass function (PMF) describes a discrete random variable. If a die has outcomes {1,2,3,4,5,6}, the PMF assigns a probability to each: P(X = k). For a fair die, P(X = k) = 1/6 for all k. The probabilities must sum to 1.

A probability density function (PDF) describes a continuous random variable. The probability that X falls in an interval [a, b] is the area under the curve: P(a ≤ X ≤ b) = ∫_a^b f(x) dx. The density f(x) itself can be greater than 1 — what matters is that the total area under the curve is 1.

The most important continuous distribution is the Gaussian (normal distribution), with two parameters: mean μ (center) and variance σ² (spread):

The Gaussian is everywhere in modeling. Sensor noise? Gaussian. Measurement errors? Gaussian. The central limit theorem says that the sum of many small independent effects is approximately Gaussian, regardless of the individual distributions. This is why it appears so often in practice.

Chapter 2: Mixtures & Flows

A single Gaussian is a unimodal bump. But the real world is often multimodal — a pedestrian at an intersection might go left, right, or straight, each with its own cluster of trajectories. A single Gaussian cannot capture this. We need richer distributions.

A Gaussian mixture model (GMM) solves this by combining K Gaussians, each weighted by π_k:

The weights π_k sum to 1. Each component has its own mean μ_k and variance σ_k². The mixture can have bumps wherever it needs them. With enough components, a GMM can approximate any continuous distribution.

For even more flexibility, normalizing flows take a simple base distribution (say, a standard Gaussian) and push it through a series of invertible transformations. Each transformation stretches and bends the distribution into a more complex shape. The key constraint: each transformation must be invertible, so we can compute both the forward mapping (sample generation) and the backward mapping (density evaluation).

Chapter 3: Conditional Distributions

The models from Chapter 1 describe distributions of individual variables. But validation models need conditional distributions — how one quantity depends on another. The three core conditional distributions from the POMDP framework are:

Each of these is a distribution parameterized by its conditioning variables. T(s'|s,a) is not a single distribution — it is a different distribution for every (s,a) pair. If there are 100 states and 5 actions, you need to specify 500 distributions (one per pair).

For continuous states, conditional distributions are often parameterized as Gaussians whose mean depends on the conditioning variables:

Here f(s,a) is a function (could be a physics simulator, a neural network, or a simple linear model) that predicts the expected next state. Σ captures the noise. The entire validation pipeline hinges on getting these conditional distributions right.

Chapter 4: Maximum Likelihood

You have data: N measurements x₁, x₂, ..., x_N. You believe they come from a Gaussian with unknown mean μ and standard deviation σ. How do you find the best μ and σ?

The idea: choose parameters that make the observed data most probable. This is maximum likelihood estimation (MLE). The likelihood is the probability of seeing your data given the parameters:

We want to maximize L. Since products are annoying, we take the log (the log is monotonic, so maximizing log L is the same as maximizing L):

Taking derivatives and setting them to zero gives the MLE solutions:

The MLE mean is the sample mean. The MLE variance is the average squared deviation. These are the "best-fit" parameters — the Gaussian that gives the observed data the highest probability.

python
import numpy as np

data = np.array([2.1, 3.5, 2.8, 3.2, 2.5])
mu_mle  = data.mean()               # 2.82
sig2_mle = ((data - mu_mle)**2).mean()  # 0.2456

def log_likelihood(data, mu, sig2):
    n = len(data)
    return -n/2 * np.log(2*np.pi*sig2) - sum((data - mu)**2) / (2*sig2)

Chapter 5: Bayesian Learning

MLE gives you a single best estimate. But what if you have prior knowledge? And what if you want to know how uncertain you are about the parameters? Bayesian learning answers both questions by treating the parameters themselves as random variables with a distribution.

Bayes' theorem says:

In words: the posterior (what we believe after seeing data) is proportional to the likelihood (how probable the data is given θ) times the prior (what we believed before seeing data). The denominator p(data) is just a normalizing constant.

The Beta distribution is the natural prior for a probability parameter θ ∈ [0, 1]. It is parameterized by α and β:

After observing h heads and t tails, the posterior is Beta(α + h, β + t). This is called a conjugate prior — the posterior has the same form as the prior, with updated parameters. It makes Bayesian computation tractable.

Chapter 6: Overfitting

A model that fits the training data perfectly is not necessarily a good model. Consider fitting a polynomial to 10 data points. A degree-9 polynomial passes through every single point — zero training error. But between the points, it oscillates wildly. On new data, it performs terribly. This is overfitting: the model has memorized the noise in the training data instead of learning the underlying pattern.

We detect overfitting by measuring performance on held-out test data — data the model was not trained on. The gap between training performance and test performance is the generalization gap. A large gap signals overfitting.

Cross-validation is a systematic way to tune model complexity. Split the data into K folds. Train on K−1 folds, test on the held-out fold. Repeat K times and average the test error. The model complexity (degree, number of parameters, regularization strength) that minimizes cross-validation error is the best trade-off between underfitting and overfitting.

Chapter 7: Cloning Agents

To validate an autonomous vehicle, you need to simulate not just your car, but all the other drivers on the road. Their policies π(a|o) are unknown — you cannot look inside their heads. But you have logged data: recordings of what they did in real traffic. Can you learn a policy from this data?

Behavioral cloning (BC) does exactly this: treat the logged state-action pairs as supervised learning data, and train a model to predict actions from states. Given data {(o₁, a₁), ..., (o_N, a_N)}, minimize:

where ℓ is a loss function (cross-entropy for discrete actions, MSE for continuous).

DAgger (Dataset Aggregation) is the fix. After the cloned agent makes errors and drifts, you ask the expert to label the new states the agent visits. Add these to the dataset and retrain. Iterate. This progressively covers the states the cloned agent actually encounters, closing the distribution shift gap.

Chapter 8: Behavior Models

Not every agent needs to be cloned from data. Sometimes we can build behavior models from first principles using decision-theoretic assumptions about how agents act. The key idea: agents are approximately rational, but not perfectly so.

The softmax response model says an agent picks actions with probability proportional to the exponentiated value of each action:

Here Q(s, a) is the value of taking action a in state s, and λ is the rationality parameter. When λ = 0, the agent is completely random (uniform over actions). When λ → ∞, the agent always picks the best action (perfectly rational). Real agents live somewhere in between.

Level-k reasoning takes this further. A level-0 agent acts randomly. A level-1 agent best-responds to level-0 agents. A level-2 agent best-responds to level-1 agents. And so on. This hierarchy captures the idea that not everyone thinks the same number of steps ahead.

Chapter 9: Validating Models

You built a model. But is it any good? A model that does not match reality is worse than useless — it gives false confidence. This chapter covers three tools for checking model quality.

Q-Q plots (quantile-quantile plots) compare the distribution of model outputs against real data. If the model is correct, the Q-Q plot is a straight diagonal line. Deviations reveal systematic mismatches: curved means the tails are wrong, offset means the mean is wrong, S-shaped means the variance is wrong.

Calibration plots check probabilistic predictions. If the model says P(crash) = 0.3, then among all situations where it predicted 0.3, about 30% should actually be crashes. A well-calibrated model has its calibration curve on the diagonal. Overconfident models curve below; underconfident models curve above.

KL divergence measures how different two distributions are:

If p is the true distribution and q is the model, D_KL(p || q) = 0 means a perfect match. Larger values mean the model is farther from reality. KL divergence is asymmetric: it penalizes the model heavily for assigning low probability to events that are actually common (q(x) small where p(x) is large).

Chapter 10: Summary

This chapter built the modeling toolkit. To validate a system in simulation, you need models of the world (transitions), the sensors (observations), and other agents (policies). Each model is a probability distribution estimated from data, and each must be validated against reality before trusting it.