When Models Meet Data

Chapter 1: Data as Vectors

Before any model can learn, we need to convert the messy real world into numbers. Every observation becomes a feature vector x ∈ R^D — a list of D numbers that describe it. A house might become x = [1200, 3, 1975, 2] (square feet, bedrooms, year built, bathrooms).

The collection of N observations forms a data matrix X ∈ R^N×D, where each row is one example and each column is one feature. The corresponding outputs stack into a vector y ∈ R^N.

But raw features are often on wildly different scales. Square footage ranges from 500 to 5000; the number of bedrooms is 1 to 6. A model that treats these equally will be dominated by whichever feature has the largest numbers.

Two essential transformations fix this:

Other encodings handle non-numeric data. A categorical feature like "color" with values {red, green, blue} becomes a one-hot vector: red = [1,0,0], green = [0,1,0], blue = [0,0,1]. This converts every observation into a numeric vector without imposing an artificial ordering.

Chapter 2: Predictors & Functions

With data in hand, we need a machine that takes in an input x and spits out a prediction. There are two very different philosophies about what this machine should look like.

Deterministic predictors give a single crisp answer: f(x) = ŷ. A straight line through a scatter plot. A neural network that outputs one number. Given the same input, the same output, every time.

Probabilistic predictors give a whole distribution: p(y | x, θ). Instead of saying "the house price is $350,000," they say "the house price is normally distributed with mean $350,000 and standard deviation $20,000." This is richer — we get a best guess and our uncertainty about it.

The second approach is strictly more informative. It includes the first as a special case (just take the mean of the distribution). And it gives us a principled way to say "I'm not sure" — which is exactly what we need to fight overfitting.

A model class is the family of functions we consider. Choosing "linear functions" means f(x) = θ₀ + θ₁x. Choosing "polynomials of degree 5" means f(x) = θ₀ + θ₁x + … + θ₅x⁵. The model class is a design choice we make before seeing data. It constrains which functions are even possible. A bad choice — too simple or too complex — dooms us before training even begins.

Chapter 3: Loss & Risk

We have a predictor with parameters θ. We have data. How do we measure whether θ is any good? We need a loss function — a number that says how bad a prediction is.

The most common choice is squared loss: the square of the difference between the prediction and the truth.

Why squared? Two reasons. First, it penalizes big errors more than small ones (an error of 10 costs 100, not 10). Second, it leads to beautiful closed-form solutions when combined with linear models and Gaussian noise, as we will see in Chapter 6.

A single data point's loss is not enough. We care about performance across all N training examples. The empirical risk is the average loss over the training set:

This is what most ML algorithms actually minimize. Gradient descent, normal equations, Newton's method — they all find the θ that makes R_emp as small as possible.

Let's compute a concrete example. Suppose we have 3 data points and a linear predictor f(x) = θx with θ = 2:

That is our empirical risk for θ = 2. Different values of θ would give different risks. The optimal θ minimizes this number over the training set.

Chapter 4: Regularization

Minimizing empirical risk alone has a dangerous failure mode. If your model is flexible enough, it can drive the training loss to zero by memorizing every data point — including the noise. The model contorts itself into wild shapes to hit every dot exactly, then makes absurd predictions on new data.

The fix: add a penalty for complexity. Regularization says: "I want small risk, but I also want simple parameters." We add a regularization term to the objective:

The term ||θ||² = ∑_d θ_d² penalizes large parameter values. The hyperparameter λ controls the trade-off: λ = 0 means pure empirical risk (no regularization); large λ forces parameters toward zero (extreme simplicity).

This specific form is called L2 regularization, Tikhonov regularization, or ridge regression (when applied to linear models). It has a deep connection to Bayesian inference that we will uncover in Chapter 7: adding ||θ||² is equivalent to placing a Gaussian prior on θ.

Other regularizers exist. L1 regularization (λ·∑|θ_d|) encourages sparsity — it drives some parameters exactly to zero, effectively doing feature selection. Dropout in neural networks is another form. All share the same spirit: trade a little training accuracy for better generalization.

Chapter 5: Cross-Validation

We have a knob λ that controls regularization. We have a knob for model complexity (polynomial degree, number of layers). How do we set these hyperparameters? We cannot use the training loss — it always prefers the most complex model.

The idea: pretend some of your training data is "unseen." Train on part of the data, evaluate on the held-out part. This simulates the generalization test.

K-fold cross-validation makes this systematic. Split the data into K equal chunks (folds). Train on K−1 folds, evaluate on the remaining fold. Rotate which fold is held out. Average the K test scores.

For hyperparameters like λ, use nested cross-validation: an outer loop to estimate generalization error, and an inner loop to select the best λ. This avoids the subtle bias of selecting hyperparameters on the same data used for evaluation.

Chapter 6: Maximum Likelihood

So far we have been vague about how to set parameters. "Minimize the loss" is intuitive, but where does the loss come from? Maximum likelihood estimation (MLE) gives a principled, probabilistic answer.

The idea: assume the data was generated by a probabilistic model p(y | x, θ). Now ask: for which θ is the observed data most probable?

If the data points are independent, the joint probability factors into a product. Products are annoying to optimize (tiny numbers multiply to underflow), so we take the logarithm. Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood (NLL):

Let's verify with numbers. Suppose σ² = 1 and our 3 data points from Chapter 3:

Each row's NLL is (1/2)(y_n − ŷ_n)² + ¹/₂log(2π). The constant 0.919 = ¹/₂log(2π) drops out during optimization, leaving just the squared errors.

Chapter 7: MAP & Priors

Maximum likelihood treats all parameter values as equally plausible before seeing data. But sometimes we have prior beliefs. Maybe we think parameters should be small. Maybe we have domain knowledge. Can we incorporate that?

Yes. Maximum a posteriori (MAP) estimation uses Bayes' theorem to combine the likelihood with a prior distribution on θ:

The posterior is proportional to the likelihood times the prior. MAP finds the peak of this posterior:

Now here is the punchline. Suppose the prior is Gaussian: p(θ) = N(0, τ²I). Then:

Let's trace the math explicitly. With Gaussian likelihood and Gaussian prior:

Multiply through by 2σ²:

That is exactly least squares + λ||θ||² with λ = σ²/τ². If the prior variance τ² is small (we strongly believe θ ≈ 0), λ is large — heavy regularization. If τ² is huge (weak prior), λ ≈ 0 — almost no regularization, and MAP reduces to MLE.

Chapter 8: Bayesian Inference

MLE gives one answer: the single best θ. MAP gives one answer with a prior bias. But both collapse the posterior into a single point. What if we kept the entire posterior distribution?

Bayesian inference does not commit to a single θ. Instead, it computes the full posterior p(θ | X, y), which tells us how plausible each θ is given the data. Predictions average over all parameter values, weighted by their plausibility:

This is the posterior predictive distribution. Instead of plugging in one θ̂, we integrate over all possible θ, weighting each by the posterior. If many different θ values are plausible, the predictive distribution will be wide — reflecting our uncertainty.

The challenge: computing the posterior requires the normalizing constant (the marginal likelihood or evidence):

This integral is often intractable. But for specific choices of likelihood and prior (called conjugate pairs), it has a closed form. The Gaussian likelihood + Gaussian prior is the most important conjugate pair, and it gives us closed-form Bayesian linear regression (Chapter 9).

Chapter 9: Model Selection

We have seen how to fit parameters inside a model. But how do we choose between models themselves? A linear model vs. a degree-5 polynomial vs. a degree-20 polynomial — which one is "best"?

Cross-validation is one answer. But there is a more principled Bayesian answer: the marginal likelihood (also called the model evidence):

This is the probability of the observed data under model M, averaging over all possible parameter values. It automatically balances fit against complexity — no hyperparameter tuning needed.

To compare two models M₁ and M₂, compute their Bayes factor:

BF > 1 favors M₁. BF < 1 favors M₂. It is the Bayesian equivalent of a hypothesis test, with a natural interpretation: how many times more likely is the data under M₁ than under M₂?

Chapter 10: Summary

This chapter built the bridge from pure math to machine learning. Every concept addresses the same question from a different angle: how do we learn from data without being fooled by noise?

The deep connections we uncovered:

What comes next: Chapter 9 takes these ideas and applies them to the single most important model in all of ML: linear regression. We will derive the closed-form MLE solution, add Bayesian uncertainty bands, and watch the posterior update as data arrives.