Model Selection

You recorded one neuron and fitted a filter of 100 weights to it. Pull the weights toward a prior and the filter calms down. Pull too hard and it goes flat. Cross-validation and evidence both find the right pull by asking how well the model predicts data it did not tune on.

Learn how to set the strength of a prior from the data alone, by held-out prediction or by the marginal likelihood, and why the two usually agree.

Drag the penalty from weak to strong and watch the estimated filter go from noisy to flat. Then press Best by evidence and Best on held-out data. Then we build, piece by piece, the prior, the penalty, the two selection rules and the two-step method called Empirical Bayes.

You need a rough idea of a prior (what you believe before you see data) and a little matrix algebra. We build the rest from zero.

Based on Lecture 13 of Princeton NEU 560: Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. Lecture slides. Every idea is re-taught here in our own words and devices, with neural examples. It is part of our NEU 560 follow-along and feeds the brain-computer interface series.

Set the penalty

Illustrative: a made-up smooth 100-weight filter, 100 noisy samples (the slide's example has the same size), white-noise inputs, noise s.d. 2, and a separate held-out set of 200 samples, all seeded. Scores are average log-likelihood per sample in nats. The evidence is computed exactly for the linear-Gaussian model.

Chapter 0

The Knob Nobody Sets for You

See why a model that fits your recording perfectly can still be the worst model you could pick

You record one neuron in motor cortex while a monkey reaches toward twelve targets arranged in a circle. On each trial you count the neuron’s spikes (the brief electrical pulses a neuron fires) and divide by the trial length. That gives a firing rate, in spikes per second.

Plot the rate against the direction of the target and a shape appears. The neuron fires most for one direction and least for the opposite one. Neuroscientists call that shape a tuning curve. You want to describe it with a smooth curve, so you can predict the rate for a direction you did not test.

A smooth curve is a model: a rule that turns an input (direction) into a prediction (rate). One simple family of rules is a polynomial. A polynomial of order 1 is a straight line. Order 2 allows one bend, order 3 allows two bends, and so on. Each extra order adds one parameter, a number the fit chooses to match the data.

More freedom always fits the data better

Here is the trap. Give the polynomial more orders and it can follow your twelve points more closely. At order 11 it passes through every point exactly, because twelve points fix twelve parameters. The error on your recording drops to zero.

But your recording was noisy. Each rate carries some random scatter that has nothing to do with direction. A curve that hits every point also hits every bit of scatter. Between the points it swings wildly. This failure has a name: overfitting, which means fitting the noise in your sample instead of the pattern behind it.

The cure is a test. Keep some data out of the fit, called held-out data, and score the curve on it. Scatter in the held-out data is new, so a curve that memorised the old scatter scores badly. Try it below.

Fit the tuning curve

Top: twelve training points (blue), sixty held-out points (purple), and the fitted curve (warm). Bottom: the error on each set for every order. Drag the order from 1 to 11.

11

Illustrative: a made-up cosine tuning curve (20 spikes per second plus 15 times a cosine), noise s.d. 4 spikes per second, seeded. The fit is ordinary least squares on a Legendre polynomial basis. Error is root-mean-square in spikes per second.

Training error falls at every step, all the way to zero at order 11. Held-out error falls at first, then turns and climbs. The best curve sits in the middle, at a low order. The model that fits best is not the model that predicts best.

The knob is not a parameter

The polynomial order is a hyperparameter: a setting that decides how flexible the model is, chosen before the fit. The fit sets the parameters. You set the hyperparameter. The fit cannot do it for you, because the fit always prefers more freedom.

Today’s lecture is about how to set such knobs from the data alone. Lecture 13 of Princeton’s NEU 560, taught by Jonathan Pillow, uses a different knob than polynomial order. It uses the strength of a prior, a statement of what weights you expect before you see data. A strong prior pulls the weights toward zero or toward a smooth shape. A weak prior lets them roam.

The lecture has three parts. First a review of fitting with a prior, called MAP inference. Then two ways to pick the prior’s strength: cross-validation, which scores held-out data, and evidence optimization, which asks how likely the data were under the prior. The last part joins evidence and MAP into one two-step method called Empirical Bayes.

A word on logistics. This is Lecture 13 of the Spring 2024 course. The slides and the written notes cover the same ground, so we follow the slides in order and add worked examples from neural data. The hero at the top of this page is the whole lecture in one picture, and each chapter explains one piece of it.

  1. MAP inference: fit with a prior, and read the prior as a penalty.
  2. Ridge and smoothing priors: two common penalties and what each believes.
  3. Overfitting in a 100-weight filter: maximum likelihood against two priors.
  4. Ridge is MAP: the exact link, and the meaning of the penalty strength.
  5. How to set the strength: why the training score cannot do it.
  6. Cross-validation: score held-out data across a grid of strengths.
  7. Folds and honest tests: n-fold cross-validation and the three data sets.
  8. Evidence: the marginal likelihood, and why it prefers simple models.
  9. Empirical Bayes: two steps, and why it needs no split.
  10. Field guide: numbers, a checklist, the code, and where to go next.
A flexible curve passes through all twelve of your training points, yet it predicts new trials badly. What went wrong?

Chapter 1

MAP Inference: Fit With a Prior

Add a belief about the weights to the data, and see how the best weight moves toward the belief

You want to know which past stimuli make a neuron fire. You show a screen of random flicker and record the neuron’s spike count in each 10 ms bin. Then you ask: how much does the spike count depend on the flicker 10 ms ago, 20 ms ago, up to 1,000 ms ago?

That is a hundred questions, one per time lag. The answers form a filter: a list of 100 weights, written w, where weight j says how strongly the flicker j bins ago pushes the spike count up or down.

The model and its data flow

The simplest model is linear-Gaussian. The expected spike count in a bin is the flicker history times the weights, and the real count scatters around it with a bell-shaped noise of fixed spread σ.

In shapes: the input X has N rows (one per time bin) and d columns (one per lag), here 100 by 100. The output Y holds N spike counts. The weights w hold d numbers. Row n of the model reads: yn = xn·w + noise.

The likelihood is the probability of the data you saw, as a function of the weights. Pick weights and the likelihood asks: how probable were these exact spike counts if those were the true weights? Weights that predict the counts well score high.

The maximum likelihood answer is the weights with the highest likelihood. It uses only the data. With 100 noisy bins and 100 weights, that is a recipe for the overfitting of Chapter 0.

Adding what you believed before

Suppose you also know something before any recording. Real filters are not wild. Their weights are usually small, and neighbouring lags usually have similar weights. A prior turns that belief into a probability over weights, p(w | θ). The symbol θ holds the prior’s settings, such as how wide it is.

Bayes’ rule combines the two. The probability of the weights after seeing data, the posterior, equals likelihood times prior, divided by a number that does not depend on the weights. So the best weights under the posterior need no division at all.

The maximum a posteriori estimate, MAP for short, is the weights with the highest posterior. Take the logarithm first, because the log turns the product into a sum and does not move the peak.

ŵmap
The MAP weights: the single list of 100 numbers that scores highest. The slide labels the whole expression the posterior.
log p(Y | X, w)
The log-likelihood. It rewards weights that predict the recorded spike counts. The slide marks it as the likelihood.
log p(w | θ)
The log-prior, also called the penalty. It rewards weights that match your belief, such as small or smooth. The slide calls this form “penalized log-likelihood estimation”.

Read it as a tug of war. The first term pulls the weights toward whatever fits the data. The second pulls them toward whatever the prior likes. The MAP estimate is where the two pulls balance.

One weight, worked by hand

Start with a single weight so you can see everything. Say four bins give a data-only best guess of 2.0 spikes per unit of flicker, with noise spread 2, and every flicker value is 1. Each bin carries 1/σ² = 1/4 of a unit of “precision” (sharpness) about the weight, so the four together carry n/σ² = 1.

The log-likelihood is then a downward parabola with its peak at 2.0 and curvature 1. Choose a prior that says “weights are near zero” with sharpness λ = 1. Its log is a downward parabola with its peak at 0. Add the two parabolas. The sum peaks where the sharpnesses balance: at (1 × 2.0 + 1 × 0) / (1 + 1) = 1.0. The prior pulled the answer halfway to zero. Now play with both knobs.

Watch the tug of war

Blue: how well each weight fits the data. Yellow: how well it fits the prior. Warm: their sum, whose peak is the MAP weight. Change the number of samples and the prior strength.

4

Illustrative: the data-only best guess is fixed at 2.0, noise s.d. 2, every flicker value 1. The log-likelihood is −(n/8)(w − 2)², the log-prior is −(λ/2)w², and the MAP weight is 2n/(n + 4λ). Each curve is shifted so its peak sits at zero.

With few samples the data are weak, the likelihood is a wide hill, and the prior wins: the MAP sits near zero. With many samples the likelihood is a narrow spike and the data win. A prior is a handicap that fades as evidence grows. That is exactly what you want from a belief.

For the real filter the same sum runs over 100 weights at once, but the picture is the same. The log-prior becomes a number you subtract for every weight that strays from the belief. That number is why people call it a penalty, and Chapter 2 writes down the two common penalties.

You double the number of recorded samples and keep the same prior. What happens to the MAP weight?

Chapter 2

Ridge and Smoothing Priors

Compare two penalties on a filter, and see which kind of filter each one expects

Picture the filter you hope to find: a 100-point curve that rises, dips and settles, like a neuron’s response to a flash. Now picture what bad data give you instead: a jagged saw-tooth, with each lag fitted on its own. A prior is how you tell the fit which of those two shapes you find believable.

The slides name two priors. Both are Gaussian, meaning bell-shaped. Each one gives weights a probability that falls off as the weights stray from what the prior expects. Each one has a single strength knob, written λ.

The ridge prior: small weights

The ridge prior says: every weight is probably near zero, and no lag is special. Each weight is drawn from a bell curve centred on zero. Its width is set by λ, which stands for the inverse variance, the width flipped upside down. A large λ means a narrow bell and a strong prior. A small λ means a broad bell and a weak prior.

log p(w | λ)
The log of the prior: the penalty, with its sign flipped. The slide writes it “penalty = −log p(w | λ)”.
λ
The ridge parameter: how hard each squared weight is punished. At λ = 0 nothing is punished.
wTw
The weight list times itself: the sum of every weight squared. The slide calls the result L2 shrinkage, because big weights are shrunk toward zero.

Take the four-weight filter [1, 3, 4, 2]. Its ridge sum is 1 + 9 + 16 + 4 = 30. At λ = 2 the penalty is ½ × 2 × 30 = 30. Notice what ridge ignores: it does not care whether the weights form a smooth curve. Shuffle them and the penalty stays the same.

The smoothing prior: neighbours agree

The smoothing prior says: neighbouring lags probably have similar weights. It punishes the step between each weight and the one before it. For the same filter, the steps are 3 − 1 = 2, 4 − 3 = 1 and 2 − 4 = −2. Their squares sum to 4 + 1 + 4 = 9, so the smoothing penalty is far smaller than the ridge penalty. The filter is smooth, and the prior approves.

The slides write this with a matrix called the graph Laplacian, L. Treat the lags as dots on a line, each linked to its neighbours. Then L has the number of neighbours on its diagonal and −1 where two dots are linked. For four lags:

# the graph Laplacian of four lags in a row
L = [[ 1, -1,  0,  0],
     [-1,  2, -1,  0],
     [ 0, -1,  2, -1],
     [ 0,  0, -1,  1]]
w = [1, 3, 4, 2]
print(w @ L @ w)   # 9 = (3-1)^2 + (4-3)^2 + (2-4)^2

The product wTLw equals the sum of squared steps. The end dots have one neighbour, so they carry a 1 on the diagonal. The middle dots have two, so they carry a 2. Check the arithmetic: L times w is [−2, 1, 3, −2], and its dot with w is −2 + 3 + 12 − 4 = 9.

One form for every Gaussian prior

Both penalties have the shape ½wT(something)w. The slides collect them into one rule. A Gaussian prior with covariance C (the matrix that records how much each weight varies and how weights vary together) has the penalty ½wTC−1w. The matrix in the middle is the inverse covariance.

Ridge is the case where C is the identity matrix divided by λ. Smoothing is the case where C−1 is λ times L. Every smoothness or sparsity wish you can express this way becomes a different matrix in the middle.

One technical wrinkle appears on slide 11. The Laplacian is not full rank: a constant filter, all ones, has no steps at all, so L maps it to zero. The smoothing penalty never punishes a filter for sitting high or low as a whole. In probability terms the prior has infinite variance along that one direction, and it is called improper. A small ridge term, ε times the identity, repairs it. The slides add that this is only about interpretation; a penalty with L alone works fine. Our code adds ε = 0.01 so the matrices can be inverted.

Draw filters from a prior

Each line is one filter the prior thinks is plausible. Pick a prior, then drag its strength. Press the button for five new draws.

Exact sampling: each filter is z divided by the square root of λ, mapped through the inverse Cholesky factor of the prior's penalty matrix (the identity for ridge, the Laplacian plus 0.01 times the identity for smoothing), with z standard normal. Seeded. The vertical scale is clipped at ±2.4.

Ridge draws look like static: every lag is its own small random number. Smoothing draws look like slow waves that wander, and they can drift away from zero as a whole, because that drift is free. Neither is right or wrong. Each is a belief. If your neuron’s true filter is a smooth wave, the smoothing prior holds the belief that matches.

Both priors still have λ to set. Too small and the prior says nothing. Too large and every filter collapses to flat. Chapters 5 to 9 answer how to choose it.

A filter has weights [2, 2, 2, 2]. Which statement is true?

Chapter 3

Overfitting a 100-Weight Filter

Run the lecture’s simulated example and see maximum likelihood fail, ridge half-fix it, and smoothing win

Slides 9 to 11 run one simulation. A neuron has a true filter of 100 weights. You record 100 noisy samples. You fit the filter three ways and compare each fit with the truth, which you know only because you built the simulation. This is the cleanest way to see what a prior buys.

We rebuilt the experiment with our own numbers. The true filter is a smooth wiggle: a central peak with side lobes, drawn in blue below. The inputs are white noise, so every input value is an independent bell-shaped random number. The noise on each spike count has a spread of 2. Everything is seeded, so you see the same data on every visit.

Failure one: maximum likelihood

With no prior, the best weights are the ones that predict the 100 training counts best. But you have 100 numbers to fit and only 100 samples. That is as many unknowns as equations, so the fit can match every count exactly, including the noise.

The symptom is a filter that looks like static. The true peak is buried in jagged spikes that are far larger than the signal. The measurement confirms it: on 200 fresh samples, the no-penalty fit scores −23.1 nats per sample, where a nat is the unit of log-probability. The fit with the true filter would score −2.11. The fix is a prior, which tells the fit that wild weights are unlikely.

Fix one: the ridge prior

The ridge prior says every weight is small. It shrinks all 100 weights toward zero together. Choosing λ well, the fit shrinks the noise spikes and the signal alike. The result is much better: the held-out score rises to −2.97. But the estimate is still rough. Ridge does not know that neighbouring lags should agree, so it leaves residual jitter on every lag.

Fix two: the smoothing prior

The smoothing prior says neighbours agree. Random noise is the opposite of neighbour-agreeing, so the prior suppresses it without flattening the real wiggle. With a well-chosen λ, the held-out score reaches −2.26, close to the ceiling of −2.11. The estimate follows the blue curve closely.

A prior that matches the true structure is worth far more than extra data would have been at a mismatched prior. Try it. Switch the sample count and watch which estimate improves most.

Compare three fits

Blue is the true filter. Warm is each estimate. Each panel uses the best penalty for that prior, found by scoring 200 fresh samples. Choose how many samples you record.

Illustrative simulation with made-up filter and noise (s.d. 2), seeded. Error is the sum of squared differences between estimate and true filter over the 100 lags. The no-penalty panel uses λ = 0.01, the smallest value on the grid.

More data help, but the story is not simply “more data, less need”. The no-penalty fit is at its worst at 100 samples, where samples equal weights and the input table is nearly impossible to invert cleanly. At 50 samples there are too few equations, so the tiny penalty picks the smallest-weight answer, which is poor but not wild. At 200 and 400 samples the data finally pin the weights down, and the no-penalty fit becomes usable. Smoothing still wins at every size, and the gap shrinks as data grow.

Reading the scores

The numbers in this chapter are average log-likelihoods per sample. For Gaussian noise with spread σ, each sample scores −½ ln(2πσ²) minus its squared error divided by 2σ². The first part is fixed. The second part is what the fit controls.

Here is the ceiling worked out. With σ = 2 the fixed part is −½ ln(2π × 4) = −1.61. The true filter leaves an average squared error equal to the noise variance, so the second part is −½. The total is −2.11. No fit can beat it on average. A score of −2.26 is therefore close to perfect, and −23.1 means the average squared error is about 43 times the noise variance.

What this chapter leaves open

Each panel used the “best” λ, and we found it by scoring 200 fresh samples. In a real experiment you do not have the truth, and you rarely have 200 spare samples. You have one recording and one question: what λ? That is the lecture’s real subject, and Chapters 5 to 9 answer it. First, Chapter 4 pins down exactly what λ is.

With 100 samples and 100 weights, why does maximum likelihood fail so badly?

Chapter 4

Ridge Regression Is MAP

Derive the exact link between a penalty and a prior, and read the penalty strength as a ratio of two variances

Two communities solve the same problem with different words. A statistician fits the filter with ridge regression: minimise the squared prediction error plus a penalty on the size of the weights. A Bayesian fits it with a Gaussian prior and takes the MAP estimate. The lecture’s in-class exercise asks: what is the exact relationship?

The answer is that they are the same estimate. Working it out also tells you what the penalty strength means, and that meaning is what makes it possible to choose it from data.

The two setups

Ridge regression chooses the weights that minimise the loss ||Y − Xw||² + λwTw. The first term is the squared prediction error over all N samples. The second is the penalty. The known solution is (XTX + λI)−1XTY. The shapes: XTX is d by d, XTY has d entries, and the answer has d weights.

The Bayesian setup has two pieces. The prior says each weight is bell-shaped around zero with variance θ, written w ~ N(0, θI). The observation model says each spike count is bell-shaped around the prediction with noise variance σ², written Y | X, w ~ N(Xw, σ²I). Note that θ is now a variance, not an inverse variance as λ was in Chapter 2. The slides warn about this switch.

The derivation, one line at a time

Start from the MAP rule of Chapter 1: maximise log-likelihood plus log-prior. Write down each term and drop constants, since constants do not move the peak.

  1. Log-prior. A Gaussian’s log is a downward parabola: log p(w) = −(1/2θ) wTw + const.
  2. Log-likelihood. Same shape on the prediction error: log p(Y | X, w) = −(1/2σ²) (Y − Xw)T(Y − Xw) + const.
  3. Add them. The log-posterior is −(1/2σ²)(Y − Xw)T(Y − Xw) − (1/2θ) wTw + const.
  4. Expand the square. (Y − Xw)T(Y − Xw) = YTY − 2wTXTY + wTXTXw. The YTY part has no w, so it joins the constant.
  5. Differentiate and set to zero. The slope with respect to w is −(1/2σ²)(−2XTY + 2XTXw) − (1/θ) w = 0.
  6. Tidy up. Multiply by −σ²: −XTY + XTXw + (σ²/θ) w = 0, so (XTX + (σ²/θ) I) w = XTY.
  7. Solve. ŵmap = (XTX + (σ²/θ) I)−1 XTY.

Compare the last line with the ridge solution. They match when λ = σ²/θ. The squared-error loss is a Gaussian likelihood in disguise, and the L2 penalty is a Gaussian prior in disguise.

λ
The ridge parameter in the loss, the number that sets how hard the penalty pulls.
σ²
The noise variance: how much a spike count scatters around its prediction. A noisier neuron means weaker trust in the data.
θ
The prior variance: how far from zero you expect the weights to wander. A broad prior means weaker trust in the prior.

Check it on the one-weight case from Chapter 1. Four bins, every input 1, data-only best guess 2.0: XTX = 4 and XTY = 8. Take σ² = 4 and θ = 1, so λ = 4. Then ŵmap = 8 / (4 + 4) = 1.0, the same 1.0 we found by adding parabolas.

What the ratio means

Because λ is a ratio, two different stories give the same fit. A noisy neuron with a broad prior and a quiet neuron with a tight prior can share one λ. Only the ratio matters to the estimate. The lecture’s conclusions follow from the ratio alone.

Between the extremes, a larger λ shrinks every weight further. Try the knobs yourself. The device finds the MAP weights two ways: with the closed-form ridge formula, and by climbing the log-posterior step by step without ever writing λ.

Two roads to one answer

Blue: the data-only (maximum likelihood) weights. Warm bars: ridge with λ = σ²/θ. Teal dots: the posterior peak found by gradient ascent. Drag the two variances.

Illustrative: six made-up samples of three inputs, true weights 1.5, −1 and 0.5, noise s.d. 2, seeded. Gradient ascent runs 4,000 steps on the log-posterior from zero, using σ² and θ directly. Variances run from 0.1 to 100 on a log scale.

Drag θ up and the warm bars grow toward the blue ones. Drag σ² up and they shrink away. Both moves change only the ratio. The teal dots sit on the warm bars throughout, to the limit of the climber’s accuracy.

This also explains why there is a knob to set at all. The noise variance σ² can be measured from residuals. The prior variance θ is a belief with no direct measurement. Setting θ, or equivalently λ, is the model selection problem.

A neuron is twice as noisy (noise variance doubles). To keep the same MAP estimate, what must you do to the prior variance θ?

Chapter 5

How Strong Should the Prior Be?

Ask the lecture’s central question, and see why the training score can never answer it

You now have a fitting machine with one dial. Feed it a recording and a value of λ, and it returns a filter. Turn λ down and the filter chases noise. Turn it up and the filter goes flat. Somewhere between lies a setting that finds the real filter. The lecture’s slide 17 asks the question in one line: how do we set the hyperparameters, or the prior parameters, λ?

The word hyperparameter deserves care. A parameter is something the fit chooses, such as the 100 weights. A hyperparameter is something that shapes the fit, such as the prior’s strength λ. Think of baking. The oven temperature is a hyperparameter. The bread’s final texture is the outcome. You set the temperature first and judge the bread afterwards.

The obvious idea, and why it fails

The obvious idea is to treat λ like a weight. Try every λ, fit the filter, and keep the λ whose fit matches the training data best. This does not work, and the reason is simple.

Without a penalty, the fit has total freedom, so it matches the training counts as closely as the model allows. Adding a penalty restricts the fit. A restricted fit can never match the training data better than an unrestricted one. So the training score falls whenever λ rises. The training score always votes for λ = 0, the weakest prior, and you are back at the overfit filter of Chapter 3.

Check it with the one weight from Chapter 1: four bins, data-only best guess 2.0, noise spread 2. The MAP weight is 2n/(n + 4λ) = 8/(4 + 4λ). The training misfit is −(n/8)(w − 2)² = −0.5(w − 2)². At λ = 0 the weight is 2.0 and the misfit is 0. At λ = 1 the weight is 1.0 and the misfit is −0.5. At λ = 4 the weight is 0.4 and the misfit is −0.5 × 1.6² = −1.28. The training score drops at every step, so it votes for λ = 0 each time. Nothing in it can notice that the shrunk weight might predict the next four bins better.

That last sentence is the whole problem. Suppose the true weight is 1.0 and the 2.0 came from noise. Then the weight 1.0 predicts fresh bins better than 2.0, yet the training score ranks it lower. A score that has seen the answer cannot tell memory from understanding. The fix must come from data the weights never touched.

Let a score be a number where higher means better. Any score computed on the same data the weights were fitted to is biased toward freedom. It rewards the fit for memorising. You need a score that the fit could not have memorised.

One more attempt deserves a mention, because it looks clever. Why not maximise the joint probability of the weights and the prior width together? That also fails. A Gaussian is tallest at its centre, and a narrower Gaussian is taller still. Shrink the prior toward zero width, put all weights at zero, and the density grows without limit. The prior collapses onto a spike. Nothing in the joint score pushes back.

Two ways out

The lecture offers two honest scores.

  1. Cross-validation. Hide part of the data. Fit on the rest. Score the fit on the hidden part. The fit never saw the hidden data, so it cannot have memorised them.
  2. Evidence optimization, also called maximum marginal likelihood. Score each λ by how probable the whole recording was, averaged over every filter the prior allows. No data are hidden, and the average punishes priors that are too loose.

Chapters 6 and 7 develop the first. Chapters 8 and 9 develop the second. They sound different, but on the lecture’s examples they land close together, and you will see both on the same curve.

A word on reading the scores. A log-likelihood of −2.5 nats per sample means the model gave the average held-out count a probability density of about e−2.5 = 0.08. Higher is better, and the units let you compare any two settings of λ on the same data.

Here is the failure and the fix side by side. The device picks λ by three rules on the same 100 training samples from Chapter 3, then judges each choice on 200 fresh samples. The rule “training score” is the obvious idea. The other two are the lecture’s answers. Cross-validation here is 5-fold on the 100 training samples, which Chapter 7 explains.

Three rules, one recording

Blue is the true filter, warm is the estimate each rule gives. Switch the prior and compare how the three rules behave.

Illustrative: the same seeded 100-sample simulation as Chapter 3 (noise s.d. 2). The held-out score uses 200 separate fresh samples. Cross-validation is 5 folds of 20 samples. The evidence is exact for this linear-Gaussian model. The λ grid has 41 values from 0.01 to 10,000.

The training-score rule picks the smallest λ on the grid every time, and its filter is the saw-tooth. The other two rules pick values near the held-out optimum. One rule used no hidden data at all, and it did as well. That is the surprise of evidence optimization, and Chapter 8 explains it.

Why does choosing λ to maximise the training log-likelihood always pick the weakest prior?

Chapter 6

Cross-Validation: Score What You Hid

Sweep a grid of penalty strengths, score each on hidden data, and read the best one off an inverted U

Imagine a teacher who wants to know whether a student understands a topic or has memorised last year’s exam. She sets a new exam with new questions. A student who memorised fails. A student who understood passes. Cross-validation is that new exam for a fitted model.

The lecture calls cross-validation the simplest and arguably the most popular approach to setting the regularisation strength. Slides 18 and 19 give the recipe. We take it in order and then run it.

The recipe

  1. Split. Divide the data into a training set (Xtrain, Ytrain) and a test set (Xtest, Ytest). The slide draws it as a table with a block of rows cut off the bottom.
  2. Make a grid. List candidate strengths {λ1, λ2, …, λm}, typically spaced evenly on a log scale.
  3. Fit on the training set. For each λi, compute the MAP weights using only the training rows: ŵi = argmax log P(w | Xtrain, Ytrain, λi).
  4. Score on the test set. Compute the log-likelihood of the hidden counts under those weights: li = log P(Ytest | Xtest, ŵi).
  5. Pick. Keep the λi with the highest test log-likelihood.
  6. Refit. Slide 19’s bullet: the final estimate is the MAP with λopt, fitted again, now on all the data.

Why a log-spaced grid? Chapter 4 showed that λ is a ratio of variances. Ratios work in factors, not steps. Moving from 1 to 2 is a big change. Moving from 1,001 to 1,002 is nothing. Equal steps in the logarithm give equal factors.

The test log-likelihood is the number to read: how probable the hidden spike counts were, according to the fitted model. For Gaussian noise it is the average squared error in disguise: a smaller error on hidden data means a higher log-likelihood. We report it per sample, in nats.

The shape it draws

The slide’s hand-drawn picture has two curves against log λ. The training log-likelihood falls steadily as λ rises, as Chapter 5 proved. The test log-likelihood rises, peaks, then falls: an inverted U.

Left of the peak is overfitting. The prior is too wide or the penalty too weak, so the fit chases features of the training set that do not help on the test set. Right of the peak is underfitting. The prior is too narrow or the penalty too strong, so the weights are too close to zero or too smooth to capture the real structure. The peak is the sweet spot.

Run it yourself. The device takes the same 100 samples as before, hides 30 of them, and walks the grid one step at a time.

Sweep the grid

The bar shows which of the 100 samples train the fit (blue) and which stay hidden (purple). Press Step to try the next λ, or Run all.

Illustrative: the Chapter 3 simulation, split at random into 70 training and 30 hidden samples (seeded). Hidden-data scores from only 30 samples are noisy, and they look rosier than fresh-data scores at the chosen λ, a gap Chapter 7 explains. The fresh-data score uses 200 separate samples.

Watch the order of events. The training curve (blue) slides down from the first step. The hidden-data curve (purple) starts low, climbs as the penalty tames the noise, and turns over when the penalty starts to flatten real structure. After the sweep, the best λ is marked and the fit is repeated on all 100 samples. Compare the two scores in the read-out. The hidden-data score at the peak looks better than the fresh-data score, because choosing λ on 30 samples tuned it to their quirks. The honest number is the fresh one.

One engineering note keeps the sweep cheap. The matrices XTX and XTY depend on the training rows but not on λ. Compute them once, then for each λ add λ times the penalty matrix to XTX and solve. With 100 weights, that is 41 solves of a 100-by-100 system, and the whole sweep takes a fraction of a second.

The code, and a failure to watch for

import numpy as np
lams = np.logspace(-2, 4, 41)               # the grid
scores = []
for lam in lams:
    w = np.linalg.solve(Xtr.T @ Xtr + lam*np.eye(d), Xtr.T @ ytr)   # MAP on training rows
    r = yte - Xte @ w                              # error on hidden rows, shape (n_test,)
    scores.append(-0.5*np.log(2*np.pi*s2) - r @ r / (2*s2*len(yte)))
lam_opt = lams[np.argmax(scores)]
w_final = np.linalg.solve(X.T @ X + lam_opt*np.eye(d), X.T @ y)   # refit on all data

The failure to watch for has a clear symptom: the best λ sits at the very edge of your grid. That means the true peak is outside the grid. The measurement is to look at where the argmax falls. The fix is to extend the grid in that direction by a few decades and run again.

A second, subtler failure: the split itself is a coin flip. A different random split gives a slightly different curve and a different peak. With only 30 hidden samples, the peak is noisy. Chapter 7 shows how to average the noise away.

On the grid sweep, the training log-likelihood falls steadily, but the held-out log-likelihood rises and then falls. What does the falling right-hand side of the held-out curve mean?

Chapter 7

Folds, Splits and an Honest Test

Average over many splits to steady the pick, and keep a third set untouched so the final number is true

Chapter 6 ended with a worry. The single split was a coin flip, and 30 hidden samples gave a noisy curve. A neuroscientist with 60 trials from one session cannot simply collect more. She must squeeze the most out of what she has. The lecture notes list three refinements that do exactly that.

Refinement one: n-fold cross-validation

Instead of one train and test split, cut the data into n equal blocks called folds. Then run n rounds. In round k, hold out fold k as the test set and fit on the other n − 1 folds. Score the held-out fold. After n rounds, every sample has been tested exactly once.

Average the n test log-likelihood curves, point by point along the λ grid. Pick the λ at the top of the averaged curve. The notes say it in one sentence: average the test log-likelihood across folds to find the best optimum for selecting λ.

A worked count shows the cost. Take 60 samples and 5 folds. Each fold holds 12 samples, so each round fits on 48 and tests on 12. Over the 41-point grid that is 5 × 41 = 205 fits instead of 41. More folds cost more fits. The extreme, one sample per fold, is called leave-one-out. It uses nearly all the data every round but needs one fit per sample per λ.

Why does averaging help? Each round’s curve carries its own luck. Luck that points up in one fold points down in another, and the average cancels part of it. Every sample also gets a vote, not just a lucky 20%.

Refinement two: three sets, not two

There is a catch. Choosing λ by the best test score means the test data helped choose. The winning score is then a little too good, the way a student who picks the exam questions that suit her gets a flattering mark. Your report would claim better performance than a new recording will deliver.

The notes give the cure: three sets. The training set fits the weights for each λ. The validation set selects λ. The test set is locked away and scored once, at the end, to report true performance. The notes say this approach scrupulously avoids overfitting because the test set is not used for fitting hyperparameters. With cross-validation, the validation role is played by the held-out folds.

Refinement three: error instead of likelihood

With Gaussian noise, the test log-likelihood is a constant minus the mean squared error (MSE) divided by 2σ². So maximising the log-likelihood is the same as minimising MSE. Most papers report MSE. The best λ is identical either way, only the curve is flipped upside down.

The device below measures the first refinement directly. It uses a smaller problem so that it can run many splits in an instant: a 40-weight filter and 60 samples. Press the button to draw 20 fresh random splits. Each dot is the λ that one split picked.

Repeat the selection 20 times

Each dot is one repeat of the whole selection on the same 60 samples with a different random split. A tight cluster means a steady pick. Choose a scheme, then press the button for new splits.

Illustrative: a made-up 40-weight smooth filter, 60 training samples, noise s.d. 2, ridge prior, seeded. The single split hides 20% of the samples (12). The marked best value is the best on 400 separate fresh samples. Spread is the standard deviation of log10 λ across the 20 picks.

A single split scatters across a wide range. Five folds and ten folds pull the dots together. Now compare the two scores in the read-out. Selecting on held-out data pushes the winning score up, because the data helped choose. Fitting on only part of the data pulls it down, because each fit sees fewer samples. The two effects pull in opposite directions, so the winning cross-validation score is not a number to report. A separate test set is.

A failure that is easy to miss

Neural data have a trap. Spike counts in neighbouring time bins are not independent: a neuron that is bursting now was probably bursting a moment ago. If you shuffle bins at random into folds, a test bin has near-twin neighbours in the training set. The fit looks excellent on the test fold without having learned anything general.

The symptom is a cross-validation score that is much better than the score on a new recording session. The measurement is to compare the two. The fix is to cut folds as long contiguous blocks of time, so test bins have no close neighbours in training.

def kfold_scores(X, y, lams, k=5, s2=4.0):
    n, d = X.shape
    blocks = np.array_split(np.arange(n), k)         # contiguous blocks, not a shuffle
    scores = np.zeros(len(lams))
    for held in blocks:
        tr = np.setdiff1d(np.arange(n), held)
        G, b = X[tr].T @ X[tr], X[tr].T @ y[tr]
        for i, lam in enumerate(lams):
            w = np.linalg.solve(G + lam*np.eye(d), b)
            r = y[held] - X[held] @ w
            scores[i] += (-0.5*np.log(2*np.pi*s2) - r @ r/(2*s2*len(held))) * len(held)/n
    return scores                                   # average held-out score per lambda
You pick λ by the best cross-validation score and then report that same score as your model’s performance. What is wrong?

Chapter 8

Evidence: Score the Prior by How Well It Predicted

Average the likelihood over every filter the prior allows, and see why that average prefers a prior that is neither too loose nor too tight

Two forecasters predict tomorrow’s spike count. Forecaster A says “somewhere between 0 and 1,000.” Forecaster B says “between 8 and 12.” Tomorrow brings 10. Both were right. But B was bolder, and B put far more probability on what happened. B deserves more credit.

That is the idea behind the second way to set λ. Treat the prior as a forecaster. Ask how much probability the prior put on the data that actually came in. Slide 20 calls this number the evidence, or the marginal likelihood.

Where the evidence comes from

You have met it before, in Chapter 1, as a quantity you were allowed to ignore. Bayes’ rule has a denominator: the posterior equals likelihood times prior divided by P(Y | X, λ). It did not depend on the weights, so MAP dropped it. Now we keep it, because it does depend on λ.

P(Y | X, λ)
The evidence: how probable the recorded spike counts were, with the weights averaged out. It depends on λ but on no particular weights.
P(Y | X, w)
The likelihood of the data for one choice of weights, as in Chapter 1.
P(w | λ)
The prior: how plausible each choice of weights is before seeing data. The integral adds up likelihood over all weights, each counted by its prior plausibility.

The slide gives it two readings. As a function of λ, P(Y | X, λ) is a likelihood, so we can maximise it. It is also a marginal of the joint P(Y, w | X, λ): integrate the joint over w, and w disappears. The lecture sums it up: set hyperparameters using maximum marginal likelihood.

One honest warning from the slide: in general this requires numerical optimisation. For our linear-Gaussian model, the value of the evidence has a neat closed form: the counts Y are bell-shaped with covariance σ²(I + XA−1XT/λ). But the λ that maximises it has no formula, so we search. The hero already did, on a 41-point grid.

Why it is an average

Read the integral as an average. Draw many weight lists from the prior. For each, ask how probable the data are. Average the answers. That average is the evidence. This connects to Monte Carlo estimation, where an average over random draws stands in for an integral, and to importance sampling, which draws from a smarter distribution when most draws are wasted.

Now the reason the evidence picks a sensible λ. A prior that is too wide sends most draws far from any weights that fit. Almost every draw gives tiny likelihood, so the average is tiny. A prior that is too narrow sends every draw near zero, where the likelihood is also small, because the true weight is not zero. Only a prior of about the right width sends a decent share of draws where the likelihood is high.

Another view says the same thing. The evidence is a probability over datasets, so it must sum to 1 over every dataset the model could produce. A very flexible prior can produce almost anything, so it spreads its probability thin and gives any one dataset a small share. A simpler prior concentrates its probability. If the data fall where it concentrated, it wins. This built-in penalty on over-flexibility is sometimes called Occam’s razor.

The device uses one weight and five samples so you can see every step. Slide the prior width and watch the draws.

Average the likelihood over the prior

Top: the likelihood of the data for each weight (blue), the prior (yellow), and 2,000 prior draws (teal ticks, height = likelihood). Bottom: the exact log evidence for every prior width (pink), with the average of the draws as a teal dot. Drag the prior variance θ.

Illustrative: five made-up samples of one weight (true value 1.5), noise s.d. 1, seeded. The exact evidence uses the closed form for a Gaussian prior and Gaussian noise. The estimate averages the likelihood over 2,000 draws from the prior. θ runs from 0.01 to 100 on a log scale; here λ = σ²/θ = 1/θ.

At a tiny θ every draw sits at zero and the likelihood there is low. At a huge θ the draws scatter across a wide range and only a sliver lands near the peak. The evidence curve peaks in between, where the prior’s width roughly matches the size of the weight the data support. The teal dot follows the exact curve, within the noise of a 2,000-draw average.

How it compares with cross-validation

The lecture notes observe that the marginal likelihood typically has an upside-down-U shape in λ, with a maximum close to the maximum of the test log-likelihood curve in many cases. Our hero showed it: with the ridge prior both peaks sit at λ = 40, and with the smoothing prior at 158 for the evidence against 224 for the held-out score. The curves are on a common scale in nats per sample, and the evidence is lower throughout, because it scores each sample using only what the prior and the earlier samples could teach, before the data have taught the model much.

The advantage is that no data are hidden. Every sample takes part in the score. The cost is that the evidence trusts the model. If the noise variance is wrong, or the prior has the wrong shape, the evidence can mislead, while cross-validation only checks predictions. The symptom of trouble is an evidence peak far from the cross-validation peak. The measurement is to plot both curves. The fix is to revisit the model: estimate σ² too, or try another prior.

Why does the evidence fall when the prior is made very wide?

Chapter 9

Empirical Bayes: Fit the Prior, Then Fit the Weights

Join evidence and MAP into one two-step method, and see why it scales to many penalty knobs where a grid cannot

A tailor measures a customer before cutting cloth. She does not cut first and measure after. The measuring step sets the pattern, and the cutting step uses it. Empirical Bayes works the same way for a fitted model: first set the prior from the data, then fit the weights under that prior.

Slide 21 gives the method in two lines. It is the lecture’s closing idea, and it uses every chapter before it.

The two steps

  1. Estimate λ by maximising the evidence. λ̂ = argmaxλ P(Y | X, λ). The slide adds, in brackets, that this fits the hyperparameters governing the prior.
  2. Find the MAP weights given λ̂. ŵmap = argmaxw P(w | Y, X, λ̂). The slide’s key note: previously we used cross-validation to set λ. Now the evidence sets it.

Step 1 is Chapter 8: score each λ by how well the prior predicted the whole recording. Step 2 is Chapter 1: a MAP fit, with λ̂ as a fixed setting. Nothing new is needed beyond gluing the two together.

The word empirical says that the prior is estimated from the data themselves, not chosen before looking. A purist Bayesian would object that the data are used twice, once to set the prior and once to fit the weights, so the final answer can be a little too confident. In practice, with one or a few hyperparameters and a lot of data per hyperparameter, the effect is small. The method is also known as type II maximum likelihood.

The slide lists two advantages. First, it does not require partitioning the data: all samples take part in both steps. Second, it is more practical with multiple λ. The second one is worth seeing, so we build a case.

Many knobs, one search

Suppose you record a neuron and use 40 lags, but you suspect that only the second half carries signal. The first 20 lags may be pure noise. Give each half its own λ. Then the model has two hyperparameters, λ1 for the first 20 weights and λ2 for the last 20. The right answer is a huge λ1, which switches the dead half off, and a moderate λ2.

Count the work. Cross-validation needs a grid. With 12 values per knob and 5 folds, two knobs mean 12 × 12 × 5 = 720 fits. Three knobs would mean 8,640 fits. The cost multiplies with every new knob.

Evidence is a smooth function of the knobs, so a numerical optimiser can climb it. The optimiser tries a value, looks at the evidence, and moves. It needs tens of evaluations, not hundreds, and the count grows slowly with the number of knobs. Each evaluation is a single score on all the data, with no folds.

In the device, the one-knob mode compares the two methods head to head. The two-knob mode shows the cross-validation grid as a map, with the evidence optimiser’s path laid over it. The optimiser here is coordinate search: it tunes one knob at a time with a golden-section line search, and cycles until it stops moving.

Grid against climb

Choose one knob for all lags, or two knobs, one per half of the filter. Purple is cross-validation, pink is the evidence.

Illustrative: a made-up 40-weight filter whose first 20 weights are zero, 60 samples, noise s.d. 2, ridge priors, seeded. Cross-validation is 5-fold. Grids: 41 values for one knob, 12 by 12 for two, from 0.01 to 10,000. Fresh-data scores use 400 separate samples after refitting on all 60.

In the one-knob mode the two methods agree closely, and the evidence method needed far fewer evaluations than the grid needed fits. In the two-knob mode the map shows a ridge of high cross-validation scores toward a large λ1. The pink path walks to the same region in a handful of steps. The counts in the read-out are the point: the gap widens with every extra knob.

The line search deserves one plain explanation. Golden-section search brackets the peak. It keeps two interior trial points, compares their evidence, and drops the outer part that cannot hold the peak. Each step shrinks the bracket to 62% of its length. Starting from a bracket six decades wide (from 0.01 to 10,000), ten steps leave a bracket of 6 × 0.61810 = 0.05 decades. Reuse of one trial point per step keeps the cost to one new evaluation each time.

Where else the idea appears

Fitting a prior’s settings by maximising a marginal likelihood is a general move. The EM algorithm of Gaussian mixtures is one way to climb such a score when some quantity is hidden. The training objective of a variational autoencoder is a lower bound on a marginal likelihood of this kind. The noise settings of a Kalman filter can be fitted the same way. Different models, the same two-step shape.

A model has four penalty knobs. Why does Empirical Bayes scale better than cross-validation on a grid?

Chapter 10

Field Guide: Setting the Knob

Keep the numbers, the terms, a checklist and the core code, then find where to go next

A fitted model has parameters that the data set and hyperparameters that you set. The prior’s strength is the most common hyperparameter in neural data analysis. MAP inference adds a prior as a penalty. Ridge regression is MAP with a Gaussian prior, and its penalty strength is λ = σ²/θ. The training score cannot choose λ, because it always prefers the weakest prior. Cross-validation scores hidden data on a grid. Evidence optimisation scores the whole recording by averaging the likelihood over the prior. Empirical Bayes sets λ by evidence and then fits the weights by MAP.

The numbers

QuantityValueWhere
Hero simulation100 weights, 100 samples, noise s.d. 2, 200 fresh samplesHero, Chapters 3, 5, 6 (illustrative)
No penalty, fresh score−23.1 nats per sampleChapter 3
Best ridge, fresh score−2.97 at λ = 40; evidence also peaks at 40Hero, Chapter 3
Best smoothing, fresh score−2.26 at λ = 224; evidence peaks at 158Hero, Chapter 3
Ceiling with the true filter−2.11 nats per sampleChapter 3
Ridge and MAPλ = σ²/θ; λ → 0 gives maximum likelihood, λ → ∞ gives zero weightsChapter 4, slide 15
Cost of 5-fold on a 41-point grid205 fitsChapters 7 and 9
Cost of two knobs, 12 by 12 grid, 5 folds720 fits, against tens of evidence evaluationsChapter 9

The terms

TermPlain meaningChapter
HyperparameterA setting that shapes the fit, chosen before fitting0, 5
Prior and penaltyA belief about the weights, and its log with the sign flipped1, 2
MAPThe weights with the highest posterior: log-likelihood plus log-prior1
Graph LaplacianThe matrix whose quadratic form sums the squared steps between neighbours2
Overfitting, underfittingFitting noise, or being too stiff to fit the pattern0, 6
FoldOne block of data held out in one round of cross-validation7
Validation and test setOne chooses λ; the other is scored once, at the end7
EvidenceThe likelihood averaged over the prior: P(Y | X, λ)8
Empirical BayesStep 1, set λ by evidence. Step 2, MAP weights given that λ9

A checklist for your next fit

  1. Name the knob. Write down every hyperparameter and its range. Use a log grid.
  2. Split with care. Cut neural data in contiguous time blocks. Lock a test set away first.
  3. Sweep and look. Plot the training and held-out curves. Check that the best value is not at the grid edge.
  4. Cross-check. Compute the evidence curve too. If the peaks differ widely, suspect the noise variance or the prior shape.
  5. Refit and report. Fit the weights on all non-test data with the chosen λ. Score the test set once.

The core as code

import numpy as np

def log_evidence(X, y, A, lam, s2):
    # y ~ N(0, s2 * (I + X A^-1 X^T / lam)) once the weights are integrated out
    n = len(y)
    K = X @ np.linalg.solve(A, X.T)                 # (n, n) kernel; A is (d, d)
    C = s2 * (np.eye(n) + K / lam)
    _, logdet = np.linalg.slogdet(C)
    return -0.5 * (n * np.log(2 * np.pi) + logdet + y @ np.linalg.solve(C, y))

lams = np.logspace(-2, 4, 41)
# step 1: set lambda by maximum marginal likelihood
lam_hat = lams[np.argmax([log_evidence(X, y, A, l, s2) for l in lams])]
# step 2: MAP weights given lambda-hat (ridge regression when A = I)
w_hat = np.linalg.solve(X.T @ X + lam_hat * A, X.T @ y)   # shape (d,)

Where it sits, and where to go

Our GLM lesson showed how penalties enter models of spike counts, and the same two rules choose their strengths. The regularisation lesson and the bias and variance lesson give the general view of why a penalty helps. Bayes estimation and probability supply the prerequisites. For the Monte Carlo reading of the evidence, see Monte Carlo and importance sampling. For latent-variable fits of the same kind, see EM for Gaussian mixtures and PCA.

In the brain-computer interface series, regularised decoders raise the same question. See linear decoders and latent variable models.

The takeaway. A penalty strength cannot be fitted on the data it penalises. Score it on data the fit did not see, or on the probability the prior gave to all the data, and the two answers will usually agree.

Credit: this lesson re-teaches Lecture 13 of NEU 560, Statistical Modeling and Analysis of Neural Data, by Jonathan Pillow at Princeton (Spring 2024). See the course site and the lecture slides. Test yourself with Teach, or go back to the penalty dial at the top and find the two peaks again.

Which sequence describes Empirical Bayes?
NEU 560 Lecture 13: Model Selection
The course site · Lecture slides · Back to Gleams