MAP Inference

A neuron’s filter hides inside noisy data, and plain least squares returns a scribble. Add a prior, a belief about the filter’s shape, and the same data give a clean answer.

Pick a prior, then slide λ, the strength of the belief. Start at maximum likelihood to see the scribble, then try ridge and smoothing. Then we build, piece by piece, why each one works and what it secretly assumes.

You need a little linear algebra (a matrix times a vector) and the idea of a probability. We build the rest from zero.

Based on Lecture 12, Regularization and MAP inference, of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. Lecture slides and lecture notes. Every idea is re-taught here in our own words and devices. It follows Lecture 11, Regularization and feeds our brain-computer-interface series, starting with Linear decoders.

Recover the filter

10

Illustrative neuron: a 100-bin filter of our own design, 100 samples of smooth random stimulus and Gaussian noise of standard deviation 1. The sizes match the lecture’s simulated example (100 weights, 100 samples) but the data are ours. Maximum likelihood is drawn as ridge with λ = 0.000001, the closest the computer gets without dividing by zero. Smoothing penalises the squared difference between neighbouring weights.

Chapter 0

Why a Neuron’s Filter Needs a Prior

See how a neuron’s recorded spikes turn into a list of weights, and why the plain answer to “which weights?” comes back as noise

Picture one neuron in the visual cortex of a mouse, recorded while a screen flickers. Every 10 milliseconds the screen picks a new random brightness. In the same 10 milliseconds you count the spikes the neuron fires.

The brightness values are the stimulus. Each 10 millisecond slice of time is a bin, and the spike count in a bin is the response. You want to know what the neuron cares about. It does not react to the brightness right now. It reacts to the last second of brightness, and some moments in that second matter more than others.

A filter is a list of numbers that captures this. It holds one weight for each recent bin: a weight for the brightness one bin ago, two bins ago, and so on. A large positive weight means a flash at that delay makes the neuron fire. A negative weight means a flash at that delay silences it. Our toy filter has 100 weights, one for each bin of the last second.

The model is a weighted sum. The response in one bin is the stimulus in each of the last 100 bins, multiplied by its weight, added up, plus noise: the random part that the stimulus does not explain. Step through it below. Pick a bin and watch one response being built.

Build one response

Top: the stimulus in the 100 bins before your chosen bin. Middle: the filter. Bottom: each bin’s stimulus times its weight. The sum of the bottom bars is the neuron’s expected response. Drag the bin.

57

Illustrative neuron, not the lecture’s data: the filter is our own bump, the stimulus is random brightness blurred over a few bins, the noise has standard deviation 1, and responses are in arbitrary units. Every device on this page uses this same neuron and these same 100 samples.

Look at the read-out. One bin gives one equation with 100 unknown weights. A recording gives you many bins, so you get many equations at once. Stack them. Row t of a table X holds the stimulus values that the response in bin t could see. Then all the equations fit in one line.

y
The measured responses, one number per bin. Here 100 numbers.
X
The design matrix: one row per bin of response, one column per delay. Row t is the stimulus the neuron could see. Here 100 × 100.
w
The filter: one weight per delay. This is the unknown we want. Here 100 numbers.
noise
Random error added to every response. We assume it is Gaussian, a bell curve around zero, the same size in every bin.

The plain answer, and why it fails

The obvious estimate is the least squares filter: the weights whose predictions Xw land closest to the measured y. When the noise is Gaussian, this is also the maximum likelihood estimate (MLE), the weights that make the measured data most probable. Our least-squares lesson derives it.

Here is the symptom. Press Maximum likelihood in the instrument at the top of the page. The estimate is not a smooth bump. It is a wild scribble that swings far past the edges of the chart. The true filter is a quiet dotted line underneath.

Here is the measurement. The scribble’s error energy is about 64,000 times the true filter’s own energy. The estimate fits the 100 noisy samples almost perfectly and says nothing true about the neuron. The lecture calls this overfitting: a model with enough freedom to memorise the noise.

The cause is arithmetic. We ask 100 samples to pin down 100 weights, and the stimulus is smooth, so neighbouring columns of X carry nearly the same information. Chapter 1 shows exactly how that breaks the solution.

Here is the fix, and the thread of this whole lesson. We tell the computer something we already believe before seeing the data: a real filter is not a scribble. Weights are modest, and neighbours are similar. That belief is a prior. Combining it with the data gives the MAP estimate, short for maximum a posteriori, the weights that are most probable after you have seen the data.

Where this lecture sits

Lecture 11 already met the problem and two fixes. It called them regularization: throw away the directions the data cannot see, or add a penalty that discourages large weights. Today’s lecture does three things. It reviews those two fixes, shows that every penalty is secretly a prior, and uses that to build new penalties on purpose, such as one that rewards smooth filters.

Logistics are short. This is one lecture from Pillow’s course, with no assigned reading. The course page lists the slides and notes, linked at the top and bottom of this page. We follow the lecture in its own order.

The road

  1. When least squares fails: more weights than samples, or columns that copy each other.
  2. Principal components regression: keep only the directions the data can see.
  3. Penalized maximum likelihood: add a bowl to the loss, and meet ridge regression.
  4. Ridge against PCR: one soft dial against one hard cut, direction by direction.
  5. MAP inference: the penalty is the log of a prior.
  6. A prior has a width: what λ means, and why data wash it out.
  7. The smoothness prior: neighbours should agree, and the graph Laplacian.
  8. Any Gaussian prior: the covariance C and its penalty.
  9. How strong a belief? Choosing λ by cross-validation.
  10. The whole lecture on one page: numbers, code, and links onward.
A least squares fit to 100 noisy samples with 100 free weights returns a wild scribble. What is the most direct cause?

Chapter 1

When Least Squares Fails

Find the two situations where the data cannot pick one answer, and see the valley that traps every method

Start with the smallest possible neuron: two weights, one recorded response. The first pixel is 1 and the second pixel is 2. The neuron fired 3. You want weights w1 and w2 with 1·w1 + 2·w2 = 3.

Try (3, 0). It works. Try (1, 1). It works. Try (−1, 2). It works too. Every point on a whole line fits the single measurement perfectly. The data do not choose among them. This is the lecture’s first failure case.

Case 1: more weights than samples

Call the number of weights d and the number of samples N. When d is bigger than N, the equations cannot fix every weight. You have fewer facts than unknowns. Our two-weight toy has d = 2 and N = 1.

The lecture draws this as the table X being wider than it is tall. A wide table always has columns that can be built from other columns. That is the link to the second case.

Case 2: columns that copy each other

Now give the toy neuron plenty of samples, but make pixel 2 always equal to pixel 1. The stimulus has rows (1, 1), (2, 2), (3, 3). The columns of X are identical. You can still learn the sum w1 + w2. You can never learn how it splits between the two.

The lecture calls such a design matrix rank deficient: some columns are not linearly independent, meaning one can be built from the others. Real stimuli are rarely this extreme. They are often close to it. A smooth stimulus makes the pixel next door almost a copy. Mathematicians call that ill-conditioned: not broken, but so close to broken that tiny noise has a huge effect.

Why the formula breaks

Least squares minimises the loss, the sum of squared prediction errors. Set its slope to zero and you get the normal equations. Solving them gives the formula below.

ŵMLE
The maximum likelihood estimate of the filter. The hat means “estimated from data”. Here 100 numbers.
(X⊤X)−1
The inverse of a d × d table that records how much each pair of pixels moves together. It exists only if no column copies the others. Here 100 × 100.
X⊤y
How strongly each pixel lines up with the responses. The ⊤ means transpose: flip rows into columns. Here 100 numbers.

Work the copied-pixel toy. With rows (1, 1), (2, 2), (3, 3), the table X⊤X has entries 1·1 + 2·2 + 3·3 = 14 in every cell. So it is [[14, 14], [14, 14]]. Its determinant is 14·14 − 14·14 = 0. A table with determinant 0 has no inverse. The lecture’s slide circles this and writes “no unique MLE”.

Now nudge the copy: make the second pixel 3.01 when the first is 3, and so on. The determinant becomes small but not zero, so an inverse exists. It is enormous. Noise in Y gets multiplied by that enormous inverse. That is the near-failure of Case 2, and it is what our hero shows.

The valley picture

Plot the loss against the two weights. When the data pin both weights, the loss is a round bowl with one lowest point. When a direction is not pinned, the bowl stretches along it into a long valley, a half-pipe with a flat floor. The lecture says exactly this. Turn it upside down, since likelihood is the negative of loss, and you get an infinite ridge.

Every point on the floor of the valley fits the data equally well. Noise decides which one the computer finds. The device below shows it. Each blue dot is the answer you would get from a fresh recording of the same neuron.

Walk the valley

Rings show the loss for two weights; the cross marks the true weights. Blue dots are least squares answers from 60 repeat recordings. Raise the correlation of the two pixels, then switch to one recording.

0.97

Toy: two weights, true values (1, 1), noise standard deviation 1, 20 samples from a fixed random seed. In the one-sample mode the stimulus is (1, 2) and the response 3, as in the text. Repeat recordings reuse the same stimulus with fresh noise.

Slide the correlation to 0.97 and watch the rings stretch into a long diagonal. The dots follow. The data pin down the sum of the two weights well, and the difference badly. Re-record, and the estimate slides along the floor of the valley while the loss barely changes.

Now the fix, which the next three chapters build. If the data leave a direction free, we must add information from somewhere else. The lecture offers two ways. One removes the free directions. The other adds a slope to the valley floor so it has one lowest point.

Two stimulus pixels always have the same value in your recording. Why is there no unique least squares filter?

Chapter 2

Principal Components Regression

Keep the few stimulus patterns the data can really see, fit only those, and throw the rest away

You flash a screen with smooth random patterns. Neighbouring pixels move together, so the screen does not really have 100 independent knobs. It has a few slow wiggles that vary a lot, and many fast wiggles that barely vary at all.

Here is the plain idea of the lecture’s first fix. The neuron can only teach you about patterns that actually varied in your stimulus. So estimate the filter only along those patterns, and leave the weights along the others at zero. The lecture states it as one sentence: penalise w so that it has fewer degrees of freedom.

Directions with a size

Each wiggle pattern is a direction in the space of 100 weights. A principal component (PC) is such a direction, chosen so that the stimulus varies the most along it. The second PC is the next-most-varying direction that is at right angles to the first. And so on, until you have 100. (Our PCA lesson and SVD lesson build them from zero.)

Each direction has a size, called an eigenvalue s. A large s means the stimulus swung a lot along that pattern, so the neuron’s response tells you a lot about the weight there. A tiny s means the stimulus hardly moved that way. Any weight along that pattern is nearly invisible in the data.

The lecture draws the invisible directions as the null space of X: the weight patterns that produce no change in the predictions at all. In the exact case the valley floor of Chapter 1 lies along the null space. The directions with a small but nonzero s are the nearly-flat ones.

Three steps

Principal components regression (PCR) does three things. The slide lists them with their formulas. We give the data shapes too, so you could code it.

  1. Project. Build the table B from the top k principal components. Compute the shrunken stimulus table X̃ = XB. Shape: 100 × 100 times 100 × k gives 100 × k. Each row is now just k numbers.
  2. Solve. Find the ordinary least squares weights for the small problem: w̃ = (X̃⊤X̃)−1X̃⊤y. Now the table being inverted is k × k with every direction well pinned. For a Poisson or Bernoulli GLM, maximise the likelihood by numerical search instead.
  3. Project back. Turn the k weights into a full filter: ŵPCR = B w̃. Shape: 100 × k times k gives 100.
ŵPCR
The final filter, all 100 weights. It lies in the span of the top k patterns, so it has only k degrees of freedom.
B
The top k principal components of the stimulus, as columns. It turns a small answer back into a full filter.
(X̃⊤X̃)−1X̃⊤y
Ordinary least squares in the small k-weight problem. It is safe because the stimulus really varies along each of these k directions.

A worked example, every step

Take the copied-pixel toy from Chapter 1. Rows of X are (1, 1), (2, 2), (3, 3) and the responses are y = (2.1, 3.9, 6.2). The table X⊤X is [[14, 14], [14, 14]]. Its two eigenvalues are 28 and 0.

Keep k = 1. The top direction is (1, 1) divided by √2, about (0.707, 0.707). Project: each row becomes (1·0.707 + 1·0.707) = 1.414, then 2.828, then 4.243. Solve: X̃⊤X̃ = 2 + 8 + 18 = 28, and X̃⊤y = 1.414·2.1 + 2.828·3.9 + 4.243·6.2 = 40.31. So w̃ = 40.31 / 28 = 1.44. Project back: 1.44 × (0.707, 0.707) = (1.02, 1.02).

PCR split the learnable sum, 2.04, evenly between the two pixels and set the unlearnable difference to zero. It made a choice the data could not make, and the choice is the quietest one.

In the neuron the same thing happens at larger scale. Slide k in the device below and watch the estimate sharpen and then break.

Keep k directions

Top: the true filter (dotted) and the PCR estimate. Bottom: the size of each stimulus direction on a log scale. Orange bars are kept, grey bars are thrown away. Drag k from 1 up to 60.

20

Our neuron and its 100 samples. The bars are the eigenvalues of X⊤X from a full eigendecomposition computed in your browser, plotted as log10. The slider stops at 60 because beyond that the error is astronomical.

Two things happen. For small k the estimate is too smooth, a flat blob that misses the filter’s fine shape: you threw away patterns the filter uses. Around k = 20 the match is excellent. Past about 40 the estimate unravels, and by 50 it explodes.

That explosion is the failure mode, and now we can name it. Symptom: spikes and wild swings in the estimate. Measurement: by k = 50 the error is over 60 times the true filter’s energy, because the 50th direction has a size of about 0.006 and we divide by it. Fix: choose k with care (Chapter 9), or use a softer cut. PCR is all-or-nothing: a direction is either fully trusted or fully ignored. The next two chapters build the soft version.

In principal components regression, why do we throw away directions with a tiny eigenvalue?

Chapter 3

Penalized Maximum Likelihood and Ridge

Tilt the flat valley floor with a bowl of cost on big weights, so that one point is lowest, and solve it in closed form

Imagine rolling a marble along the floor of the valley from Chapter 1. The floor is flat, so the marble stops anywhere. Now tilt the floor a little toward the middle. The marble rolls to one spot and stays there.

That is the lecture’s second fix. It does not delete directions. It adds a cost for large weights, so the combined landscape has one lowest point. The cost is the penalty function, written p(w). Maximising the log-likelihood minus the penalty is called penalized maximum likelihood.

The lecture draws the picture with two surfaces. The likelihood is an upside-down valley. The penalty is a downward bowl centred at the origin. Add them and the ridge acquires a peak.

The penalty for a linear Gaussian neuron

When the noise is Gaussian, maximising likelihood is the same as minimising squared error. So the penalized problem becomes penalized least squares: make the prediction error small, but pay a price for each weight you use. The price is the penalty.

The lecture’s specific penalty is the sum of the squares of the weights, scaled by a number λ. That choice is called ridge regression. The sum of squares is the L2 penalty, because it uses the squared length of the weight list. The scale λ is the ridge parameter: a dial for how much you dislike big weights. The lecture also names it L2 shrinkage and Tikhonov regression, all the same thing.

ŵridge
The ridge estimate of the filter. Here 100 numbers.
(X⊤X + λI)−1
The same table as Chapter 1, but with λ added to every number on its diagonal. I is the identity table: ones on the diagonal, zeros elsewhere. Here 100 × 100.
X⊤y
How strongly each pixel lines up with the responses, unchanged from Chapter 1.

Where does it come from? Write the cost as ||y − Xw||2 + λ||w||2. Take its slope with respect to w: −2X⊤(y − Xw) + 2λw. Set it to zero, move terms, and you get (X⊤X + λI)w = X⊤y. Multiply both sides by the inverse. That is the formula on the slide.

The fix works because adding λ to the diagonal lifts every eigenvalue of X⊤X by λ. An eigenvalue of 0 becomes λ, which is positive. So the table now always has an inverse, even for the copied pixels.

A worked example, every step

Return to the copied-pixel toy: rows (1, 1), (2, 2), (3, 3), responses (2.1, 3.9, 6.2). We found X⊤X = [[14, 14], [14, 14]], which has no inverse. Compute X⊤y = (1·2.1 + 2·3.9 + 3·6.2, and the same again) = (28.5, 28.5).

Choose λ = 1. Then X⊤X + λI = [[15, 14], [14, 15]]. Its determinant is 15·15 − 14·14 = 29, which is not zero. Its inverse is (1/29)[[15, −14], [−14, 15]]. Multiply by (28.5, 28.5): each weight is (15 − 14)·28.5 / 29 = 0.983.

The answer is (0.983, 0.983). As λ shrinks toward zero, it moves to (1.018, 1.018), the same split PCR chose in Chapter 2. As λ grows, both weights shrink toward zero. Ridge also splits the learnable sum evenly. It just does so with a smooth dial.

Add the bowl

Blue rings: the loss valley around the least squares answer (blue dot). Purple rings: the ridge bowl around zero. The orange curve is every ridge answer as λ grows; the orange dot is the answer at your λ. Slide λ.

3.2

Toy: two weights, true values (1, 1), 20 samples whose pixels have correlation 0.97, one fixed noise draw, noise standard deviation 1. The orange path is computed exactly from (X⊤X + λI)−1X⊤y at 200 values of λ.

At small λ the orange dot sits near the blue one, far out along the valley. As you raise λ it slides down the floor toward the centre. The bowl cares little about the steep direction, where the data are certain, and a lot about the flat one, where they are not.

Biased, and better

The ridge answer is biased: on average it is not equal to the true weights, because it pulls toward zero. A lecture slide notes that this still gives provably better performance than maximum likelihood for a good choice of λ, a result going back to James and Stein in 1960. The pull costs a little accuracy and removes the huge swings. A small, steady bias is better than a wild, unbiased guess. Our bias and variance lesson unpacks the trade.

Now the failure mode of the dial itself. Symptom: at too small a λ the estimate is still a scribble, and at too large a λ it is a flat line near zero. Measurement: on our neuron the error is 213 times the filter’s energy at λ = 0.001, 5% at λ = 10, and 56% at λ = 1000. Fix: choose λ by testing on held-out data. Chapter 9 does it.

Why does adding λI to X⊤X make ridge regression solvable even when two pixels are exact copies?

Chapter 4

Ridge Against PCR, Direction by Direction

Answer the lecture’s question: ridge keeps the same directions as PCR but trusts each one by a smooth fraction instead of a hard yes or no

You now own two fixes. PCR cuts: it keeps the top k stimulus patterns and drops the rest. Ridge bends: it adds λ to the diagonal. They look different. The lecture asks a sharp question about them. How does the ridge estimate compare with the PCR estimate, in terms of what each does to the singular values and vectors?

Think of 100 volume knobs, one for each stimulus pattern. The plain least squares answer sets every knob to full. PCR switches the first k knobs to full and the rest to zero. What does ridge do to the knobs? To find out, we follow the lecture’s warm-up problem.

The warm-up: one eigenvector, shifted

Suppose ui is an eigenvector of X⊤X with eigenvalue si. An eigenvector is a direction that the table only stretches, never turns. The eigenvalue is the stretch factor. So X⊤X ui = siui. The lecture asks: what is (X⊤X + λI)ui?

Split it in two. The first part gives siui. The second part, λI ui, gives λui, because the identity table changes nothing. Add them: (si + λ)ui.

(X⊤X + λI)ui
The ridge table applied to the i-th stimulus pattern.
(si + λ)ui
The same pattern, pointing the same way, with its size raised from si to si + λ. Ridge never turns a direction. It only adds λ to its size.

So ridge has the same eigenvectors as plain least squares. Only the eigenvalues change, from si to si + λ. That is the first half of the lecture’s answer. Ridge and PCR use the very same directions.

The second half: a knob for each direction

Now see what the inverse does. Because the table acts on ui by multiplying by si + λ, its inverse acts by dividing by it. Plain least squares divides the i-th component of X⊤y by si. Ridge divides by si + λ. The ratio between them is the volume knob.

keep fraction
How much of the plain least squares weight along direction i ridge keeps. A number between 0 and 1.
si
How much the stimulus varied along direction i. Big for the strong patterns, tiny for the weak ones.
λ
The ridge dial. A direction with si equal to λ is kept exactly half.

PCR has a keep fraction too. It is 1 for the first k directions and 0 for the others. Ridge is a smooth ramp between the same two ends: close to 1 where si is much bigger than λ, close to 0 where it is much smaller. Ridge is PCR with the cliff sanded into a slope. This is the second half of the lecture’s answer.

A worked example on our neuron

Take λ = 10. Direction 1 has s = 973, so ridge keeps 973 / 983 = 0.99. Direction 20 has s = 66.1, so it keeps 66.1 / 76.1 = 0.87. Direction 40 has s = 0.539, so it keeps 0.539 / 10.54 = 0.05. Direction 50 has s = 0.006, so it keeps 0.0006, which is nearly nothing. PCR with k = 20 keeps 1 for directions 1 to 20 and 0 for all the others.

The two sets of numbers are close for the strong and weak ends. They differ in the middle, around the direction where the data stop being informative. Ridge trusts a middling direction partly. PCR trusts it fully or not at all.

Compare the knobs

Top: both estimates against the true filter (dotted). Bottom: for each of the 100 directions, how much of its least squares weight each method keeps. Solid orange is ridge, dashed orange is PCR. Move λ and k until the two curves cross.

10
20

Our neuron and its 100 samples. Keep fractions are computed exactly from the eigenvalues of X⊤X: s/(s + λ) for ridge, 1 or 0 for PCR. The faint blue line is log10 of the direction sizes, only to show where the weak directions begin.

Try it. At λ = 10 the ridge ramp passes through 0.5 near direction 29, where the size of a direction equals λ. Set k = 29 and the two methods give very similar filters. That is the point: they are two ways to say “do not trust the weak directions”, one with a slope and one with a cliff.

The failure mode of the cliff is the one from Chapter 2. A direction just inside k is trusted fully, even if it is nearly noise, and one just outside is thrown away even if it carried some signal. Symptom: the PCR error jumps from 6% at k = 30 to 91% at k = 45. Measurement: the ridge error moves from 5% at λ = 10 to 14% at λ = 100, a gentler change. Fix: the slope. Ridge forgives a poor choice better than PCR does.

A direction of the stimulus has size s = 2 and you use ridge with λ = 8. What fraction of that direction’s least squares weight does ridge keep?

Chapter 5

MAP Inference: The Penalty Is a Log Prior

Read every penalty as a belief you held before the data arrived, and see that the penalized answer is the most probable weight after them

Before you record anything, a colleague tells you: “For this cell, the weight on that pixel is probably small. It is almost never above 2.” Then you record four trials and the average response is 1.6. Do you believe 1.6, or something between 1.6 and your colleague’s hunch?

Most people say something in between. Bayesian inference turns that instinct into arithmetic. The lecture’s next move, done on the board, is to show that the penalties of the last three chapters are exactly such hunches, written in a different form.

Four named pieces

Bayes rule has four parts, and each one is a plain thing. The likelihood is how probable the recorded data are if the filter were a particular w. The prior is how probable that w was before you saw any data. It depends on a few settings, called hyperparameters and written θ, that set its shape and strength. The posterior is how probable w is after seeing the data. The evidence (also called the marginal likelihood) is a single number that makes the posterior add up to 1.

P(w | Y, X, θ)
The posterior: belief about the filter after the data. The bar reads “given”.
P(Y | X, w)
The likelihood: the chance of the measured responses if the filter were w. This is the term that least squares maximises.
P(w | θ)
The prior: belief before the data, shaped by the hyperparameters θ.
P(Y | X, θ)
The evidence. It does not contain w, so it is the same number for every candidate filter. It only rescales.

(Our Bayes rule lesson builds this rule from a two-by-two table of counts.) The key reading is: posterior is proportional to likelihood times prior. Multiply the two bell shapes together and you get a new one.

From posterior to penalty

The MAP estimate, short for maximum a posteriori, is the filter at the peak of the posterior. To find it, take three small steps, the same three steps the lecture notes give.

  1. Take the log. The logarithm turns a product into a sum and keeps the peak in the same place. The log-posterior is log-likelihood + log-prior − log-evidence.
  2. Drop the evidence. It has no w in it, so it cannot move the peak. Throw it away.
  3. Maximise what is left. The sum of the log-likelihood and the log-prior.
ŵMAP
The filter at the top of the posterior.
log P(Y | X, w)
The log-likelihood: how well w explains the data. The same term as in plain maximum likelihood.
log P(w | θ)
The log-prior: how sensible w looks before any data. This is the negative of the penalty.

Compare this with Chapter 3, where we maximised log-likelihood minus a penalty p(w). The two lines are identical if p(w) = −log P(w | θ). That is the lecture’s punchline: every cost function is a log-prior, and penalized maximum likelihood is MAP inference. A penalty that charges for big weights is a prior that thinks big weights are rare.

A worked example, every step

Take one weight, the neuron’s gain to one feature. You record n = 4 trials. Each trial has noise with standard deviation σ = 1, and the average response is 1.6. By itself the data say: the gain is 1.6, give or take 1/√4 = 0.5. In terms of precision, which is one over the variance, the data carry precision 4/1 = 4.

The prior says the gain is centred at 0 with standard deviation τ = 1, so it has precision 1/1 = 1. Precisions add when you multiply bell shapes: 4 + 1 = 5. The peak is the precision-weighted average: (4 × 1.6 + 1 × 0) / 5 = 1.28.

So the MAP gain is 1.28, which sits 80% of the way from the prior’s 0 to the data’s 1.6. The data had four times the precision, so they get four fifths of the say. Move the sliders to see the shapes.

Multiply two beliefs

Blue: what the data say about the gain. Purple: the prior. Orange: their product, the posterior. The orange line is the MAP gain. Change how much data you have and how wide the prior is.

4
1.00

Illustrative numbers: average response 1.6 from n trials with noise standard deviation 1, and a zero-centred Gaussian prior of standard deviation τ. All three curves are exact Gaussians. Each curve is drawn at its own height scale so the peaks stay on screen.

Now the failure modes. Symptom: with a very narrow prior, τ = 0.2, the posterior hugs zero and ignores the data. Measurement: at n = 4 the MAP gain is 0.22 although the data say 1.6. Fix: either widen the prior or collect more trials. At n = 40 the data precision of 40 beats the prior’s 25 and the MAP rises to 0.98. The opposite symptom is a very wide prior: the MAP equals the data and inherits all their noise.

Why can we ignore the evidence P(Y | X, θ) when we look for the MAP estimate?

Chapter 6

A Prior Has a Width

Find the bell curve hiding inside the ridge penalty, read λ as its inverse variance, and watch more data wash the prior out

Suppose you could ask the ridge penalty to imagine filters. What would it draw? The answer shows what ridge secretly believes, and it makes λ easy to feel.

Recall from Chapter 5 that the penalty is the negative log of the prior. The ridge penalty is ½λΣwi2, a bowl. Which probability has a log that looks like a bowl? The Gaussian, the bell curve. Its log is a downward parabola.

So the ridge prior is a zero-centred Gaussian on every weight, all of them independent. Each weight is a bell curve centred at 0 with variance 1/λ. The lecture writes it w ~ N(0, (1/λ)I). Read N(mean, covariance) as “a Gaussian with this centre and this spread”. The covariance is the table of variances and co-movements of the weights. For ridge it is a diagonal table: every weight has the same variance and no two weights move together.

w
The filter before any data, treated as a random list of 100 numbers.
0
The prior’s centre: each weight is most likely zero. This is the pull toward zero.
(1/λ) I
The covariance. Every weight has variance 1/λ, so its standard deviation is 1/√λ. Big λ, narrow bell, strong belief that weights are tiny.

This is the lecture’s reading of λ: the inverse variance of the prior, also called its precision. At λ = 0 the prior has infinite width. It rules nothing out, so MAP equals maximum likelihood. At the other extreme, λ = ∞, the prior is an infinitely tall spike at zero, a Dirac delta, and the estimate is zero whatever the data say. Everything useful lives between.

What the prior imagines

Each purple curve is one filter drawn at random from the ridge prior. The dotted curve is the true filter. Slide λ and see how wild or how tame the prior’s imagination is.

5

The five draws use fixed random numbers, each scaled by the prior’s standard deviation 1/√λ, so they change smoothly with the slider. The noise variance is 1, so this λ is the noise-to-prior ratio of the closed form below. The true filter’s root-mean-square weight (0.45) is our own toy value.

Notice what ridge dreams up: jagged static, with no structure. Even at the best λ, none of the five curves looks like a smooth bump. The prior only says “weights are small”. That is a weak belief, and Chapter 7 gives a better one.

The closed form, with the noise in it

Chapter 3 hid one thing: the noise. If responses have noise variance σ2, the log-likelihood is −||y − Xw||2 / (2σ2) plus a constant. The log-prior is −½w⊤C−1w, where C is the prior covariance. Add them, expand the squares, and collect the terms with w. The log-posterior is −½w⊤(X⊤X/σ2 + C−1)w + w⊤X⊤y/σ2 + constant.

It is a downward parabola in w, so its peak is where the slope is zero: (X⊤X/σ2 + C−1)w = X⊤y/σ2. Multiply through by σ2 and solve.

ŵMAP
The posterior peak for a linear model with Gaussian noise and a Gaussian prior. This is exact, not an approximation.
(X⊤X + σ2C−1)−1
The data table plus a prior table scaled by the noise variance σ2. Noisier data, or a narrower prior, make the prior table matter more.
X⊤y
The same data summary as before.

For ridge, C = θI with prior variance θ, and the formula becomes (X⊤X + (σ2/θ)I)−1X⊤y. So the ridge number from Chapter 3 is the ratio of noise variance to prior variance. The lecture notes make this point: the larger the ratio, the more the weights shrink toward zero. Our noise has variance 1, so λ = 1/θ.

A worked example, every step

Set λ = 10 on our neuron. Then θ = 1/10 = 0.1, so each weight has standard deviation √0.1 = 0.32, and about 95% of draws fall within ±0.63. The true filter has 100 weights with energy 20.4, so a typical weight is √(20.4/100) = 0.45. Its peak weight is 1.6, which is five prior standard deviations out. The ridge prior finds our filter surprising, and it still works well. A rough belief is better than none.

More data wash the prior out

X⊤X adds up one contribution per sample, so it grows with N. The prior term λ stays the same size. With more data, the prior becomes a smaller share of the sum and the estimate moves toward maximum likelihood. Try it below.

Add more samples

Each dot is a fresh recording of our neuron with that many samples. Blue is maximum likelihood, orange is ridge at your λ. Both axes are logarithmic. Move λ, then step through the sample counts.

10
100

Illustrative: the same neuron and noise, with new smooth stimulus for each sample count (20, 30, 50, 70, 100, 150, 200, 300, 400). Below 100 samples there are fewer samples than weights, and “maximum likelihood” means ridge with λ = 0.000001. The stimulus is smooth, so its weak directions stay weak, and maximum likelihood is still poor at 400 samples.

Step N from 20 up to 400. The orange line falls fast, then flattens. The blue line is awful throughout, because our smooth stimulus keeps its weak directions weak however long you record. The read-out compares λ with the typical size of the diagonal of X⊤X. That is the lecture’s point in numbers: a fixed prior is a smaller push once the data are rich.

The failure mode is a prior that is wrong, not a prior that is weak. Symptom: with plenty of data, a strong ridge prior still biases the estimate toward zero. Measurement: at N = 400, λ = 1000 leaves the error about 18 times what λ = 100 achieves (6.8% against 0.4%). Fix: shrink λ as data grow, which is what choosing it by held-out data does automatically (Chapter 9).

With noise variance σ2 = 1 and a ridge prior of variance θ = 0.04, what ridge λ does the closed form use, and how strong is the pull?

Chapter 7

The Smoothness Prior and the Graph Laplacian

Charge for the jump between neighbouring weights instead of the size of each weight, and see why the prior is a random walk

Look at the dotted true filter in the hero. It is a smooth bump. Neighbouring weights are nearly equal, because the neuron integrates over time and does not flip sign every 10 milliseconds. Ridge knows none of this. It only says each weight should be small.

The lecture’s next prior encodes what you can see. Real filters are smooth, so charge for the jump between neighbouring weights. Two weights that differ by a lot cost a lot. Two that match cost nothing, however large they are.

The smoothness penalty

For our 100-weight filter the penalty is λ times the sum, over every neighbouring pair, of the squared difference (wi − wi−1)2. The lecture calls it the L2 smoothing prior and labels the sum the smoothness penalty. Picture the weights as 100 beads on a line, joined to their neighbours by springs. The penalty is the total stretch energy of the springs. A flat line has no stretch. A zigzag is stretched everywhere.

Why it is written with a matrix

To use this in the closed form from Chapter 6 you need it as w⊤Mw for some table M. The lecture names that table the graph Laplacian, Γ. A graph here is just dots joined by lines, and ours is a chain. Work three weights to see where Γ comes from. The penalty is (w2 − w1)2 + (w3 − w2)2. Expand: w12 − 2w1w2 + 2w22 − 2w2w3 + w32. Read off the coefficients into a table.

Γ
The graph Laplacian of the chain. The diagonal counts each weight’s neighbours (1 at the ends, 2 inside). Every neighbour pair gets a −1. For 100 weights it is 100 × 100.
λ w⊤Γw
The smoothness penalty written as one product of the filter with itself through Γ. The same number as the sum of squared jumps.

Now try three filters on three weights. The flat filter (1, 1, 1) has jumps 0 and 0, so the penalty is 0. The ramp (1, 2, 3) has jumps 1 and 1, so the penalty is 2. The zigzag (1, −1, 1) has jumps −2 and 2, so the penalty is 4 + 4 = 8. Compare the ridge penalty, the sum of squares: it gives 3, 14 and 3. Ridge treats the flat line and the zigzag as equally good. The smoothness penalty does not.

Plugging it in

Swap I for Γ in Chapter 3. The estimate becomes (X⊤X + λΓ)−1X⊤y, which is the formula on slide 16. Everything else is the same. The device below shows what each penalty is comparing. Choose a curve and read both costs.

Measure the stretch

Pick a filter. Thicker, darker stretches of the curve are bigger jumps between neighbouring weights. The read-out gives both penalties for that filter.

Our neuron and its 100 samples. Ridge penalty is the sum of squared weights; smoothness penalty is the sum of squared jumps between neighbours. The maximum likelihood filter is ridge with λ = 0.000001. The smoothing solver adds a leak of 0.0001λ to the diagonal so the arithmetic stays stable.

Read the numbers. The true filter costs 20.4 under ridge and 1.4 under smoothing. The maximum likelihood scribble costs about 1.3 million under ridge and 4.2 million under smoothing. Both priors know the scribble is absurd. The smoothing penalty is the one that cares about its zigzag; it separates it from the truth by a factor of three million.

A prior that cannot be written down

Chapter 6 said a penalty w⊤C−1w is a Gaussian prior with covariance C. For smoothing, C = (1/λ)Γ−1. There is a catch the lecture notes point out. The flat filter has a penalty of 0, so Γ has a zero eigenvalue and no inverse. The table is missing one dimension: it has rank d − 1.

That means the prior has infinite variance along the flat direction. It puts no limit on the average level of the filter. The notes call such a prior improper or degenerate. It is not a problem here, because the data term X⊤X supplies the missing direction. A new reading follows: each step between neighbours is an independent bell with variance 1/λ, and the level is unconstrained. The prior is a random walk.

What each prior imagines

Switch between the two priors at the same λ and compare their five random filters. Ridge draws are static. Smoothing draws are random walks.

32

Smoothing draws are built exactly from the cosine eigenvectors of the chain Laplacian, each scaled by 1/√(λ times its eigenvalue), with the infinite-variance flat direction left out so the curve stays on screen. Ridge draws are independent bells of standard deviation 1/√λ. Fixed random numbers, so both change smoothly with λ.

The read-out measures the difference. In the ridge draws, neighbours differ by about 1.4 times the size of a typical weight, which is the signature of static. In the smoothing draws they differ by a small fraction of it. The smoothing prior imagines filters that look like the real thing.

Its failure mode is a prior that is too confident about smoothness. Symptom: a sharp feature in the true filter, such as a narrow spike, is blurred away. Measurement: on our smooth neuron, error grows from 1.2% at λ = 30 to 8.8% at λ = 1000, as the whole curve flattens. Fix: lower λ, or choose a covariance with a shorter reach, which is Chapter 8.

The filters (1, 1, 1) and (1, −1, 1) have the same sum of squares, 3. What do the ridge and smoothness penalties say about them?

Chapter 8

Any Gaussian Prior: Who Moves Together

Meet the one table, the prior covariance C, that holds both ridge and smoothing, and learn to read what it says about your filter

Ridge and smoothing look like two separate tricks. They are two settings of one dial. The lecture’s summary names it: a general Gaussian prior, w ~ N(0, C). Pick the table C, and you pick the belief.

Think of C as a seating chart for the weights. A big number between weight i and weight j says “these two tend to rise and fall together.” A zero says “they ignore each other.” Ridge is a chart where everyone sits alone. Smoothing is a chart where neighbours are tied.

The penalty that goes with any C

The log of a Gaussian density is a quadratic. Take its log and flip the sign, and you get a penalty. The lecture gives the log-prior directly.

penalty
The cost you add to the squared error. Minus the log of the prior, as in Chapter 5.
½
A convention from the Gaussian density. It matches the ½λ used in the lecture notes.
w⊤C−1w
The filter measured through the inverse of the covariance. Weights that the prior expects to move together are cheap together and expensive apart. The inverse of C is called the precision table.

Two tables, one idea. C says who moves together. Its inverse says what costs. Ridge has C = (1/λ)I, so the cost table is λI, a diagonal. Smoothing has a cost table λΓ, which is nearly empty: only neighbours appear. But its C is a full table, because tying neighbours ties everyone, like a random walk. The lecture’s slides also name two relatives, ARD for sparse filters and ASD for automatic smoothness. Today’s lecture only names them, and so do we.

Three charts to compare

Ridge has a diagonal covariance and a diagonal precision. Each weight is its own island, and the filter has no shape at all. Smoothing flips the pattern. Its precision is nearly empty, with a 2 on the diagonal and a −1 beside it, because only neighbours are charged. Its covariance is full everywhere. A random walk remembers where it has been, so two weights far apart still share most of their history.

The third prior is our own example, with a reach you control. Weights within about one reach of each other move together, and weights farther apart are nearly independent. A short reach gives something close to ridge. A reach as long as the whole filter gives something close to a single flat level. Between them sits a family of smooth filters, with the width of the bumps set by one number. Its precision table looks fussy, with ripples of alternating sign. That is normal: inverting a smooth table always gives ripples.

A worked example, every step

Take two weights with prior covariance C = [[1, 0.9], [0.9, 1]]. They are strongly tied. The determinant is 1 − 0.81 = 0.19. The inverse is (1/0.19)[[1, −0.9], [−0.9, 1]] = [[5.26, −4.74], [−4.74, 5.26]].

Price the filter (1, 1). The product w⊤C−1w is 5.26 − 4.74 − 4.74 + 5.26 = 1.05, and half of it is 0.53. Now price (1, −1). It is 5.26 + 4.74 + 4.74 + 5.26 = 20, and half is 10. Both filters have the same sum of squares, yet this prior says the second is 19 times costlier. The prior has opinions about shape, not size.

That is the whole power of the general form. To build a new prior, choose how weights should move together, invert that table, and plug it into (X⊤X + σ2C−1)−1X⊤y from Chapter 6. Nothing else changes. Use the device to see the two tables for three choices, and to price two filters under each.

Read the seating chart

Left square: the covariance C, who moves together. Right square: the precision C−1, what costs. Purple is positive, blue is negative. Below: three filters drawn from the prior. Choose a prior and slide.

10
6

Illustrative priors on our 100-bin filter. Ridge: C = I/λ. Smoothing: C = (λΓ)−1 with its flat direction left out. Bump reach is our own example of a smooth covariance, C = (1/λ)·exp(−d2/2·reach2) for bins d apart, plus 0.001 on the diagonal to keep it invertible. The reach slider acts only on this prior. Prices are ½λ·w⊤(shape table)−1w.

Slide the reach and watch. With a short reach the covariance square collapses toward the diagonal, which is ridge. With a long reach it becomes a wide blob: weights far apart are tied. At a reach near 6 bins, matching our bump’s own width, the true filter is cheap. The read-out prices the true filter against the maximum likelihood scribble; every sensible prior says the scribble is millions of times costlier.

Now the failure mode, which is a prior with the wrong structure. Symptom: the estimate takes the prior’s shape instead of the data’s. Measurement: on our neuron, the bump prior with reach 25 and λ = 10 gives an error of 145% of the true filter’s energy, much worse than plain ridge’s 5%, while reach 6 at λ = 1 gives 0.6%. Fix: treat the reach as another setting to choose from data, exactly as we choose λ in the next chapter.

Two weights have prior covariance [[1, 0.9], [0.9, 1]]. Which filter does this prior charge more for?

Chapter 9

How Strong a Belief? Choosing λ

Hold some data back, score each λ on it, and see why the best score for predicting spikes is not always the best filter

You now have a dial and no rule for it. Too low and the filter is a scribble. Too high and it is a flat line. The lecture asks the question and gives its simplest answer on the slide: use cross-validation.

The notes name two approaches. The first is cross-validation, which this chapter builds. The second, maximising the evidence (the marginal likelihood from Chapter 5), is left for the next lecture, along with the Laplace approximation for models that are not Gaussian. We stay with the first.

Test on data the fit never saw

Training error always improves when you give the model more freedom, so it cannot choose λ. A model that memorised the noise scores perfectly on the data it memorised. The honest test is on data it did not see. These are the held-out samples.

K-fold cross-validation does this without wasting data. Split the N samples into K blocks, called folds. For each fold in turn, fit on the other K − 1 folds, predict the held-out fold, and record the squared prediction errors. Average over all samples. Repeat for each λ on a grid and pick the lowest.

CV(λ)
The cross-validation score for one λ: the mean squared prediction error on held-out samples. Lower is better.
yt
The measured response in a held-out bin.
xt⊤
The stimulus row for that bin: 100 numbers.
ŵ−fold(λ)
The MAP filter fitted without that fold, at this λ. The minus sign means “left out”.

Two practical points. First, the fold fits are only for scoring. Once you have picked λ, refit once on all the samples and report that filter. Second, a bigger K uses more data for each fit but costs more fits. In Bayesian terms λ is a hyperparameter, one of the settings θ that shape the prior. Cross-validation sets the prior’s width from data by asking which width predicts best. The evidence method of the next lecture asks which width made the recorded data most probable.

A worked example, every step

Our neuron has N = 100 samples, so K = 4 folds hold 25 each. We use contiguous blocks, because neighbouring bins share almost all their stimulus. For the smoothing prior, take four values of λ. For each, fit four times and score the four blocks. The averaged scores are: λ = 0.17 gives 32.4, λ = 3.04 gives 10.0, λ = 12.7 gives 10.3, and λ = 53 gives 14.0.

The lowest is λ = 3.04. The full grid of 30 values refines it to λ = 4.9, with a held-out score of 9.7. The score is far above the noise floor of 1 because each fit sees only 75 samples. The ranking, not the level, is what we use.

Score every λ

Top: prediction error against λ, on the data the fit saw (blue) and on held-out data (orange). Bottom: how far the filter is from the truth, which you can only know in a toy. The green dot is the cross-validation pick. Slide λ and compare priors.

12.6

Our neuron and its 100 samples; 4 contiguous folds of 25; 30 values of λ from 0.01 to 10,000, spaced evenly in the logarithm. Prediction errors are mean squared errors in response units. Filter error is as everywhere else on this page, a share or multiple of the true filter’s energy. All solves run in your browser.

Look at the top panel. The blue training curve rises as λ grows, because the filter gets less freedom. The orange held-out curve is a U. At low λ the filter chases noise and predicts new data badly. At high λ it is too stiff. The bottom of the U is the pick.

Now the surprise, and the failure mode. Switch to ridge. Symptom: the cross-validation pick is λ = 0.28, yet the filter error there is 35%, while λ near 12.7 would give 4.9%. Measurement: held-out prediction cares about directions in proportion to how much the stimulus moves along them. The weak directions barely affect predictions, so the held-out score cannot see the damage a noisy weak direction does to the filter. Fix: choose a prior that tames those directions. Switch to smoothing and the pick moves to λ = 4.9 with a filter error of 4.1%, close to the best possible 1.1%. A good prior does more than a good λ.

One last note from the lecture. All of this used the Gaussian noise model, where the MAP estimate has a closed form. For the Bernoulli and Poisson GLMs of earlier lectures, the lecture notes say you find the maximum by numerical optimisation instead. The prior and the penalty are unchanged, so everything here carries over to spike counts. See our GLM lesson and, for the spike-train version in a brain-computer interface, the point-process GLM.

Why can’t we choose λ by picking the value with the smallest error on the training samples?

Chapter 10

The Whole Lecture on One Page

Keep the numbers, the checklist and the code, and carry one idea out: a penalty is a prior in disguise

Here is the lecture in one paragraph. When the data cannot pin down every weight, because there are too few samples or the columns of X copy each other, least squares has no unique answer or a wild one. You can cut the weak directions away (PCR) or add a penalty that tilts the valley floor (ridge). Both keep the same directions, and ridge trusts each by the smooth fraction s/(s + λ). Every penalty is minus the log of a prior, so penalized maximum likelihood is MAP inference. The prior’s covariance C says who moves together. Ridge is one choice, a smoothness prior is another, and you set the strength by testing on held-out data.

The numbers to remember

QuantityWhat it saysOn our neuron
Maximum likelihood error100 weights from 100 noisy samples: a scribbleabout 64,000 times the filter’s energy
Ridge, λ = 10The bowl tilts the valley floor5.0% of the filter’s energy
Smoothing, λ = 30Charge for jumps between neighbours1.2% of the filter’s energy
PCR, k = 20 and k = 50A hard cut: the right k works, a wrong k explodes5.4%, then 62 times the energy
Keep fraction s/(s + λ)The share of each direction ridge keeps0.99, 0.87 and 0.05 for directions 1, 20 and 40 at λ = 10
Prior width at λ = 10λ is inverse variance, θ = 1/λeach weight has standard deviation 0.32
One-weight MAPPrecision-weighted average of data and prior(4 × 1.6 + 1 × 0) / 5 = 1.28
Chain penaltiesRidge cannot tell flat from zigzagflat 3 and 0, zigzag 3 and 8 (ridge, smoothing)
Cross-validation picksBest prediction, not always the best filterridge λ = 0.28 (35% error), smoothing λ = 4.9 (4.1% error)

The terms, in plain words

TermPlain meaning
Design matrix XOne row per bin of response, one column per delay. Here 100 × 100.
Penalty p(w)The price added to the squared error for a filter you dislike.
Prior P(w | θ)Belief about the filter before the data. The penalty is its negative log.
PosteriorBelief after the data: likelihood times prior, rescaled by the evidence.
MAP estimateThe filter at the top of the posterior.
Ridge parameter λInverse variance of the prior, or noise variance over prior variance.
Graph Laplacian ΓThe table that writes the sum of squared neighbour jumps as w⊤Γw. Rank d − 1.
Prior covariance CThe table of who moves together under the prior. Its inverse is the penalty table.

The checklist

The core as code

import numpy as np

def map_filter(X, y, lam, P=None):
    """MAP filter for y = X w + noise, with penalty lam * w' P w.
    X: (N, D) stimulus rows    y: (N,) responses
    P: None for ridge (identity), or a (D, D) shape such as a graph Laplacian."""
    N, D = X.shape
    P = np.eye(D) if P is None else P
    A = X.T @ X + lam * P            # (D, D) data table plus prior table
    b = X.T @ y                      # (D,)
    return np.linalg.solve(A, b)     # (D,) solve, never invert

def chain_laplacian(D):
    G = np.zeros((D, D))
    for i in range(1, D):
        G[i, i] += 1; G[i-1, i-1] += 1
        G[i, i-1] -= 1; G[i-1, i] -= 1
    return G                         # w' G w = sum of squared neighbour jumps

def pcr_filter(X, y, k):
    s, U = np.linalg.eigh(X.T @ X)   # eigenvalues ascending
    U = U[:, ::-1][:, :k]            # (D, k) top-k directions
    Xt = X @ U                       # (N, k) project
    wt = np.linalg.solve(Xt.T @ Xt, Xt.T @ y)   # (k,) solve
    return U @ wt                    # (D,) project back

def cv_lambda(X, y, lams, P=None, K=4):
    N = len(y); idx = np.arange(N); scores = []
    for lam in lams:
        sse = 0.0
        for fold in np.array_split(idx, K):          # contiguous blocks
            tr = np.setdiff1d(idx, fold)
            w = map_filter(X[tr], y[tr], lam, P)
            sse += np.sum((y[fold] - X[fold] @ w) ** 2)
        scores.append(sse / N)
    return lams[int(np.argmin(scores))]

Where it sits in the field

Ridge is the same L2 penalty that deep-learning libraries call weight decay, and the MAP reading says the same thing: a bell-curve prior on the weights. A Gaussian prior on a smooth function, with a covariance that sets the reach, is the starting point for Gaussian processes. Every step of a Kalman filter is a Gaussian posterior built from a Gaussian prior. In brain-computer interfaces the same machinery decodes movement from spikes, as in linear decoders and the point-process GLM.

Keep going

The takeaway. A penalty is a prior in disguise. Say what you believe about the filter, and the penalty, the closed form and the estimate follow.

Now press Present or Teach and explain it from memory: why the data leave the weights free, what PCR cuts and what ridge softens, why a penalty is a log prior, what λ means as a variance, and what the smoothing prior believes that ridge does not. Then go back to the instrument at the top, choose smoothing at λ near 30, and explain why it beats ridge on this neuron.

Credit. This lesson re-teaches Lecture 12 of NEU 560, Statistical Modeling and Analysis of Neural Data (Princeton, Spring 2024), by Jonathan Pillow, using the lecture’s slides and notes. The neuron, the data and every device are our own.

Which statement about penalized maximum likelihood and MAP inference is correct?
NEU 560 Lecture 12: Regularization and MAP inference, Jonathan Pillow, Princeton (Spring 2024)
The course site · Lecture slides · Lecture notes · Back to Gleams