A neuron model with 100 knobs and only 100 noisy samples fits the noise, not the neuron. A small penalty on the knobs pulls the estimate back toward the true filter.
Learn how to fit spike-train models with more unknowns than data, and why a penalty on the knobs is the fix.
Start with no penalty and see the wreck. Then press Ridge and slide the penalty up. Try Smooth too. Then we build, piece by piece, the models, the matrices and the maths underneath.
You need a little algebra and the idea of a spike train (the times a neuron fires). We build the rest from zero.
Based on Lecture 11 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow, with his lecture notes. Lecture slides. Every idea is re-taught here in our own words and devices.
Illustrative toy: a made-up 100-number filter, 100 samples (N = 100) of a white random stimulus, Gaussian noise of 0.5, and the least-squares estimate with the chosen penalty, solved live in your browser. The size 100 by 100 follows the lecture’s simulated example (slide 31). “Error” is the straight-line distance between the estimate and the true filter.
Chapter 0
See why a model with as many unknowns as data points cannot find the neuron, and why a penalty on the unknowns can
Picture a retinal ganglion cell. It is a neuron at the back of the eye that sends spikes (brief electrical pulses) to the brain. You show it a screen that flickers black and white 100 times a second, and you record when it fires. You want to know one thing: what pattern in the flicker makes this cell fire?
The usual answer is a stimulus filter. It is a list of weights, one for each recent moment. The first weight says how much the screen one time bin ago pushes the cell. The second says how much the screen two bins ago pushes it. And so on back to bin 100. Add up the weighted flicker and you have the cell’s drive.
That list holds 100 numbers. Each number is a knob that the fit must set. Each time bin you record gives one sample: one measured response with noise on it. Here the lecture asks a blunt question. What if you have 100 knobs and only 100 samples?
School algebra has the answer. Two unknowns and two equations have one solution. Two unknowns and one equation have many: a whole line of answers fits equally well. Fewer equations than unknowns means no unique solution.
Work a tiny case. A cell has two knobs, w1 and w2, and you record one sample: w1 + 2 w2 = 5. The pairs (1, 2), (3, 1) and (5, 0) all fit it exactly. So do (−1, 3) and (7, −1). Every point on a line fits, and nothing in the one equation prefers a point. Now scale up. A 100-knob filter with 60 samples has a 40-dimensional flat of exact fits, not a line.
The flat is not harmless. Each exact fit can have very different weights, some huge and of both signs. Two of them predict the training data equally well and predict a new stimulus very differently. A model that cannot say which one it prefers will pick by accident.
Real data adds noise. With exactly as many samples as knobs, the fit has just enough freedom to hit every noisy sample exactly. It then treats the noise as signal. The lecture calls this overfitting: parameters fit details of the training data that do not help predict new data (slide 31).
Drag the number of samples below. Watch the count of knobs that nothing pins down.
The warm bar is the 100 knobs. The teal bar is the samples you have. Red marks the knobs that no sample pins down. Drag the samples below and above 100.
The knob count D = 100 is the lecture’s simulated example (slide 31). The counting rule is exact for a linear model: with N independent equations, at most N of the D knobs can be pinned down. Noise is not shown here. Chapter 8 adds it.
Even past 100 samples the trouble is not gone. With 120 samples there are enough equations on paper. But the noise in each one still bends the answer, and some directions of the 100-knob space get very little information. Chapter 8 measures that.
Lecture 11 does not start with the penalty. It first finishes the generalized linear model (GLM) story from the lecture before: a recipe that turns a stimulus into a firing rate and then into spikes. It adds a neuron’s own spike history, adds links between neurons, and lists real uses. It then proves one lovely fact: the fit for the Poisson GLM has a single peak, so a computer cannot get stuck.
Only then does it ask what to do when there are too many knobs. The answer is regularization: add a cost that grows when the knobs look bad, so that the fit trades a little accuracy on the training data for a lot less noise. The slides give this idea one title slide and a promise: “details on board”. The lecture notes give the maths, and Chapters 9 and 10 teach it.
One short note on logistics. The slides open with a warm-up quiz that is not for credit. We turn it into Chapter 1. A neighbouring lecture builds the GLM itself; this lesson recaps it in two paragraphs where needed and links the full lesson on generalized linear models.
Chapter 1
Measure how far one spike-count model sits from another, and write down the score a GLM tries to maximise
You record one neuron for ten minutes. You count its spikes in each 100 ms bin. Then you write down two models of those counts. Which model is closer to the truth? You need a number for “closer”.
The lecture opens with five warm-up questions that are not for credit. They are the toolkit for everything after. Questions 1 and 2 are about KL divergence, question 3 about mutual information, and questions 4 and 5 about the log-likelihood of two GLMs.
Suppose the real spike-count distribution is P. You do not know P, so you use a codebook built for a model Q. A codebook here means a rule that gives short bit strings to likely counts and long ones to rare counts. The KL divergence from P to Q is the average number of extra bits you pay per sample for using Q’s codebook when the data come from P (slide 3).
Take a worked case. The truth P gives a 50% chance each to “no spike” and “spike”. Your model Q gives 90% to no spike and 10% to spike. The first term is 0.5 × log2(0.5 / 0.9) = 0.5 × (−0.848) = −0.424. The second is 0.5 × log2(0.5 / 0.1) = 0.5 × 2.322 = 1.161. The sum is 0.737 bits. Using the wrong codebook costs about three quarters of a bit per bin.
Now the two questions the slide asks. When is KL zero? When P = Q: every ratio is 1, every log is 0. The slide also notes that KL is never negative. When is it infinite? If Q(x) = 0 where P(x) is not zero. The model says an outcome can never happen, the data show it happening, and the ratio P/Q divides by zero.
Try both facts in the device. P is a neuron that fires 3 spikes per bin on average. Q is your model. Move Q’s mean, then forbid high counts.
Blue is the true distribution of spikes per bin. Warm is your model. Slide the model’s mean, then switch the model to one that forbids counts above 6.
Illustrative: the truth is a Poisson distribution with mean 3 spikes per bin. The model is a Poisson with your mean, or the same shape cut off above 6 and rescaled to add to 1. KL is summed over counts 0 to 40 in bits, live in your browser.
With the cut-off model, the truth still gives a count of 7 about 2% of the time. The model gives it exactly 0. The divergence jumps to infinity. This is the failure mode: a model that forbids something that happens scores minus infinity, however good it is elsewhere. The fix is a model that never says exactly zero. Chapter 7 shows that GLM nonlinearities such as the exponential do exactly this.
Question 3 asks you to write the mutual information I(x, y) as a KL divergence. Mutual information measures how much knowing the stimulus x tells you about the spike response y. The slide’s answer: it is the KL divergence from the joint distribution P(x, y) to the product P(x)P(y) (slide 3). It also equals H(y) − H(y | x): the entropy (the unpredictability, in bits) of the response, minus what is left once you know the stimulus.
The reason is simple. If x and y have nothing to do with each other, the joint equals the product, and KL is 0. The more x and y depend on each other, the further the joint sits from the product, and the larger the KL.
Work a small case. A stimulus x is on or off, 50% each. When it is on, the neuron spikes with probability 0.8. When it is off, 0.2. The joint probabilities are 0.4 (off, no spike), 0.1 (off, spike), 0.1 (on, no spike) and 0.4 (on, spike). The product P(x)P(y) is 0.25 for every cell, because the cell spikes half the time overall. Then I = 2 × 0.4 × log2(0.4/0.25) + 2 × 0.1 × log2(0.1/0.25) = 0.5425 − 0.2644 = 0.278 bits. The check from the other formula agrees: H(y) = 1 bit, H(y | x) = 0.722 bits, and 1 − 0.722 = 0.278.
Questions 4 and 5 are the score a GLM maximises. A likelihood is the probability the model gives to the data you actually saw. Its logarithm, the log-likelihood, turns a product over time bins into a sum. In both GLMs the stimulus enters through the linear drive xTw (the weighted sum of the stimulus) and then through a nonlinearity f.
The Bernoulli GLM is for binary data: one bin, spike or no spike. The coin-flip probability is p = f(xTw), and P(y | p) = py(1 − p)1−y. The log is
log P(y | x, w) = y log f(xTw) + (1 − y) log[1 − f(xTw)]
Check it with numbers. If f(xTw) = 0.8 and the cell spiked (y = 1), the score is log 0.8 = −0.223. If it stayed silent (y = 0), the score is log 0.2 = −1.609. A model that expected a spike is more upset by silence.
The Poisson GLM is for counts: how many spikes in the bin. The mean count is λ = f(xTw), and P(y | λ) = λy e−λ / y!. Its log, up to a constant that does not depend on w, is
log P(y | x, w) = y log f(xTw) − f(xTw) + const
With a mean of λ = 2 and an observed count of 3, the score is 3 ln 2 − 2 = 0.079, plus the constant −ln 3! = −1.792. Over many bins you add one such score per bin. Chapters 5 to 7 use this sum.
Chapter 2
Add the cell’s own recent spikes to the model and watch refractory pauses and bursts appear
A neuron fires. For the next millisecond or two it cannot fire again, however hard the stimulus pushes. A bit later it can, but it is still a little reluctant. Other cells do the opposite: one spike makes a second spike more likely, and the cell fires in bursts.
A model that looks only at the stimulus cannot capture either habit. It gives the same firing rate to the moment after a spike as to any other moment. This chapter fixes that, and it is the first half of the lecture’s recap (slides 4 to 6).
Slide 4 lists what the lecture before taught. A design matrix X stacks the stimulus history for every time bin, one row per bin. Least-squares regression is the same as maximum likelihood in a linear model with Gaussian noise. Other GLMs swap the noise: the Bernoulli GLM for spike or no spike, the Poisson GLM (also called the linear-nonlinear-Poisson model) for spike counts. You choose the nonlinearity f. The last item is the topic here: spike-history filters. Our GLM lesson teaches the first five items in full, and the course’s own GLM lecture comes just before this one.
The plain Poisson GLM gives a rate that depends on the stimulus alone: λ(t) = f(k · x(t)). Here x(t) is the stimulus history at time t, and k is the stimulus filter from Chapter 0. To add memory, slide 5 (after Truccolo and colleagues, 2004) feeds the neuron’s own recent spikes back in through a second filter h:
The right-hand side uses the rule exp(a + b) = exp(a) exp(b), which the slide writes out. It says something useful. With the exponential, stimulus and history multiply. A negative history drive scales the rate down. A positive one scales it up. The history acts as a gain control on top of the stimulus.
The slide also draws the lesson in two bullets. The output is no longer a Poisson process, because the rate now depends on the past spikes. And the model is also called an autoregressive (AR) model: a model that predicts the present from its own past. Given the past, each bin is still Poisson. That is why the rate is called “conditional”.
Let the stimulus alone give 40 spikes per second. Let the cell just fire. Take a post-spike filter that starts at −3 and decays with a time constant of 16 ms. The multiplier on the rate is eh.
Now flip the sign. A filter that starts at +0.4 multiplies the rate by 1.5 right after a spike, so one spike invites the next. Play with both in the device. Each trial uses 2 ms bins.
A stimulus switches on at 100 ms and off at 700 ms. Top: the rate in trial 1. Middle: spikes from eight trials. Bottom: the post-spike filter h. Drag its strength below zero for a refractory pause and above zero for bursts.
Illustrative toy, not the lecture’s data: a baseline of 5 spikes per second, a stimulus drive that lifts the rate to 40, a post-spike filter h(τ) = strength × e−(τ−1)/8 over the last 25 bins, and spikes drawn live with a fixed seed. Counts come from 60 trials of 800 ms; the rate per bin is capped at 0.5 spikes so a strong burst filter cannot run away.
At strength 0 the counts come out close to Poisson, where the variance equals the mean: the ratio is 0.85 here. At −3 the cell is more regular than Poisson, with a ratio of 0.37. At +0.4 the ratio climbs to 3.5, because bursts make some trials very busy and others quiet. Real recordings show all three behaviours, so a model with no history term will mismatch the spread of the counts even when it matches their average.
Slide 6 shows how far this goes. It borrows a catalogue of firing patterns from Izhikevich’s neuron model: tonic spiking, phasic spiking, tonic bursting, phasic bursting, spike-frequency adaptation and more, labelled A to P. For each pattern it shows a GLM, with its own stimulus and spike-history filters, that reproduces the spikes (Weber and Pillow 2017). The lesson: a handful of filters is enough to imitate a rich range of cell behaviour.
Chapter 3
Let each cell hear the spikes of its neighbours and see how coupling creates the correlations in real recordings
An electrode array records eleven retinal cells at once. You notice that when cell 5 fires, cell 6 tends to fire a few milliseconds later. Two cells that watch the same patch of screen would do this even if they never spoke. Or cell 5 might really excite cell 6. How can a model tell the two stories apart?
Slide 7 sets the goal: model the dependencies in multi-neuron spike trains. The input x is the stimulus, here a movie. The output y is a set of spike trains, one per neuron. The encoding model is the probability P(y1, y2, …, yn | x, θ), where θ stands for all the model’s knobs. An encoding model answers: given this stimulus, how likely is this pattern of spikes across all cells?
Slide 8 starts with the simplest version. Take the spike-history GLM from Chapter 2 and make one copy per neuron. Each copy has its own stimulus filter and its own post-spike filter. The copies do not talk. Two cells that share a stimulus will still be correlated, because the same stimulus drives both. But this model can only explain correlation that the stimulus causes.
Slide 9 adds the missing wire. The spikes of neuron 1 now also feed into neuron 2 through a coupling filter: a filter that weights the recent spikes of another cell, exactly as the post-spike filter weights the cell’s own spikes. Neuron 2 then has three inputs inside its exponential: its stimulus drive, its own history, and the spikes of neuron 1. The reverse wire, from 2 to 1, has its own filter.
Slide 10 writes the whole thing as one line, for neuron i:
Read the sign of hij as a verdict on the wire. Positive means cell j makes cell i more likely to fire: excitation. Negative means inhibition. Zero means no wire. Try it with two cells in the device. Neuron 1 fires at about 20 spikes per second on its own, and so does neuron 2. Only the filter from 1 to 2 changes.
Top: spikes of both cells over five trials. Bottom: how neuron 2’s firing rate changes just before and after a spike of neuron 1. The dashed line is neuron 2’s average rate. Drag the coupling from inhibitory to excitatory.
Illustrative toy, not the paper’s data: both cells have a constant drive worth 20 spikes per second and no history filter. The coupling filter from 1 to 2 is strength × e−(τ−1)/2 over 12 bins of 2 ms. 150 trials of 800 ms, drawn live with a fixed seed. The curve is the average rate of neuron 2 in the 2 ms bin at each delay from a neuron 1 spike.
At zero coupling the bars are flat: knowing that neuron 1 just fired tells you nothing about neuron 2. With excitatory coupling, a bump rises just after the spike of neuron 1. With inhibitory coupling, a dip opens in the same place. The bump or dip is the cross-correlogram, a plot of one cell’s rate against the delay from another cell’s spike. It is the tool that makes coupling visible.
Slide 11 shows the real version (Pillow and colleagues, 2008). On the left are the receptive fields of eleven cells, drawn as circles on the screen. On the right is a triangle of cross-correlograms, one for each pair. Each panel overlays three curves: the recorded data, the full GLM with coupling, and an uncoupled GLM. The point of the slide is that the cross-correlation between cells is something a model must earn, and coupling filters are how it earns it.
Here is where regularization starts to matter. Count the knobs, with illustrative sizes. Give every filter 10 time lags. Neuron i has a stimulus filter (10 knobs). It has a history or coupling filter from every cell, itself included (n × 10 knobs). With n = 11 cells that is 10 + 110 = 120 knobs per cell, and 1,320 knobs for the whole population.
The same stimulus-and-spike recording must pin down all 1,320 knobs. Double the cells to 22 and the count nearly quadruples to 22 × (10 + 220) = 5,060. Spike data are scarce, because a cell fires only a few times a second. So a population GLM is a prime case of more knobs than data, the trouble Chapters 8 to 10 treat.
The data flow is simple. For each neuron i you build a table X with one row per time bin. Its columns hold the stimulus history and the recent spikes of every cell. The target is yi, a column of spike counts. Given the past, the bins are independent Poisson draws, so the joint log-likelihood splits into one sum per neuron and per bin. Each neuron’s fit is its own optimisation. Chapter 5 builds this table.
Chapter 4
See how the same filter-and-nonlinearity recipe explains a decision in cortex and a map cell in the entorhinal cortex
So far the GLM has had a stimulus filter, a history filter and a coupling filter. It sounds like a model for retinal cells and flashing screens. But nothing in the recipe cares what the inputs are. Anything you can write as a time series can go in: a button press, a sound, the position of a rat.
Slides 12 to 15 show two examples from other brain areas. The lecture ends the tour with a note: many ideas for course projects lie along these lines. Treat each example as a template.
Area LIP (the lateral intraparietal area) is a part of the monkey cortex that helps turn a visual decision into an eye movement. Slides 12 and 13 describe a study by Park, Meister, Huk and Pillow (2014). A monkey watches a field of moving dots and decides which way they drift. It reports the choice with an eye movement, called a saccade.
The model reads three task variables: when the targets appear, when the dots move (and how strongly they move to one side, the coherence), and when the saccade happens. Each variable is a time series, mostly zeros with a pulse where the event happens. Each gets its own filter. The slide labels the three pieces: a target-locked component, a dots component and a choice component.
Add up the three filtered series and pass the sum through the nonlinearity. You get the cell’s firing rate. The slide checks it by plotting the model’s prediction against the real average firing rate (the PSTH, a peri-stimulus time histogram) at several coherences. The predicted and recorded curves line up. A decision signal in cortex comes out as a sum of a few filtered events.
The entorhinal cortex holds cells that map space. Slides 14 and 15 follow Hardcastle, Ganguli and Giocomo (2017). The animal runs around an arena. Four things are tracked at every moment: position (P), head direction (H), running speed (S) and theta phase (T), the phase of a slow brain rhythm.
One cell can care about several of these at once. The lecture calls that mixed selectivity. The slide’s pipeline is: each variable passes through its own learned tuning curve, the curves are summed, the sum goes through an exponential nonlinearity, and a Poisson process makes the spikes. The example cell on slide 15 has a clear position map, a head-direction peak, a flat speed curve and a theta peak.
Because of the exponential, the sum of tuning curves becomes a product of factors. Each variable acts as a multiplier on the rate. A worked case with illustrative numbers: the cell fires 2 spikes per second at rest. Standing in the middle of its place field multiplies the rate by 2.46. Facing its preferred direction multiplies by 2.23. Its speed is irrelevant, so the factor is 1. Its theta phase gives 1.46. The rate is 2 × 2.46 × 2.23 × 1 × 1.46 = 16.0 spikes per second.
Use the device to switch variables off and on. Watch what happens to the rate.
Each row shows how much one variable adds to the cell’s log-rate. Switch a variable off to remove it from the model. Move the sliders to change the animal’s head direction, speed and theta phase.
Illustrative cell, not the paper’s fit: a baseline of 2 spikes per second, a place field centred at 50 on a 0 to 100 track (log-rate +0.9 in the middle, falling to −0.6 away from it), head direction 0.8 × cos(angle − 120°), a flat speed curve, and theta 0.4 × cos(phase − 200°). The structure, a sum of tuning curves inside an exponential, is the one on slide 14.
Switch off the position factor and the rate falls by a factor of about 2.5, no matter where the other variables sit. That multiplication is the exponential at work, and the reason this model is easy to read: each variable’s effect is its own factor.
How do the task variables of LIP and the maps of entorhinal cortex enter the fit? Exactly as the stimulus did in Chapter 2. A variable that changes in time becomes a set of columns in the design matrix: its value at the current time bin, one bin ago, two bins ago and so on, or a handful of smooth bumps that span a window. A variable that lives in space, like head direction, is cut into bins or smooth bumps too. The knobs are the weights on those columns.
As data flow, take an illustrative trial of 2 seconds cut into 10 ms bins. That gives 200 rows. The design matrix X has one column for each lag of each variable, so a model with 4 variables and 20 columns each has 80 columns, an array of shape 200 × 80. The target y is a column of 200 spike counts. The fit returns one weight per column, 80 numbers, and the tuning curves on the slides are just those weights plotted against their column index.
This is why the GLM travels so well. The recipe stays fixed: build columns, weight them, apply f, draw spikes. Only the columns change. It also explains the final warning of the chapter. Every variable you add brings dozens of new knobs. A rich model on a short recording is again a case of more knobs than data, and again it needs a penalty.
Chapter 5
Turn a flicker, a spike train and a few neighbours into one table that a GLM can fit
Slide 16 asks the question every student has: what does this look like in code? The answer starts with one table, and building it is most of the work. Slides 17 to 21 build it in three steps, on real data.
The data (slides 17 and 18) come from a parasol retinal ganglion cell, one of the large cell types of the primate retina, recorded by Uzzell and colleagues (J Neurophysiology, 2004). The stimulus is a binary flicker: the whole screen is black or white, and it flips at random between the two. The slides plot the raw flicker as a blue square wave and the spike times as dots below it. Then they cut time into bins and count the spikes in each bin, which gives a column of small whole numbers.
To predict the spike count in bin t, the model needs the last 20 or so flicker values: the screen now, one bin ago, two bins ago. Stack those values in a row. The row for bin t then reads [s(t), s(t−1), s(t−2), …]. Do this for every bin and you get a table with one row per time bin and one column per lag. That table is the design matrix X. The spike counts are the column Y.
Look at the matrix on slide 19, labelled “stimulus-only GLM”. It shows diagonal stripes. Each column is the previous column pushed down by one row: a copy of the same flicker, delayed by one more bin. The stripes are not a feature of the cell. They are just what delayed copies look like.
The model P(Y | X) then says: the spike count in row t has a Poisson distribution whose mean is f applied to row t of X times the weight vector w. The weights w are the stimulus filter of Chapter 0. Fitting the filter is fitting one weight per column.
Slide 20 adds the spike-history columns. Next to the stimulus block it places a block whose row t reads [y(t−1), y(t−2), …]: the cell’s own recent spikes. The slide labels the two halves “stimulus portion” and “spike-history portion”. The history starts at lag 1, not lag 0, because a bin cannot use its own spike to predict itself. That would be cheating, and the fit would learn nothing.
Slide 21 adds three more neurons. For each neighbour you add another block of delayed copies, this time of that neighbour’s spikes. The slide shows five blocks: the stimulus, then four spike-history blocks, one for the cell itself and one for each of three neighbours. The weights on a neighbour’s block form the coupling filter from Chapter 3.
The device builds all three versions. The picture is a small piece of the real thing: 30 rows out of 1,000.
Each row is one time bin, each column one delayed copy. Blue is the stimulus. Teal is the cell’s own spikes. Purple is the neighbours’ spikes. Pick a model, then change the number of lags.
Illustrative toy: a random binary flicker and random sparse spikes, seeded so they are the same on every visit, for a record of 1,000 bins with 38 spikes in the target cell. The block layout follows slides 19 to 21. The constant column of ones that carries the baseline rate is left out of the column count.
Watch the column count. Stimulus only, with 20 lags, gives 20 knobs. Adding 10 lags of own history gives 30. Adding three neighbours with 10 lags each gives 60. The row count stays 1,000, but the number of spikes in the record, the events that carry the information, stays at 38. The ratio of knobs to spikes keeps rising.
The lecture points to a tutorial with the full code in Python and MATLAB (slide 16, from the Pillow lab). The core is one short function that makes a block of delayed copies.
python import numpy as np def lagmat(v, n, start): # column j is v delayed by (start + j) bins M = np.zeros((len(v), n)) for j in range(n): d = start + j M[d:, j] = v[:len(v) - d] return M X = np.hstack([lagmat(stim, 20, 0), # stimulus, lags 0..19 lagmat(y_self, 10, 1), # own spikes, lags 1..10 lagmat(y_n2, 10, 1), lagmat(y_n3, 10, 1), lagmat(y_n4, 10, 1)]) X = np.hstack([X, np.ones((len(stim), 1))]) # constant column: the baseline print(X.shape) # (1000, 61)
Read the shape. The first number, 1,000, is the number of time bins. The second, 61, is 20 stimulus lags plus 4 × 10 history lags plus 1 constant. The stimulus block starts at lag 0, and every spike block starts at lag 1. The first few rows have zeros where the data would reach before the start of the record. That is an edge effect: the model sees silence there, not missing data. It is usually fine, but note it when a fit looks odd in the first few bins.
Links to the full tutorials, as given on the slide: Python and MATLAB.
The symptom: the fitted filters look jagged, with large weights of both signs, and the model predicts the training spikes well and fresh spikes badly. The measurement: count the knobs and count the spikes. A record with 38 spikes and 60 knobs has fewer events than unknowns. The fix has two parts. You can shorten the filters, which costs flexibility. Or you can keep them long and add a penalty, which costs a little bias and buys a lot of stability. The second path is the rest of this lesson.
Chapter 6
Learn the one shape property that guarantees an optimiser cannot get stuck on a false peak
You stand on a foggy hillside and want the lowest point. You cannot see. You can only feel which way the ground tilts under your feet, so you step downhill, again and again. In a smooth round bowl, this works from anywhere. In a range of hills and dips, you may stop in a small dip and never learn that a deeper valley lies over the ridge.
Fitting a GLM is this walk. The ground is the negative log-likelihood, and every point in it is one setting of the knobs. Slide 22 opens a short section with a title: the mathematical properties of the log-likelihood for a Poisson GLM. This chapter gives the vocabulary. Chapter 7 uses it.
Slide 23 lists the vocabulary. A function is concave if it has downward curvature everywhere: a hill, like an upside-down bowl. It is convex if it has upward curvature everywhere: a bowl. Turn one over and you get the other: f is concave exactly when −f is convex.
That gives the slide’s fourth bullet. Maximising a concave function is the same job as minimising a convex one. We maximise a log-likelihood, a hill, or we minimise its negative, a bowl. These are two names for one task.
The fifth bullet is the prize. Both shapes preclude non-global local optima. A local optimum is a point that beats all its near neighbours. A global optimum is a point that beats every point anywhere. In a bowl or a hill, any local optimum is the global one. There is no small dip to get stuck in.
The slide gives a test you can run on any function. Pick two points x1 and x2, and a number a between 0 and 1. The point a x1 + (1 − a) x2 lies between them: at a = 1 it is x1, at a = 0 it is x2, and at a = ½ it is the midpoint.
In words: the curve may never rise above any of its chords. A bowl passes. A hill fails, because its curve sits above its chords.
Do one by hand. Take f(x) = x2, x1 = −1, x2 = 2 and a = ½. The in-between point is 0.5, and f(0.5) = 0.25. The chord height is ½ × f(−1) + ½ × f(2) = ½ × 1 + ½ × 4 = 2.5. Since 0.25 ≤ 2.5, the test passes here. Now take f(x) = −x2 with the same numbers. The curve is −0.25 and the chord is ½(−1) + ½(−4) = −2.5. The curve is above the chord, so this function is not convex. It is concave.
You rarely check chords by hand. For a smooth function of one variable, the slide gives the shortcut: f″(x) ≥ 0 everywhere. The second derivative measures curvature, so this says “never curving downward”. For a function of many variables, the shortcut uses the Hessian H, the table of all second derivatives ∂2f / ∂xi∂xj. The function is convex if all eigenvalues of H are ≥ 0, or equivalently xTHx ≥ 0 for every vector x. An eigenvalue here is the curvature along one special direction. All must be non-negative: no direction curves downward.
Two-variable examples, with the numbers worked. For f(x, y) = x2 + 3y2 the Hessian is diagonal with entries 2 and 6, both positive: a bowl, steeper along y. For f(x, y) = x2 − y2 the entries are 2 and −2. One direction curves up, the other down: a saddle, not convex.
Try these ideas on the curves below. Choose a function, move the two points and the mixing number a, and then drop two balls and let gradient descent roll them downhill.
The dashed line is the chord between the two points. The warm dot is the curve at the mixing point and the teal dot is the chord there. Press Roll the balls to start two balls at the two points and let each one slide downhill.
Toy curves on the range −3 to 3: x²/2, 4.5 − x²/2, 0.12x4 − 0.9x² + 0.4x + 3, and ex/2. The curvature test samples the second derivative at 120 points. The balls follow plain gradient descent for 150 steps and stop at the edge of the range if the curve keeps falling.
On the bowl, both balls end at the same place, whatever their start. On the two-valley curve, a ball that starts on the left of the central bump and a ball that starts on the right end in different valleys, and only one of those is the global minimum. That is the failure mode in its pure form. The symptom: two runs of the optimiser end at different answers. The measurement: compare the final value of the objective. The fix is to use a problem that is convex.
The least-squares error ||Y − X w||2 is a bowl in the knobs w. Its Hessian is 2 XTX, a matrix whose eigenvalues are never negative. When one eigenvalue is exactly zero, that direction is flat: the bowl has an infinite trough, and many knob settings tie for best. That is the situation of Chapters 0 and 8. The penalty of Chapter 9 adds a little upward curvature in every direction and ends the tie.
Chapter 7
Check the log-likelihood of a spiking GLM piece by piece and see why any stimulus gives a fit you can trust
You have a big recording and a long list of knobs. You start a computer search for the best weights. A worry nags: what if the search stops on a false peak, and a second search from a different start finds a better answer? For many models this is a real danger. For the Poisson GLM it is not, and slides 25 to 27 say exactly why.
The result is due to Paninski (2004). It says that the log-likelihood of the Poisson GLM is concave in the parameters, the stimulus filter k and the history filter h, for any stimulus X and any spike data Y, as long as the nonlinearity f meets two conditions.
Slide 25 lists three nonlinearities that pass: f(x) = ex, the rectified linear function f(x) = ⌊x⌋ (the “ReLU”, which is x when x is positive and 0 otherwise), and the soft-plus f(x) = log(1 + ex), a smooth version of the rectifier. It also gives a rule of thumb: f must grow at least linearly and at most exponentially. Convexity stops f from flattening out. Log-concavity stops it from growing faster than an exponential.
Test the first example by hand. For f = ex, f″ = ex > 0, so f is convex. And log f = x is a straight line, which is concave (and convex). Both conditions hold. Now take f = ex². It is convex, but log f = x2 is a bowl, not a hill. It grows faster than any exponential, and it fails.
Slide 26 lists the three facts the proof needs (marked optional in the lecture, but worth seeing). First, an affine map (multiply by a number and add a number) keeps convexity: if f(x) is convex, so are a f(x) + b for a ≥ 0 and f(ax + b). Second, a sum of convex functions is convex. Third, a linear function is both concave and convex.
Slide 27 then writes the log-likelihood of the Poisson GLM and splits it in two:
Look at what the proof never used: the stimulus values, the spike counts (except that they are not negative) and the filter lengths. That is why the theorem has no side conditions on the data.
See it happen. The device below fits one knob w to 14 bins of toy data. Pick the nonlinearity, and watch the log-likelihood curve. Where the curve bends downward it is green. Where it bends upward it is red.
The curve is the log-likelihood of the data at each value of the knob w. Choose a nonlinearity. The first two satisfy the theorem. The last three do not.
Toy data, labelled illustrative: 14 time bins with a random stimulus, and spike counts drawn from an exponential-nonlinearity model with true weight 0.6; the same counts are scored under each nonlinearity. The rectifier from the slide is left out: its log is minus infinity wherever the drive is zero or negative, which a one-knob toy cannot show. “Square” is x². “Too fast” is ex², which grows faster than any exponential. “Bump” is e−x² plus 0.05. Concavity is checked by second differences on 240 grid points.
With the exponential the curve is a single green hill. With soft-plus it is green too, with one peak. With the square, two peaks appear, one at a positive w and one at the mirror negative value: the model cannot tell w from −w, so a search can land on either. The too-fast function also gives two peaks, with red patches where the curve bends upward. The bump shows the subtle case: it usually has one visible peak, yet red runs along much of the curve, so a different data set can break it. The theorem promises concavity only when both conditions hold.
The slide puts it this way: there is no restriction on the choice of stimuli, and you can easily find the maximum-likelihood fits to GLM parameters for any stimuli and spikes. Natural movies, correlated noise, sounds, your own spike history: all of them give one hill. A simple uphill search, such as gradient ascent, finds the top from any start.
Slide 28 collects the whole GLM story so far. It adds two items to the earlier list: coupling filters from other neurons, and convexity and concavity. The next chapter turns to what this guarantee does not give you. A concave log-likelihood has a top, but when the data have fewer samples than knobs, the top is a flat ridge, and every point on it ties.
Chapter 8
Find out where maximum likelihood breaks, why it is worst at exactly as many samples as knobs, and what overfitting looks like
Slide 29 opens a new section: MAP inference and regularization. Slide 30 gives its first title, “Modern statistics”, and one bold claim: there are now more dimensions than samples. In symbols, D ≥ N. Neural recordings have long filters, many neurons and many task variables. They often have fewer informative samples than that.
The slide draws the setup as a picture of matrices. The N observations y1 to yN form a column. The design matrix X has N rows and D columns, one regressor per column. The weights form a column of D unknowns. Add noise. In one line:
The formula needs the inverse of XTX, a D by D matrix. The lecture notes name two cases where that inverse does not exist. One: there are more features D than observations N. Two: the columns of X are not independent, so X is singular. This second case is called co-linearity: one column can be built from the others, for example a stimulus lag that is a copy of its neighbour. In both cases the formula fails, and many different w give exactly the same fit.
The notes give the geometry. The squared error ||Y − X w||2 is a bowl in the knobs (Chapter 6). In these two cases the bowl has an infinite trough: a whole line or plane of knob settings at the same lowest height. In log-likelihood terms it is an infinite ridge. The data cannot say which point on it is right.
There is a quieter third case. If XTX is merely ill-conditioned, the inverse exists, but the trough is only slightly tilted. You could walk a long way along some directions and barely change the fit. The estimate exists on paper. In practice noise decides where you land.
Work a two-knob case. Let X have two nearly identical columns: rows (1, 1) and (1, 1.01). The truth is w = (1, 1), so Y = (2, 2.01). The inverse of X is 100 times larger than the entries of X: X−1 = [[101, −100], [−100, 100]]. Check: it gives (101 × 2 − 100 × 2.01, −100 × 2 + 100 × 2.01) = (1, 1). Now add a noise of just 0.01 to the second measurement, so Y = (2, 2.02). The answer is (202 − 202, −200 + 202) = (0, 2). A change of 0.01 in the data moved each weight by 1, a hundred times more. That factor is the noise amplification, and it grows as the columns get closer to identical.
The lecture runs the real thing: a 100-element filter (D = 100) and 100 noisy samples (N = 100). It draws the true filter as a dotted curve with one sharp peak, and the maximum-likelihood estimate beside it, a spiky red mess. Its comment: “overfitting”, parameters fit to details of the training data that are not useful for predicting new data.
The device repeats the experiment with our own toy filter. Drag the number of samples from 50 to 400. The top plot is the estimate. The bottom curve is the error at every N.
Top: the blue dashed true filter and the warm maximum-likelihood estimate. Bottom: the error at every sample count, with a marker at yours. Red dots mark values off the top or bottom of the plot.
Illustrative toy, the same as the hero: D = 100 knobs, a made-up filter, a white random stimulus and Gaussian noise of 0.5, one fixed data set. Where N is below 100 no unique answer exists, so the plot shows the shortest filter that fits the data exactly (a tiny penalty of 0.000001 picks it). The all-zeros guess is 2.55 from the truth.
Read the bottom curve. Past 100 samples the error falls as more data arrive: it is 0.43 at 200 samples and 0.31 at 400. Below 100 the exact-fit answer is rough but bounded: 1.88 at 50 samples. The worst place is right at N = 100, where the error is 3.93, about one and a half times the length of the true filter itself (2.55). At exactly as many samples as knobs, there is just enough freedom to hit every noisy sample and no room left to average the noise away. The smallest eigenvalue of XTX is then closest to zero, and the noise is amplified the most.
The symptom, then: a jagged filter with huge weights of both signs, and a fit that predicts its own training data perfectly and new data badly. The measurement: error on held-out data, or the ratio of knobs to samples. The fix is the subject of the next chapter. The slide gives it in two words: solution, regularize.
Chapter 9
Add a cost for large knobs, derive the closed-form answer, and watch a single dial trade fit against noise
Slide 32 says it in one line: solution, regularize. It adds “details on board”. The lecture notes carry those details, in sections 3, 4.1 and 5, and this chapter follows them.
Think back to the trough of Chapter 8: many knob settings fit the data equally well, and noise picks one. Regularization is a tie-breaker with a taste. Among all the settings that fit about equally well, prefer the one with small, tame weights.
The notes call the idea penalized maximum likelihood (or penalized least squares, in the Gaussian case). Instead of maximising the log-likelihood alone, subtract a cost c(w) for settings you dislike, scaled by a number λ:
The notes spell out how λ behaves. At λ = 0 you get the plain ML estimate. Larger λ pushes the estimate further from the ML estimate. In the least-squares case the same recipe reads: minimise ||Y − X w||2 + λ c(w).
The most popular choice is the ridge or L2 penalty: the squared length of the weight vector, c(w) = w12 + w22 + … + wD2 = wTw. It charges a lot for one big weight and little for many small ones. This is also called L2 shrinkage, because it shrinks weights toward zero. In the linear-Gaussian model, the estimator is ridge regression, and it has a closed form that looks like the ML formula with one extra term:
ŵridge = (XTX + λI)−1 XTY
Here I is the identity matrix: ones on the diagonal, zeros elsewhere. The notes derive it in five lines (equations 15 to 20). Here they are, one move each.
The matrix XTX has no negative eigenvalues, but some can be zero or tiny. Those are the flat or nearly flat directions of the trough. Adding λI raises every eigenvalue by λ. None can now be below λ, so the inverse always exists, and noise is amplified by at most 1/λ in any direction.
Return to the two-knob case of Chapter 8. With the noisy data Y = (2, 2.02), plain least squares gave the wild answer (0, 2). With λ = 0.1 the matrix becomes [[2.1, 2.01], [2.01, 2.1201]], XTY is (4.02, 4.0402), and the solution is (0.975, 0.981). The truth was (1, 1). The penalty gave up a tiny bit of exactness on the data and avoided the 100-fold amplification.
A note on names. Frequentists call this “penalized maximum likelihood”. The title of slide 29, MAP inference, uses the Bayesian name: maximum a posteriori, the single most probable knob setting once a prior belief about the knobs is included. The two are the same computation. A Gaussian prior on the weights, with standard deviation τ, gives the ridge penalty with λ = σ2/τ2, where σ is the noise level. This is a standard result, not one the slides prove. In the toy below, σ = 0.5 and λ = 3 means a prior standard deviation of about 0.29.
Try the dial. For each number of samples, drag λ from 0.1 to 1,000.
Top: the true filter and the ridge estimate. Bottom: two curves against λ. Warm is the distance from the true filter. Blue is the misfit to the recorded data. The teal dot marks your λ.
Illustrative toy, the same data and filter as the hero (D = 100, noise 0.5). The estimate is the closed form (XTX + λI)−1XTY, solved live. “Misfit” is the root-mean-square gap between the data and the fit on the same samples used to fit. The best λ shown by the faint line needs the true filter, which a real experiment does not have; in practice one picks λ on held-out data.
At N = 100 with no penalty, the training misfit is 0.00: the model hits every sample, and is 3.93 from the truth. At λ = 3.2 the misfit rises to 0.19, and the error falls to 1.13, more than three times better. Push λ to 1,000 and the estimate shrinks toward zero: the misfit climbs to 2.13 and the error to 2.35, close to the all-zeros score of 2.55. That is the failure mode on the other side: too much penalty. The symptom is a flat, tiny filter. The measurement is a misfit that rises toward the size of the data itself. The fix is a smaller λ.
Change to N = 200. The best λ is smaller (about 0.6) and the gain is tiny, 0.43 against 0.43 with no penalty: more data need less help. The next lecture, MAP inference, develops the prior view in full. Our regularization lesson and bias and variance lesson give the wider picture. A smoothness penalty, which suits filters, is next.
Chapter 10
Charge for jumps between neighbouring weights, build the graph Laplacian that does it, and compare the result to ridge
Look at the true filter in the hero. It is a smooth hill with a small dip. A filter of a real neuron looks like that too: the weight on the screen 40 ms ago is close to the weight on the screen 41 ms ago. Ridge knows nothing of this. It would charge the same for a smooth bump and for a jagged saw of the same total size.
Section 4.2 of the lecture notes fixes that. If you have reason to believe the weights should be not merely small but smooth, as when they weigh a quantity that varies over time or space, then penalise the differences between neighbours.
The notes give the penalty as the sum of squared differences between adjacent weights:
Work a four-weight example. Let w = (1, 2, 4, 3). The jumps are 2 − 1 = 1, 4 − 2 = 2 and 3 − 4 = −1. Squared and added: 1 + 4 + 1 = 6. A flat vector like (2, 2, 2, 2) costs 0, and a saw (1, 4, 1, 4) costs 9 + 9 + 9 = 27.
The notes then show a trick that puts the penalty in the form the ridge derivation needs. Collect the differences with a matrix D. For four weights, with one row per neighbouring pair:
D = [[−1, 1, 0, 0], [0, −1, 1, 0], [0, 0, −1, 1]], so ||D w||2 = wTDTD w = wTΓw
The product Γ = DTD is the graph Laplacian of the chain of weights. Its rule is simple (equation 10 of the notes). The diagonal entry Γii is the degree of weight i: how many neighbours it has. The off-diagonal entry Γij is −1 if i and j are neighbours and 0 otherwise. For four weights in a row it is the tri-diagonal matrix [[1, −1, 0, 0], [−1, 2, −1, 0], [0, −1, 2, −1], [0, 0, −1, 1]]. The end weights have one neighbour, so they get 1. The middle weights have two, so they get 2.
Check it on w = (1, 2, 4, 3). Multiply: Γw = (1 − 2, −1 + 4 − 4, −2 + 8 − 3, −4 + 3) = (−1, −1, 3, −1). Then wTΓw = 1 × (−1) + 2 × (−1) + 4 × 3 + 3 × (−1) = −1 − 2 + 12 − 3 = 6. It matches the direct sum.
The graph view reaches past one line of weights. For a receptive field drawn on an image, each pixel has up to four neighbours, and Γ has degree 4 on the diagonal of the interior pixels. The same formula penalises jumps in any direction.
The algebra of Chapter 9 goes through unchanged, with Γ where I stood. The closed form (equation 14 of the notes) is:
ŵsmooth = (XTX + λΓ)−1 XTY
One difference matters. Ridge pulls every weight toward zero. The smoothness penalty does not mind a high weight, as long as its neighbours are high too: a flat filter at any level costs 0. So it leaves the overall level to the data and only removes the wiggles. In matrix terms Γ has an eigenvalue of zero for the all-ones direction, so it cannot fix a problem along that direction on its own. In practice the data pin down the level, and the sum XTX + λΓ can be inverted.
Now compare the two on the same data. In the device each penalty has its own dial.
The blue dashed line is the true filter. The purple curve is the ridge estimate and the warm curve is the smoothness estimate, both from the same 100 samples. Turn each dial, or press the button to set each to its best value.
Illustrative toy, the same data as the hero (N = D = 100, noise 0.5). Both estimates are closed forms solved live. The best λ for each is found by scanning 41 values against the true filter, which a real experiment does not have. The peak height is the largest weight; the true filter’s is 0.96.
At the best settings on this data set, ridge gets within 1.11 of the truth (at λ about 5) and smoothness within 0.36 (at λ about 63): about 3 times closer. The smoothness estimate also keeps the peak (0.87, against a true 0.96) and the small dip, where ridge’s curve is noisier. The reason is plain. The penalty uses extra knowledge, that neighbouring lags behave alike, and the knowledge is true here.
The failure mode is knowledge that is false. Press Smooth too hard. At λ = 1,000 the peak falls to 0.59 and the dip to −0.23: the filter is blurred. The symptom is a smooth but squashed filter. The measurement is a peak far below the height the unpenalised data suggest. The fix is a smaller λ, or a different penalty if the true filter has sharp edges.
Chapter 11
Keep the numbers, the checklist and the core code of Lecture 11 on one page
The lecture has two halves. The first finishes the GLM: a neuron’s rate is f applied to a sum of filtered inputs (stimulus, its own spikes, its neighbours’ spikes), its log-likelihood has one peak for any data, and the inputs become columns of a design matrix. The second half asks what to do when the columns outnumber the samples. Add a penalty on the knobs. Ridge shrinks them. The smoothness penalty smooths them. Both give a closed-form answer by adding one matrix to XTX.
| What | Value | Where |
|---|---|---|
| KL divergence | zero when P = Q; infinite if Q(x) = 0 where P(x) ≠ 0 | Slide 3, Chapter 1 |
| Poisson GLM log-likelihood | y log f(xTw) − f(xTw) + const | Slide 3, Chapter 1 |
| Spike-history GLM | λ(t) = f(k · x(t) + h · yhist(t)) | Slide 5, Chapter 2 |
| Multi-neuron GLM | λi(t) = exp(ki · x(t) + ∑j hij · yj(t)) | Slide 10, Chapter 3 |
| Knobs for 11 cells, 10 lags each | 11 × (10 + 11 × 10) = 1,320 (illustrative) | Chapter 3 |
| Design matrix, 20 stimulus lags, 4 × 10 history lags, 1 constant | 1,000 × 61 (illustrative) | Chapter 5 |
| Convex test | f(ax1 + (1 − a)x2) ≤ af(x1) + (1 − a)f(x2); f″ ≥ 0; Hessian eigenvalues ≥ 0 | Slide 24, Chapter 6 |
| Paninski theorem | log-likelihood concave if f is convex and log-concave: ex, ⌊x⌋, log(1 + ex) | Slide 25, Chapter 7 |
| Simulated example | D = 100 knobs, N = 100 samples | Slide 31, Chapter 8 |
| Maximum likelihood, linear-Gaussian | (XTX)−1 XTY | Notes eq. 3, Chapter 8 |
| Ridge | (XTX + λI)−1 XTY | Notes eq. 7, Chapter 9 |
| Smoothness | (XTX + λΓ)−1 XTY, Γ = DTD | Notes eq. 14, Chapter 10 |
| Our toy, N = D = 100 | distance from truth: no penalty 3.93, ridge 1.11, smoothness 0.36, all zeros 2.55 (illustrative) | Chapters 8 to 10 |
| Term | In one line |
|---|---|
| Conditional intensity | The spike rate at time t given the stimulus and the past spikes. |
| Post-spike filter | Weights on a cell’s own recent spikes: refractory pauses and bursts. |
| Coupling filter | Weights on another cell’s recent spikes: excitation or inhibition. |
| Design matrix | One row per time bin, one column per delayed copy of an input. |
| Convex, concave | A bowl and a hill: no false local optimum. |
| Overfitting | Fitting noise that will not repeat on new data. |
| Penalized likelihood, MAP | Log-likelihood minus λ times a cost on the knobs; the Bayesian name is MAP. |
| Ridge, L2 | Cost is the squared length of the weights. |
| Graph Laplacian | The matrix that writes the sum of squared neighbour differences as wTΓw. |
Both penalised least-squares estimators are one function. The Poisson version adds a gradient of the penalised log-likelihood, which is concave for the exponential, so a plain optimiser finds the one peak.
pythonimport numpy as np def penalty_matrix(D, kind): if kind == "ridge": return np.eye(D) # c(w) = w^T w diff = np.diff(np.eye(D), axis=0) # (D-1, D) difference matrix return diff.T @ diff # Gamma = D^T D, the graph Laplacian def penalized_ls(X, Y, lam, kind="ridge"): # Y: (N,), X: (N, D) G = penalty_matrix(X.shape[1], kind) return np.linalg.solve(X.T @ X + lam * G, X.T @ Y) # (D,) def poisson_nll(w, X, y, lam, G): # exponential nonlinearity: log f(u) = u u = X @ w nll = -(y @ u - np.exp(u).sum()) + lam * (w @ G @ w) grad = -X.T @ (y - np.exp(u)) + 2 * lam * (G @ w) return nll, grad # convex in w: one minimum
With scipy, minimize(poisson_nll, w0, args=(X, y, lam, G), jac=True) returns the fit. Here λ weighs the penalty against the log-likelihood, so its scale differs from the least-squares λ above.
This lesson is the spine of our brain-computer interface series. The next stops show the same ideas at work: point-process GLMs for decoding, latent-variable models for populations, and linear decoders, which meet the same overfitting problem.
For the maths in more depth, read the lessons on generalized linear models, regularization and optimization and bias and variance. Latent models and state estimation come up later in the course, and our lessons on PCA, Gaussian mixtures and EM, hidden Markov models, the Kalman filter and RTS smoothing cover them. Monte Carlo and importance sampling cover the sampling tools.
Credit: this lesson re-teaches Lecture 11 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data, Spring 2024, taught by Jonathan Pillow. See the course site, the lecture slides and his Lecture 11 notes, on which Chapters 8 to 10 rest. The Python tutorial from slide 16 lives at the Pillow lab GitHub. Papers named on the slides: Truccolo and colleagues 2004; Weber and Pillow 2017; Pillow and colleagues 2008; Park, Meister, Huk and Pillow 2014; Hardcastle, Ganguli and Giocomo 2017; Paninski 2004; Uzzell and colleagues 2004.
Now press Present or Teach and explain from memory why 100 knobs and 100 samples is the worst case, and what the ridge term does to XTX. Then go back to the filter at the top, press None and then Smooth, and explain the difference.