Maximum Likelihood and GLMs

One knob sets how strongly a model neuron fires. Turn it until the recorded spikes look least surprising. That setting is the maximum likelihood estimate.

Learn how to fit a model of a neuron to spike counts, why the fit has a one-line answer, and how a small change turns it into the generalized linear model.

Drag θ and watch the purple score. Find the peak by eye, then press Jump to the best θ and check your guess. Then we build, piece by piece, the model, the score, the maths of the peak, and the step from spike counts to spike trains.

You need the idea of a probability (a number from 0 to 1 for how likely something is) and a little calculus (the slope of a curve). We teach the rest from zero.

Based on Lecture 9 of Princeton NEU 560: Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. Lecture slides. Every idea is re-taught here in our own words and devices.

Fit the neuron

1.90

Illustrative: 12 simulated trials from a made-up Poisson neuron (rate = θ × contrast), at contrasts 5, 20 and 35, four trials each. The score is the log-likelihood of Chapter 4 and the best θ is the lecture’s closed form, total spikes divided by total contrast (slide 23). The data are seeded, so each experiment number gives the same trials.

Chapter 0

A Neuron Never Answers the Same Way Twice

See why a model of a neuron must give a probability, and learn the one formula this whole lecture is about

You record one neuron in visual cortex. You show it a striped pattern at a fixed contrast and count its spikes for one second. You get 18. You show the same pattern again. This time you get 23. A third trial gives 15.

Nothing about the stimulus changed. The neuron still answered differently each time. So a model that says “contrast 20 gives 18 spikes” is wrong on the very next trial. A better model says which counts are likely and which are rare.

One formula, three parts

That idea has a standard written form. It is called a regression model, and the lecture writes it as one expression.

P( … )
A probability: a number from 0 to 1 that says how likely something is. The bar reads “given”.
y
The response: what you measure from the brain. Here it is a spike count.
x
The stimulus: what you control or know, such as contrast or a task variable.
θ
The parameters: the settings of the model, like knobs. Fitting a model means finding good settings.

Read it aloud: “the probability of the response, given the stimulus and the knob settings.” For each stimulus it gives a whole list of probabilities, one for every response that could happen. The list is the model’s answer.

The lecture stresses how general this is. The response y can be a spike count, a membrane potential, EEG power, an fMRI BOLD signal (the blood-flow signal of a brain scan) or calcium fluorescence (a glow that rises when a neuron fires). The same formula covers all of them.

A model like this does three jobs. It can simulate: pick a stimulus, and draw a response from the list. It can be fitted: given recorded pairs of x and y, find the θ that suits them. And it can be reversed: given a response, ask which stimuli could have caused it. The chapters ahead take these jobs one at a time, and every job rests on the same formula.

The stimulus x need not be one number. It can be a list, such as contrast, orientation and size together, or a task variable such as the direction of an arm movement. We start with one number, so you can see every step, and add more later.

Encoding and decoding: the same formula, read two ways

When you use the formula to predict the response from the stimulus, you build an encoding model. It describes how the brain turns the world into activity. When you go the other way and guess the stimulus from the response, you do decoding.

Decoding is what a brain-computer interface does. It reads neural activity and asks what the person intended. This lecture builds the encoding side first. Chapter 6 shows how the very same formula gives you the decoder for free.

To feel the two directions, take the lecture’s small table of a neuron that sees houses and faces. Each row lists how likely each spike count is for one stimulus. Try both readings below.

Read the table two ways

Choose Encoding, pick a stimulus, and see the spike counts it can cause. Then choose Decoding, pick a spike count, and see how well each stimulus explains it.

The table is the lecture’s practice problem (slide 41). The row and column readings are our own.

A row always adds up to 1, because the neuron must fire some number of spikes. A column does not add up to 1. It says how well each stimulus explains one count, which is a different job. This difference returns in Chapter 6 and sits at the heart of the word “likelihood”.

The plan, and a word on logistics

The lecture has two parts. First, maximum likelihood for simple models of spike counts. Second, models of whole spike trains, which leads to the generalized linear model (GLM). We follow the same order.

The opening slides cover course business: homework 2 is a Colab notebook due Friday 3/8, and a survey asked about pace, slides versus chalkboard, and class-time problems. The comments asked for more neuroscience examples, more code and more practice problems. This lesson does that: every example is a neuron, and the last chapter works the practice problems.

  1. Three distributions: coin flips, spike counts and noise.
  2. A linear Poisson neuron: the first encoding model.
  3. Maximum likelihood: turn a knob, score the spikes.
  4. Log-likelihood and the closed form: solve for the peak.
  5. A Gaussian neuron: and why it is least squares.
  6. Three readings of one formula: encoding, likelihood, stimulus likelihood.
  7. The generalized linear model: linear, then nonlinear, then noise.
  8. From spike counts to spike trains: the next lecture’s starting point.
  9. Practice problems: Bayes rule and a Bernoulli estimate.
  10. Field guide: numbers, checklist and code.
Which sentence best describes an encoding model of one neuron?

Chapter 1

Three Distributions for Neural Data

Build the Bernoulli, Poisson and Gaussian from a single coin flip, and see why spike counts follow the Poisson

Cut one second of a recording into 1,000 slices of one millisecond each. In every slice the neuron either fires or stays silent. It almost never fires twice in 1 ms, so each slice is a coin flip with a small chance of heads.

That picture gives us all three distributions of the lecture. A distribution is a rule that gives each possible outcome a probability. We need one for a single slice, one for the count over the whole second, and one for smooth signals like voltage.

Bernoulli: one coin flip

The Bernoulli distribution describes one flip. The outcome x is 1 for a spike and 0 for silence. The parameter p is the probability of a spike. The lecture writes its rule in one line.

P(x | p)
The probability of outcome x when the spike chance is p.
px
Equals p when x = 1 and equals 1 when x = 0. It switches on only for a spike.
(1 − p)1 − x
Equals 1 − p when x = 0 and equals 1 when x = 1. It switches on only for silence.

Check it with p = 0.1. A spike (x = 1) gives 0.11 × 0.90 = 0.1. Silence (x = 0) gives 0.10 × 0.91 = 0.9. The two add up to 1, as they must. The exponents are just a switch that picks the right factor.

Poisson: count the heads in a second

Now count the spikes over all 1,000 slices. With a 0.01 chance per slice, you expect 1,000 × 0.01 = 10 spikes, but a given second may give 7 or 13. The lecture’s key step is to let the slices get thinner and thinner.

Make n slices and set the chance per slice to p. Let n grow without limit and p shrink toward zero, but keep the product λ = n × p fixed. The slide gives three cases that share λ = 1: 10 coins with p = 0.1, 100 coins with p = 0.01, and 1,000 coins with p = 0.001. In the limit the count follows the Poisson distribution.

P(k | λ)
The probability of exactly k spikes when the average count is λ.
λk
Grows with k when λ is large, so big counts become likelier as the rate rises.
e−λ
The chance of seeing no spikes at all. It shrinks fast as λ grows.
k!
k factorial: 1 × 2 × … × k. It makes sure all the probabilities add up to 1.

Work it by hand for λ = 2. A count of 0 has probability e−2 = 0.135. A count of 1 has 2 × 0.135 = 0.271. A count of 2 has 4 × 0.135 / 2 = 0.271. A count of 3 has 8 × 0.135 / 6 = 0.180. The most likely counts sit near 2, and the tail thins out to the right.

The Poisson has one parameter, λ, and that one number is both the mean (the average count) and the variance (how far counts spread around it). Slide 11 marks this: mean = variance. Hold that fact. It decides how neurons with a high rate differ from neurons with a low rate. Slide the knobs below to watch the coin flips turn into a Poisson.

Thin the slices, keep λ

Blue bars are the spike counts from n coin flips. The warm dots are the Poisson with the same average. Pick more coins and watch the dots land on the bars.

3.0

Exact probabilities, computed live in your browser: the binomial for n coins with chance λ/n each, and the Poisson formula above. Only the choice of λ values is illustrative.

At 10 coins the bars and dots still disagree a little, since 10 coins can never give more than 10 heads. At 1,000 coins the two are almost the same picture. That is the limit the slide describes.

Gaussian: the bell curve

The third distribution is for smooth quantities such as a membrane voltage or a blood-flow signal. The Gaussian is the bell curve, with a centre μ (the mean) and a width σ (the standard deviation, whose square is the variance).

The slide labels it “of CLT fame”. The central limit theorem says that adding up many small independent effects gives a bell curve, whatever the effects looked like alone. A voltage that sums many small inputs is a natural fit, so the lecture uses the bell curve for smooth signals.

Its formula is P(x | μ, σ2) = (1 / √(2πσ2)) × e−(x − μ)2 / 2σ2. At x = μ the exponent is zero, so the curve peaks at 0.399 / σ. A narrow curve is tall, a wide one is low, and the area under it is always 1.

One difference matters later. The Gaussian has a separate width σ, free to choose. The Poisson has none: its spread is tied to its mean. Chapter 5 shows what that changes when you fit a neuron.

Here is the failure to watch for. Suppose you record 50 trials at one stimulus and the counts spread out more than the Poisson allows. The symptom is a variance well above the mean. Measure it with the Fano factor, the variance divided by the mean: a true Poisson gives 1. If you find 2.5, the Poisson model is too tight, and you need a distribution with a free width, such as the Gaussian below, or a richer noise model.

A neuron has a 0.003 chance of firing in each of 1,000 slices, and slices are independent. About how many spikes do you expect, and how much do counts vary?

Chapter 2

The Linear Poisson Neuron

Write the simplest encoding model, a rate that grows with contrast and a Poisson count around it, and see its noise grow too

Turn up the contrast of a pattern, and a neuron in visual cortex usually fires more. Double the contrast and it often fires about twice as much. The lecture takes that observation and turns it into the simplest encoding model it can.

The model has two layers. The first layer sets an average. The second layer adds the noise around it. We take them in that order.

Layer one: the rate

The spike rate, written λ, is the average spike count you would get if you ran the same trial many times. The lecture sets it proportional to the stimulus: λ = θx. Here x is the contrast and θ is one number that says how strongly this neuron responds.

This is a linear model, because the rate is a straight line in x. It also passes through zero: no contrast, no spikes. Take θ = 1. Then contrast 5 gives a rate of 5, contrast 20 gives 20 and contrast 35 gives 35. With θ = 0.5 the same contrasts give 2.5, 10 and 17.5.

Layer two: the noise

A rate of 20 does not mean exactly 20 spikes on every trial. The actual spike count y varies. The lecture assumes the count follows a Poisson distribution whose average is the rate. In symbols, y ~ Poisson(λ), read as “y is drawn from a Poisson with mean λ”.

Put the two layers together. Replace λ by θx in the Poisson formula of Chapter 1. The result is the full encoding model, with the three named parts in the lecture’s slide: the parameter, the stimulus, and the count.

P(y | x, θ)
The probability of seeing y spikes at contrast x when the neuron’s strength is θ.
y!
y factorial. It does not depend on θ, a fact that helps in Chapter 4.
(θx)y
The rate raised to the observed count. A higher rate favours higher counts.
e−θx
The chance of silence at that rate. A higher rate makes silence rarer.

Work one case. Take θ = 1 and x = 5, so λ = 5. The chance of exactly 5 spikes is 55 × e−5 / 5! = 3,125 × 0.00674 / 120 = 0.175. The chance of 10 spikes is far smaller: 510 × e−5 / 10! = 0.018. The model prefers counts near the rate and punishes the rest.

A picture of the whole model

The lecture draws the model as a grey image. The horizontal axis is contrast x. The vertical axis is spike count y. Brightness at (x, y) is P(y | x, θ). Each vertical column is one distribution, the answer for one stimulus.

The bright band rises along a line, because the average count is θx. The band also gets wider to the right. That is the “mean = variance” fact at work: the spread (standard deviation) is √(θx). At θ = 1 it is about 2.2 spikes at contrast 5, 4.5 at contrast 20, and 5.9 at contrast 35.

The slides cut the image at x = 5, 20 and 35. Each cut gives a Poisson bump centred higher and wider than the last. The device below lets you place the cut anywhere and draw trials from it.

Cut the model at a contrast

Left: brightness is the probability of each spike count at each contrast. Right: the cut at your contrast. Drag the sliders, then press Draw 10 trials to scatter blue dots from the model.

20
1.0

The brightness is the exact Poisson probability P(y | x, θ). The dots are simulated draws from it, different on every press. The model and its two layers are the lecture’s (slides 11 to 14); the sliders are ours.

Notice what the cut shows when you move x to the right: the bump moves up and flattens. A high-contrast trial is less predictable in absolute terms, yet its centre is easier to see against zero.

Signal and noise from one number

The spread grows, but more slowly than the average. The spread is √λ and the average is λ, so the ratio of average to spread is also √λ. At a rate of 5 that ratio is 2.2. At a rate of 20 it is 4.5. At a rate of 35 it is 5.9.

So a strong response is noisier in spikes yet cleaner in proportion. This is why a neuron is a better messenger at high contrast. It also hints at decoding. Suppose you see a count of 20. A neuron with θ = 1 could have produced that at contrast 20, but also at contrast 16 or 25, only less often. One count never pins the stimulus down. It only makes some stimuli more believable than others.

Where this model breaks

The symptom: your neuron fires a few spikes with no stimulus at all, but the model insists on silence. Measure it. At x = 0 the rate is θ × 0 = 0, so P(y = 1) = 0. A single spontaneous spike then has probability zero, and its log score, which Chapter 4 introduces, is minus infinity. No setting of θ can ever repair that. The fix is a baseline term: λ = θx + b. Chapter 7 will make this the first step toward the full GLM. For the encoding of firing rates over time see Spikes to Rates.

For a linear Poisson neuron with θ = 2, what are the average and the variance of the spike count at contrast x = 10?

Chapter 3

Maximum Likelihood: Score the Spikes You Saw

Turn the model’s knob until the recorded spike counts become as likely as possible

You have run an experiment. On twelve trials you showed a pattern at some contrast x and counted the spikes y. You trust the linear Poisson model of Chapter 2, but the strength θ is unknown. Which value of θ should you report?

The lecture’s answer is the oldest idea in statistics. Pick the θ under which the data you actually saw would have been most probable. If one setting makes your recording look ordinary and another makes it look like a miracle, believe the first.

One trial, then all of them

For one trial, the model gives a probability: P(yi | xi, θ). The lecture calls it the single-trial probability. Trial i has contrast xi and count yi. We want the probability of all the counts together, written P(Y | X, θ). Here Y is the list of all counts and X the list of all contrasts.

To combine the trials we need an assumption, and the slide asks for it by name. The assumption is conditional independence across trials: once you know the stimulus and θ, one trial’s count says nothing about another’s. Then probabilities multiply, as the chance of two fair coins both landing heads is 0.5 × 0.5.

P(Y | X, θ)
The probability of the whole data set: every count, given every stimulus and the knob setting.
∏i=1N
Multiply the terms for trial 1 up to trial N. A capital pi means “product”, as a capital sigma means “sum”.
P(yi | xi, θ)
The single-trial probability from the encoding model of Chapter 2.

The same expression, a new job

Now a second question from the slide: when do we call P(Y | X, θ) a likelihood? The answer is a change of viewpoint. When you hold θ fixed and ask how probable different data are, it is a probability. When you hold the data fixed at what you recorded and vary θ, it is a likelihood.

Nothing in the formula changes. Only the question changes. A likelihood is not the probability that θ is correct. It only scores how well each θ explains the data you have.

A worked score, by hand

Take three trials. Contrasts are 5, 20 and 35 and the counts are 4, 22 and 31. First try θ = 1, so the rates are 5, 20 and 35. The single-trial probabilities are 0.175, 0.077 and 0.056. Their product is 0.175 × 0.077 × 0.056 = 7.6 × 10−4.

Now try θ = 0.5, with rates 2.5, 10 and 17.5. The first count is still fine (0.134). But a count of 22 against a rate of 10 has probability 0.0004, and 31 against 17.5 has 0.0010. The product is 5.6 × 10−8.

The data are about 13,500 times more probable under θ = 1 than under θ = 0.5. That ratio is the whole argument for choosing θ = 1. The lecture says you could find the best value “by turning a knob”, which is what slides 17 to 19 show for θ = 1.5, 1 and 0.5. Do exactly that below, trial by trial.

Turn the knob, trial by trial

Each purple bar is the probability the model gives one recorded trial. The contrast and count are under each bar. Move θ and watch the bars, and the product of all twelve, change.

1.50

Illustrative data: 12 simulated trials from a made-up Poisson neuron, at contrasts 5, 20 and 35, four trials each. The three chip values are the knob settings on slides 17 to 19. All probabilities are exact Poisson values computed in your browser.

Watch the red outline. It marks the trial the model finds hardest to believe at the current setting. Move θ far from the best value in either direction, and the outline lands on a contrast-35 trial. The reason: a small error in θ is multiplied by x, so the predicted rate is off by the most at high contrast. High-contrast trials therefore have the loudest say in the fit.

Now zoom out. Imagine sliding θ from 0 to 3 and plotting the product at each stop. You get a hill. Its top is the maximum likelihood estimate, written θ̂ML (“theta hat”, the hat marking an estimate made from data). The hero at the top of this page draws that hill for you. The rest of the lecture is about finding the top without turning a knob by hand.

Each trial pulls on the hill. A trial at contrast 35 has a big count, so it pulls the peak toward a large θ. A trial at contrast 5 with a small count pulls it down. The peak is where those pulls balance.

Two ways the recipe can fail

The first failure is the independence assumption. Suppose the neuron adapts: after a strong stimulus it fires less on the next trial. The symptom is a pattern in the leftovers, where a high count tends to follow a high count or a low one follows a high one. Measure it by correlating each trial’s count error with the next trial’s. If the correlation is far from zero, shuffle the trial order in the experiment or add the previous trial to the model.

The second failure is numerical. Twelve probabilities near 0.1 multiply to something like 10−12. A thousand trials would give 10−1000, which a computer rounds to exactly zero. Every θ would then score zero and you could not tell them apart. The next chapter fixes this with one move: take the logarithm.

When do we call P(Y | X, θ) a likelihood?

Chapter 4

Take the Log, Find the Peak

Turn the product into a sum, set the slope to zero, and get the best θ in one line, then see why more data makes it better

Chapter 3 ended with a hill: the likelihood of the data at each value of θ. Its top is the answer. You could climb it by trial and error, but the Poisson neuron lets us do far better. We can write the top down in one line.

Three moves get us there. Take a logarithm so the product becomes a sum. Differentiate to find where the slope is zero. Solve. The lecture’s recipe on slide 39 is exactly these three steps.

Move 1: take the log

A logarithm turns multiplication into addition: log(a × b) = log a + log b. It also never changes which θ is highest, because the log is an increasing function. A taller hill stays taller after you take its log. So we maximise the log-likelihood instead.

Start from the Poisson likelihood. Take the log of each trial’s term and add them up.

log P(Y | X, θ)
The log-likelihood: the score of θ for the whole data set.
log θ · ∑ yi
The log of θ, times the total number of spikes. It rewards a large θ when there are many spikes.
θ ∑ xi
θ times the total contrast. It punishes a large θ when much contrast was shown, since a strong neuron should have fired even more.
c
Everything that does not contain θ, such as the log of y!. It shifts the curve up or down and cannot move the peak, so we ignore it.

Each trial contributed yi log(θxi) − θxi − log yi!. Split log(θxi) into log θ + log xi. The log xi and log yi! parts have no θ in them, so they go into c. What is left is the line above. The data enter only through two totals: the total spikes and the total contrast.

Moves 2 and 3: differentiate and solve

At the top of a smooth hill the slope is zero. The slope of the log-likelihood with respect to θ is (1 / θ) ∑ yi − ∑ xi. Set it to zero.

(1 / θ) ∑ yi − ∑ xi = 0   ⇒   θ̂ML = ∑ yi / ∑ xi

The best strength is the total number of spikes divided by the total contrast. In words: spikes per unit of contrast, averaged over the whole experiment. It is a very natural answer, and the maths confirms it. Take the three trials of Chapter 3, with counts 4, 22 and 31 at contrasts 5, 20 and 35. The totals are 57 spikes and 60 units of contrast, so θ̂ = 57 / 60 = 0.95.

Check that it is a peak and not a valley. The second slope is −(∑ yi) / θ2, which is negative for any positive count. A negative second slope means the curve bends downward, so the point is a top.

The lecture says this closed form exists because the Poisson belongs to the exponential family, a large group of distributions (including the Gaussian and Bernoulli) whose log has this friendly shape. When a model falls outside that group, there may be no closed form. Then you climb the hill in small steps, a method called gradient ascent: move θ a little in the direction of the slope, and repeat.

Does the answer improve with more data?

Slide 24 names two good properties of the maximum likelihood estimator. It is consistent: as the number of trials grows without limit, θ̂ converges to the true θ. It is efficient: it converges as fast as any estimator can, reaching the smallest possible error for large data sets. Slide 39 words the second as saturating the Fisher information, a number that measures how sharply the log-likelihood peaks.

For our neuron the sharpness is easy to state. The typical error of θ̂ is √(θ / ∑ xi). With θ = 1 and twelve trials (total contrast 240) that is 0.065. With 120 trials it falls to 0.02. Try it. Each press runs 25 fresh experiments and shows where each estimate landed.

Run the experiment again

The purple curve is the log-likelihood of one experiment, lifted so its peak sits at zero. The green mark is its peak. The teal ticks are the peaks of 25 other experiments. The true θ is 1.00.

12

Illustrative: simulated Poisson neuron with true θ = 1, contrasts 5, 20 and 35 in turn. The estimate is the lecture’s closed form, total spikes over total contrast. The predicted error, the square root of θ over total contrast, is the standard result for this model.

Slide the number of trials up. The curve gets narrower, the ticks crowd around 1.00, and the observed spread follows the formula closely. That is consistency and efficiency, seen rather than claimed.

When the recipe needs care

The symptom: you try to apply the three moves to a model with a nonlinearity inside, and the equation “slope = 0” has no tidy solution. Measure by trying to isolate θ. If you cannot, use gradient ascent. The check that keeps you honest is simple: the log-likelihood must rise at every step. If it falls, your step is too big, so halve it. Chapter 7 returns to this, and the lesson on Generalized Linear Models shows the full machinery.

For a linear Poisson neuron, what is the maximum likelihood estimate of θ?

Chapter 5

A Gaussian Neuron Is Least Squares

Keep the straight-line rate, swap the noise layer for a bell curve, and watch maximum likelihood turn into a familiar fit

Not every response is a count of spikes. A membrane voltage, an EEG power value or a calcium signal varies smoothly. Even spike counts, when they are large, look like a bell curve. For these, the lecture swaps the second layer of the model.

The first layer stays. The rate is still a straight line through zero: μ = θx. The second layer is now a Gaussian with that mean and a fixed width: y ~ N(μ, σ2). Slide 26 notes the key difference: all slices have the same width. The Poisson fan widened with contrast. This band does not.

The same three moves

Chapter 4 gave us a recipe. Apply it. The log of a Gaussian density is −(y − μ)2 / 2σ2 minus a term that has no θ in it. Add that up over trials, with μi = θxi.

log P(Y | X, θ)
The log-likelihood of the whole data set under the Gaussian neuron.
∑ (yi − θxi)2
The sum of squared errors: how far each count is from the line, squared, then added.
2σ2
A fixed scale. It stretches the score but cannot change where the peak is.
c
Terms with no θ in them, which only shift the curve.

The minus sign matters. To make the log-likelihood as large as possible, make the sum of squared errors as small as possible. Maximum likelihood with Gaussian noise is least squares. The scale σ2 drops out of the answer.

Differentiate and set to zero: ∑ (yi − θxi) xi / σ2 = 0. Solve for θ. The slide gives the result, and labels it the “least squares regression” solution.

θ̂ML = ∑ yixi / ∑ xi2   (Poisson: ∑ yi / ∑ xi)

For the full story of fitting lines by least squares, see the Least Squares lecture and the lesson on Linear Regression.

Two weighted averages

The two estimators look alike, so compare them on the three trials of Chapter 3. The contrasts are 5, 20 and 35 and the counts are 4, 22 and 31. Each trial alone suggests a strength y / x of 0.80, 1.10 and 0.89.

The Poisson answer is 57 / 60 = 0.950. The Gaussian answer is (20 + 440 + 1,085) / (25 + 400 + 1,225) = 1,545 / 1,650 = 0.936. They differ. Why?

Both are weighted averages of the single-trial ratios. The Poisson weights trial i by xi. The Gaussian weights it by xi2. So the Gaussian trusts high-contrast trials much more. That fits its own noise story: with equal noise everywhere, a trial at high contrast gives a cleaner ratio, since the same noise is divided by a bigger x. The Poisson already has more noise at high contrast, so it trusts those trials less.

Test this on the device below. It shows the fit of each model and its noise band. Raise the last trial’s count to see how each estimator reacts to a surprising high-contrast trial.

Fit the same spikes two ways

Pick a noise model. The line is its best fit, and the shaded band is two standard deviations of its noise. Then raise the count of the last trial and watch the line move.

29

Illustrative data: 12 simulated trials from a Poisson neuron with θ = 1; the last one is yours to change. Both fits use the lecture’s closed forms. The Gaussian width is the usual maximum likelihood value, the root mean squared error of the fit.

The Gaussian line swings more. One large count at contrast 35 carries weight 352 = 1,225 in the Gaussian sum and only 35 in the Poisson. Also look at the bands. At low contrast the Gaussian band can dip below zero, a count nobody can record. The Poisson band never does.

Estimating the width too

The width σ is a second knob. Maximum likelihood handles it the same way: differentiate with respect to σ2 and set to zero. The answer is the average squared error, σ̂2 = (1 / N) ∑ (yi − θ̂xi)2.

Work it for the three trials with θ̂ = 0.936. The predictions are 4.68, 18.72 and 32.76. The errors are −0.68, 3.28 and −1.76. Their squares are 0.46, 10.76 and 3.10, which sum to 14.32. Divide by 3 to get 4.77. The width is the square root, σ̂ = 2.2 spikes. The device above reports this number for the full twelve trials.

Choosing the noise on purpose

The symptom of a wrong noise layer is easy to see. Plot the leftover error of each trial against contrast. If the leftovers fan out to the right, as they do for spikes, a fixed-width Gaussian is wrong: it overweights the noisy high-contrast trials and underweights the quiet ones. Measure the fan by computing the variance of the leftovers in a few contrast bins. The fix is either to use the Poisson, or to take the square root of the counts first, a standard trick that makes the variance nearly constant.

Use the Gaussian for smooth, symmetric signals such as voltage or BOLD. Use the Poisson for small spike counts. Both fit in one framework, and the next chapters build on that.

Why does maximum likelihood with Gaussian noise give the least squares solution?

Chapter 6

One Formula, Three Readings

See that P(y | x, θ) is an encoding distribution, a likelihood or a stimulus likelihood, depending on which letter you let move

The lecture pauses for a quick quiz. It points at P(y | x, θ) and says the formula can be seen as a function of y, of x or of θ. What is it in each case? The answers sort out words we have been using loosely.

Think of the formula as a machine with three dials: the spike count y, the stimulus x and the strength θ. Turn one dial, hold the other two still, and you get a curve. Three dials give three curves, and each has its own name and its own job.

The three answers

First, as a function of y, with x and θ fixed. This is the encoding distribution: a probability distribution over spike counts. It is what Chapter 2 drew as one cut through the grey image. Its bars add up to 1.

Second, as a function of θ, with the data fixed. This is the likelihood function of Chapter 3: how probable the recorded data are under each model setting. It is the hill we climbed in Chapter 4.

Third, as a function of x, with the spike count and θ fixed. The lecture calls this the stimulus likelihood function, and says it is useful for maximum likelihood decoding. Here the model is already fitted and you have just observed a count. You ask which stimulus makes that count most probable. The top of the curve is x̂ML, your best guess of the stimulus.

The peak is in the same place, the area is not

Work the Poisson case with θ = 1 and an observed count of 20. As a function of x, the formula is x20 e−x / 20!. The log of it is 20 log x − x, with slope 20 / x − 1. That is zero at x = 20. So the decoded stimulus is x̂ = 20. In general x̂ML = y / θ.

The curve is not a sharp spike. At x = 20 it reaches 0.0888. At x = 15 it is 0.0418, and at x = 25 it is 0.0519. A count of 20 is quite compatible with a stimulus a few units away. A decoder that reports only x̂ hides that width.

Now the key difference. The area under each curve is not the same. Sum the bars over y, and you get exactly 1. Add up the curve over θ, and you get 1 / x, which is 0.05 at x = 20. Add it up over x, and you get 1 / θ. A likelihood is not a probability distribution over its argument. It is a score. Use the device below to cut the formula all three ways and watch the area.

Turn one dial at a time

Choose which letter moves. The curve is P(y | x, θ) as that letter varies. The other two stay at the values of their sliders.

20
20
1.0

Exact Poisson probabilities, summed (for y) or integrated numerically (for θ and x) in your browser. The three readings are the lecture’s quiz (slides 29 to 31); the Poisson numbers are ours.

Pick “Vary x” and set θ to 0.5. The peak jumps to 40, since a weaker neuron needs a stronger stimulus to produce 20 spikes. The area is now 2. The curve is a score, not a probability.

Where decoding goes next

How wide is that stimulus likelihood? For a Poisson neuron, the count y has a spread of about √y, and the stimulus estimate is y / θ. So the decoded stimulus is uncertain by about √y / θ. At y = 20 and θ = 1 that is 4.5 contrast units. A stronger neuron, with θ = 2, halves the uncertainty in stimulus units, since the same noise is spread over a steeper line.

A second neuron adds a second curve. If the neurons are conditionally independent given the stimulus, the same rule as Chapter 3 applies: multiply their stimulus likelihoods, or add their logs. The product is narrower than either curve alone. This is how a population of neurons decodes better than one.

Recall the house and face table of Chapter 0. A column of it is a stimulus likelihood, and it did not add up to 1. Maximum likelihood decoding picks the stimulus whose column entry is largest. If you also know how common each stimulus is, Bayes rule combines that prior with the likelihood. Chapter 9 works it.

This is the seed of every neural decoder. A brain-computer interface fits an encoding model during calibration, then reads the likelihood backwards during use. The lessons on Linear Decoders and Point-Process GLMs build on exactly this step.

The failure to avoid is a bookkeeping slip. Someone plots the likelihood over θ, sees a peak of 0.0888 and reports “an 8.9% chance that θ is right”. The symptom is a number that sounds like a probability of θ but is not. Measure the area: it is 0.05, not 1. The fix is to remember that a likelihood scores the data. To get the probability of θ, you need a prior and Bayes rule.

A model is already fitted and you observe a spike count. You plot P(y | x, θ) against x. What is this curve, and what is it good for?

Chapter 7

The Generalized Linear Model

Add one nonlinear step between the straight line and the noise, and fit neurons that no straight line can describe

The lecture now hands you an unknown neuron and asks: be the computational neuroscientist, what model would you use? You scatter its spike counts against contrast, from −25 to +25. For negative contrast the counts sit at zero. For positive contrast they climb faster and faster, to about 75 spikes at +25.

Try the model you know. A line through zero predicts negative rates for negative contrast, which cannot happen. A line that stays at zero until the stimulus turns on would miss the steady climb. The data bend upward. We need a curve, and still want the fitting to stay easy.

Linear, then nonlinear, then noise

The lecture’s answer is three boxes in a row. The first box is linear: it takes the stimulus x and gives one number, θ · x. The second box is a nonlinearity f: it bends that number into a valid rate, λ = f(θ · x). The third box is the noise: it draws a spike count around the rate, such as y ~ Poisson(λ). Slide 33 says it plainly: this is a generalized linear model.

y
The observed spike count.
Poisson( … )
The noise box. The lecture calls this the “distribution function”, and it must come from the exponential family.
f
The nonlinearity. It turns any number into a rate that is never negative.
θ · x
A dot product: multiply each stimulus feature by its weight and add. It squeezes a long list of features to one number.

The stimulus can be many numbers. Let x = (contrast, a second feature) = (10, 5) and θ = (0.4, −0.2). Then θ · x = 0.4 × 10 + (−0.2) × 5 = 3.0. Pass 3.0 through f = exp and the rate is e3 = 20.1 spikes. The linear box acts as the lecture’s “dimensionality reduction”, because many inputs become one score.

The slides name the parts. The parameter is θ. The link function is the inverse of f, written f−1: it maps the rate back to the linear score. The noise box is the distribution function. People often say the model uses a “log link” when f = exp, because the log undoes the exp.

Two models, one abbreviation

Slide 34 warns about a naming trap. GLM can mean the General Linear Model: linear box, then noise, with no nonlinearity. For a Gaussian this is y = θ · x + ε, ordinary regression, which Chapter 5 fitted. Or it can mean the Generalized Linear Model of Nelder (1972), which adds the nonlinearity. The slide strikes the word “linear” out of the second name, because the whole input-to-rate map now bends. Only the first box stays linear. The word “generalized” carries the difference.

Fitting it: no closed form, but still easy

Take f = exp and Poisson noise. The log-likelihood of one trial is y θ x − eθx − log y!. Differentiate and sum over trials: ∑ (yi − eθxi) xi = 0. You cannot isolate θ from that, so the lecture’s alternative applies: gradient ascent.

Notice the shape of the equation. It says “error times input, summed, equals zero”, where the error is the count minus the predicted rate. Chapter 5’s Gaussian condition had the same shape. This is not a coincidence: it is the signature of the exponential family. And with f = exp the log-likelihood has one peak, so gradient ascent cannot get stuck on a false hill.

Use the device to bend the model yourself. It holds 44 simulated trials of an unknown neuron. Pick a nonlinearity, then slide θ to find the best fit. The device scores each choice by log-likelihood.

Choose the nonlinearity

Pick f, then drag θ to make the warm curve pass through the blue spike counts, or press Jump to the best θ. A higher score is a better fit.

0.100

Illustrative data: 44 trials, four at each contrast from −25 to +25 in steps of 5, simulated from a Poisson neuron with rate e0.165x, so the exponential is the true model here. The best θ for each choice is found by searching a grid for the highest log-likelihood. The rectified line floors the rate at 0.01 so a surprise spike is not infinitely unlikely.

The exponential wins by a wide margin, since it generated the data. Its best score is −66.0. The rectified line and the softplus reach only −152.2 and −150.9, about 85 points lower on the log scale. Both are close to straight for the large inputs here, so neither rises fast enough at +25 and still stays near zero for negative contrast. The data chose the model, which is the point of the lecture.

What can go wrong with f

The symptom is a fit that returns not-a-number or an infinite score. Measure it: compute the smallest predicted rate over all trials. If any rate is zero or negative while a count is above zero, the Poisson probability is zero and its log is minus infinity. The fix is to choose f so that it stays positive (exp, softplus), or to add a small floor.

A second symptom is subtler. The exponential grows without bound, so a stimulus far outside your data predicts an absurd rate, such as e0.165×60 = 20,000 spikes. Measure the predicted rate on the extremes of your stimulus range, and compare it with the highest rate the neuron could physically reach. If it is far above, use softplus or a saturating f. For a fuller treatment of these models see Generalized Linear Models.

What separates a Generalized Linear Model from a General Linear Model?

Chapter 8

From Spike Counts to Spike Trains

Let the stimulus change every moment, give the neuron a memory of the recent past, and see why the first model to try fails

So far each trial gave one count for one stimulus. A real neuron lives in a world that never holds still. Contrast flickers. Sounds rise and fall. Watch one second of it, cut into 1 ms bins, and you see a long row of zeros with the occasional 1. That row is a spike train.

The lecture ends its main content with a preview: from spike counts to spike trains. The stimulus is now a signal that changes with time, and the response is a train. At every moment t we want a model of how the neuron responds. Slide 38 gives the first idea.

A window on the past

A neuron does not react only to the stimulus at this instant. It reacts to what happened over the last few tens of milliseconds. So the input at time t is not one number. It is a list, the vector stimulus xt, made of the most recent stimulus values: now, one bin ago, two bins ago, and so on.

The model needs one weight for each of those moments. The list of weights is the linear filter, written k. Where k is large and positive, the neuron likes the stimulus at that lag. Where k is negative, it dislikes it. The linear box of Chapter 7 does the work: it takes the dot product of the filter with the window.

yt
The response at time t. For a spike train this is 0 or 1 spikes in a small bin.
k · xt
The filter dotted with the window of recent stimulus. It slides along time, one bin at a time.
noise
Gaussian noise with mean zero and variance σ2, as in Chapter 5.

Work one bin with a window of three. Let xt = (0.5, −1.0, 2.0), most recent first, and k = (1.0, 0.5, −0.5). Then k · xt = 1.0 × 0.5 + 0.5 × (−1.0) + (−0.5) × 2.0 = 0.5 − 0.5 − 1.0 = −1.0. A real filter has dozens of taps, but the arithmetic is the same.

Here is the data flow. Say you record T = 10,000 bins and let the window span K = 100 bins. Stack the windows into a table X with one row per bin and one column per lag, so X has shape [10000, 100]. The filter k has shape [100]. The response y has shape [10000]. The model says y is about X k. The least squares answer of Chapter 5, written for many weights at once, is k̂ = (XTX)−1 XTy.

The lecture calls this the first idea: a linear-Gaussian model. It is just Chapter 5, with a window in place of one contrast value. Everything you learned there still holds: the best filter is the least squares solution, and the Gaussian noise width falls out of the leftover errors.

Why the first idea is not enough

Look at the response row on slide 38: zeros, with a few spikes. The data are counts of 0 or 1. A Gaussian cannot give that. Its output is continuous, so it will write 0.37 or −0.8. A neuron cannot fire a negative spike or a third of one.

The Chapter 7 fix applies directly. Keep the filter. Pass its output through a nonlinearity that stays positive, and draw the response from a Poisson. The result is yt ~ Poisson(f(k · xt)). This is the GLM for spike trains, and it is where the next stage of the course picks up. For the full version, with spike history and coupling between neurons, see Point-Process GLMs.

See the difference yourself. The device builds a filter output from a made-up stimulus. Then it asks each model to write a response train. Look for impossible values.

Let each model write a spike train

Choose a filter shape and a model. The warm line is the model’s predicted average response. The blue marks are one response it wrote. Red marks are values a spike train can never take. Press Write a new response for another draw.

1.5

Illustrative: a made-up stimulus of 60 bins, an 8-tap filter and a baseline of −1.5 inside the exponential, all chosen for a clear picture. The linear-Gaussian model and the filter idea are slide 38’s; the Poisson GLM is Chapter 7’s nonlinearity and noise applied to the same filter output.

With the Gaussian model, many bins come out negative or fractional, shown in red. The GLM never does that. Every bin is a whole number, and the average response, exp of the filter output, stays above zero. Raise the gain and the GLM fires in bursts where the filter output is high, as a real neuron would.

A check on the filter length

The symptom of a window that is too short is a model that reacts too early or too late. Measure it by looking at the last few weights of the fitted filter. If the final tap is still large, the neuron cares about lags beyond your window. Lengthen the window until the tail sits near zero. If it is too long you waste data: each extra tap is another number to estimate. A check that the tail settles is the fix, and it is the same guard against too many parameters that Chapter 4 showed with the number of trials.

Why does the linear-Gaussian model fit a spike train badly, and what does the GLM change?

Chapter 9

Practice: Decode a Neuron and Estimate a Coin

Work the lecture’s extra problems from scratch: Bayes rule on a house and face neuron, and the maximum likelihood of a coin

The lecture closes with a summary slide and two sets of practice problems. The summary is a recap of everything so far: regression models P(y | x, θ), the three distributions, the three-step recipe for maximum likelihood, consistency and efficiency, and the two kinds of GLM. Chapter 10 collects it. Here we do the problems, because the survey asked for more of them.

Problem set 1: Bayes rule on a neuron

Take the house and face table from Chapter 0. The columns are spike counts 0 to 4. The house row is 0.1, 0.1, 0.3, 0.4, 0.1. The face row is 0.1, 0.4, 0.3, 0.2, 0. The two stimuli are equally common: P(house) = P(face) = 0.5. Five questions follow.

1. Which does the neuron prefer? Compare the average counts. For houses it is 0 × 0.1 + 1 × 0.1 + 2 × 0.3 + 3 × 0.4 + 4 × 0.1 = 2.3 spikes. For faces it is 0.4 + 0.6 + 0.6 + 0 = 1.6 spikes. It responds more to houses.

Questions 2 to 5 decode. Given a count, which stimulus was more likely? This needs Bayes rule, the recipe for turning the likelihood into the probability of the stimulus. In the lecture’s notation: P(x | y) = P(y | x) P(x) / P(y). Here P(x) is the prior, how common each stimulus is before you look at the spikes. P(x | y) is the posterior, your belief after.

2. Two spikes. Houses give P(2 | house) = 0.3 and faces give 0.3. With equal priors the products are equal (0.15 each), so the posterior is 0.5 and 0.5. Two spikes cannot tell the stimuli apart.

3. Three spikes. The products are 0.4 × 0.5 = 0.20 for a house and 0.2 × 0.5 = 0.10 for a face. Their total is 0.30, which is P(y). Dividing gives P(house | 3) = 0.667 and P(face | 3) = 0.333. A house is the best guess.

4. Any count that is certain? Yes. Four spikes. P(4 | face) = 0, so the face product is zero and the posterior for a house is 1.00.

5. Change the prior. Now P(house) = 0.2 and P(face) = 0.8. With three spikes the products are 0.4 × 0.2 = 0.08 and 0.2 × 0.8 = 0.16. The posterior becomes 0.333 for a house and 0.667 for a face. The decision flips. A strong prior can outvote a modest likelihood. Try every case below.

Decode with a prior

Pick the spike count you observed, then slide the prior. The bars show the likelihood, the prior and the posterior for a house and a face.

0.50

The likelihood table and the priors for questions 3 and 5 are the lecture’s practice problem (slide 41). The prior slider is ours.

The failure to watch here is overconfidence. The face probability for four spikes is exactly 0, so four spikes certify a house. In a real experiment that zero may only mean you have not yet seen a face fire four spikes. Measure how many trials stand behind each table cell. If a cell has few, add a small pseudo-count to every cell, so no stimulus is ever ruled out for ever.

Problem set 2: maximum likelihood of a coin

Now the Bernoulli problem. You flip a coin N times and get y1 to yN, each 0 or 1. Find the maximum likelihood estimate of p, using the three moves of Chapter 4. The likelihood is ∏ pyi (1 − p)1 − yi.

Move 1, log. Let k = ∑ yi be the number of heads. The log-likelihood is k log p + (N − k) log(1 − p). Move 2, differentiate. The slope is k / p − (N − k) / (1 − p). Move 3, solve. Set it to zero: k (1 − p) = (N − k) p, so k = N p, and p̂ = k / N. The best estimate is the fraction of heads. With 3 heads in 10 flips, p̂ = 0.3.

Check it with scores. At p = 0.3 the log-likelihood is 3 log 0.3 + 7 log 0.7 = −6.11. At p = 0.5 it is 10 log 0.5 = −6.93. The fraction of heads wins.

The second part asks for the variance of a Bernoulli. The mean is E[y] = 1 × p + 0 × (1 − p) = p. The variance is E[(y − p)2] = p (1 − p)2 + (1 − p) p2 = p (1 − p) [(1 − p) + p] = p (1 − p). It is largest, 0.25, at p = 0.5.

Tie this to Chapter 1. For a neuron, p is the spike chance in one tiny bin. If p = 0.012, the variance is 0.012 × 0.988 = 0.0119, almost equal to p. For small p the Bernoulli variance matches its mean, which is why counts of many such bins look Poisson.

You flip a coin 20 times and see 6 heads. What is the maximum likelihood estimate of p, and the variance of one flip at that estimate?

Chapter 10

Field Guide: Fit a Neuron in Three Moves

Keep the numbers, the terms, a checklist and the core code on one page, and see where the lecture leads next

Here is the whole lecture in one paragraph. A neuron never answers the same way twice, so we model a response with a probability, P(y | x, θ). To fit θ, we score every setting by how probable the recorded data are, multiply over independent trials, take the log, and find the peak. For a Poisson neuron the peak is total spikes over total contrast. For a Gaussian neuron it is least squares. Add a nonlinearity between the line and the noise and you have the generalized linear model, which has no closed form but is still easy to climb.

The numbers and formulas

ModelFitted byWorked value in this lesson
Poisson countMean = variance = λ. P(k | λ) = λk e−λ / k!λ = 2 gives P(0) = 0.135, P(2) = 0.271
Linear Poisson neuronθ̂ = ∑y / ∑x. Typical error √(θ / ∑x)Counts 4, 22, 31 at contrasts 5, 20, 35: 57 / 60 = 0.95
Linear Gaussian neuronθ̂ = ∑xy / ∑x2. σ̂2 = mean squared errorSame trials: 1,545 / 1,650 = 0.936, σ̂ = 2.2
Bernoulli coinp̂ = k / N. Variance p(1 − p)3 heads in 10 flips: 0.3, variance 0.21
Stimulus likelihood (Poisson)x̂ = y / θ. Area under the curve 1 / θy = 20, θ = 1: x̂ = 20, spread about 4.5
Poisson GLM, f = exp∑(y − eθx) x = 0, solved by gradient ascentBest score −66.0 against −152.2 for a rectified line

The terms

TermPlain meaning
Encoding modelP(y | x, θ) read as stimulus in, response out
LikelihoodThe same formula read as a function of θ, data held fixed. It does not add up to 1
Conditional independenceGiven x and θ, one trial says nothing about another, so probabilities multiply
ConsistentThe estimate converges to the true θ as data grow without limit
EfficientIt converges as fast as possible, saturating the Fisher information
General linear modelLinear box then noise: y = θ · x + ε
Generalized linear modelLinear box, nonlinearity f, then exponential-family noise. The link function is f−1

A checklist for fitting a neuron

  1. Name the three parts. What is y, what is x, and what are the knobs θ?
  2. Pick the noise from the data. Counts of spikes: Poisson. Smooth signals: Gaussian. Check the Fano factor.
  3. Check independence. Look for a pattern between each trial’s error and the next.
  4. Write the log-likelihood. Add the trial terms and drop everything without θ.
  5. Solve for the peak. Use the closed form when the model has one. Otherwise climb, and check the score rises at every step.
  6. Check the predictions. No negative rates, no absurd rates at the edges of your stimulus range.
  7. Read it backwards when you decode. Vary x, hold y, and bring in a prior with Bayes rule.

The core as code

This is the whole lecture as a few lines of Python. The arrays are x of shape [N] for the contrasts and y of shape [N] for the spike counts.

pythonimport numpy as np
from scipy.special import gammaln

# Poisson neuron, rate = theta * x: the closed form (slide 23)
theta_pois = y.sum() / x.sum()

# Gaussian neuron, mean = theta * x: least squares through the origin (slide 28)
theta_gauss = (x * y).sum() / (x * x).sum()
sigma2 = ((y - theta_gauss * x) ** 2).mean()

def loglik(X, y, theta):
    lam = np.exp(X @ theta)                        # X: [N, D], theta: [D], lam: [N], f = exp
    return np.sum(y * np.log(lam) - lam - gammaln(y + 1))

def fit_glm(X, y, lr=1e-3, steps=3000):
    theta = np.zeros(X.shape[1])
    for _ in range(steps):
        lam = np.exp(X @ theta)
        theta += lr * (X.T @ (y - lam))                # gradient: error times input, summed
    return theta

Run loglik after every few steps of fit_glm. It must rise. If it falls, halve lr.

Where this sits, and where to go

Maximum likelihood and the GLM are the backbone of statistical neuroscience. Every decoder in a brain-computer interface starts from an encoding model like these. To go on, take Point-Process GLMs for spike trains, Linear Decoders for the decoding side, and Spikes to Rates for binning. For the maths in depth, see Generalized Linear Models, Linear Regression, Logistic Regression and Bayes Rule. The earlier lectures of this course are Least Squares and Probability. Related models for hidden state and sampling are in Hidden Markov Models, The Kalman Filter, EM and Gaussian Mixtures, PCA and Monte Carlo.

The takeaway. Fitting a neuron means choosing the knob setting that makes your recorded spikes least surprising. For a Poisson neuron that is total spikes over total contrast. Everything else in the lecture is the same idea with a different box in the middle.

Credit. This lesson follows Lecture 9 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. The course site has the slides and the homework. All words, numbers on the devices, and pictures here are our own re-teaching, and the toy data are marked as illustrative.

Now press Teach and explain, from memory, how to fit a Poisson neuron: the model, the likelihood, the log, the peak, and what changes when you add a nonlinearity. Then go back to the hero and find the peak by eye once more.

NEU 560 Lecture 9, re-taught
Course site · Back to Gleams