A spike count can never be negative, yet a straight line happily predicts minus two spikes. Swap the line for an exponential and the noise for a Poisson, and the model respects the neuron.
Learn how one recipe, a linear step, a bend, then noise, turns a stimulus into a spike train, and how to fit it.
Pick a model, then press Fit the best curve. Compare what each one predicts for the quietest stimulus. Then we build it piece by piece: the design matrix, least squares, coin flips, Poisson counts, and the spike-history filter that makes a model burst and adapt.
You need a rough idea of a probability and the dot product of two lists of numbers. We build the rest from zero.
Based on Lecture 10 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. Lecture slides. Every idea is re-taught here in our own words and devices. It follows Lecture 6, Probability and the least-squares lecture, Lecture 4.
Illustrative data: 36 time bins from a made-up neuron whose true rate is exp(0.3 + 0.9 × stimulus) spikes per bin, with Poisson counts drawn from a fixed seed. The line is fitted by least squares; the exponential by maximum likelihood (Newton steps). The slides use the same two ideas, not this data.
Chapter 0
See why the same stimulus gives a different spike count every time, and why that noise decides which model you may use
Picture a recording room. A thin wire sits next to one neuron in visual cortex. A screen flashes the same pattern again and again. You count the spikes the neuron fires in each short stretch of time after the flash.
Trial one gives 3 spikes in the best stretch. Trial two gives 1. Trial three gives 5. The pattern never changed, and the answer never repeats. This is not a faulty wire. A neuron is a noisy device, and the noise is part of what you must model.
That fact shapes the whole lecture. A model of a neuron must say two things. First, how strongly the neuron tends to fire for a given stimulus. Second, how the real count scatters around that tendency. The lecture gives a single recipe for both, and it calls the recipe a generalized linear model, or GLM.
The firing rate is the average number of spikes per time bin when the stimulus is the same. Think of a rain gauge on a day with steady drizzle. The drizzle has a rate. Each minute the gauge catches a slightly different amount. The rate is the drizzle. The catch is the scatter.
The noise model is the rule that turns a rate into a random count. For spikes, the natural rule is the Poisson distribution: it gives the chance of seeing 0, 1, 2, 3 and more events when events arrive independently at a known average rate. Its key trait is that the scatter grows with the rate. A neuron that averages 0.2 spikes per bin mostly gives 0. A neuron that averages 8 gives numbers between 4 and 12.
Run it yourself below. The warm line is the true rate across twenty time bins. Each tick row is one trial. Add trials and watch the average of the counts find the rate.
Top: spikes on each trial, in 20 time bins. Bottom: the blue bars are the average count per bin over your trials; the warm line is the true rate. Drag the number of trials, then press Draw again.
Illustrative: a made-up rate that peaks at 2.6 spikes per bin, and counts drawn live in your browser from a Poisson distribution with that rate.
With four trials the bars are ragged. A bin that truly averages 2.6 can show 1.5 or 3.8. With thirty trials the bars hug the line. This is the law of large numbers: averages of noisy counts settle on the true mean. It is also why a model must be fitted to many bins, not a few.
Here is the whole lecture in one line. Take the stimulus, mix it with a list of weights (linear). Bend the result so it is a valid rate (nonlinear). Draw a random count from that rate (noise). The slides write the three boxes as Linear, Nonlinear, Noise. Chapter 2 opens each box.
Why does the order matter? Each box fixes one problem. The linear box compresses a huge stimulus into one number. The bend fixes the range, because a rate cannot be negative. The noise box says how real counts scatter. You will see each problem bite in turn.
The lecture begins with five warm-up problems about two neurons and information, then recaps last time, then builds the GLM family one noise model at a time: Gaussian, Bernoulli, Poisson. It ends with spike-history filters and a summary, then extra practice problems. There are no assigned readings for this session. Logistics (homework, exam dates) live on the course site.
The hero above is a first taste. A straight line, the simplest model, predicts a negative count at the quiet end. The exponential curve cannot. The rest of the lesson shows why, and then how to fit the curve to real counts.
Chapter 1
Measure the information two spike counts share, using the lecture’s warm-up table, in bits you can count by hand
Two neurons sit side by side. The lecture calls them x and y. In every time bin each one fires 1, 2 or 3 spikes, never more and never fewer. You record both for a long time and tabulate how often each pair of counts happens. That table is a joint distribution: for every pair (x, y), the share of bins that showed it.
The lecture asks five questions about this table. They are a good workout, and they teach three tools you will keep: the average, the correlation, and mutual information, a number of bits that says how much knowing one neuron reduces your doubt about the other.
Read the lecture’s table by rows. When y = 1, the pair (1, 1) happens half the time. When y = 2, the pairs (2, 2) and (3, 2) each happen one eighth of the time. When y = 3, the pairs (2, 3) and (3, 3) each happen one eighth of the time. All the other cells hold zero, and the whole table adds to 1.
Question 1 asks for the average count of x. First collapse the table over y by adding down each column. This gives the marginal distribution of x, so called because you write it in the margin of the table: P(x) = [0.5, 0.25, 0.25]. Now weight each count by its chance.
E[x] = 0.5 × 1 + 0.25 × 2 + 0.25 × 3 = 1.75 spikes
Question 2 asks whether x and y are uncorrelated, positively or negatively correlated. Two counts are positively correlated when large values tend to come together. Check the table: x = 1 only ever pairs with y = 1, and the larger counts pair with larger counts. The same average works for y (it also equals 1.75). The average of the product is E[xy] = 0.5 × 1 + (1/8)(4 + 6 + 6 + 9) = 3.625, while E[x]E[y] = 3.0625. The gap, the covariance, is 0.5625. It is positive, so the answer is positively correlated.
Now to bits. Play twenty questions. If a neuron fired 1 spike half the time and 2 or 3 spikes a quarter of the time each, how many yes/no questions do you need on average to name the count? Ask first, “Was it 1?” Half the time you are done after one question. Otherwise ask, “Was it 2?” and you are done after two. Average: 0.5 × 1 + 0.5 × 2 = 1.5 questions.
That average is the entropy, written H(x), measured in bits. The formula gives the same number. A rare outcome is more surprising, and the surprise of an outcome with chance p is −log2 p bits. The slide shows three values: −log2(1) = 0, −log2(1/2) = 1 and −log2(1/4) = 2. Entropy is the average surprise.
Question 3. H(x) = 0.5(1) + 0.25(2) + 0.25(2) = 1.5 bits. It matches the twenty-questions count. (The slide writes the sum as 0.5 × 1 + 0.5 × 2, folding the two equal quarters together.)
Now suppose a friend tells you y. Does that shrink your doubt about x? Look at the table by rows. Given y = 1, x must be 1: no doubt at all, 0 bits. Given y = 2, x is 2 or 3 with equal chance: one yes/no question settles it, 1 bit. The same holds for y = 3. Row chances, from the slide: P(y) = 0.5 for y = 1 and 0.25 each for y = 2 and 3.
Question 4. The conditional entropy H(x | y) averages the leftover doubt over every y: 0.5 × 0 + 0.25 × 1 + 0.25 × 1 = 0.5 bits.
Question 5. Mutual information is the doubt that disappeared: I(x, y) = H(x) − H(x | y) = 1.5 − 0.5 = 1 bit. Knowing y answers one of the 1.5 yes/no questions about x. Try the same arithmetic on other tables below.
Left: the joint table, brighter cells for likelier pairs. Right: the three bar lengths in bits. Pick a table, or drag the blend between two neurons that ignore each other and two that always match.
The lecture table is the slide’s. The other three are made up for contrast. Blend 0 is two neurons with the lecture’s marginals that ignore each other; blend 1 is two that always fire the same count.
Drag the blend from 0 to 1. At 0 the neurons are independent, so I = 0: knowing y tells you nothing. At 1 they always match, so H(x | y) = 0 and I = H(x) = 1.5 bits: y tells you everything. The lecture table lies in between, at exactly 1 bit.
Symptom: you compute the correlation of two neurons and get zero, so you drop one from your analysis. Measurement: take x as 1, 2 or 3 with equal chance, and let y = 3 when x is 1 or 3, and y = 1 when x = 2. The covariance works out to exactly 0. Yet y tells you whether x is in the middle. The mutual information is H(x) − H(x | y) = 1.585 − 0.667 = 0.918 bits. Press “Dependent, uncorrelated” to see it.
Fix: correlation only sees straight-line links. Mutual information sees any link. Use it when the relation could bend, as it nearly always does in a neuron. The next chapters build models that bend on purpose.
Chapter 2
Open the three boxes of a generalized linear model and learn the one rule that tells you which bend suits which noise
You have a stimulus: a short clip of a flickering screen, boiled down to 20 numbers, one per pixel patch. You want one prediction: the number of spikes in the next time bin. Twenty numbers in, one number out. How do you get from one to the other without inventing a new method for every neuron?
Last time set the tools. A regression model is a rule P(y | x, θ). Give it inputs x and knobs θ, and it says how likely each output y is. We tune θ by maximum likelihood: pick the knobs that make the recorded data least surprising. In practice, write the log of the chance of the data, take its slope with respect to θ, set the slope to zero, and solve. Sometimes no formula solves it. Then you climb the hill step by step, called gradient ascent.
The lecture also names two quality marks for a fit. A fit is consistent if it homes in on the true θ as the data grow without limit, like a poll that improves as more people answer. A fit is efficient if it gets there as fast as any method can. The lecture states this as “saturates the Fisher information”, a technical limit on how fast data can teach you. You can skip that phrase for now.
Last lecture ended with two names. A general linear model is linear, then noise. A generalized linear model is linear, then nonlinear, then noise. The extra middle box is the whole upgrade. The slide labels the three boxes: linear (“dimensionality reduction”), nonlinear (“warping”), and noise (from the exponential family, a set of distributions that includes the Gaussian, Poisson and Bernoulli).
Follow one bin through it. The linear box takes the 20 stimulus numbers x and the 20 weights θ and computes one number, their dot product: z = θ · x. It squeezes 20 numbers into one. The bend f takes z and returns a number that suits the noise. The noise box uses that number as the centre of a random draw and returns the data y.
The slide gives three examples: Gaussian noise, Poisson noise, and Bernoulli noise. A Bernoulli draw is a single coin flip that gives 0 or 1. Chapters 4, 5 and 6 build each one.
Now the slide asks a question: what restrictions, if any, must the bend f obey in each case? Think about what each noise box needs as its centre. A Gaussian is happy with any real number as its mean. A Poisson centre is a rate, and a rate is an average count, which cannot be negative. A Bernoulli centre is a probability, which must sit between 0 and 1.
The lecture’s answers: Gaussian, none. Poisson, non-negative. Bernoulli, bounded between 0 and 1. Try every pairing in the device below. A curve turns green where its output is valid for the chosen noise, and red where it is not.
Pick a noise model and a bend. Drag z, the output of the linear box. The panel shows f(z) and a few draws from the noise box.
The three noise families and the range rule are the slides’. Draws are made live in your browser. A Gaussian draw uses a standard deviation of 1.
Pick Poisson with the identity bend and drag z below zero. The curve turns red, and the panel says it cannot draw. Switch the bend to exp and the curve is green everywhere. Pick Bernoulli with exp and the curve is green only while the output stays at or below 1. Only the logistic is safe for Bernoulli at every z.
Take stimulus x = [1, 2] and weights θ = [0.5, −0.3]. The linear box gives z = 0.5 × 1 + (−0.3) × 2 = −0.1. With the identity bend and Gaussian noise the mean is −0.1, fine. With exp and Poisson noise the rate is e−0.1 = 0.905 spikes per bin. With the logistic and Bernoulli noise the spike chance is 1/(1 + e0.1) = 0.475. Same z, three valid but different stories. Shapes: x is [d], θ is [d], z is a scalar, f(z) is a scalar, y is a scalar.
Symptom: you pair Poisson noise with the identity bend, and your fitting code returns an error value instead of a number. Measurement: the Poisson log-likelihood contains log of the rate. At z = −0.4 the rate is −0.4, and the log of a negative number does not exist. One negative bin ruins the whole sum. Fix: choose a bend whose output is never negative. Exp and softplus both qualify.
Chapter 3
Slice time into bins, slide a window along the stimulus, and predict the response one bin at a time
So far each trial gave one number: the spike count. Real neurons do not wait for trials. They fire all the time, while the world flickers past. A recording of ten seconds holds thousands of moments, and each moment has its own stimulus and its own response. The lecture calls this the step from spike counts to spike trains.
A spike train is the record of when a neuron fired. To handle it with regression we chop time into equal time bins, say 10 ms each, and write down the number of spikes in each bin. The slide shows a train as a string of digits, 001101100000100001000. It has 21 bins. Six of them hold a spike, so the spike count for the whole stretch is 6.
The count threw away timing. The string keeps it. The count says “6 spikes”. The string says “two pairs close together, then a lone spike, then a lone spike”. To model when a neuron fires, we must model the string bin by bin.
A neuron does not answer the stimulus at one instant only. A photoreceptor signal arrives a few milliseconds late, and a cell may pool the last 50 ms of input. So the prediction for bin t should use the stimulus over a short stretch that ends at bin t. Call that stretch a window and write it as the list xt. Say the window holds five bins.
The slide draws it as a dashed red box that rides along the stimulus. At t = 1 the box sits at the start, at t = 2 it has moved one bin right, and so on through t = 6. At each stop it covers a fresh five-bin slice of the stimulus, and the neuron’s response in that bin is written under it as yt.
Now give the neuron a list of weights, a linear filter k, with one weight per bin of the window. The model says the response is the filter laid over the window, multiplied bin by bin, and added up. Plus noise, the Gaussian that the slide writes N(0, σ2).
A worked bin. Let the filter be k = [−0.1, 0.1, 0.3, 0.6, 0.5], oldest weight first. Let the window be xt = [0.4, −0.2, 1.0, 0.8, 1.2]. Multiply pairs and add: −0.04 − 0.02 + 0.30 + 0.48 + 0.60 = 1.32. The model’s prediction for this bin is 1.32 plus noise. The next bin slides the window one step right: drop the 0.4, add the next stimulus value at the right end.
Walk the window yourself. Drag t or press Play, and watch the teal box move while the five products and their sum update.
Top: the stimulus (blue steps) and the spike train under it. The teal box is the window at bin t. Bottom: each window value, its filter weight, and their product. Drag t, or press Play.
Illustrative: a made-up stimulus of 40 bins and a made-up five-weight filter. The spikes are drawn as coin flips whose chance rises with the filter output; the lecture’s own train and stimulus are on the slides.
Notice two things as you play. First, the prediction is high when big positive stimulus values land under the big positive weights, and low otherwise. The filter is a pattern detector. Second, the recorded spikes often agree: they cluster where the prediction is high. They do not agree every time, because spikes are random.
Two details bite in practice. The first is bin size. Too wide, and the bin hides timing: two spikes 3 ms apart and two spikes 30 ms apart both read “2”. Too narrow, and almost every bin holds 0, which gives a model little to learn from. The right size depends on how fast the cell fires, so try a few.
The second is the edge. Symptom: the first few predictions look odd. Measurement: at t = 2 the window should cover bins −2 to 2, but bins before 1 do not exist. Fix: pad the stimulus with zeros, or start the walk at the first bin where a full window exists. The slider above starts at t = 5 for that reason.
Chapter 4
Write the whole walk as one matrix equation, then see why least squares and maximum likelihood give the same answer
Walking the window one bin at a time is easy to picture and slow to code. A recording has 100,000 bins. You do not want a loop of 100,000 dot products. You want a single operation that does them all, and linear algebra offers exactly that.
The trick is to stack the windows. Write the window of bin 5 as a row of five numbers. Write the window of bin 6 as the row below it. Keep going down the page. The result is a table with one row per time bin and one column per window position. The lecture calls it the design matrix and names it X.
Look at two neighbouring rows. Bin 6’s window equals bin 5’s window moved one place left, with one new stimulus value entering on the right. So each row is a shifted copy of the row above it. Matrices with this look have a name, which the slide gives: a Hankel matrix. (A close cousin, the Toeplitz matrix, is the same pattern mirrored.) The practical fact: every stimulus value appears again and again along a diagonal. Follow one value with the device below.
Top: eight rows of the design matrix, one window per row, beside the recorded response y. Teal cells all hold the same stimulus bin. Bottom: bars for the least-squares filter and the spike-triggered average, from all 36 rows.
Illustrative: the same made-up stimulus, filter and spikes as the window walk in Chapter 3. The bars are rescaled to the same height so their shapes compare. The full matrix X has 36 rows and 5 columns.
Now the whole walk is one line. Put the responses in a column Y with one entry per bin. Put the filter in a column k. Then the model is Y = Xk + noise. Matrix times vector does all 36 dot products at once. Shapes: X is [T × 5], k is [5], Y is [T].
Chapter 2 gave the three-step recipe for maximum likelihood. Apply it. First write the likelihood. The model says each yt is a Gaussian centred on k · xt with variance σ2. The slide writes one bin’s chance as a bell curve. Because the bins are independent across time, the chance of the whole data set is the product over bins. Take the log and the product becomes a sum.
This is the lecture’s “general point”: minimizing a sum of squares is always the same as maximizing the likelihood under a Gaussian noise model. The minus sign flips the problem. Making the squares small makes the log-likelihood big. Try it on one weight in the second device.
Twelve points, one weight k. The yellow curve is the squared error and the warm curve is the log-likelihood, each rescaled to fit. Drag k and the noise level σ. The peak and the trough do not move.
Illustrative: 12 made-up points from y = 1.5 x plus Gaussian noise. The log-likelihood includes its constant, −(n/2) log(2πσ2).
Now differentiate. Write the sum of squares in matrix form: (Y − Xk)T(Y − Xk) is the column of misses multiplied by itself. The best k is where its slope is zero. Set the slope to zero. You get XTY = XTXk. Solve:
The slide labels exactly these two parts. In code: k_hat = np.linalg.solve(X.T@X, X.T@Y). A tiny worked case with one weight and three bins: x = [1, 2, 3], y = [2, 3, 7]. Then XTX = 1 + 4 + 9 = 14 and XTY = 2 + 6 + 21 = 29, so k̂ = 29/14 = 2.071. Squared error at k = 2 is 2.00; at 2.071 it drops to 1.93.
The STA is the filter you would guess by eye: average what the stimulus did just before each spike. Least squares fixes the guess when the stimulus is correlated, by dividing out XTX. The bars in the first device show the two shapes differ here, because the toy stimulus drifts smoothly.
Symptom: the fitted model predicts −0.3 spikes in one bin and 1.4 in another. Measurement: y is 0 or 1, but a Gaussian centre can sit anywhere. Fix: the rest of the lecture. Bernoulli and Poisson noise respect the data’s range. This chapter’s linear-Gaussian fit stays useful as a quick first look, and as the closed form the other models lack.
Chapter 5
Make each bin a weighted coin, see how its chance of heads comes from the filter, and fit the filter by climbing the likelihood
Look at a spike train recorded with 1 ms bins. Almost every bin holds 0. A few hold 1. Nearly none hold 2, because a neuron needs a brief moment to recover after a spike. The data are not a smooth number that wanders around a mean. They are a string of yes and no.
Chapter 4’s linear-Gaussian model fits this badly. Its predictions float anywhere, including below 0 and above 1, and its noise can give 0.37 of a spike. The lecture says it plainly: noise is not Gaussian for binary responses. We need a noise box that only ever returns 0 or 1.
The simplest such box is the Bernoulli distribution: one coin flip with a chance p of heads. The slide calls it the coin-flipping model. Heads means “a spike in this bin”, so y = 1. Tails means y = 0. The chance of heads is the number we must predict, and it can change from bin to bin.
Now run the recipe from Chapter 2. The linear box computes z = k · xt. The bend f turns z into a chance pt = f(k · xt). The bend must give numbers between 0 and 1, as Chapter 2 showed. The noise box flips a coin with that chance.
The slide writes the outcome probabilities: P(yt = 1) = pt and P(yt = 0) = 1 − pt. One line covers both cases:
The bend the lecture picks is the logistic function, f(z) = 1/(1 + e−z). It is an S-shaped curve that squeezes any z into the range 0 to 1. Very negative z gives a chance near 0. Very positive z gives a chance near 1. And z = 0 gives exactly 0.5. The lecture notes that logistic regression, the familiar classifier, is a special case of a Bernoulli GLM.
Chapter 4 told us to multiply the chances of all bins, then take the log. Do it. The log of a product is a sum of logs, and the exponents come down in front:
A worked case with three bins. The spike train is y = [0, 1, 1] and the model chances are p = [0.2, 0.7, 0.9]. The bins give log 0.8 + log 0.7 + log 0.9 = −0.223 − 0.357 − 0.105 = −0.685. A model that gave p = 0.5 everywhere would score 3 × log 0.5 = −2.079, much worse. A higher L means the model found the data less surprising.
In code this is two lines, the slide’s own: z = f(X @ k) to get P(heads) for every bin, then L = np.dot(y, np.log(z)) + np.dot(1-y, np.log(1-z)). Shapes: X is [T × d], k is [d], z and y are [T].
Unlike least squares, setting the derivative of L to zero has no closed-form solution. So we climb. Start with some k, compute the slope of L, take a small step uphill, and repeat. This is the gradient ascent the recap mentioned. Try it on one weight and one baseline.
Dots are 30 bins: a spike (top) or none (bottom) at each stimulus value. The warm curve is your model’s chance of a spike. Yellow stems show how far the model was from each outcome. Drag the knobs, or press Climb.
Illustrative: 30 made-up bins from a true chance of sigmoid(2 × stimulus − 0.5), fixed seed. Climbing uses plain gradient ascent on the log-likelihood, with a small fixed step, then lands on the exact optimum.
Press Climb and watch two things. The curve slides toward the dots, and the log-likelihood rises toward zero. Each step is small and uphill. The rise slows near the top, where the slope flattens. That slow finish is normal for gradient ascent.
Symptom: the fit runs and the weight keeps growing. Measurement: if every spike sits at a larger stimulus than every silence, a steeper curve always scores better, so no finite best weight exists. A chance of 10−9 for a bin that did spike costs log(10−9) = −20.7 in one bin. Fix: add a penalty on large weights, called regularization, which later lectures cover. For this lesson, remember the check: a fit that never settles is a sign of perfectly separable data, not of a bug in the climb.
Chapter 6
Let each bin hold any whole number of spikes, draw it from a rate, and choose the bend that keeps that rate honest
Now widen the bins to 100 ms. A bin can hold 0, 1, 2 or 7 spikes. The Bernoulli coin cannot say “7”. We need a noise box that returns a whole number, zero or more, with no upper limit, and that scatters more when the rate is higher.
That box is the Poisson distribution from Chapter 0. The lecture calls the model Poisson regression or the Poisson GLM. It is the workhorse for spike counts.
Follow the recipe. The linear box gives z = k · xt. The bend gives a firing rate λt = f(k · xt), in spikes per second. The slide also calls it the conditional intensity: the rate conditioned on the stimulus. Bend rule from Chapter 2: λ must be at least zero.
The noise box needs the expected count in one bin. If the bin lasts Δ seconds, the expected count is Δλt. So a rate of 9 spikes per second in a 0.1 s bin gives an expected count of 0.9. The count itself is a Poisson draw with that mean:
Same steps as before. Multiply across bins, take the log, and drop the pieces that do not contain k. The factorial and the log Δ terms are constants. What remains is the slide’s formula:
Reward against penalty is why the fit lands in the right place. A worked bin: one 1-second bin with 3 spikes and a free rate λ. The score is 3 log λ − λ. At λ = 2 it is 2.079 − 2 = 0.079. At λ = 3 it is 3.296 − 3 = 0.296. At λ = 4 it is 4.159 − 4 = 0.159. The best rate is 3, the observed count. In vectors, the slide’s code reads z = f(X @ k) then L = np.dot(y, np.log(z)) - dt*np.sum(z). Shapes: X is [T × d], z and y are [T].
The lecture adds notes on f. In the GLM literature f is usually fixed in advance, but you can test different choices or even fit parameters of f. A few facts to keep. Linear-Gaussian has a closed-form solution only when f is the identity. For Bernoulli, the canonical bend, the one the theory pairs with that noise, is the logistic. For Poisson, the canonical bend is ex, and another common choice is the softplus, log(1 + ex).
The exponential has a nice reading. Since ea + b = ea eb, each input multiplies the rate. A stimulus feature that adds 0.7 to z doubles the rate, because e0.7 ≈ 2. The softplus grows only linearly for large z, so it stays tamer. Fit both below, and also try the identity.
Dots are spike counts in 40 bins of 0.1 s, against stimulus. The warm curve is the expected count in a bin. Pick a bend, drag the knobs, or press Fit for the maximum-likelihood curve.
Illustrative: 40 made-up bins of Δ = 0.1 s from a true rate of exp(2.2 + 0.8 × stimulus) spikes per second, fixed seed. The rate is the bend of the knobs, in spikes per second. The log-likelihood is the slide’s formula without its constant. The identity bend has no log-likelihood whenever any bin gets a rate of zero or less.
Fit with exp and with softplus. Both land close together inside the data. Now look at the extrapolation in the read-out. At a stimulus of 6, far past the data, the exp fit predicts about 1,000 spikes per second while the softplus fit predicts about 50. Try the identity fit too: it gives a negative rate at the quietest stimulus. Which is right depends on the neuron, which is why the lecture says you can test different bends.
Symptom: a very bright test stimulus gives a predicted rate of nearly 10,000 spikes per second. Measurement: z = 9.2 gives e9.2 ≈ 9,900, while softplus gives about 9.2. Real neurons have a ceiling on how fast they can fire, far below that. Fix: keep test stimuli inside the range of the training data, use softplus, or cap the rate. A second slip: forgetting Δ gives a rate per bin, not per second. A 0.9 per bin with 0.1 s bins is 9 spikes per second, ten times larger.
Chapter 7
Feed the neuron its own recent spikes through a second filter, and watch a pure Poisson process become a regular, refractory one
A neuron has just fired. For the next few milliseconds it cannot fire again, however hard you push it. This pause is the refractory period. A model that ignores it will happily predict two spikes 1 ms apart. The real neuron never does that.
The model we have built so far cannot see this. It looks at the stimulus and nothing else. This chapter fixes that with one extra filter.
The lecture gives the Chapter 6 model its classic name: Linear-Nonlinear-Poisson, or LNP. A stimulus x(t) goes through a stimulus filter k, then an exponential nonlinearity f, and out comes a rate λ(t) = f(k · x(t)). A Poisson box turns that rate into spikes. The slide draws it as three boxes in a row.
The slide adds two bullets. The output is a Poisson process: a stream of spikes in which each moment fires independently at the current rate. And the problem: it assumes spiking depends only on the stimulus. A spike a moment ago changes nothing in this model.
The fix, from Truccolo and colleagues (the slide cites ‘04), is to feed the neuron’s own spikes back in. Let yhist(t) be the list of the last few bins of the neuron’s own spike train. A second filter h, the post-spike filter, weights those bins. Its output joins the stimulus filter’s output before the nonlinearity:
The slide reminds us that exp(a + b) = exp(a) exp(b). So the two filters multiply the rate. The stimulus sets a factor. The recent spikes set another factor. A worked case: let the stimulus drive alone be 3.0, giving a rate of e3.0 = 20.1 spikes per second. One bin after a spike, let the post-spike filter read −2.5. Now the drive is 0.5 and the rate is e0.5 = 1.65 spikes per second. The spike cut the rate by a factor of e−2.5 = 0.082, about twelve times.
As data flow, the design matrix of Chapter 4 simply grows extra columns. The first block holds stimulus windows. The second block holds lagged copies of the response y itself. Fit both blocks together with the Poisson likelihood from Chapter 6. Shapes: stimulus block [T × dk], history block [T × dh], joined [T × (dk + dh)].
The output is no longer a Poisson process, because the rate now depends on past spikes. The slide notes the model is also known as an autoregressive (AR) model: it predicts a quantity partly from its own past values. See the effect below. Slide the history depth from 0 up and watch the interval histogram.
Top: twelve simulated trials of a neuron with steady drive. Bottom left: how long it waited between spikes. The dashed line is what a Poisson neuron at the same rate would give. Bottom right: the purple post-spike filter. Drag the depth.
Illustrative: 1 ms bins; a base rate of 20 spikes per second multiplied by edrive; a post-spike filter that starts at minus the depth and decays with an 8 ms time constant. The coefficient of variation (CV) is the standard deviation of the intervals divided by their mean: 1 for a Poisson process. Spikes are drawn live, 20 trials of 1,000 bins.
At depth 0 the intervals follow the dashed curve, tall at zero and falling away. Short gaps are the likeliest, so spikes clump, and the CV sits near 1. As you add depth, the first few milliseconds empty out and a hump forms. The neuron becomes more regular, and the CV falls below 1.
The lecture ends the section with a link to an older model. A classic integrate-and-fire neuron is a leaky bucket. Input pours in, the bucket fills, and when the level reaches a threshold the neuron fires and the bucket empties. The slide draws its spike rate as a hard step: zero below the threshold, effectively infinite above it.
The GLM is the soft-threshold version. Its exponential nonlinearity rises smoothly instead of jumping. Near the threshold the neuron may or may not fire, and the chance climbs with the drive. The post-spike filter plays the part of the emptied bucket. This reading matters because it ties a statistical model to biophysics.
Symptom: the model predicts pairs of spikes 1 ms apart, which the real neuron never produces. Measurement: the real interval histogram is empty for the first few milliseconds, while the Poisson curve peaks at zero. Fix: add the post-spike filter and refit. A caution: a positive h makes each spike raise the next rate, and because exp grows fast the fitted model can run away. Check simulated spikes before you trust a fit.
Chapter 8
Change only the shape of the post-spike filter and see a neuron turn irregular, regular, adapting or bursting
Imagine four neurons that all receive the same signal. A switch flips on at time 25 and stays on until time 75. The first neuron fires at random moments, like rain on a roof. The second fires like a metronome. The third starts fast and then slows. The fourth fires in short bursts with silences between.
You might expect four different models. The lecture shows that one GLM covers all four. Only the post-spike filter h changes. The slides run through four traces and label them irregular spiking, regular spiking, adaptation and bursting.
Each slide has the same layout, top to bottom. The stimulus is the switch. The filter outputs, which the slide calls “currents”, are the sum of the stimulus drive and the history drive from Chapter 7. The p(spike) panel shows the chance of a spike in each bin after the bend squeezes the current into the range 0 to 1. And in the corner the slide draws the post-spike filter h(t): the pattern of weights that a spike leaves behind for the bins that follow.
In the lecture’s traces, the shapes of h are the whole story. For irregular spiking, h is flat at zero: the neuron has no memory, so spikes arrive at random, as in Chapter 6. For regular spiking, h dips deeply negative just after a spike and then returns to zero. Each spike briefly shuts the neuron down, then the drive wins again, and the next spike comes after a steady wait.
For adaptation, h dips and stays negative for a long time. Each spike adds a lasting penalty, and the penalties pile up. A neuron that fires fast at the switch-on builds up a drag and slows down. For bursting, h is first positive and then negative. A spike makes another spike likelier right away, which gives the burst. The negative tail then ends the burst and creates the silence.
Chapter 7 showed that the history filter adds inside the exponent. Every past spike contributes its own copy of h, shifted to its own time. So the history drive at time t is the sum, over all recent spikes, of h evaluated at the time since that spike. Ten spikes with a long negative h give ten overlapping negative tails. This is the mechanism of adaptation, and you can read it off the sum.
A worked bin. Suppose the stimulus drive at time t is +2. Three spikes happened recently, and h reads −1.0, −0.8 and −0.6 at their three lags. The total current is 2 − 1.0 − 0.8 − 0.6 = −0.4, and the logistic bend gives p(spike) = 1/(1 + e0.4) = 0.40. Without the three spikes, p would be 1/(1 + e−2) = 0.88. Three spikes have cut the chance from 88% to 40%.
Try each shape. Pick a behaviour, press Run for a fresh set of trials, and drag the strength to blend the filter in or out.
Left, top to bottom: the stimulus switch, the chance of a spike on trial 1, and ten trials of spikes. Right: the purple post-spike filter. Pick a behaviour, then drag the strength of the filter.
Illustrative: 100 time bins, a logistic bend, and hand-picked filters, not the lecture’s fitted ones. The statistics come from 100 simulated trials. The CV is the standard deviation of the intervals divided by their mean.
With the irregular filter, the spike count in the first 20 bins of the switch equals the count in the last 20, and the CV is near 1. With the adaptation filter, the first 20 bins hold about twice as many spikes as the last 20. With bursting, the CV jumps to about 2, because gaps are either tiny or long. With the regular filter, the CV falls to about 0.5.
The lecture closes the section with a comparison. Izhikevich’s neuron is a small set of equations that is known for reproducing many distinct firing patterns of real cells. The slide shows its panels beside GLM spikes and the GLM parameters that match them, from Weber and Pillow (the slide cites 2017). The slide shows sixteen panels, labelled A to P. They include tonic and phasic spiking, bursting, spike-frequency adaptation, spike latency and bistability. Under each panel sit the two fitted GLM filters.
This is why the GLM earns its place. It is a statistical model whose likelihood has a single peak to climb, yet it can mimic a biophysical model with many dynamic behaviours. The slide for regular spiking carries a Weber and Pillow 2016 label.
Symptom: a neuron you expect to adapt speeds up as the switch stays on. Measurement: the long tail of h is positive, so each spike makes the next more likely, and the penalties of Chapter 7 become rewards. Fix: inspect the fitted h before you interpret it. Set the strength slider on Adaptation to 0 and the neuron stops adapting. At 1.5 it adapts harder.
Chapter 9
Keep the numbers, the checklist and the core code, and work the lecture’s two extra practice problems
The lecture’s own summary has five lines. A generalized linear model is a linear step, a bend and a noise box. Its design matrix stacks one window of stimulus per row. Least squares is maximum likelihood under Gaussian noise. Two more GLMs handle spikes: Bernoulli for binary bins and Poisson (the Linear-Nonlinear-Poisson model) for counts. And a spike-history filter lets the model fire like a real neuron, not like a Poisson process.
| Term | What it says | Number from this lesson |
|---|---|---|
| Warm-up table | Average count of x, then the three information measures. | E[x] = 1.75 spikes; H(x) = 1.5, H(x | y) = 0.5, I = 1 bit |
| Recipe | z = k · x, rate or chance = f(z), then noise. | z = −0.1 gives 0.905 (exp) or 0.475 (logistic) |
| Range rule | Gaussian: any f. Poisson: f ≥ 0. Bernoulli: 0 ≤ f ≤ 1. | the log of a negative rate does not exist |
| Least squares | k̂ = (XTX)−1XTY, the maximum-likelihood fit under Gaussian noise. | x = [1, 2, 3], y = [2, 3, 7] gives k̂ = 29/14 = 2.071 |
| Bernoulli GLM | L = ∑ y log f + (1 − y) log(1 − f). Canonical bend: logistic. | p = [0.2, 0.7, 0.9], y = [0, 1, 1] gives L = −0.685 |
| Poisson GLM | L = ∑ y log f − Δ f. Canonical bend: ex. Another: softplus. | 3 spikes in a 1 s bin: best rate = 3 |
| Spike-history filter | Add h · yhist inside the bend. Filters add, rates multiply. | h = −2.5 cuts the rate by e−2.5 = 0.082 |
| Bursting vs adapting | Bursts: h positive, then negative. Adaptation: long negative tail. | illustrative: CV about 2 for bursting, about 0.5 for regular |
python import numpy as np def hankel(stim, d): # stim [T] to X [T-d+1, d]: each row is a window n = len(stim) - d + 1 return np.stack([stim[i:i+n] for i in range(d)], axis=1) def lagged(y, d): # the neuron's own past: Z[t, j] = y[t-j-1] Z = np.zeros((len(y), d)) for j in range(d): Z[j+1:, j] = y[:len(y)-j-1] return Z def fit_gaussian(X, Y): # linear-Gaussian: least squares = maximum likelihood return np.linalg.solve(X.T @ X, X.T @ Y) def loglik_bernoulli(k, X, y): z = 1 / (1 + np.exp(-(X @ k))) # P(heads) for each time bin return np.dot(y, np.log(z)) + np.dot(1 - y, np.log(1 - z)) def loglik_poisson(k, X, y, dt): z = np.exp(X @ k) # conditional intensity, spikes per second return np.dot(y, np.log(z)) - dt * np.sum(z) def fit_poisson(X, y, dt, lr=0.01, steps=2000): # gradient ascent on the log-likelihood k = np.zeros(X.shape[1]) for _ in range(steps): k += lr * X.T @ (y - dt * np.exp(X @ k)) # d L / d k return k # with memory: stack both blocks, fit once, split the weights afterwards # X = np.hstack([hankel(stim, 20), lagged(y, 10)[19:]]); k = fit_poisson(X, y[19:], dt)
The gradient line uses the fact that the derivative of y log ez − Δ ez with respect to z is y − Δ ez, which is the miss in the expected count. Shapes: X is [T × d], y is [T], k is [d].
The lecture ends with two extra problems. First, two Bernoulli distributions: P1 = [0.5, 0.5] and P2 = [0, 1]. Compute the KL divergence in both directions. KL(P, Q) is the average number of extra bits you pay if you code data that follow P with a code built for Q. It is the sum of P(x) log2 (P(x) / Q(x)).
KL(P1, P2). The first outcome has P1 = 0.5 but P2 = 0. The term is 0.5 log2(0.5/0), which is infinite. So KL(P1, P2) = ∞. A model that gives an outcome zero chance is infinitely surprised when it happens. KL(P2, P1). The first outcome has P2 = 0, so its term is 0 (zero times anything is zero). The second gives 1 × log2(1/0.5) = 1. So KL(P2, P1) = 1 bit. The two directions differ, so KL is not a distance.
Second, suppose y1, …, yN are N counts from a Poisson distribution with unknown mean λ. Derive the maximum-likelihood estimate. The log-likelihood is ∑ (yi log λ − λ − log yi!). Differentiate: ∑ yi / λ − N. Set to zero: λ̂ = (1/N) ∑ yi. The best rate is the sample average. Check with counts [0, 2, 1, 0, 3]: the sum is 6 and N = 5, so λ̂ = 1.2.
These are the GLMs used on real spike data. The Linear-Nonlinear-Poisson model and its spike-history version are the standard encoding models of systems neuroscience. The next steps on the site use the same recipe.
Credit. This lesson follows Lecture 10 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. See the course site and the lecture slides. The spike-history GLM is from Truccolo and colleagues; the dynamic behaviours are from Weber and Pillow, both as cited on the slides. Every device and number here is ours, labelled illustrative where made up.
Press Teach to explain it back on a whiteboard, or Present to rehearse. Then go back to the first instrument and fit the two models again.