A neuron fires 0 to 7 spikes in a window. How many yes/no questions does it take to learn the count? A quiet neuron needs fewer than a bursty one.
Learn the Gaussian rules, how to estimate a hidden number from spikes, two ways to think about probability, and how many bits a spike count carries.
Pick a neuron, then pick a plan for guessing its count. Watch the number of questions change, and see the one number that no plan can beat. Then we build, piece by piece, the Gaussians, the estimators and the entropy underneath it.
You need what a probability is (a number from 0 to 1 that says how likely something is) and a little algebra. We build the rest from zero.
Based on Lecture 7 of Princeton NEU 560: Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 6, Probability.
Illustrative neurons. Bursty is the lecture’s second example: probability 1/8 on four values and 1/2 on the top value, shown here as spike counts 0 to 7 (the slides call them 1 to 8, with 0 on the first three). Quiet is made up: 60, 20, 8, 5, 3, 2, 1 and 1 percent. One by one counts the question “is it 7?” too. The halve-the-chance plan is the optimal question tree, built live in your browser.
Chapter 0
See why a single spike count raises two different questions, and which half of the lecture answers each
An electrode sits next to one neuron in the visual cortex. A grating flashes on a screen for 200 milliseconds. The recording shows 5 spikes. You flash the same grating again and see 3. Again: 6. Again: 4.
Nothing about the grating changed, yet the count did. Every record of a neuron looks like this. The count wobbles from trial to trial around some typical value.
That wobble raises two separate questions. They are the two halves of Lecture 7 of Princeton’s NEU 560, taught by Jonathan Pillow.
Suppose the neuron has a true average rate, say 4.5 spikes per window, that you cannot see. You only see counts. Each count is a noisy view of the rate. How do you turn a handful of counts into a best guess of the rate? How sure can you be?
Take the four counts above: 5, 3, 6 and 4. Their sample mean (add them up, divide by how many) is 18/4 = 4.5 spikes. Each count sits away from that mean by 0.5, 1.5, 1.5 and 0.5. Square those gaps, add them, and you get 5. Divide by 3 for the usual sample variance, 1.67, so the spread is about 1.29 spikes. Four counts tell you the typical value and the wobble, but only roughly: another four trials would give a slightly different mean. Chapters 4 to 6 ask how far such a guess can be trusted.
This is statistics: learning a hidden number from noisy data. The first half of the lecture is about it. It starts with the Gaussian (the bell curve), because the Gaussian makes the maths easy. Then it defines an estimator (a recipe that turns data into a guess) and asks what makes a recipe good. It ends by splitting statisticians into two camps, frequentists and Bayesians, who disagree about what “probability” means.
Now forget the hidden rate. Take the count itself. A friend sees it and you want to learn it. You may ask only yes/no questions. How many questions do you need?
If the neuron is quiet and the count is nearly always 0, one question may be enough: “is it 0?” If the count could be anything from 0 to 7 with equal chance, you will need more. The average number of questions is a measure of how surprising the neuron is. Claude Shannon named it entropy in his 1948 paper “A mathematical theory of communication”. The second half of the lecture builds it from the questions up.
The hero at the top of this page is that game. Go back to it after Chapter 7 and you will be able to explain every number it shows.
The lecture has 32 slides and no assigned reading. It opens with the title slide, then walks through six Gaussian facts, estimators, frequentist and Bayesian views, a summary of the probability unit, and then information theory. The lecture’s own summary slide lists what Lecture 6 and this lecture together covered: distributions, marginalization, conditioning, Bayes’ rule, expectations, Monte Carlo, independence, Gaussians, estimation and the two camps. This lesson follows the slides in order and uses neural examples throughout.
One note on the maths. We prove little and show a lot. Where a topic has its own full lesson on this site, we say so and link it. Lecture 6 is the base: Probability.
Chapter 1
Scale, add and multiply bell curves, and see that the answer is always another bell curve
You are decoding where a bar of light sits in the visual field. One group of neurons says the bar is at 10 degrees, give or take 4. A second group says 16 degrees, give or take 2. Both reports are noisy. You want one best guess, and you want to know how sure you can be.
To answer, you need to combine two uncertain numbers. This is easy when both uncertainties are bell curves, and Pillow’s slide title says why: “the amazing Gaussian”.
A Gaussian (or normal) distribution is the bell curve. It has two numbers. The mean μ is where the peak sits. The variance σ² says how wide the bell is. Its square root σ is the standard deviation, in the same units as the data. The lecture writes “X is distributed as a Gaussian” as X ~ N(μ, σ²).
Two numbers describe the whole curve. That is a gift. If an operation turns one Gaussian into another, you only have to track two numbers instead of a whole function.
The slide says the Gaussian family is closed under many operations. Closed means: do the operation, and the result is still a member of the family. The slide lists three.
One: scaling. Multiply a Gaussian variable X by a constant a, and aX is Gaussian. The mean scales by a. The variance scales by a², because variance is a squared quantity. A spike count over 200 ms becomes a rate in spikes per second when you multiply by 5. A count with mean 5 and standard deviation 2 becomes a rate with mean 25 and standard deviation 10.
Two: sums. Add two independent Gaussian variables X and Y, and X + Y is Gaussian. The means add. The variances add too. The standard deviations do not. Take X with σ = 3 and Y with σ = 4. The variances are 9 and 16, so the sum has variance 25 and σ = 5. The lecture adds that any linear function of Gaussian variables, such as 2X − 3Y + 7, is Gaussian too, since it is built from scalings and sums.
Why do variances add? Think of two independent wobbles. Sometimes they push the same way and sometimes they cancel. On average the spread grows less than the plain sum of the two widths. That is why the answer is 5, not 7, in the example above.
Three: products. This one is subtler, and the slide words it with care. It does not say that X times Y is Gaussian. It says that the product of the two density curves, P(X) times P(Y), is proportional to another Gaussian density, P(Z). Take two bell curves, multiply them height by height at every point, and rescale so the area is 1. You get a new bell curve.
This is exactly how you combine the two cues. Each cue is a bell curve over the bar’s position. A position is believable only if both cues find it believable, so you multiply. The result is a narrower bell that sits between the two, closer to the sharper cue.
The rule uses precision, which is 1 divided by the variance. Narrow bells have high precision. Precisions add, and the new mean is an average of the old means weighted by precision.
Work the example. Cue one: μ1 = 10, σ1 = 4, so precision 1/16 = 0.0625. Cue two: μ2 = 16, σ2 = 2, so precision 1/4 = 0.25. The precisions add to 0.3125, so the variance is 3.2 and σ = 1.79. The mean is 3.2 × (10 × 0.0625 + 16 × 0.25) = 3.2 × 4.625 = 14.8. The answer sits 80% of the way from cue one to cue two, because cue two carries 0.25 of the 0.3125 precision.
Try it. Switch between the two operations and drag the second cue.
The blue solid curve is cue one, N(10, 4²). The blue dashed curve is cue two, which you control. The warm curve is the result. Choose Sum or Product, then move the sliders.
Illustrative cues in degrees of visual angle. Sum assumes X and Y are independent. The product curve is rescaled to area 1, as in the lecture’s P(X)P(Y) ∝ P(Z). All numbers are computed exactly from the formulas above.
Push the second spread up to 8 in Product mode. The result moves only from 10 to 11.2: a vague cue has little precision to add. Set it to 4 and the result lands exactly halfway. In Sum mode the result gets wider, never narrower. Adding uncertainty can only grow it.
The data flow is short. Input: two means and two spreads, four numbers. Output: two numbers, a mean and a spread. No integrals, no sampling. That is why neuroscientists reach for Gaussians so often. A decoder that merges cues, or a filter that merges a prediction with a measurement, runs in a few multiplications. The Kalman filter is this product rule applied once per time step, and the multivariate Gaussian lesson shows it in many dimensions.
Chapter 2
Average enough lopsided spike counts and watch a bell curve appear, whatever you started with
One trial of one neuron gives you a count: 0, 1, 2 or maybe 4 spikes. A histogram of single counts is lopsided. It piles up at 0 and 1 and trails off to the right. It looks nothing like a bell curve.
Now an experimenter does what experimenters always do. She repeats the stimulus 20 times and averages the counts. She does this on many days and plots the averages. The histogram of averages is a smooth, symmetric bell. The lopsidedness is gone.
This is the fourth Gaussian fact on Pillow’s slide, and it is the reason Gaussians are everywhere.
The central limit theorem (CLT) says this. Take n independent random numbers X1 to Xn that all follow the same distribution, with mean μ and variance σ². Form their average Sn = (X1 + … + Xn)/n. As n grows, Sn becomes Gaussian, with mean μ and variance σ²/n.
Notice what it does not say. It does not require the X values to be Gaussian. The starting distribution can be a coin flip, a spike count, or anything with a finite variance. The slide shows coin flipping: a bar chart for n = 1 (two bars, tails and heads), then n = 5, 10 and 20, each one smoother and narrower than the last. The lecture adds why this matters: it explains why many things in nature are approximately Gaussian. A measurement that is the sum of many small independent nudges ends up Gaussian.
Chapter 1 gave us the tool. A sum of independent variables has variance equal to the sum of the variances. Add n copies and the variance is nσ². Dividing by n scales the variance by 1/n². So the average has variance nσ²/n² = σ²/n, and standard deviation σ/√n.
Worked example with the slide’s coin. One fair flip, counted as 0 or 1, has mean 0.5 and σ = 0.5. The average of n flips has σ = 0.5/√n. For n = 1 that is 0.5. For n = 5 it is 0.224. For n = 10, 0.158. For n = 20, 0.112. Those are the four shrinking bell widths in the slide’s picture. Now a spike count with mean rate 0.7 per bin has variance 0.7 and σ = 0.84. Over 20 trials, σ = 0.84/√20 = 0.19. Each step of the way you can compute the answer with a calculator.
The reason the bell shape appears is the same as in Chapter 1: sums of independent pieces keep smoothing the distribution. Each new term blurs sharp corners and evens out lopsidedness.
Run the experiment below. It draws 3,000 datasets, each with n draws, and plots the 3,000 averages.
Blue bars are the histogram of 3,000 averages. The warm curve is the Gaussian the theorem predicts. Choose a starting distribution, then drag n up from 1.
Simulated live in your browser with a fixed random seed, so the picture is the same on every visit. The skewness is measured from the 3,000 averages; the theory column is 1/√(nλ) for spike counts and (1−2p)/√(np(1−p)) for the coin and the rare spike. Skewness is 0 for a perfect bell.
Start with the fair coin and slide n from 1 to 10. Two bars become a hill almost at once. Now pick Rare spike. At n = 5 the histogram is still piled against the left wall, and its skewness is large. Even at n = 60 it is still visibly lopsided, with skewness near 0.5.
This is the failure mode. The theorem is a statement about the limit. It does not say how fast you get there. The symptom: you average 5 trials of a neuron that rarely fires, and your confidence intervals from the Gaussian formula are wrong, too narrow on one side. The measurement: compute the skewness of the averages. For a rare spike with p = 0.05 and n = 5 it is about 1.8, far from 0. The fix: use more trials or longer time bins, until skewness falls near 0. The skewness shrinks as 1/√n, so waiting for 4 times as many trials halves it.
There is a second, rarer failure. If the distribution has infinite variance, such as the Cauchy distribution with its very heavy tails, the average never becomes Gaussian. Neural spike counts do not have this problem, since a neuron has a maximum firing rate.
The CLT is also why the Gaussian shows up in the next chapters. When we estimate a rate from many trials, the estimate is an average, so for large n its error is a Gaussian with a known width. The Monte Carlo lesson uses the same 1/√n rule to say how many samples an estimate needs.
Chapter 3
Slice and squash a cloud of two neurons’ spike counts and see that every view is again a bell curve
Two neighbouring neurons both listen to the same stimulus. On each trial you record both spike counts. Plot trial after trial as a dot: the first neuron’s count across, the second’s up. The dots form a fuzzy cloud. If the neurons share noise, the cloud leans. When one fires more than usual, so does the other.
A real recording has hundreds of neurons, but two are enough to see every idea. The slide calls this a population response: one vector r = (r1, r2, …) per trial, one entry per neuron.
A multivariate Gaussian is the bell curve for a whole vector. The slide writes X ~ N(μ, C). Now μ is a list of means, one per neuron. Shape: N numbers. And C is the covariance matrix, an N by N table. Its diagonal holds each neuron’s variance. Each off-diagonal entry says how two neurons move together. A positive entry means they rise and fall together.
The density formula on the slide looks heavy, but it has one job. It measures how far x is from the centre μ, using distances that stretch along the cloud’s own shape, and then turns the distance into a height.
Our device standardises both neurons: mean 0 and variance 1. Then the only free number is the correlation ρ, which lives in the off-diagonal of C. We will use three of the slide’s facts.
Marginals. Ignore the first neuron and look only at the second. This is called marginalizing: you squash the cloud onto the second axis. The slide says the marginal is Gaussian. Its mean and variance are just the matching entries of μ and C. In our standardised cloud, the marginal of r2 is N(0, 1), whatever ρ is.
Conditionals. Now fix the first neuron’s count at some value c and ask what the second does. This is a slice: you keep only the trials where r1 was c. The slide says slices are Gaussian too. For our standardised pair, r2 given r1 = c has mean ρc and variance 1 − ρ². Worked example: ρ = 0.6 and c = 1.5. The mean is 0.9 and the variance is 1 − 0.36 = 0.64, so σ = 0.8. Knowing the first neuron moved the best guess for the second to 0.9 and made the guess sharper, from 1.0 to 0.8.
Linear projections. Finally, the slide takes a weighted sum of the neurons, Y = AX. The matrix A sets the weights. Y is Gaussian with mean Aμ and covariance ACAᵀ. Shape: if A is k by N, Y has k numbers. The slide’s example is A = [1 1], which just adds the two neurons’ counts. Its variance is 1 + 1 + 2ρ. With ρ = 0.6 that is 3.2, so σ = 1.79. With ρ = −0.6 it is only 0.8.
That last number matters for brains. A downstream neuron that pools two inputs gets a noisier signal when their noise is positively correlated, because the shared noise does not cancel. Negative correlation helps. A linear readout, the simplest decoder, is exactly a projection, so this one rule explains how noise correlations hurt or help it.
Try all three views at once. Drag ρ and the slice, then turn the projection direction.
Each dot is one trial of two neurons, in standardised units. The warm ring is the 2-spread contour. The teal band is your slice. The blue line is your projection direction. Below, three bell curves show what each view looks like.
Toy data: 260 trials drawn with a fixed seed from a standard two-neuron Gaussian, then tilted by the correlation. The curve formulas are exact: marginal N(0, 1); conditional N(ρc, 1 − ρ²); a projection onto the direction at angle φ has variance 1 + ρ sin 2φ. At 45° the unit direction is A = [1 1]/√2, the slide’s A scaled to length 1.
Set ρ to 0.9 and the slice to 1.5. The conditional curve becomes needle thin and centred near 1.35: nearly everything about neuron 2 is known from neuron 1. Now turn the angle to 135 degrees, across the lean. The projection narrows to a spread of 0.32. Looking across the leaning direction shows almost no variation, because the cloud is a thin ellipse. This is the idea behind principal component analysis: find the direction of largest variance, and drop the rest.
The multivariate Gaussian lesson derives the conditional formulas in general and in code. The full-size version of this chapter is the same rule with larger matrices.
Chapter 4
Turn a handful of spike counts into a guess, then measure how good the guessing recipe is
A neuron fires at some true average rate to a fixed stimulus. You cannot read that rate off the cell. You can only record counts: 6, 3, 4, 7, 5. You must announce one number as your best guess. Your lab partner announces a different one. Whose recipe is better?
To answer, we need to say what a recipe is, and what “better” means. That is Pillow’s two estimation slides.
The slide draws a loop. On the left is the parameter θ, the hidden number you want. In the slide it is a stimulus. In our device it is the neuron’s mean count. In the middle is the model p(x | θ), a rule that says how likely each dataset is for a given θ. On the right is the dataset x = {r1, r2, …, rn}, the measured counts. The slide calls it a population response when the r values are many neurons, and plots spike count against neuron number.
The model points from θ to data: given the cause, here is how the data scatter. Estimation runs the arrow backwards. An estimator is any function f that takes the dataset and returns a guess, written θ̂(x), or just θ̂. The hat means “estimated”. Shape: x holds n numbers, and θ̂ is one number.
Here are three simple recipes for the neuron’s mean count. The sample mean: average the n counts. The constant: always answer 7, whatever the data say. The shrunk mean: (n × sample mean + 2 × 7) / (n + 2), an average that pulls toward 7 as if you had 2 extra counts of 7 in the data.
The data are random, so θ̂ is random too. Run the experiment again and you get a different dataset and a different guess. To judge a recipe, imagine repeating the experiment many times with the same true θ, and look at the cloud of guesses. Two questions matter about that cloud. Where is its centre, and how wide is it?
The bias is the centre minus the truth: b(θ) = E[θ̂] − θ. The letter E means expected value, the average over many draws of the data x. An estimator is unbiased if the bias is 0. The variance is the width: var = E[(θ̂ − E[θ̂])²], the average squared distance of a guess from the cloud’s own centre.
Think of darts. Bias is how far the cluster is from the bullseye. Variance is how spread out the cluster is. A tight cluster in the wrong place has low variance and high bias.
The slide adds one more word. An estimator is consistent if both bias and variance go to zero as n grows without limit. A consistent recipe homes in on the truth when you give it enough data.
Now the slide’s own question: what is the bias of the estimator that says θ̂(x) = 7 for every dataset? The estimate never changes, so E[θ̂] = 7 and the bias is 7 − θ. The variance is 0, because the guess never moves. If the truth is 5, this recipe is wrong by exactly 2, every time, with perfect confidence.
Compare the recipes. Set the true mean and the number of trials. The device runs each recipe on 2,000 fresh datasets.
Each row is a histogram of 2,000 guesses. The warm line is the true mean count. The teal triangle marks each cloud’s centre. Drag the true mean and the number of trials, and watch each cloud’s centre and width.
Simulated live: counts are Poisson with the true mean, a standard model of spike counts; seeded, so repeatable. MSE is bias squared plus variance. Theory for the shrunk mean: bias 2(7−θ)/(n+2), variance nθ/(n+2)².
Start at θ = 5, n = 10. The sample mean is centred on 5 with variance 0.50. The constant is a single bar at 7. The shrunk mean is slightly off centre, but narrower than the sample mean. Its total error is lower. Move θ down to 2 and the order flips: the pull toward 7 now costs more than it saves.
Total error combines both parts. The mean squared error is MSE = E[(θ̂ − θ)²] = bias² + variance. Squaring the bias means a small bias is cheap and a large one is costly. That is why a slightly biased recipe can beat an unbiased one when data are scarce. You trade a little bias for a lot less variance.
Now slide n to 40 at θ = 5. The sample mean and the shrunk mean both tighten around 5. The constant does not move. Only the first two are consistent.
The symptom: a recipe gives nearly identical answers on every run, and that feels reassuring. The measurement: compare the answers with a known truth, or simulate as above. A stable recipe with a large bias is just confidently wrong. The fix: judge a recipe by both numbers together, as MSE, and check that the bias shrinks as n grows. Code for the check is five lines:
def bias_var(estimator, theta, n, reps=2000): est = np.array([estimator(rng.poisson(theta, n)) for _ in range(reps)]) return est.mean() - theta, est.var() # (bias, variance)
Finally, notice what these properties rest on. Bias and variance are averages over repeated datasets with θ held fixed. The slide stresses that they are frequentist properties. The next chapter says what that word means. For the matching ideas in their own lesson, see estimator evaluation and Bayesian estimation.
Chapter 5
Decide what is random, the data or the parameter, and see how the same spikes give two different kinds of answer
You recorded 10 windows from a neuron and counted 62 spikes in all. A colleague asks: “What is the chance that the neuron’s true rate is between 5 and 7 spikes per window?”
One kind of statistician says: that question has no answer. The true rate is one fixed number. It is either between 5 and 7 or it is not. Chance does not enter. Another kind says: of course it has an answer, and here it is. Both are doing honest maths. They disagree about what the word “probability” may be attached to.
Slide 8 puts the two camps side by side. For a frequentist, probability is a long-run frequency in repeated experiments. The probability of heads is the fraction of heads you would see in 1,000 flips. For a Bayesian, probability measures subjective uncertainty, a degree of belief. The probability of heads is how strongly I believe this particular flip will land heads.
The slide notes what the Bayesian reading allows. You can attach a probability to things a frequentist cannot, such as “the probability that the billionth digit of pi is 7”. That digit is already fixed. No experiment can be repeated. But you can still hold a belief about it.
Slides 9 and 10 turn the difference into a picture with two circles: a parameter space (all the possible values of θ) and a data space (all the datasets you might record).
The frequentist picture: θ is a single, fixed point. The data are random. An arrow labelled P(x | θ) runs from the parameter circle to the data circle. The slide says frequentists sit in the parameter space and contemplate what happens as we sample from the model repeatedly. Bias and variance from Chapter 4 live here.
The Bayesian picture reverses it. The data x are fixed, because you have them in your hand. The parameter θ is random, in the sense that you are unsure of it. The arrow now points backwards, labelled P(θ | x). The slide says Bayesians sit in the data space and contemplate what inferences we can make from our data and our prior beliefs.
The reversal is Bayes’ rule, from Lecture 6. The new belief, the posterior, is the old belief, the prior, reweighted by how well each θ explains the data, the likelihood.
Work the opening example with a standard recipe for counts. Model each count as Poisson with unknown rate θ, and give θ a Gamma prior, a flexible bump over positive numbers. With the prior Gamma(3, 1), the posterior after n windows with total T spikes is Gamma(3 + T, 1 + n). For T = 62 and n = 10 that is Gamma(65, 11): mean 5.91, spread 0.73. The Bayesian answer to the colleague is a number, the area of that curve between 5 and 7, about 0.82.
The frequentist answers a different question. Imagine the experiment repeated again and again. A recipe that builds an interval from the data, here the sample mean plus or minus 1.96 standard errors, will contain the true rate in about 95% of those repeats. This is a confidence interval. The 95% describes the recipe over many datasets. It does not describe this one interval.
The device puts both side by side. The top panel is the frequentist world: the true rate is fixed at 6, and the histogram shows what the sample mean does across 1,500 repeated datasets. The bottom panel is the Bayesian world: one dataset, one posterior.
Top: the data vary, the rate is fixed at 6. Bottom: your dataset is fixed, the rate is uncertain. The blue tick is your dataset’s mean count. Press New dataset, change n, or check how often each interval catches 6.
Toy model, all computed live: Poisson counts with true rate 6, prior Gamma(3, 1) (mean 3, so the prior is wrong and the data must fix it), posterior Gamma(3 + total, 1 + n). Confidence interval: mean ± 1.96√(mean/n). Credible interval: the middle 95% of the posterior. Seeded, so repeatable.
Press New dataset a few times. The top panel never changes, because it describes the whole population of possible datasets. The bottom panel jumps, because each dataset brings its own belief. Now press Check 500 datasets at n = 5 and then at n = 40. At n = 40 both intervals catch 6 close to 95% of the time (94.6% and 96.0%). At n = 5 both fall short, at 92.8% and 91.8%, and the Bayesian one is pulled toward the wrong prior mean of 3. The prior helps when it is right and costs you when it is wrong.
Slide 13 puts it in one line. A Bayesian says “I use probabilities”, and the slide’s examples are maximum-likelihood decoding and Fisher information. The non-Bayesian says “I don’t use probability, per se.” In practice, the working neuroscientist mixes the two. Both camps need the likelihood P(x | θ). They differ in what they do next: average over datasets, or reweight a prior.
The failure mode is the cross-reading. The symptom: a report says “there is a 95% chance the true rate lies in this interval”, and the interval came from a frequentist recipe. The measurement: that sentence claims something about this one interval, which the recipe never promised. The 95% describes coverage across datasets, as the Check button measures. The fix: say “this recipe covers the truth 95% of the time” for a confidence interval, and use the posterior for belief statements. Our Bayes’ rule lesson builds the prior and posterior from zero.
Chapter 6
Compute the best accuracy any recipe could reach from spikes, and check a recipe against that limit
A neuron’s count changes a little when you shift the stimulus by one degree. If it changes a lot, the count tells you a lot about the stimulus. If it barely changes, you are guessing. Is there one number for how much a count tells you?
There is, and it also sets a floor. No recipe, however clever, can guess more accurately than that number allows. Slides 11 and 12 pack this into one dense table, with the frequentist column on the left and the Bayesian column on the right. We unpack the left column first.
The data are random, so a likelihood curve log p(r | θ) wobbles from dataset to dataset. Look at how sharply it peaks near the true θ. A sharp peak means the data pin θ down. A flat one means they do not. Sharpness is the curvature, the second derivative. Fisher information is the average curvature, with a minus sign so it comes out positive.
Work it for a Poisson neuron whose mean count is θ. The log-likelihood of one count r is r log θ − θ − log r!. The first derivative is r/θ − 1. The second is −r/θ². Take the expectation, using E[r] = θ: IF = θ/θ² = 1/θ. For n independent windows the information adds, so IF = n/θ. A neuron with a low rate tells you more per window about its rate, since each count is relatively less noisy.
The slide states the unbiased form of the bound. Its efficiency line says that an estimator is efficient when its squared error equals 1/IF, the Cramér-Rao bound. The general form is standard theory, and we add it here. For any estimator with bias b(θ), the variance obeys var(θ̂) ≥ (1 + b′(θ))² / IF(θ). The prime means derivative with respect to θ. If the estimator is unbiased, then b = 0 and the bound is simply 1/IF. The standard proof uses the Cauchy-Schwarz inequality, E[A²] · E[B²] ≥ E[AB]². An estimator that reaches it is called efficient: its efficiency, 1/(IF × var), equals 1.
For the Poisson neuron, the bound is θ/n. Compare the sample mean: its variance is θ/n exactly. The sample mean is efficient. No unbiased recipe for the Poisson mean can beat it. That is why everyone uses it.
Now the shrunk mean from Chapter 4, (n × mean + 14)/(n + 2). Its bias is 2(7 − θ)/(n + 2), so b′ = −2/(n + 2) and 1 + b′ = n/(n + 2). Its bound is n²/(n + 2)² × θ/n = nθ/(n + 2)², which is the variance we found before. So the biased recipe is efficient for its own bias. Because it leans on 14 imagined counts, it ends up with a lower bound than the unbiased one. Biased recipes are not breaking the rule; they play a different game.
Let a neuron’s mean count depend on the stimulus s through a tuning curve f(s). A Poisson neuron has Fisher information IF(s) = f′(s)² / f(s) per window: the squared slope of the tuning curve, divided by the noise level f. Take illustrative numbers: f = 10 spikes and a slope f′ = 2 spikes per degree. Then IF = 4/10 = 0.4 per square degree, and the bound is a spread of 1/√0.4 = 1.58 degrees. Four independent neurons like it give IF = 1.6 and a spread of 0.79 degrees. The information is high where the tuning curve is steep, not where it peaks. Neurons that fire most at the preferred stimulus tell you least about small changes there.
Check three recipes against the bound. Pick the true mean and the number of windows. The device measures each recipe’s variance on 4,000 datasets.
Each bar is a recipe’s measured variance. The green mark is its Cramér-Rao bound. A recipe can reach the mark, never go clearly below it. Drag the true mean and n.
Simulated live with Poisson counts and a fixed seed; measured variances wobble by about 2% around their exact values. Bounds: θ/n for the two unbiased recipes, nθ/(n+2)² for the shrunk mean. Half the data means the first ⌊n/2⌋ windows. Illustrative recipes, not from the slides.
At θ = 5, n = 10, the full sample mean measures near 0.50, on its mark. Half the data costs you a factor of 2: variance near 1.00, efficiency about 50%. You threw away half your spikes. Now raise n to 40. Every bar shrinks like 1/n, and the sample mean still sits on its bound.
The slides’ right column asks the same question with the other bookkeeping. The Bayesian quantity is Bayes’ risk: the average squared error of an estimate, with the average taken over the posterior, ∫ (θ̂(x) − θ)² P(θ | x) dθ. The data x are fixed. Slide 12 marks the contrast in colour. For the frequentist, all integrals run over the data x. For the Bayesian, integrals always run over the parameter θ.
The failure mode here is quoting a bound without its conditions. The symptom: your estimator seems to beat 1/IF, for instance variance 0.20 where the bound says 0.50. The measurement: check the bias. The plain bound only holds for unbiased recipes. A biased one must use (1 + b′)²/IF. The constant answer 7 has variance 0, but its bias slope is b′ = −1, so its bound is 0 too, and the “violation” vanishes. The fix: always report the bias next to the variance. See the Fisher information lesson for the full derivation.
Chapter 7
Guess a hidden number with yes/no questions, and discover that the best plan needs log base 2 of K questions
Statistics is done for now. The second half of the lecture changes the subject. Slide 15 opens it with Claude Shannon’s 1948 paper, “A mathematical theory of communication”. It lists the topics information theory covers in this course: entropy, conditional entropy, mutual information, KL divergence, the data processing inequality, the efficient coding hypothesis (Barlow, 1961) and maximum-entropy distributions. This lecture covers only the first.
Slide 16 gives that first idea in one line. Entropy is the number of yes/no questions you need, on average, to learn the value of a random variable. We build it from a game.
A friend thinks of a number from 1 to 8. You may ask yes/no questions. Your goal is to name the number with as few questions as possible. The same game applies to a neuron: a count between 1 and 8 spikes, a friend who saw it, you with a line to the recording room.
Strategy 1: one by one. Ask “is it 1?”, then “is it 2?”, and so on. If the number is 8, you ask 8 questions. That is the worst case. The slide says the average is 4 if every number is equally likely. Counting every question exactly, including the last, the average is (1 + 2 + … + 8)/8 = 4.5, which the slide rounds down. Then it asks: can we do better?
Strategy 2: halve the numbers. Ask “is it more than 4?”. Whatever the answer, four numbers remain: 1 to 4 or 5 to 8. Ask “is it more than 2?” (or “more than 6?”). Two numbers remain. One last question, “is it 2?”, finishes the job. The slide steps through this one question at a time, building a branching tree. Every number is found in exactly 3 questions. The average is 3.
The saving is big, and it comes from one idea. A good question cuts the possibilities in half. A bad question, such as “is it 1?”, cuts off only one of eight and leaves seven.
Each answer is one bit, the smallest unit of information: a choice between two. After N answers you can tell apart 2N possibilities, since each answer doubles what you can distinguish. The slide writes this as: 2N numbers need N binary questions. Turn it around. For K possibilities you need N such that 2N = K. That N is the logarithm base 2 of K.
Worked examples. K = 8 gives 3, as above. K = 16 gives 4. K = 1,000 gives about 10, because 210 = 1,024 is the first power of 2 above 1,000. A spike count with 64 equally likely values needs 6 questions, so 6 bits. When K is not a power of 2, such as 5, the best plan mixes 2-question and 3-question endings: three numbers take 2 questions and two take 3, an average of 2.4, just above log2(5) = 2.32.
Try it. Choose a plan, set K, and click any bar to see the questions for that secret number.
Each bar is how many questions the plan needs when the secret is that number. The yellow line is the average. The dashed line is log base 2 of K. Click a bar or use the buttons to see the questions asked.
Exact counts, every secret equally likely. One by one counts the question “is it K?” as the slide does, so the worst case is K questions. Halving asks “is it above the middle?” of the numbers still possible, and stops when one is left.
Set K to 8 and choose Halve the numbers. Every bar is 3 and the yellow line sits exactly on the dashed one. Switch to One by one: the bars climb from 1 to 8 like stairs and the average is 4.5. Now set K to 5 with halving. The bars are 3, 3, 2, 2, 2, the average is 2.4, and the dashed line is just below at 2.32. The dashed line is a floor: the best any plan can do on average. Halving reaches it when K is a power of 2 and comes within one question otherwise.
The data flow is simple. Input: a secret index from 1 to K, hidden from you. Each question is a function from the secret to one bit. Output: after about log2(K) answers, you hold a list of bits that spells the secret. That list of answers is a code for the number, and the next chapters call its length the code length. A neuron that sends a count to the next brain area, or an implant that sends spikes over a radio link, is doing the same job; our implant compression lesson shows the cost of sending those bits.
The failure mode is the lopsided question. The symptom: a plan that works but takes 8 questions on some numbers. The measurement: the bar chart, where the One by one stairs reach 8 while halving never exceeds 3. The fix: pick each question so that both answers are about equally likely. That rule only works as stated while all numbers are equally likely. Chapter 8 takes away that assumption. For more practice, the information theory workbook has exercises.
Chapter 8
Give the common numbers short question lists and the rare numbers long ones, and cut the average below log base 2 of K
Now a neuron that is not fair. It fires 7 spikes in a window half of the time. The rest of the time it gives 3, 4, 5 or 6 spikes, about equally often. It never gives 0, 1 or 2. You play the guessing game again.
This is slide 24, “motivating example #2”. The slide writes the probabilities P(X) above the numbers 1 to 8: zero for 1, 2 and 3, one eighth each for 4, 5, 6 and 7, and one half for 8. (In our hero, these are the spike counts 0 to 7, and the half-probability value is the top count. Here we keep the slide’s numbering.)
The slide notes that 3 questions would still suffice. Halving the numbers works exactly as in Chapter 7. But it ignores something. Half the time the answer is 8, and a single question could settle that: “is it 8?” The slide asks whether we can do better.
The slide builds the new plan one question at a time. Q1: is it 8? If yes, done in 1 question. If no, 4, 5, 6 or 7 remain, each equally likely. Q2: is it more than 5? That leaves 4 and 5, or 6 and 7. Q3: is it 5? (or “is it 7?” on the other branch.) Done.
Count the average. Half the time you finish in 1 question. The other half you need 3. The average is 1/2 × 1 + 1/2 × 3 = 1/2 + 3/2 = 2 questions. That beats the 3 questions of halving the numbers. The first question, “is it 8?”, is a perfectly fair coin flip in disguise: it splits the probability into two halves, 1/2 and 1/2, even though it splits the numbers into 1 and 7.
The slide states the rule. Divide the probability in half with each question. Every number reached after N answers has probability 1/2N, because each answer halved the chance. So a number with probability 1/2N needs N questions.
Flip the rule round, as in Chapter 7. A number with probability 1/K needs log2(K) questions. Write 1/K as p. Since log2(K) = −log2(1/K) = −log2(p), a number with probability p needs −log2(p) questions. The slide braces this quantity and labels it code length.
Check it against the slide. The number 8 has p = 1/2 and −log2(1/2) = 1. The numbers 4 to 7 have p = 1/8 and −log2(1/8) = 3. Slide 30 writes the answers out as a code, one list of Y and N for each number: 8 is Y; 7 is NYY; 6 is NYN; 5 is NNY; 4 is NNN. Anyone who hears the list of answers can rebuild the number, and no code is the start of another, so there is never any doubt where one number ends.
This is the same trick as everyday speech. Common words are short (“the”, “a”), rare words are long. A quiet neuron works the same way. Its most common count, silence, gets the shortest code, and a rare burst of 7 spikes gets the longest. That is why a recording from a mostly quiet channel can be compressed so much.
Play with the neuron. Slide the chance of the top number and watch the best question plan, its codes and its average.
Bars show each number’s chance. Under each: its yes/no code, then its surprise, −log base 2 of p. The yellow meter is the average number of questions; the dashed line is the entropy. Start at 50% on the top number, as in the lecture.
Exact, computed live. The other numbers share the rest of the probability equally. The plan is the optimal question tree (repeatedly merging the two least likely groups). At 50% in the lecture’s layout it matches slide 30 exactly: 8 is Y, 4 to 7 are NNN, NNY, NYN, NYY.
At 50% in the lecture’s layout the average is 2.00 and the entropy is 2.00. They match, because every chance is a power of one half. Now slide to 60%. The plan stays the same (8 gets a one-question code), but the average is 1 × 0.6 + 3 × 0.4 = 1.80 and the entropy is 1.77, a little below. Single questions come in whole numbers, but −log2(p) is not whole unless p is a power of one half. The entropy is the floor, and a plan that asks one number at a time can sit above it. A plan that codes several numbers together can close the gap, a standard result that the lecture does not take up.
Now choose All eight numbers and slide to 12.5%. All numbers are equally likely, every code has 3 letters, and the average is 3.00. This is Chapter 7 again: the uniform case is the hardest, since no number is cheaper than another. Slide up to 90% and the average falls to about 1.3 questions. A neuron that almost always says the same thing is cheap to describe.
The failure mode is using the wrong chances. The symptom: your question plan is tuned to a neuron’s old firing pattern, and the neuron changes. The measurement: average the plan’s code lengths under the new chances, instead of the old. Set a plan for 90% and test it on the uniform neuron: its average becomes 3.5 instead of the 3.0 that a plan made for the uniform neuron achieves. The fix: rebuild the plan from the current chances. Later in the course this gap, between a plan made for one distribution and used on another, becomes the KL divergence.
Chapter 9
Write the average number of questions as one formula, and see where it is largest and where it is zero
A neuron fires in about 2 of every 100 one-millisecond bins. Most bins are silent. When a spike does arrive, it feels like news. When silence arrives, nobody is surprised. How do we put a number on that feeling?
Chapter 8 already did the hard part. A value with probability p needs −log2(p) questions. Slide 32 gives that quantity a second name, the surprise of a value. A rare value is a big surprise. A certain value, p = 1, has surprise −log2(1) = 0: no news at all.
Slide 31 closes the loop. The number of questions for one value x is its surprise, −log p(x). The number of questions you need on average is that surprise averaged over how often each value occurs. Multiply each value’s chance by its surprise and add them up. The slide annotates the formula in two colours: the sum over x, weighted by p(x), is the “averaged over p(x)” part, and log p(x) is the “# questions for x” part. The leading minus sign makes the whole thing positive.
Slide 32 sums up in two lines: the surprise function is −log p(x), and entropy is the average surprise, or the average number of yes/no questions, written −∑ P(x) log P(x), “in bits”. If you use the natural logarithm instead of base 2, the unit is the nat, and you divide by ln 2 = 0.693 to get bits.
Take the neuron with a spike chance of p = 0.02 per bin. A spike has surprise −log2(0.02) = 5.64 bits. Silence has chance 0.98 and surprise −log2(0.98) = 0.029 bits. Now average: 0.02 × 5.64 + 0.98 × 0.029 = 0.113 + 0.029 = 0.141 bits per bin. A bin of this neuron needs on average only 0.14 of a yes/no question. If the 1,000 bins in a second were independent (an assumption, not a fact), the second would carry about 141 bits at most. A fair coin, p = 0.5, would carry a full 1 bit per flip, seven times as much.
The slide ends with two facts about a distribution on K bins. The maximum entropy is log K, reached by the uniform distribution. This matches Chapter 7: with K equally likely values you need log2(K) questions, and nothing is easier to predict than a lopsided guess. The minimum entropy is 0, reached when all the probability sits in one bin. Then you know the value before you ask anything.
Use the device below to see both ends and everything between. Choose the number of bins. Then slide the chance of the first bin. The other bins share the rest equally.
The top strip shows the chance of each bin and its surprise in bits. The curve shows entropy as the first bin’s chance moves from 0 to 1. The dot is where you are; the dashed line marks the uniform case, the maximum.
Exact, computed live from H = −∑ p log2 p, with the other bins sharing the remaining chance equally. The 2-bin curve is the binary entropy function. The 0.141 bits for p = 0.02 is the lesson’s own worked example with an illustrative spike chance.
Choose 2 bins and slide to 50%. The curve peaks at exactly 1.00 bit: a coin flip is the hardest to predict. Slide toward 2%: entropy falls to 0.14 bits, the example above. Now choose 8 bins and set the chance near 12.5%. Every bin has the same chance and the entropy reaches its peak of 3.00 bits, which is log2(8). Pile 99.9% into one bin and the entropy is near 0.
Notice the curve is not a straight line. Moving from 50% to 60% on the two-bin curve costs only 0.03 bits. Moving from 5% to 15% adds 0.32 bits. Entropy is most sensitive to changes when the distribution is nearly certain.
The data flow: a vector p of K chances that sum to 1 goes in, and one number, in bits, comes out. In code the whole formula is one line, with one trap.
def entropy(p): # p: shape (K,), sums to 1 p = np.asarray(p, dtype=float) p = p[p > 0] # drop zero-chance bins return -(p * np.log2(p)).sum() # bits
The failure mode is the zero bin. The symptom: the function returns nan for the lecture’s own example, where the numbers 1 to 3 have chance 0. The measurement: np.log2(0) is minus infinity, and 0 times minus infinity is nan. The fix: the maths defines 0 × log 0 as 0, because a value that never happens adds no surprise, so drop zero-chance bins first, as the second line does. With that fix, the lecture’s example gives 2.00 bits, exactly the average we counted in Chapter 8.
Entropy returns later in the course, and in machine learning as the training loss for classifiers; see loss functions for where it shows up. For exercises, use the information theory workbook.
Chapter 10
Keep the numbers, the checks and the code, and find where each idea goes next
Lecture 7 did two jobs. First it finished the statistics toolkit. Gaussians stay Gaussian when you scale, add or multiply them. Averages of many draws turn Gaussian. A multivariate Gaussian keeps its slices, squashes and projections Gaussian. An estimator turns data into a guess, and bias and variance measure how good it is. Frequentists hold the parameter fixed and let the data vary. Bayesians hold the data fixed and let the parameter be uncertain. Fisher information sets a floor on variance. Second, it began information theory: entropy is the average number of yes/no questions, which is the average surprise, −∑ p log2 p.
| Idea | Rule | A number from this lesson |
|---|---|---|
| Scale a Gaussian | aX has mean aμ and variance a²σ² | Count to rate, times 5: spread 2 becomes 10 |
| Add Gaussians | Means add, variances add | Spreads 3 and 4 give spread 5 |
| Multiply Gaussian densities | Precisions add; mean is precision-weighted | 10 ± 4 with 16 ± 2 gives 14.8 ± 1.79 |
| Central limit theorem | The average of n draws has spread σ/√n and tends to a Gaussian | Fair coin, n = 20: spread 0.112 |
| Conditional Gaussian | Standardised pair: mean ρc, variance 1 − ρ² | ρ = 0.6, c = 1.5: mean 0.90, spread 0.80 |
| Linear projection | AX has mean Aμ and covariance ACAᵀ | A = [1 1], ρ = 0.6: variance 3.2 |
| Bias and variance | MSE = bias² + variance | Always answering 7 when the truth is 5: bias 2, variance 0 |
| Fisher information, Poisson | IF = n/θ; bound = θ/n for unbiased recipes | θ = 5, n = 10: variance at least 0.50 |
| Questions for K equal values | log2 K | K = 8: 3 questions; K = 1,000: about 10 |
| Code length of a value | −log2 p | p = 1/2: 1 question; p = 1/8: 3 questions |
| Entropy | H = −∑ p log2 p; from 0 to log2 K | Lecture example 2: 2.00 bits; spike chance 0.02: 0.141 bits |
nan.import numpy as np rng = np.random.default_rng(0) def gauss_product(m1, s1, m2, s2): # combine two cues: precisions add p1, p2 = 1 / s1**2, 1 / s2**2 var = 1 / (p1 + p2) return var * (m1 * p1 + m2 * p2), np.sqrt(var) # (10,4),(16,2) -> (14.8, 1.79) def bias_var(est, theta, n, reps=2000): # data x: shape (n,), guess: one number g = np.array([est(rng.poisson(theta, n)) for _ in range(reps)]) return g.mean() - theta, g.var() mean_est = lambda x: x.mean() shrunk_est = lambda x: (x.sum() + 14) / (len(x) + 2) cr_bound = lambda theta, n: theta / n # Fisher information n/theta def entropy(p): # p: shape (K,), sums to 1 p = np.asarray(p, float); p = p[p > 0] return -(p * np.log2(p)).sum() def question_lengths(p): # best question plan: merge the two least likely groups = [(q, [i]) for i, q in enumerate(p) if q > 0] L = np.zeros(len(p), int) while len(groups) > 1: groups.sort(key=lambda g: g[0]) (qa, ia), (qb, ib) = groups[0], groups[1] L[ia + ib] += 1; groups = groups[2:] + [(qa + qb, ia + ib)] return L # [0,0,0,1/8,1/8,1/8,1/8,1/2] -> 3,3,3,3,1 on the live bins
This lecture is the second of the probability pair. Lecture 6 built distributions, Bayes’ rule and Monte Carlo. Here we used them to estimate and to count questions. The slides list what information theory brings next in this course: conditional entropy, mutual information, KL divergence, the data processing inequality, the efficient coding hypothesis and maximum-entropy distributions.
For neural data, the spike-count and Poisson models of this lecture feed straight into generalized linear models. For the Gaussian machinery in action, try the Kalman filter and PCA. For the maths in more depth, read Fisher information, estimator evaluation, Bayesian estimation, the multivariate Gaussian, Monte Carlo and the information theory workbook. For the first spikes of the brain-computer interface series, start with The Spike and From Spikes to Rates.
Credit. This lesson follows Lecture 7 of NEU 560: Statistical Modeling and Analysis of Neural Data, Princeton, Spring 2024, taught by Jonathan Pillow. The slides are at slides07_Entropy.pdf. All text, devices and numbers outside the slides are our own, and the toy neurons are illustrative.
Now press Teach and explain one spike count out loud, from memory: how two cues combine, why the average of trials turns Gaussian, what bias and variance mean for a recipe, how a frequentist and a Bayesian answer the same question, and why a quiet neuron needs fewer yes/no questions than a bursty one. Then go back to the neurons at the top, pick the bursty neuron, switch the plan to halving the chance, and explain the number you see.