A neuron’s spikes are a code. Information theory counts the yes/no questions that code can answer, and shows why the best neurons use every spike count equally often.
Learn entropy, mutual information and KL divergence from zero, with spike counts as the running example, and see how Barlow’s efficient coding idea turns them into predictions about real neurons.
Pick how many stimuli the world can show, then drag the neuron’s top firing rate. Watch the bits climb, then stall well short of the ceiling. Then we build, piece by piece, the numbers underneath.
You need to know what a probability is (a number from 0 to 1 that says how likely something is) and what a logarithm does. We rebuild both in the first two chapters.
Based on Lecture 8 of Princeton NEU 560: Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. Lecture slides. Every idea is re-taught here in our own words and devices.
Toy model, illustrative: stimulus k of n makes the mean count 1 + (top rate − 1) × k/(n−1) spikes in one window, and the count is Poisson. The bits are computed exactly in your browser as I = H(count) − H(count | stimulus).
Chapter 0
See how one plain idea, counting yes/no questions, measures what a neuron tells you and what it wastes
You record one neuron in visual cortex. A screen shows a grating, a set of stripes, at one of eight angles chosen at random. In each half second you count the neuron’s spikes. This time the count is 4. Which angle was on the screen?
You cannot say for sure. Several angles could give a count of 4. But you are not clueless either: a count of 4 makes some angles unlikely and others likely. The neuron told you something. “Something” is not a number, and a scientist needs a number. Information theory supplies one, and the unit is the bit.
This is Lecture 8 of Jonathan Pillow’s Princeton course, and it is the course’s shortest path from probability (Lecture 6) to a deep idea about brains: that neurons are shaped to carry as much information as they can. We get there in two halves. First the tools: entropy, conditional entropy, mutual information, KL divergence and the data processing inequality. Then the theory: Barlow’s efficient coding hypothesis and maximum entropy distributions.
Play twenty questions with the screen. The angle is one of eight, all equally likely. Ask: “Is it in the left four?” Whatever the answer, four angles are gone. Ask again and two are left. Ask once more and you have it. Three questions found one angle out of eight.
The pattern is halving. Each question cuts the candidates in half, so the number of questions is the number of times you can halve n candidates before one remains. That count has a name. It is the logarithm base 2 of n, written log2(n). Read it as “how many times must I double 1 to reach n?” Doubling 1 three times gives 8, so log2(8) = 3.
One bit is the information in one fair yes/no answer. Eight equally likely angles hold 3 bits of uncertainty, because three answers settle it. Sixty-four hold 6 bits. Doubling the number of candidates adds exactly one question. Try it.
Each square is one possible stimulus. A hidden one is the answer. Pick how many there are, then press Ask a question to learn whether it lies in the left half of what is still lit.
Toy: the hidden stimulus is drawn at random on each restart, and the question always splits what is lit into a left half and a right half. The count of questions is exact: log2(n).
The device shows two facts. Questions grow slowly: 64 candidates cost only 6 questions. And the count never depends on luck, because halving always works when all candidates are equally likely.
When they are not equally likely, a cleverer asker needs fewer questions on average. If angle 3 appears 90% of the time, you ask “Is it angle 3?” first and you are done nine times in ten. So the bits in a quantity depend on how its probabilities are spread, and not only on how many outcomes exist. That spread is called entropy. Chapter 2 makes it exact.
The lecture opens with a short warm-up (slide 2): nine practice problems on a small table of probabilities and on a list of eight. Chapter 1 works every one. Slide 3 lists the plan. Entropy is crossed out on it, and slide 4 calls it a recap, so the class had met it before. We rebuild it anyway. The list ends with the efficient coding hypothesis and maximum entropy distributions. A note on the slide says the class “ran out of time and didn’t really cover” the maximum entropy part. We cover both fully, because they are the payoff.
Three questions drive the lecture. How much does a neuron’s response tell you about the stimulus? That is mutual information. How far is a model’s idea of the data from the data itself? That is KL divergence. And what code should a neuron use? That is the efficient coding hypothesis.
A brain-computer interface (BCI) lives on the first question. A decoder reads spikes and guesses movement or speech. It can never recover more than the spikes carry, a limit Chapter 6 proves. Counting bits tells you how much room a decoder has, and how much a noisier electrode costs. The same maths reappears in implant compression, where an implant must squeeze spikes into a thin wireless link.
Chapter 1
Slice, add and renormalise one small table until marginals, moments and conditionals feel automatic
Picture a recording that is easy to count by hand. A lamp shines at one of three brightness levels, called y: 1 is dim, 2 is medium, 3 is bright. A neuron answers with a spike count class, called x: 1, 2 or 3. Over thousands of trials you tally how often each pair (x, y) happened and divide by the total.
The result is a joint distribution, written P(x, y): one probability for every pair, all nine adding to 1. It is slide 2’s table. Dim light with a count of 1 happened in 20% of trials. Medium light with a count of 2 happened in 30%. Every question in the warm-up is a way of reading this one table.
The marginal P(y) asks: ignoring the neuron, how often was each brightness shown? Add across each row. You are summing x out of the table. For y = 1 that is 0.2 + 0.1 + 0.2 = 0.5. The name comes from the old habit of writing these totals in the margin.
The expectation E[y] is the average brightness over many trials. Weigh each value by its probability and add. The second moment E[y2] does the same with each value squared first.
The conditional P(x | y > 1) asks a different kind of question: among only the trials where the light was medium or bright, what did the neuron do? Two steps. Slice: keep the rows that match. Renormalise: divide by the slice’s total probability so the numbers add to 1 again.
Now work all nine problems. Pick each one below. The device lights the cells it uses and draws the distribution the answer lives in.
Warm cells are the table, darker for higher probability. Teal outlines mark the cells a question uses. Choose a question, then read the steps under the picture.
The table and the list of eight probabilities are the lecture’s (slide 2). The brightness and spike count labels are our illustration. All answers are computed live from the table.
Look at question 6, P(x | x = y). The condition keeps only the diagonal cells: 0.2, 0.3 and 0.1. They total 0.6, not 1. Dividing each by 0.6 gives 1/3, 1/2 and 1/6. A condition does not have to be a whole row. It can be any set of cells.
Questions 7 and 8 stay inside one row, y = 3. That row holds 0.1 at x = 1 and 0.1 at x = 3, so its total is 0.2 and the conditional is (0.5, 0, 0.5). The mean is 1 × 0.5 + 3 × 0.5 = 2, a value the neuron never produces. Keep this in mind: an average can sit where no data lives.
Question 8 asks for the entropy of (0.5, 0, 0.5). Two outcomes, equally likely, and one never happens. One fair yes/no question settles it, so the entropy is 1 bit. Question 9 gives eight probabilities, 1/32 up to 1/4. A cleverer asker names the likely outcomes first. The device shows how many questions each outcome costs: an outcome of probability p costs log2(1/p) questions, so 1/4 costs 2 and 1/32 costs 5. Averaging gives 2.6875. Chapter 2 explains why that is the right recipe.
The marginals give a quick test for independence. Two variables are independent when every cell equals the product of its row and column totals. Look at the cell x = 2, y = 2. It holds 0.3. The column total for x = 2 is 0.1 + 0.3 + 0 = 0.4, and the row total for y = 2 is 0.3. Their product is 0.12, far from 0.3. So the neuron’s count does depend on the brightness. Chapter 4 turns that yes-or-no verdict into a number of bits.
Two habits catch most slips. First, every marginal and every conditional you compute must add to 1. Second, every probability you write must lie between 0 and 1. Run both checks on each answer above, and you will find the errors before they spread.
The classic slip is slicing and forgetting to renormalise. Symptom: your “conditional” for y > 1 reads (0.1, 0.3, 0.1), and a quick sum gives 0.5, not 1. Measurement: always add up a distribution you build. Fix: divide by the slice’s total, here 0.5, to get (0.2, 0.6, 0.2). The same slip in a decoder means probabilities that do not add to one, and every later number is wrong.
The table above also hides a second lesson for neuroscientists. Rows (y) are what the world did, columns (x) are what the neuron did. Conditioning on a row asks “given this stimulus, what does the neuron do?” Conditioning on a column asks the decoder’s question: “given this response, what was the stimulus?” The same table answers both. Lecture 6 builds this view in depth, and Bayes’ rule converts one into the other.
Chapter 2
Turn the number of yes/no questions into a formula, and see why a quiet neuron carries so little per bin
Slice a spike train into bins of 10 milliseconds. In each bin the neuron either fires or stays silent. This neuron is sparse: it fires in 5 bins out of 100. You will watch it for a minute. Which bins are worth your attention?
The rare ones. A silent bin is what you expected, and it tells you little. A spike is news. The size of the news depends on how unlikely the event was. Information theory makes that precise with one function of a single probability.
The surprise of an outcome with probability p is −log2(p), in bits. A certain event (p = 1) has surprise 0. A coin flip outcome (p = 1/2) has surprise 1 bit, one question. A one-in-eight event has 3 bits. The spike with p = 0.05 has −log2(0.05) = 4.32 bits, and a silent bin with p = 0.95 has only 0.07 bits.
Why a logarithm, and not, say, 1/p? Take two independent events: a spike now and a spike in the next, unrelated bin. Their probabilities multiply. But news should add: two surprises are twice the news of one. A logarithm turns products into sums, since log(pq) = log p + log q. That is the only simple function with that property, and it matches the question count: one more halving, one more question.
Over many bins, how surprised are you on average? Weigh each outcome’s surprise by how often it happens. The result is the entropy of the distribution, written H(x).
Work the sparse neuron. Spike: 0.05 × 4.32 = 0.216. Silence: 0.95 × 0.074 = 0.070. Add them: 0.29 bits per bin. If a spike were instead as likely as silence (p = 0.5), each bin would carry a full bit. The sparse neuron wastes most of its time saying what you already knew. Drag the spike probability and watch.
The curve is entropy, the average surprise per bin. The two bars below it are the surprise of each outcome. Drag the spike probability and find where the curve peaks.
Illustrative: each 10 ms bin is treated as an independent spike or no-spike event, which real neurons are not (refractory periods, bursts). Entropy peaks at 1 bit when p = 0.5.
The peak at p = 0.5 is the whole story of efficient coding in miniature. A bin is most informative when both outcomes are equally likely. Chapters 7 to 9 turn that into a prediction about how neurons should use their firing rates.
The slide’s other reading of H is the average number of yes/no questions needed to guess x. Here is how a questioner actually gets there. Give every outcome a codeword, a string of yes/no answers (1 for yes, 0 for no). Likely outcomes get short codewords, rare ones long. The Huffman code builds such codewords by repeatedly merging the two least likely outcomes. A fair code costs about H bits per outcome, never less.
Pick a distribution over eight outcomes. Each bar is an outcome’s probability. The string beside it is its question code, built live. Compare the average length with the entropy.
The list is the lecture’s (slide 2, question 9). The skewed and almost-sure distributions are illustrative. Codes come from the Huffman algorithm run in your browser; the average is the sum of probability times code length.
For the slide’s list every probability is a power of one half, so the code lengths equal the surprises exactly and the average, 2.6875 questions, equals H. For the skewed list the average lands slightly above H, because a question can only have whole-number length. Entropy is the floor no questioner can beat.
The slide marks its formula “(in bits)”, which means the logarithm is base 2. Natural logarithms give the same quantity in a different unit, called nats, and 1 bit equals 0.693 nats. Nothing else changes. Every comparison in this lesson holds in either unit, and the ratio between two entropies is the same. Machine-learning code often reports nats, because natural logs are the default in software. Keep the unit in mind when you compare numbers from different sources. A fair eight-sided die has 3 bits, which is 2.08 nats.
Symptom: someone says a neuron with 1,000 possible spike counts carries log2(1000) = 10 bits per window. Measurement: compute H from the counts you actually see. If one count appears 90% of the time, H is under 1.5 bits. Fix: log2 of the number of outcomes is the ceiling, reached only when all outcomes are equally likely. The entropy is what the distribution really delivers. Also, 0 × log 0 is defined as 0, so impossible outcomes add nothing, as in Chapter 1.
This formula also trains language models. Cross entropy, the loss in Loss functions, is the average surprise of the true outcome under the model’s probabilities. Chapter 5 shows how it splits into entropy plus KL divergence.
Chapter 3
Measure the entropy of a spike time, and see why the answer changes when you switch from seconds to milliseconds
A neuron fires a spike a short time after a flash. The delay is not always the same. Across trials it scatters around 20 milliseconds with a spread of 4 milliseconds. A delay can be any real number, not one of eight values. How many bits does that scatter hold?
Try the old recipe. A delay of exactly 20.0000 ms has probability zero, as does every other exact value. The probability lives in intervals, not in points. For a continuous variable we describe the scatter with a probability density p(x): a curve whose area over an interval is the probability of landing in it. The curve’s height is a probability per millisecond, so it can exceed 1. A delay squeezed into a 0.1 ms window has a density of about 10 per millisecond.
The natural step is to swap the sum for an integral. The result is the differential entropy h(x).
The lecture lists the warnings. Differential entropy can be negative, can be infinite, and changes if we change units (the slide’s example is seconds to milliseconds). A negative number of yes/no questions is nonsense, so h is not a question count. What is it?
Chop the delay axis into bins of width Δ and count the questions needed to name the bin. That is an ordinary discrete entropy, H. For a smooth density and a small Δ, each bin has probability about p(x) × Δ, and the surprise of a bin is −log2(p(x)Δ) = −log2 p(x) − log2 Δ. Average it:
H(bins of width Δ) ≈ h(x) − log2(Δ)
So h is the entropy of the variable relative to a ruler of width 1, whatever unit that is. Halve the bin width and you need one more question, so H grows by one bit. Let Δ shrink toward zero and H grows without limit. The bits needed to name an exact real number are infinite. Only h, the part that does not depend on Δ, stays finite.
Now units make sense. Switch from seconds to milliseconds and every number gets 1,000 times bigger, as if the ruler got 1,000 times finer. The entropy changes by log2(1000) = 9.97 bits. Nothing about the neuron changed. Only the ruler did.
For a bell curve with spread (standard deviation) σ, the lecture gives h = log σ + ½ log(2πe). In bits that is log2 σ + 2.05. Check the pieces with σ = 4 ms: log2(4) = 2, plus 2.05 gives h = 4.05 bits relative to a 1 ms ruler. In seconds, σ = 0.004 and log2(0.004) = −7.97, so h = −5.92 bits. Same neuron, two answers, one negative. The slide notes this too: it can be arbitrarily negative or positive.
The bell curve is the scatter of a spike delay. Bars are the delay chopped into bins. Drag the spread and the bin width. Then switch the unit and watch h change while the bins’ entropy stays put.
Illustrative Gaussian delay centred at 20 ms. h = log2 σ + 2.05 bits in the chosen unit. The binned entropy is summed exactly from the Gaussian’s cumulative distribution; the approximation h − log2Δ is shown beside it.
With σ = 4 ms and 1 ms bins, the binned entropy is about 4.05 bits and h agrees. Make the bins 0.5 ms and H climbs to about 5.05, one bit more, as the derivation predicts. The unit toggle moves h by 9.97 bits but leaves H alone, since H describes real bins on a real clock.
Read h = log2 σ + 2.05 for what it says about precision. Doubling the spread adds exactly one bit, because log2(2) = 1. Halving it removes one bit. A neuron whose spike delay scatters over 4 ms holds one bit more timing variety than one that scatters over 2 ms. For a neuroscientist, that is the right way to hear “this cell is twice as precise”: one bit of timing information gained per halving of the jitter, when the jitter is measured against a fixed ruler.
Symptom: a paper reports “the neuron’s timing entropy is −3 bits” and a reader asks how a distribution can carry less than nothing. Measurement: re-express in another unit and see it move. Fix: never read h alone. Compare differences, such as how much a stimulus reduces h. Differences cancel the ruler. That is exactly what mutual information does in Chapter 4, and why it works for continuous signals when entropy by itself does not.
Chapter 4
Measure how many of the stimulus’s bits survive inside a neuron’s response, and how many are lost to noise
A faint tone is either played or not, each on half the trials. A neuron in auditory cortex fires a spike in the following window on 80% of tone trials, and also on 20% of silent trials. You see only the response. How much of your doubt about the tone did the neuron remove?
Before the neuron, you have one bit of doubt: tone or no tone, a fair coin. After a spike you lean toward “tone”, but not with certainty. Some doubt remains. The bits that went away are the bits the neuron told you. Measuring that gap is the entire topic of this chapter.
The conditional entropy H(x | y) is the average entropy of x once you know y. Do it in two steps, as the slide’s colour boxes do. For each response value, take the entropy of the stimulus distribution given that response. Then average those entropies, weighted by how often each response happens.
The slide then merges the two sums using p(x | y)p(y) = p(x, y), giving H(x | y) = −∑ p(x, y) log p(x | y), a single sum over the whole joint table. The last line of slide 7 notes that H(x | y) = H(x) when p(x, y) = p(x)p(y). Independent means knowing y leaves your doubt untouched.
The bits removed are the starting doubt minus the remaining doubt. That is mutual information.
Run the tone example. The four cells of the joint table are 0.4 (tone, spike), 0.1 (tone, silent), 0.1 (no tone, spike) and 0.4 (no tone, silent). H(x) = 1 bit and H(y) = 1 bit, since each is a fair split. The joint entropy is −(2 × 0.4 log2 0.4 + 2 × 0.1 log2 0.1) = 1.058 + 0.664 = 1.722 bits. So I = 1 + 1 − 1.722 = 0.278 bits, and H(x | y) = 1.722 − 1 = 0.722. An 80% hit rate against a 20% false alarm rate removes just over a quarter of a bit of your one bit of doubt.
Slide 9 draws all of this as a Venn diagram. Two overlapping circles, H(x) and H(y). The overlap is I(x, y). The left crescent is H(x | y), what only x has. The right crescent is H(y | x), the neuron’s own noise. The whole shape is H(x, y). Build it yourself.
Blue is the stimulus, warm is the response. The purple overlap is the mutual information. Pick a preset, then adjust the three knobs.
Illustrative neuron with a binary stimulus (tone or none) and a binary response (spike or silence). Circle areas are proportional to entropies, and the overlap area is solved to equal I exactly. All bits are computed from the joint table.
Set the hit rate equal to the false alarm rate. The circles pull apart until they just touch, I = 0, and H(x | y) equals H(x). A neuron that does the same thing whatever the tone is deaf to it. Set the rates to 95% and 5%: the circles nearly merge, because the response almost determines the stimulus.
The lecture adds two plain-language readings of I: “How much does X tell me about Y (or vice versa)?” and “How much is your uncertainty about X reduced from knowing Y?” They are the same number. Note, too, what the third form buys you. To compute I you need only the entropies of x, y and the pair, so you can estimate it from a table of counts. That is how it is estimated from spike data in practice.
Symptom: you find a response after which you are more unsure than before, and conclude the neuron is harmful. Measurement: with a tone on 90% of trials, a hit rate of 20% and a false alarm rate of 60%, the prior doubt is H(x) = 0.47 bits. After a spike, doubt is H(0.75) = 0.81 bits, higher. After silence it is 0.30. Averaged by how often each occurs, 0.24 × 0.81 + 0.76 × 0.30 = 0.42 bits, lower than 0.47. Fix: conditioning on a particular y can raise uncertainty, but on average it never does. That is the statement I(x, y) ≥ 0.
Chapter 5
Count the extra yes/no questions you ask when your model of a neuron is wrong, and see why the price is lopsided
You model how many spikes a neuron fires in a window, from 0 to 7. Your model, Q, says every count is equally likely. The neuron, P, does something else: it never fires fewer than 3 spikes, and it fires 7 half the time. You must ask yes/no questions to name each count, and your question strategy comes from your model. How many questions do you waste?
From Chapter 2, the best strategy for a distribution asks about log2(1/p) questions for an outcome of probability p. Your strategy was built for Q, so it spends log2(1/Q(x)) questions on count x. Here that is 3 questions for every count. But the counts actually arrive with probabilities P(x), so your average cost is ∑ P(x) log2(1/Q(x)): 3 questions per count.
A questioner who knew the truth would average only H(P). For this P that is 2 bits: the count 7 (half the time) costs 1 question, and the four counts at 1/8 cost 3 each, and 0.5 × 1 + 4 × 0.125 × 3 = 2. You paid 3 and could have paid 2. The extra 1 bit is the cost of the wrong codebook.
That extra cost has a name: the Kullback-Leibler divergence, or KL divergence, also called relative entropy.
It is the cross cost minus the true cost: ∑ P log(1/Q) − ∑ P log(1/P). Check with the example. Only the four counts at 1/8 and the count at 1/2 have P > 0. For the four: P log2(P/Q) = 1/8 × log2((1/8)/(1/8)) = 0. For the big one: 1/2 × log2((1/2)/(1/8)) = 1/2 × 2 = 1. Total: 1 bit, as counted above.
The lecture states three facts. KL is at least 0 for all P and Q, because no codebook beats the true one. KL equals 0 only when P = Q, because only the exact codebook is exact. And KL is not symmetric: DKL(P || Q) differs from DKL(Q || P) in general.
Slide 11 asks both directions for two distributions over eight outcomes. P1 is uniform, 1/8 each. P2 is zero on outcomes 1 to 3, then 1/8 on four outcomes and 1/2 on outcome 8. The device below computes it live. First, think about the reverse direction by hand: DKL(P1 || P2). Outcome 1 has P1 = 1/8, but P2 gives it probability 0. The term is 1/8 × log2((1/8)/0), a log of infinity. A codebook built for P2 has no codeword for outcome 1, and P1 uses it one time in eight. The cost is infinite.
Warm bars are the truth P, blue bars are your model Q, over spike counts 0 to 7. Beneath each count is its share of the bill, P log(P/Q). Choose both, then try the smoothing slider.
Slide 11’s P1 and P2 are the lecture’s (outcomes 1 to 8 are shown as counts 0 to 7). The Poisson options are truncated to 0 to 7 and rescaled, an illustration. Smoothing mixes Q with the uniform distribution by the chosen percent.
With P = slide’s P2 and Q = uniform, the forward price is 1.00 bit and the reverse price is infinite (shown as ∞), because the reverse direction charges for outcomes where P2 is zero. The lecture’s exercise has exactly this answer: DKL(P2 || P1) = 1 bit and DKL(P1 || P2) = ∞.
Now swap roles: choose P uniform and Q the slide’s P2. The forward price is infinite, because Q has no word for the first three counts. Drag smoothing up to a few percent. Q now gives every count a sliver of probability, every codebook has a word for every count, and the price drops to a finite number. That is why people smooth a model before scoring it.
The rule of thumb: the first argument decides where the bill is charged, and the second argument must cover it. Wherever P has mass, Q must too.
Slide 12 walks a long chain of equalities. Follow it in three moves. First, write both entropies as sums over the joint table: I = −∑ p(x, y) log p(x) + ∑ p(x, y) log p(x | y). Second, combine the two logs: I = ∑ p(x, y) log [p(x | y) / p(x)]. Third, replace p(x | y) by p(x, y) / p(y):
I(x, y) = ∑ p(x, y) log [ p(x, y) / (p(x)p(y)) ] = DKL( p(x, y) || p(x)p(y) )
So mutual information is the extra cost of pretending x and y are independent when they are not. Check it on the tone neuron from Chapter 4. The joint is (0.4, 0.1, 0.1, 0.4). The product of marginals is 0.25 in every cell. The sum is 2 × 0.4 log2(0.4/0.25) + 2 × 0.1 log2(0.1/0.25) = 0.543 − 0.264 = 0.278 bits. The same number, by a second route. Starting from the other side, p(y | x) / p(y), gives H(y) − H(y | x), so I is symmetric and the chain confirms all the forms of Chapter 4.
Symptom: you compare two fitted neuron models by DKL(data || model) and then by DKL(model || data) and get rankings that disagree. Measurement: compute both, as the device does. Fix: do not call KL a distance. It is a cost with a direction. Pick the direction that matches your question. “What does my model waste when coding real data?” is DKL(data || model). Chapter 6 shows that this is the direction maximum likelihood minimises.
Chapter 6
Prove that no amount of clever processing recovers bits the recording already lost, then link likelihood to KL
A tone is played. A neuron fires or stays silent. An electrode picks up that spike, but it also picks up noise, so some spikes are missed. The implant bins the electrode’s signal and sends one bit per bin over a wireless link, and the link sometimes flips a bit. A decoder reads the bits and guesses the tone. At each stage some information goes missing. Can a smart enough decoder get it back?
No, and that is the data processing inequality. Name the stages: S is the stimulus, R1 is the first recording and R2 is anything computed from R1 alone. The lecture writes S → R1 → R2 and says the three form a Markov chain.
A Markov chain means each link only looks at the one before it. Once you know R1, S adds nothing to your guess about R2. In symbols, slide 13 writes the joint as
P(R1, R2 | S) = P(R2 | R1) P(R1 | S)
Read it left to right: the stimulus produces R1, and R1 alone produces R2. R2 never sees S directly. Its information about S arrives only through R1. The slide’s conclusion is
Here is a short reason, in plain steps. Measure the information that R1 and R2 together hold about S, and count it two ways. Way one: R1 first, then whatever R2 adds. R2 adds nothing beyond R1, since it is built from R1 alone, so the total is I(S, R1). Way two: R2 first, then whatever R1 adds on top. That second piece cannot be negative, because information is never below zero (Chapter 4). So the total is at least I(S, R2). Both ways give the same total. Therefore I(S, R1) is at least I(S, R2).
Take the simplest chain. S is a fair coin, so it holds 1 bit. Each stage copies its input but flips it with some probability. For a flip probability e, the stage keeps 1 − H(e) bits, where H is the binary entropy of Chapter 2. At e = 0.1 that is 1 − 0.469 = 0.531 bits. Chain a second stage with e = 0.1 and a flip happens if exactly one of the two stages flips, a probability 0.1 × 0.9 + 0.9 × 0.1 = 0.18. The bits left are 1 − H(0.18) = 0.320. The second stage cost another 0.21 bits.
A fair stimulus bit passes through the electrode and then the link. Drag each stage’s flip probability. Then choose what to do with the final signal, and try to make the bars rise.
Illustrative binary chain with a fair stimulus bit. For a flip probability e, a stage keeps 1 − H(e) bits; chained flips combine as e1(1−e2) + e2(1−e1). Relabelling is a lossless reprocessing; adding noise is lossy.
Choose “Swap labels”. The bars do not move: a one-to-one relabelling is processing that loses nothing, so the inequality holds with equality. “Add 10% noise” drops the last bar. “Always say 0” drops it to zero. No choice raises it, and none ever will. The best a decoder can do is lose nothing.
Every step between the neurons and a cursor is processing: the electrode, the spike detector, the binning, the radio link, the decoder. Each can only keep or lose bits about the intended movement. So a decoder’s accuracy is capped by what the earliest stage kept. If the spike detector throws away weak units, no later network can bring them back. This is why implant designers measure information before compressing it, as in implant compression, and why a better electrode beats a better decoder when the recording is the bottleneck.
The inequality has a loophole worth knowing. It says processing cannot add information about S. It does not say processing is useless. A good decoder moves the same information into a form that is easy to read. Rearranging keeps the bits, and that is the job.
Slide 14 gathers the formulas on one page: surprise, entropy, differential entropy, mutual information, KL divergence and the data processing inequality. The Field Guide chapter reproduces them with the numbers from this lesson. Slide 15 then names two topics the class did not discuss: how log-likelihood relates to entropy, and how the maximum likelihood estimator relates to KL divergence. They fit together in one step.
Suppose the true distribution of spike counts is P and you fit a model qθ with parameter θ. The average log-likelihood of data from P under the model is ∑ P(x) log qθ(x). Split the log: this equals −H(P) − DKL(P || qθ). The entropy H(P) is a fixed number that no model parameter can touch. So maximising the likelihood is the same as minimising the KL divergence from the data to the model. Maximum likelihood finds the model with the cheapest codebook for your data. The lesson on generalised linear models fits spike counts this way, and point process GLMs do it for whole spike trains.
Warm bars are 40 observed spike counts. Blue bars are a Poisson model with the rate you choose. The curve below is the KL cost of each rate. Slide to the bottom of the curve.
Toy data, illustrative: 40 trials with counts 0 to 6 occurring 3, 7, 10, 9, 6, 3 and 2 times, mean 2.625. The model is an ordinary Poisson distribution; KL is summed over counts 0 to 7 with the data as P.
The curve bottoms out at λ = 2.625, the sample mean, which is the well-known maximum likelihood estimate for a Poisson rate. At that rate the model still pays a small KL bill, because a Poisson cannot match every wiggle of the data. Move away in either direction and the bill rises.
Chapter 7
Ask what a neuron with a limited range of responses should say, and see why the answer is “use all of it”
A retinal ganglion cell sends its message down the optic nerve with a limited range of firing: from silence up to perhaps a few hundred spikes a second, and no more. Each moment it must report something about the light. It can use its range well or badly. Suppose it sat at a few spikes a second almost all the time and rarely moved. Most of its range would carry nothing. What would a good use look like?
Horace Barlow answered in 1961, and the lecture calls it one of the core “big ideas” in theoretical neuroscience. The efficient coding hypothesis says that the goal of the nervous system is to maximise the information it carries about the environment. Atick and Redlich put numbers on it in 1990. Everything in this chapter is one line of arithmetic you already own from Chapter 4.
Let x be the stimulus and y the neuron’s response. Mutual information has the form I(x, y) = H(y) − H(y | x). Slide 18 gives the two parts names. H(y) is the response entropy: how varied the neuron’s output is overall. H(y | x) is the noise entropy: how much the response still varies when the stimulus is held fixed, which is variation the stimulus does not explain.
Think of the response entropy as a bucket and the noise entropy as a leak. What reaches the reader is what is in the bucket minus what leaked. A large, varied output is only useful if it varies because of the stimulus.
Now suppose the physical hardware fixes the range of responses and how noisy they are. Over every possible way of using that hardware, there is a best one, the way that carries the most information. That best value is the channel capacity, C. In symbols C = sup I(x, y). Here sup means the largest value reachable, taken over all possible stimulus distributions. The slide notes two facts: it is an upper bound on mutual information, and it is determined by the physical properties of the encoder.
Barlow defines redundancy as the fraction of capacity that you fail to use.
A neuron with C = 2 bits that carries I = 1.5 bits has R = 1 − 0.75 = 0.25. A quarter of its capacity is redundant. The hypothesis says evolution drives R toward 0.
Barlow’s original 1961 version assumed responses without noise. Then H(y | x) = 0, the leak is shut, and I = H(y). The slides cross out the noise entropy to show this, so R = 1 − H(y)/C. To make redundancy small you must make the response entropy large. The lecture draws two consequences. The brain should use the full dynamic range, so that responses sit at every level and not just a few. And it should decorrelate (“reduce redundancy”): different neurons should not repeat each other. Chapter 10 covers the second.
The lecture points out the “mega impact” of this idea: a huge number of theory and experimental papers asked whether the brain uses decorrelated, information-maximising codes.
The neuron can say one of 8 response levels, 0 to 7 spikes. Choose how often it uses each level, then add noise that blurs a level into its neighbours. Watch information, capacity and redundancy.
Illustrative: each intended level is blurred by a discrete Gaussian of the chosen width, clipped to the range 0 to 7. Capacity is found by the Blahut-Arimoto algorithm run in your browser. The four named codes are our own examples.
At the lowest noise, pick “Even use”. All 8 levels are used equally, the response entropy is 3.00 bits, the noise entropy is nearly 0, and I equals the capacity of about 3 bits: redundancy near 0. Now pick “Mostly silent”. The neuron uses level 0 about 60% of the time, its entropy collapses to about 1.81 bits, and redundancy is about 0.40. Barlow’s prediction in one picture: fill the range.
Now raise the noise. The best code stops being even. With heavy noise, levels in the middle blur into each other, while the two end levels can be confused with only one neighbour. At a blur of 1.0 the solved code uses only four levels, with 34% on each end level and 16% on two middle ones, and capacity falls from 3.00 to 1.36 bits. The neat rule “maximise response entropy” is exact only when noise is absent. With noise you maximise I, which is the Atick and Redlich extension in Chapter 9.
Symptom: a model neuron sits at 1 or 2 spikes for 90% of stimuli and one reader says it is “efficient because it saves spikes”. Measurement: compute response entropy against log2 of the number of levels. If it is far below, the range is wasted. Fix: efficiency has two sides. Saving spikes is a cost saving, which the lecture does not model. For the information goal, the question is whether the range is spread so that the response tells you something every time.
Chapter 8
Pick the most honest distribution for a neuron when all you know is its range, its mean or its variance
You have recorded a neuron for a minute. You know its firing never exceeds 20 spikes per window. That is all. A colleague asks you to model the full distribution of its spike counts. You cannot list 21 probabilities from one fact. Which distribution do you choose?
Choose the one that assumes the least. A distribution with a sharp peak at 7 claims you know something about 7 that you never measured. The distribution with the highest entropy, subject to what you do know, claims nothing more. This is the maximum entropy principle. It also explains where the neural-coding prediction of Chapter 7 will land: the best code, under a limit on the hardware, is the one with the most entropy the limit allows.
Entropy alone has no maximum worth finding: spreading probability over more outcomes always raises it. We need to maximise subject to a constraint. The standard tool is a Lagrange multiplier. Add the constraint to the quantity you are maximising, multiplied by an unknown number λ. Then λ acts as a price on breaking the rule, and you tune it until the rule holds.
Slide 21 works the simplest case. The only constraint is that the counts are 0, 1, …, rmax with probabilities pi that add to 1.
At the best point, the slope of L with respect to each pj is zero. Take that slope (using natural logs): −log pj − 1 + λ = 0. So log pj = λ − 1, and pj = eλ − 1. Nothing in this answer depends on j. Every outcome gets the same probability. The second slope, with respect to λ, just restates that they add to 1, so pj = 1/Nbins. The most honest distribution on a bounded range is the uniform distribution.
Add other facts you know and the same method gives other shapes. Slide 22 lists three pairings, and the next two slides state the general rule. If your constraint is that the average of some function f(x) equals a number c, then the maximum-entropy distribution has the form below.
Read off the three cases. Range constraint: f is nothing, the exponent is flat, so you get the uniform. Mean constraint with non-negative values: f(x) = x, so p(x) ∝ e−|λ|x, an exponential that falls from its largest value at zero. Variance constraint: f(x) = (x − mean)2, so p(x) ∝ e−|λ|(x−mean)², the Gaussian. The bell curve is not magic. It is what you get when you know a centre and a spread and nothing else.
Work one number. Spike counts are 0, 1, 2, … with a mean of 3. The exponential form on integers is pk = (1 − θ)θk, a geometric distribution, with mean θ/(1 − θ). Setting that to 3 gives θ = 0.75. Its entropy is [−0.25 log2 0.25 − 0.75 log2 0.75] / 0.25 = (0.5 + 0.311)/0.25 = 3.25 bits. A Poisson distribution with the same mean of 3 has only 2.79 bits. The Poisson is a fine model of spiking, but it is not the most uncertain way to have a mean of 3. It packs in more structure than the mean alone demands.
Choose what you know about the neuron. The purple shape is the maximum entropy answer. The blue shape is a rival that satisfies the same constraint. Drag the number and check who has more bits.
Illustrative. Range: uniform against a Binomial(R, 0.5). Mean: geometric against Poisson at the same mean (shown for counts 0 to 23). Variance: Gaussian against Laplace and uniform of equal variance, entropies in bits against a 1-unit ruler. The maximum entropy shapes are exact.
Under range, the uniform beats the binomial at every R, which is the slide 21 result. Under mean, the geometric beats the Poisson at every rate. Under variance, the Gaussian beats the Laplace and the uniform: with σ = 1 it has 2.05 bits, against 1.94 and 1.79. No rival wins. That is the theorem, and the device cannot break it.
Slides 23 and 24 pose the general question and leave it open. The slide says the class ran out of time for this part, so the lecture never got to the neurons. Chapter 9 applies the idea to neurons, using the lecture’s slides 25 to 28.
Symptom: a model of firing rates uses a Gaussian, and it assigns probability to negative rates. Measurement: integrate the Gaussian below zero. For a mean of 3 spikes and a spread of 3, about 16% of the mass is negative, which is impossible. Fix: choose constraints that reflect real physical facts. A non-negative rate with a known mean points to the exponential. A Gaussian assumes a free-running real number with a known spread and is fine only when the mean sits many spreads above zero.
Chapter 9
Design the best tuning curve for a single neuron, check it against a fly’s eye, then add noise and watch the answer change
A neuron in a fly’s eye reports contrast to the cells behind it. In natural scenes most contrasts are mild and a few are extreme. The neuron can respond with only a limited number of distinct voltage levels. Where should it place those levels along the contrast axis: evenly, or crowded where the scene usually sits?
Slide 25 poses exactly this, in clean notation. The stimulus x is drawn from a prior P(x), the distribution of stimuli the world delivers. Here it is a Gaussian bell curve. The neuron encodes with no noise and with discrete levels: y = f(x), where y takes one of n values. The question on the slide: what f carries the most information?
With no noise the noise entropy is zero, so I(X, Y) = H(Y), exactly as slide 26 crosses out. The task is to maximise the entropy of the output, subject to there being n levels. Chapter 8 already solved that: the uniform distribution. Every output level should be used equally often.
Translate that back to the input. If each of 20 levels must be used 5% of the time, then each level must catch exactly 5% of the stimuli. The first level takes the lowest 5% of the prior, the next level the following 5%, and so on. The boundaries between levels are the prior’s quantiles. The function that maps a stimulus to its quantile is the prior’s cumulative distribution function, the CDF: the probability that a stimulus falls below x. So the best tuning curve is the CDF of the stimulus, scaled to the output range. The lecture calls this histogram equalization, since it flattens the output histogram.
Work the Gaussian. Twenty equal slices of probability have boundaries at −1.64, −1.28, −1.04, and so on up the bell. Near the middle, one level covers a stimulus range of just 0.13 wide. At the far edge, one level covers everything below −1.64. The curve is steep where stimuli are common and flat where they are rare. It spends resolution where it pays.
Left: the tuning curve (warm) over the stimulus prior (blue). Right: how often each of 20 output levels is used. Choose a tuning, then change how widely the stimulus varies. Only one tuning keeps up.
Illustrative: 20 output levels, a Gaussian prior with the chosen spread, and a noiseless neuron, so I = H(output) with a ceiling of log2(20) = 4.32 bits. The linear tuning spans −3 to 3; the S-curves are logistic with slopes 3 and 1. The entropies are integrated numerically in your browser.
At a spread of 1.0, the linear tuning gives 3.78 bits, the steep S-curve 3.99, the gentle one 4.05, and the prior’s CDF reaches 4.32, the full log2(20). Now narrow the stimulus to 0.5. The linear curve falls to 2.81 bits, because the stimuli bunch into a few middle levels and the rest sit idle. The CDF still gives 4.32, because it re-shapes itself to the new prior. That is also what Laughlin found, next.
In 1981, Simon Laughlin measured how a blowfly’s visual neurons respond to contrast, and counted how often natural scenes contain each contrast. The slide overlays them. The measured response curve follows the cumulative distribution of light levels in nature, point by point. The lecture calls it the first major validation of Barlow’s theory. A neuron had been predicted to equalise its output histogram, and it did.
Real neurons are noisy, and slide 28 turns to Atick and Redlich (1990), who extended the theory. With noise, the objective is the full I = H(y) − H(y | x), and a new tension appears. Boosting the faint, fine details of a scene (whitening, as Chapter 10 explains) raises response entropy. But when the input is noisy, boosting also boosts the noise. The best strategy depends on the signal-to-noise ratio, SNR: how strong the signal is compared with the noise.
The slide shows the receptive fields predicted for the retina at three SNRs. With high SNR (low noise) the cell whitens, or decorrelates: it subtracts a neighbourhood average, giving a sharp centre and a negative surround. With middle SNR it whitens partly. With low SNR (high noise) it averages, or correlates: it pools over neighbours to cancel noise, giving a broad positive centre. The device builds a toy version.
Top: the filter’s gain at each spatial frequency. Bottom: the receptive field it makes, warm where it excites and blue where it inhibits. Drag the signal-to-noise ratio from high to low.
Toy model, ours: natural scenes have power S(f) = 1/(f² + 0.09) and the filter gain is √S / (S + N) with N = 1/SNR, a whitening term times a noise-averaging term. The shape follows the idea of slide 28, not its data.
At SNR 10 the gain climbs with frequency and the receptive field has a sharp warm centre with a clear blue surround: the filter erases what neighbours share. At SNR 0.1 the gain is highest at low frequency, the blue surround fades, and the field is a smooth warm blob that averages. The same cell, the same goal, a different best strategy.
Symptom: you apply a sharp edge-enhancing filter to a dim, grainy recording and the result is dominated by speckle. Measurement: estimate SNR in the frequencies you are boosting. Fix: the optimal filter backs off as the noise rises, trading sharpness for averaging. Efficient coding is not “always decorrelate”. It is “decorrelate as far as the noise lets you”.
Chapter 10
See why a population carries the most information when its neurons are independent, and what that asks of the code
Two neighbouring photoreceptors look at a photograph. Natural images are smooth, so the two points almost always have nearly the same brightness. If the first reports “fairly bright”, the second will report “fairly bright” too. The second report tells you almost nothing new. You paid for two neurons and got a little more than one neuron’s worth of news.
Slide 30 shows this as two scatter plots. On the left, the brightness of pixel i against pixel i + 1: a thin diagonal cloud, because neighbours agree. On the right, the neural responses the theory wants: a cloud that fills the whole square, with each neuron’s response telling you nothing about the other’s. The red arrow between them is the “desired encoding”.
Slide 29 states the arithmetic. For two neurons with responses r1 and r2, the joint entropy follows from the Venn diagram of Chapter 4:
H(r1, r2) = H(r1) + H(r2) − I(r1, r2)
The joint entropy is the variety of the pair. It is the sum of the two individual varieties minus the part they share. Suppose the hardware caps each neuron’s own entropy, as Chapter 7’s range limit does. Then the best you can do is make the shared part zero. The slide says it directly: this is clearly maximised when neurons are independent, I(r1, r2) = 0.
A second route, which is our own and not on slide 29, reaches the same result. Constrain the marginals, the distribution of each neuron alone, and ask for the joint distribution with the most entropy. This is a maximum entropy problem of Chapter 8 with many constraints. By the general rule the answer is an exponential of a sum of one term per neuron, which splits into a product: p(r1, r2) = p1(r1) p2(r2). A product is independence.
For two bell-shaped responses with correlation ρ, the shared information has a closed form, found by plugging the Gaussian entropies of Chapter 3 into the formula above:
At ρ = 0.5 the redundancy is 0.21 bits. At ρ = 0.9 it is −½ log2(0.19) = 1.20 bits. At ρ = 0.99 it is 2.83 bits. Highly similar neighbours waste more than two bits of the pair’s total variety. Natural images have strong correlations between neighbouring pixels, which is why this matters.
Left: two neighbouring pixels across 260 scenes. Right: the responses of two neurons that read them. Drag the correlation, then switch the code and see where the cloud goes and how many bits it carries.
Illustrative Gaussian pixel pairs, 260 seeded points. Each neuron may spend at most unit variance. Bits are the closed forms for Gaussians: pair entropy = log2(2πe) + log2(σ1σ2) + ½ log2(1 − ρout2), in bits against a 1-unit ruler.
At ρ = 0.9 the raw code carries 4.09 + ½ log2(0.19) = 2.90 bits of pair entropy, with 1.20 bits of redundancy. “Decorrelate only” turns the cloud into an upright ellipse (the redundancy is now 0) but the second neuron barely moves, so the pair entropy actually falls to 1.97 bits. The cloud is round only in the third mode, where both neurons spend their full range: 4.09 bits, the maximum. Both halves of slide 20’s advice are needed: decorrelate and use the full dynamic range.
The rotation in “sum and difference” is the same move as principal component analysis (PCA and Lecture 3): find the axes along which the data are uncorrelated. Dividing each axis by its spread is whitening. Neural populations that share signals are a theme of neural manifolds, where the shared part is exactly the redundancy measured here.
Slide 31 closes the lecture with two sentences. First, for single neurons: given a constraint on the response, such as a maximum or a mean firing rate, information transfer is maximised if the marginal response distribution is the maximum entropy distribution for that constraint, p(r) ∝ eλ f(r). That is Chapters 8 and 9. Second, for multiple neurons: information is maximised when the response distributions are independent, p(r1, …, rn) = p(r1)p(r2)…p(rn), known as redundancy reduction. That is this chapter.
Symptom: a population has zero pairwise correlation, yet a decoder reads one neuron’s output off from the other’s. Measurement: take x bell-shaped around 0 and y = x2. Their correlation is 0, since the positive and negative sides of x cancel, but y is fully determined by x, so I is large. Fix: independence means the whole joint equals the product of the marginals, a stronger test than zero correlation. For Gaussians the two coincide, so the toy device is safe. Real spike counts are not Gaussian, and the lecture’s word for the goal is “independent”.
Chapter 11
Keep the numbers, the checklist and the code, and see where the ideas lead next
Information theory counts yes/no questions. Entropy is the average number of questions needed to name an outcome. Mutual information is how many of those questions a second variable answers. KL divergence is the extra questions you ask when your model is wrong. Information can only leak along a chain of processing. Barlow’s efficient coding hypothesis then asks what a neuron with limited range should do, and the answer is: use every level equally, and do not repeat your neighbours. Maximum entropy gives the shape of that answer for each constraint.
| Idea | What it measures | Number from this lesson | Chapter |
|---|---|---|---|
| Surprise, −log2 p | Questions an outcome costs | A spike at p = 0.05 costs 4.32 bits; silence 0.07 bits | 2 |
| Entropy H(x) | Average questions to name x | The slide’s list of eight: 2.6875 bits; a sparse neuron: 0.29 bits per bin | 1, 2 |
| Differential entropy h(x) | Entropy of a density, ruler dependent | Gaussian, σ = 4 ms: 4.05 bits in ms, −5.92 in s | 3 |
| Conditional entropy H(x | y) | Doubt left after seeing y | Tone neuron (80% hit, 20% false): 0.72 of 1 bit | 4 |
| Mutual information I(x, y) | Doubt removed, symmetric | The same neuron: 0.278 bits | 4 |
| KL divergence DKL(P || Q) | Extra bits from the wrong codebook | Slide 11: 1 bit one way, infinite the other | 5 |
| Data processing inequality | No processing adds bits about S | Two 10% flip stages: 0.53 then 0.32 bits left of 1 | 6 |
| Redundancy R = 1 − I/C | Fraction of capacity wasted | Mostly-silent code R = 0.40; even use R = 0.00 (no noise) | 7 |
| Maximum entropy | Most honest distribution per constraint | Mean 3: geometric 3.25 bits against Poisson 2.79 | 8 |
| Histogram equalization | Best noiseless tuning curve | 20 levels: CDF 4.32 bits against linear 3.78 | 9 |
| Pair redundancy | Shared bits of two Gaussian neurons | ρ = 0.9: 1.20 bits; ρ = 0.99: 2.83 bits | 10 |
Every number in the table comes from a few lines of numpy. Run them to check the lesson’s claims.
pythonimport numpy as np from math import erf, sqrt def H(p): # entropy in bits; 0 log 0 = 0 p = np.asarray(p, float).ravel(); p = p[p > 0] return -(p * np.log2(p)).sum() def mutual_info(J): # J[x, y] is the joint table return H(J.sum(1)) + H(J.sum(0)) - H(J) def cond_entropy(J): # H(x | y) = H(x, y) - H(y) return H(J) - H(J.sum(0)) def kl(p, q): # D_KL(p || q): inf if q misses mass of p p, q = np.asarray(p, float), np.asarray(q, float); m = p > 0 return np.inf if (q[m] == 0).any() else (p[m] * np.log2(p[m] / q[m])).sum() J = np.array([[.4, .1], [.1, .4]]) # tone neuron: rows tone / none, cols spike / silent print(mutual_info(J), cond_entropy(J)) # 0.278 0.722 P1 = np.full(8, 1/8); P2 = np.array([0, 0, 0, 1/8, 1/8, 1/8, 1/8, 1/2]) print(kl(P2, P1), kl(P1, P2)) # 1.0 inf (slide 11) x = np.random.default_rng(0).standard_normal(200_000) # stimulus ~ Gaussian prior u = np.array([0.5 * (1 + erf(v / sqrt(2))) for v in x]) # its CDF, in [0, 1] y = np.minimum(np.floor(20 * u), 19).astype(int) # 20 output levels: histogram equalization print(H(np.bincount(y, minlength=20) / len(y))) # 4.32 = log2(20) rho = 0.9 print(-0.5 * np.log2(1 - rho**2)) # 1.20 bits of pair redundancy
Read it top to bottom. The first four functions are the whole of Chapters 2 to 5, each in one line of arithmetic. The tone neuron and the two slide distributions reproduce Chapters 4 and 5. The last block applies the CDF to a Gaussian stimulus, so that each of the 20 output levels is used 5% of the time. Swap the CDF for a straight line and the entropy falls to 3.78 bits.
This lecture follows Lecture 6’s probability review and Lecture 7, and supplies the vocabulary for everything that asks “how much does this signal tell me?” Its two loose ends, log-likelihood and maximum likelihood as KL minimisation, are the bridge to Lecture 9’s maximum likelihood and GLMs. Its decorrelation story connects to the PCA of Lecture 3. The course site posts the slides.
The slides cite Barlow (1961) for the efficient coding hypothesis, Laughlin (1981) for the blowfly data and Atick and Redlich (1990) for the noisy extension. Chapter 9’s retinal filter is our toy of that idea, not their data.
Now press Present or Teach and explain, from memory, how many yes/no questions a spike count answers, why an unequal use of response levels wastes bits, and why two neighbouring neurons that agree cost you capacity. Then go back to the neuron at the top, set 8 stimuli and a top rate of 6, and explain why raising the rate to 30 still falls short of 3 bits.