One typing session, three honest scores. A brain interface that picks the right target 9 times in 10 is not a 9 out of 10 interface in bits. Count the errors, and the number shrinks.
Learn how a brain-computer interface is scored: bits per selection, bits per second, cursor throughput, and word and character error rates. Learn why two papers can quote very different speeds for the same device.
Pick a grid size, then drag the accuracy down and the time per pick up. Watch the three bars split apart. Then we build, piece by piece, every score behind them: the bit, the Wolpaw rate, net-correct bitrate, Fitts throughput, word and character errors, and the tricks behind headline claims.
You need a feel for percentages and what a logarithm does (it counts how many times you can halve a number). We build the rest from zero.
Built from primary sources: Pandarinath et al., eLife 2017, Willett et al., Nature 2021, Willett et al., Nature 2023, Jayalath, Ballyk and Parker Jones, 2026, and MacKenzie’s Fitts’ law slides. Neuralink figures are labelled as company claims. Lesson 22 of the Brain-Computer Interfaces series.
Illustrative session, not a patient’s. Every pick takes the same time and is right with the accuracy you set. Gross is picks per second times log2 N. Wolpaw is explained in Chapter 2, net-correct in Chapter 3 (the rule Pandarinath et al. 2017 used).
Chapter 0
See why two honest papers can report very different speeds for the same kind of device, and learn the three families of score that sort them out
Imagine two headlines on the same morning. One says a man with paralysis typed 90 characters per minute with a brain implant. The other says a different man moved a cursor at 10 bits per second. Which implant is faster?
You cannot tell. One headline counts letters. The other counts bits, a unit of information we define in the next chapter. One task is typing a sentence. The other is clicking targets on a grid. The two numbers do not live on the same ruler.
This lesson builds the rulers. A brain-computer interface (BCI) turns brain activity into a command for a computer. A decoder is the program that does the turning. Measuring a BCI means scoring how well that decoder lets a person act. It sounds simple. It is full of choices, and every choice moves the number.
Almost every BCI result belongs to one of three families. Each family answers a different question, so we teach them in the order the field grew.
The first family scores selection. The person picks one item from a set, such as one cell of a grid. The score is bits per selection, then bits per second. This family contains the Wolpaw information transfer rate and the achieved bitrate of the Stanford typing studies (Chapters 1 to 3).
The second family scores pointing. The person steers a cursor to a target. The score is Fitts throughput, a measure borrowed from the study of mice and touch screens (Chapter 4).
The third family scores language. The person types or speaks, and we count how many characters or words come out wrong. The scores are the character error rate (CER) and the word error rate (WER), plus speed in characters or words per minute (Chapters 5 and 6).
Here is the failure mode, told as a story. The symptom: a paper reports 4 bits per second, a company reports 10, and a reader asks if the company is 2.5 times better. The measurement: open both methods sections and list four things: the formula, the task, whether errors were counted, and which stretch of the session was reported. The fix: put both results on the same formula and the same task before you compare.
You already met this in the hero. Drag the accuracy down to 70% and the gross bar barely moves, because gross ignores errors. The other bars fall fast. Same session, same hands, same brain. Only the rule changed.
Every score in this lesson is a function. It takes a log of a session and returns a number. For the selection family the log is a list of (target shown, target chosen, time) triples. For the pointing family it is a list of (start, end point, movement time) records. For the language family it is two strings, (what was meant, what was decoded), plus a stopwatch.
Notice what the log does not contain. It does not contain the brain signal. A score judges the whole system: the electrodes, the decoder, the screen, and the person. Change any part and the score changes. That is why a good paper names the task, the person, and the day.
We rebuild every score from papers we opened ourselves. Pandarinath et al. (2017) gives the typing numbers and the bitrate rule for grid tasks. Willett et al. (2021) gives handwriting and its error rates. MacKenzie’s course slides give the Fitts formulas. A 2026 preprint from Oxford gives a new way to compare speech decoders. Where a figure comes only from a company page or the press, we mark it company claim and treat it as a claim, not a result.
One more promise. Every toy number in a device is labelled illustrative. Every number from a paper names its paper.
Chapter 1
Count how many yes or no questions it takes to find one target, and see why that count is the unit of every selection score
A friend hides a coin under one of 16 cups in a row. You may ask yes or no questions. “Is it in the left half?” Then “Is it in the left half of that half?” How many questions do you need to find the coin?
Four. Sixteen cups become 8, then 4, then 2, then 1. Each answer cuts the field in half. After four answers only one cup is left.
That count is the idea of a bit. One bit is the information in the answer to one fair yes or no question. Finding one of 16 equal cups takes 4 questions, so a choice among 16 is worth 4 bits.
The logarithm base 2, written log2, answers one question: how many times can I halve this number before I reach 1? Halve 16 four times and you reach 1, so log2 16 = 4. Halve 64 six times, so log2 64 = 6.
Try it on the board below. Pick a grid, press Ask a question and watch half the cells fade each time.
One cell, outlined in teal when you find it, is the target. Each question keeps the half that holds it. Change the grid size and compare the number of questions.
Illustrative toy. The target is chosen by a seeded random number, so you see the same cell on each visit. A grid of 36 needs up to 6 whole questions, while its information is log2 36 = 5.17 bits: whole questions round up.
Many typing studies use a 6 by 6 grid of letters, so N = 36. How many bits does one perfect pick carry?
log2 36 = log2 6 + log2 6 = 2.585 + 2.585 = 5.17 bits
The answer is not a whole number, and that is fine. Think of it as an average over many picks. A grid of 36 sits between a grid of 32 (5 bits) and a grid of 64 (6 bits).
Here is the failure mode. Symptom: a lab doubles the grid from 16 to 32 targets, and its bits per second rise, with no change to the decoder. Measurement: each doubling adds exactly 1 bit to every perfect pick, since log2 32 = 5 and log2 16 = 4. Fix: always report N next to a bitrate. A rate without N is half a sentence.
The data flow is small. The input is an integer from 0 to N minus 1, the index of the target. A perfect decoder returns the same integer. The information moved is a string of about log2 N yes or no answers.
pythonimport math def bits_per_pick(n_targets): return math.log2(n_targets) # halvings needed to reach one target print(bits_per_pick(16), bits_per_pick(36)) # 4.0 5.169925...
Why use a logarithm at all? Because it turns a product into a sum. Say a person makes two picks in a row, each from 4 targets. There are 4 × 4 = 16 possible pairs. Each pick carries 2 bits, and the pair carries 2 + 2 = 4 bits, which is log2 16. The bits of two independent picks simply add.
That is why a rate in bits per second is meaningful. Ten picks carry ten times the bits of one pick. Divide by the seconds, and you have a rate that does not depend on how you chopped the session into picks. A count of “picks per second” has no such property, because a pick from 4 targets and a pick from 64 are not the same amount of news.
Chapters 2 and 3 ask what happens to this clean picture when the decoder makes mistakes. The ceiling stays log2 N. The real number falls below it.
The formula B = log2 N assumes every target is equally likely. Real text breaks that rule. The letter E shows up far more often than Q. A guess of “E” carries less news than a guess of “Q”, because you expected it. When some targets are likelier, the true information per pick falls below log2 N.
Grid tasks dodge the problem. The target is drawn at random, so every cell is equally likely, and log2 N is honest. Chapter 8 returns to this when we score real language.
Chapter 2
Derive the classic formula that discounts a perfect pick for the chance of a wrong one, and see how much a 10% error rate really costs
Back to the cups. This time your friend answers some questions wrongly. Nine times in ten the answer is true, and once in ten it is a lie. Can you still find the coin? You can, but you need more questions or less certainty. The wrong answers cost you information.
A BCI is that unreliable friend. The person means one target. The decoder sometimes names another. A perfect pick carries log2 N bits. A pick that is wrong now and then carries fewer. The question is how many fewer.
Information is uncertainty removed. Before the pick, you do not know the target: the uncertainty is log2 N bits. After the pick, you still doubt the decoder. Subtract the leftover doubt and you have the bits that got through.
To compute the leftover doubt we need two assumptions. First, the decoder is right with probability P. Second, when it is wrong, it picks each of the other N minus 1 targets equally often. Under those assumptions the leftover doubt has a closed form. This is the result of Pierce (1980), made standard for BCI by Wolpaw and colleagues in 1998 and 2002.
If P = 1 the two doubt terms vanish and B = log2 N. If P = 1/N, the decoder is guessing and B = 0. Between those two ends the curve bends. Slide the accuracy below.
Each line is one grid size. The teal marker sits at your accuracy on the grid you choose. The dashed line is the ceiling for that grid.
Computed live from the Wolpaw formula, for equally likely targets and errors spread evenly. Below chance (accuracy under 1/N) the formula gives no meaning, so the line stops at zero.
Take N = 4 targets and P = 0.9. Compute each term.
A 10% error rate removed 31% of the information. The loss is more than 10% because a wrong pick does more than miss. It adds doubt about every other target.
Now the rate. If the user makes one pick every 1.5 s, there are 40 picks per minute. The information transfer rate is B times the pick rate: 1.372 × 40 = 54.9 bits per minute, or 0.91 bits per second.
Look at the 4 target line in the device, and read off five points: 25% accuracy gives 0.00 bits (a pure guess), 50% gives 0.21, 75% gives 0.79, 90% gives 1.37, 95% gives 1.63, and 100% gives 2.00.
The curve is not a straight line. Going from 50% to 75% buys 0.58 bits. Going from 90% to 100% buys 0.63 bits, with only ten points of accuracy. The last few percent are the expensive ones, and they are the most valuable. That is why labs fight for every point of accuracy near the top, and why a headline that quotes the best participant at 98% says so much less than one that quotes everyone.
Here is the failure mode. Symptom: a paper reports a high ITR from an accuracy measured on the best participant on the best day. Measurement: the formula is steep near the top. Moving from 90% to 95% on 36 targets adds about 0.4 bits per pick, so a small accuracy gain buys a big rate gain. Fix: ask which accuracy went in, and for how many trials.
A second trap hides in the assumptions. Real errors are not spread evenly. A decoder that confuses neighbouring cells makes errors that are partly predictable, so the true information can be higher than the formula says. The formula also assumes a fixed trial time and that every trial ends in a pick. A system with a backspace key or an abort button breaks that. The 2026 preprint we meet in Chapter 8 also states these assumptions plainly: equal chance for every symbol, and errors spread evenly.
Chapter 3
Run a real self-paced grid task, count right and wrong picks over one minute, and turn the count into bits per second the way the Stanford typing papers do
A woman with paralysis sits in front of a screen. A 6 by 6 grid of squares shows. One square lights up, and she steers a cursor onto it and holds it there for one second to select it. Then another square lights up. She does this for a full minute. How fast is her interface?
This is the grid task from Pandarinath et al. (2017), run with three people who had one or two arrays of 96 electrodes in the motor cortex. The task has no fixed trial clock. The person goes as fast as she can. So the clean trial structure that the Wolpaw formula expects is gone, and we need a rate that works on a stopwatch.
The authors count net correct selections: all right picks minus all wrong picks. Each wrong pick cancels one right pick, so every error must be paid back by an extra correct selection. Then they multiply by the bits one selection is worth and divide by the time.
The paper calls this a conservative measure, following the earlier method of Nuyujukian et al. (2015). The authors give one check: on an 8 target task, a net rate of one target per second is log2 7 = 2.8 bits per second.
Run a minute below. Choose a grid, set the accuracy and the time per pick, and compare the three ways of scoring the very same list of picks.
Each tick is one pick: green is right, red is wrong. The bars score the same ticks three ways. Press Run a minute again for a new random minute.
Illustrative simulation: every pick takes the same time and is right with the chance you set. The paper’s own results were 2.2 (T6), 3.7 (T5) and 1.4 (T7) bits per second on the 6 by 6 grid, and 4.16 for T5 on a 9 by 9 grid (Pandarinath et al. 2017).
Take the 6 by 6 grid, 40 picks in 60 s, 36 right and 4 wrong.
Compare. Gross counting gives 40 × 5.17 / 60 = 3.45 bits per second. The Wolpaw rate with 90% accuracy gives 4.19 × 40 / 60 = 2.79. The net rule sits at the bottom, close to Wolpaw here. With only 4 wrong picks they agree. Raise the errors in the device and they drift apart.
The hold time gives a handy sanity check, and the arithmetic is ours, not the paper’s. If each pick needs a 1 s dwell on the target, no one can pick faster than once per second. The paper’s click decoder could end a pick sooner for two of the three people, so read the ceiling as a rough guide. On the 6 by 6 grid the best possible net rate is then log2 35 = 5.13 bits per second, with zero travel time and zero errors.
T5 reached 3.7, which is 72% of that ceiling. On the 9 by 9 grid the ceiling is log2 80 = 6.32, and the 4.16 result sits at 66% of it. Such a calculation tells you how much headroom a number has before you start to believe it. A claim above the ceiling for its stated task and hold time needs a different task from the one described.
In a speller, a wrong letter forces a correction. The user must delete the letter and type again. That costs time, and the net rule charges for it: each error cancels one good pick. The Wolpaw formula does not see the backspace. It treats each trial as separate and forgets the repair.
The real-data side matters too. In the paper, the neural signal was cut into 15 ms bins. A ReFIT Kalman filter turned the bins into cursor motion (you can build one in the Kalman decoder lesson, and see ReFIT in the closed-loop lesson). For T6 and T5, a separate click decoder (a hidden Markov model) could also fire a selection. The dwell on the target was 1 s for T6 and T5, and 1.5 s for T7, who had no click decoder. Every part of that chain feeds the clock T.
Here is the failure mode. Symptom: the same person scores very differently on two keyboards. In the paper, T6 reached 23.9 correct characters per minute on a QWERTY layout and 31.6 on the OPTI-II layout, a gain of a factor of 1.3 with the same brain and decoder. Measurement: only the screen layout changed. Fix: report the layout and the task next to any bitrate.
pythonimport math def achieved_bitrate(n_targets, correct, wrong, seconds): net = max(0, correct - wrong) # each error cancels one right pick return net * math.log2(n_targets - 1) / seconds print(achieved_bitrate(36, 36, 4, 60)) # 2.735...
Chapter 4
Measure speed and accuracy in one number with effective width, and see why a fast, sloppy hand does not earn a free win
You reach for a light switch in the dark. A big switch close by is easy. A tiny switch far away is hard. Everyone knows this. Paul Fitts turned it into a law in 1954: the difficulty of a pointing move depends on how far away the target is and how small it is.
The modern form, used in human-computer interaction, is the Shannon formulation. Call the distance to the target A and its width W. The index of difficulty (ID) is the number of bits the move must resolve:
This is the same bit as in Chapter 1, now applied to a continuous move. A 400 unit move to a 60 unit target has ID = log2(400/60 + 1) = 2.94 bits. Do it in 0.9 s and the nominal throughput is 3.3 bits per second.
There is a catch. Suppose you ignore the target and slap the cursor toward it as fast as you can. MT is tiny, so TP looks huge. Half your clicks miss. The formula never noticed, because it used the target width W that the screen showed, not the width you achieved.
The fix is effective width. Take every landing point of a run, measure its spread along the line of approach, and call the standard deviation SD. Then:
The slides state the logic in four words: speed plus accuracy equals throughput. A wide spread raises We, which lowers IDe, which lowers TP. Sloppiness is charged. Try it.
The start is on the left and the shaded band is the target. Each dot is one of 25 landings. The teal bracket is the effective width the spread earns. Drag the spread and the time, or try a preset.
Illustrative units: distance A = 400, target width W = 60. The 25 landings are a fixed bell shaped sample rescaled to the spread you set. Throughput uses the effective width formulas from MacKenzie’s Fitts’ law slides; real studies compute it over many moves, never from one.
Take the presets. Fast and sloppy: SD = 40 and MT = 0.5 s.
Slow and careful: SD = 8 and MT = 1.2 s. We = 33, IDe = log2(400/33 + 1) = 3.72 bits, and TP = 3.1 bits per second. The nominal formulas said 5.9 and 2.5. They disagree wildly. The effective ones land close together, since the two people trade speed for accuracy.
The slides say We holds the central 96% of a normal spread, and that 4.133 equals √(2πe). Here is a way to see why, in our own words. Information is a width measured in bits. A flat, uniform spread of width W carries log2 W bits. A bell shaped spread with standard deviation SD carries ½ log2(2πe SD2) bits.
Set the two equal and solve. The uniform width that carries the same information as the bell is W = SD × √(2πe) = 4.133 SD. So effective width is the width of a flat target that a perfect, evenly spread user would need to land with the same scatter. This ties Fitts back to the bit of Chapter 1.
A BCI cursor moves because a decoder says so. It wobbles, overshoots and drifts. Effective width captures that wobble honestly. A grid task like Chapter 3 hides it, because a pick is either right or wrong. Neither ruler is better. They answer different questions, so do not trade them across papers.
Here is the failure mode. Symptom: a demo shows a fast cursor and quotes a big bits per second figure from the target size on screen. Measurement: count the share of landings inside the target, then compute We. If the hit rate is far from 96%, the nominal figure is off. Fix: quote throughput with effective width, over many moves.
pythonimport math, statistics as st def throughput(moves): # moves: list of (distance_moved, landing_error, seconds) a_e = st.mean(m[0] for m in moves) w_e = 4.133 * st.pstdev(m[1] for m in moves) # spread along the approach axis id_e = math.log2(a_e / w_e + 1) mt = st.mean(m[2] for m in moves) return id_e / mt # bits per second, from aggregate data
Chapter 5
Count the fewest edits that turn what the decoder wrote into what the person meant, and see why one wrong letter is a small error or a whole wrong word
A man tries to write “the class was about to begin”. The screen shows “the glass was about begin”. Is that decoder good or bad? One word is wrong. One word is missing. The rest is perfect. We need a number for that, and the number must work for any pair of texts.
For language, a rate in bits is awkward. Nobody speaks in equal-chance symbols. The field uses a simpler idea: count the mistakes. A word error rate (WER) counts mistakes in whole words. A character error rate (CER) counts mistakes in single characters. Both rest on one notion, the edit distance.
The edit distance is the smallest number of single edits that turn one text into another. There are three kinds of edit. A substitution swaps one item for another. A deletion is an item the person meant that the decoder dropped. An insertion is an extra item the decoder made up.
These names look from the side of the reference, the text the person meant. Speech recognition uses the same names. The device below does too.
Type your own decoded text below and watch the alignment change.
The person meant the sentence on top. Edit the decoded text or press a preset. Green words match. Red words are errors: solid red is a substitution or an insertion, a struck dashed word is a deletion.
The reference sentence is a toy example written for this lesson. The edit distance is the standard dynamic programme, computed live in your browser. Spaces count as characters in the CER.
Reference: the class was about to begin (N = 6 words, 28 characters with spaces). Decoded: the glass was about begin.
Now see how the two rates differ. The change “class” to “glass” costs one word out of six, which is 16.7% of the WER. In the CER it costs one character out of 28, which is 3.6%. A typo is cheap in characters and costly in words.
Before you count edits, both texts are usually cleaned. Cleaning may lower-case everything, strip punctuation, and collapse spaces. These small rules change the score. Take “don’t” and “dont”. Without cleaning they differ and count as one word error. With punctuation stripped they match.
A decoder can also be scored on a closed list of sentences or on free text. On a closed list, a sentence with one wrong letter is easy to repair by snapping to the nearest listed sentence. On free text there is no list. The two settings have the same formula and very different difficulty. A paper that gives the formula and hides the cleaning rules has told you only half the method. When you read an error rate, look for the sentence that says how the text was cleaned and what list, if any, the decoder chose from.
The trick is a table. Cell (i, j) holds the edit distance between the first i items of the reference and the first j items of the decoded text. Each cell takes the cheapest of three moves:
pythondef edit_distance(ref, hyp): # ref, hyp: lists of words or of characters D = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)] for i in range(len(ref) + 1): D[i][0] = i # delete everything for j in range(len(hyp) + 1): D[0][j] = j # insert everything for i in range(1, len(ref) + 1): for j in range(1, len(hyp) + 1): same = 0 if ref[i-1] == hyp[j-1] else 1 D[i][j] = min(D[i-1][j] + 1, # word the decoder missed D[i][j-1] + 1, # word the decoder made up D[i-1][j-1] + same) # match or swap return D[-1][-1] wer = edit_distance(ref.split(), hyp.split()) / len(ref.split()) cer = edit_distance(list(ref), list(hyp)) / len(ref)
The data flow is two strings in and one fraction out. The only knobs are what counts as an item (word or character) and how the text is cleaned first: lower case, punctuation, spaces. Two labs that clean differently report different rates for the same output.
Here is the failure mode. Symptom: a decoder shows a 3% CER yet readers find it hard to use. Measurement: compute the WER on the same text. If one wrong letter in every few words spoils each word, a 3% CER can hide a 15% WER. Fix: report both, and say which cleaning you used.
Chapter 6
Walk the typing and speech results from 2017 to 2024 and learn which unit, which setting and which vocabulary hides behind each famous number
Four papers set the pace for brain-to-text. Each reports a speed and an error figure. Read them side by side and the units change under your feet. This chapter lines them up honestly.
Pandarinath et al. measured copy typing, where the person types a given sentence, in correct characters per minute (ccpm): only characters that end up right count. The best marks were 39.2 ccpm for T5 and 31.6 for T6, both on the OPTI-II keyboard layout. Free typing, where the person writes their own words, gave T6 24.4 ccpm. The best earlier result without word completion was 9.4 ccpm.
Willett et al. asked T5, who has two 96 channel arrays, to imagine writing each letter by hand. A neural network turned the signals into letters. The headline is 90 characters per minute at 94.1% raw online accuracy. Raw means no cleanup after the fact. By the usual rule of five characters per word, 90 characters per minute is about 18 words per minute.
The same paper reports more than one error rate. The text gives a raw online CER of 5.4%, and the table lists 5.9%. With an offline language model the CER falls to 0.89% and the WER to 3.4%. A language model here is a program that knows which letter strings are likely English and fixes unlikely ones. Free typing ran at 73.8 characters per minute, with an 8.54% CER online and 2.25% with the language model.
Willett et al. (2023) decoded attempted speech from T12, a participant with ALS who cannot speak clearly. The decoder reads sounds, called phonemes, then a language model picks words. It reached 62 words per minute. The word error rate was 9.1% with a 50 word vocabulary and 23.8% with 125,000 words. The paper says natural conversation runs near 160 words per minute, and that 62 is 3.4 times the previous record of 18.
Card et al. (2024, New England Journal of Medicine) report a man with ALS who reached 99.6% accuracy on a 50 word vocabulary on the first day, and 90.2% on a 125,000 word vocabulary on the second day. The paper adds 97.5% sustained accuracy (a 2.5% word error rate over the final five evaluation sessions) at about 32 words per minute in self-paced conversation, over 248 cumulative hours across 8.4 months.
Do not rank these two speech papers by one number. They use different people, different vocabularies and different metrics: Willett reports a word error rate, and Card reports accuracy. Chapter 8 gives a way to put them on one scale.
Blue bars are numbers printed in the papers. Switch the view. The first view puts speed on one ruler, the second shows the headline error, the third shows how one paper gives several errors.
All values are printed in the cited papers (Pandarinath 2017, Willett 2021, Willett 2023). Words per minute are drawn as characters per minute at the usual five characters per word, a convention, not a measurement. Card 2024 figures come from the paper’s abstract and results.
In 2024 Willett and colleagues ran the Brain-to-Text Benchmark ’24. Teams decoded the same neural recordings and were ranked by WER on a private set of 1,200 sentences. The baseline scored 9.7% and the top entry scored 5.8%. All three top teams merged several decoders and rescored them with a fine-tuned large language model. A shared test removes the argument about vocabulary and day, which makes comparison honest. Chapter 9 shows why a test set of this size still has noise.
The 2017 paper reports both bits per second and correct characters per minute. The later handwriting and speech papers lead with characters or words per minute and an error rate. The reason is practical. Text has an obvious unit that readers understand. And language is not equal-chance: some letters and words are common, so the Wolpaw assumption of Chapter 2 breaks.
You lose something by dropping bits. Bits let you compare a grid task with a cursor task. Words per minute let you compare a speech BCI with a speaker. Neither lets you compare across both. Chapter 8 shows one attempt to build a bridge.
Symptom: a reader says handwriting is more than twice as fast as point-and-click, 90 against 39. Measurement: the 39 counts only correct characters. The 90 counts all characters typed, with a 5.4% error rate. Remove the errors and 90 becomes about 85 correct characters per minute (90 × 0.946, our illustrative conversion). The gap is still large, near 2.2 times, but the units differ. Fix: convert both to correct characters per minute before you say “twice”.
Chapter 7
Score one session nine ways, then learn the questions that turn a headline into a number you can trust, including for company claims
Picture a reporter who calls two labs. Both say their volunteer reached the best bitrate in the field. The numbers are 3.7 and 10. Neither lab is lying. Both ran a grid task. Yet they may have used different grids, different formulas, and different minutes. Here is how much room those choices give.
A bitrate comes from a chain of choices. Each one is legal. Each one moves the answer. Four matter most.
Fifth, there is the question of who checked. A peer reviewed paper has a methods section and referees. A blog post or a press release has neither. That does not make a company wrong. It means you hold its number as a claim until a method can be read.
Use the device. It scores one illustrative session of ten one-minute blocks on a 6 by 6 grid. Choose a formula and a window and watch the headline move.
The bars are the ten minutes of one session under the formula you pick. The teal line is the headline for the window you pick. The table shows all nine headlines at once.
Illustrative: ten minutes on a 6 by 6 grid, each with 38 to 46 picks and 84% to 97% accuracy drawn from a seeded random generator. This is not any person’s data. Real sessions vary in the same way, which is why the choice of window matters.
Here is a quick estimate, with illustrative numbers. Say a person’s minute-by-minute rate wobbles around a mean of 2.5 bits per second with a spread (standard deviation) of 0.3. Take 30 minutes and keep only the best. For 30 draws from a bell shaped spread, the best is typically about two standard deviations above the mean.
So the best minute lands near 2.5 + 2 × 0.3 = 3.1 bits per second. That is 24% above the average, from nothing but selection. No change of decoder, no change of brain. A record from a long session, with many minutes to choose from, is nearly always a maximum like this one. The device above shows the same effect on its ten minutes.
Companies announce numbers for good reasons: they show progress. We hold each one as a company claim until we can read how it was made. Here is the table we can build today.
| Figure | Who says it | What we could check | What is missing |
|---|---|---|---|
| 11.32 bits per second, a “record” claim | Neuralink, October 2026, as reported by a press site | The press report repeats it. We could not read the company page itself with our tools. | Task, grid, formula, session length, number of participants, independent replication |
| Median near 10 bits per second claim | Neuralink, same report | Same as above | Which participants, which days, how many |
| Previous record 10.39 claim | Neuralink, same report | Same as above | Same task and formula as 11.32? |
| About 8 bits per second, 2024 claim | Press coverage of the first participant | Not checked against a primary page | Formula and task |
| 4.16 bits per second, 9 by 9 grid | Pandarinath et al., eLife 2017, peer reviewed | Method read: log2(N−1) times net correct over time, 8 blocks over 2 days, participant T5 | Nothing the paper hides, but it is one person and one decade ago |
Neuralink says its public Webgrid game scores in bits per second from net correct targets per minute, the grid size and the number of click types. That is the same family as Chapter 3. If the claims use that rule, a bigger grid is the easy way to a bigger number, and the grid size is the first thing to ask. We state it as the company’s description, not as something we verified.
Symptom: a record number, followed by silence. Measurement: ask six questions. What formula? What grid? What window? How many people? How many days? Who else checked? Fix: a good claim answers all six in one paragraph, or points to a methods section that does.
Chapter 8
Weigh a decoder’s information by how much of what a person wants to say it can even represent, and see a 100% accurate decoder lose to a 50% accurate one
A speech BCI offers the person a menu of 50 words. It decodes them perfectly. Every word comes out right. Its accuracy is 100%. Now the person wants to say “I am hungry”, and “hungry” is not on the menu. The decoder is perfect and the person is stuck.
Accuracy and word error rate both have this blind spot. They are computed only over words the system supports. A short menu can score beautifully and still say very little of what a person means. A September 2026 preprint by Jayalath, Ballyk and Parker Jones at Oxford makes this point exact and offers a fix.
The preprint asks two questions. First: what words should the interface let the user say? Call the distribution of words a user may want to say the reference distribution. Second: how much information about those words gets through? The old scores answer only the second question, and only inside the menu.
The key quantity is lexical coverage, written C(S): the probability that a word the user wants is on the decoder’s menu S. The new score, open-vocabulary mutual information (OVMI), weighs the in-menu information by that coverage.
Look at the last bracket. It is the Wolpaw formula from Chapter 2, with V in place of N. OVMI is not a rival to the Wolpaw rate. It is the Wolpaw rate multiplied by a gate: the chance that the word was even on the menu. When coverage is 100%, the two agree.
The user may want any of 1,000 equally likely words. The teal marker is your decoder. The warm curve is its OVMI as the menu grows. The dashed line is a perfect decoder. Try the two presets.
The toy follows the preprint’s own example: 1,000 equally likely words, so coverage is menu size over 1,000. Real language is Zipfian (a few words are very common), and the preprint uses real word frequencies. Its Figure 4 estimates, from its own reference distribution, put Willett’s 125,000 word system at 72.0% and Card’s at 93.7% of the lexical information, against 2.4% to 6.7% for the 50 word systems it lists from Moses and Willett.
System A: V = 50 words, P = 100%. System B: V = 1,000 words, P = 50%. Reference: 1,000 equally likely words.
Accuracy ranks A over B, 100% against 50%. OVMI ranks B over A by a factor of 14. The preprint uses this same pair as its toy case.
The toy gives every word the same chance. Real speech does not. A few words, like “the” and “I”, make up a large share of everything we say, and a long tail of rare words makes up the rest. Language with this shape is called Zipfian.
Two effects follow, and the preprint tracks both. A menu of the commonest words covers more speech than its size suggests, so coverage grows quickly at first and slowly later. Yet the Wolpaw term log2 V overstates the information of a menu where some words are common, because a common word is less news. The preprint replaces log2 V with the actual entropy of the words on the menu.
The paper is a preprint. Its simple form assumes errors spread evenly over the menu, and that a word outside the menu produces no output. Treat its numbers as a careful first proposal for a common ruler, not as settled fact.
Recall Chapter 6. Willett reported a 9.1% WER on 50 words and 23.8% on 125,000. The smaller error belongs to the system that could say far less. OVMI puts both on one scale. The preprint reports that the big historical jump among invasive systems came from larger vocabularies, after decoding quality inside small menus had already saturated.
The preprint also shows that selecting the vocabulary to maximise OVMI improved accuracy by up to 16.3% relative, in three speech domains. We take that as the authors’ result. It is a preprint, not yet peer reviewed.
Here is the failure mode. Symptom: a decoder with a tiny vocabulary reports a very low WER. Measurement: ask what share of everyday speech uses only those words. If it is 5%, the WER describes 5% of what the person wants to say. Fix: report vocabulary size with every error rate, and add coverage against a stated reference text.
Chapter 9
Repeat one evaluation many times to see the noise in a word error rate, and learn the rules that keep an evaluation honest
Two teams test the same kind of decoder. Team A reports a 10% word error rate. Team B reports 9%. Is B better? That depends on a question the papers rarely answer in the headline: how many sentences did each team test?
An error rate from a finite test is an estimate. Run the same decoder on another set of sentences and you get a slightly different number. The fewer the sentences, the more the number jumps. A fair test says how big the jump is.
Suppose each word is wrong with probability p, independently. Test W words. The measured error rate has a spread, the standard error, of about the square root of p(1 − p)/W.
Worked example, with illustrative numbers: true p = 10%, 10 words per sentence.
So at 50 sentences, 9% against 10% is noise. At 1,200 sentences, a gap of 3.9 points, like the benchmark’s 5.8% against 9.7%, is far outside the noise. A gap of 0.3 points is not.
Press the button below to see this for yourself. The device repeats the whole evaluation 200 times and draws where the measured rate lands each time.
Each bar counts how many of 200 repeated tests measured that error rate. The vertical line is the true rate. The shaded band holds 95% of the repeats. Drag the test size.
Illustrative simulation: 10 words per sentence, a seeded random generator, 200 repeats. With the toggle on, some sentences are harder than others, which makes real test sets noisier than the plain binomial formula says. The slider is logarithmic: 10 to 1,000 sentences.
In an online test the person watches the output and the decoder runs live. The person can adapt, and the clock runs. In an offline test the lab replays recorded brain signals through a model later. Offline models may look at future signal, run slowly, and use a large language model. Offline numbers are valuable for research, but they are not what the person experienced. Willett 2021 shows both: 5.4% CER online and 0.89% offline with a language model.
The 2021 paper also tells us how its test was built: five days of evaluation, with four blocks of 7 to 10 sentences each day. The 2023 speech paper pooled 80 trials a day for the large vocabulary. Small, careful sets, and the numbers carry that noise.
When you compare two decoders, test both on the very same sentences. Some sentences are simply harder. If the same hard sentences hit both decoders, that difficulty cancels in the difference between them.
Compare two separate error rates from two different test sets, and the sentence difficulty adds noise to each number. A paired test, which looks at the sentences where the two decoders disagree, gives a tighter answer from the same amount of data. This is one reason a shared benchmark with one private test set is so useful: every team faces the same sentences.
Brain signals drift. The same electrode may hear a slightly different set of neurons next week. A decoder trained on Monday may do worse on Friday. So a test on sentences from the same hour as the training data is a kind. A test on a later day is a truer one.
The Card paper makes this a feature of the score: it reports day one and day two separately, with the vocabulary growing from 50 to 125,000 words. Calibration time belongs in the report, next to the error rate.
Symptom: a new decoder beats the old one by half a point on a test of 100 sentences. Measurement: the repeat device shows a range several points wide at that size, and the gap sits inside it. Fix: report the number of sentences and an interval, and use a shared test set when you can, as the 2024 benchmark did.
Chapter 10
Keep the terms, the numbers, the checklist and the code in one place, then read the next BCI headline with new eyes
Every BCI score answers one question about one task. Bits per selection ask how much a choice carries. The Wolpaw rate discounts for errors. Achieved bitrate counts net-correct picks against a clock. Fitts throughput prices a steered move in effective bits. Error rates count the edits a text needs. Open-vocabulary information asks how much of the person’s language the system can even say. A fair claim names the task, the formula, the window, the people and the test size.
| Term | Formula | Use it when |
|---|---|---|
| Bits per pick | log2 N | N equally likely targets, no errors |
| Wolpaw bits | log2 N + P log2 P + (1−P) log2((1−P)/(N−1)) | Fixed trials, equal chance, errors spread evenly |
| Achieved bitrate | (C − W) log2(N−1) / T | Self-paced grid task (Pandarinath 2017 rule) |
| Fitts throughput | log2(Ae/We + 1) / MT, with We = 4.133 SD | Cursor pointing, over many moves |
| WER and CER | (S + D + I) / N over words or characters | Text out; report both |
| OVMI | C(S) × in-menu information | Comparing decoders with different vocabularies |
| What | Number | Source and status |
|---|---|---|
| Grid bitrate, 6 by 6 | T6 2.2, T5 3.7, T7 1.4 bits per second | Pandarinath et al., eLife 2017, peer reviewed |
| Grid bitrate, 9 by 9 | T5 4.16 bits per second | Same paper |
| Copy typing | T5 39.2, T6 31.6 correct characters per minute | Same paper |
| Handwriting | 90 characters per minute, CER 5.4% raw, 0.89% with offline language model | Willett et al., Nature 2021, peer reviewed |
| Speech, large vocabulary | 62 words per minute, WER 23.8% (125,000 words), 9.1% (50 words) | Willett et al., Nature 2023, peer reviewed |
| Speech, calibration | 99.6% day one (50 words), 90.2% day two (125,000 words) | Card et al., NEJM 2024, peer reviewed |
| Benchmark | WER 9.7% baseline, 5.8% best, 1,200 private sentences | Willett et al., arXiv 2412.17227 |
| Open vocabulary toy | 0.28 bits (50 words, 100%) against 3.98 bits (1,000 words, 50%) | Jayalath et al., arXiv 2609.02887, preprint |
| Cursor record | 11.32 bits per second, median near 10 claim | Neuralink, October 2026, read through a press report only |
pythonimport math, statistics as st def wolpaw_bits(n, p): # bits per pick, equal chance, errors spread evenly if p >= 1: return math.log2(n) if p <= 1 / n: return 0.0 return math.log2(n) + p * math.log2(p) + (1 - p) * math.log2((1 - p) / (n - 1)) def achieved_bitrate(n, correct, wrong, seconds): # Pandarinath 2017 rule return max(0, correct - wrong) * math.log2(n - 1) / seconds def fitts_throughput(moves): # moves: (distance_moved, landing_error, seconds) a_e = st.mean(m[0] for m in moves) w_e = 4.133 * st.pstdev(m[1] for m in moves) return math.log2(a_e / w_e + 1) / st.mean(m[2] for m in moves) def edit_distance(ref, hyp): # lists of words or characters prev = list(range(len(hyp) + 1)) for i, r in enumerate(ref, 1): cur = [i] for j, h in enumerate(hyp, 1): cur.append(min(prev[j] + 1, cur[j-1] + 1, prev[j-1] + (r != h))) prev = cur return prev[-1] def wer(ref, hyp): return edit_distance(ref.split(), hyp.split()) / len(ref.split()) def cer(ref, hyp): return edit_distance(list(ref), list(hyp)) / len(ref) def ovmi_uniform(v, p, lexicon): # menu of v words out of `lexicon` equally likely words return (v / lexicon) * wolpaw_bits(v, p) print(round(wolpaw_bits(4, 0.9), 3), round(ovmi_uniform(50, 1, 1000), 2), round(ovmi_uniform(1000, 0.5, 1000), 2)) # 1.373 0.28 3.98
Measuring comes after decoding. The decoders you scored here come from earlier lessons: linear decoders, the Kalman decoder (see also the Kalman filter itself), and closed-loop refitting, which is the ReFIT step behind the 2017 numbers. The P300 speller lesson is the classic place the Wolpaw rate is used. Drift and recalibration explains why a score from Monday may not hold on Friday. For the information theory underneath, try the information theory workbook.
Now press Present or Teach and explain, out loud, how one session becomes three different bitrates and why the error rate in a paper depends on the setting. Then go back to the instrument at the top, drag the accuracy down to 70%, and say in one sentence which bar falls fastest and why.