Measuring a BCI

One typing session, three honest scores. A brain interface that picks the right target 9 times in 10 is not a 9 out of 10 interface in bits. Count the errors, and the number shrinks.

Learn how a brain-computer interface is scored: bits per selection, bits per second, cursor throughput, and word and character error rates. Learn why two papers can quote very different speeds for the same device.

Pick a grid size, then drag the accuracy down and the time per pick up. Watch the three bars split apart. Then we build, piece by piece, every score behind them: the bit, the Wolpaw rate, net-correct bitrate, Fitts throughput, word and character errors, and the tricks behind headline claims.

You need a feel for percentages and what a logarithm does (it counts how many times you can halve a number). We build the rest from zero.

Built from primary sources: Pandarinath et al., eLife 2017, Willett et al., Nature 2021, Willett et al., Nature 2023, Jayalath, Ballyk and Parker Jones, 2026, and MacKenzie’s Fitts’ law slides. Neuralink figures are labelled as company claims. Lesson 22 of the Brain-Computer Interfaces series.

Score one session

90%
1.5 s

Illustrative session, not a patient’s. Every pick takes the same time and is right with the accuracy you set. Gross is picks per second times log2 N. Wolpaw is explained in Chapter 2, net-correct in Chapter 3 (the rule Pandarinath et al. 2017 used).

Chapter 0

Why One Number Is Never Enough

See why two honest papers can report very different speeds for the same kind of device, and learn the three families of score that sort them out

Imagine two headlines on the same morning. One says a man with paralysis typed 90 characters per minute with a brain implant. The other says a different man moved a cursor at 10 bits per second. Which implant is faster?

You cannot tell. One headline counts letters. The other counts bits, a unit of information we define in the next chapter. One task is typing a sentence. The other is clicking targets on a grid. The two numbers do not live on the same ruler.

This lesson builds the rulers. A brain-computer interface (BCI) turns brain activity into a command for a computer. A decoder is the program that does the turning. Measuring a BCI means scoring how well that decoder lets a person act. It sounds simple. It is full of choices, and every choice moves the number.

Three families of score

Almost every BCI result belongs to one of three families. Each family answers a different question, so we teach them in the order the field grew.

The first family scores selection. The person picks one item from a set, such as one cell of a grid. The score is bits per selection, then bits per second. This family contains the Wolpaw information transfer rate and the achieved bitrate of the Stanford typing studies (Chapters 1 to 3).

The second family scores pointing. The person steers a cursor to a target. The score is Fitts throughput, a measure borrowed from the study of mice and touch screens (Chapter 4).

The third family scores language. The person types or speaks, and we count how many characters or words come out wrong. The scores are the character error rate (CER) and the word error rate (WER), plus speed in characters or words per minute (Chapters 5 and 6).

Think of three rulers. A selection score is a ruler for information. A pointing score is a ruler for steering. A language score is a ruler for text. Each ruler is fine on its own job. Trouble starts when a headline lays one ruler against another job.

What goes wrong, in three steps

Here is the failure mode, told as a story. The symptom: a paper reports 4 bits per second, a company reports 10, and a reader asks if the company is 2.5 times better. The measurement: open both methods sections and list four things: the formula, the task, whether errors were counted, and which stretch of the session was reported. The fix: put both results on the same formula and the same task before you compare.

You already met this in the hero. Drag the accuracy down to 70% and the gross bar barely moves, because gross ignores errors. The other bars fall fast. Same session, same hands, same brain. Only the rule changed.

The data flow of a score

Every score in this lesson is a function. It takes a log of a session and returns a number. For the selection family the log is a list of (target shown, target chosen, time) triples. For the pointing family it is a list of (start, end point, movement time) records. For the language family it is two strings, (what was meant, what was decoded), plus a stopwatch.

Notice what the log does not contain. It does not contain the brain signal. A score judges the whole system: the electrodes, the decoder, the screen, and the person. Change any part and the score changes. That is why a good paper names the task, the person, and the day.

Where the sources come from

We rebuild every score from papers we opened ourselves. Pandarinath et al. (2017) gives the typing numbers and the bitrate rule for grid tasks. Willett et al. (2021) gives handwriting and its error rates. MacKenzie’s course slides give the Fitts formulas. A 2026 preprint from Oxford gives a new way to compare speech decoders. Where a figure comes only from a company page or the press, we mark it company claim and treat it as a claim, not a result.

One more promise. Every toy number in a device is labelled illustrative. Every number from a paper names its paper.

The road

  1. The bit: how much information one choice among N targets can carry.
  2. The Wolpaw rate: how errors shrink the bits, with the formula derived.
  3. Achieved bitrate: time, net-correct picks, and the Stanford grid rule.
  4. Fitts throughput: cursors, effective width, and why a sloppy fast hand does not win.
  5. Word and character errors: edit distance, worked by hand and in code.
  6. The published records: typing and speech numbers, read with care.
  7. Why claims differ: nine ways to score one session, and company claims.
  8. Open vocabulary: what accuracy hides when a decoder knows few words.
  9. A fair test: how many sentences make a number trustworthy.
  10. The Field Guide: numbers, checklist, code and sources.
Paper A reports a typing speed in characters per minute. Paper B reports a cursor speed in bits per second. What is the most honest way to compare them?

Chapter 1

The Bit: What One Choice Is Worth

Count how many yes or no questions it takes to find one target, and see why that count is the unit of every selection score

A friend hides a coin under one of 16 cups in a row. You may ask yes or no questions. “Is it in the left half?” Then “Is it in the left half of that half?” How many questions do you need to find the coin?

Four. Sixteen cups become 8, then 4, then 2, then 1. Each answer cuts the field in half. After four answers only one cup is left.

That count is the idea of a bit. One bit is the information in the answer to one fair yes or no question. Finding one of 16 equal cups takes 4 questions, so a choice among 16 is worth 4 bits.

The logarithm is just a halving counter

The logarithm base 2, written log2, answers one question: how many times can I halve this number before I reach 1? Halve 16 four times and you reach 1, so log2 16 = 4. Halve 64 six times, so log2 64 = 6.

B
Bits carried by one perfect selection. It is a count of halvings.
N
The number of targets the person could have chosen, all equally likely.

Try it on the board below. Pick a grid, press Ask a question and watch half the cells fade each time.

Find the hidden target

One cell, outlined in teal when you find it, is the target. Each question keeps the half that holds it. Change the grid size and compare the number of questions.

Illustrative toy. The target is chosen by a seeded random number, so you see the same cell on each visit. A grid of 36 needs up to 6 whole questions, while its information is log2 36 = 5.17 bits: whole questions round up.

Worked example: a 36 cell speller grid

Many typing studies use a 6 by 6 grid of letters, so N = 36. How many bits does one perfect pick carry?

log2 36 = log2 6 + log2 6 = 2.585 + 2.585 = 5.17 bits

The answer is not a whole number, and that is fine. Think of it as an average over many picks. A grid of 36 sits between a grid of 32 (5 bits) and a grid of 64 (6 bits).

Why the bit matters for a BCI

Here is the failure mode. Symptom: a lab doubles the grid from 16 to 32 targets, and its bits per second rise, with no change to the decoder. Measurement: each doubling adds exactly 1 bit to every perfect pick, since log2 32 = 5 and log2 16 = 4. Fix: always report N next to a bitrate. A rate without N is half a sentence.

The data flow is small. The input is an integer from 0 to N minus 1, the index of the target. A perfect decoder returns the same integer. The information moved is a string of about log2 N yes or no answers.

pythonimport math

def bits_per_pick(n_targets):
    return math.log2(n_targets)      # halvings needed to reach one target

print(bits_per_pick(16), bits_per_pick(36))   # 4.0  5.169925...

Why bits add up when choices stack

Why use a logarithm at all? Because it turns a product into a sum. Say a person makes two picks in a row, each from 4 targets. There are 4 × 4 = 16 possible pairs. Each pick carries 2 bits, and the pair carries 2 + 2 = 4 bits, which is log2 16. The bits of two independent picks simply add.

That is why a rate in bits per second is meaningful. Ten picks carry ten times the bits of one pick. Divide by the seconds, and you have a rate that does not depend on how you chopped the session into picks. A count of “picks per second” has no such property, because a pick from 4 targets and a pick from 64 are not the same amount of news.

Chapters 2 and 3 ask what happens to this clean picture when the decoder makes mistakes. The ceiling stays log2 N. The real number falls below it.

The hidden assumption: all targets equally likely

The formula B = log2 N assumes every target is equally likely. Real text breaks that rule. The letter E shows up far more often than Q. A guess of “E” carries less news than a guess of “Q”, because you expected it. When some targets are likelier, the true information per pick falls below log2 N.

Grid tasks dodge the problem. The target is drawn at random, so every cell is equally likely, and log2 N is honest. Chapter 8 returns to this when we score real language.

A perfect decoder chooses among 32 equally likely targets. How many bits does one correct pick carry?

Chapter 2

The Wolpaw Rate: Errors Eat Bits

Derive the classic formula that discounts a perfect pick for the chance of a wrong one, and see how much a 10% error rate really costs

Back to the cups. This time your friend answers some questions wrongly. Nine times in ten the answer is true, and once in ten it is a lie. Can you still find the coin? You can, but you need more questions or less certainty. The wrong answers cost you information.

A BCI is that unreliable friend. The person means one target. The decoder sometimes names another. A perfect pick carries log2 N bits. A pick that is wrong now and then carries fewer. The question is how many fewer.

From uncertainty to the formula

Information is uncertainty removed. Before the pick, you do not know the target: the uncertainty is log2 N bits. After the pick, you still doubt the decoder. Subtract the leftover doubt and you have the bits that got through.

To compute the leftover doubt we need two assumptions. First, the decoder is right with probability P. Second, when it is wrong, it picks each of the other N minus 1 targets equally often. Under those assumptions the leftover doubt has a closed form. This is the result of Pierce (1980), made standard for BCI by Wolpaw and colleagues in 1998 and 2002.

B
Bits that get through in one selection, on average.
log2 N
The uncertainty before the pick, and the ceiling: a perfect decoder reaches it.
P log2 P
A negative number. It is the doubt caused by the right picks, because even a right pick is not certain.
(1 − P) log2(…)
A negative number. It is the doubt caused by wrong picks, spread over N minus 1 other targets.

If P = 1 the two doubt terms vanish and B = log2 N. If P = 1/N, the decoder is guessing and B = 0. Between those two ends the curve bends. Slide the accuracy below.

Bend the Wolpaw curve

Each line is one grid size. The teal marker sits at your accuracy on the grid you choose. The dashed line is the ceiling for that grid.

90%

Computed live from the Wolpaw formula, for equally likely targets and errors spread evenly. Below chance (accuracy under 1/N) the formula gives no meaning, so the line stops at zero.

Worked example, every step

Take N = 4 targets and P = 0.9. Compute each term.

  1. Ceiling. log2 4 = 2.000 bits.
  2. Right-pick doubt. 0.9 × log2 0.9 = 0.9 × (−0.152) = −0.137.
  3. Wrong-pick doubt. 0.1 × log2 (0.1 / 3) = 0.1 × (−4.907) = −0.491.
  4. Sum. B = 2.000 − 0.137 − 0.491 = about 1.37 bits.

A 10% error rate removed 31% of the information. The loss is more than 10% because a wrong pick does more than miss. It adds doubt about every other target.

Now the rate. If the user makes one pick every 1.5 s, there are 40 picks per minute. The information transfer rate is B times the pick rate: 1.372 × 40 = 54.9 bits per minute, or 0.91 bits per second.

Reading the curve

Look at the 4 target line in the device, and read off five points: 25% accuracy gives 0.00 bits (a pure guess), 50% gives 0.21, 75% gives 0.79, 90% gives 1.37, 95% gives 1.63, and 100% gives 2.00.

The curve is not a straight line. Going from 50% to 75% buys 0.58 bits. Going from 90% to 100% buys 0.63 bits, with only ten points of accuracy. The last few percent are the expensive ones, and they are the most valuable. That is why labs fight for every point of accuracy near the top, and why a headline that quotes the best participant at 98% says so much less than one that quotes everyone.

When the formula misleads

Here is the failure mode. Symptom: a paper reports a high ITR from an accuracy measured on the best participant on the best day. Measurement: the formula is steep near the top. Moving from 90% to 95% on 36 targets adds about 0.4 bits per pick, so a small accuracy gain buys a big rate gain. Fix: ask which accuracy went in, and for how many trials.

A second trap hides in the assumptions. Real errors are not spread evenly. A decoder that confuses neighbouring cells makes errors that are partly predictable, so the true information can be higher than the formula says. The formula also assumes a fixed trial time and that every trial ends in a pick. A system with a backspace key or an abort button breaks that. The 2026 preprint we meet in Chapter 8 also states these assumptions plainly: equal chance for every symbol, and errors spread evenly.

A decoder on 4 targets goes from 100% to 90% accuracy. By what share do its bits per pick fall?

Chapter 3

Achieved Bitrate: Net-Correct Picks per Second

Run a real self-paced grid task, count right and wrong picks over one minute, and turn the count into bits per second the way the Stanford typing papers do

A woman with paralysis sits in front of a screen. A 6 by 6 grid of squares shows. One square lights up, and she steers a cursor onto it and holds it there for one second to select it. Then another square lights up. She does this for a full minute. How fast is her interface?

This is the grid task from Pandarinath et al. (2017), run with three people who had one or two arrays of 96 electrodes in the motor cortex. The task has no fixed trial clock. The person goes as fast as she can. So the clean trial structure that the Wolpaw formula expects is gone, and we need a rate that works on a stopwatch.

The rule the paper used

The authors count net correct selections: all right picks minus all wrong picks. Each wrong pick cancels one right pick, so every error must be paid back by an extra correct selection. Then they multiply by the bits one selection is worth and divide by the time.

R
The achieved bitrate in bits per second.
C − W
Correct selections minus incorrect selections, the net count.
log2(N − 1)
Bits credited to one net selection. The paper uses N minus 1, not N.
T
Seconds on the clock, including the one second hold on each target.

The paper calls this a conservative measure, following the earlier method of Nuyujukian et al. (2015). The authors give one check: on an 8 target task, a net rate of one target per second is log2 7 = 2.8 bits per second.

Run a minute below. Choose a grid, set the accuracy and the time per pick, and compare the three ways of scoring the very same list of picks.

Run a minute of grid picks

Each tick is one pick: green is right, red is wrong. The bars score the same ticks three ways. Press Run a minute again for a new random minute.

90%
1.5 s

Illustrative simulation: every pick takes the same time and is right with the chance you set. The paper’s own results were 2.2 (T6), 3.7 (T5) and 1.4 (T7) bits per second on the 6 by 6 grid, and 4.16 for T5 on a 9 by 9 grid (Pandarinath et al. 2017).

Worked example: one minute, every step

Take the 6 by 6 grid, 40 picks in 60 s, 36 right and 4 wrong.

  1. Net count. 36 − 4 = 32 net correct selections.
  2. Bits per net selection. log2 35 = 5.129 bits.
  3. Total bits. 32 × 5.129 = 164.1 bits.
  4. Rate. 164.1 / 60 = 2.74 bits per second.

Compare. Gross counting gives 40 × 5.17 / 60 = 3.45 bits per second. The Wolpaw rate with 90% accuracy gives 4.19 × 40 / 60 = 2.79. The net rule sits at the bottom, close to Wolpaw here. With only 4 wrong picks they agree. Raise the errors in the device and they drift apart.

A ceiling set by the clock

The hold time gives a handy sanity check, and the arithmetic is ours, not the paper’s. If each pick needs a 1 s dwell on the target, no one can pick faster than once per second. The paper’s click decoder could end a pick sooner for two of the three people, so read the ceiling as a rough guide. On the 6 by 6 grid the best possible net rate is then log2 35 = 5.13 bits per second, with zero travel time and zero errors.

T5 reached 3.7, which is 72% of that ceiling. On the 9 by 9 grid the ceiling is log2 80 = 6.32, and the 4.16 result sits at 66% of it. Such a calculation tells you how much headroom a number has before you start to believe it. A claim above the ceiling for its stated task and hold time needs a different task from the one described.

Why net-correct fits a self-paced task

In a speller, a wrong letter forces a correction. The user must delete the letter and type again. That costs time, and the net rule charges for it: each error cancels one good pick. The Wolpaw formula does not see the backspace. It treats each trial as separate and forgets the repair.

The real-data side matters too. In the paper, the neural signal was cut into 15 ms bins. A ReFIT Kalman filter turned the bins into cursor motion (you can build one in the Kalman decoder lesson, and see ReFIT in the closed-loop lesson). For T6 and T5, a separate click decoder (a hidden Markov model) could also fire a selection. The dwell on the target was 1 s for T6 and T5, and 1.5 s for T7, who had no click decoder. Every part of that chain feeds the clock T.

Here is the failure mode. Symptom: the same person scores very differently on two keyboards. In the paper, T6 reached 23.9 correct characters per minute on a QWERTY layout and 31.6 on the OPTI-II layout, a gain of a factor of 1.3 with the same brain and decoder. Measurement: only the screen layout changed. Fix: report the layout and the task next to any bitrate.

pythonimport math

def achieved_bitrate(n_targets, correct, wrong, seconds):
    net = max(0, correct - wrong)                 # each error cancels one right pick
    return net * math.log2(n_targets - 1) / seconds

print(achieved_bitrate(36, 36, 4, 60))   # 2.735...
In a one minute grid task there are 30 right picks and 10 wrong picks on a 6 by 6 grid. What is the net count the rule uses?

Chapter 4

Fitts Throughput: Scoring a Steered Cursor

Measure speed and accuracy in one number with effective width, and see why a fast, sloppy hand does not earn a free win

You reach for a light switch in the dark. A big switch close by is easy. A tiny switch far away is hard. Everyone knows this. Paul Fitts turned it into a law in 1954: the difficulty of a pointing move depends on how far away the target is and how small it is.

The modern form, used in human-computer interaction, is the Shannon formulation. Call the distance to the target A and its width W. The index of difficulty (ID) is the number of bits the move must resolve:

ID
How hard the move was, in bits. Far and small means high.
A
Distance from the start to the target centre.
W
Width of the target, along the direction of motion.
TP
Throughput in bits per second: difficulty achieved per second of movement.
MT
Movement time in seconds, from start to selection.

This is the same bit as in Chapter 1, now applied to a continuous move. A 400 unit move to a 60 unit target has ID = log2(400/60 + 1) = 2.94 bits. Do it in 0.9 s and the nominal throughput is 3.3 bits per second.

The cheat, and the fix

There is a catch. Suppose you ignore the target and slap the cursor toward it as fast as you can. MT is tiny, so TP looks huge. Half your clicks miss. The formula never noticed, because it used the target width W that the screen showed, not the width you achieved.

The fix is effective width. Take every landing point of a run, measure its spread along the line of approach, and call the standard deviation SD. Then:

We
The width of the spread you actually produced. In MacKenzie’s slides it holds the central 96% of landings, which is a 4% error rate. The constant 4.133 is √(2πe).
IDe
The difficulty you really achieved, not the difficulty you were shown.
Ae
The average distance you actually moved. This stops anyone from stopping short.

The slides state the logic in four words: speed plus accuracy equals throughput. A wide spread raises We, which lowers IDe, which lowers TP. Sloppiness is charged. Try it.

Spread the landings

The start is on the left and the shaded band is the target. Each dot is one of 25 landings. The teal bracket is the effective width the spread earns. Drag the spread and the time, or try a preset.

15
0.90 s

Illustrative units: distance A = 400, target width W = 60. The 25 landings are a fixed bell shaped sample rescaled to the spread you set. Throughput uses the effective width formulas from MacKenzie’s Fitts’ law slides; real studies compute it over many moves, never from one.

Worked example, every step

Take the presets. Fast and sloppy: SD = 40 and MT = 0.5 s.

  1. Effective width. We = 4.133 × 40 = 165.
  2. Effective difficulty. IDe = log2(400/165 + 1) = 1.77 bits.
  3. Throughput. 1.77 / 0.5 = 3.5 bits per second. The nominal formula said 2.94 / 0.5 = 5.9.

Slow and careful: SD = 8 and MT = 1.2 s. We = 33, IDe = log2(400/33 + 1) = 3.72 bits, and TP = 3.1 bits per second. The nominal formulas said 5.9 and 2.5. They disagree wildly. The effective ones land close together, since the two people trade speed for accuracy.

Where the constant 4.133 comes from

The slides say We holds the central 96% of a normal spread, and that 4.133 equals √(2πe). Here is a way to see why, in our own words. Information is a width measured in bits. A flat, uniform spread of width W carries log2 W bits. A bell shaped spread with standard deviation SD carries ½ log2(2πe SD2) bits.

Set the two equal and solve. The uniform width that carries the same information as the bell is W = SD × √(2πe) = 4.133 SD. So effective width is the width of a flat target that a perfect, evenly spread user would need to land with the same scatter. This ties Fitts back to the bit of Chapter 1.

What this means for BCI cursors

A BCI cursor moves because a decoder says so. It wobbles, overshoots and drifts. Effective width captures that wobble honestly. A grid task like Chapter 3 hides it, because a pick is either right or wrong. Neither ruler is better. They answer different questions, so do not trade them across papers.

Here is the failure mode. Symptom: a demo shows a fast cursor and quotes a big bits per second figure from the target size on screen. Measurement: count the share of landings inside the target, then compute We. If the hit rate is far from 96%, the nominal figure is off. Fix: quote throughput with effective width, over many moves.

pythonimport math, statistics as st

def throughput(moves):               # moves: list of (distance_moved, landing_error, seconds)
    a_e = st.mean(m[0] for m in moves)
    w_e = 4.133 * st.pstdev(m[1] for m in moves)   # spread along the approach axis
    id_e = math.log2(a_e / w_e + 1)
    mt = st.mean(m[2] for m in moves)
    return id_e / mt                  # bits per second, from aggregate data
A user hurries and lands all over the screen, so the spread of landings is large. What happens to effective throughput, if the movement time stays the same?

Chapter 5

Word and Character Errors

Count the fewest edits that turn what the decoder wrote into what the person meant, and see why one wrong letter is a small error or a whole wrong word

A man tries to write “the class was about to begin”. The screen shows “the glass was about begin”. Is that decoder good or bad? One word is wrong. One word is missing. The rest is perfect. We need a number for that, and the number must work for any pair of texts.

For language, a rate in bits is awkward. Nobody speaks in equal-chance symbols. The field uses a simpler idea: count the mistakes. A word error rate (WER) counts mistakes in whole words. A character error rate (CER) counts mistakes in single characters. Both rest on one notion, the edit distance.

Edit distance, in plain words

The edit distance is the smallest number of single edits that turn one text into another. There are three kinds of edit. A substitution swaps one item for another. A deletion is an item the person meant that the decoder dropped. An insertion is an extra item the decoder made up.

These names look from the side of the reference, the text the person meant. Speech recognition uses the same names. The device below does too.

WER
Word error rate. The same formula over characters gives CER.
S
Substitutions: words swapped for a wrong word.
D
Deletions: words the person meant that the decoder dropped.
I
Insertions: extra words the decoder added.
N
Number of words in the reference, the text the person meant.

Type your own decoded text below and watch the alignment change.

Score a decoded sentence

The person meant the sentence on top. Edit the decoded text or press a preset. Green words match. Red words are errors: solid red is a substitution or an insertion, a struck dashed word is a deletion.

Meant
Decoded (you can edit it)
Alignment

The reference sentence is a toy example written for this lesson. The edit distance is the standard dynamic programme, computed live in your browser. Spaces count as characters in the CER.

Worked example, every step

Reference: the class was about to begin (N = 6 words, 28 characters with spaces). Decoded: the glass was about begin.

  1. Align the words. the=the, class→glass, was=was, about=about, to missing, begin=begin.
  2. Count. S = 1 (class became glass), D = 1 (to was dropped), I = 0.
  3. WER. (1 + 1 + 0) / 6 = 33.3%.
  4. CER. One letter changed (c to g), and the three characters of “to ” are gone: 4 edits over 28 characters, 14.3%.

Now see how the two rates differ. The change “class” to “glass” costs one word out of six, which is 16.7% of the WER. In the CER it costs one character out of 28, which is 3.6%. A typo is cheap in characters and costly in words.

Clean the text first, and say how

Before you count edits, both texts are usually cleaned. Cleaning may lower-case everything, strip punctuation, and collapse spaces. These small rules change the score. Take “don’t” and “dont”. Without cleaning they differ and count as one word error. With punctuation stripped they match.

A decoder can also be scored on a closed list of sentences or on free text. On a closed list, a sentence with one wrong letter is easy to repair by snapping to the nearest listed sentence. On free text there is no list. The two settings have the same formula and very different difficulty. A paper that gives the formula and hides the cleaning rules has told you only half the method. When you read an error rate, look for the sentence that says how the text was cleaned and what list, if any, the decoder chose from.

How edit distance is computed

The trick is a table. Cell (i, j) holds the edit distance between the first i items of the reference and the first j items of the decoded text. Each cell takes the cheapest of three moves:

pythondef edit_distance(ref, hyp):            # ref, hyp: lists of words or of characters
    D = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)]
    for i in range(len(ref) + 1): D[i][0] = i       # delete everything
    for j in range(len(hyp) + 1): D[0][j] = j       # insert everything
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            same = 0 if ref[i-1] == hyp[j-1] else 1
            D[i][j] = min(D[i-1][j] + 1,        # word the decoder missed
                          D[i][j-1] + 1,        # word the decoder made up
                          D[i-1][j-1] + same)  # match or swap
    return D[-1][-1]

wer = edit_distance(ref.split(), hyp.split()) / len(ref.split())
cer = edit_distance(list(ref), list(hyp)) / len(ref)

The data flow is two strings in and one fraction out. The only knobs are what counts as an item (word or character) and how the text is cleaned first: lower case, punctuation, spaces. Two labs that clean differently report different rates for the same output.

Here is the failure mode. Symptom: a decoder shows a 3% CER yet readers find it hard to use. Measurement: compute the WER on the same text. If one wrong letter in every few words spoils each word, a 3% CER can hide a 15% WER. Fix: report both, and say which cleaning you used.

The person meant a 10 word sentence. The decoder got 8 words right, swapped 1 word, and left out 1 word. What is the word error rate?

Chapter 6

The Published Records, Read With Care

Walk the typing and speech results from 2017 to 2024 and learn which unit, which setting and which vocabulary hides behind each famous number

Four papers set the pace for brain-to-text. Each reports a speed and an error figure. Read them side by side and the units change under your feet. This chapter lines them up honestly.

2017: point-and-click typing, in correct characters

Pandarinath et al. measured copy typing, where the person types a given sentence, in correct characters per minute (ccpm): only characters that end up right count. The best marks were 39.2 ccpm for T5 and 31.6 for T6, both on the OPTI-II keyboard layout. Free typing, where the person writes their own words, gave T6 24.4 ccpm. The best earlier result without word completion was 9.4 ccpm.

2021: imagined handwriting, in raw characters

Willett et al. asked T5, who has two 96 channel arrays, to imagine writing each letter by hand. A neural network turned the signals into letters. The headline is 90 characters per minute at 94.1% raw online accuracy. Raw means no cleanup after the fact. By the usual rule of five characters per word, 90 characters per minute is about 18 words per minute.

The same paper reports more than one error rate. The text gives a raw online CER of 5.4%, and the table lists 5.9%. With an offline language model the CER falls to 0.89% and the WER to 3.4%. A language model here is a program that knows which letter strings are likely English and fixes unlikely ones. Free typing ran at 73.8 characters per minute, with an 8.54% CER online and 2.25% with the language model.

2023 and 2024: speech

Willett et al. (2023) decoded attempted speech from T12, a participant with ALS who cannot speak clearly. The decoder reads sounds, called phonemes, then a language model picks words. It reached 62 words per minute. The word error rate was 9.1% with a 50 word vocabulary and 23.8% with 125,000 words. The paper says natural conversation runs near 160 words per minute, and that 62 is 3.4 times the previous record of 18.

Card et al. (2024, New England Journal of Medicine) report a man with ALS who reached 99.6% accuracy on a 50 word vocabulary on the first day, and 90.2% on a 125,000 word vocabulary on the second day. The paper adds 97.5% sustained accuracy (a 2.5% word error rate over the final five evaluation sessions) at about 32 words per minute in self-paced conversation, over 248 cumulative hours across 8.4 months.

Do not rank these two speech papers by one number. They use different people, different vocabularies and different metrics: Willett reports a word error rate, and Card reports accuracy. Chapter 8 gives a way to put them on one scale.

Read the records three ways

Blue bars are numbers printed in the papers. Switch the view. The first view puts speed on one ruler, the second shows the headline error, the third shows how one paper gives several errors.

All values are printed in the cited papers (Pandarinath 2017, Willett 2021, Willett 2023). Words per minute are drawn as characters per minute at the usual five characters per word, a convention, not a measurement. Card 2024 figures come from the paper’s abstract and results.

2024: a shared test

In 2024 Willett and colleagues ran the Brain-to-Text Benchmark ’24. Teams decoded the same neural recordings and were ranked by WER on a private set of 1,200 sentences. The baseline scored 9.7% and the top entry scored 5.8%. All three top teams merged several decoders and rescored them with a fine-tuned large language model. A shared test removes the argument about vocabulary and day, which makes comparison honest. Chapter 9 shows why a test set of this size still has noise.

Why the language papers stopped quoting bits

The 2017 paper reports both bits per second and correct characters per minute. The later handwriting and speech papers lead with characters or words per minute and an error rate. The reason is practical. Text has an obvious unit that readers understand. And language is not equal-chance: some letters and words are common, so the Wolpaw assumption of Chapter 2 breaks.

You lose something by dropping bits. Bits let you compare a grid task with a cursor task. Words per minute let you compare a speech BCI with a speaker. Neither lets you compare across both. Chapter 8 shows one attempt to build a bridge.

Failure mode: 90 against 39

Symptom: a reader says handwriting is more than twice as fast as point-and-click, 90 against 39. Measurement: the 39 counts only correct characters. The 90 counts all characters typed, with a 5.4% error rate. Remove the errors and 90 becomes about 85 correct characters per minute (90 × 0.946, our illustrative conversion). The gap is still large, near 2.2 times, but the units differ. Fix: convert both to correct characters per minute before you say “twice”.

Paper A reports 90 characters per minute with a 5.4% character error rate. Paper B reports 39 correct characters per minute. What is the fairest first step before comparing?

Chapter 7

Why Bitrate Claims Differ

Score one session nine ways, then learn the questions that turn a headline into a number you can trust, including for company claims

Picture a reporter who calls two labs. Both say their volunteer reached the best bitrate in the field. The numbers are 3.7 and 10. Neither lab is lying. Both ran a grid task. Yet they may have used different grids, different formulas, and different minutes. Here is how much room those choices give.

The choices hidden inside a bitrate

A bitrate comes from a chain of choices. Each one is legal. Each one moves the answer. Four matter most.

  1. Which formula. Gross counts every pick. Wolpaw discounts for errors. Net-correct subtracts errors. Chapters 2 and 3 showed the gaps.
  2. Which grid. A bigger N raises log2 N. Chapter 1 showed that doubling the grid adds one bit.
  3. Which minute. The best minute of a day is higher than the average minute. A record is a maximum, and a maximum of noisy numbers rises with the number of tries.
  4. Which person and day. A record from the best participant says little about the median participant, and the median participant says little about the first day after a new implant.

Fifth, there is the question of who checked. A peer reviewed paper has a methods section and referees. A blog post or a press release has neither. That does not make a company wrong. It means you hold its number as a claim until a method can be read.

Use the device. It scores one illustrative session of ten one-minute blocks on a 6 by 6 grid. Choose a formula and a window and watch the headline move.

Shop for a headline

The bars are the ten minutes of one session under the formula you pick. The teal line is the headline for the window you pick. The table shows all nine headlines at once.

Illustrative: ten minutes on a 6 by 6 grid, each with 38 to 46 picks and 84% to 97% accuracy drawn from a seeded random generator. This is not any person’s data. Real sessions vary in the same way, which is why the choice of window matters.

How big is the maximum effect?

Here is a quick estimate, with illustrative numbers. Say a person’s minute-by-minute rate wobbles around a mean of 2.5 bits per second with a spread (standard deviation) of 0.3. Take 30 minutes and keep only the best. For 30 draws from a bell shaped spread, the best is typically about two standard deviations above the mean.

So the best minute lands near 2.5 + 2 × 0.3 = 3.1 bits per second. That is 24% above the average, from nothing but selection. No change of decoder, no change of brain. A record from a long session, with many minutes to choose from, is nearly always a maximum like this one. The device above shows the same effect on its ten minutes.

Reading a company claim

Companies announce numbers for good reasons: they show progress. We hold each one as a company claim until we can read how it was made. Here is the table we can build today.

FigureWho says itWhat we could checkWhat is missing
11.32 bits per second, a “record” claimNeuralink, October 2026, as reported by a press siteThe press report repeats it. We could not read the company page itself with our tools.Task, grid, formula, session length, number of participants, independent replication
Median near 10 bits per second claimNeuralink, same reportSame as aboveWhich participants, which days, how many
Previous record 10.39 claimNeuralink, same reportSame as aboveSame task and formula as 11.32?
About 8 bits per second, 2024 claimPress coverage of the first participantNot checked against a primary pageFormula and task
4.16 bits per second, 9 by 9 gridPandarinath et al., eLife 2017, peer reviewedMethod read: log2(N−1) times net correct over time, 8 blocks over 2 days, participant T5Nothing the paper hides, but it is one person and one decade ago

Neuralink says its public Webgrid game scores in bits per second from net correct targets per minute, the grid size and the number of click types. That is the same family as Chapter 3. If the claims use that rule, a bigger grid is the easy way to a bigger number, and the grid size is the first thing to ask. We state it as the company’s description, not as something we verified.

Failure mode: the record that no one can repeat

Symptom: a record number, followed by silence. Measurement: ask six questions. What formula? What grid? What window? How many people? How many days? Who else checked? Fix: a good claim answers all six in one paragraph, or points to a methods section that does.

Lab A quotes the best minute of its best participant. Lab B quotes the whole-session average of its median participant. Why is a direct comparison unfair?

Chapter 8

Open Vocabulary: What Accuracy Hides

Weigh a decoder’s information by how much of what a person wants to say it can even represent, and see a 100% accurate decoder lose to a 50% accurate one

A speech BCI offers the person a menu of 50 words. It decodes them perfectly. Every word comes out right. Its accuracy is 100%. Now the person wants to say “I am hungry”, and “hungry” is not on the menu. The decoder is perfect and the person is stuck.

Accuracy and word error rate both have this blind spot. They are computed only over words the system supports. A short menu can score beautifully and still say very little of what a person means. A September 2026 preprint by Jayalath, Ballyk and Parker Jones at Oxford makes this point exact and offers a fix.

Two questions, not one

The preprint asks two questions. First: what words should the interface let the user say? Call the distribution of words a user may want to say the reference distribution. Second: how much information about those words gets through? The old scores answer only the second question, and only inside the menu.

The key quantity is lexical coverage, written C(S): the probability that a word the user wants is on the decoder’s menu S. The new score, open-vocabulary mutual information (OVMI), weighs the in-menu information by that coverage.

OVMI
Bits conveyed per word the user wants to say, counting the words the system cannot represent as zero.
C(S)
Lexical coverage: the share of the user’s intended words that sit on the menu.
[ … ]
The Chapter 2 expression for a menu of V words, with the decoder right with probability P and equal chance for every word on the menu. It is the form the preprint gives for uniform words and symmetric errors.

Look at the last bracket. It is the Wolpaw formula from Chapter 2, with V in place of N. OVMI is not a rival to the Wolpaw rate. It is the Wolpaw rate multiplied by a gate: the chance that the word was even on the menu. When coverage is 100%, the two agree.

Trade menu size for accuracy

The user may want any of 1,000 equally likely words. The teal marker is your decoder. The warm curve is its OVMI as the menu grows. The dashed line is a perfect decoder. Try the two presets.

50 words
100%

The toy follows the preprint’s own example: 1,000 equally likely words, so coverage is menu size over 1,000. Real language is Zipfian (a few words are very common), and the preprint uses real word frequencies. Its Figure 4 estimates, from its own reference distribution, put Willett’s 125,000 word system at 72.0% and Card’s at 93.7% of the lexical information, against 2.4% to 6.7% for the 50 word systems it lists from Moses and Willett.

Worked example, every step

System A: V = 50 words, P = 100%. System B: V = 1,000 words, P = 50%. Reference: 1,000 equally likely words.

  1. A, coverage. 50 / 1,000 = 5%.
  2. A, bracket. P = 1, so the bracket is log2 50 = 5.64 bits.
  3. A, OVMI. 0.05 × 5.64 = 0.28 bits per intended word.
  4. B, coverage. 100%.
  5. B, bracket. 9.966 + 0.5 × log2 0.5 + 0.5 × log2(0.5/999) = 9.966 − 0.5 − 5.48 = 3.98 bits.
  6. B, OVMI. 1 × 3.98 = 3.98 bits.

Accuracy ranks A over B, 100% against 50%. OVMI ranks B over A by a factor of 14. The preprint uses this same pair as its toy case.

Real words are not equally likely

The toy gives every word the same chance. Real speech does not. A few words, like “the” and “I”, make up a large share of everything we say, and a long tail of rare words makes up the rest. Language with this shape is called Zipfian.

Two effects follow, and the preprint tracks both. A menu of the commonest words covers more speech than its size suggests, so coverage grows quickly at first and slowly later. Yet the Wolpaw term log2 V overstates the information of a menu where some words are common, because a common word is less news. The preprint replaces log2 V with the actual entropy of the words on the menu.

The paper is a preprint. Its simple form assumes errors spread evenly over the menu, and that a word outside the menu produces no output. Treat its numbers as a careful first proposal for a common ruler, not as settled fact.

Why this matters for the numbers you meet

Recall Chapter 6. Willett reported a 9.1% WER on 50 words and 23.8% on 125,000. The smaller error belongs to the system that could say far less. OVMI puts both on one scale. The preprint reports that the big historical jump among invasive systems came from larger vocabularies, after decoding quality inside small menus had already saturated.

The preprint also shows that selecting the vocabulary to maximise OVMI improved accuracy by up to 16.3% relative, in three speech domains. We take that as the authors’ result. It is a preprint, not yet peer reviewed.

Here is the failure mode. Symptom: a decoder with a tiny vocabulary reports a very low WER. Measurement: ask what share of everyday speech uses only those words. If it is 5%, the WER describes 5% of what the person wants to say. Fix: report vocabulary size with every error rate, and add coverage against a stated reference text.

A decoder gets 100% of its 200 supported words right. Those 200 words make up 20% of what the user wants to say. What does the open-vocabulary view add?

Chapter 9

A Fair Test: How Many Sentences Make a Number

Repeat one evaluation many times to see the noise in a word error rate, and learn the rules that keep an evaluation honest

Two teams test the same kind of decoder. Team A reports a 10% word error rate. Team B reports 9%. Is B better? That depends on a question the papers rarely answer in the headline: how many sentences did each team test?

An error rate from a finite test is an estimate. Run the same decoder on another set of sentences and you get a slightly different number. The fewer the sentences, the more the number jumps. A fair test says how big the jump is.

The size of the noise

Suppose each word is wrong with probability p, independently. Test W words. The measured error rate has a spread, the standard error, of about the square root of p(1 − p)/W.

Worked example, with illustrative numbers: true p = 10%, 10 words per sentence.

  1. 50 sentences. W = 500 words. Standard error = √(0.1 × 0.9 / 500) = 1.34 points.
  2. 95% range. About 2 standard errors either side: 10% ± 2.6 points, from 7.4% to 12.6%.
  3. 1,200 sentences (the size of the 2024 benchmark’s private test). W = 12,000. Standard error = 0.27 points. Range: 9.5% to 10.5%.

So at 50 sentences, 9% against 10% is noise. At 1,200 sentences, a gap of 3.9 points, like the benchmark’s 5.8% against 9.7%, is far outside the noise. A gap of 0.3 points is not.

Press the button below to see this for yourself. The device repeats the whole evaluation 200 times and draws where the measured rate lands each time.

Repeat the evaluation 200 times

Each bar counts how many of 200 repeated tests measured that error rate. The vertical line is the true rate. The shaded band holds 95% of the repeats. Drag the test size.

50
10%

Illustrative simulation: 10 words per sentence, a seeded random generator, 200 repeats. With the toggle on, some sentences are harder than others, which makes real test sets noisier than the plain binomial formula says. The slider is logarithmic: 10 to 1,000 sentences.

Online and offline are different tests

In an online test the person watches the output and the decoder runs live. The person can adapt, and the clock runs. In an offline test the lab replays recorded brain signals through a model later. Offline models may look at future signal, run slowly, and use a large language model. Offline numbers are valuable for research, but they are not what the person experienced. Willett 2021 shows both: 5.4% CER online and 0.89% offline with a language model.

The 2021 paper also tells us how its test was built: five days of evaluation, with four blocks of 7 to 10 sentences each day. The 2023 speech paper pooled 80 trials a day for the large vocabulary. Small, careful sets, and the numbers carry that noise.

Compare two decoders on the same sentences

When you compare two decoders, test both on the very same sentences. Some sentences are simply harder. If the same hard sentences hit both decoders, that difficulty cancels in the difference between them.

Compare two separate error rates from two different test sets, and the sentence difficulty adds noise to each number. A paired test, which looks at the sentences where the two decoders disagree, gives a tighter answer from the same amount of data. This is one reason a shared benchmark with one private test set is so useful: every team faces the same sentences.

Days are part of the test

Brain signals drift. The same electrode may hear a slightly different set of neurons next week. A decoder trained on Monday may do worse on Friday. So a test on sentences from the same hour as the training data is a kind. A test on a later day is a truer one.

The Card paper makes this a feature of the score: it reports day one and day two separately, with the vocabulary growing from 50 to 125,000 words. Calibration time belongs in the report, next to the error rate.

Failure mode and checklist

Symptom: a new decoder beats the old one by half a point on a test of 100 sentences. Measurement: the repeat device shows a range several points wide at that size, and the gap sits inside it. Fix: report the number of sentences and an interval, and use a shared test set when you can, as the 2024 benchmark did.

  1. Freeze the decoder before the test sentences are chosen.
  2. Keep test sentences out of training, ideally from later days than the training days.
  3. Say online or offline, and say whether a language model ran.
  4. Report the count of sentences and words, with an interval.
  5. Report calibration time and the vocabulary size.
  6. Report every participant, not only the best.
You multiply the number of test sentences by 4. By roughly what factor does the standard error of the measured error rate shrink?

Chapter 10

Field Guide: Measuring a BCI

Keep the terms, the numbers, the checklist and the code in one place, then read the next BCI headline with new eyes

Every BCI score answers one question about one task. Bits per selection ask how much a choice carries. The Wolpaw rate discounts for errors. Achieved bitrate counts net-correct picks against a clock. Fitts throughput prices a steered move in effective bits. Error rates count the edits a text needs. Open-vocabulary information asks how much of the person’s language the system can even say. A fair claim names the task, the formula, the window, the people and the test size.

The terms

TermFormulaUse it when
Bits per picklog2 NN equally likely targets, no errors
Wolpaw bitslog2 N + P log2 P + (1−P) log2((1−P)/(N−1))Fixed trials, equal chance, errors spread evenly
Achieved bitrate(C − W) log2(N−1) / TSelf-paced grid task (Pandarinath 2017 rule)
Fitts throughputlog2(Ae/We + 1) / MT, with We = 4.133 SDCursor pointing, over many moves
WER and CER(S + D + I) / N over words or charactersText out; report both
OVMIC(S) × in-menu informationComparing decoders with different vocabularies

The numbers that matter

WhatNumberSource and status
Grid bitrate, 6 by 6T6 2.2, T5 3.7, T7 1.4 bits per secondPandarinath et al., eLife 2017, peer reviewed
Grid bitrate, 9 by 9T5 4.16 bits per secondSame paper
Copy typingT5 39.2, T6 31.6 correct characters per minuteSame paper
Handwriting90 characters per minute, CER 5.4% raw, 0.89% with offline language modelWillett et al., Nature 2021, peer reviewed
Speech, large vocabulary62 words per minute, WER 23.8% (125,000 words), 9.1% (50 words)Willett et al., Nature 2023, peer reviewed
Speech, calibration99.6% day one (50 words), 90.2% day two (125,000 words)Card et al., NEJM 2024, peer reviewed
BenchmarkWER 9.7% baseline, 5.8% best, 1,200 private sentencesWillett et al., arXiv 2412.17227
Open vocabulary toy0.28 bits (50 words, 100%) against 3.98 bits (1,000 words, 50%)Jayalath et al., arXiv 2609.02887, preprint
Cursor record11.32 bits per second, median near 10 claimNeuralink, October 2026, read through a press report only

The checklist for any headline

  1. Which family? Selection, pointing or language.
  2. Which formula? Gross, Wolpaw, net-correct, throughput, WER, CER.
  3. Which task size? The grid N, the vocabulary V, the target width.
  4. Which window? Best minute, median minute, whole session.
  5. Which people? One participant or all, which days.
  6. Which test? Online or offline, how many sentences, with an interval.
  7. Who checked? A methods section, a referee, a second lab, or only the author.

The core as code

pythonimport math, statistics as st

def wolpaw_bits(n, p):                         # bits per pick, equal chance, errors spread evenly
    if p >= 1: return math.log2(n)
    if p <= 1 / n: return 0.0
    return math.log2(n) + p * math.log2(p) + (1 - p) * math.log2((1 - p) / (n - 1))

def achieved_bitrate(n, correct, wrong, seconds):  # Pandarinath 2017 rule
    return max(0, correct - wrong) * math.log2(n - 1) / seconds

def fitts_throughput(moves):                    # moves: (distance_moved, landing_error, seconds)
    a_e = st.mean(m[0] for m in moves)
    w_e = 4.133 * st.pstdev(m[1] for m in moves)
    return math.log2(a_e / w_e + 1) / st.mean(m[2] for m in moves)

def edit_distance(ref, hyp):                    # lists of words or characters
    prev = list(range(len(hyp) + 1))
    for i, r in enumerate(ref, 1):
        cur = [i]
        for j, h in enumerate(hyp, 1):
            cur.append(min(prev[j] + 1, cur[j-1] + 1, prev[j-1] + (r != h)))
        prev = cur
    return prev[-1]

def wer(ref, hyp): return edit_distance(ref.split(), hyp.split()) / len(ref.split())
def cer(ref, hyp): return edit_distance(list(ref), list(hyp)) / len(ref)

def ovmi_uniform(v, p, lexicon):                # menu of v words out of `lexicon` equally likely words
    return (v / lexicon) * wolpaw_bits(v, p)

print(round(wolpaw_bits(4, 0.9), 3), round(ovmi_uniform(50, 1, 1000), 2), round(ovmi_uniform(1000, 0.5, 1000), 2))  # 1.373 0.28 3.98

Where this sits

Measuring comes after decoding. The decoders you scored here come from earlier lessons: linear decoders, the Kalman decoder (see also the Kalman filter itself), and closed-loop refitting, which is the ReFIT step behind the 2017 numbers. The P300 speller lesson is the classic place the Wolpaw rate is used. Drift and recalibration explains why a score from Monday may not hold on Friday. For the information theory underneath, try the information theory workbook.

Sources we opened

The takeaway. A bitrate is not a property of a brain. It is a property of a brain, a decoder, a task, a formula and a test. Change any of the five and the number moves. When you read a headline, ask which five it used.

Now press Present or Teach and explain, out loud, how one session becomes three different bitrates and why the error rate in a paper depends on the setting. Then go back to the instrument at the top, drag the accuracy down to 70%, and say in one sentence which bar falls fastest and why.

Measuring a BCI, lesson 22 of the Brain-Computer Interfaces series
Pandarinath 2017 · Willett 2021 · Jayalath 2026 · Back to the series start