A man who cannot move his hand imagines writing letters, and a recurrent network turns his motor cortex activity into text at 90 characters per minute, more than twice the cursor record.
Learn how a brain-computer interface (a device that reads brain activity and turns it into commands) decodes attempted handwriting, why it beats cursor typing, and how it was trained on a few hours of data.
Drag the clock, then switch between the four ways of writing. Watch how many characters of the same sentence each one has typed. Then we build, piece by piece, everything underneath: the signal, the letter patterns, the network, the labels, the training tricks and the reason handwriting is easy to decode.
You need a rough idea of a neuron (a brain cell that sends brief electrical pulses) and a little probability. We build the rest from zero.
Based on Willett, Avansino, Hochberg, Henderson and Shenoy, “High-performance brain-to-text communication via handwriting”, Nature 593, 249–254 (2021), doi:10.1038/s41586-021-03506-2. Every idea is re-taught here in our own words and devices; it is a reading of the lecture Linear Decoders.
Speeds are the paper’s: EEG spellers 1 to 5 characters per minute (the lane uses 5), intracortical point-and-click at most 40, the handwriting BCI 90, and able-bodied smartphone typing 115 for the participant’s age group. Each lane types at a steady rate; real typing is bursty. The three red letters on the handwriting lane are illustrative: 3 of 57 characters, 5.3%, where the paper reports a raw error rate near 5.4% to 5.9%. The sentence is one of the paper’s own.
Chapter 0
See why typing speed limits communication BCIs, and follow one letter from brain to screen
Picture a man who cannot move from the neck down. His name in the study is T5. A spinal cord injury cut the wires between his brain and his body years ago. His hands twitch, and that is all. He is clear-headed and has plenty to say.
How does he say it? Today there are three main answers. Each one is slow, and slowness is the real wall. A conversation that runs at a few letters per minute is not a conversation.
The first way reads EEG (electroencephalography: electrodes on the scalp that pick up faint brain waves). An EEG speller flashes letters on a screen and watches for the brain’s reaction to the letter you want. The paper reports 1 to 5 characters per minute for the common kinds. A faster kind, based on visually evoked potentials, reached 60 characters per minute, but it ties up your eyes and needs flashing panels.
The second way is an intracortical BCI. A brain-computer interface (BCI) is a device that reads brain activity and turns it into commands. “Intracortical” means the electrodes sit inside the cortex, the brain’s wrinkled outer layer, so they hear single cells instead of a blurred crowd. These BCIs let the user steer a cursor on a screen and click letters on a keyboard. The best result before this paper peaked at 40 characters per minute.
The third way is to decode speech. The paper calls this exciting, but says accuracy and vocabulary size still needed work for general use.
Here is the idea of this paper, in plain words. Cursor typing asks the brain for a simple move: slide left, slide right. But a person who can write already owns a fast, rich skill. Handwriting packs a whole letter into one flowing stroke.
After the injury, the hand cannot write. But the brain may still send the “write an a” command. Place sensors on the part of the brain that sends it, record, and a computer can guess the letter. That guess, repeated, is typing.
The open question was whether such a fast, delicate skill survives paralysis. Earlier BCIs had shown that gross movements such as reaching, grasping and moving a cursor stay encoded in motor cortex. Nobody knew about handwriting. T5 answered it by trying to write.
Follow one letter through the system. The stepper below shows five stages. Each stage hands the next one a thing of a known shape. We name those shapes now, because every later chapter zooms into one stage.
T5 attempts to write while imagining a pen on ruled paper. Two electrode arrays in the hand “knob” area of the precentral gyrus (a premotor region of the brain’s surface) record 192 channels. The recorder counts threshold crossings: moments when a channel’s voltage dips below a set line, which signals a nearby neuron firing. Counts are binned every 20 ms and smoothed.
So the network sees a table with one row per 20 ms and 192 columns. A recurrent neural network (RNN: a network that reads a sequence one step at a time and keeps a memory of what it saw) turns that table into two things: a probability for each of 31 characters, and a single number that rises when a new character begins.
Each row is one stage, with the shape of what it hands on. Press Next stage to light the path from the brain to the screen.
The stage names and numbers are the paper’s (Fig. 2a, Extended Data Fig. 1): 192 electrodes, 20 ms bins, a two-layer GRU, 31 characters, a 0.3 s emission delay and a 50,000-word bigram language model that the authors applied offline. The one-line summaries are ours.
T5 reached 90 characters per minute at 94.1% raw accuracy while typing sentences he had never trained on. With a language model that fixes errors afterwards, offline, accuracy rose above 99%. That speed is more than double the 40 characters per minute of the best cursor BCI, and it approaches the 115 characters per minute that able-bodied people of his age type on a smartphone.
The paper does more than set a record. It explains why handwriting is easier to decode than straight-line movements (Chapter 7). It shows how to train an RNN when you do not know when each letter was written (Chapter 4). And it shows how little daily calibration the decoder needs (Chapter 8).
Keep one limit in mind from the start. This is one participant. The authors call the system a proof of concept, not a clinical product.
This paper reads the lecture Linear Decoders. There you meet the simple recipe: fit a matrix from firing rates to movement. This paper keeps that recipe for one analysis (Chapter 1) and then replaces it with a recurrent network. For how spikes become rates, see Spikes to Rates.
Chapter 1
Read a pen’s velocity out of 192 channels of firing rates with one matrix
Imagine a hall with 192 listeners, each facing a stage. Every listener cheers when the pen on stage moves in their own favourite direction. One cheers for up, another for down-left, another for right. You cannot see the stage. You can only hear the cheering. Could you redraw the pen’s path from the noise?
That hall is the recording in this paper, and the question is the first thing T5’s team asked. Before building any network, they checked whether the pen’s movement is written in the neural activity at all.
Each electrode sits next to a few neurons. A neuron fires brief pulses called spikes. The authors did not sort spikes into single neurons. They counted threshold crossings: every time the voltage on a channel dropped below a line set at 4.5 times the channel’s background noise (RMS: the typical size of its noise). The count in a short window is a firing rate, so the recording becomes 192 numbers per window. The lesson Spikes to Rates builds this step in detail.
The windows are 20 ms long, and the rates are smoothed. After one second you hold a table with 50 rows and 192 columns. Call the row at time t the vector r: one rate per channel.
Motor cortex neurons are tuned. A neuron’s rate rises when the hand moves in its preferred direction and falls when the hand moves the opposite way. In between, the rate follows a cosine. In vector form, neuron i adds up a weighted sum of the two velocity components. Stack all neurons and you get one matrix equation.
Why this D? Because we want the v that makes B v as close to the observed r as possible. Setting the slope of the squared error to zero gives exactly (BTB)−1BTr. This is the linear decoder from the lecture Linear Decoders. In practice, the matrix is fitted from data instead of known.
Take three neurons with preferred directions 0°, 120° and 240°. Their rows of B are (1, 0), (−0.5, 0.866) and (−0.5, −0.866). Let the pen move with v = (0.6, 0.8).
We get v back exactly, because there was no noise. Real counts are noisy. That is the failure to watch for.
Grey: the true pen path. Blue: the path decoded from the noisy rates of the neurons shown as a heat map. Raise the noise until the letter breaks, then add neurons to rescue it.
A toy, not T5’s data: neurons have evenly spaced preferred directions and equal gain, the noise is independent Gaussian, and the noise unit is the pen’s typical speed. The decoder is the least-squares D from the equation above. The paper decoded real velocity from trial-averaged activity (Fig. 1d).
Add noise and the blue path wobbles. Add neurons and it settles again. Each neuron is wrong in its own way, and D averages many of them. With N independent neurons the noise in the estimate shrinks roughly as 1 divided by the square root of N. That is why a 192-channel array is useful even though each channel alone is poor.
Real recordings are messier than the toy. The authors ran principal components analysis (PCA: a way to find the few directions along which a cloud of data varies most; see PCA) on the 192 rates. The top three dimensions already showed strong, repeatable patterns for each letter (Fig. 1b).
Then they decoded pen tip velocity linearly from the trial-averaged activity and integrated it to get a path. Recognizable letter shapes came out for all 31 characters tested (Fig. 1d). The neural dimensions that carried velocity accounted for 30% of the total neural variance. So pen velocity is a large, readable part of the signal, but far from all of it.
The rest of the variance holds other things, such as the identity of the letter and the timing. That is why the final decoder does not stop at a linear map. It uses a network that learns whatever is useful.
Chapter 2
Align trials in time, and watch 31 characters fall into separate clusters
Ask a friend to say “hello” five times. Each take has the same sounds in the same order. But one take is quick, one is slow, and one lingers on the “o”. Now lay the five recordings on top of each other and average them. You get mush. The peaks no longer line up.
Handwriting on a paralysed hand has the same problem. T5 never writes the same letter at exactly the same speed. If every letter has a neural signature, the speed wobble hides it.
Before decoding whole sentences, the authors ran a simple test. T5 attempted to write one character at a time, following a prompt on a screen: 31 characters, 27 repeats each. The characters were the 26 lowercase letters, commas, apostrophes, question marks, periods (written as ‘~’) and spaces (written as ‘>’). The odd symbols make periods and spaces easier to detect.
Plot the activity in the top 3 principal components and each repeat looks like a strong, repeatable wave. But the peaks and valleys land at different times from trial to trial. The authors say this is probably because of changes in writing speed.
Time warping asks, for each trial, which stretching and squeezing of its clock makes it best match the others. A warp is a monotone function: it can speed up or slow down parts of the trial, but it cannot reorder events. Apply the inverse warp to every trial, and the peaks line up. The paper used a published time-alignment technique for this (its reference 11). Then it plotted the aligned trials (Fig. 1c).
After alignment, the authors saw “remarkably consistent underlying patterns of neural activity that are unique to each character.” The inset of Fig. 1c shows the warps for the letter ‘m’. They lie close to the identity line, so T5’s speed varies only modestly.
The device below uses a simpler cousin of that technique, dynamic time warping (DTW): a dynamic-programming search for the best monotone match between two signals. See Dynamic Programming for the idea behind it. It is our teaching tool, not the paper’s exact method.
Thin lines are six repeats of the same neural pattern, each written at a slightly different speed. The thick line is their average. Raise the speed variability, then switch between the two ways of averaging.
A toy: one made-up channel, six repeats of a three-bump pattern, each repeat warped by a random smooth speed change plus a little noise. The DTW here has a band limit of 20 of 100 samples. The paper’s own alignment method and its warping functions are in Fig. 1c.
With no variability, both averages agree. Raise it, and the as-recorded average flattens: its peaks shrink and its valleys fill in. The warped average keeps the true height. That height is exactly the information a classifier needs.
Next the authors looked at every single trial, without averaging. They used t-SNE, a method that squashes high-dimensional points into a 2D picture so that points that were close stay close. Each dot is one trial, 27 per character. The dots fell into tight clusters, one per character (Fig. 1e).
The layout was motoric: characters that are written in a similar way sit near each other. This matches the idea that the activity tracks the movement.
Then came a harder test with a k-nearest-neighbour classifier (k-NN). To label a new trial, find the k labelled trials that are closest to it, and take a vote. Closeness is the ordinary distance between the two activity patterns: flatten each pattern into one long list of numbers and compute the straight-line distance.
A tiny worked case with k = 3. A new trial sits at distances 2.1 and 2.4 from two trials of ‘a’, 2.2 from one trial of ‘b’, and 5.0 from a second ‘b’. Its three nearest are a (2.1), b (2.2) and a (2.4). The vote is two a against one b, so the label is ‘a’. (These distances are illustrative.)
Offline, this classifier labelled the characters with 94.1% accuracy (95% confidence interval 92.6 to 95.8). So even years after paralysis, the neural representation of handwriting is, in the authors’ words, “likely strong enough to be useful for a BCI.”
Look at what this test assumed. It cut out one trial per letter, starting at a known go cue. A real sentence has no cues. Letters follow each other without gaps, and nobody knows where one ends. A neighbour vote cannot find the boundaries.
The next chapters solve this. First comes the network that reads a continuous stream (Chapter 3). Then comes the trick of finding the letter times in training sentences when the hand does not move (Chapter 4).
Chapter 3
Turn a recurrent network’s two outputs into typed characters in real time
A court stenographer listens to a witness. The witness never pauses between words, and the stenographer must still type each word as it ends. She cannot wait for the whole speech. She must also not type a word twice because the witness coughed.
The handwriting decoder has the same job. T5 writes letter after letter. The decoder hears 192 channels of firing rates and must print each letter once, soon after he finishes it.
The decoder works on the rate table from Chapter 1. Every 20 ms a new row arrives: 192 smoothed threshold-crossing counts. The authors z-scored each channel, which means they subtracted its mean and divided by its spread, so every channel lives on the same scale. Call the stream x, with shape [T, 192] where T is the number of 20 ms bins.
One letter lasts about a second. That is roughly 50 rows. The right answer at any row depends on rows that came earlier. A model for this needs memory.
A recurrent neural network reads one row at a time and carries a hidden state, a list of numbers that summarises everything it has seen. At each step it mixes the new row into the state. A GRU (gated recurrent unit) adds two gates. A gate is a number between 0 and 1 that decides how much to keep or to overwrite. This lets the state hold a fact for a long time without being washed out. The lesson RNN and GRU derives the gates.
The paper uses a two-layer GRU. The bottom layer reads every 20 ms row. The top layer runs more slowly and updates only once every 100 ms, so it sees five rows at a time. The authors found that this makes it easier to hold information for long periods and speeds up training. The network’s outputs therefore arrive every 100 ms, ten times per second.
At each output step the network produces two things (Extended Data Fig. 1).
Why two? Because y alone cannot tell “still writing the same ‘l’” from “the second ‘l’ in hello”. A double letter has the same y for two seconds. The pulse in z marks each start, so a double letter makes two pulses.
A letter looks like another letter until its last stroke. An ‘n’ and an ‘m’ share a start. So the authors gave the network a one-second output delay. At time t, the network’s outputs describe the moment one second earlier (the paper calls them p at t minus d). It has seen the whole letter before it commits.
The cost is time. Characters showed up on screen after T5 finished them, with a delay the authors estimate at 0.4 to 0.7 s.
For live typing, the rule is simple. Watch z. When z crosses a threshold at time t, take the most likely character in y at time t + 0.3 s, and print it. That is all (Fig. 2a, “Raw Online Output”). The 0.3 s shift gives y a moment to settle after the pulse.
Here is one pulse with illustrative numbers. The threshold is 0.5. At t = 2.30 s, z moves from 0.41 to 0.57, so it has crossed. At 2.60 s, y gives ‘h’ 0.81, ‘n’ 0.11 and the rest 0.08 in all. The decoder prints ‘h’.
The threshold is a dial between two failures. Set it too low, and a twitch in z that is not a letter prints an extra character. Set it too high, and a weakly written letter never prints, so it is missed. The authors report no backspace in this system. T5 was told to keep writing when an error occurred.
The blue line is a toy new-character signal z while T5 writes “hello”. Green ticks mark where each true letter starts. Drag the threshold and watch which letters print.
A toy: the pulses, the twitch and the noise are made up (one weak letter at height 0.58 and one twitch near 0.4), and a false trigger prints a random letter. The emission rule is the paper’s: threshold crossing in z, then the most likely character in y 0.3 s later. A new crossing within 0.6 s of the last one is ignored here.
A sweet spot exists in the middle. Below it, a twitch prints a stray letter. Above it, the weak letter vanishes. In the real system the network learns to make z clean, so the threshold is easy to set.
To train y and z the network needs to know, for every moment, which letter was written and when it began. That is a hard demand. T5’s hand does not move, so nobody can watch the pen and note the start times. The next chapter shows how the authors recovered them.
Chapter 4
Align known text to unlabelled brain activity with a hidden Markov model
You own a film with no subtitle timing. You also own a script that lists every line in order. For each line, when does the actor start speaking? You know what is said and in what order. You do not know when.
The handwriting team faced exactly this puzzle, in a harder form. Their film was the neural recording, and their script was the sentence on the screen.
To train the network of Chapter 3 you need, for every 100 ms, the letter being written and the start of each letter. A person who can move gives you this for free: a pen records every stroke. T5’s hand is paralysed. The authors name this as the first of two challenges: the time that each letter was written is unknown, which makes standard supervised learning hard to apply. The second is that the data set is small, and Chapter 5 deals with it.
The fix borrows from speech recognition, where the same problem appears: you have a recording and its transcript, but no word boundaries. The tool is a hidden Markov model (HMM). Our lesson Hidden Markov Models builds one from zero.
First, the single-character data from Chapter 2 is used. T5 wrote each character many times. The authors time-warped and averaged the repeats to get one template per character: a pattern of neural activity over time that stands for that character (Extended Data Fig. 2b). Each day’s sessions start with this letter block.
For a training sentence such as “few black taxis drive up major roads on quiet hazy nights” the HMM has one state for each character, in the order they appear: f, e, w, space, b, l, and so on. The model starts in the first state. At each time bin it can do exactly two things: stay in the current state or move on to the next one. It can never go back or skip ahead. This is a forced alignment, because the text forces the order.
What decides the path? Each state scores each time bin by how well the data in that bin matches that character’s template. Call the score of state k at time t by sk(t). The paper checks the match with cross-correlation heat maps (Extended Data Fig. 2d).
There are astronomically many ways to cut T bins into K stretches. The Viterbi algorithm (a standard dynamic program for this kind of alignment; the part of the paper we read names the forced-alignment HMM but not its recurrence, so this recurrence is our teaching version) finds the best cut. Let Vk(t) be the best total score of any path that is in state k at time t.
Fill the table left to right. Then walk back from the last state at the last bin, always following whichever choice won the max. The bins where the walk steps from letter k−1 to letter k are the letter start times.
Three letters a, b, c and six bins. The match scores are illustrative.
| Letter | t = 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| s for a | 5 | 4 | 1 | 0 | 0 | 0 |
| s for b | 0 | 1 | 5 | 4 | 1 | 0 |
| s for c | 0 | 0 | 0 | 1 | 5 | 4 |
The start times are a at bin 1, b at bin 3 and c at bin 5. Exactly where the scores peak.
Each row is a letter of the word, each column a 0.1 s bin; bright cells match a template. Green ticks are the true starts. Teal lines are the starts the Viterbi path finds. Raise the noise until they part.
A toy: scores are smooth bumps in each letter’s true interval plus Gaussian noise, and the true letter lengths are made up. The recurrence is the one above, run in your browser. The paper’s HMM uses neural templates and its real data (Extended Data Fig. 2c, d).
With little noise, the found starts sit on the true ones. As noise grows, a start can slip by a few bins, and at the highest settings whole letters go wrong. Note that a slip of a bin or two hardly matters for training. The network only needs approximately correct pulses.
Only the start time of each character is needed to build the targets (Extended Data Fig. 2d caption). The pulse target for z sits at each start. The target for y is the labelled character. The authors checked quality in one more way: they plotted cross-correlation heat maps between the character templates and the real data. The HMM start times land on clear hot spots.
Chapter 5
Build synthetic sentences, add noise, and teach a network that generalizes
Imagine learning a language from one short book. You read it fifty times. Soon you can recite it perfectly. Then someone hands you a new page, and you are lost. You did not learn the language. You memorised the book.
Neural networks do the same. The word for it is overfitting: the network fits the training data so closely that it fails on anything new. Big networks on small data overfit fast. This is the second challenge the authors name: the data set was small compared with typical RNN data sets.
A person who writes with a paralysed hand gives you a few hours a day. Before the first test day, the team had 242 sentences from three pilot days. On each later day they added more data to recalibrate. By the last day the network trained on 572 sentences, which add up to 7.3 hours and 30.4 thousand characters.
The team evaluated on sentences the network never trained on, so the numbers in the paper measure real generalisation. To get there, they used three families of tricks, and each one has a clear reason.
Chapter 4 gave every training sentence a set of character start times. That means you can cut the data into snippets: one piece of neural activity per written character. Put the pieces in a library. Then build a new sentence by pulling snippets from the library at random and gluing them in a row (Extended Data Fig. 2e).
The new sentence is not real handwriting, but it is a fair picture of how the brain writes each letter. And it comes with perfect labels, because you know which snippet you used and where it starts. You can make as many as you like.
One more step makes the sentence harder to memorise. Each snippet is stretched or squeezed in time by up to 30%, which mimics T5’s own speed changes from Chapter 2.
Real recordings carry noise. A network that has only ever seen clean data may freak out on a noisy day. So the authors added white noise to the inputs during training. Because the inputs were z-scored, the noise level is in units of standard deviations of the input.
They also added noise to the feature means: random offsets and slow changes in the baseline firing rate of each channel. This matters for a particular reason. Baselines drift slowly over a session, so a block recorded later has a different baseline from a block recorded earlier. Training with drifted baselines taught the network to ignore them. The caption of Extended Data Fig. 3b says this helps with later blocks in the session, but not with new trials in blocks the network already trained on.
The trace is one channel of a made-up sentence glued from character snippets. Letters mark the snippet starts. Change the three augmentations, then press New sentence to build another.
A toy: each character’s snippet is a made-up smooth curve of about one second, standing in for real neural snippets. The 30% stretch limit is the paper’s (Extended Data Fig. 2e); the noise and drift units here are in standard deviations of the toy signal, and the paper tuned its own values by grid search (Extended Data Fig. 3a).
Electrode signals change from day to day. A neuron that fired on channel 17 on Monday may be quiet on Friday. Two choices follow from that.
Build the three-character sentence “cab” from the library. Suppose the snippets last 0.9 s, 1.1 s and 1.0 s. Draw three random stretch factors, each between 0.7 and 1.3: say 1.20, 0.80 and 1.00.
The start times are the labels. The network trains on 2.96 s of activity it has never seen in exactly this form, with exact targets. Add input noise and baseline offsets on top, and each pass over the data looks a little different. Memorising becomes impossible.
The paper tests each trick on its own (Extended Data Fig. 3). Both synthetic sentences and input white noise were essential: leaving out either hurt performance, even when the other was set to its best value. Synthetic data mattered most when training on one day only, where data is scarcest. Each parameter gave a statistically significant improvement for at least one condition, judged with bootstrap confidence intervals. The text of the paper does not list the numbers, so we do not quote any. Look at the figure for them.
The pattern is the point. Small data does not need a smaller idea. It needs a way to invent honest variety.
Chapter 6
Read the online results and see how a language model polishes the raw letters
You type a text message with a few typos. Your phone underlines one word and quietly swaps it for the word you meant. It knows that “black taxis” is a likelier phrase than “blach tazis”. A smart reader fixes your slips without ever seeing your fingers.
The handwriting decoder gets the same help. This chapter reads the paper’s results, then opens the language model that lifts them.
The evaluation took five days. Each day had four blocks of 7 to 10 sentences, and the network had never trained on any of them. T5 copied each sentence from a prompt on the screen, attempting to write it letter by letter. The decoded letters appeared on screen as the network emitted them. No backspace existed, so he kept writing after an error. Letters showed up 0.4 to 0.7 s after he finished them (the authors’ estimate).
The paper scores text with edit distance: the fewest single-letter changes (a swap, an insertion or a deletion) that turn the output into the target. The character error rate (CER) divides that by the length of the target. The word error rate (WER) does the same over words.
A small example. The target is “black taxis”, 11 characters including the space. The decoder outputs “blach tazis”. Two letters differ: the h for a k, and the z for an x. The edit distance is 2, so the CER is 2 / 11 = 18%. Both words are wrong, so the WER is 2 / 2 = 100%. One typo kills a word, which is why WER runs higher than CER.
| Setting | Speed | Error |
|---|---|---|
| Copy typing, raw online output | 90 characters per minute | 94.1% accuracy (abstract); 5.4% CER averaged over blocks (Fig. 2c) |
| Copy typing + language model (offline) | same | 0.89% CER, 3.4% WER (all days) |
| Non-causal offline RNN (whole sentence at once) | not real time | 0.17% CER |
| Free typing, real time | 73.8 characters per minute | 8.54% CER raw; 2.25% with a language model |
| Previous best intracortical free typing | 24.4 correct characters per minute | not compared here |
Two details deserve care. The paper gives the raw error both as 94.1% accuracy (a 5.9% error, in the abstract and in the calibration text) and as a 5.4% CER averaged across the blocks in Fig. 2c. We use the 94.1% as the headline and report both. Second, the free-typing run used open questions: T5 wrote his own answers instead of copying. It was a bit slower and less accurate, but it still shows the system works for self-generated text.
The 0.89% CER and 3.4% WER are in the range of speech recognition, where the paper cites word error rates of 4 to 5%. The non-causal result, 0.17% CER, is the ceiling: a network that may look at the whole sentence at once. It cannot give letter-by-letter feedback, but it shows the signal carries enough information.
A language model assigns a probability to text. The paper used a custom 50,000-word bigram model: it knows, for each word, how likely every other word is to follow it. It never looks further back than one word. The rest of the context is handled by the RNN’s character probabilities.
The combined decoder picks the text that is likely under both: it matches what the RNN saw in the brain, and it reads like real language.
This is the common recipe in speech recognition. The paper does not publish its weight in the text we read, so the weight λ in our device is illustrative. The device scores three word slots with made-up numbers. Move the weight and see who wins.
For each word slot the decoder compares three candidates. Each bar is a cost (lower is better): blue is the cost from the network, pink is the weighted language cost. Drag the language weight.
A toy with invented numbers: the network and language costs are negative log probabilities that we made up, and each slot is scored on its own. The target is “few black taxis”. The paper used a 50,000-word bigram model with a real search; it does not state its weight in the main text.
With the weight at zero, the brain signal alone decides, and the output has three typos. A moderate weight lets the language model rescue each word. At a very high weight, the model starts to overrule the brain with common words. That is the failure to watch for: a language model that is too strong turns a rare but real word into a likelier wrong one.
The live system printed raw letters with no language correction. The language model was applied afterwards in a retrospective analysis (Fig. 2a), because it needs to see a few words around each choice. In a future live system it could correct a word a moment after it appears. The authors present the offline results as the performance the full system would reach, not as something T5 saw on screen.
Chapter 7
Measure how temporal variety pushes neural patterns apart
A locksmith has a tray of keys. In the first tray, every key has one cut at a different depth. In the second, every key has five cuts of varied depth. Which tray is easier to sort by feel in the dark? The second. The keys in the first tray differ in one thing only, and a small slip confuses neighbours.
This chapter makes that picture exact. It answers a puzzle from the introduction: how can handwriting be decoded more than twice as fast as point-and-click, with similar accuracy?
A point-and-click BCI moves a cursor to a target and selects it. Its speed is limited mainly by accuracy. The designers raise the cursor gain (how far the cursor moves for a given neural signal) until it runs so fast that decoding errors make it uncontrollable. The best intracortical result peaked at 40 characters per minute. T5 wrote at 90.
The authors propose an explanation: handwritten letters differ from each other in many more ways than straight-line movements do. Letters have more variety in their patterns across time. This makes them easier to tell apart.
They took two sets of 16 movements, both written by T5 as attempted handwriting. One set held 16 letters that need no pen lift, each lasting about 1 s. The other held 16 straight-line movements in different directions, each lasting 0.6 s. For each movement they averaged the time-warped trials and resampled to a fixed length, giving a matrix of shape 192 × 100: 192 electrodes, 100 time steps (Fig. 4b).
Now measure how far apart the movements are. Take the distance between two matrices as the straight-line distance, treating each as one long list of numbers. The key quantity is the nearest-neighbour distance of a movement: its distance to the closest other movement.
Why the nearest neighbour and not the average distance? Because a decoder fails at the closest pair. A set can have a large mean distance and one confusable pair. The paper found that the mean distance did not grow for characters, but the nearest-neighbour distance was 72% larger for characters than for lines (95% CI 60% to 86%, Fig. 4c). A simulation of classification against neural noise confirmed it: characters were easier to classify (Fig. 4d).
Where does the extra distance come from? The authors split the answer into two kinds of dimension. Spatial dimensionality counts how many independent patterns across electrodes the movement set uses. Temporal dimensionality counts how many independent patterns across time it uses. Both are measured with the participation ratio, a number that is about the count of dimensions needed to explain the bulk of the variance (the paper says about 80%).
Three quick cases with five directions. If all the variance sits in one direction (1, 0, 0, 0, 0): PR = 12 / 1 = 1. If it is spread evenly (1, 1, 1, 1, 1): PR = 52 / 5 = 5. If one direction dominates (4, 1, 1, 1, 1): PR = 82 / 20 = 3.2.
The result: spatial dimensionality was only modestly larger for characters (1.24 times, 95% CI 1.19 to 1.30). Temporal dimensionality was far larger (2.65 times, CI 2.58 to 2.72). Letters differ mostly in how their activity evolves over time.
The authors built a toy model: four movements and two neurons whose activity is held along one line. If each movement is a flat level, the four sit at four points on a line. Add one bend in time, and the four trajectories can fan out into a bigger space of shapes (Fig. 4f to h). Our device uses the same idea with numbers of our own.
Each line is the activity pattern of one movement over time. At variety 0 all four are flat levels. At variety 1 each line has one bend. Add noise and compare the two bars.
A toy, in the spirit of the paper’s Fig. 4f to h but with our own numbers: 4 patterns of 20 time steps, activity limited to the range −1 to 1, independent Gaussian noise on every step, and a nearest-pattern classifier run on 800 simulated trials. The 72% and 2.65 times figures come from T5’s data, not from this toy.
The patterns share the same limit on activity, from −1 to 1. Flat levels must pack four values into that range, so neighbours are close. Bent trajectories spend the same range twice, once per half, and so can be much farther apart. Noise then fails the flat set first.
They hold that time-varying movements, such as letters, are fundamentally easier to decode than point-to-point movements. They add that this is one, not necessarily the only, factor behind the speed. The result also held for noise that was correlated across time and neurons (Extended Data Fig. 5).
A bonus: using the same principle of maximising nearest-neighbour distance, they designed an alphabet that is theoretically easier to classify than the Latin one, since most Latin letters start with a downstroke or a counter-clockwise curl (Extended Data Fig. 6). Nobody has tested it with a person yet in this paper.
Chapter 8
See how recordings change across days, and what keeping up costs the user
A piano is tuned in the morning. By evening it has drifted a little, and by next month a lot. Nothing is broken. Strings stretch, wood breathes. A pianist who tunes before every concert plays beautifully. One who never tunes slowly plays out of key.
A brain-computer interface is the same. The array sits in living tissue. The decoder you trained on Monday is slightly out of tune on Friday.
The paper names two likely causes: neural plasticity (the brain slowly changes how it codes a movement) and electrode array micromotion (the array shifts a tiny distance, so each channel hears different cells). The effect is that the same letter looks a little different on a new day.
The standard fix is to recalibrate: at the start of each day, the user writes a block of known sentences, and the decoder retrains on them. This works, but it uses the user’s time and energy before he can say anything. T5’s team used 50 calibration sentences per day. Ideally, little or none would be needed.
The authors asked this offline, by replaying the recorded copy-typing data with less calibration. This is a simulation: they cannot test it live without risking T5’s time, and the paper says plainly that offline simulations are not always predictive of online performance.
Choose a retraining option. The bars show its error and the calibration time it costs. The grey tick on each bar is the daily 50-sentence baseline.
Numbers are the paper’s offline simulations (Fig. 3a, b), except the baseline row: 5.9% raw is the raw online error rate and 0.89% is the character error rate with a language model averaged over days. The paper does not give the language-model error for 30 sentences, so that bar is left empty. Minutes are our arithmetic, scaled from 8.7 minutes for 10 sentences. “None” applies to sessions 2 to 7 days apart; “Unsupervised” to sessions up to 7 days apart.
Read the trade. Dropping from 50 sentences to 10 saves about 35 minutes of writing and raises the raw error from 5.9% to 8.5%. Skipping calibration entirely costs more raw error (11.1%), but the language model brings it back to 1.5%. And a decoder that teaches itself from corrected text lands close to the fully calibrated one.
The authors measured how much each character’s neural pattern changed between days. When sessions were 7 days apart or less, the patterns stayed highly similar: the mean correlation was 0.85. The changes seemed to build up at a steady, predictable rate (Extended Data Fig. 4).
A closer look shows the shape of the drift. Think of each day’s patterns as points in a space. Seen in the first day’s top 10 principal components, the layout of the characters stays almost the same on later days, but the points shrink toward the centre. The authors read this as the code slowly rotating into a new space, so that less of it is visible in the old one. A practical hack follows: scale the decoder inputs up. A scale of 1.5 helped when long gaps passed between sessions, and the paper used 1.5 in the no-retraining conditions of Fig. 3.
The symptom: on a morning after a long gap, the decoder prints more wrong letters than it did last week, and T5 notices. The measurement: the error rate climbs, and the correlation between today’s character patterns and the old ones falls as the gap grows (the paper’s mean was 0.85 within a week). The fixes sit on a ladder. Scale the inputs up a little. Retrain on a short calibration block. Or let the decoder retrain on its own corrected text. Each rung costs the user less than the one before, and each needs more trust in the decoder.
The authors are direct that this is a proof of concept in one participant, and not yet a clinical system. The limits they list:
The paper also lists competing interests: some authors consult for BCI companies and are inventors on a patent application that covers the decoding approach. We note it so you can weigh the claims knowing it.
Chapter 9
Keep the numbers, the terms, the core code and the way onward
A participant with paralysis attempts to handwrite letters, and two microelectrode arrays in the hand area of motor cortex record 192 channels of threshold-crossing rates. Pen movement is readable from them, and each character has a consistent neural signature once trials are time-warped. A two-layer GRU turns the stream, binned every 20 ms, into 31 character probabilities and a new-character signal. A threshold on that signal prints letters. Because the hand does not move, training labels come from a forced-alignment HMM, and the small data set is stretched with synthetic sentences and noise. The result is 90 characters per minute online at 94.1% raw accuracy, and above 99% offline with a bigram language model. Letters are easy to decode because their activity varies over time.
| What | Number | Where in the paper |
|---|---|---|
| Typing speed, copy typing | 90 characters per minute | Abstract, Fig. 2c |
| Raw online accuracy | 94.1% (5.4% CER averaged over blocks) | Abstract, Fig. 2c |
| With a language model, offline | 0.89% CER, 3.4% WER; above 99% accuracy | Table 1, text |
| Non-causal offline RNN | 0.17% CER | Text |
| Free typing | 73.8 characters per minute; 8.54% raw, 2.25% with LM | Extended Data Table 2 |
| Best earlier intracortical typing | 40 characters per minute (24.4 correct, free typing) | Text |
| Letter-by-letter k-NN | 94.1% (CI 92.6 to 95.8) on 31 characters | Fig. 1e |
| Training set at the end | 572 sentences, 7.3 hours, 30.4k characters | Text |
| Nearest-neighbour distance, letters vs lines | 72% larger (CI 60 to 86) | Fig. 4c |
| Dimensionality, letters vs lines | spatial 1.24 times, temporal 2.65 times | Fig. 4e |
| Short calibration | 10 sentences (8.7 min): 8.5% raw, 1.7% with LM | Fig. 3a |
| Unsupervised retraining | 7.3% raw, 0.84% with LM (sessions up to 7 days apart) | Fig. 3b |
| Term | Meaning |
|---|---|
| Threshold crossing | A moment when a channel’s voltage dips below a set line, counted as a nearby spike. |
| Time warping | Stretching each trial’s clock so repeats line up. |
| y and z | The network’s 31 letter probabilities and its new-character signal. |
| Forced alignment | Finding letter start times for a known text with an HMM and Viterbi. |
| Synthetic sentence | A new training sentence glued from labelled snippets, stretched up to 30%. |
| CER and WER | Edit distance divided by the number of target characters or words. |
| Nearest-neighbour distance | Distance from a movement to its closest rival; it sets how easily noise confuses them. |
| Participation ratio | About how many dimensions the variance spreads over. |
pythonimport numpy as np, random, torch, torch.nn as nn class HandwritingRNN(nn.Module): # Sketch. Hidden size is illustrative; the paper's sizes are in its Supplement. def __init__(self, n_days, n_ch=192, n_char=31, h=256): super().__init__() self.day_in = nn.ModuleList([nn.Linear(n_ch, n_ch) for _ in range(n_days)]) # one input layer per day self.low = nn.GRU(n_ch, h, batch_first=True) # reads every 20 ms bin self.high = nn.GRU(h, h, batch_first=True) # runs every 100 ms (every 5th bin) self.head = nn.Linear(h, n_char + 1) # 31 letters + 1 new-character logit def forward(self, x, day): # x: [B, T, 192] z-scored rates lo, _ = self.low(self.day_in[day](x)) # [B, T, h] hi, _ = self.high(lo[:, 4::5]) # [B, T/5, h] (stride is a simplification) out = self.head(hi) return out[..., :31].softmax(-1), out[..., 31].sigmoid() # y: [B, T/5, 31], z: [B, T/5] def emit(y, z, chars, thr=0.5, delay=3): # 3 steps of 100 ms = 0.3 s text, armed = "", True for t in range(len(z)): if z[t] > thr and armed: text += chars[int(y[min(t + delay, len(y) - 1)].argmax())] armed = False if z[t] < thr: armed = True return text def forced_align(S): # S[k, t]: template match of sentence char k at bin t K, T = S.shape V = np.full((K, T), -np.inf); back = np.zeros((K, T), bool); V[0, 0] = S[0, 0] for t in range(1, T): for k in range(K): stay, adv = V[k, t - 1], (V[k - 1, t - 1] if k else -np.inf) back[k, t] = adv > stay V[k, t] = max(stay, adv) + S[k, t] starts, k = [0] * K, K - 1 for t in range(T - 1, 0, -1): if back[k, t] and k: starts[k] = t; k -= 1 return starts def synth_sentence(library, n=8, stretch=0.3, noise=0.3): parts = [] for _ in range(n): ch, snip = random.choice(library) # snip: [t, 192] idx = np.linspace(0, len(snip) - 1, int(len(snip) * (1 + random.uniform(-stretch, stretch)))) parts.append((ch, np.stack([np.interp(idx, range(len(snip)), snip[:, c]) for c in range(192)], 1))) x = np.concatenate([p[1] for p in parts]) x = x + np.random.randn(*x.shape) * noise + np.random.randn(192) * 0.3 # white noise + baseline offset return x, [p[0] for p in parts]
The authors’ own offline reproduction, with their data, is at github.com/fwillett/handwritingBCI. The recordings (1,000 sentences, 43.5 thousand characters, 10.7 hours) are on Dryad. Our sketch is simpler than theirs.
This paper turned a motor skill into text. Its method (an RNN, a delay, a language model and a way to label unlabelled sequences) carried over to speech. The authors say it could apply to any sequential behaviour that cannot be observed directly. A later paper by an overlapping group decoded attempted speech: Willett et al., Nature 2023, doi:10.1038/s41586-023-06377-x. A benchmark and competition report followed: arXiv 2412.17227. We have not re-taught those here.
Now press Present or Teach and explain the decoder out loud, from memory: what the arrays record, why y and z both exist, how the HMM finds letter times, why synthetic sentences help, and why a letter beats a straight line. Then go back to the race at the top, move the clock to 38 s, and explain why the handwriting BCI finishes the sentence about when the cursor BCI is at 25 characters.
Credit. This lesson re-teaches Francis R. Willett, Donald T. Avansino, Leigh R. Hochberg, Jaimie M. Henderson and Krishna V. Shenoy, “High-performance brain-to-text communication via handwriting”, Nature 593, 249–254 (2021). Read the original: doi:10.1038/s41586-021-03506-2 or the author manuscript. All figures and numbers are theirs; the toy devices and the wording are ours.