NDT2 and NDT3

A decoder pretrained on 2,000 hours of other people’s brain spikes learns your brain from far less of your data, until you hand it 1.5 hours of your own. Then pretraining stops helping.

Learn how two papers pretrain a Transformer on spikes from motor cortex, why that shortens the calibration a brain-computer interface needs, and the exact point where the gain ends.

Pick a decoder, then drag how much of your own data it gets. Watch the gap between the pretrained curve and the from-scratch curve close. Then we build, piece by piece, how the papers made that gap, and what they found when they looked for its limits.

You need a rough idea of what a Transformer does (it reads a list of tokens and mixes them with attention). We build the spikes, the decoder and the pretraining from zero.

Papers: Ye, Collinger, Wehbe and Gaunt, Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity (NeurIPS 2023), and Ye and 23 co-authors, A Generalist Intracortical Motor Decoder (NeurIPS 2025). Code: NDT2, NDT3. Every idea is re-taught here in our own words and devices. This is the paper reading of the lecture LFADS and latent dynamics.

Spend your calibration time

9 min

The paper’s numbers are the two crossover marks: NDT2’s gains last up to 3,000 one-second trials, about 50 minutes (Fig. 4C), and NDT3’s pretrained models match from-scratch ones at 1.5 hours of downstream data (Fig. 1B). The curve heights are illustrative shapes, not measured scores. The two papers use different tasks, so compare the marks, not the heights.

Chapter 0

Why a Decoder Starts Over Every Day

See why a brain-computer interface needs fresh calibration data each session, and what pretraining on other sessions, people and tasks tries to change

A woman with a spinal cord injury sits down at a computer. Two small grids of electrodes sit in the motor cortex of her brain, the strip of cortex that plans movement. She imagines moving her hand, and a cursor on the screen should follow.

Before it can, the system needs to learn what her brain activity means today. So she spends the first part of the session on calibration: she watches a cursor move to targets and tries to move along with it, while the computer records. Tomorrow she repeats the whole routine. This lesson is about two papers that try to shorten that routine.

What a decoder is

An intracortical brain-computer interface (iBCI) is a device that reads electrical activity from inside the cortex. A small array of wires, a microelectrode array, records from dozens to hundreds of channels. Each channel is one wire. On each channel the device counts spikes, the brief electrical pulses that neurons use to signal. Chapter 1 builds this from zero.

A decoder is the program that turns those spike counts into an intended action, such as the velocity of a cursor. Think of a translator who has just met a new speaker. The translator needs a few sentences from that speaker before the translation works. Calibration data are those sentences.

Why the translator must start again

The papers name three kinds of change, and they use one word for all of them: a context is one setting of the recording. A new session is a new sitting, perhaps the next day. A new subject is a different person or monkey. A new task is a different behaviour, such as reaching to random targets instead of a centre-out pattern.

Spiking activity shifts across all three. Electrodes move by a fraction of a millimetre. A neuron that fired on Monday goes quiet on Tuesday, and a new one appears. The NDT2 paper says that iBCIs typically need recalibration within hours, while the larger ECoG signal (recorded from the brain surface) may hold for days. Fine detail brings high accuracy and high fragility together.

Feel it yourself. Below, 24 model neurons each prefer one direction of motion. The decoder is fitted at hour 0 and then left alone. Each half hour, every neuron's preference drifts a little, and a few neurons are replaced by new ones. Drag the clock.

Let the neurons drift

Top: each dot is a neuron’s preferred direction now, joined by a line to where it pointed when the decoder was fitted. Bottom: how well the decoder predicts velocity. Drag the hours, then press Recalibrate here.

4.0 h

A toy model, not data from the paper. Each neuron fires with a Poisson count of mean 5 + 4 (velocity dotted with its preferred direction). Every half hour a preferred direction drifts by a random 6.4 degrees and a neuron is replaced with chance 3%. The decoder is a population vector with one fitted gain. Score is R2 on 200 fresh trials, shown from 0.

The score falls for two reasons that you can see in the ring. Lines stretch where preferences drifted. Dots jump where a neuron was replaced. A decoder that remembers the old picture reads the new activity wrongly. Recalibrating redraws the picture, and the score comes back.

The idea of the two papers

Calibration is costly because each session starts from nothing. Yet the introduction of NDT2 points out that something stays put. The dominant principal components of neural activity, the few directions in which the whole population varies most, can stay stable across sessions, subjects and tasks. If a lot is shared, a model could learn the shared part once, from many contexts, and then adapt to a new context from a small sample.

That is the recipe that worked for language and images: train one generic model on broad data with no labels (pretraining), then adjust it to the target with a little data (fine-tuning). The two papers test whether it works for spikes.

NDT2 (Neural Data Transformer 2, NeurIPS 2023) pretrains a Transformer on about 20,000 trials from other sessions, subjects or tasks, and at most about 100,000 trials. NDT3 (NeurIPS 2025) pretrains on 2,000 hours from over 30 monkeys and humans, with models of 45 million and 350 million parameters. The second paper also asks the harder question: where does pretraining stop helping, and why?

The headline, in one line. Pretraining on other contexts helps most when you have little data from the new one, and its benefit vanishes at about 1.5 hours of downstream data in NDT3’s tests. The rest of the lesson shows how the papers found that line and what lies beyond it.

The road

  1. Spikes and decoders: what the data look like, how a decoder is scored.
  2. Hide and guess: masked autoencoding, the pretraining task, and its Poisson loss.
  3. NDT2: neural patches, the asymmetric encoder and decoder, context tokens.
  4. What NDT2 showed: transfer across sessions, subjects and tasks, and a closed-loop pilot.
  5. NDT3: one flat stream: spikes and behaviour as tokens in one sequence.
  6. Training NDT3: the losses, covariate dropout, phase and return tokens.
  7. Scale and the 1.5 hour wall: 2,000 hours, two model sizes, 31 settings.
  8. Where it breaks: shuffled inputs, held-out angles, and the benchmark it loses.
  9. Field guide: numbers, a checklist, core code, links onward.
Why does an intracortical decoder usually need new data each session?

Chapter 1

Spikes, Bins and a Score for a Decoder

Turn a stream of electrical pulses into a table of counts, and learn the one number both papers use to grade a decoder

Stand outside a stadium and listen. You cannot hear one person, but you can tell when a goal is scored, because the roar changes. A microelectrode in the cortex does something similar for neurons. It listens at one spot, and it hears the nearby cells.

A neuron is a brain cell that signals with short electrical pulses. Each pulse is a spike, and it lasts about a millisecond. A cell fires slowly when it is quiet and fast when it is busy. The number of spikes per second, its firing rate, carries the message.

From wire to counts

Here is the path the NDT3 paper describes. The electrode records a voltage 30,000 times per second. A band-pass filter keeps the part of the signal where spikes live. Then a threshold, which the researcher sets, marks each moment the voltage crosses it. Each crossing counts as one spike on that channel, which is one electrode wire.

One channel may hear several neurons at once. Spike sorting separates them by the shape of their pulses. The papers call the result sorted data. Skipping the step leaves unsorted (multi-unit) data, where a channel is simply a place. Unsorted data are more practical for devices, and NDT2 found they decode slightly better on its test set.

Next comes binning. Choose a bin width, say 20 milliseconds, and count the spikes on each channel inside each bin. Both papers use 20 ms bins, which means 50 time steps per second. NDT2 notes that control applications benefit from rates of 50 to 100 Hz. Drag the bin width below and watch the picture change.

Bin the spikes

Top: the velocity the person is trying to produce. Middle: spike times on 16 channels for one second. Bottom: the same second as counts, one cell per channel and bin, brighter for more spikes. Drag the bin width.

20 ms

Spikes are simulated, not recorded: each channel is a Poisson process whose rate rises and falls with the velocity trace. The shape arithmetic is exact for any recording: bins per second = 1000 divided by the bin width, and counts per second = bins times channels (96 here, one common array size).

What you see in the bottom panel is the whole input of a decoder. It is a table with one row per time step and one column per channel. Each entry is a small whole number: 0, 1, 2 and rarely more. Computer scientists call this table a tensor with shape [T, C], where T is the number of bins and C the number of channels.

What the decoder predicts

The person tries to move a cursor. At each time step the program knows the cursor's intended velocity, a pair of numbers: how fast to the right and how fast upward. The papers call such signals covariates (behaviour recorded alongside the neural data). A monkey's covariates come from real limb tracking. For a person with paralysis, the computer moves the cursor and assumes the person intends to follow.

The decoder is a function from the [T, C] table to a [T, 2] table of velocities. The classic choice is the Wiener filter. In the NDT3 paper it is ridge regression with several past time steps as input. Ridge regression fits a straight-line rule between inputs and outputs, and it adds a small penalty on large weights so the fit does not chase noise. The papers use it as the baseline to beat.

Grading with R-squared

How do we say that one decoder is better? Both papers report R2, the coefficient of determination. It compares your errors to the errors of the laziest possible guess, which is to predict the average velocity every time.

R2
The score. It is 1 for perfect prediction and 0 for a decoder no better than guessing the average. It can go negative for a decoder that is worse than the average.
∑ (v − v̂)2
The decoder's total squared error. Here vt is the true velocity at step t and v̂t is the decoder's prediction.
∑ (v − v̄)2
The total squared error of the lazy guess, where v̄ is the mean velocity. This sum is the spread of the true velocity.

Work one example with five time steps. True velocities: 2, −1, 0, 3, −2. The decoder says 1.5, −0.5, 0.5, 2, −1. The mean of the true values is 0.4, so the lazy errors are 1.6, −1.4, −0.4, 2.6, −2.4. Their squares add to 2.56 + 1.96 + 0.16 + 6.76 + 5.76 = 17.2. The decoder's errors are 0.5, −0.5, −0.5, 1, −1. Their squares add to 0.25 + 0.25 + 0.25 + 1 + 1 = 2.75.

R2 = 1 − 2.75 / 17.2 = 1 − 0.160 = 0.84

The decoder explains 84% of the variation in velocity. NDT2 reports R2 for monkey reaching, with one number per test session. NDT3 computes R2 for each behaviour dimension, then averages them weighted by how much each one varies.

The second score: how well the counts are predicted

Chapter 2 introduces a second grade that looks only at the neural data. NDT2 calls it the unsupervised score: how well the model can predict spike counts that it did not see. Lower is better. The model never needs behaviour labels for this, and that is the key to pretraining on data from anywhere.

A decoder always predicts the average velocity of the test data. What R2 does it get?

Chapter 2

Hide and Guess: Pretraining Without Labels

Cover half of the spike counts, ask the model to fill them in, and see why that forces it to learn how a neural population behaves

A friend covers half the words of a sentence with her thumb and asks you to read it anyway. “The cat sat on the ___.” You guess “mat”. You can only do this because you know how sentences work. The game teaches the structure without anyone writing down a single answer.

Researchers use the same game to pretrain models. In language it gave us BERT. In pictures it gave the masked autoencoder (MAE) of He and colleagues: hide most of an image's patches and train a network to paint them back. NDT2 and NDT3 apply it to the table of spike counts from Chapter 1.

Why the game works on neurons

The neurons in a motor cortex do not act alone. During a reach, large groups rise and fall together, and the whole population follows a few smooth patterns. Researchers call the space of these patterns a neural manifold (see the manifolds lesson and latent variable models). The NDT2 introduction notes that the dominant principal components can stay stable across sessions, subjects and tasks.

Because of that shared pattern, one channel's count helps to predict another's. If the visible channels all surge, the hidden one probably surges too. A model that fills the gaps well has learned the pattern. NDT2 says high mask ratios work well here, in line with the low dimension and high redundancy of the code, compared with language. NDT2 pretrains with 50% masking.

The best part: the game needs no labels. Any recording of spikes is training data, with no velocity, no task and no cursor. That is what lets NDT2 pool sessions, subjects and tasks that share nothing else.

A loss that suits counts

To train, the model must say how wrong a guess is. The model does not guess a count. It guesses a rate λ, the expected number of spikes in that bin, and counts scatter around it. The standard model for this scatter is the Poisson distribution: if the rate is λ, the chance of seeing exactly k spikes is

P(k | λ)
The probability of counting k spikes in one bin when the true rate is λ.
λk
Grows when the rate is high and the count is high. A count of 5 is likelier under a rate of 5 than under a rate of 1.
e−λ
Shrinks as the rate grows. It makes the probabilities add to 1.
k!
k factorial, 1 × 2 × … × k. It does not depend on λ.

Training should make the true counts likely. The negative log-likelihood (NLL) turns that into a cost: take the log of the probability and flip the sign, so a likelier count costs less. Taking the log of the formula gives

NLL(k, λ) = λ − k ln λ + ln k!

Try k = 3 spikes. A guess of λ = 3 costs 3 − 3 ln 3 + ln 6 = 3 − 3.296 + 1.792 = 1.50. A guess of λ = 1 costs 1 − 0 + 1.792 = 2.79. A guess of λ = 6 costs 6 − 5.375 + 1.792 = 2.42. The cost is lowest when the rate equals the count, and it grows when the guess is too low or too high. The last term, ln k!, does not depend on the model and is often dropped, but it stays in the numbers below so that they match the toy device.

The network outputs the logarithm of the rate and applies an exponential, so the rate can never be negative. The NDT2 appendix calls this a linear-exponential readout. NDT2 evaluates the model by the NLL on bins it never saw, which it calls the unsupervised score. Lower is better.

Try it

The device below plays the game on a toy population of 10 channels. Two guessers fill the hidden cells. The first only knows each channel's average count. The second has learned the shared pattern, and for each time bin it uses the visible channels to infer how strongly the population is surging. Raise the mask ratio and watch what happens to each.

Hide and guess the counts

Top grid: counts the model can see; dark cells are hidden. Middle grid: the shared-pattern guess for the hidden cells. Bars: the loss on the hidden cells, lower is better. Drag the mask ratio, or draw a new pattern.

50%

A toy population, not paper data: 10 channels, 36 bins, rates exp(a + b z) with one shared smooth signal z. The pattern guesser knows a and b, as a pretrained model would have learned them, and infers z in each bin from the visible cells. The paper pretrains with a 50% mask ratio.

With the default seed, the channel-average guess stays well above the other two bars at every ratio. It uses only each channel’s own average. The shared-pattern guess stays close to the true-rate floor until most of the table is hidden, and then it drifts upward. The gap between those two bars is what the game teaches. (The toy is small and random, so the bars wobble a little when you draw a new pattern.)

The three stages of use

NDT2 runs the game in stages, shown in its Figure 2. Pretraining plays it on the pooled data from other contexts. Unsupervised tuning plays it again on the neural data of the target session. Last, a supervised probe learns to read velocity from the tuned model, using whatever labelled trials exist. Every labelled score in the paper has had the benefit of the two unsupervised stages.

Why can NDT2 pretrain on sessions that have no behaviour recorded?

Chapter 3

NDT2: Patches, Two Halves and Context Tokens

Cut the spike table into tokens a Transformer can afford, and see how three small design choices let one model learn from many sessions

Imagine a library clerk with a rule: she can compare any two index cards, but only if both lie on her desk at once. A big desk makes her fast. A small desk makes her slow. Every extra card also adds a new comparison with every card already there.

That clerk is a Transformer (see the Transformer lesson). It reads a list of tokens, small vectors that each stand for one piece of the input. Attention lets every token look at every other token. With N tokens there are about N × N pairs, so the cost grows with the square of the list length. The first design question for neural data is: what is a token?

Tokens: from one big token to patches

The earlier model, NDT1, used one token per time step. Each token held the whole population at that moment. That keeps the list short, but it gives the model no way to treat channels as separate things. Across sessions the meaning of a channel changes, so the authors wanted spatial attention too.

The other extreme is one token per channel per time step. For 288 channels over 2.5 seconds at 20 ms bins, that is 125 × 288 = 36,000 tokens. The pairs number over a billion per layer. The NDT2 paper calls full neuron-wise attention too costly.

Their middle path borrows from the Vision Transformer. A ViT cuts an image into square patches. NDT2 groups K channels into a patch, pads with zeros if the last patch is short, and gives each patch at each time step one token. Each channel's count in the patch passes through its own linear projection, and the projections are joined into one vector.

The authors fixed the other numbers first. Bins are 20 ms, so 50 Hz. The window is 2.5 s, which is 125 time steps. The context budget is about 2,000 tokens, which is what their GPUs hold. Dividing, 2,000 / 125 = 16 tokens per time step at most. With common arrays of 100 to 200 channels, that forces patches of 16 to 32 channels. Try the numbers.

Patch the array

Each cell is one token: a column is one 20 ms time step, a row is one patch of channels. The bar shows tokens against the 2,000 token budget. Change the array size, the patch size and the window.

96
2.5 s

The paper’s numbers: 20 ms bins, a 2.5 s window (125 steps), a 2,000 token budget, patches of 16 to 32 channels, at most 288 channels in its sorted data. The token count is time steps times ceil(channels / patch); the pair count is N times (N + 1) / 2, because pretraining attention is causal.

Patch size is a trade. The authors tested 8 channels against 32. Smaller patches sometimes gave a slightly better count loss, but they decoded velocity reliably worse. They cost more to train, and they cost more data to fit the decoder. NDT2 therefore kept the larger patches. The paper adds that adjacent channels in open datasets often sit at unrelated places on the array, so a patch is not a neighbourhood the way an image patch is.

Two halves: an encoder and a decoder

NDT2 follows the masked autoencoder layout. The encoder reads only the tokens that are not hidden. The decoder is a second, shallow Transformer. It receives the encoder's output plus a placeholder mask token at every hidden position, and it predicts the rates there. Most of the work happens in the deep encoder, which sees about half the tokens.

That saves memory because attention cost depends on the square of the length. The encoder in most NDT2 runs has 6 layers and about 3 million parameters. The decoder has 2 layers and 0.7 million. Pretraining attention is causal: a token cannot look at future time steps, as in real-time use. Dropout is 0.1, the mask ratio is 50%, and the hidden size is 256 in the base setting.

Who sees which tokens

Top strip: 375 tokens (96 channels, patches of 32, 125 steps). Dark ones are hidden. Bars: attention pairs per forward pass, counting layers. Drag the mask ratio.

50%

Our arithmetic, not a paper table: pairs are counted as N squared per layer (full attention), with a 6 layer encoder over N tokens as the uniform baseline, and a 6 layer encoder over the visible tokens plus a 2 layer decoder over all N for the asymmetric design.

At 50% masking, the asymmetric design uses 6 × (187.5)2 + 2 × 3752 pairs against 6 × 3752, which is about 58% of the cost. With no masking it would cost more, not less, because the decoder is extra. The saving comes from the mask.

Context tokens: telling the model where the data came from

Sessions differ. The model needs a way to know which session, which subject and which task a sequence came from. NDT2 adds context tokens: learned vectors, one per unique session, subject or task, put in front of the data tokens. It is like a name tag on each card. The idea mirrors prompt tuning in language models.

The ablation is modest and honest. Context tokens mainly speed up learning. The final loss improves by about 0.001, small but not nothing. Giving more tokens per context changed little. The authors kept one token each for session, subject and task because it costs almost nothing. They also tried cross-attention instead of adding tokens, and saw no difference.

The rival: stitching

Before NDT2, a common way to pool sessions was stitching: give every session its own linear read-in layer, which maps its channels into a shared space, and its own read-out layer. A single 96 by 256 read-in already holds 96 × 256 = 24,576 weights, per session. The paper warns that clinical sessions with only dozens of trials can overfit such layers. In its experiments NDT-Stitch did much worse than NDT2.

Why does the NDT2 encoder skip the hidden tokens?

Chapter 4

What NDT2 Showed: Transfer, Its Price and a Pilot

Read the evidence for pretraining across sessions, subjects and tasks, including the cases where it did not help

A fair test of pretraining asks one plain question. Give two decoders the same small amount of data from a new day. One has seen other data first. Does it win, and by how much? NDT2 asks this on three kinds of other data and reports the answer in each case.

The test bench

The main dataset is a public one with monkeys reaching to random targets that appear one after another, called RTT. It has two monkeys and 47 sessions, about 20,000 seconds per monkey. The authors cut each session into trials of one second. They pick five spaced-out sessions of one monkey, Indy, as the test sessions, and give the model 5 minutes of data per session: 300 trials. That amount is enough to fit a reasonable single-session model while staying a practical calibration time.

Pretraining uses about 20,000 trials of one of three kinds. Multi-session: other sessions of the same monkey. Multi-subject: sessions from the other monkey. Multi-task: reaching data that is not RTT. The kinds are matched in size, so a difference in score is a difference in relevance, not in amount.

Each run reports two scores from Chapters 1 and 2: the Poisson loss on hidden counts (lower is better) and R2 for velocity (higher is better). Baselines include NDT1 with stitching and a published nonlinear decoder from the dataset authors, rEFH.

Result 1: every kind of data helped, but not equally

On sorted data, all pretrained NDT2 models beat both rEFH and the single-session models, on both scores. Surprisingly, multi-subject data worked about as well as multi-session data. Multi-task data helped by a smaller but real amount. NDT-Stitch did much worse in every case. After cross-task pretraining it even fell below the single-session baseline.

The best result used unsorted data, pretrained on other sessions: R2 above 0.7. Unsorted channels keep a physical meaning across sessions, because they are tied to electrode places. So the authors used unsorted data for the rest of the RTT analysis. The single-session NDT2 and NDT1 scored below rEFH. The authors say plainly that this is not unexpected, since the models vary widely with tuning, and that part of the value of pretraining is that it simplifies the tuning.

Result 2: the three kinds scale at different slopes

The second experiment grows the pretraining set and gives each model only 100 trials from the new session. It compares against single-session models given more and more of their own trials. Here are the paper's own comparisons:

The authors found no negative transfer here, meaning no case where pretraining made the tuned model worse than training from scratch. They note that their pretraining is modest: the largest set has two orders of magnitude more data than the largest single-context model, and no more.

The same section carries a warning. If your goal is the best possible explanation of neural variance, a moderately large amount of in-day data still beats pretraining alone. Count loss and decoding R2 told different stories for cross-task and cross-subject data. The paper observes that language models also show such mismatches.

Result 3: people

Human data are harder. Sessions are short, and the intended velocity is a guess: the cursor moves by itself and the program assumes the person follows. The authors tune one model per block and test on 50% of the trials. Table 2 of the paper reports R2 for two participants, P2 and P3. Explore it.

Read Table 2

Each bar pair is one row of the paper’s Table 2: the score for participant P2 and for P3. Tap a row to compare it with row 1, the broad human mix.

Values are copied from Table 2 of the NDT2 paper (velocity R2, SEM over 3 fine-tuning seeds, up to 0.020). Every row also uses each person’s own task and subject data. Human data total 100K trials for P2 and 30K for P3. Row 8 is smoothed spike ridge regression, called OLE in the paper.

The reading is fourfold. Removing other people's data (row 2) costs little. Removing other tasks (row 3) costs more. That ablation drops the data collected during online control and leaves only open-loop observation trials. Adding monkey data in any form (rows 4 to 7) lowers the score. That is the paper's first sign that broad pretraining can hurt, and it asks for more study. And the simple linear baseline (row 8) is far behind on this task.

Result 4: a person at the cursor

The last test is closed loop: one person, P4, implanted about two months earlier, controlling a cursor with the decoder live. Three decoders took turns for 40 center-out trials each: a linear one, and two NDT2 models, one pretrained on 10 minutes of the person's own data and one on all human and monkey data. NDT2's per-step compute ran in under 15 ms on a consumer GPU, inside the 20 ms bin. Tuning took under 5 minutes on a remote server.

NDT2 gave consistent control with no tuning at all, while the linear decoder sometimes failed in that setting. Both NDT2 models improved after tuning, with or without labels. But with supervised data the linear decoder gave the best control. The authors link this to a failure they call “pulsar” behaviour: the cursor moves in bursts, mimicking the ballistic reaches seen in the open-loop training data, while the simple decoder always gives smooth, if less stable, motion.

The largest cross-session NDT2, given 100 trials of a new session, reached a count loss like which from-scratch model?

Chapter 5

NDT3: One Flat Stream of Tokens

Put spikes and movement into a single sequence, so that decoding becomes the same next-token game a language model plays

Picture a conveyor belt in a sorting plant. Boxes of three kinds arrive in a fixed order: a box of neural counts, a box of cursor velocity, a small tag. The worker at the belt does one job: look at everything that has passed and say what the next box holds. Nothing else is needed. Neither the belt nor the worker cares what the boxes mean.

NDT3 is that worker. Its authors changed the design in two big ways from NDT2. They dropped the masked autoencoder for a plain autoregressive Transformer, which predicts each token from the tokens before it, the way a language model predicts the next word (see the language model lesson). And they put behaviour into the sequence next to the spikes. Decoding is then just predicting the behaviour tokens.

The data: 2,000 hours, mostly reaching

NDT3 pretrains on spiking activity from motor cortex of more than 30 monkeys and humans, paired with movement. The text rounds the total to 2,000 hours (2 khr). The precise count is about 1,750 hours, 3 million sequences and 1 billion neural tokens. Most of it is reaching and grasping in short repeated trials.

The paired signals come in three types. Kinematics are positions of a limb, converted to velocities. EMG (electromyography) is the electrical activity of muscles. Force is how hard someone pushes against a handle. A person with paralysis cannot move, so the paired signal is what the screen asked for, and the person attempts or imagines it. The authors normalise each dataset so that the largest absolute value of each variable is 1.

They also include closed-loop BCI data, in which a decoder, not the person's body, produced the cursor movement. That data teaches the model less about the brain, so it needs special handling (Chapter 6).

Preprocessing is deliberately light: cut or join the data into fixed sequences of two seconds, roughly the time scale of the behaviour, with no marks between trials. Discard sequences that have no spikes or no movement. This is how language-model training packs documents, and it keeps every GPU batch full.

The tokens

Each 20 ms step of real time becomes several tokens:

Every kind of token has its own linear read-in (a matrix that lifts the raw numbers into the model's width) and its own linear read-out. There are no task-specific layers. A different dataset just supplies a different number of tokens per step, so NDT3 handles any array size or any number of behaviour dimensions with the same weights.

The order and the clock

Inside one step, the tokens are sorted in a fixed, canonical order, drawn in the paper's Figure 2D, so that decoding stays causal. Neural tokens come before the behaviour tokens of the same step. So when the model predicts behaviour at a step, the neural activity of that same step is already in view. That is the order a real-time decoder needs.

The model must know when each token happened. NDT3 uses rotary embeddings (RoPE). The idea: rotate each token's query and key vectors by an angle proportional to its time. Then the attention score between two tokens depends on how far apart they are in time, not on where they sit in the list. All the tokens of one step share one time. A small table of 48 learned labels tells the tokens within a step apart: 10 for neural, 16 for covariates, 16 for covariate constraints, 1 for reward or return, and 1 for dummy tokens.

Build one time step

The row shows two consecutive steps of the stream. Choose training or use, then change the array and the behaviour dimensions. Hatched boxes are hidden from the model.

96
2

Token counts follow the paper’s rules: 32 channels per neural token, one token per behaviour dimension, a phase token and a return token at pretraining, 100 steps in two seconds. The 350M model fits 4 to 8 thousand tokens per context. The order of boxes in the picture is our drawing of Figure 2D; the paper fixes one canonical causal order and gives no single count of tokens per step.

Count the tokens: 96 channels give 3 neural tokens, 2 behaviour dimensions give 2, plus a phase token and a return token. That is 7 tokens per step, 700 for the 100 steps in a sequence. A 288 channel array with 10 behaviour dimensions makes 9 + 10 + 2 = 21 per step, so 2,100 tokens.

The network and what it dropped

NDT3 ships in two sizes. Base has 6 layers, width 1,024, 8 heads, and 45 million parameters. Big has 12 layers, width 2,048, 16 heads, and 350 million. For comparison, NDT2's published model has 4 layers at width 256 and about 6 million. The feed-forward width equals the model width, which is unusually narrow, and the authors flag that as a possible place to gain.

Three modern parts keep training stable and fast. FlashAttention 2 speeds up attention (how it works). QK normalisation adds a layer norm on queries and keys, which helped the 350M model train. And the authors removed NDT2's context tokens. Which task a sequence belongs to is now settled in context, by what the earlier tokens show (Chapter 6 explains how). Inference takes 4 ms for the 45M model and 9 ms for the 350M model on an RTX 4090, inside the 20 ms step.

Fine-tuning keeps the same objectives and updates every parameter. No new head is needed, which the authors give as a reason NDT3 behaves better than NDT2 downstream: NDT2 must start new decoding layers for every task.

An array has 160 channels and the task has 3 behaviour dimensions. At use, how many neural and behaviour tokens does one time step contain?

Chapter 6

Training NDT3: Losses and the Copy Shortcut

See why a model fed the previous movement just copies it, and how masking part of the behaviour inputs forces it to read the brain

A student takes a test where each question says, in small print, “the answer to the previous question was 17”. Most answers are close to the one before. The student stops reading the questions and writes the previous answer, plus a little. She scores well on the practice test. Then the small print disappears, and she scores almost nothing.

NDT3 faces this problem exactly. Its training sequence holds both the spikes and the behaviour. Cursor velocity changes slowly, so the behaviour at one 20 ms step is almost the same as at the step before. A model that is allowed to see the last behaviour token can predict the next one without looking at a single spike.

The two losses

Training needs one number that says how wrong the model is. NDT3 adds up two kinds of error, one for each part of the stream.

L
The total loss for one batch. Training lowers it. The return tokens add a cross-entropy term of their own.
wn, wb
Fixed weights. The paper sets them so the two parts are roughly balanced.
CE(spike counts)
Cross-entropy. Each spike count is treated as a class (0, 1, 2 and so on). The model gives every class a probability, and the loss is minus the log of the probability of the true count.
MSE(behaviour)
Mean squared error. The average of the squared gap between true and predicted behaviour.

A worked case. The true count in one bin is 2. If the model gives the class “2” a probability of 0.25, the loss is −ln 0.25 = 1.39. If it gives 0.5, the loss is −ln 0.5 = 0.69. The better the guess, the smaller the cost. For behaviour, a true velocity of 0.6 and a prediction of 0.5 cost (0.1)2 = 0.01. NDT3 swapped NDT2's Poisson loss for cross-entropy; the paper says this choice should be tested on neural-prediction tasks, since its tests only measure decoding.

The shortcut, in numbers

The NDT3 appendix names the problem. Its default next-step objective fails to learn decoding of highly autocorrelated behaviour. It suggests the model relies on the teacher-forced behaviour inputs as a shortcut. Teacher forcing means feeding the true previous token during training, as language models do.

Here is the smallest model that shows it. Let the model's guess be a weighted sum of two clues: the previous behaviour p, weighted by a, and the neural clue n, weighted by w. The velocity has autocorrelation ρ: its value correlates with the previous step at ρ. The neural clue alone could explain 60% of the variance. Training picks the weights that minimise the average squared error. When the previous behaviour is always visible, this works out to w = (1 − ρ2) / (σ2 + 1 − ρ2), where σ2 is the noise of the neural clue. With ρ = 0.98 that is about 0.06. The model nearly ignores the brain.

Break the shortcut

Left: the weights the toy learns. Right: its score when the previous behaviour is visible and when it is masked, against the share of masked inputs in training. Drag the mask share, then the autocorrelation.

0%
0.98

A toy, solved exactly with least squares, not a trained network and not a figure from the paper. The neural clue explains 60% of the variance by itself (illustrative). The paper’s observed effect is the qualitative one: without masking, the ablated model decodes worse than a trivial guess when it must use its own predictions (appendix Fig. 7).

Set the mask share to 0% with ρ = 0.98. The score with visible inputs is high, and the score once they are masked is near 0.1. That is the student without the small print. Now drag the mask share up. The weight on the neural clue rises, and the masked score climbs toward the 0.6 ceiling. Set ρ to 0 and the problem disappears: when there is no autocorrelation, the previous value is no clue.

The fix: covariate dropout

NDT3 uses a simple procedure, with no architecture change. On every training batch, draw two random numbers:

  1. M, uniform between 0 and 1: the fraction of behaviour inputs to hide.
  2. On 90% of batches, T, uniform between 0 and 2 seconds: the hiding starts only after time T. Before T, the true behaviour is shown. That stretch is a prefix prompt.

The loss is still computed on the prefix too. Validation always uses a prefix and masks the rest. At use, all behaviour inputs are masked. The prefix does a second job: it can show the model, in context, what the behaviour dimensions mean, so one network serves many tasks without task embeddings. In the ablation without masking (appendix Fig. 7), the model performed “subtrivially” when it had to use its own earlier predictions.

The other training choices

Training settings: batch size 16,000 sequences; early stopping on validation loss with a limit of 400 epochs; peak learning rate 4e−4 for the 45M model, lowered to 1e−4 because the 350M model diverged at the higher rate. Fine-tuning tries three rates and three seeds.

With ρ = 0.98 and no masking in training, why does the toy decode poorly once its behaviour input is masked?

Chapter 7

Scale and the 1.5 Hour Wall

Follow NDT3 from 1.5 to 2,000 hours of pretraining and from 45 to 350 million parameters, and find where the extra data stops paying

A baker has three questions about a bigger oven. Does more dough need a bigger oven? Does any dough benefit? And is there a loaf size past which a bigger oven changes nothing? The NDT3 authors ask the same three about data and model size, and they answer each with a number.

The ladder

The team trained models at several data sizes: 1.5, 25, 70, 200 and 2,000 hours. Each larger set contains all the smaller ones. At 45 million parameters they trained the whole ladder. At 350 million they trained the 200 and 2,000 hour rungs. All pretraining data include 1.5 hours of calibration data for the main test, so even the smallest model has seen a little of the test task.

Then they fine-tuned. They tuned about 2,000 models across 31 evaluation settings: four human and four monkey datasets, each used at several downstream sizes. All the test monkeys, and (for models below 2,000 hours) all the test humans, were held out of pretraining. So the tests measure transfer to a new person or animal, which is the case a BCI company cares about.

Test 1: does 200 hours of other monkeys help?

The first test (Figure 3A) asks the question on the pretraining task itself. Take the 1.5 hour model as the reference. Add 200 hours from ten other monkeys: no improvement. Add 200 hours from the same monkey in other sessions: an improvement. Cross-subject data are too distant to help a model that already has 1.5 hours of its own.

The surprise comes at 2,000 hours. The 45 million model, trained on the full set, scores below the 45 million model trained on 200 hours. Dissimilar data interfered with learning the test task. Raising the size to 350 million repairs most of it, though the result stays near the 200 hour reference. The paper cites the standard advice to grow data and model together.

Test 2: the whole spread of tasks

The aggregate (Figure 3C) averages the 31 settings. Here is the shape:

Individual datasets disagree. In a human bimanual cursor task, pretraining helps at every downstream size. In the monkey self-paced reaching task, a from-scratch NDT3 is competitive at all but the smallest size. The paper says plainly that studying one dataset gives variable conclusions, and so a foundation model has to be graded across many.

Test 3: the wall

Now the key figure (3E). For each setting the authors divide performance by that of the 350M, 200 hour model, then plot it against the hours of downstream data. The spread among the models is wide at small downstream sizes. It vanishes by 1.5 hours of downstream data. Pretrained and from-scratch models meet.

That explains the first test, too. The upstream test already had 1.5 hours of calibration data, so no pretrained model could pull ahead of it by much. The authors conclude, provisionally, that further scaling of NDT-style decoders will be productive mainly when the new user brings less than 1.5 hours of data. A few sessions of data collection can reach 1.5 hours. Drag the two controls below.

Climb the ladder

Left: the models the paper trained, by pretraining hours and parameters. Right: how much a pretrained model still gains over scratch as your own data grows. Pick a model and drag your hours.

0.3 h

The seven models and the 1.5 hour crossover are the paper’s (Figs. 3C and 3E; A100-hours from Sec. 2.2). The gain curve is a straight line on a log axis that reaches zero at 1.5 hours: an illustration of Fig. 3E’s message, not a measurement.

What it costs

The paper gives the bill. The 45M, 200 hour model trains for 480 A100 GPU-hours. The 350M, 2,000 hour model needs 20,000. Let us check the authors' FLOP estimate (a FLOP is one arithmetic operation). Kaplan and colleagues give a forward pass per token of 2N + 2 · nlayer · nctx · dattn. For the 350M model the paper reports about 0.9 billion FLOPs per token. Training also needs a backward pass, about twice the forward cost, so a rule of thumb is three times the forward cost. With about 0.9 trillion neural tokens processed:

3 × 0.9×109 × 0.9×1012 ≈ 2.4 × 1021 FLOPs

That matches the paper's figure of about 2.4e21. The multiplication by three is our reading of the standard rule, not a line from the paper. For use, FlashAttention 2 keeps the 350M model at 9 ms per step on an RTX 4090. See scaling laws for why data and parameters are scaled together.

Two more honest findings

Upstream and downstream do not always rise together. A small test found that the downstream gain of a 45M, 25 hour model appeared by the first checkpoint, at epoch 20, and then stayed flat, while the pretraining loss kept falling. And on handwriting decoding (FALCON H2, a sequence of letters trained with a CTC loss), pretraining did not help. Fine-tuning even narrowed the range of settings that trained at all.

The 45M model pretrained on 2,000 hours scored below the 45M model pretrained on 200 hours on the upstream test. What repaired most of the loss?

Chapter 8

Where It Breaks, and Where It Holds

Test NDT3 with shuffled inputs, held-out reach angles, and a benchmark it loses, to see which limits come from the data and which from the method

A good mechanic does not only show you the car running. She also shows you what happens when you remove a spark plug. The NDT3 authors do the same with their model. After showing that pretraining helps, they poke it in the places where it should be strong, and report where it fails.

Two sources of variability

The introduction separates two reasons why one person's brain data differ from another's. Sensor variability is physical: different electrodes, different neurons recorded. Picture two toy subjects with two neurons each. In one, neuron A fires for leftward motion and B for rightward. In the other, it is the reverse. No amount of pretraining can say which neuron is which. The new subject must supply data to tell. Residual variability is the rest: differences in behaviour and in the underlying dynamics. Shared structure between datasets fades quickly beyond the top principal components.

Both put a floor under what pretraining can do. The question is how high the floor sits, and whether it is the data or NDT3's design that sets it.

Test 1: scramble the inputs

The experiment (Figure 4A and 4B) works like this. Take a model, pretrained or not. Tune it on an intermediate dataset: several hours of data from another session, or from another subject. Then tune on a single minute of the real target session. If intermediate data help, they should lift the final score above tuning on the one minute alone.

Cross-session intermediate data keep helping, even after 2,000 hours of pretraining. Cross-subject data do not. For the pretrained model, they add nothing beyond the one-minute baseline. So pretraining already took all the cross-subject benefit that was on offer, and what is left is blocked by the subject differences.

To pin down the cause, the authors alter the cross-session data so that they look like another subject's, in three strengths. Each device cell below is one channel. Tokens hold 32 channels.

Scramble the channels

The top strip is 96 channels in 3 tokens of 32, shaded by original token (1, 2, 3). The bottom strip shows what the model receives after the alteration. Choose one.

The 96 channels, 32 per token, and the three alterations are the paper’s (Fig. 4A and 4B); the purity numbers are computed here from one seeded shuffle. Whether transfer survived each alteration is the paper’s finding.

Only the mildest alteration, the token shuffle, kept the pretrained model's gain. With the half-token shift or the full channel shuffle, no run beat the pretrained baseline. The conclusion: NDT3's cross-session transfer, and so its downstream scaling, depends on consistent neural inputs at the level of whole tokens. This is a sharp limit. A new subject's electrodes will never line up with the training set channel for channel. The authors also tried neuron-level tokens, as POYO does, which still require knowing which channel is which, and they performed poorly in the authors' hands.

Test 2: held-out reach angles

The second test asks whether a pretrained decoder can predict a reach it never saw. In a monkey force task with eight reach angles, the authors train on three and hold out five. Neural activity during centre-out reaching lies close to a plane, so a simple method can do this by construction. They reduce the data with PCA and LDA, then fit a Wiener filter. That linear model handles the held-out leftward reach.

NDT3 does not. A from-scratch model predicts only along the angles it has seen. The pretrained model improves overall and in the three held-in angles, but still fails on the five held-out ones. Its predictions fall onto the nearest trained direction. The authors call these attractors. The paper adds a caution: this does not show that pretraining cannot learn a smooth decoder. A light post-training step might discourage the attractors.

Reach where it never trained

Dots on the ring are the eight true reach angles. Filled dots were in training. Arrows show a decoder’s prediction for each. Switch the decoder.

A schematic of the paper’s Fig. 4D. Which three angles are held in here is our arbitrary choice, and the linear model’s small errors are illustrative. The attractor rule (predict the nearest trained angle) is a description of the behaviour the paper reports, not NDT3’s computation.

Where it holds

Section 3.3 is the good news, under real shifts:

The benchmark it loses

The Neural Latents Benchmark scores how well a model predicts the firing of held-out neurons, a pixel-level task. NDT3 does poorly there (co-smoothing bits per spike, higher is better):

DatasetNDT1Best publishedNDT3 scratchNDT3 350M 2 khr
RTT0.16430.20100.15330.1695
Maze0.35970.36500.30930.2775
Maze Large0.37390.38310.28920.2781
Maze Medium0.30810.33290.18590.1937
Maze Small0.27880.34580.18530.2116

From Table 1 of the NDT3 appendix. The best published entries are LangevinFlow (RTT and Maze) and MINT (the Maze scaling sets). Pretraining improved NDT3 on three of the five rows (RTT, Maze Medium and Maze Small), yet the model stayed well behind. Both papers say so openly. NDT2 also did not exceed the benchmark's best on its motor sets.

Which alteration of cross-session data did NDT3’s pretrained model survive, still gaining from it?

Chapter 9

Field Guide: NDT2 and NDT3 on One Page

Keep the numbers, the terms, a checklist and the core code, and find where to read next

Two papers, one thread. NDT2 showed that a Transformer pretrained by masked autoencoding on spikes from other sessions, subjects and tasks tunes faster to a new session. NDT3 turned that into a 45 and 350 million parameter autoregressive model on 2,000 hours, and then tested its limits. Pretraining helps most when the new user has under 1.5 hours of data. It is brittle when the input channels change, and it snaps unseen reaches onto trained ones.

The numbers

WhatValueWhere
Time bin20 ms (50 Hz), both papersNDT2 Sec. 3.1; NDT3 Sec. 2.2
NDT2 window and budget2.5 s = 125 steps; 2,000 tokensNDT2 Sec. 3.1
Patch size16 to 32 channels (NDT3 mostly 32)NDT2 Sec. 3.1; NDT3 Fig. 2
NDT2 mask ratio50%, dropout 0.1NDT2 Sec. 4
NDT2 size6-layer encoder (about 3M) + 2-layer decoder (0.7M)NDT2 Sec. 4
NDT2 test budget300 trials, 5 minutes per sessionNDT2 Sec. 4.1
Cross-session gain100 trials match 1,000 scratch trials in count lossNDT2 Fig. 4B
NDT2 crossoverGains last to 3,000 trials, about 50 minutesNDT2 Fig. 4C
Best unsorted decodeR2 above 0.7 (cross-session)NDT2 Fig. 3B
Human offline R2P2 0.503, P3 0.515 (all human data)NDT2 Table 2
Real-time loopUnder 15 ms; tuning under 5 minutesNDT2 App. A.3
NDT3 pretraining data2,000 h (about 1,750 h, 3M sequences of 2 s, 1B neural tokens), 30+ subjectsNDT3 Sec. 2.1
NDT3 sizes45M (6 layers, width 1,024); 350M (12 layers, width 2,048)NDT3 Table 4
NDT3 compute480 A100-h (45M, 200 h); 20,000 A100-h and about 2.4e21 FLOPs (350M, 2 khr)NDT3 Sec. 2.2, A.3.2
NDT3 evaluationAbout 2,000 models, 31 settings, 8 datasetsNDT3 Sec. 3.1
The wallPretrained and scratch models meet at 1.5 h of downstream dataNDT3 Fig. 3E
Inference4 ms (45M), 9 ms (350M) on an RTX 4090NDT3 App. A.3.6
Covariate dropoutM uniform on [0, 1]; prefix T uniform on [0, 2] s in 90% of batchesNDT3 App. A.2.2
NLB RTT co-bpsNDT3 0.1695 against 0.2010 best publishedNDT3 App. Table 1

The terms

TermMeaning
iBCIA brain-computer interface that records inside the cortex.
ContextOne setting of the recording: a session, a subject or a task.
Masked autoencoderHide part of the input and train the model to restore it. NDT2.
PatchA group of channels at one time step that becomes one token.
Context tokenA learned vector for a session, subject or task. In NDT2, not NDT3.
AutoregressivePredicts each token from the tokens before it. NDT3.
Covariate dropoutHide a random share of behaviour inputs so the model must read the spikes.
Phase and return tokensTags that say whether behaviour came from the brain or a program, and how well the task went.
Sensor variabilityDifferent electrodes and neurons from one recording to the next.
AttractorA decoder output that snaps to a trained value instead of following the input.

A checklist for the next neural foundation model paper

  1. Which data scale does the claim cover? NDT3’s gains vanish at 1.5 hours of downstream data.
  2. Is the new subject held out of pretraining? Cross-session transfer is easy. Cross-subject transfer is the case that matters.
  3. What happens to the input mapping? Shuffle channels and tokens and see if the gain stays.
  4. Does the decoder extrapolate? Hold out behaviours, such as reach angles, and check for attractors.
  5. Is the score offline or closed loop? NDT2’s linear decoder won in the live pilot.
  6. Which loss and which score? Count loss and decoding R2 told different stories.
  7. Is a trial clock leaking in? Test continuous streaming against trial-aligned data.
  8. Which benchmark does it lose? Both papers report a negative NLB result.

The core as code

This is our compact sketch of NDT3’s recipe, not the authors’ code. The released code is at github.com/joel99/ndt3.

pythonimport torch, torch.nn as nn, torch.nn.functional as F

def patchify(spikes, K=32):                 # spikes: [T, C] counts, 20 ms bins
    T, C = spikes.shape
    P = -(-C // K)                          # ceil(C / K) patches per step
    spikes = F.pad(spikes, (0, P * K - C))  # zero-pad the last patch
    return spikes.float().view(T, P, K)     # [T, P, K]

def covariate_dropout(beh):                 # beh: [T, B], T = 100 steps (2 s)
    T = beh.shape[0]
    M = torch.rand(())                      # share of behaviour inputs to hide
    prefix = int(torch.rand(()) * T) if torch.rand(()) < 0.9 else 0
    hide = torch.rand_like(beh) < M
    hide[:prefix] = False                   # the prefix prompt stays visible
    return torch.where(hide, torch.zeros_like(beh), beh)

class Stream(nn.Module):                    # our sketch, not the authors' code
    def __init__(s, W=1024, K=32, max_count=20, n_kinds=48):
        super().__init__()
        s.read_n = nn.Linear(K, W)          # one read-in per modality
        s.read_b = nn.Linear(1, W)
        s.kind = nn.Embedding(n_kinds, W)   # which token inside a step
        s.write_n = nn.Linear(W, K * max_count)   # count classes per channel
        s.write_b = nn.Linear(W, 1)
        s.body = None                       # causal Transformer with rotary time

    def forward(s, spikes, beh_in, beh):
        T, P, K = spikes.shape; B = beh.shape[1]
        n = s.read_n(spikes) + s.kind.weight[:P]                       # [T, P, W]
        b = s.read_b(beh_in[..., None]) + s.kind.weight[16:16 + B]     # [T, B, W]
        x = torch.cat([n, b], 1).flatten(0, 1)                         # [T*(P+B), W]
        h = s.body(x, steps=torch.arange(T).repeat_interleave(P + B))
        h = h.view(T, P + B, -1)
        logits = s.write_n(h[:-1, :P]).view(T - 1, P, K, -1)           # next-step counts
        loss_n = F.cross_entropy(logits.flatten(0, 2), spikes[1:].long().flatten().clamp(max=19))
        loss_b = F.mse_loss(s.write_b(h[:, P:]).squeeze(-1), beh)      # behaviour, same step
        return loss_n + loss_b                                         # weights ~ balanced

# train:  loss = model(patchify(spikes), covariate_dropout(beh), beh)
# use:    beh_in = zeros(T, B): the inputs are masked, the outputs are the decode

Where it sits in the field

NDT2 compares its setting with pretraining on other neural recordings: BrainBERT on SEEG, BENDR on EEG, and BERT itself as the language yardstick. NDT3 names POYO and NDT-MTM as other scaled Transformers on spikes. Its tokenization follows GATO and Decision Transformers. Both papers belong to the thread that starts with latent-variable models of spikes, such as LFADS.

Keep going

Credit

NDT2: Joel Ye, Jennifer L. Collinger, Leila Wehbe and Robert Gaunt, University of Pittsburgh and Carnegie Mellon University. NeurIPS 2023. NDT3: Joel Ye and 23 co-authors from Carnegie Mellon, Pittsburgh, Northwestern, Chicago, the University of Kansas Medical Center and Lawrence Berkeley National Laboratory, with Robert Gaunt as the corresponding author. NeurIPS 2025. Every table value and claim above comes from these two papers; where we added arithmetic or a toy, the device says so. This page is a reading aid, not the papers.

The takeaway. Pretraining on other people’s spikes is a head start, not a replacement. It shortens calibration for a new user while that user has under about 1.5 hours of data, and it leaves two open problems that more data alone did not fix: inputs that change from person to person, and outputs that snap to what the model has seen.

Now press Present or Teach and explain from memory why a decoder that copies the previous movement fails, what covariate dropout changes, and where the 1.5 hour wall comes from. Then go back to the dial at the top and move both marks: the gap closes where the paper says it does.

A lab has 5 hours of calibration data from a new user. According to NDT3, what should it expect from a pretrained model compared with one trained from scratch?
NDT2 and NDT3
Read the NDT3 paper · Read the NDT2 paper · Back to LFADS and latent dynamics