Neural Data Transformers

A transformer reads 70 bins of spikes at once and recovers the hidden firing rates as well as a recurrent network does, in 3.9 ms instead of 26 ms.

Pick a model, then drag the bin width. Watch how late the cursor learns what the neurons just did. Then we build, piece by piece, the spikes, the attention, the masked training and the tests behind that 3.9 ms.

You need what a spike is (a brief electrical pulse from one neuron) and a feel for a matrix multiply. We build the rest from zero.

Based on Representation learning for neural population activity with Neural Data Transformers (arXiv 2108.01210, August 2021) by Joel Ye and Chethan Pandarinath. Authors’ code. This reading belongs to the lecture LFADS. Every idea is re-taught here in our own words and devices.

Race the cursor

10 ms

Inference times are the paper’s Table 2 (70 bins, one CPU: 26 ms, 3.9 ms, 2.2 ms, 0.98 ms). The paper adds the bin width to the compute time: 10 + 4 = 14 ms for the NDT and 10 + 26 = 36 ms for LFADS. The circling hand, one turn every 0.8 s, is illustrative, and times for other bin widths keep the 70-bin figures.

Chapter 0

Why a Cursor Cannot Wait

See why a brain-computer interface needs the newest spikes turned into a state within a few milliseconds, and why a recurrent model struggles to do it

A person with paralysis thinks about moving their hand. Tiny electrodes in the motor cortex (the strip of brain that plans and sends movement commands) record what the neurons do. A computer reads that activity and moves a cursor on a screen. The person watches the cursor, adjusts their intent, and the loop goes round again.

This loop is called closed loop, because the person sees the result of every command. Think of steering a car with a delay in the wheel. A short lag is fine. A long lag makes you overshoot, correct, and overshoot again. The paper cites several studies showing that delays of only a few milliseconds can lower the quality of closed-loop control.

What the computer receives

A spike is a brief electrical pulse from one neuron. An electrode channel reports when spikes happen. The software cuts time into short slices and counts the spikes in each slice. Each slice is a bin, and the count is the spike count for that bin. In this paper a bin is usually 10 ms wide.

Many neurons are recorded together. The set of all channels is the neural population. For the monkey data in the paper, that is 202 neurons. So every 10 ms the computer gets one new column of 202 small whole numbers.

Raw counts are noisy. A neuron that fires at the same average speed will give 1 in one bin and 0 in the next. What the software really wants is the neuron’s firing rate: the average number of spikes per second, behind the noise. The next chapter makes this precise. For now, treat the rate as the smooth hidden signal and the counts as a noisy peek at it.

The older tool: LFADS

A method called LFADS (latent factor analysis via dynamical systems; the lecture LFADS, builds it) infers smooth rates from noisy counts, trial by trial. It works well. It is built from a recurrent neural network, or RNN: a network that reads one bin at a time and carries a memory, called its state, from each step to the next.

Picture reading a book with your finger under each word. You cannot understand page 70 until your finger has passed page 69. An RNN works the same way. Step 5 needs the state from step 4. The steps form a chain, and a chain cannot be sped up by adding more processors.

LFADS adds a second problem. It first reads the whole window of bins with a two-direction RNN. It uses that summary to set the starting state of a second RNN, which then runs forward for the full window length. So every new bin means re-reading and re-running the whole window. The paper measures this: 26 ms for a window of 70 bins on one CPU.

The idea of the paper

Language researchers hit the same wall. Reading words one at a time was too slow to train on huge text, so they moved to the transformer. A transformer looks at all the words together, like glancing at a whole page at once. Joel Ye and Chethan Pandarinath ask a simple question: can the same trick infer firing rates?

Their model is the Neural Data Transformer, or NDT. The answer in the paper is yes, for one kind of data. On a monkey reaching dataset it matches LFADS at decoding hand velocity (0.918 against 0.915 R², a score Chapter 6 explains) and runs in 3.9 ms, which is 6.7 times faster. The authors conclude that an explicit dynamics model is not necessary to model autonomous neural population dynamics. Chapter 8 explains what “autonomous” limits.

Count the steps in a row

Speed here is not about a faster chip. It is about how many steps must wait for the one before. Drag the window length. The recurrent model works through its steps in order. The six-layer NDT finishes six layers, and each layer handles every bin together.

Count the steps in a row

Top: a recurrent model reads all bins, then runs forward over them, one step at a time. Bottom: six transformer layers, each one pass over every bin at once. Both rows advance one step per tick.

70

A counting model, not a clock. The recurrent row counts T encoder steps and T decoder steps in a row (the two reading directions run side by side). A transformer layer does more work than one recurrent step, so the measured gap at 70 bins is 6.7 times (26 ms against 3.9 ms, Figure 6), smaller than the step ratio.

The recurrent chain grows with the window. The transformer’s depth stays at six. That is why the paper’s Figure 6 shows LFADS time rising with window length while the NDT stays almost flat. The authors note that the NDT time still creeps up a little with longer windows.

One honest point. The paper tests speed on a rolling window, where each update re-processes the window from scratch for both models. A recurrent filter such as a Kalman filter can instead carry its state and add one bin at a time. The authors say no one has shown that for LFADS, and that its two-direction encoder is needed for good performance.

The road

  1. From spikes to rates: counts, the Poisson idea and the loss that scores a guess.
  2. Attention over time bins: queries, keys, values and a worked example.
  3. The NDT encoder: every tensor shape from spikes to rates.
  4. Training by hiding bins: masked modeling with a zero mask.
  5. Three choices that make it train: log-rates, the zero mask, heavy dropout.
  6. Does it recover the rates? Synthetic data, the monkey and what R² means.
  7. Speed against size: Table 2 and a deadline you set.
  8. Where it breaks: inputs the model cannot see, and what the authors propose next.
  9. Field Guide: the numbers, a checklist, the core code and where to go next.

A timeline of one answer

Follow one bin through the system. The bin closes at time 0. Compute starts. The new answer is ready after the compute time, and the cursor can show it.

With the 6-layer NDT the answer is ready at 3.9 ms. No new bin has arrived yet, since the next one closes at 10 ms. With LFADS the answer is ready at 26 ms. By then two newer bins have already closed, at 10 ms and 20 ms. The answer describes a moment that is two bins old.

There is a second cost. A fresh window arrives every 10 ms, and each window is processed from scratch. A model that needs 26 ms on one CPU can finish only 38 windows per second (1000 / 26), while 100 windows arrive. On one core it would fall further behind each bin. This is our arithmetic on the paper’s numbers, not a measurement from the paper.

Why does the NDT infer rates faster than LFADS on a window of 70 bins?

Chapter 1

From Spikes to Rates

Turn noisy spike counts into a hidden firing rate, and learn the one loss that scores every guess in this paper

Watch one neuron in the motor cortex while a monkey reaches. In ten bins of 10 ms you see the counts 0, 0, 1, 0, 0, 2, 0, 1, 0, 0. Is the neuron quiet or busy? You cannot tell from ten numbers. Run the same reach again and you get a different list.

Something steady sits behind the noise. The neuron has a firing rate: the average number of spikes per second it would produce if you could repeat the moment forever. The rate rises and falls smoothly as the reach unfolds. Spikes are random draws that follow it. The job of this whole paper is to read the rate back out of the spikes.

The Poisson idea

How random is a spike count? The standard model is the Poisson distribution: events that arrive independently at a steady average speed. Think of raindrops on a paving stone. You know the average is two drops per second. You never know exactly when the next lands, and one drop does not make the next more likely.

Let r be the average number of spikes in one bin. A neuron firing at 50 spikes per second, seen through a 10 ms bin, has r = 50 × 0.01 = 0.5. The Poisson rule gives the chance of seeing exactly k spikes:

P(k | r)
The chance of counting exactly k spikes in a bin whose average count is r. The bar reads “given”.
rk
Grows with the rate. A bigger r makes a bigger k more likely.
e−r
Makes the chances add up to 1. It is also the chance of seeing no spike at all.
k!
k factorial: 1 × 2 × … × k. It counts the orderings, so equal spike patterns are not double counted.

Work it for r = 0.5. A bin with no spike has chance e−0.5 = 0.607. One spike: 0.5 × 0.607 = 0.303. Two spikes: (0.25 / 2) × 0.607 = 0.076. Three: (0.125 / 6) × 0.607 = 0.013. These four add to 0.998, and longer counts share the rest. So a neuron with a rate of 50 spikes per second stays silent in 61 of 100 bins.

The loss: how bad is a guess?

Training needs a number that says how well a guess for r explains what was seen. Use the chance itself. A good guess makes the observed count likely. The standard form takes the negative logarithm, so a likely count gives a small loss. This is the Poisson negative log-likelihood, or NLL:

L(r)
The loss for one bin of one channel. Smaller is better.
r
The model’s guess for the average count. This is the number the network outputs.
k
The spike count that was actually observed in that bin.
log r
The natural logarithm of the guess. It punishes a guess near 0 when spikes did occur.
log k!
Does not depend on r, so training ignores it.

Take one bin where k = 1. Guess r = 0.5: loss = 0.5 − 1 × ln 0.5 = 0.5 + 0.693 = 1.193. Guess r = 1: loss = 1 − 0 = 1.000. Guess r = 2: loss = 2 − 0.693 = 1.307. The best guess is r = 1, the observed count. For k = 0 the loss is just r, so the best guess is as close to 0 as possible.

This shows a trap. If the model sees a bin and is scored on the same bin, the lowest loss is to copy the count. That teaches nothing about the hidden rate. Chapter 4 fixes it. The paper reports the NLL averaged over every bin and every channel.

Why many neurons help

One neuron gives a thin signal. But the neurons are not independent. The theory behind this field says a population is driven by a small hidden state that changes over time by its own rule, like a ball rolling in a landscape. Each neuron’s rate is a fixed blend of that state. Many noisy neurons share one low-dimensional story, so reading all of them together recovers it.

The paper tests this on made-up data. A 3-number state follows the Lorenz equations (a classic chaotic system) and is projected up to 29 channels to make rates. Spikes are then drawn from a Poisson distribution at those rates. Because the true rates are known, the model can be graded against them. Chapter 6 does exactly that.

The baseline: just smooth

The simplest way to read a rate is to blur the counts. Replace each bin by a weighted average of its neighbours, with weights shaped like a bell curve. The bell’s width, its standard deviation, is the smoothing width. The paper uses a width of 30 ms as its spike smoothing baseline. Try it. Too narrow leaves noise. Too wide flattens the real peaks.

Smooth the spikes

Blue bars are spike counts per 10 ms bin for one made-up neuron. The dashed warm line is its true rate, which a real recording never shows. The teal line is your smoothed estimate. Drag the width, or press Resample for new spikes.

30 ms

Toy data, seeded: a rate of 10 spikes per second plus two bumps (peaks near 60 and 35 spikes per second), 70 bins of 10 ms, counts drawn from a Poisson distribution. Error is the root mean square gap, in spikes per second, between the teal line and the true rate. Width 0 means raw counts divided by the bin width.

Smoothing treats every neuron alone and every bin by its distance in time. It cannot use what the other neurons say. It also cannot know that a rate follows a dynamical rule. A model that learns from the whole population can do better, and that is what LFADS and the NDT try to do.

A model is scored on the Poisson NLL of a bin it can see. What does it learn to output for that bin?

Chapter 2

Attention Over Time Bins

Let every bin ask which other bins help explain it, and build its new description as a weighted average of their answers

You want the firing rate in bin 40. The count in bin 40 alone is a noisy peek. Bins 39 and 41 help, since rates change smoothly. A bin from 60 ms earlier, where the population showed the same pattern, might help too. Which bins deserve your trust, and by how much?

Self-attention answers with a simple recipe. Every bin looks at every bin, scores how useful each one is, and builds a new description of itself as a weighted average of all of them. The weights are not fixed by hand. The network learns how to score.

Bins become vectors

A network cannot work with a list of spike counts one number at a time. It first turns each bin into a vector, a short list of numbers, written xi for bin i. In the NDT the starting vector is the bin’s counts across all channels. The paper also lets each channel’s count be copied into a small embedding of size n, so the vector has C × n numbers. Chapter 3 shows the shapes.

Queries, keys and values

Think of a library. You walk in with a question (a query). Every book has a label on its spine (a key) and contents inside (a value). You compare your question with each label, pick the best matches, and read a blend of their contents.

In attention each bin plays all three roles. Three learned weight matrices, WQ, WK and WV, each of size d × d, turn a bin’s vector into its query, its key and its value. So qi = WQ xi, kj = WK xj and vj = WV xj. Here d is the vector size, kept the same all through the network.

yi
The new description of bin i. It is the output of the attention block.
∑j=1T
Add over all T bins in the window, including bin i itself.
wij
The attention bin i pays to bin j. Weights are positive and add up to 1 over j.
vj
The content bin j offers.
qi · kj
A dot product: multiply matching entries and add. It is large when the query and key point the same way.
√d
The scaling from the original transformer paper, which the NDT keeps (Section 7.2).

The softmax turns any list of scores into positive weights that add to 1. It does this by taking e to the power of each score and dividing by the total. Big scores get big weights. Because it uses exponentials, a gap of 3 in score already means a weight about 20 times larger.

Why divide by √d? A dot product of two d-sized vectors grows as d grows. Large scores push the softmax toward one winner and nearly zero for everyone else, and then learning slows. Dividing by √d keeps the scores at a steady size whatever d is.

A worked example, three bins

Take d = 2 and pretend the weight matrices are the identity, so q, k and v all equal x. Let x1 = (1, 0), x2 = (2, 1) and x3 = (0, 1). Compute the output for bin 2.

  1. Scores. q2 = (2, 1). Dot with each key: 2 × 1 + 1 × 0 = 2; 2 × 2 + 1 × 1 = 5; 2 × 0 + 1 × 1 = 1.
  2. Scale. Divide by √2 = 1.414: 1.414, 3.536, 0.707.
  3. Softmax. e raised to each: 4.11, 34.33, 2.03. They total 40.47. The weights are 0.102, 0.848 and 0.050.
  4. Blend. y2 = 0.102 × (1, 0) + 0.848 × (2, 1) + 0.050 × (0, 1) = (1.80, 0.90).

Bin 2 listens mostly to itself, since its query matches its own key best. Without the scaling step the weights would be 0.047, 0.936 and 0.017, which is sharper. Try both below.

Order has to be added

There is one trap. Look at the formula: shuffle the bins and every output is unchanged, only reordered. Attention alone has no idea which bin came first. Time order matters for spikes, so the NDT adds a position embedding to each input: a learned vector that acts as a name tag for that time step. Now a query can prefer keys that sit close in time.

The paper also mentions multi-head attention: several copies of the recipe side by side, with outputs joined. Language models use eight or more. The authors found no benefit from it and keep h = 1 or 2.

Watch one bin attend

Twelve bins in a row. Pick the query bin with the slider. Purple bars show how much weight it gives each bin. Switch scaling and position tags and watch the weights change.

5

Toy, not trained: d = 4, the vectors follow a looping path so that bins 7 apart look alike, and WQ = WK = WV = identity stands in for learned weights. Position tags are a fixed cosine and sine pattern here; the paper learns them. The count of bins averaged is 1 / ∑w².

Notice what comes back: with plain attention, the query bin listens to bins far away that happen to look alike. With position tags it also favours near neighbours. A real NDT learns which mix it needs, layer by layer.

Shuffle the order of the bins fed to a plain self-attention block with no position embedding. What happens to the outputs?

Chapter 3

The NDT Encoder, Shape by Shape

Follow one trial of spikes through the network and name the size of every tensor from input to inferred rates

Imagine handing a monkey trial to the network like a parcel on a conveyor belt. The parcel is a table of counts. At each station the table is reshaped, mixed or squashed, and a label tells you its size. This chapter is the list of stations. If you can recite the sizes, you can rebuild the model.

The parcel

A tensor is a table of numbers with named sides. The input is a table of spike counts with T rows, one per bin, and C columns, one per channel. For the monkey reaching data the paper trains on the span from 250 ms before movement onset to 450 ms after. At 10 ms bins that is T = 70 rows. There are C = 202 channels. So the input is 70 × 202, which holds 14,140 whole numbers.

Station 1: embed, then tag with position

The paper allows an optional step: each channel’s count is copied into an embedding of n numbers, with n either 1 or 2. The vector for one bin then has d = C × n numbers, so d = 202 for n = 1 and d = 404 for n = 2. The tensor becomes T × d.

Next the network adds a learned position embedding of the same shape, one vector per time step, as Chapter 2 explained. Adding keeps the shape at T × d. Then a dropout layer acts on it. Dropout sets a random share of the numbers to zero during training, so the network cannot lean on any single number. It is a kind of insurance against memorising.

Station 2: the encoder layers

The body is a stack of identical encoder layers. The main experiments use 6, and Chapter 7 tests 2 and 1. One layer has two blocks.

  1. Self-attention. Three d × d weight matrices make Q, K and V, each T × d. The score table Q KT / √d is T × T: 70 × 70 = 4,900 scores for each head. After the softmax, multiplying by V gives T × d again.
  2. Add and normalise. The block output is added to its input, the circled plus in the paper’s Figure 2. A layer norm then rescales each bin’s vector to zero mean and unit spread, which keeps numbers in a healthy range from layer to layer.
  3. MLP. A multi-layer perceptron is two linear layers joined by a ReLU (it keeps positive numbers and zeroes negative ones). It treats each bin on its own, so it adds no mixing across time. Dropout follows each linear layer.
  4. Add and normalise again. Same idea, shape still T × d.

One detail matters for speed and memory. The paper sweeps a context span, the number of bins forward and backward that each bin may gather from, between 10 and 50 on the reaching data. Attention is also bidirectional: a bin may use bins after it in the window. In a live decoder the newest bin sits at the window’s end, so it sees only the past.

Station 3: the head

After the last layer a final dropout acts. A linear layer maps each bin’s d numbers down to C numbers, so the tensor is T × C again. These numbers are treated as log firing rates. An exponential turns each into a positive rate, since e raised to any real number is above 0. The result, 70 × 202, is the inferred rate for every bin and channel. The Poisson loss from Chapter 1 compares it with the observed counts.

Count the cost. The paper reports 1.36 million trainable parameters for the 6-layer model, 480 thousand for 2 layers and 270 thousand for 1 layer. The LFADS model it compares against has 280 thousand.

StationWhat happensShape (n = 1, monkey data)
Input spikesBinned counts, 10 ms70 × 202
Embed (n = 1 or 2)Each count becomes n numbers70 × 202 or 70 × 404
Position embeddingAdd a learned vector per time step70 × 202
Attention scoresQ KT / √d, per head70 × 70
Layer outputAttention, norm, MLP, norm70 × 202
HeadLinear to C, then exp70 × 202 rates

Now follow the parcel yourself. The slider moves along the stations. Switch the embedding size and see which tensors double.

Ride the conveyor

Each box is one tensor. Its width follows the number of columns and its height the number of rows. Slide the station, or press Play to send a parcel through.

Spikes

Shapes follow the paper (Section 2 and 7.2) for the monkey reaching setup: 70 bins, 202 channels. The paper does not give the MLP hidden width, so the device leaves it out; read the authors’ code for it. The paper’s text lists attention, norm, MLP, norm, while its Figure 2 draws the norms slightly differently, so check the code for exact placement.

Everything in the body works on a T × d table. That is why the model is easy to run in parallel: the matrix multiplies for all 70 bins are one big multiply.

Check the sizes by hand

Count the weights in two pieces for n = 1, where d = 202. The final linear layer maps 202 numbers to 202 numbers. That is 202 × 202 = 40,804 weights plus 202 biases, so 41,006 numbers.

The query, key and value matrices are each 202 × 202 too. That is 3 × 40,804 = 122,412 weights per attention block, before biases. Layers repeat such blocks, so the total grows with depth. The MLP width is not in the paper, so we do not total the rest. These counts use our teaching shapes, so they will not match the paper’s 1.36 million. Do not try to.

With 70 bins, 202 channels and n = 1, what is the shape of the score table inside one attention head?

Chapter 4

Training by Hiding Bins

Cover some bins, ask the network to guess their rates from the rest, and score only the guesses

A teacher shows you a sentence with one word covered: “The cat sat on the ___.” You must guess the word from the others. To do well you have to learn how words fit together. Nobody gave you labels. The sentence itself was the exam.

This is masked modeling, the training idea behind the language model BERT. The NDT borrows it. The neural recording is the sentence, the bins are the words, and the covered bins are the blanks. No one needs to label the true firing rates, which no one knows anyway. The method is unsupervised: it learns from the data alone.

Why hide anything

Recall the trap from Chapter 1. If the network sees a bin and is scored on that same bin, the cheapest answer is to copy the count. It would learn nothing about rates. Hiding the bin removes the shortcut. To guess a hidden bin well, the network must use the visible bins around it and the other channels. If counts in the visible neighbours are high, the hidden bin likely had a high rate too.

The old LFADS has a cousin of this idea, called coordinated dropout. It hides single channel values at random. The NDT hides whole time steps instead. A hidden step has all its channels hidden together.

One training step, in order

  1. Take a batch of trials: spikes S of shape B × T × C, with B trials in the batch.
  2. Pick the hidden bins. Choose random bins, often in short runs called spans. The reaching experiments use spans of 1 to 7 bins. The set of hidden bins is the mask M.
  3. Zero them. For a hidden bin, set its whole row of counts to 0 in the input. The paper calls this the zero mask.
  4. Run the network on the damaged input. Out comes a rate table B × T × C.
  5. Score only the hidden bins. Average the Poisson loss L(r) = r − k log r over the cells in the mask. The target k is the original count, before zeroing.
  6. Update the weights to lower that loss, and repeat.

Why zeros and not a special token

BERT covers a word with a special symbol, [MASK]. That symbol never appears in real text at test time. The paper says this creates a large shift between training and use. The NDT avoids it. A zero count is an ordinary spike-count value, and most bins in spiking data hold few spikes anyway. So a zeroed bin looks like data the network will meet again. (That last reason is our gloss; the paper reports the effect, and Chapter 5 shows how large it is.)

The paper adds two BERT details as knobs. Of the hidden bins, only a share is truly zeroed. For the reaching data the zeroed share is swept between 0.6 and 1.0. Of the remainder, 0.6 to 1.0 are replaced by random counts, and the rest are left unchanged. All three treatments teach the network not to trust any single input blindly.

At test time nothing is hidden

When you use the trained model, the whole window goes in with no mask. The network outputs a rate for every bin. For a live decoder the window ends at the current bin. The authors also pick checkpoints this way: validation NLL is measured without masking, and early stopping keeps the checkpoint with the lowest value.

This matters for choosing models. The true rates are unknown for real data, so the search cannot score against them. Likelihood is available everywhere. The paper checks that it is a good stand-in. On synthetic data, models with better held-out likelihood also recover the true rates better (Figure 4b). The same trend holds for matching the empirical PSTH on the monkey data (Figure 5b), where the PSTH is the average of rates over repeats of the same reach.

The search itself is plain: a grid of 20 models, three searches per experiment, 60 models in all. Dropout between 0.2 and 0.6, the context span, mask lengths and ratios are the swept knobs. LFADS gets a smarter search called population-based training, and the authors report that NDT does about the same with either.

Hide, then guess

Top: spike counts for 8 channels over 30 bins. Pink columns are hidden and zeroed. Bottom: a stand-in guess for every cell, warm where the rate is high. Change how much is hidden and how long each hidden run is.

25%
2 bins

Toy, not a trained NDT: 8 channels, 30 bins, rates between 0.1 and 1.6 spikes per bin, counts drawn from a Poisson distribution. The stand-in guess averages the visible bins of the same channel with a bell-shaped weight of width 2 bins. NLL here is r − k log r per cell, with the constant dropped. The “true rate” score uses the hidden generating rates, which real data never gives.

Look at what the toy shows. Short runs are easy to fill: both neighbours are visible. Long runs leave a gap with no nearby evidence, and the guess drifts away from the true rate. A real NDT also uses the other channels, which is why a population model can beat a per-channel smoother.

How many cells does a mask cover?

Take our code sketch in Chapter 9 with a ratio of 0.25 and a span of 2. Each bin starts a hidden run with chance 0.25 / 2 = 0.125. A bin is hidden if either of the two starts that cover it fired. The chance is 1 − (1 − 0.125)2 = 0.234.

On 70 bins that hides about 16 bins. Each hidden bin has 202 channels, so about 3,300 cells carry the loss for one trial. The other 10,800 or so cells only serve as context. The paper sweeps its own ratios and spans, so treat 0.25 and 2 as a teaching choice.

In one NDT training step, which cells count toward the loss?

Chapter 5

Three Choices That Make It Train

See what happens to the score when you remove the log-rate output, the zero mask or the heavy dropout, one at a time

A baker leaves out the salt, the yeast or the oven heat, one at a time, and tastes each loaf. That is an ablation: remove one part of a method and measure how much worse it gets. A part whose removal changes little was not needed. A part whose removal wrecks the result was essential.

The paper runs this test on three design choices of the 6-layer NDT. All three look small. All three matter a lot. The score below is the R² of decoding hand velocity from the inferred rates on the monkey data. Chapter 6 explains this score. Higher is better, and 1 is perfect.

Variant (paper Table 3)R² (mean ± SEM)
Baseline NDT-60.918 ± 0.003
Output rates, not log-rates0.292 ± 0.104
Use a “[MASK]” token, not zeros0.022 ± 0.012
Constrain dropout to 0.0 to 0.20.549 ± 0.053

Each interval is the SEM (standard error of the mean) over three separate searches of 20 models. So a score of 0.292 ± 0.104 is both poor and unstable. Let us take the choices in turn.

Choice 1: output log-rates

The last linear layer of the network produces a number z for each cell. The NDT treats z as the log-rate and sets the rate r = ez. The alternative is to let the layer output r directly.

The paper finds that outputting rates trains less stably and converges to poor solutions over a wide set of settings (its Figure 7). This happens for 2-layer and 6-layer models alike. The authors conclude that the NDT cannot learn to exponentiate well on its own, even with long training.

Here is one simple reason to see in a toy. This is our illustration, not the paper’s proof. Take one bin with observed count k. The loss in terms of the rate is L = r − k ln r. Its slope is 1 − k/r. When the guess r is small and k is large, that slope is huge and negative, so one step overshoots. With k = 0 the slope is a constant 1, so a big enough step pushes r below zero, where ln r does not exist.

In terms of the log-rate the loss is L = ez − kz, with slope ez − k = r − k. The guess r = ez can never fall to zero or below. Work one step by hand. Start at r = 1 and k = 0, with step size 1.5. Rate space: r becomes 1 − 1.5 × 1 = −0.5, which is invalid. Log-rate space: z = 0 becomes 0 − 1.5 × 1 = −1.5, so r = 0.22, which is still valid.

Step downhill two ways

The dashed line is the best guess, the observed count k. Pick how the guess is parameterised, set k and the step size, then press Run. Watch whether the guess stays valid and where it settles.

3
0.30

One bin, one parameter, plain gradient steps from r = 1 (z = 0). Only the Table 3 and Figure 7 results are the paper’s. This toy shows one simple way rate space can break; it does not prove why the paper’s networks struggle. Log-rate steps can also be slow or jumpy at large step sizes, but they never give an invalid guess.

Choice 2: zeros, not a [MASK] token

Chapter 4 told this story. Swapping zeros for a special [MASK] token drops the score from 0.918 to 0.022, which is close to no decoding skill at all. At test time the token never appears, so the network meets a kind of input it never practised on.

Choice 3: heavy dropout

Neuroscience datasets are small. The monkey set holds 2,296 trials. The subsets used in the data-size test shrink to 92 trials, which the authors put at 0.2 Mb. A model with over a million parameters can memorise a set that small. Dropout fights this by randomly zeroing numbers while training, so no path through the network can carry the whole answer.

The search sweeps dropout between 0.2 and 0.6, which is heavy. Limiting it to 0.0 to 0.2 drops the score to 0.549. A smaller 2-layer NDT still needs it: confined to 0.0 to 0.3, its score fell to about 0.4 (reported as not shown in the paper).

Not every transformer trick helped. The authors tried variable-length mask spans and extra embedding layers and found contributions of only about 1 to 2% R² on this dataset. A lesson here: recipes from huge language corpora do not carry over unchanged to a few thousand trials.

How big is each gap?

Compare each row with the baseline, 0.918. Rates instead of log-rates lose 0.626, which is about six SEMs. The [MASK] token loses 0.896, about 75 SEMs. Narrow dropout loses 0.369, about seven SEMs. Each SEM comes from only three searches, so these ratios are rough. Still, no gap is close to noise.

The same step, worked by hand

Use the device’s defaults: k = 3, step size 0.3, start at r = 1 (z = 0). In rate space the slope is 1 − k / r. Step one: r = 1 − 0.3 × (1 − 3) = 1.60. Step two: slope 1 − 3 / 1.6 = −0.875, so r = 1.86. Step three: r = 2.05.

In log-rate space the slope is ez − k. Step one: z = 0 − 0.3 × (1 − 3) = 0.60, so r = 1.82. Step two: slope 1.82 − 3 = −1.18, so z = 0.95 and r = 2.59. Step three: r = 2.93. After three steps the log-rate guess sits at 2.93 and the rate guess at 2.05. The target is 3.

This toy has one parameter and a fixed step size. It shows the shape of the slopes. It does not show why a deep network trained with an adaptive optimiser struggles in rate space. The paper’s Figure 7 shows that on real runs.

What the table cannot tell you

Each row changes one thing on one dataset, the monkey reaching set. The paper does not test, say, a different bin size or a different monkey. The zero mask result also depends on the rest of the recipe, such as the random replacement share. So read Table 3 as strong evidence for these three choices on this task. It is not a law for all spiking data.

For your own rebuild the lesson is practical. Put an exponential after the last linear layer. Zero the hidden bins, not a token. Search dropout from 0.2 to 0.6 and expect to need the high end on small sets.

In the NDT, what does the last linear layer output before the exponential?

Chapter 6

Does It Recover the Rates?

Grade the NDT against LFADS twice: on made-up data where the true rates are known, and on a real monkey recording where they are not

How do you grade a model that guesses a hidden quantity? If the quantity is truly hidden, you cannot. So the authors use two kinds of exam. In the first, they build the hidden rates themselves, draw spikes from them, and ask the model to find the rates again. In the second, they use a real recording and ask whether the inferred rates still hold the information about what the monkey’s hand did.

The score: R²

Both exams use the coefficient of determination, written R². It asks: how much of the up-and-down in the true values does your estimate explain? A score of 1 is perfect. A score of 0 is no better than always guessing the average. Negative is worse than that.

Work a small case. The true values are 2, 4, 6, 8, with average 5. The total spread is (2−5)² + (4−5)² + (6−5)² + (8−5)² = 9 + 1 + 1 + 9 = 20. Your estimates are 3, 4, 5, 9. The errors are 1, 0, 1, 1, so the squared errors add to 1 + 0 + 1 + 1 = 3. Then R² = 1 − 3/20 = 0.85.

R²
The share of the spread in the truth that the estimate explains.
∑(y − ŷ)²
The summed squared error between the truth y and the estimate ŷ. In the example, 3.
∑(y − ȳ)²
The summed squared gap between the truth and its own average ȳ. In the example, 20.

Exam 1: known rates

The Lorenz dataset starts from a three-number state that follows the Lorenz equations, a classic chaotic system. The state is projected up to 29 channels to make rates, and spikes are drawn from them. The set has 1,560 trials of 50 bins, in 65 conditions with 24 trials each. A condition is one starting state, so all its trials share the same rates.

The chaotic RNN dataset is harder. Its rates come from a random recurrent network with 50 channels, 100 bins and 1,300 trials in 100 conditions. The authors measure difficulty by the number of principal components in the generating system. Both sets keep 80% of trials for training and 20% for validation.

Table 1 reports R² between inferred and true rates, as mean ± SEM over searches. The NDT gets 0.934 ± 0.004 on Lorenz against 0.921 ± 0.005 for LFADS. On the chaotic RNN LFADS leads, 0.869 ± 0.001 against 0.846 ± 0.011. The authors call the gaps small. They also note that NDT rates look less smooth in the plots.

Exam 2: a real monkey

The real data come from the Monkey J Maze set. It has 202 neurons from the primary motor and dorsal premotor cortices, recorded as a monkey made delayed reaches, both straight and curved. There are 2,296 trials in 108 reach conditions. Each trial has a random pause between seeing the target and the go cue, which gives the neurons time to plan.

The authors train on the span from 250 ms before movement onset to 450 ms after, since earlier work found this stretch well described by an autonomous system. Results held for bin sizes from 2 ms to 20 ms (not shown in the paper).

No true rates exist here, so there are two checks. First, the PSTH check: average inferred rates over repeats of the same reach and compare with the average of the raw spikes. Both models give consistent rates across trials, while smoothing (a 30 ms bell-shaped kernel) gives noisier single trials. Second, the decoding check: use a linear map to read the hand velocity from the inferred rates. If the rates have captured the neural state, the map works. The paper fits it with ridge regression at a 90 ms lag between neural activity and movement. The result is 0.918 R² for the NDT and 0.915 for AutoLFADS.

Compare the scores

Choose a test. Warm bars are the NDT and purple bars are LFADS. Thin marks show the paper’s SEM where it gives one. The bars start at 0.7, not 0.

Numbers from Table 1 (synthetic) and Section 3.2 (reaching, no SEM given). The reaching score is for the 6-layer NDT against AutoLFADS. The y axis is cut at 0.7 to make small gaps visible, so bar lengths do not show ratios.

Less data, and the likelihood link

Two more findings sit behind the numbers. First, on the synthetic data, models with better likelihood on held-out spikes also match the true rates better (Figure 4b). This does not hold for LFADS models trained without coordinated dropout, which the authors take as confirming that the masking works as intended. Second, with fewer trials the NDT still beats smoothing down to 92 training trials. The 6-layer NDT falls behind AutoLFADS at 92 trials, while a 2-layer NDT scores 0.866 and outperforms it (reported as not shown). Chapter 7 returns to size.

How the decoding check works, step by step

Take the inferred rates for each trial, 70 bins by 202 channels. At each time step, a ridge regression (least squares with a small penalty, here α = 0.01) maps the 202 rates to the 2D hand velocity. The map is linear, so it cannot hide a poor input.

The hand moves after the neurons fire. So the paper pairs neural activity with the velocity 90 ms later. Rates are inferred over −250 to 450 ms around movement onset. Velocity is predicted only over −160 to 450 ms, because the first 90 ms of rates have no later velocity to pair with.

How big are the gaps?

On Lorenz, the NDT leads by 0.934 − 0.921 = 0.013, about two combined SEMs. On the chaotic RNN, LFADS leads by 0.869 − 0.846 = 0.023, also about two. The authors call both gaps small. On decoding, the gap is 0.003 and the paper gives no SEM. Read these as ties with a slight tilt, not as wins.

An estimate has summed squared error 5 against a total spread of 20 around the average. What is its R²?

Chapter 7

Speed Against Size

Read Table 2, set a deadline for your own decoder, and find which model is the best one that still meets it

You are building the software for a cursor. A fresh bin lands every 10 ms. Your plan says the decoder gets a few milliseconds to turn the latest window into rates. A model that scores well but takes 26 ms misses the deadline. A model that finishes in 1 ms but scores poorly wastes the signal. You need the best score that fits.

How the paper measured speed

The authors time one forward pass: spikes in, rates out. They average over 1,300 trials on a CPU machine with 32 GB of memory and a 4-core i7-4790K processor at 4.2 GHz. LFADS uses one posterior sample. No graphics card is involved. Window length is 70 bins.

In a live system the window rolls forward by one bin. Both models process each new window from scratch. The paper’s example: if the recording module reports every x = 10 ms, an NDT-based interface reflects the latest activity after 10 + 4 = 14 ms. An LFADS-based one needs 10 + 26 = 36 ms. A smaller x widens the benefit. For comparison, earlier work with high-performing online decoding used 20 bins of 15 ms, and even with that shorter window the NDT is about 4 times faster.

Table 2

ModelTrain timeInference (70 bins)ParametersReaching R²
AutoLFADS45 min26 ms280K0.915
NDT-69.4 hr3.9 ms1.36M0.918
NDT-245 min2.2 ms480K0.921
NDT-120 min0.98 ms270K0.886

Read the table in three steps. First, the 6-layer NDT is much faster at inference, 3.9 ms against 26 ms, but it trains for 9.4 hours against 45 minutes and has almost five times the parameters. Second, shrinking helps a lot. A 2-layer NDT trains in 45 minutes, infers in 2.2 ms and scores 0.921, the highest of all four. Third, one layer is too small: it scores 0.886, about 0.03 lower, but infers in under 1 ms and trains in 20 minutes.

The authors stress that training time across a search varies widely. The 6-layer models took between 3 and 18 hours. The fair comparison with AutoLFADS, which has 280K parameters, is the 2-layer or 1-layer NDT.

Why parallel models train faster too

Training an RNN uses backpropagation through time. The error signal must walk backwards through the unrolled chain, one step after another, to reach the early steps. This is the same chain that slows inference. A transformer layer sends the error back through all bins at once. Fewer steps wait in line, so each training update is cheaper.

The size result also matters for data. On just 92 trials, a 2-layer NDT reaches 0.866, above AutoLFADS. The authors suggest that the larger 6-layer model is oversized for small sets, and that Transformers may suit neuroscience data if they are scaled to the data.

Why not just carry the state?

A Kalman filter avoids re-reading the window. It keeps a state and folds in one new measurement per step. Could LFADS do the same? The authors say that nobody has shown it. Its two-direction encoder gives the decoder its starting state from the whole window, and in unpublished tests a forward-only encoder lost performance. For a multi-layer NDT too, most work must be redone when the window shifts. So the paper compares both models on whole windows and expects the trend to hold online.

Set a deadline

Now play the integrator. The slider is your compute budget for one window. Models that fit turn green. Those that miss it turn red. The read-out names the best one that fits.

Meet the deadline

Each dot is a row of Table 2. Left is faster, up is more accurate. Drag the budget line. Dots left of it fit.

5 ms

All four points are Table 2 (one sweep each, a 70-bin window, one CPU). The budget is yours and covers compute only; waiting for the bin to fill comes on top. The horizontal axis is logarithmic. One sweep per variant means the small gap between NDT-6 and NDT-2 is not strong evidence that fewer layers is better.

Table 2 as ratios

Divide each row by AutoLFADS. Inference is 26 / 3.9 = 6.7 times faster for NDT-6, 26 / 2.2 = 11.8 times for NDT-2, and 26 / 0.98 = 26.5 times for NDT-1. Parameters are 1.36M / 280K = 4.9 times more for NDT-6, 1.7 times more for NDT-2, and about the same for NDT-1.

Training time runs the other way for NDT-6: 9.4 hours is 12.5 times the 45 minutes of AutoLFADS. NDT-2 matches that 45 minutes. NDT-1 takes 20 minutes. The paper’s text rounds the 1-layer result to 0.89 and says “under 30 minutes”. The table says 0.886 and 20 minutes. We use the table.

The rolling-window budget

A 70-bin window holds 700 ms of data, yet a new one starts every bin. At 10 ms bins that is 100 windows per second. One core running NDT-6 can finish 1000 / 3.9 = 256 windows per second. NDT-1 can finish 1,020. So all three NDT sizes keep up on this CPU. AutoLFADS, at 38 per second, would need extra cores. This is our arithmetic on Table 2, which timed one window at a time.

A live decoder gets bins every 10 ms and must finish computing within 5 ms. Using Table 2, which statement is right?

Chapter 8

Where It Breaks

Find the one kind of data the NDT cannot model well, and separate what the authors showed from what they only propose

Roll a ball down a landscape of hills and valleys. Once you release it, its path depends only on where it started and on the shape of the land. Nobody touches it. Now let a stranger flick the ball halfway down. The path bends in a way that no one could predict from the starting point.

These two cases have names. A system whose future depends only on its own state is autonomous. A system that also receives outside pushes is non-autonomous. The whole paper is about the first kind, and its title claim is limited to it.

Why motor cortex is treated as autonomous here

In the monkey task, the neurons spend a stretch of time planning the reach. That plan sets the starting state. Then, during the reach, activity unrolls by the circuit’s own rule. Earlier analyses found that this stretch is well described by an autonomous system. So the authors train only on the span from 250 ms before movement onset to 450 ms after, and the tests in Chapter 6 live inside it.

Real life is not always like that. Brain areas the electrodes do not monitor can send signals into the recorded area. A cue first processed in the visual or touch areas will arrive and perturb the motor dynamics. To the recorded neurons this input looks like a flick on the ball.

The measured weakness

LFADS has a part built for this case: a controller, a small network that infers the unseen inputs from the data. The NDT has no such part. The authors ran a preliminary test on a synthetic non-autonomous set, the chaotic RNN with inputs. There LFADS outperformed the NDT by over 20% in R². The paper calls this the most critical limitation of the NDT.

The authors say that fixing it would likely need architecture changes that model both the dynamics and the inputs explicitly. They do not offer a tested fix. So this model, in this paper, is a tool for autonomous stretches of activity.

Feel the failure

Here is a toy of the problem. A hidden two-number state spins and slowly shrinks, and you watch one number. The first half of the window is untouched. A model that assumes the system is autonomous infers the starting state from that clean part and rolls it forward. Then you kick the real system. Watch the model carry on as if nothing happened.

Kick the system

The warm line is the true signal. The dashed purple line is an autonomous model that never sees the kick. Drag the kick size and time. Then switch on a model that can infer inputs.

1.2
bin 30

Toy: a 2-number state turned by 0.22 radians per bin and shrunk by 1.5% per bin, 60 bins, kicked once. The “infers inputs” line is an idealised model that recovers the kick exactly, to show what a controller is for; it is not LFADS’s measured error. The paper’s own figure is the over 20% R² gap.

Before the kick, the autonomous model is perfect. After it, the model cannot know why the world changed. The bigger the kick, and the earlier it comes, the longer the gap lasts.

Other limits the paper names

What the authors propose, and what came next

The discussion section looks ahead. It is a proposal, not a result. The authors expect that larger datasets, perhaps collected without measured behaviour, could be used to pretrain a model on neural activity in a self-supervised way, much as text models are pretrained on the web. They also suggest one model that sees several brain areas or several recording types, and one that finds both fast and slow features of activity.

The same first author later tested pretraining on motor-cortex spiking across many sessions, subjects and tasks in Neural Data Transformer 2 (Ye, Collinger, Wehbe and Gaunt, NeurIPS 2023). Its conclusions are the authors’ own, and we have not re-taught it here. The lecture LFADS shows where recurrent latent-dynamics models fit.

What the controller does, and why the paper switched it off

A controller is a small extra network inside LFADS. It looks at the data and proposes an outside push at each time step, which the main dynamics then absorb. Think of a coach shouting corrections to the rolling ball. The LFADS lecture builds it.

The paper states that the LFADS controller is kept off in all main results, because those results study autonomous data. So the 0.915 and the 26 ms are for LFADS without its input-inferring part. The only test with the controller on is the preliminary synthetic one above.

Sort the claims by how well they are supported

Reading the limit fairly

Our reading, not the paper’s: a task with sudden cues, such as a target that flashes mid-reach, would send fresh input into motor cortex. A model that assumes self-driven dynamics could then smooth over the change. Check for that before trusting rates on such data.

What is the most critical limitation the paper names for the NDT?

Chapter 9

Field Guide

Keep the numbers, check what you can rebuild, read the core code, and see where this paper sits in the field

The Neural Data Transformer is a BERT-style encoder for spike counts. It hides short runs of bins, zeroes them, and learns to predict their firing rates under a Poisson loss. It outputs log-rates, and it uses heavy dropout. On autonomous reaching data it matches LFADS in accuracy and runs 6.7 times faster. It does not model unseen inputs.

The numbers

QuantityValueWhere
Inference, NDT-6 against AutoLFADS (70 bins)3.9 ms against 26 ms (6.7×)Table 2, Fig. 6
Monkey J Maze data202 neurons, 2,296 trials, 108 conditionsSec. 3.2
Input window−250 to 450 ms, 10 ms bins, 70 binsSec. 3.2
Velocity decoding R², NDT-6 against AutoLFADS0.918 against 0.915Sec. 3.2
Lorenz R², NDT against LFADS0.934 against 0.921Table 1
Chaotic RNN R², NDT against LFADS0.846 against 0.869Table 1
Ablation: rates, [MASK] token, narrow dropout0.292, 0.022, 0.549Table 3
Parameters, NDT-6, NDT-2, NDT-1, AutoLFADS1.36M, 480K, 270K, 280KTable 2
Search per experiment20 models × 3 searches = 60Sec. 3
Dropout sweep, mask span sweep0.2 to 0.6, 1 to 7 binsSec. 3, 7.4
Smallest training set tested92 trials (about 0.2 Mb)Fig. 5d, Sec. 4
Non-autonomous gapLFADS over 20% R² higherSec. 4

The terms

TermMeaning in this paper
BinA short slice of time (10 ms) in which spikes are counted.
Firing rateThe hidden average spike speed behind the counts. The model outputs it.
Poisson NLLThe loss r − k log r, averaged over cells.
Masked modelingHide bins, predict them from the rest, score only the hidden ones.
Zero maskHidden bins have their counts set to 0 instead of a [MASK] token.
AutonomousActivity set by its own state alone, with no outside inputs.
R²One minus the summed squared error over the total spread.

Can you rebuild it?

The core, as code

This is a compact PyTorch sketch of the ideas in Chapters 3 to 5. It is ours, not the authors’ code. The MLP width, the placement of the norms and the random-replacement masking are details to check in their repository.

pythonimport torch, torch.nn as nn

class NDT(nn.Module):
    def __init__(self, C=202, T=70, n=1, layers=6, heads=2, drop=0.4, span=30):
        super().__init__()
        d = C * n
        self.n = n
        self.embed = nn.Linear(1, n)                       # each count -> n numbers
        self.pos = nn.Parameter(torch.zeros(T, d))          # learned position embedding
        layer = nn.TransformerEncoderLayer(
            d_model=d, nhead=heads, dim_feedforward=2 * d,   # width is our guess
            dropout=drop, batch_first=True)               # attention, norm, MLP(ReLU), norm
        self.body = nn.TransformerEncoder(layer, layers)
        self.drop = nn.Dropout(drop)
        self.head = nn.Linear(d, C)                         # d -> C log-rates
        idx = torch.arange(T)
        self.register_buffer("far", (idx[None] - idx[:, None]).abs() > span)  # context span

    def forward(self, spikes):                           # (B, T, C) counts
        B, T, C = spikes.shape
        x = self.embed(spikes[..., None]).reshape(B, T, C * self.n)   # (B, T, d)
        x = self.drop(x + self.pos[:T])
        x = self.drop(self.body(x, mask=self.far[:T, :T]))        # (B, T, d)
        return self.head(x)                                  # (B, T, C) LOG-rates

def make_mask(B, T, ratio=0.25, span=2):
    starts = torch.rand(B, T) < ratio / span
    mask = torch.zeros(B, T, dtype=torch.bool)
    for s in range(span):
        mask[:, s:] |= starts[:, :T - s]                    # each start hides `span` bins
    return mask

def train_step(model, opt, spikes):                   # spikes: (B, T, C) float
    mask = make_mask(*spikes.shape[:2])
    x = spikes.masked_fill(mask[..., None], 0.0)           # zero mask, no [MASK] token
    logr = model(x)
    nll = torch.exp(logr) - spikes * logr                   # r - k log r  (log k! dropped)
    loss = nll[mask].mean()                                 # hidden bins only
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

# inference: no mask, the whole window in, rates out
# model.eval(); rates = model(window).exp()               # (1, 70, 202)

Where it sits in the field

Before the NDT, the strong tool for single-trial rates was the sequential autoencoder LFADS, tuned at scale by AutoLFADS. The NDT shows that a plain BERT-style encoder can match it on autonomous data and run faster. It sits between hand-set smoothing and heavy recurrent latent-dynamics models, and it opens the door to training one large model on many recordings, which the authors propose as future work.

Keep going

The takeaway. If a signal is a smooth hidden rate seen through Poisson noise, hide bins and let a transformer fill them in. It can match a recurrent model in accuracy and finish in a few milliseconds. The cost: it assumes the neurons follow their own dynamics, so it needs more work when unseen inputs push the system.

Credit: the Neural Data Transformer is the work of Joel Ye and Chethan Pandarinath (Georgia Institute of Technology and Emory University). Read the original at arXiv 2108.01210 and see the authors’ code. Now press Present or Teach and explain, from memory, why hiding bins teaches a rate model and why the NDT is quick. Then go back to the race at the top, pick NDT-1, and say what you gave up for 0.98 ms.

Neural Data Transformers (Ye & Pandarinath 2021)
Read the paper · Authors’ code · The LFADS lecture · Back to Gleams