POYO

POYO turns every spike into a token, so one model learns from many monkeys and sessions, then reads a brand-new session by learning only one small vector per new neuron.

A neural population decoder (a model that reads the electrical chatter of many brain cells and tells you what the body is doing) that no longer needs the same cells on every day.

Picture a monkey moving a cursor while a computer reads its brain and guesses the hand velocity (blue is the truth, warm is the guess). Tap each decoder below and watch how closely the warm line follows the blue one. R² is a score from 0 to 1, and 1 is a perfect match. “Pretrained” means trained earlier on many sessions, and “frozen” means its weights stay fixed. Then we build every part of POYO, piece by piece.

You need a rough idea of what a neuron is (a cell that sends short electrical pulses) and what a transformer does (see the transformer Gleam). We build the rest from zero.

Based on Azabou et al., “A Unified, Scalable Framework for Neural Population Decoding”, NeurIPS 2023 (Georgia Tech, Mila, Université de Montréal, McGill). Lecture: Drift and recalibration.

Decode a session it never saw

Scores are the paper’s Table 2 (R², mean over the held-out sessions; the other half of the table, POYO-1, is not drawn). The two lines are a toy: a made-up velocity curve plus made-up wobble, scaled so its R² equals the table value exactly. The paper’s own traces are in its Figure 2.

Chapter 0

A New Day, a New Cast of Speakers

See why a brain decoder trained on Monday fails on Tuesday, and what POYO changes about that

A monkey sits in a chair and moves a cursor across a screen. A small array of electrodes sits in its motor cortex, the part of the brain that drives arm movements. Each electrode listens to nearby cells. When a cell fires, it sends a short electrical pulse called a spike.

A computer reads those spikes and guesses where the hand is moving. That guess is neural decoding. It is the core of a brain-computer interface (BCI), a system that turns brain activity into a command. If the guess is good, a paralysed person could one day steer a cursor or a robot arm with thought alone. You can read how the spikes are made in The Spike.

Now the hard part. On Monday, an engineer sorts the recorded pulses into units, one unit per cell (or per small group of cells). Unit 1 is a cell that fires when the hand moves right. Unit 2 fires for up. The engineer trains a decoder that expects unit 1 in slot 1, unit 2 in slot 2, and so on.

On Tuesday the wires have shifted a little. The sorting finds a different set of cells. Slot 1 now holds a cell that fires for down. The decoder still trusts slot 1 to mean right. So it is wrong, and nobody told it. The lecture on drift and recalibration shows how much effort labs spend on this one problem.

Feel the problem

Here is the problem in a toy you can run. Eight cells sit on a ring. Each cell has a favourite direction, and it fires most when the hand moves that way. The old decoder learned on Day 1, where slot k held the cell with favourite direction k times 45 degrees. Pick Day 2 or Day 3 and the cells are shuffled between slots.

Run yesterday’s decoder on a new day

Each dot is one slot; its size is how hard that slot’s cell fires. The blue arrow is the true hand direction. The warm arrow is what the decoder guesses. Pick a day, drag the direction, then switch on Name the cells.

40°

Toy model, not the paper’s data: eight cells, each firing at 0.5 × (1 + cos of the angle between the hand and its favourite direction). The decoder adds up each slot’s rate times the favourite direction it learned for that slot on Day 1. The two shuffles are made up.

On Day 1 the guess is perfect. On Day 2 or Day 3 it is far off, and the error depends on the shuffle, not on any noise. Nothing is wrong with the decoder as such. Its inputs changed meaning.

Now switch on Name the cells. The decoder is told which cell sits in which slot, so it uses each cell’s own favourite direction. The error drops to zero. That is the whole idea of this paper: give every cell a name the model can look up, so the model never relies on slot numbers.

Why this blocks big models

Language models got good by training on huge piles of text. Every document shares one vocabulary, so the word “cat” means the same thing in every file. Brain data has no such shared vocabulary. Each recording has its own cast of cells, and nobody knows how Monday’s cells match Tuesday’s, let alone another monkey’s.

The authors put it as a conversation. Each recording listens to a new set of speakers. We do not know who they are or what they respond to. But they may well talk about the same subjects: movement, touch, body state. The lexicon changes, the topics do not.

Older work handled this with a second step. Train on one day, then align a later day to it. Alignment needs extra machinery for every new day. It also assumes the same animal, the same brain area, and the same task. POYO asks a bigger question: can one model train on many animals, labs and tasks at once, with no alignment at all?

What POYO does, in one paragraph

The name stands for Pre-training On manY neurOns. It makes three moves. First, it turns every single spike into a token (one small unit of input, like a word for a language model), tagged with which unit fired and when. Second, it squeezes thousands of those tokens into a few hundred summary tokens, using a design called Perceiver. Third, it asks questions of that summary, such as “what is the hand velocity at time t?”. To use a new session, it freezes almost everything and learns only one small vector for each new unit.

The data are real: over 100 hours of motor cortex recordings from several monkeys, with numbers that Chapter 8 and the Field Guide set out. The paper reports them in two slightly different ways, and we will flag that when we reach it.

The road

  1. Spikes as events: why cutting time into bins wastes space.
  2. One token per spike: a unit embedding plus a timestamp, and the start and end markers.
  3. The Perceiver bottleneck: thousands of spike tokens into a few hundred latents.
  4. Rotary time encoding: attention that sees only how far apart two events are.
  5. Asking questions of the latents: output queries and session embeddings.
  6. Training and inference: shapes, loss, batches and the sliding window.
  7. Unit identification: adapting to a new session by learning one vector per unit.
  8. What the tables show: transfer, scaling and where it falls short.
  9. Field Guide: numbers, code, POYO+, and where to go next.
Why does a decoder trained on Monday often fail on Tuesday with no change in the animal’s behaviour?

Chapter 1

Spikes Are Events, Not Columns

Count what binning costs, and see why the paper keeps each spike as its own event

Imagine you record one second from 40 cells. Cell 7 fires at 0.113 s and again at 0.642 s. Cell 12 fires five times. Most cells fire a handful of times, and most of the second is silence. What do you hand to a neural network?

Start with the words. A spike train is the list of times at which one unit fired. A raster is a picture of many spike trains stacked up, one row per unit and one tick per spike. A firing rate is how many spikes a unit makes per second, written in hertz (Hz). A unit at 10 Hz fires about ten times a second. The chapter on spikes to rates goes deeper on rates.

The standard answer: bins

Most neural models do not take spike times. They cut time into equal slices called bins, such as 10 ms each, and count the spikes of each unit inside each bin. The result is a grid of whole numbers. It has one row per unit and one column per bin. A grid is easy to feed to a network.

Bins have two costs. The first is a choice you must make. A wide bin (50 ms) blurs the exact timing of spikes. A narrow bin (1 ms) keeps timing but makes the grid huge. The second cost is waste. At a typical firing rate most bins hold zero spikes, so most of the grid is zeros that the model still has to read.

A small count shows it. Take 3 units, a one second window and 10 ms bins. The grid has 3 x 100 = 300 numbers. If the three units fire 6, 3 and 5 times, only 14 spikes exist. At most 14 of the 300 cells are non-zero, so at least 286 are empty.

Count bins against spikes

The raster shows one second of toy spiking, one row per unit. The two bars compare tokens: one per unit per bin, against one per spike. Change the bin width and the number of units.

40

Toy data, seeded so every visit looks the same: each unit gets a made-up rate between 2 and 20 Hz (mean near 8 Hz), and spikes fall at random times. The paper’s real windows hold between 1,000 and 20,000 spike tokens (its Appendix A.4.4).

Drag the bin width down to 1 ms. The bin tokens explode, and almost all of them are empty. The spike tokens do not move, because the number of spikes does not depend on how you slice time. That is the first reason POYO leaves bins behind.

Real numbers make the point. A unit that fires at 8 Hz makes about 8 spikes in a second. A population of 200 such units makes about 1,600 spikes. In 1 ms bins the grid has 200 x 1,000 = 200,000 cells, so at least 198,400 are empty. Summing across units would shrink the grid, but it would also erase which unit fired. The model needs both the identity and the time.

The second problem: a fixed set of columns

The grid has a deeper flaw for our goal. Its rows are tied to a specific list of units. A model that reads a 40-row grid expects exactly those 40 units, in that order. That is the slot problem from Chapter 0 again. A second animal with 90 units cannot fit.

The related work in the paper shows this pattern in every earlier transformer for neural data. The neural data transformer (NDT) puts the firing rates of all units into one token per bin. The embedded interaction transformer (EIT) makes tokens over both units and time. The spatiotemporal NDT (STNDT) uses both layouts together. All three bin the data. All three train on one session with a fixed set of units. You can see NDT in its own paper Gleam, NDT.

The failure, as symptom and measurement

The symptom: when you shrink the bin to keep spike timing, the model slows down and runs out of memory. The measurement: tokens per window grow in proportion to units divided by bin width. With 200 units and 1 ms bins, one second gives 200,000 tokens, and about 199,000 of them hold nothing.

The fix the paper chooses: stop storing silence. Keep only what happened. A spike is an event with two facts, which unit fired and when. A window with 1,000 spikes becomes 1,000 tokens, whatever the bin width would have been. The next chapter builds that token.

This also explains a design choice you will meet in Chapter 6. The paper cuts windows by time, one second each, not by token count. A one second window from 300 fast units can hold ten times more spikes than one from 30 slow units. The token count is whatever the brain produced.

No resolution to choose. Because the model reads exact spike times, there is no bin width to tune and no information thrown away by rounding. The cost is that the number of tokens now varies from window to window. Chapter 6 shows how the paper copes with that.
You halve the bin width from 10 ms to 5 ms. What happens to the number of spike tokens in a one second window, and to the number of bin tokens?

Chapter 2

One Token Per Spike

Build the spike token from a unit name and a time, and add markers so silent units still speak

A language model reads a sentence as a list of word tokens. Each token is a number that points to a row in a table. The row holds a list of learned numbers, the word’s embedding. Two words that behave alike end up with similar rows. The table is called a lookup table because you look a word up by its index.

POYO does the same for brain cells. Give each unit an index. Keep a table with one row per unit, UnitEmbed. Each row has D numbers, and the paper uses D = 128. The row is the unit’s name tag, and the model learns it during training. The authors say the embedding space encodes something about the “speaker’s” identity and role.

The spike token

A spike has two facts: which unit fired, and when. The paper writes a spike token as the pair (xi, ti). The first part is the embedding of the unit that fired:

xi
The vector for spike number i. It has D = 128 numbers.
UnitEmbed
The lookup table. One learned row for every unit the model has ever met.
unit of spike i
The index of the unit that fired. Every spike from one unit gets the same vector.

The second part, ti, is the time of the spike. Notice what the model does not do: it does not add the time into the vector. Time travels beside the vector and enters attention through a rotation, which Chapter 4 derives. So two spikes from the same unit have identical x and differ only in t.

A window with M spikes becomes a list of M vectors and M timestamps. In shapes: X: M × 128 and t: M. M is not fixed. It depends on how many units you record and how fast they fire. The model never sees “how many units are there” as a setting. It sees only the tokens that exist.

Turn spikes into tokens

Five units fire in one second. Unit E stays silent. Drag the time slider to reveal tokens in order, and read the newest one below. Then switch the markers on.

0.360 s

Toy spike times, chosen by hand: A fires 8 times, B 5, C 6, D 4, E never. The embedding itself is not drawn, because a real one is 128 learned numbers with no readable meaning.

Does the order of tokens in the list matter? No. Attention treats its input as a set, and each token carries its own timestamp. You can shuffle the list and get the same answer. This is why the paper can concatenate the spikes of any number of units into one list, with no padding rows for missing cells. New units simply add new rows to the lookup table, and the table grows as the model meets more animals.

The silent unit problem

Look at unit E with markers off. It is not in the token list at all. The model cannot tell whether E was silent, or was never recorded. That matters. A unit that is inhibited (pushed to stay quiet) tells the brain something, just as a loud one does. A tokenizer that records only activity throws away that information.

There is a second problem. With rotary encoding the model sees only relative times (Chapter 4). A window has no obvious start or end, since nothing marks time zero.

The paper fixes both with delimiters (Appendix A.3). The model learns two special vectors, x_start and x_end. For every unit in the population it adds two tokens. Each is the sum of the delimiter vector and that unit’s own embedding. The start token gets time t (the window start). The end token gets time t + T (the window end).

xstart + UnitEmbed(u)
“I am tuning in to unit u from now.” One per unit, stamped at the window start t.
xend + UnitEmbed(u)
“Unit u’s window ends here.” Stamped at t + T, with T = 1 s in the paper’s runs.

Count with the toy. Without markers the window holds 23 tokens. With markers each of the five units adds two, so 23 + 10 = 33 tokens. Unit E now appears twice, once at 0 s and once at 1 s, with nothing between. That gap says: E was watched, and E said nothing.

A realization check: what the table holds

The table has one row per unit across all training sessions. POYO-1 trains on 27,373 units (the paper’s abstract figure). At 128 numbers each, the table holds about 3.5 million numbers. That is simple arithmetic, not a figure from the paper. The row for a unit comes from nowhere but training: it moves by gradient descent until it helps predict behaviour. Two units from different animals can end up with similar rows if they play similar roles.

Two spikes from different units at the same instant are two tokens with the same time and different vectors. Two spikes from the same unit at different instants are two tokens with the same vector and different times. Name and time stay separate, and attention can use either one alone.

Why this solves Chapter 0. The model reads a unit’s name tag, not its slot. A new day with new cells just means new rows in the table. Nothing else about the input format changes, so the model can ingest populations of any size.
Why does POYO add a start and an end token for every unit, even a unit that never fires in the window?

Chapter 3

Squeeze Thousands of Spikes Into a Few Hundred Latents

Compress the spike tokens with cross-attention, then do the heavy thinking on the small summary

You now hold up to 20,000 spike tokens for one second. A transformer would let every token look at every other token. That is the right idea, and it is too costly here. Let us see exactly why, starting with a short primer on attention. If you know it, skip to the next heading. A fuller tour is in the transformer Gleam.

Attention in three sentences

Each token makes three vectors by multiplying its embedding with learned matrices: a query (what I am looking for), a key (what I offer) and a value (what I hand over if you pick me). A token scores every key against its query with a dot product. A softmax turns the scores into weights that add to 1, and the token takes the weighted mix of the values.

In symbols, with dk the size of a key vector:

Z1
The new vectors, one per query. This is the output of the attention step.
Q KT
A table of scores. Row i, column j is the dot product of query i with key j.
√dk
A fixed shrink factor. Without it, long vectors give huge scores and the softmax goes spiky.
softmax
Turns each row of scores into weights between 0 and 1 that sum to 1.
V
The values. Each output row is a weighted average of these rows.

A tiny run. Say the scaled scores for one query against three keys are 1.0, 0.0 and 2.0. The exponentials are 2.718, 1 and 7.389, which sum to 11.107. The weights are 0.245, 0.090 and 0.665. With values [1, 0], [0, 1] and [2, 2], the output is 0.245 × [1, 0] + 0.090 × [0, 1] + 0.665 × [2, 2] = [1.575, 1.420]. A second query with all scores 0 gives equal weights of one third and the output [1, 1].

The cost problem

When all M tokens attend to all M tokens, the score table has M x M entries. For M = 10,000 that is 100 million entries in one layer, for one head. Stack six layers and you reach 600 million. The cost grows with the square of M. The paper cites input lengths between 1,000 and 20,000 tokens, so squares are the enemy.

The Perceiver trick

The paper borrows the Perceiver design. Make a short list of N latent tokens. These are learned vectors that do not come from any spike. Think of N reporters sent to a crowd of M witnesses. Each reporter asks questions of the whole crowd and writes one summary.

In attention terms, the queries come from the latents, Q = WqZ0, and the keys and values come from the spike tokens, K = WkX and V = WvX. That is cross-attention: queries from one list, keys and values from another. The score table is now N x M, not M x M. This is equation (1) in the paper, with Z0 the learned latents and X the spike tokens.

The output Z1 has N rows, one per latent, each of size D = 128. Whatever the spike count M was, the answer has shape N × 128. The paper wraps this in a standard transformer block with pre-normalization and a feed-forward network.

Think on the small list

Now every later layer works only on the N latents. These are ordinary self-attention layers (equation 2): queries, keys and values all come from Zl. Each costs N x N. After L layers you get ZL, still N × 128.

Count the attention scores

Bars use a log scale: each tick is ten times more. Slide the number of spike tokens M and latents N, and pick a depth L.

10,000
512

Counts of score-table entries for one head, not seconds or FLOPs. Ordinary self-attention scores L × M² entries. The Perceiver design scores M × N once (the input cross-attention) plus L × N² (the latent layers); the output queries of Chapter 5 add a small extra term not drawn. The depths 6, 14 and 26 are the paper’s scaling study (Section 3.3). The paper’s text gives N = 512 (Section 3.1) but also describes 256 latents (Appendix A.3) and 128 for single-session models (Table 4).

At M = 10,000, N = 512 and L = 6, the Perceiver design needs about 6.7 million scores against 600 million. That is about 90 times fewer. Raise M to 20,000 and the gap grows to about 200 times, because the square term grows faster than the linear one.

The failure mode, as symptom, measurement and fix

The symptom: a recording with 300 fast-firing units makes a window of 20,000 tokens, and plain self-attention will not fit on a GPU. The measurement: 20,000 × 20,000 = 400 million scores per layer per head. The fix: pay M x N once, to summarize, then pay N x N for the layers. The cost of reading the input is linear in M, and the cost of thinking is fixed.

There is a price. The latents are a bottleneck. If N is too small, detail is lost. The paper does not run a table that sweeps N, so we cannot quote the loss curve. It reports only that single-session models were stable over a wide range of hyperparameters (its Appendix C.3, Figure 8).

Reporters, not witnesses. The latents never see the raw crowd again after the first layer. Everything the later layers know about the spikes passed through one round of questions. That makes the first cross-attention the most important layer in the model.
The number of spike tokens M doubles. How does the cost of the input cross-attention (latents reading spikes) change, with N fixed?

Chapter 4

Rotary Time: Attention That Feels Only the Gap

Rotate queries and keys by their timestamps, and watch the score depend on the gap alone

A spike at 0.20 s and a spike at 0.25 s from the same unit are close in time. A spike at 0.20 s and one at 0.90 s are far apart. The model must feel that difference, yet the tokens carry no time in their vectors. Attention, as you built it in Chapter 3, treats the tokens as an unordered bag. So where does time go?

Language models face the same issue with word order. A common cure is to add a position encoding to each embedding. POYO instead uses rotary position encoding (RoPE), which acts inside the attention score and gives relative time for free.

Why only the gap matters

POYO trains on any one second slice of a recording, with no trial boundaries (Appendix A.2). A window may begin at 3.2 s or at 91.7 s. Absolute time then means nothing. What matters is the gap between two events: this spike came 30 ms after that one. We want an attention score that depends on the gap only.

Clock hands

Picture a clock hand. At time t it points at angle 2πt/T, where T is the time for one full turn. A second hand has a short T. An hour hand has a long one. Give each token a hand, and rotate its query (or key) vector by the angle for its timestamp. Two tokens then have hands at different angles, and the difference of the two angles depends only on the gap between their times.

The paper writes the rotation as a 2 x 2 matrix, for a time t and a period T:

R(t, T)
A rotation of a 2-D arrow. Given an arrow, it turns it by the angle θ.
[ cosθ −sinθ ; sinθ cosθ ]
The standard matrix for turning a plane by θ. The semicolon separates the two rows.
θ = 2πt / T
The angle for time t. After one period T the arrow has turned a full circle.

A head vector has 64 numbers, so it splits into 32 pairs. Each pair is a plane with its own hand. The paper spaces the periods geometrically between Tmin = 1 ms and Tmax = 4 s: Ti = Tmin (Tmax / Tmin)2i/D. Fast hands resolve small gaps. Slow hands resolve large ones, just as a clock uses seconds, minutes and hours together.

Now the key fact. Query q at time ti is rotated by R(ti); key k at time tj by R(tj). The score is their dot product:

(R(ti) q)T (R(tj) k)
The attention score after both vectors are rotated by their own times.
qT R(tj − ti) k
The same score, written with one rotation by the gap. Only tj − ti appears. The two times are gone.

The step in the middle is R(ti)T R(tj) = R(tj − ti): turning back by one angle and then forward by another leaves only the difference. Try it in the device.

Shift both tokens, keep the score

Each dial is one plane with its own period. The teal arrow is the query at time i, the warm arrow the key at time j. The number under the dial is the score, the cosine of the angle between them. Drag the shift and watch every score stay put.

100 ms
350 ms
0 ms

Five of the 32 planes, at exponents 0, 0.25, 0.5, 0.75 and 1 of the paper’s period formula (about 1 ms, 8 ms, 63 ms, 0.50 s and 4 s). Here q and k are the same unit arrow in every plane, so the score is the cosine of the angle difference. Real q and k are learned, so real scores also depend on their content.

A worked case. Take the plane with period 0.5 s. A query at 100 ms and a key at 350 ms differ by 250 ms, which is half a period. The angle between the arrows is π, and cos π = −1. Shift both by 300 ms, to 400 ms and 650 ms. The arrows turn together, the gap is still 250 ms, and the score is still −1. In the 4 s plane, the same gap gives an angle of 0.39 radians, and a score of 0.92.

Time also enters the values

In ordinary RoPE the rotation touches only queries and keys, so time shapes only which tokens get weight. The authors add one extension (Appendix A.1). They also rotate the values, so the output carries the relative time of what it read. The slow way would compute R(ti − tj) for every pair. The fast way factors it: rotate each value by R(ti) first, run attention, then rotate each output by R(−tj).

The paper states that the model performed better with value rotation in early experiments, and that rotating only half the head dimensions also helped. It does not show an ablation table for either choice, so treat both as reported design choices, not measured results.

Timestamps for the latents

Latents need times too, or they could not use RoPE. The paper spreads latent timestamps evenly across the window [0, T]. In Appendix A.3, 256 latents come as 32 shared learned vectors, each repeated at 8 evenly spaced timestamps. The vectors in each repeat start equal; they differ only in time, then evolve separately after the first cross-attention. The idea is that every time slot should query the spikes in the same way, just centred on a different moment.

Symptom, measurement, fix. Symptom: a model with absolute time works on one window and fails on the same activity shifted by a second. Measurement: its scores change when both times shift. Fix: rotary encoding, where the score depends only on tj − ti, as the device shows.
Query at 100 ms and key at 350 ms. You move both by +1 s. What happens to the attention score in every plane?

Chapter 5

Ask the Latents a Question

Read out behaviour at any times you choose, using output queries tagged with a time and a session

The latents now hold a summary of one second of brain activity. You want hand velocity. One lab measures the hand 100 times a second. Another measures it 1,000 times a second. A decoder with a fixed output layer of 100 numbers fits the first lab and breaks on the second. How do you build one readout for both?

The paper’s answer is to treat the readout as a question. You write down the times you care about. For each time you create an output token, a small vector that says “tell me the behaviour at this moment”. The model answers each one separately.

The output token

Each output token has two parts, just like a spike token: a vector y0,i and a timestamp tiout. At the start, every output token in a session has the same vector, one learned vector y0 of size D = 128. Only the timestamps differ. The number of output tokens P is up to you. It is the number of time points you want to predict.

Now the readout is one more cross-attention, this time with the queries from the output tokens and the keys and values from the final latents ZL:

Y
The answers. One vector of size 128 for each of the P output tokens.
Q KT
A table of P rows and N columns. Row i says how much output token i trusts each latent.
Q = Wq Y0
Queries come from the output tokens. Keys and values come from the latents ZL.
V
The latents’ values. Each answer mixes them by those weights.

Because RoPE (Chapter 4) rotates by timestamps, a query at 0.35 s scores high against latents whose timestamps and contents help at that moment. Nothing about sampling rate is built into the weights. A 1 kHz lab asks 1,000 questions, and a 100 Hz lab asks 100.

Ask at your own sampling rate

Purple dots are latent timestamps. Thin warm ticks are output queries. The teal line is one query you choose, and the bars show where it looks. Change the rate and the query slider.

0.35 s

Toy attention: the weights are a smooth bump around the query time, chosen for illustration. A trained model learns its own pattern, which also depends on content. Eight latent timestamps follow the paper’s Appendix A.3 description (256 latents, groups of 8). The range 100 Hz to 1 kHz is from its Section 3.1.

Telling the model which session it is

Each recording has quirks that no vector of spikes shows: the electrode array, the task rules, the apparatus. The paper adds a session embedding, another lookup table with one learned row per session. The row is added into the output token y0. The authors read the whole readout as a sentence:

The query, in words. “Predict [BEHAVIOR] at time [TIMESTAMP] under the experimental conditions of [SESSION ID].”

So the question has three slots: what (the learned y0), when (the timestamp, via rotation) and where (the session row). The answers Y then pass through a small MLP (multi-layer perceptron, a stack of simple layers) that makes the behaviour numbers, such as a 2-D hand velocity.

A worked query. Ask for the velocity at 0.35 s. The output token starts as y0 plus the session row, 128 numbers. Its query is rotated by 0.35 s. It scores all N latents, each rotated by its own timestamp. The softmax picks the latents whose content and time fit best. The answer is their weighted mix of values, 128 numbers again. The MLP then maps those to two numbers, the x and y velocity at 0.35 s. The velocities were z-scored per lab during training, so the output is on that scale.

The shapes, end to end

For a 1 s window with M spikes and 100 behaviour samples: spike tokens M × 128, latents N × 128 after the Perceiver and L self-attention layers, output tokens 100 × 128, answers 100 × 128, and after the MLP 100 × 2. Only M and P change from session to session. The weight matrices never see them.

A useful side effect shows up in the paper’s Figure 5D. After training, the learned session embeddings were projected to two dimensions with PCA. Sessions from the same lab and task landed together, and the datasets named I and L formed their own clusters. The model was never told which sessions were related.

The failure mode, in the right order

Symptom: a fixed output layer cannot serve two labs with different sampling rates. Measurement: the output size must match P, and P is 100 for one lab and 1,000 for another. Fix: P becomes a property of the request, not of the model. The paper does not ablate the session embedding, so we do not know how much of the score it contributes.

Why this is a language-model idea. The output tokens work like the prompt of a language model: the same weights answer whatever you ask. The ask changes, the model does not. Chapter 9 meets POYO+, the successor that decodes several kinds of task.

Lab A records hand velocity at 100 Hz and Lab B at 1,000 Hz. How does one POYO model serve both?

Chapter 6

Training on Many Sessions, Decoding With a Sliding Window

Follow one batch through training and one recording through inference, with every shape and every loss choice

You now have every part. Spikes become tokens, tokens become latents, latents answer queries. This chapter puts the parts in motion: what one training step looks like, how the loss avoids a trap, and how a long recording is decoded one second at a time.

The window is time, not tokens

A language model cuts text into windows of a fixed number of tokens. For neural data that would be a mistake. A window of 1,000 tokens covers a long time when few units are recorded and a short time when many are. So POYO fixes the window in time: any 1 s slice of a recording, taken without regard to trial boundaries (Appendix A.2). Trial structure is used only at test time, to decide which segments count in the score.

A fixed time window means a variable token count M. Batches must be padded to the longest sequence in the batch, and the first cross-attention uses an attention mask so the padding is ignored. Padding wastes work when lengths differ by 20 times. The paper’s fix is load balancing (Appendix A.4.4). It makes one data bucket per GPU, each holding sequences of similar length, and gives long-sequence GPUs a smaller local batch.

One training step

  1. Sample a 1 s window from a random session. Apply unit dropout: keep a random subset of the units, never fewer than 30 (Appendix A.4.2).
  2. Tokenize its spikes, plus the start and end markers: M × 128 and M timestamps.
  3. Compress with cross-attention into N latents, then run L self-attention layers: N × 128.
  4. Query with P output tokens, each carrying its time and the session embedding: P × 128.
  5. Read out through an MLP: P × 2 for planar hand velocity.
  6. Score against the true velocity with mean squared error, and update every weight, the unit table and session table included.

Unit dropout is a cheap augmentation. Because the model reads only the tokens that exist, a sub-population is a valid input. This teaches the model to work with whatever units show up on a new day.

The loss and its traps

The loss is mean squared error (MSE) on hand velocity. Three details keep it fair across labs (Appendix A.4.3).

One ruler per lab. Velocities are z-scored with one scale per lab, so a lab that reports centimetres per second and another that reports a different range land on a common scale.

Fair counting. A 1 kHz session has ten times as many velocity samples in a window as a 100 Hz one, and would dominate the loss. So the error is evaluated on 100 time points, picked at random on every pass.

A heavier weight for good data. The center-out task is structured, with a planning phase before each reach, and has been studied for years. The loss on the reaching segments is multiplied by 5. The paper compares this to up-weighting good-quality text when training language models. It reports no ablation of this factor.

The optimizer is LAMB with weight decay, a learning rate held constant and then decayed by a cosine schedule over the last 25% of epochs. Large models use 8 GPUs with a total batch of 1,400; single-session models use one GPU and a batch of 128. Both large models ran 400 epochs on 8 Nvidia A40 GPUs: POYO-mp took 2 days and POYO-1 took 5 days. A single-session model trains in under an hour on one RTX 3090.

SettingValueSource
Embedding size D128Section 3.1, Table 4
Heads, head size8 heads, 64 per headTable 4
Single-session model6 layers, 128 latents, dropout 0.3, lr 4e-3, batch 128Table 4
Multi-session modelN = 512 latents (text); POYO-mp 24 layersSections 3.1, 3.3
Context window1 s, step 500 ms at inferenceAppendix A.2
Split per session20% of trials test, 10% validationSection 3.1

Inference: slide, predict, average

A recording lasts minutes, and the model sees one second at a time. At test time the paper slides the 1 s window forward in steps of 500 ms. Neighbouring windows overlap by half, so every moment, except at the very ends, is predicted twice. The two predictions are averaged.

Why bother? A window’s edges are its weakest part. A prediction at the first or last instant has only one-sided context: it cannot see the spikes before or after. Overlap gives each moment a second prediction from a window that sees it near the middle.

Slide the window, average the overlaps

The blue line is the true signal. Each pink bar is one 1 s window. The warm line is the average of the windows that cover each moment. Pick a step size and compare the error.

Illustrative. The paper states the procedure (1 s windows, 500 ms step, overlaps averaged) but not an error curve. Here each window’s error is made-up noise that grows toward the window edges, with a different random draw per window. The error is measured from 1 s to 3 s, where every setting has full coverage.

The error in the toy drops when windows overlap, because independent errors partly cancel in the average. The 250 ms step goes further, with four views of each moment, at four times the compute. In a real model the benefit depends on how the real edge errors behave, which the paper does not measure.

Symptom, measurement, fix

Symptom: one lab’s sessions dominate the training loss. Measurement: the number of velocity samples per window differs by up to 10 times between labs. Fix: score 100 random time points per window, and z-score by lab. A second symptom: GPUs idle while one waits for a very long batch. Measurement: M varies from about 1,000 to 20,000. Fix: bucket by length.

Why does the loss use 100 randomly chosen time points per window, not every velocity sample?

Chapter 7

Unit Identification: Learn One Vector Per New Cell

Freeze the whole model and fit only the name tags of the new units, then see when that is enough

It is Tuesday. A new set of cells is on the wires, and you have a pretrained POYO. You have a few minutes of recording with hand velocity labels. How do you get a working decoder?

You have three options. Train a new model from scratch on the few minutes. Fine-tune every weight of the pretrained model. Or do what the paper calls unit identification: keep every weight frozen and learn only the name tags of the new units.

The recipe

In Section 2.5 the recipe is short. Freeze all existing weights. Add new rows to the UnitEmbed table, one for each new unit, and one new row to the session embedding table. Run gradient descent on the labelled data, changing only those new rows. The bulk of the model, which maps population activity to behaviour, is simply transferred.

Why can this work? The frozen network already knows how to read a population described by name tags. It learned, across thousands of units, what tag means “fires for rightward reaches”. A new cell only needs a tag in the right region of the embedding space. Finding it is a search over 128 numbers per unit, with everything else fixed.

A toy you can run

The device mimics this at tiny scale. The frozen decoder reads eight cells, each with a tag ek of two numbers. It outputs the sum over cells of (rate − 0.5) times its tag. The new day has cells with unknown favourite directions. You give the decoder some labelled samples and press Learn the embeddings. Gradient descent changes the sixteen tag numbers and nothing else.

Fit the name tags, freeze the rest

Pink ticks mark where each cell’s tag should point. Purple arrows are the tags the model has learned so far. The curve below is the training error. Choose how many labelled samples you have, then press the button.

A toy of the idea, not the paper’s model. Eight cells with unknown favourite directions, each rate is 0.5 × (1 + cosine of the angle to its favourite direction) plus noise of up to 0.3 either way, 150 full-batch gradient steps on squared error. The decoder is linear and frozen; only the eight 2-number tags move. Samples and noise are seeded, so the same press gives the same result.

With 64 samples the tags land close to their targets and the held-out error falls to about 1.5 degrees. With 16 samples it is about 5 degrees. With 4 samples there are fewer equations than the sixteen unknowns, so the noise leaks into the tags and the error stays near 29 degrees. More labelled data averages the noise away.

What the paper shows

Table 2 compares the options for a held-out session of Monkey C, the same animal on a new day. The numbers are R² for hand velocity.

OptionWhat changesR², Monkey C new day
POYO, trained from scratchAll weights, on the new data only0.9682
POYO-mp + unit identificationNew unit and session rows only0.9675
POYO-mp + fine-tuningEverything, after the embeddings0.9708

Freezing the model costs almost nothing here: 0.9675 against 0.9682. Fine-tuning goes beyond the from-scratch model, to 0.9708. The same table shows a harder case. On a new animal doing random-target reaches, unit identification gives 0.7759 (from scratch 0.7800), while fine-tuning gives 0.8105. So the frozen route is best when the new data look like training data, and less sufficient when they do not.

How fine-tuning is done

Appendix A.6 describes gradual unfreezing. For the first few steps only the unit embeddings and the session embedding train. This is unit identification. Then all the other weights unfreeze. The optimizer is LAMB with batch size 128, learning rate 10-4 and weight decay 10-4. The authors use the same settings for every experiment. They argue this shows the method needs no expert tuning.

The cost is small. Unit identification converges in under a minute on a GPU, or a few minutes on a CPU (Appendix A.4.1). Fine-tuning also uses a single GPU.

The paper also checks how fast this learns. Its Figure 4 plots score against the number of training samples for a held-out session. Fine-tuning the pretrained model keeps a gap above single-session training across the whole range of sample sizes. A new recording usually has few labels, a few minutes and not hours, so this is the case that matters.

Does it find the true cells?

The authors test this honestly (Appendix C.2). Take a session the model already saw in pretraining, throw away its learned unit rows, start new rows from random numbers, and run unit identification. If the method works, each new row should land near the original row for the same unit.

SessionUnitsUnit-ID accuracyCosine similarity
Monkey C, center-out, 2013/10/03731.0000.845
Monkey M, center-out, 2014/02/031160.9660.802

Accuracy here means the share of re-learned rows whose nearest neighbour, among the original rows, is their own original. In the first session all 73 units are found. In the second, 96.6% of 116 are. The average cosine similarity is 0.845 and 0.802, so the rows are close but not identical. The authors call the method robust. Note that this tests re-finding units the model has seen, not a brand-new cell.

In unit identification, which parameters are updated by gradient descent?

Chapter 8

What the Tables and Figures Actually Show

Read each result with its number, its comparison, and the things the paper leaves open

A decoder score needs a ruler. The paper reports R², the share of the hand velocity’s variation that the decoder explains. R² = 1 is perfect. R² = 0 is no better than always guessing the average. Each score below is a mean over test sessions, and the test trials were never seen in training.

Step 1: one model per session works well

The authors first trained 100 separate POYO models, one per session, on three monkeys (C, M and Ja). Each session holds 10 to 106 minutes of data. The average R² is 0.9347 on the center-out task and 0.8402 on the random-target task. The center-out task is a stereotyped reach to one of eight targets. The random-target task is continuous and self-paced, so it is harder. POYO beats a Wiener filter, a GRU and an MLP, with the largest gains on random-target sessions.

Step 2: one model for all sessions works better

Now train one model, POYO-mp (mp for multi-session; 24 layers), on all 100 sessions together: 9,789 units and 4,367 neuron-hours (units times hours recorded). The average test R² rises to 0.9512 on center-out and 0.8738 on random-target. In the paper’s Figure 2C and 2D, nearly every session improves, and sessions with few trials gain the most. So training across sessions pays, even though the units are all different.

Step 3: scale

The scaling study (Figure 3) trains three depths: 6 layers (3.8 million parameters), 14 (7.4 million) and 26 (13 million). Even at 6 layers, the multi-session model beats the average single-session model. More depth helps further, and random-target helps most. Data matters too. At depth 6, going from single-session training (about 1.2 million tokens) to all random-target sessions (26 million) helps. Adding the center-out sessions too (134 million tokens in total) and deepening the model lifts random-target R² by 4% over the single-session baseline. These are three points on a curve. The paper does not fit a scaling law, and it says only that it “offers insights” into one.

Step 4: transfer to the unseen

Chapter 7 showed the same-monkey case. For a new monkey (T, held out entirely), Table 2 gives: center-out, from scratch POYO 0.9194, POYO-mp with unit identification 0.9012, with fine-tuning 0.9379; random-target 0.7800, 0.7759 and 0.8105. The best non-POYO baseline, the MLP, scores 0.8577 and 0.7467. These are means over 6 sessions each, with large spreads (0.0471 to 0.0561 for the random-target pretrained rows).

Table 3 tests data from outside the training labs: the Neural Latents Benchmark Maze and RTT datasets, new monkeys with a touch screen and different equipment. Use the device to see all twelve rows.

Read Table 3 row by row

Pick a benchmark dataset. Purple bars are pretrained POYO models, grey bars are trained from scratch. The warm outline marks the best from-scratch baseline that is not POYO.

Numbers copied from Table 3 of the paper. Axis starts at 0.40, not 0, to make differences readable. A paper-text note: for NLB-Maze the text says unit identification gives 0.8952, while the table prints 0.8962; this page uses the table.

On Maze, POYO-1 with fine-tuning scores 0.9482. The best baseline is AutoLFADS with a linear readout at 0.9062, a gap of 0.0420. On RTT, POYO-1 with fine-tuning scores 0.7378 and the best baseline is the MLP at 0.6953, a gap of 0.0425. Unit identification alone, with a frozen model, already beats every baseline on RTT (0.7107 for POYO-mp, 0.7294 for POYO-1), but not on Maze for POYO-mp (0.8962, below AutoLFADS at 0.9062).

Step 5: two pretrained models, two strengths

POYO-1 adds data from other labs: seven monkeys across three labs, 27,373 units and 16,473 neuron-hours. It transfers better under unit identification (Maze 0.9329 against 0.8962 for POYO-mp). POYO-mp does better on same-lab random-target sessions, 0.8788 against 0.8664. More diverse data helps the strange case and costs a little on the familiar one. In Figure 5D the session embeddings of POYO-1 cluster by lab and task without any labels.

What the paper does not show

The honest summary. The consistent pattern is that pretraining helps, more for harder tasks, and that learning only the unit tags often gets most of the way. The size of each piece of the design is not measured.
Which result best supports the claim that a frozen pretrained POYO can adapt to a new day of the same monkey?

Chapter 9

Field Guide

Keep the numbers, the terms and the core code, and see where POYO and its successor POYO+ sit

POYO is a recipe in five moves. Turn each spike into a token made of a unit name and a time. Compress the tokens with cross-attention into a few hundred latents. Mark time by rotating queries and keys, so attention feels only the gap. Read out behaviour with output queries that carry a time and a session. Adapt to a new session by learning name tags for the new units, with the rest frozen.

The numbers to keep

QuantityValueWhere in the paper
Embedding size D128Section 3.1
Context window1 s; inference step 500 ms, overlaps averagedSection 3.1, A.2
Input tokens per window Mabout 1,000 to 20,000A.4.4
Latents N512 in the main text; 256 in A.3; 128 for single-session (Table 4)3.1, A.3, Table 4
Rotary periods1 ms to 4 s, geometric; half of each head rotatedA.1
LossMSE on hand velocity; 100 random time points; 5x on reachingA.4.3
Single-session average R²0.9347 center-out, 0.8402 random-target (100 models)3.2
POYO-mp average R²0.9512 center-out, 0.8738 random-target3.3
Scaling studyL = 6, 14, 26 layers = 3.8M, 7.4M, 13M parametersFigure 3
POYO-1 training data27,373 units, 16,473 neuron-hours, 7 monkeys3.5
Compute8 A40 GPUs; POYO-mp 2 days, POYO-1 5 days, 400 epochsA.4.1
Unit identification timeunder a minute on a GPUA.4.1
Fine-tuningLAMB, lr 1e-4, weight decay 1e-4, batch 128, gradual unfreezingA.6

Terms in one line each

Can you do these from memory?

  1. Say why a slot-based decoder fails on a new day.
  2. Give the token count with and without start and end markers for a window of 23 spikes and 5 units (23 and 33).
  3. Compare the score counts of self-attention and the Perceiver design for M = 10,000, N = 512, L = 6.
  4. Show that rotary scores depend only on tj − ti.
  5. List the shapes from spike tokens to velocity output.
  6. State what unit identification freezes and what it trains.

The core as code

python# A sketch of POYO's data flow, written for this lesson. Not the authors' code.
# Official code and weights: poyo-brain.github.io  (library: torch_brain)
class POYO(nn.Module):
    def __init__(self, n_units, n_sessions, D=128, N=512, L=6):
        self.unit_emb = nn.Embedding(n_units, D)          # one row per unit, ever seen
        self.sess_emb = nn.Embedding(n_sessions, D)       # one row per session
        self.x_start, self.x_end = Param(D), Param(D)     # delimiter vectors
        self.z0 = Param(N, D)                             # learned latents
        self.t_lat = spread_over_window(N)                # latent timestamps in [0, T]
        self.y0 = Param(D)                                # the shared query vector
        self.enc = CrossAttnBlock(D)                      # latents read spikes
        self.proc = [SelfAttnBlock(D) for _ in range(L)]
        self.dec = CrossAttnBlock(D)                      # queries read latents
        self.head = MLP(D, 2)                            # hand velocity (x, y)

    def forward(self, unit_ids, t_spk, markers, mask, t_out, session):
        x = self.unit_emb(unit_ids)                       # (B, M, D)
        x = x + markers_vec(markers)                      # start / end tokens add x_start / x_end
        z = self.enc(self.z0, self.t_lat, x, t_spk, mask) # (B, N, D)   RoPE inside, M is masked
        for blk in self.proc:
            z = blk(z, self.t_lat)                        # (B, N, D)   N x N attention
        q = self.y0 + self.sess_emb(session)              # (B, 1, D)
        q = q.expand(-1, len(t_out[0]), -1)               # (B, P, D)   same vector, P times
        y = self.dec(q, t_out, z, self.t_lat)             # (B, P, D)
        return self.head(y)                               # (B, P, 2)

# Unit identification on a new session: freeze everything, train only new rows.
for p in model.parameters(): p.requires_grad = False
model.unit_emb = grow(model.unit_emb, n_new_units)        # new rows, trainable
model.sess_emb = grow(model.sess_emb, 1)
opt = LAMB([model.unit_emb.new_rows, model.sess_emb.new_rows], lr=1e-4)
for batch in labelled_windows:                          # 1 s windows, random 100 time points
    loss = ((model(*batch.inputs) - batch.velocity) ** 2).mean()
    loss.backward(); opt.step(); opt.zero_grad()

POYO+ in context

The same group followed POYO with POYO+ (Azabou, Pan, Arora, Knight, Dyer and Richards; ICLR 2025). Per its project page, POYO+ extends POYO in three ways. It handles regular time series, such as calcium imaging, through a value projection layer. It handles recordings across many brain regions and cell types. It decodes several kinds of task at once: regression, classification and segmentation. It trains on the Allen Institute Brain Observatory data, which the page describes as responses from over 100,000 mouse neurons in 6 brain areas to visual stimuli, recorded by two-photon calcium imaging.

The page reports that pooling data from different sources helps several downstream decoding tasks, and that the model transfers between brain regions and cell sub-types. Its latent representations separate regions and sub-types even though the model was never given those labels. This lesson covers POYO only. For POYO+ details, read the paper at poyo-plus.github.io.

Where POYO sits

POYO is one of several neural foundation-model attempts. NDT masks binned spikes and learns to fill them in. NDT2 and NDT3 scale that idea across sessions. CEBRA learns latent spaces with contrastive learning. LFADS models single-trial dynamics. POYO’s difference is the token: events, not bins, with names, not slots. The authors say a next step is a self-supervised objective, such as next-event prediction or masked modelling, so that unlabelled data can be used too.

Limits to remember

Keep going

Read the lecture this paper belongs to, drift and recalibration, to see the older alignment methods POYO sidesteps. Then see the building blocks again: neural manifolds, linear decoders, spike sorting (where units come from) and the transformer.

Credit: Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael J. Mendelson, Blake Richards, Matthew G. Perich, Guillaume Lajoie and Eva L. Dyer, at Georgia Tech, Mila, Université de Montréal and McGill. arXiv 2310.16046, NeurIPS 2023. Project page and code: poyo-brain.github.io. All figures here are our own toys unless a line says they come from the paper’s tables.

The takeaway. Give every cell a name tag the model can look up, give every spike a time, and the same network can read many animals, many days and many labs. A new day then costs a few new tags, not a new model.
You have a pretrained POYO and 20 minutes of labelled data from a new session with new units. Following the paper’s recipe, what is the first step?

Now press Present or Teach and explain POYO from memory: why slots fail, what a spike token holds, how the Perceiver cuts the cost, why rotary scores see only the gap, how a query asks for behaviour, and what unit identification freezes. Then go back to the decoder at the top and pick a new monkey again.

POYO: A Unified, Scalable Framework for Neural Population Decoding
Read the paper · Project page · POYO+ · Back to Gleams