EEG caps differ in channels and length. LaBraM cuts every recording into 1-second patches, learns to name hidden ones, and lifts TUEV balanced accuracy from 0.5281 to 0.6409.
Learn how one pre-trained model reads brain recordings from any cap: the patches, the spectrum tokenizer, the masked prediction, and the criss-cross attention that followed.
Pick a cap, then drag the share of hidden patches. Watch one 256-patch budget hold a 23, 32, 62 or 64 channel recording. Then we build every piece, one mechanism at a time: the patch, the encoder, the Fourier target, the codebook, the mask, the results and the ablations, and CBraMod’s criss-cross attention.
You need a rough idea of what a Transformer does (it lets every piece of an input look at every other piece) and a little Python for the code. We teach EEG, the Fourier transform and vector quantization from zero.
Based on Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI (LaBraM), Wei-Bang Jiang, Li-Ming Zhao and Bao-Liang Lu, ICLR 2024, and on CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding, Jiquan Wang, Sha Zhao and colleagues, ICLR 2025. EEGPT (Wang et al., NeurIPS 2024) appears as context. This paper is a reading of the lecture Spatial Filters and CSP. Every idea is re-taught here in our own words and devices.
The caps are datasets named in the LaBraM paper (channel counts from its Appendix D and Section 3.1). The rule is the paper’s: patches of 200 samples (1 second at 200 Hz) and at most 256 patches per sample, so 64 channels give 4 seconds and 32 channels give 8 seconds. Which patches hide is a seeded toy draw. The paper hides 50% in its main setting.
Chapter 0
See why every EEG network built so far fits one cap, one task and one recording length, and what a foundation model would change
Picture two labs. A hospital lab records patients with a cap of 23 electrodes, sampled 256 times a second. A psychology lab shows film clips to volunteers and records with 62 electrodes, sampled 1,000 times a second.
A team trains a neural network on the hospital data to tell normal brain recordings from abnormal ones. It works. Now someone hands it a file from the psychology lab. The network expects 23 rows of numbers. The file has 62. The network cannot even read it.
This is the daily situation in EEG research, and it is the problem the two papers in this lesson attack. First we need the basic words.
Electroencephalography (EEG) measures the tiny electrical activity of the brain with metal discs on the scalp. Each disc is an electrode. The voltage at one electrode, followed over time, is one channel. Think of each channel as one microphone in a large hall, each hearing the crowd from its own seat.
Electrodes sit at standard spots named by the international 10–20 system, such as Fp1, Cz or O2. The letters name a brain region and the number names a side and a place. A recording is then a table of numbers.
The paper states that EEG values usually fall between −0.1 mV and 0.1 mV. That is a ten-thousandth of a volt. Every number in X is tiny, and noise from muscles, eyes and mains power can be as large as the signal.
Both papers list the same walls. First, labels are scarce. A label is an expert’s answer for a recording, such as “abnormal”. Experts are slow and expensive, so most labeled EEG sets cover a few dozen people and less than tens of hours.
Second, formats clash. Caps differ in channel count, labs differ in sampling rate, and tasks differ in recording length. A network with a fixed input shape fits exactly one of these.
Third, the signal is weak. The LaBraM authors call out the low signal-to-noise ratio (SNR), the size of the useful signal against the noise. Many researchers still design features by hand because learning from raw EEG is hard.
Try the second wall yourself. The chart lists ten datasets named in the LaBraM paper. Each bar is the count of numbers in ten seconds of that dataset. Pick the dataset your imagined network targets.
Each bar is channels × 10 seconds × sampling rate. A green tick means the shape matches the network you picked. A red cross means the network cannot take that file as it is.
Channel counts and sampling rates are from the LaBraM paper (Appendix D, and Section 3.1 for Temple’s TUAB and TUEV). “Fits” is the toy rule of a fixed input: the same channel count and the same sampling rate. Real caps also differ in electrode names and reference wiring, which makes the gap wider.
Almost nothing fits. The only dataset that matches is the one the network targets. A lab that wants to reuse a model must rebuild it, retrain it, and find new labels.
Text faced a similar wall. A language model reads huge amounts of unlabeled text, learns by filling in hidden words, and only then gets a small labeled task. This recipe is self-supervised pre-training: the data supplies its own exercises, so no expert is needed. The model that results is called a foundation model because many tasks build on it.
LaBraM, the Large Brain Model, tries the same recipe on EEG. The authors collected about 2,500 hours of EEG from about 20 datasets, mostly public and some recorded by themselves. The authors kept the four test datasets out of pre-training. They trained three sizes, from 5.8 million to 369 million parameters, and call these the largest models in the field so far.
To make the recipe work they must answer three questions. How do we cut any recording into a standard unit? What do we ask the model to predict, given the noise? And how do we hide pieces so the exercise teaches something? Each answer is one chapter.
Chapter 1
Turn a table with any number of rows into a list of equal tiles that a Transformer can read
Imagine tiling a floor. Rooms come in every shape, but if every tile is the same one-foot square, you can cover any room. You only need to count the tiles.
LaBraM does the same to EEG. A recording has any number of channels and any length. The model never sees that table. It sees a list of identical tiles, each one a single channel for a single second.
A Transformer reads a list of equal-size pieces called tokens. In text a token is a word piece. In images the Vision Transformer made a token from a 16 by 16 block of pixels (the Vision Transformer lesson builds this). The LaBraM paper copies that idea.
Take a sample x with C channels and t time points. Pick a window length w. Cut every channel into non-overlapping windows of w samples. Each window is an EEG channel patch.
The paper sets w to 200 samples. After resampling to 200 Hz, one patch is exactly 1 second of one channel. A longer recording is first cut into samples of t time points, moving ahead by a stride. During pre-training the stride is 4 seconds.
The authors apply “very little” preprocessing. Each step has one job.
The patches of one sample become one sequence. The model compares every patch with every other patch, so the cost grows with the square of the sequence length. For stable cost the paper caps the sequence at 256 patches. Then 256 × 256 = 65,536 pairs are compared in each attention head.
A fixed budget in patches means a flexible length in seconds. The paper gives two cases: a 64 channel cap gets 4 seconds per sample, and a 32 channel cap gets 8 seconds.
Now play with it. Choose a channel count and a number of seconds. The grid shows the patches the model would receive, and what breaks when you ask for too many.
Each tile is one channel for one second, 200 numbers. Blue tiles fit the budget. Red tiles do not. Drag the sliders, or press a preset.
The rule (200 samples per patch, at most 256 patches) and the 64 and 32 channel examples are the paper’s (Section 3.2). The 5 s and 10 s presets are the TUEV and TUAB sample lengths from Section 3.1. The pair count is the exact number of query-key pairs in full attention over the sequence.
Try the last preset. A 64 channel cap with 10 seconds makes 640 patches. That is 2.5 times the budget and 6.25 times the pairs. The fix is the stride: cut the 10 seconds into windows of 4 seconds and treat each as its own sample.
What does a patch lose? Inside one patch, the model sees one electrode only. Links between electrodes appear later, in attention, where all patches meet. Links inside the one second appear earlier, in the convolutions of Chapter 2. The cut is a trade: a standard unit now, in return for pushing every cross-channel question into the Transformer.
The failure mode is easy to meet. Symptom: you feed a 64 channel, 10 second file and the sequence is 640 patches. Measurement: 640 × 640 = 409,600 pairs, 6.25 times the pairs the budget allows, and 2.5 times the memory. Fix: cut the file into 4 second samples before patching.
Take one sample of SEED, an emotion dataset with 62 channels recorded at 1,000 Hz. The data flow runs as follows.
Notice what the model does not know yet. After flattening, a patch has no label for which channel it came from or when. A patch from Cz at second 3 looks like a bag of 200 numbers, same as any other. Chapter 2 fixes that.
Chapter 2
Compress 200 voltages into one vector, add a time tag and a place tag, and feed the Transformer
A mail room receives 115 envelopes. Each holds a sheet with 200 numbers, but no return address. The sorter cannot tell which electrode sent a sheet, or at which second. Before sorting, someone must summarize each sheet and stamp it with a place and a time.
That someone is the first part of LaBraM, shown in the paper’s Figure 1. Three steps turn a patch into a token: encode, tag, then pass to the Transformer.
Why not feed the 200 raw numbers straight to attention? The authors note that EEG is rich in fine time detail. So they pull out time features first, patch by patch, using convolutions.
A 1-D convolution slides a short window of weights, the kernel, along the signal. At each stop it multiplies and adds, so it lights up where the signal looks like the kernel. The stride is how far the window jumps. Padding adds zeros at both ends. The output length follows one formula.
The paper’s Table 3 gives three convolution blocks. Each block is a convolution, then group normalization (it rescales groups of channels inside one sample, so it does not depend on batch size), then GELU (a smooth switch that passes positive values and softly blocks negative ones).
The settings are: input channels 1, 8, 8; output channels 8, 8, 8; kernel 15, 3, 3; stride 8, 1, 1; padding 7, 1, 1. Work the first block: L = ⌊(200 + 14 − 15) / 8⌋ + 1 = ⌊24.875⌋ + 1 = 25. The next two blocks keep 25, since (25 + 2 − 3) / 1 + 1 = 25.
After three blocks the patch is a table of 8 channels by 25 steps. Flattened, that is 8 × 25 = 200 numbers, the same as the Base hidden size d = 200. The Large model uses 16 channels and d = 400. The Huge model uses 32 channels and d = 800. The match holds all three times. The paper does not write out the flatten, so this is our reading of its tables.
Set the dials below to check the arithmetic, and break it.
Choose a model size, then change the first block’s kernel, stride and padding. The last bar must land on the hidden size.
Kernel 15, stride 8, padding 7 and the blocks after it (kernel 3, stride 1, padding 1), with 8, 16 and 32 output channels, are Tables 3 and 4 of the paper. The hidden sizes 200, 400 and 800 are Table 4. That the encoder output is flattened to d numbers is inferred from these tables, not stated in the text.
Only the paper’s setting lands on 200. Slide the stride to 4 and the patch becomes 50 steps, so the flatten gives 400. The block would no longer fit the Transformer without another layer.
Now each patch is a vector e of d numbers. The paper adds two embeddings, which are learnable vectors looked up from a table.
The temporal embedding list TE has one entry per time slot, up to a limit tmax. The spatial embedding list SE has one entry per electrode of the universal 10–20 set. Cz has one entry, whatever the cap.
This is how one model serves many caps. A 23 channel and a 62 channel recording each pick their own rows of SE. The same electrode name always gets the same vector. Missing electrodes simply never appear in the sequence.
The tagged tokens go into a standard Transformer encoder (the Transformer lesson builds it). Base has 12 layers, 10 heads and a feed-forward width of 800. Two edits come from the ViT-22B recipe of Dehghani and colleagues.
First, a layer norm (LN, which rescales a vector to zero mean and unit spread) is applied to queries and keys before the dot product. Second, the bias terms in the Q, K and V projections are dropped, which speeds up training with no loss.
Why this helps: a normalized vector of width 20 has length √20. By the Cauchy–Schwarz bound, a dot product of two such vectors is at most 20. Divide by √20 and every logit stays within ±4.5. Without the LN, logits can grow without limit, and the softmax collapses onto one patch. (The bound is exact at unit gain, the layer norm’s starting setting.)
For downstream tasks the paper averages the output vectors over all patches and adds a task head. Base has 5.8 million parameters.
Chapter 3
See why LaBraM asks its tokenizer to rebuild the Fourier amplitude and phase of a patch instead of its raw voltages
You record the same piano chord twice in a noisy room. The two sound files look different, wiggle by wiggle, because the room noise differs. But a musician hears the same three notes both times. The notes are the stable part.
EEG has the same trouble, and worse. A model that must rebuild every wiggle is asked to copy noise. The LaBraM authors report that when they tried this, the loss did not converge. So they changed the target.
Any wave is a sum of pure sine waves of different pitches. The Fourier transform finds the recipe: for each frequency, how much of that sine is present, and where in its cycle it starts. The discrete Fourier transform (DFT) does this for a list of samples. (The DFT lesson derives it from scratch.)
The brain makes rhythms at typical pitches. The alpha rhythm sits near 10 Hz and appears when you close your eyes. The DFT shows it as a tall spike at 10 Hz. See the brain rhythms lesson.
The paper writes the DFT of one patch x with w samples. It tests each frequency index m.
A worked number. Take a patch that is a pure 10 Hz sine of height 1, sampled 200 times. Index 10 gives X̃ = 100, so the amplitude is 100 (half of the 200 samples, times the height). Every other index gives 0. A sine of height 0.5 would give 50. The spectrum tells us what is present and how strongly.
Because the voltages are real numbers, the second half of the spectrum mirrors the first (the DFT is conjugate symmetric). So a patch of 200 samples has 100 useful frequencies, 1 Hz to 100 Hz. The paper’s Figure 7 shows half the spectrum for this reason.
Every patch gets an amplitude list and a phase list. Within each sample the paper applies a z-score to both: subtract the mean, divide by the spread. This makes the numbers comparable and the training stable. A decoder then predicts both lists, and the loss is a plain mean squared error on each.
Phase is the harder half. The paper reports that the amplitude trend is rebuilt well, the phase less so, though the reconstruction loss still falls steadily. Try the toy below to feel why amplitude is the steadier target.
Top: two recordings of the same 10 Hz and 20 Hz rhythm with different random noise. Bottom: the amplitude of each, 1 to 60 Hz. Raise the noise, then draw new noise.
Illustrative toy, not EEG data: a 10 Hz sine of height 1 plus a 20 Hz sine of height 0.5, 200 samples at 200 Hz, plus white noise. Real EEG noise is not white and not stationary. The patch length and the DFT definition are the paper’s (Eqs. 5 to 8).
At noise level 1.0 the two waveforms agree only weakly, while the two spectra still show the same two peaks. That is the argument for the spectrum: it keeps the part of the patch that is repeatable.
Phase is an angle, so it wraps around: 359 degrees and 1 degree are close, but their difference looks huge. It also reacts to tiny timing shifts. Shifting a patch by Δt seconds moves the phase of frequency f by 360 × f × Δt degrees. A 1 ms shift moves a 10 Hz wave by 3.6 degrees. It moves an 80 Hz wave by 28.8 degrees.
Amplitude has no such trouble. A 1 ms shift leaves it untouched. This is why the paper’s Figure 7 shows amplitude rebuilt well and phase less so. The training still gains from phase, since the loss falls steadily.
The failure mode, then, has a clear shape. Symptom: a tokenizer whose decoder must copy raw voltages stalls, and its loss does not converge (the authors report this for their tokenizer, not for the later masked prediction). Measurement: in the toy above, two noisy copies of one rhythm barely correlate as waves. Fix: aim at the part that repeats, the spectrum.
The spectrum is not the final prediction. It is the exam the tokenizer must pass. The tokenizer squeezes each patch into one code from a small list. The decoder must rebuild the spectrum from those codes alone. If the codes keep what the spectrum needs, they keep the rhythm content of the patch. Chapter 4 builds that tokenizer.
Chapter 4
Build the neural tokenizer: match each patch to its nearest code, and train the codes by rebuilding the spectrum
A paint shop sells exactly 8,192 colors. You bring a photo of a wall, and the clerk names the closest shop color for every square inch. You walk out with a list of color numbers. You lost fine shades, but you kept what mattered, and the list is short.
A language model predicts a word out of a fixed vocabulary. EEG has no words. The LaBraM authors build a vocabulary by learning it, and call the result a neural tokenizer with a neural codebook.
The tokenizer reuses the encoder of Chapter 2 (a temporal encoder, tags, and 12 Transformer layers) to turn each patch into a vector p of fixed width. The codebook V holds K = 8,192 learned vectors, each of width D = 64 (Table 3 of the paper). Vector quantization replaces p with the nearest entry of the codebook.
Why normalize? Without it a long vector and a short vector pointing the same way would pick different codes, since distance also measures length. After normalization only direction counts. The paper cites earlier work showing this improves codebook utilization, meaning more of the 8,192 codes actually get used. Try it on two dimensions.
Pink dots are 16 toy codes. The teal dot is a patch vector. Turn it, then stretch it. Strip: the code chosen as the length sweeps from short to long.
Toy: 16 codes in 2 dimensions with seeded positions and lengths. The real codebook has 8,192 codes of width 64 (Table 3). The nearest-code rule with L2 normalization is the paper’s Eq. 4. That normalization improves utilization is the paper’s citation of Peng et al. 2022, not something this toy measures.
In raw mode, stretching the vector can send it to a far code, so the strip changes color along the way. In L2 mode the strip is one color from end to end, because length no longer enters.
A tiny worked case. Let a patch vector be (3, 4), which has length 5. Divided by 5 it becomes (0.6, 0.8). A code vector (0.8, 0.6) has cosine similarity 0.6 × 0.8 + 0.8 × 0.6 = 0.96 with it. A code (1, 0) scores only 0.6. The patch picks the first code, and would pick it again as (6, 8) or (0.3, 0.4).
The codes must mean something. The paper trains them with a rebuild task. Each patch becomes its nearest code vector. A three-layer Transformer decoder reads the code vectors, averages them, and two small heads predict the patch’s z-scored amplitude and phase (Chapter 3). Training makes the codes carry what the spectrum needs.
The first two terms are mean squared errors on amplitude and phase. The last two are the standard VQ-VAE pair (the VQ-VAE lesson derives them). The paper also updates the codebook with an exponential moving average, a running average of the patches assigned to each code. That makes the codebook update steadier.
Take a TUEV sample: 23 channels, 5 seconds. Patches: (115, 200). Tokenizer encoder: (115, 64). Nearest code, per row: 115 whole numbers between 0 and 8,191. This list is the sample’s neural token sequence, a sentence in the EEG vocabulary. It is the answer key for the next chapter.
The tokenizer trains first, with 12 encoder layers, 3 decoder layers, a batch of 1,024 and 100 epochs. After that it is frozen. The paper’s Appendix B views the whole recipe as a two-stage variational autoencoder, a discrete one: stage one learns the codes, stage two learns to predict them from masked input.
One failure to know. If all patches fall on a handful of codes, the vocabulary collapses, and a model that predicts the common code looks accurate while learning nothing. The symptom is a few codes dominating the counts. L2 normalization and the moving-average update are the paper’s guards.
Chapter 5
Pre-train the Transformer with masked EEG modeling, and reuse every mask twice with symmetric masking
A teacher hands out a page of text with some words blacked out. You fill in each blank from the words around it. Do this a million times and you learn the language, without anyone telling you what any sentence means.
LaBraM plays this game on brain recordings. It covers some patches of the grid, and the model must name them. The twist is what “name” means here: the model must give the tokenizer’s code for the hidden patch.
Use the sample from Chapter 4: 115 patches, and for each one a code from the tokenizer. The step has five moves.
This is a cross-entropy loss: a classification with 8,192 classes. Guessing at random scores 1/8,192, about 0.012%. Training raises the score. The paper’s Figure 3 shows the loss falling and the masked accuracy rising, with a larger model reaching a lower loss and a higher accuracy.
Look at why the spatial embedding matters here. A blank carries no signal. Only its place tag says which patch to guess. The paper reports that without spatial embeddings the pre-training loss could not converge.
What must the model learn to answer? A hidden patch at Cz in second 3 has neighbors: Cz in seconds 2 and 4, and other electrodes in second 3. The model can only guess well by using both kinds of context. Slow rhythms carry across seconds and across nearby electrodes. That is the knowledge pre-training stores.
The paper’s Appendix B gives a second view. The whole recipe is a variational autoencoder over discrete codes. Stage one trains the tokenizer to rebuild the spectrum from codes. Stage two, this chapter, learns to predict the codes from a damaged input. Splitting it in two keeps each stage simple.
The tokenizer costs one extra forward pass per sample, since it must produce the answer key. The authors want to get more from that pass. So they flip the mask.
Take the mask M. Its inverse M̃ hides exactly the patches M left visible. Run the same sample again with M̃. The same codes are the answer key, so the tokenizer is not called again.
The paper gives two reasons. It reuses the discrete codes, so training is more efficient. And it gives the batch more masking views, which acts like extra data. Appendix I tests it, and Chapter 7 shows what it found.
See both masks at work on a toy grid of 8 channels by 8 seconds.
Orange tiles are hidden and become targets. Blue tiles are seen. Switch on symmetric masking to add the inverse pass, then change the share.
A toy grid of 64 patches, not a real sample (real samples hold up to 256). The rule is the paper’s (Sections 2.3 and Appendix I): the mask hides a share r, the inverse hides 1 − r, and the tokenizer is called once. The paper’s default is r = 0.5.
With r = 0.5 the two masks split the grid in half and every patch becomes a target once. With another r the inverse mask hides the rest, such as 60% after 40%, as in the paper’s Table 8 where “0.4 (0.6)” names the pair.
The mask share is a dial with two failure modes. Hide too little and the guess is trivial, since neighbors give it away. Hide too much and nothing is left to reason from. The paper tested r from 0.1 to 0.9 and found a broad plateau, with 0.5 the safest (Chapter 7).
The paper pre-trains with a batch of 512 samples, a peak learning rate of 5e-4, 50 epochs and the AdamW optimizer. It uses eight A800 GPUs. All of this runs on the 2,534.78 hours of EEG, with the four test datasets held out. Chapter 6 shows what that bought.
Chapter 6
Fine-tune LaBraM on two clinical exams, learn what each metric means, and see what more parameters and more data buy
A student finishes years of general reading. Now two exams arrive, each from a different subject. The honest question is not “did they pass” but “by how much, against whom, and how steady was the score”. This chapter reads the paper’s two main tables that way.
TUAB asks: is this recording normal or abnormal? It comes from the Temple University Hospital corpus. It has 23 channels at 256 Hz and 409,455 samples of 10 seconds each.
TUEV asks a harder question. Which of six event types is in this 5-second window: spike and sharp wave, generalized periodic discharge, periodic lateralized discharge, eye movement, artifact, or background? It has 112,491 samples. Both sets are held out of pre-training.
Fine-tuning means continuing to train a pre-trained model on a labeled task. LaBraM averages the output vectors over all patches, adds a small task head, and trains the whole network. It uses binary cross-entropy for TUAB and cross-entropy for TUEV.
Settings from the paper’s Table 5: batch 512, peak learning rate 5e-4, 50 epochs for Base (30 for Large and Huge), and a layer-wise learning rate decay of 0.65 for Base. That decay gives layers near the input a smaller learning rate, so pre-trained features move gently. The splits follow the BIOT paper, with training patients divided 80% and 20% into training and validation. Every number is the mean over five random seeds, with its spread.
Balanced accuracy is the average recall of each class. Say a test set has three classes with recalls 0.9, 0.5 and 0.1. Balanced accuracy is (0.9 + 0.5 + 0.1) / 3 = 0.5. Plain accuracy could look far better if the first class dominates. Balanced accuracy refuses to hide a neglected class.
Cohen’s kappa measures agreement beyond chance. If the model matches the experts 80% of the time and chance alone would match 50% of the time, kappa = (0.8 − 0.5) / (1 − 0.5) = 0.6. AUROC and AUC-PR are areas under the receiver-operating and precision-recall curves, both across all decision thresholds. Weighted F1 combines precision and recall per class and weights each by its size.
Now read the tables. Pick an exam and a metric. The bars start above zero so differences show, which the axis label states.
Warm bars are LaBraM. Blue bars are the earlier models the paper compares against. The thin line on each bar is plus or minus one standard deviation over five seeds.
Values are Tables 1 and 2 of the LaBraM paper (mean over five seeds; the baselines are the best results from the BIOT paper). The bars start at 0.70, 0.80 and 0.80 for the TUAB metrics, at 0.30, 0.30 and 0.60 for the TUEV metrics, not at zero.
On TUEV, LaBraM-Base reaches 0.6409 balanced accuracy where BIOT, the strongest earlier model, reaches 0.5281. Kappa goes from 0.5273 to 0.6637 and weighted F1 from 0.7492 to 0.8312. The jump is large on the harder, six-class exam.
On TUAB the gain is smaller. AUROC moves from 0.8815 (BIOT) to 0.9022, and balanced accuracy from 0.7966 (ST-Transformer) to 0.8140. The authors suggest TUAB is a lower-level task, where normal and abnormal can often be told apart by eye.
Size helps a little. Going from Base (5.8M) to Huge (369M) adds 0.0207 balanced accuracy on TUEV and 0.0118 on TUAB. That is small beside the leap from earlier models to Base.
For the regression task in Appendix F the metrics change. Pearson’s correlation says how well predictions rise and fall with the truth. R2 says what share of the variation the model explains. RMSE is the typical error size, where lower is better.
Appendix F adds two tasks. On SEED-V emotion recognition (five classes), Base scores 0.4095 accuracy against BIOT’s 0.3802. On MoBI gait regression, Base is on par with the best baseline (R2 0.2876 versus 0.2911 for ST-Transformer), and Large and Huge pass it (0.3093 and 0.3145).
The paper also scales the pre-training data. Per Figure 5, the Base model at 500 hours already beats its own 2,500 hour version on TUAB and reaches over 90% of it on TUEV. Large grows more slowly after 1,000 hours. Huge keeps rising. From this and the scaling law of Kaplan and colleagues, the authors infer that Huge would need on the order of at least ten thousand hours.
Two more checks. Adding TUAB and TUEV themselves to pre-training changed little (Figure 4), and training from scratch without pre-training dropped steeply (Figure 8).
Three cautions. The baselines come from the BIOT paper, and the authors chose its best numbers. The pre-training pool holds much Temple hospital data: TUAR, TUEP, TUSZ and TUSL add up to 1,842.56 of the 2,534.78 hours, or 72.7%. The paper says TUAB and TUEV recordings are disjoint from these, but the source is the same hospital family. And the whole network is fine-tuned, which costs compute and memory. The paper lists this as one of its three limits.
Chapter 7
Read the paper’s ablation tables to learn which design choices carry the gain, and which are within the noise
A cook thinks a stew needs seven ingredients. To test the idea, she makes the stew seven more times, each time without one ingredient, and tastes. Where the taste collapses, that ingredient mattered. Where nothing changes, it may be decoration.
An ablation does exactly this to a model: remove or swap one part, keep everything else, and compare scores. The LaBraM appendix runs five of them. Each one answers a question you may ask of any design.
Every score has a spread. The paper repeats each run with five seeds and prints mean and standard deviation. On TUEV the kappa spread is often 0.01 to 0.03, so a gap of 0.01 is not a finding. On TUAB the AUROC spread is 0.001 to 0.004.
Use two numbers per variant: AUROC on TUAB, and Cohen’s kappa on TUEV. These are the monitor metrics the authors chose. Pick an experiment below, then read the rows.
Each row is a variant of LaBraM-Base. Blue bar: TUAB AUROC. Warm bar: TUEV kappa. Look for gaps bigger than the spread.
Values are Tables 7 to 11 of the LaBraM paper (Appendices G, H, I, K and L), mean of five seeds. The TUAB bars run from 0.86 to 0.93 and the TUEV bars from 0.30 to 0.70, so they do not start at zero. Large and Huge rows in the symmetric masking experiment are those models, not Base.
Compare a gap with the combined spread of the two runs, which is the square root of the sum of the squared spreads. Take TUEV kappa in Table 7. Codes score 0.6637 ± 0.0093. Rebuilding raw patches scores 0.5910 ± 0.0156.
The gap is 0.0727. The combined spread is the square root of 0.00932 + 0.01562, about 0.0182. The gap is 4.0 spreads wide. That is a real difference.
Now try “predict tokenizer output” (0.6376 ± 0.0168). The gap is 0.0261, the combined spread 0.0192, so the gap is only 1.4 spreads. That is weak evidence. And on TUAB the 0.0045 AUROC edge of raw patches (0.9067 against 0.9022) is about 1.8 spreads, so it is a hint, not a result. Five seeds is a small number.
1. The target (Table 7). Four ways to teach the same Transformer. LaBraM predicts tokenizer codes. Setting 1 predicts the tokenizer’s output vectors by cosine similarity instead of codes. Setting 2 drops the tokenizer and rebuilds raw patches. Setting 3 drops it and rebuilds the spectrum with mean squared error.
On TUEV, kappa is 0.6637 for codes against 0.6376, 0.5910 and 0.5643. On TUAB, plain reconstruction does slightly better (0.9067 and 0.9053 AUROC against 0.9022). The authors explain it this way. TUAB is a low-level task, where bad EEG looks different. TUEV needs the semantic grouping that codes provide.
2. Symmetric masking (Table 9). Without it, Base even gains a little on TUAB (0.9065 against 0.9022). But on TUEV every size drops, and Huge drops most: kappa 0.6745 falls to 0.6391. The authors say the larger the model, the more it needs the extra data views.
3. Mask ratio (Table 8). On TUAB all ratios score close together, with r = 0.4 best (0.9049). On TUEV, r = 0.5 leads the others by about 0.05 kappa. The authors pick 0.5 because its spread is smaller on both exams.
4. How much to fine-tune (Table 10). Training the last 8 of 12 blocks gives the best TUEV kappa (0.6820), a gain about the size of the run-to-run spread, so treat it as level with full fine-tuning. A linear probe, which trains only a new output layer on frozen features, holds TUAB at 0.8835 but drops TUEV kappa to 0.3968. The pre-trained features need adapting.
5. Spatial embeddings (Table 11). Without them AUROC falls from 0.9022 to 0.8888 and kappa from 0.6637 to 0.6069. During pre-training, the loss could not converge at all without them.
One more result sits in the text. Training LaBraM from scratch, with no pre-training, falls steeply on both tasks (Figure 8). Pre-training matters more than any single tweak above.
Four habits follow from the tables. First, write down the spread before you run the comparison, so a lucky seed cannot fool you. Second, test on more than one exam, since the target ablation shows that two exams can disagree. Third, test a frozen version and a fine-tuned version, to learn how far the features travel alone.
Fourth, keep the cheap defaults unless a table beats them by several spreads. The mask ratio of 0.5 and the symmetric mask cost nothing to keep. A new tokenizer costs weeks. The tables tell you which piece deserves the weeks: here, the tokenizer codes and the spatial tags move the six-class score the most.
Chapter 8
Follow the ICLR 2025 successor: criss-cross attention, a position code computed from neighbors, and a plain rebuild loss, with EEGPT as context
Look at a spreadsheet of sales: stores down the side, months across the top. A cell relates most to its own row (the same store, other months) and its own column (the same month, other stores). It relates little to a random cell far away.
EEG patches have the same grid: channels down, seconds across. LaBraM lets every patch attend to every other. A group at Zhejiang University argued that this wastes effort and blurs two different kinds of relation.
CBraMod (Wang, Zhao and colleagues, ICLR 2025) names two problems with models like LaBraM. First, full EEG modeling treats all patches alike. But space and time are different: patches in one channel or in one time slot may relate more closely than patches far apart in both.
Second, absolute position codes tied to electrode numbers assume a fixed link between a channel and a place. In real data a channel also depends on the reference, the electrode or average that voltages are measured against (earlobe, average, REST or bipolar). The same electrode name can mean different things across datasets.
Notice the contrast with LaBraM. There is no tokenizer and no codebook. The target is the raw patch itself. The encoder gets frequency help directly at the input instead.
Now see the pattern. Choose how heads attend, and move the query patch. The counts at the bottom use CBraMod’s real pre-training shape: 19 channels and 30 seconds, which is 570 patches.
Rows are channels and columns are seconds. The teal ring is the query patch. Purple tiles are the patches it attends to. Move the query with the sliders.
The grid is a toy of 6 channels by 8 seconds. The pair counts use the real shape from the CBraMod paper (19 channels, 30 s, 570 patches, 8 heads: 4 spatial and 4 temporal). Pair count means query-key pairs in one head of one layer. The paper’s measured FLOPs (Table 22) are the cost figure that counts.
The saving is large in pairs and smaller in total cost. One full head compares 570 × 570 = 324,900 pairs. A spatial head compares 30 slots of 19 × 19, which is 10,830. A temporal head compares 19 channels of 30 × 30, which is 17,100. Per layer: 8 × 324,900 = 2,599,200 pairs against 4 × 10,830 + 4 × 17,100 = 111,720. But the measured cost drops less, from 469.4M to 318.9M FLOPs (Table 22, measured on CHB-MIT with 16 channels and 10 seconds), about 32%. We expect the feed-forward layers, which attention changes do not touch, to explain the gap; the paper does not say.
CBraMod pre-trains on TUEG, 27,062 hours in 69,652 recordings. After cutting recordings to 19 channels, 30 second samples, and dropping any sample over 100 µV, 1,109,545 samples remain, over 9,000 hours. That is about 3.5 times LaBraM’s 2,534.78 hours. The model has 4.0M parameters (LaBraM-Base has 5.8M) and trained for about 5 days on four RTX A5000 GPUs.
It is tested on up to 10 task types with 12 public datasets. Selected comparisons, all from the CBraMod paper’s own runs:
| Task and metric | CBraMod | Best earlier |
|---|---|---|
| FACED emotion, kappa | 0.5041 | 0.4698 (LaBraM-Base) |
| SEED-V emotion, kappa | 0.2569 | 0.2386 (LaBraM-Base) |
| PhysioNet-MI motor imagery, kappa | 0.5222 | 0.4912 (LaBraM-Base) |
| SHU-MI motor imagery, AUROC | 0.6988 | 0.6609 (BIOT) |
| TUEV events, balanced accuracy | 0.6671 | 0.6616 (LaBraM-Huge) |
| TUAB abnormal, AUROC | 0.9227 | 0.9162 (LaBraM-Huge) |
The ablations tell a clear story. Full attention scores worst of the four attention types tried (Figure 4). A 4:4 split of spatial and temporal heads beats 2:6 and 6:2 (FACED kappa 0.5041 against 0.4921 and 0.4910). Time and frequency together beat either alone (0.5041 against 0.4956 and 0.4652). Cleaning the pre-training data matters: kappa 0.5041 with clean data, 0.4768 with dirty data, 0.4615 with none.
Two checks echo LaBraM. A frozen CBraMod fails as a fixed feature extractor (FACED kappa 0.2579 against 0.5041 fine-tuned). And with only 30% of the labels, CBraMod keeps 0.3239 on FACED against 0.2672 for LaBraM and 0.2573 for BIOT.
EEGPT (Wang and colleagues, NeurIPS 2024) is a third design. Its abstract describes about 10 million parameters, a mask-based dual self-supervised method that aligns spatial and temporal representations instead of raw signals, and a hierarchical design that handles space and time separately. It reports strong linear-probing results. We read only its abstract and the way CBraMod describes it, so we state no numbers for it here.
CBraMod re-ran most baselines itself, using the BIOT code. Only LaBraM-Base weights are public, so on new tasks it compares against Base, and on TUEV and TUAB it compares against LaBraM’s published Huge numbers. Its own Appendix G tests the architecture on the same data and finds the architecture matters more than the larger data. Its stated limits: crude cleaning of TUEG loses data, it still needs more compute than small task models, and it has not been scaled toward billions of parameters.
Chapter 9
Keep the whole paper on one page: what to remember, what to check, and where to read next
LaBraM makes one model out of many EEG datasets. It cuts each recording into one-second channel patches, tags them with time and electrode, learns an 8,192 word vocabulary by rebuilding Fourier spectra, and pre-trains by naming the codes of hidden patches. After full fine-tuning it beats earlier models on TUAB, TUEV and SEED-V, and its Large and Huge sizes lead on MoBI. CBraMod then drops the tokenizer, splits attention into spatial and temporal heads, and reaches higher scores with fewer parameters.
| Quantity | Value | Where |
|---|---|---|
| LaBraM pre-training data | 2,534.78 hours, about 20 datasets | LaBraM, Appendix D |
| Patch and budget | 200 samples (1 s at 200 Hz), at most 256 patches | LaBraM, Section 3.2 |
| Model sizes | 5.8M, 46M, 369M parameters | LaBraM, Section 3.2 |
| Layers, width, heads | 12, 200, 10 · 24, 400, 16 · 48, 800, 16 | LaBraM, Table 4 |
| Codebook | 8,192 codes of width 64, L2 normalized | LaBraM, Table 3 and Eq. 4 |
| Mask | r = 0.5, plus the inverse mask | LaBraM, Table 4, Eq. 12 |
| TUEV, Base | balanced accuracy 0.6409, kappa 0.6637, weighted F1 0.8312 | LaBraM, Table 2 |
| TUAB, Base | balanced accuracy 0.8140, AUROC 0.9022 | LaBraM, Table 1 |
| CBraMod size and data | 4.0M parameters, 1,109,545 clean samples (over 9,000 hours) | CBraMod, Section 3.1 |
| CBraMod heads and cost | 4 spatial + 4 temporal; 318.9M FLOPs against 469.4M for full attention | CBraMod, Table 22 |
| Term | Plain meaning |
|---|---|
| Channel patch | One electrode for one second: 200 numbers. The unit every model here reads. |
| Spatial / temporal embedding | Learned tags for the electrode name and the time slot, added to each patch. |
| Neural tokenizer | A network that maps each patch to one of 8,192 codes, trained by rebuilding the spectrum. |
| Vector quantization | Replace a vector by the nearest entry of a fixed list of vectors. |
| Masked EEG modeling | Hide patches, then predict the tokenizer’s code for each hidden one. |
| Symmetric masking | Run the inverse mask on the same sample and add the two losses. |
| Criss-cross attention | Half the heads attend across channels in one slot, half across time in one channel. |
| ACPE | A position code made by a convolution over neighboring patches, not a lookup by electrode ID. |
| Linear probe | Train only a new output layer on frozen features. |
| Balanced accuracy, kappa | Average per-class recall; agreement beyond chance. |
This sketch is ours, not the authors’ code. It shows the data flow with shapes. The authors’ implementation is at github.com/935963004/LaBraM.
python import torch, torch.nn as nn, torch.nn.functional as F W = 200 # samples per patch: 1 s at 200 Hz def patchify(x): # x: (C, T), resampled to 200 Hz, scaled by 0.1 mV C, T = x.shape n = T // W return x[:, : n * W].reshape(C, n, W) # (C, n, 200) class PatchEncoder(nn.Module): # Table 3, Base: channels 8, kernels 15/3/3, strides 8/1/1 def __init__(self, co=8): super().__init__() blk = lambda i, o, k, s, p: nn.Sequential(nn.Conv1d(i, o, k, s, p), nn.GroupNorm(4, o), nn.GELU()) # 4 groups: our choice self.net = nn.Sequential(blk(1, co, 15, 8, 7), blk(co, co, 3, 1, 1), blk(co, co, 3, 1, 1)) def forward(self, p): # p: (N, 200) return self.net(p[:, None, :]).flatten(1) # (N, 8 * 25) = (N, 200) def spectrum_target(p): # p: (N, 200) patches of ONE sample X = torch.fft.rfft(p, dim=-1)[:, 1:101] # bins 1..100 Hz z = lambda v: (v - v.mean()) / v.std() # z-score within the sample return z(X.abs()), z(torch.angle(X)) # amplitude, phase: each (N, 100) def tokenize(p, codebook): # p: (N, 64), codebook: (8192, 64) sim = F.normalize(p, dim=-1) @ F.normalize(codebook, dim=-1).T # cosine, (N, 8192) return sim.argmax(-1) # (N,) ids in [0, 8191] def masked_loss(model, head, e, tags, ids, mask_token, r=0.5): # e: (N, d) patch vectors; tags: (N, d) = time tag + electrode tag; ids: (N,) from the frozen tokenizer N = e.shape[0] hide = torch.zeros(N, dtype=torch.bool) hide[torch.randperm(N)[: int(r * N)]] = True loss = 0.0 for m in (hide, ~hide): # symmetric masking: the mask, then its inverse x = torch.where(m[:, None], mask_token.expand_as(e), e) + tags logits = head(model(x)) # (N, 8192) loss = loss + F.cross_entropy(logits[m], ids[m]) # hidden patches only; mean, not the paper's sum return loss
The LaBraM authors list three limits. It is still far below the scale of vision and language models. It needs full fine-tuning, which is costly in compute and memory. And it uses only EEG, with no other signal. They point to adapters, prompt tuning and LoRA, and to pairing EEG with images, speech or text, as next steps.
CBraMod adds its own: crude data cleaning, higher cost than small task models, and no scaling study past 9,000 hours and 4M parameters. Both papers test on offline datasets. Neither shows a closed-loop device. Treat the scores as a step in representation learning, not as a ready product.
Credit. LaBraM: Wei-Bang Jiang, Li-Ming Zhao and Bao-Liang Lu (Shanghai Jiao Tong University), ICLR 2024, arXiv 2405.18765. CBraMod: Jiquan Wang, Sha Zhao, Zhiling Luo, Yangxuan Zhou, Haiteng Jiang, Shijian Li, Tao Li and Gang Pan (Zhejiang University and partners), ICLR 2025, arXiv 2412.07236. EEGPT: Wang et al., NeurIPS 2024, abstract page. All figures here come from these papers, except toy numbers marked as illustrative.
Now press Present or Teach and explain the pipeline from memory: how a 62 channel file becomes 248 patches, why the tokenizer rebuilds spectra, how the inverse mask reuses the codes, and why CBraMod splits its heads. Then go back to the cap picker at the top, choose 23 channels and 64 channels, and say why both fit one model.