Magenta RealTime 2

Chapter 1: Sound as Tokens — the Neural Audio Codec

A language model predicts the next token from a finite vocabulary — word, sub-word, whatever. But raw audio isn't tokens. A single second of CD-quality stereo is 96,000 floating-point numbers (48,000 samples per second, two channels). You cannot ask a Transformer to predict the next floating-point sample 96,000 times a second; the sequence would be astronomically long and the values continuous. We need to turn the river of audio into a manageable stream of discrete symbols. The device that does this is a neural audio codec.

Think of it as a learned MP3. A classical codec (MP3, AAC) compresses audio using hand-designed math. A neural codec learns the compression: an encoder network squashes a waveform into a short sequence of integer codes, and a decoder network expands those codes back into a waveform that sounds (almost) identical. The integers in the middle are exactly the "tokens" our language model will learn to predict. MRT2's codec is called SpectroStream.

Let's nail the dimensions precisely, because every later chapter leans on them. Write the input waveform as

Decode that notation one symbol at a time, because it's our whole foundation:

• a is the audio — bold because it's a big array of numbers, not a single value.
• ℝ (the real numbers) tells you each entry is a continuous amplitude, e.g. 0.0123.
• T is the clip's length in seconds.
• f_s is the audio sampling rate — how many amplitude samples we store per second. For SpectroStream, f_s = 48,000 (48 kHz).
• The × 2 is the two stereo channels (left, right).

So a 1-second clip is a 48,000 × 2 grid of real numbers. The encoder turns it into a much smaller grid of integers:

Again, symbol by symbol:

• x is the token grid the language model will actually see.
• V_c is the codec vocabulary — the set of integer codes a token may take. SpectroStream uses 2¹⁰ = 1024 possible codes per slot.
• f_k is the token frame rate: how many frames of tokens we emit per second. SpectroStream runs at f_k = 25 Hz. One frame therefore covers 1/25 s = 40 ms of sound. Remember that 40 ms number — it is the heartbeat of the entire model.
• d_c is the depth: how many tokens describe a single frame. SpectroStream uses d_c = 12. So one 40 ms slice of music isn't one token — it's a little stack of 12.

Why does one frame need twelve tokens instead of one? Because of how the codec quantizes: a scheme called residual vector quantization (RVQ). Here's the intuition, built from zero.

Suppose you must describe a precise colour using only a fixed palette of 1024 paint chips. Your first chip is the closest single match — but it's a bit off. So you look at the residual (the leftover difference between the true colour and your first chip) and pick a second chip that best captures that leftover. Then the leftover of the leftover, with a third chip. Each chip refines the last. Stack 12 such chips and you've described the colour far more exactly than any single one could. RVQ does exactly this with sound: token 1 is the coarse approximation of the 40 ms frame, token 2 corrects its error, token 3 corrects the remaining error, down to token 12, the finest detail. Coarse-to-fine, twelve levels deep.

Now the magic of the compression ratio. SpectroStream squeezes 48 kHz stereo down to about 3 kbps (kilobits per second). Let's sanity-check what that buys us. One second is 25 frames, each frame 12 tokens, so 25 × 12 = 300 tokens per second. Compare that to 96,000 raw samples per second: the codec has shrunk the thing the language model must predict by a factor of 320×. That shrinkage is what makes real-time prediction even thinkable.

And the round trip closes cleanly. At generation time the language model produces a fresh token grid x', and the codec's decoder turns it back into audio you can hear:

where Dec is the SpectroStream decoder. So the language model never touches a raw waveform — it lives entirely in the world of these 300-tokens-per-second integer grids, and the codec is the translator at both ends.

Chapter 2: Chunks vs Frames — the Unit of Autoregression

Now we get to the idea that is MRT2. Everything else is support structure for this one decision: what size piece of audio does the model commit to at a time?

An autoregressive model generates a sequence by repeatedly predicting "the next piece" and feeding it back in. The crucial design knob is how big "the next piece" is. Call that the unit of autoregression. And here's the iron law that ties it to our latency wall: the model cannot react to your input any faster than the unit it's currently committed to generating. If it's mid-way through producing a piece, your control change can only take effect at the start of the next piece. The unit size is the minimum reaction delay.

Original Magenta RealTime chose a big unit: a 2-second chunk. In codec terms that's 2 × 25 = 50 frames... actually v1 used a different codec at 16 RVQ levels, giving 25 frames × 16 = 400 tokens per 2-second chunk. The architecture matched: a T5-style model with a bidirectional encoder (which reads a whole chunk at once, looking both forward and backward inside it) feeding a causal decoder. Bidirectional attention is powerful — but it forces you to work a whole chunk at a time, because "looking forward" only makes sense if you have the whole window in hand. The consequence is brutal for latency: your control change can't land until the next 2-second boundary. Minimum reaction delay ≈ 2 seconds. That's the wall, baked into the architecture.

MRT2 shrinks the unit all the way down to a single frame — 12 tokens, 40 ms. Instead of committing to two seconds of the future, it commits to forty milliseconds, then reconsiders. To do that it had to drop the bidirectional encoder (you can't peek forward inside a 40 ms frame — there's nothing there yet) and become a decoder-only model with strictly causal attention: each token sees only the past. That's the trade. v1 bought richer within-chunk context at the cost of a 2-second reaction delay; MRT2 buys a 40 ms reaction granularity at the cost of never peeking ahead. For a live instrument, that's exactly the right trade.

Read the gap between the two reaction markers in that widget. The bottom one (MRT2) is never more than 40 ms away from where you acted. The top one (v1) can be nearly 2 full seconds away. That gap is the 15× improvement, made visible.

But shrinking the unit creates two new problems, and the rest of the lesson is largely about solving them. First: predicting a frame still means predicting 12 tokens, and doing that naively, frame after frame, 25 times a second, could be too slow — so we need a cheap way to roll out 12 tokens (Chapter 3). Second: a causal decoder that runs forever will accumulate an ever-growing history; attention cost grows with history length, and unbounded growth kills real-time performance — so we need to attend to the past without paying for all of it (Chapter 4). Hold those two questions; we answer them next.

Chapter 3: Two Transformers — Temporal × Depth

We just committed to generating one 40 ms frame at a time. But a frame is 12 RVQ tokens, and those 12 are not independent — remember the paint-chip ladder: token 5 only makes sense as a correction to tokens 1–4. So within a frame the tokens are ordered, coarse-to-fine, and each depends on the ones above it. Meanwhile, across frames, frame i depends on the whole musical past, frames 1…i−1. We have dependency in two directions: along time (frame to frame) and along depth (token to token within a frame).

The naive fix — flatten everything into one long sequence of 300 tokens/second and run one giant Transformer — works but is expensive: the model would re-attend over the entire flattened history for every one of the 12 tokens in every frame. MRT2 instead splits the labour between two Transformers, a design borrowed from the "RQ-Transformer" family:

The beautiful efficiency: the costly "attend over all of history" work happens once per frame in the temporal Transformer, and the 12 within-frame decisions are made by a small depth Transformer that never looks back across time at all — it just unrolls the ladder for the one frame it was handed. You pay for history once, then cheaply expand it into 12 tokens.

Let's write the probability the two networks jointly model. We want the probability of the whole token grid x. It factorises along both axes — frames on the outside, depth on the inside:

This looks dense; it is actually a precise statement of the two paragraphs above. Walk it left to right:

• ∏_i=1^T·f_k — multiply over every frame i, from 1 to "seconds × 25." This is the time axis.
• ∏_j=1^d_c — and within each frame, multiply over every depth level j, from 1 (coarsest) to 12 (finest). This is the depth axis.
• x_i^j — the token we're predicting: depth-level j of frame i.
• x_i^<j — the coarser tokens already chosen in this same frame (levels 1…j−1). This is the depth Transformer reading the ladder above it.
• Temporal_θ(x_<i) — the context vector the temporal Transformer produced by compressing all earlier frames. This is "the past," delivered as one summary.
• P_φ(·) — the depth Transformer turns those two inputs into a probability over the 1024 possible codes, via a softmax: P_φ = SoftMax(Depth_φ(·)).

So to generate frame i: the temporal Transformer fires once to make Temporalθ(x<i); then the depth Transformer fires 12 times, each time conditioning on that vector plus the tokens it has already laid down this frame, until all 12 levels are filled. Decode those 12 codes with SpectroStream and you've got 40 ms of fresh stereo audio. Repeat 25 times for a second of music.

Chapter 4: Streaming Forever — Sliding Windows & Attention Sinks

MRT2 is meant to play indefinitely — a jam can last an hour. But a causal Transformer's attention cost grows with the length of the history it attends to. If frame 90,000 (an hour in) must attend over all 89,999 prior frames, the cost balloons and real-time collapses. We need to bound the history the model looks at, without the music falling apart. That bound is sliding window attention (SWA).

The idea is simple: each frame attends only to the most recent w frames — a fixed-size window that slides forward as time passes. Older frames fall out of view. The attention cost per frame becomes constant (it depends on w, not on how long you've been playing), so the model can stream forever at steady speed. The implementation is a rolling cache (a "KV cache" of the keys and values each past token contributes): when a new frame arrives, its entries are written in, and entries older than w are evicted.

But there's a notorious failure mode lurking here, and MRT2 hits it head-on. When the very first tokens of a stream finally slide out of the window and get evicted, models tend to deteriorate sharply — the audio degrades, sometimes collapses into noise. Why would dropping ancient, long-irrelevant tokens hurt? The answer is one of the more surprising results in recent Transformer research: those first tokens were acting as attention sinks.

Here's the intuition. The softmax inside attention forces the attention weights to sum to 1 — every query must distribute its attention somewhere, even when it has nothing important to attend to. In practice models learn to dump that excess attention onto the first few tokens of the sequence, using them as a kind of "null" or pressure-release valve. Those early tokens become load-bearing not for their content but for their role as a sink. Evict them, and suddenly every query's leftover attention has nowhere stable to go — the distribution destabilises and quality craters.

MRT2's fix is elegant: add a learnable attention-sink embedding — a small set of permanent, trained "tokens" pinned at the start of the window that never get evicted. They give every query a stable place to park excess attention, so the model stays healthy even as real musical history scrolls past and falls out of the window. The sink stays; the music flows through.

There's one more subtlety MRT2 handles, about position. A Transformer needs some way to know token order. v1 used learned positional embeddings; but those don't generalise past the length you trained on — fatal for a model meant to run arbitrarily long. The team first tried RoPE (rotary position encoding), then ultimately leaned on a striking finding: with causal masking and a sliding window, a decoder can infer order implicitly — the so-called NoPE ("no positional encoding") approach. Because each token can already tell "how far back" something is from the structure of the causal mask itself, you can drop explicit position signals entirely and gain clean generalisation to any sequence length. One careful constraint ties it together: the window w is sized so the model's effective reach never exceeds the crop length it was trained on — so it's never asked to extrapolate beyond what it has seen.

Chapter 5: Steering the Stream — Frame-Rate Conditioning

So far we can generate endless, coherent music — but a fixed jam isn't an instrument. The point of MRT2 is that you steer it, live. This chapter is the spine of all control: how an external signal gets injected into the generator fast enough to matter.

Formally, we move from modelling P(x) to the conditional distribution P_θ,φ(x | c), where c is everything you're telling the model to do. MRT2 bundles three control streams:

• c_style — the vibe: a text prompt ("dreamy lo-fi") or a reference audio clip. (Chapter 7.)
• c_notes — the pitches: which MIDI notes are sounding, from a keyboard or sequencer. (Chapter 6.)
• c_drums — a drums on/off control. (Chapter 8.)

The single most important design decision about control is the one that's easy to skim past: every conditioning signal is tokenised at the audio frame rate — 25 Hz, the very same 40 ms grid as the audio tokens. The notes you're holding, the style embedding, the drum flag — all are sampled into one value (or vector) per frame. Why does that matter so much? Because it means a control change can be absorbed by the next frame the model generates. If conditioning lived on a slower clock — say, once per second — then no matter how fast the generator ran, your input could wait up to a second to be seen. Frame-rate conditioning is what lets the ~40 ms generation granularity from Chapter 2 actually translate into ~40 ms responsiveness.

Mechanically: the per-frame control values are concatenated into one conditioning vector for that frame, mapped into multi-channel embeddings, and injected into the temporal decoder through streaming cross-attention. "Cross-attention" means the generator's frame query attends to the conditioning embeddings (as opposed to self-attention over its own past). "Streaming" means it happens frame-by-frame as new control arrives, not as a one-shot prompt at the start. The upshot: the model can register and react to your change within a single 40 ms frame.

Chapter 6: Playing Notes — the Pianoroll & Auto-Strum

The notes lane, c_notes, is what lets you actually play MRT2 like an instrument: hold a chord on a MIDI keyboard and the model renders that chord in its current style. To make this work we need a representation of "which notes are sounding right now" that lives on the 40 ms frame grid. That representation is a pianoroll.

Picture a sheet of graph paper. The vertical axis has one row per MIDI pitch — 128 channels, covering every note from the lowest rumble to the highest ping. The horizontal axis is time, one column per frame (25 columns per second). A cell is "lit" when its pitch is sounding in that frame. That grid — 128 rows, frame-rate columns — is the pianoroll, and it's exactly c_notes.

Where do the pianorolls to train on come from? MRT2 was trained on roughly 71,000 hours of instrumental stock music — but that music came as audio, not MIDI. So the team ran a transcription model called MT3 over all of it to infer the notes, producing a pianoroll label for each audio clip. The model thus learns the mapping "these pitches, in this style → this audio" from 71k hours of automatically-transcribed examples.

Now the subtle part: a row being "lit" isn't enough information. There's a difference between striking a note (a fresh attack, the hammer hitting the string) and holding one that's already ringing. A held chord and a re-struck chord sound very different. So each pitch-row, each frame, carries one of four tokens:

This four-token vocabulary unlocks the model's signature note feature: Auto-Strum. It comes in two modes, and the difference is entirely about who decides the attacks:

• Auto-Strum ON — you specify only which pitches are active (the "generic note on" tokens). The model decides when each note attacks — it strums, arpeggiates, re-voices on its own, musically. You hold the harmony; it performs it.
• Auto-Strum OFF — you specify exact onsets (the "note onset" tokens). The model attacks precisely when you say. You get note-level rhythmic control; the model fills in timbre and style.

How does the model become good at Auto-Strum's "you decide the attack" behaviour? Through a training trick called onset-masking augmentation. During training, the system stochastically replaces some onset/continuation tokens with a special onset-mask token — effectively hiding when notes were struck. Forced to generate plausible audio anyway, the model learns to invent musically plausible attacks when onset info is absent. At inference, feed it masked onsets and it strums tastefully; feed it explicit onsets and it obeys them. One trained behaviour, two playing styles.

Chapter 7: The Vibe — MusicCoCa & Mean Flow

The style lane, c_style, answers "what should it sound like?" — the genre, mood, instrumentation. The tricky part is that you might specify style two very different ways: by text ("warm analog synthwave") or by an audio reference (a clip you want it to match). The model needs a single internal notion of "style" that both can map into. That shared space comes from MusicCoCa.

MusicCoCa is a model trained to embed music and text describing music into the same vector space — a "Contrastive Captioning" model for audio, in the CLIP family. Train it so that a clip of bossa nova and the words "relaxed bossa nova guitar" land near each other, and you get a space where proximity means stylistic similarity regardless of whether the point came from audio or text. MRT2 uses MusicCoCa's quantised (RVQ) style embeddings as c_style, and — importantly — keeps MusicCoCa frozen (its weights are not trained further inside MRT2).

But freezing exposes a real problem — the train/test modality mismatch. During training, style usually comes from audio (you have the clip, you embed it). At inference, a user often gives text. Text and audio, even in a shared space, don't land in exactly the same places: a single text prompt like "lo-fi piano" corresponds to many valid audio renditions, scattered across a region of the space — a one-to-many relationship. Feed the generator a lone text point and it may sit in a spot that's slightly "off-distribution" relative to the audio embeddings it trained on.

MRT2's fix is a small bridge network: a Pixel Mean Flow (pMF) mapper. Its job is to take a text embedding and map it into the distribution of plausible audio embeddings — learning that one-to-many relationship explicitly. "Mean Flow" is a generative technique (a cousin of flow-matching / diffusion) that here is tuned for high-quality one-step inference: instead of many denoising steps, it jumps from the text point to a valid audio-style point in a single shot — essential when you have a ~40 ms budget per frame. It's trained on a mix of short tags ("techno") and long-form captions ("driving four-on-the-floor techno with a detuned acid bassline"), so it handles both terse and verbose prompts.

Chapter 8: The Mixer — Multi-Signal Guidance & Creative Masking

We now have three controls riding the conditioning bus: style, notes, and the third one we've deferred — c_drums. This chapter does two things: explains the drums control, then shows how MRT2 lets you balance all three like channels on a mixing desk, and even leave channels deliberately blank for creative effect.

First, drums. You might expect a drum control to work like the pianoroll — specify exact hits. But there's a catch: hitting a drum is an instantaneous event, and the model's end-to-end reaction time (the ~200 ms we keep meeting) is too slow to place individual kicks and snares with sample-accurate tightness. So MRT2 uses drums differently: it transcribes drum stems from training audio with a model called OaF Drums ("Onsets and Frames" for drums), and learns a frame-wise drum-presence signal. At inference, c_drums is used mainly as a switch: generate with drums, or generate a "drumless" texture — not as a step sequencer for individual hits. It's a coarse but musically useful control: "drop the drums for this section, bring them back for the chorus."

How does a single model both honor your controls and let you dial their strength — or switch one off entirely? The answer is classifier-free guidance (CFG), extended to multiple signals.

CFG, built from zero: train the model to generate both with a condition and without it (by sometimes dropping the condition during training). At generation time you have two predictions — the conditional one (what it'd do given your control) and the unconditional one (what it'd do freely). You then extrapolate: push the output in the direction that the condition pulls it, by an adjustable amount. A guidance weight near 1 means "follow the control about as much as training suggested"; higher means "exaggerate adherence to the control"; lower (toward 0) means "barely heed it." MRT2's contribution is doing this independently for each of the three signals at once — so you can crank note-adherence while loosening style, or vice versa. Three guidance knobs, one stream.

That multi-signal CFG is also how the drums switch works: "drumless" is just running the drum signal's guidance toward its unconditional/negative direction — steering away from drums rather than toward them. Same machinery, used as an on/off.

There's one more control idea, and it's the most creative: masking as a paintbrush. Recall that any conditioning signal can be partially specified. During training, MRT2 stochastically masks contiguous regions of each control signal — sometimes a stretch of time, sometimes a band of pitches — varying both how much and where. This teaches the model to follow controls where given and improvise where they're blank, staying robust to noisy or missing input. At inference, that robustness becomes expressive power:

• Mask the pitch dimension — leave most of the pianoroll blank, holding only the notes you're pressing — and the model is free to add its own harmonies and embellishments around your notes. You play a melody; it voices the chords.
• Constrain to only your notes — specify the full pianoroll tightly — and it plays only what you gave it.
• Partial conditioning in time — condition the next bar but leave the one after blank — and it improvises a continuation in your style.

Masking turns "missing input" from a bug into a dial for how much creative freedom you grant the model. Full control at one extreme; open-ended jamming at the other; anything between.

Chapter 9: The Budget — Adding Up 200 Milliseconds

We've built every piece. Now the showcase: let's account for the 200 ms. Where does it go? Because here's a puzzle — the model generates a 40 ms frame at a time, so why is the end-to-end reaction ~200 ms and not ~40 ms? The answer is that "reaction time" is a chain, and the model's own step is only one link. Roughly five frame-durations' worth of overhead stack up from real-world buffering and decoding. Let's enumerate every link.

Add those up and you land near 200 ms — about 5× the 40 ms frame. The model's compute (links 2–4) is a minority of the budget; the buffering at the edges (links 1, 5, 6) is the real tax, and it's the price of running glitch-free on a general-purpose laptop inside real audio software. The widget lets you push each link and see whether you stay under the "playable" line.

And the reason the compute link can stay so small on a laptop is the deployment stack. MRT2 ships as a C++ inference engine built on MLX (Apple's array framework for Apple Silicon), which compiles a model into a .mlxfn container that DAW plugins and standalone apps can load. The 2.4B-parameter base model streams in real time on an M3 Pro / M2 Max; a smaller 230M variant runs on any MacBook Air. (For contrast, v1 needed a TPU/GPU in the cloud.) The architecture made 40 ms frames possible; MLX made them fast enough on hardware you own.

Chapter 10: Connections & Cheat Sheet

You can now look at the Magenta RealTime 2 write-up and explain every line: why frames instead of chunks, why two Transformers, why attention sinks, why MusicCoCa is frozen, why the latency is 200 and not 40. Let's lock it in.

The one-paragraph summary

MRT2 turns 48 kHz stereo into 300 codec tokens/second via SpectroStream's 12-deep RVQ. A decoder-only model generates one 40 ms frame at a time — a Temporal Transformer summarises the windowed past once per frame, a small Depth Transformer unrolls the frame's 12 tokens — using sliding-window attention with a learnable attention sink (and NoPE) so it streams forever at constant cost. Three controls (style via frozen MusicCoCa + a one-step Mean-Flow mapper; notes via a 128-channel four-token pianoroll with Auto-Strum; drums on/off) ride a 25 Hz conditioning bus, each independently weighted by classifier-free guidance and maskable for creative freedom. The whole thing runs on Apple Silicon via an MLX C++ engine, reacting in ~200 ms — fast enough to jam.

Cheat sheet — the numbers that matter

The architecture, top to bottom

# MRT2 generation loop (one frame), in pseudo-Python
def generate_frame(history, control, guidance, masks):
    # 1. CONTROL: sample each signal at this frame (25 Hz bus)
    c_style  = mean_flow(musiccoca_text(control.prompt))  # frozen encoder + 1-step mapper
    c_notes  = pianoroll_tokens(control.midi, auto_strum=control.strum)  # 128ch x 4-vocab
    c_drums  = control.drums_on
    c = concat(c_style, c_notes, c_drums)
    c = apply_masks(c, masks)                 # blank out regions for creative freedom

    # 2. TEMPORAL: summarise the windowed past ONCE (SWA + sink + NoPE)
    ctx = temporal_theta(history[-W:], sink=learned_sink)   # constant cost in W, not len(history)

    # 3. DEPTH: unroll 12 RVQ tokens, coarse -> fine, each sees the coarser ones
    frame = []
    for j in range(12):
        logits = depth_phi(frame, ctx, cross_attend=c)       # multi-signal CFG applied here
        logits = cfg_mix(logits, guidance)                   # independent weight per signal
        frame.append(sample(softmax(logits)))

    # 4. CODEC: 12 codes -> 40 ms of 48kHz stereo audio
    return spectrostream_decode(frame)          # a' = Dec(x')

Where this sits in the field

• Lineage. The codec-language-model recipe (audio → RVQ tokens → predict tokens) traces to SoundStream/EnCodec + AudioLM + MusicLM. The two-Transformer temporal×depth split is the RQ-Transformer idea. MRT2's contribution is bending all of it toward real-time, controllable, on-device use.
• Attention sinks & NoPE. Borrowed from the streaming-LLM literature — the same tricks that let a chat model run on an endless conversation let MRT2 run an endless jam.
• Classifier-free guidance & Mean Flow. Straight from the diffusion world, repurposed for tokens and for one-step style mapping under a real-time budget.
• What's next (per the authors). Fine-tuning on your own datasets, and performance tools built with musician Manaswi Mishra — turning the model from a fixed instrument into one you can train and perform.

Related lessons on Engineermaxxing

• The Transformer and attention fundamentals — the substrate under both Transformers here.
• Diffusion & Flow Matching — the family Mean Flow and CFG come from.
• Tokenizers & VQ-VAE — the quantization idea behind RVQ codecs.
• RoPE / positional encodings — context for why NoPE is surprising and useful.

Now press Present or Teach Mode and explain the 200 ms budget back — out loud, from memory. If you can account for every link, you own this model.