Magenta RealTime 2

Sit down at a piano with a friend. You play a chord; they answer with a bassline; you nudge the rhythm; they follow. It works because the gap between your move and their response is tiny, a few tens of milliseconds. You don't experience a conversation. You experience playing together.

Now give that friend a two-second delay. You hit a chord; two seconds of silence; then their reply, for the chord you've already moved on from. That is the first partner on the pads above. The magic is gone, not because the friend is unmusical, but because of latency: the delay between an input and the system's reaction to it.

This is the wall every "AI music generator" has hit. Most models are offline: you type a prompt, wait, and a finished clip drops out. Wonderful for producing a track; useless for playing one. The original Magenta RealTime (MRT, or v1) was a landmark because it generated music continuously and let you steer it as it played. But its reactions lagged by around two seconds. You could conduct it, the way you'd conduct an orchestra over a slow phone line. You could not jam with it.

Magenta RealTime 2 (MRT2) cuts the reaction time from roughly two seconds to about 200 milliseconds, a ~15× improvement, while running on a consumer laptop. Two hundred milliseconds is where steering stops feeling like remote control and starts feeling like play. That one number, and everything the architecture had to become to hit it, is what this lesson is about.

Let's measure the wall on the beat grid. At 120 beats per minute each beat is 500 ms apart. You strike on beat 1; the partner answers after whatever latency you dial in.

So the design goal of MRT2, in one sentence: generate high-fidelity stereo music continuously, let a musician change MIDI notes, audio style, and drums at any instant, and react within ~200 ms, all on an Apple-Silicon MacBook. Every architectural choice ahead is downstream of that sentence. Whenever something looks strange ("why not just use a normal Transformer?"), the answer will almost always be: because of the wall.

Here is the road, in order:

1. Sound as tokens: turn audio into discrete symbols a language model can predict.
2. Chunks vs frames: predict one 40 ms frame at a time instead of a two-second chunk. This is the central trick.
3. Two transformers: make each frame cheap to predict.
4. Streaming forever: attend to an unbounded past without running out of memory.
5. Steering, notes, the vibe, the mixer: the controls, and how they share one fast clock.
6. The budget: add up every millisecond of the 200.

A language model predicts the next token from a finite vocabulary. Raw audio isn't tokens. One second of CD-quality stereo is 96,000 floating-point numbers: 48,000 samples per second, two channels. You cannot ask a Transformer to predict the next floating-point sample 96,000 times a second; the sequence would be astronomically long and the values continuous. We need to turn the river of audio into a manageable stream of discrete symbols. The device that does this is a neural audio codec.

Think of it as a learned MP3. A classical codec compresses with hand-designed math. A neural codec learns the compression: an encoder network squashes a waveform into a short sequence of integer codes, and a decoder network expands those codes back into a waveform that sounds almost identical. The integers in the middle are exactly the tokens our language model will predict. MRT2's codec is called SpectroStream.

Every later chapter leans on the dimensions, so let's pin them down. Hover or tap any symbol to see what it means. The input waveform is:

a

The audio. Bold because it's a big array of numbers, not one value.

ℝ

The real numbers: each entry is a continuous amplitude, like 0.0123.

T

The clip's length in seconds.

f_s

The sampling rate, amplitude samples stored per second. For SpectroStream, 48,000 (48 kHz).

× 2

The two stereo channels, left and right.

So a one-second clip is a 48,000 × 2 grid of real numbers. The encoder turns it into a much smaller grid of integers:

x

The token grid the language model actually sees.

V_c

The codec vocabulary, the integer codes a token may take: 2¹⁰ = 1,024 per slot.

f_k

The token frame rate: 25 frames per second, so one frame covers 40 ms. That number is the heartbeat of the whole model.

d_c

The depth: how many tokens describe a single frame. SpectroStream uses 12.

Why does one frame need twelve tokens instead of one? Because of how the codec quantizes, a scheme called residual vector quantization (RVQ). Suppose you must describe a precise colour using a fixed palette of 1,024 paint chips. Your first chip is the closest single match, but it's a bit off. So you look at the residual, the leftover difference, and pick a second chip that best captures that. Then the leftover of the leftover. Stack 12 chips and you've described the colour far more exactly than any single one could. RVQ does this with sound: token 1 is the coarse approximation of the 40 ms frame, token 2 corrects its error, and so on down to token 12, the finest detail. Coarse to fine, twelve levels deep.

You can hear that ladder. The device below quantizes a short chord one rung at a time.

Now the compression ratio. SpectroStream squeezes 48 kHz stereo to about 3 kbps. One second is 25 frames of 12 tokens: 300 tokens per second. Against 96,000 raw samples, the codec has shrunk what the language model must predict by 320×. That shrinkage is what makes real-time prediction thinkable.

1 second → 25 frames × 12 tokens = 300 tokens (vs 96,000 samples)

The round trip closes cleanly. At generation time the language model produces a fresh token grid x′, and the decoder turns it back into audio: a′ = Dec(x′). The language model never touches a waveform. It lives entirely on these integer grids, and the codec is the translator at both ends.

This is the idea that is MRT2. Everything else supports one decision: what size piece of audio does the model commit to at a time?

An autoregressive model generates a sequence by repeatedly predicting the next piece and feeding it back in. The crucial knob is how big that piece is: the unit of autoregression. And here is the iron law that ties it to the wall: the model cannot react to your input faster than the unit it's committed to generating. If it's midway through a piece, your change can only take effect at the start of the next one. The unit size is the minimum reaction delay.

The original Magenta RealTime chose a big unit: a 2-second chunk, generated by a T5-style encoder-decoder. The encoder reads its context with bidirectional attention, looking forward and backward across the window. That's powerful, but "looking forward" only makes sense when you hold a whole window, so the model works a chunk at a time. The consequence for latency is brutal: your control change can't land until the next 2-second boundary. Minimum reaction delay, about 2 seconds. The wall, baked into the architecture.

MRT2 shrinks the unit to a single frame: 12 tokens, 40 ms. Instead of committing to two seconds of the future, it commits to forty milliseconds, then reconsiders. To do that it drops the bidirectional encoder (there's nothing ahead to peek at inside a frame that doesn't exist yet) and becomes a decoder-only model with strictly causal attention: each token sees only the past. That's the trade. v1 bought richer context at the cost of a 2-second reaction; MRT2 buys 40 ms granularity at the cost of never peeking ahead. For a live instrument, that's the right trade.

Read the gap between the two reaction markers. MRT2's is never more than 40 ms after you act. v1's can be nearly two full seconds. That gap is the 15× improvement, made audible.

Shrinking the unit creates two new problems, and much of the rest of the lesson solves them. First: a frame is still 12 tokens, and rolling those out naively 25 times a second could be too slow, so we need a cheap way to produce them (Chapter 3). Second: a causal decoder that runs forever accumulates an ever-growing history, and attention cost grows with it, so we need to attend to the past without paying for all of it (Chapter 4).

We just committed to generating one 40 ms frame at a time. But a frame is 12 RVQ tokens, and they aren't independent: token 5 only makes sense as a correction to tokens 1 to 4. Meanwhile frame i depends on the whole musical past. We have dependency in two directions: along time, frame to frame, and along depth, token to token within a frame.

The naive fix is to flatten everything into one sequence of 300 tokens per second and run one giant Transformer. It works, but it re-attends over the entire flattened history for every one of the 12 tokens in every frame. MRT2 splits the labour between two Transformers, a design from the RQ-Transformer family:

You pay for history once per frame, then cheaply expand it into 12 tokens. Here is the probability the two networks model together. It factorises along both axes, frames outside, depth inside:

P_θ,φ(x)

The probability of the whole token grid, under both networks' weights.

∏_i

Multiply over every frame, 1 to seconds × 25. The time axis.

∏_j

Inside each frame, multiply over every depth level, 1 (coarsest) to 12 (finest).

x_i^j

The token being predicted: level j of frame i.

x_i^<j

The coarser tokens already chosen in this same frame. The depth network reading the ladder above it.

Temporal_θ(x_<i)

The context vector the temporal network made from all earlier frames. "The past," as one summary.

P_φ

The depth network's softmax over the 1,024 possible codes: SoftMax(Depth_φ(·)).

So to generate frame i, the temporal network fires once; then the depth network fires 12 times, each time conditioned on that vector plus the tokens it has laid down this frame. Decode those 12 codes with SpectroStream and you have 40 ms of fresh stereo. Repeat 25 times for a second.

MRT2 is meant to play indefinitely; a jam can last an hour. But a causal Transformer's attention cost grows with the history it attends to. If frame 90,000 (an hour in) had to attend over all 89,999 before it, real time would collapse. We need to bound the history without the music falling apart. That bound is sliding window attention (SWA).

Each frame attends only to the most recent w frames, a fixed-size window that slides forward. Older frames fall out of view. The cost per frame becomes constant (it depends on w, not on how long you've played), so the model can stream forever at steady speed. The implementation is a rolling cache of the keys and values each past token contributes (a "KV cache"): new entries are written in, entries older than w are evicted.

There's a notorious failure lurking, and MRT2 meets it head on. When the very first tokens of a stream slide out of the window, models tend to deteriorate sharply. Why would dropping ancient, long-irrelevant tokens hurt? Because those first tokens were acting as attention sinks.

The softmax inside attention forces the weights to sum to 1. Every query must put its attention somewhere, even when nothing is important. Models learn to dump that excess onto the first few tokens, a pressure-release valve. Those tokens become load-bearing for their role, not their content. Evict them, and every query's leftover attention has nowhere stable to go; quality craters.

MRT2's fix: a learnable attention-sink embedding, a few permanent trained "tokens" pinned at the start of the window that are never evicted. Every query gets a stable place to park its excess attention, while real musical history scrolls past. The sink stays; the music flows through.

One more subtlety: position. A Transformer needs some way to know token order. v1 used learned positional embeddings, which don't generalise past the length you trained on; fatal for a model meant to run arbitrarily long. The team tried RoPE (rotary position encoding), then leaned on a striking finding: with causal masking and a sliding window, a decoder can infer order implicitly, the NoPE ("no positional encoding") approach. The causal mask itself tells each token how far back things are, so explicit position signals can go, and the model generalises to any length. One constraint ties it together: the window is sized so the model's reach never exceeds the crop length it trained on, so it's never asked to extrapolate.

We can now generate endless, coherent music. But a fixed jam isn't an instrument. The point is that you steer it, live. This chapter is the spine of every control: how an outside signal gets into the generator fast enough to matter.

Formally we move from modelling P(x) to the conditional distribution P_θ,φ(x | c), where c is everything you're telling the model. MRT2 bundles three control streams:

c_style

The vibe: a text prompt ("dreamy lo-fi") or a reference audio clip. Chapter 7.

c_notes

The pitches: which MIDI notes are sounding, from a keyboard or sequencer. Chapter 6.

c_drums

A drums on or off control. Chapter 8.

The most important decision about control is easy to skim past: every conditioning signal is tokenised at the audio frame rate, 25 Hz, the same 40 ms grid as the audio tokens. The notes you hold, the style embedding, the drum flag: each is sampled into one value per frame. That means a change can be absorbed by the next frame the model generates. If conditioning lived on a slower clock, say once per second, your input could wait up to a second to be seen, however fast the generator ran. Frame-rate conditioning is what turns Chapter 2's 40 ms granularity into 40 ms responsiveness.

Mechanically, the per-frame control values are concatenated into one conditioning vector for that frame, mapped into multi-channel embeddings, and injected into the temporal decoder through streaming cross-attention. Cross-attention means the generator's frame query attends to the conditioning embeddings, rather than to its own past. Streaming means it happens frame by frame as new control arrives, not once as a prompt at the start.

The notes lane, c_notes, is what lets you play MRT2 like an instrument: hold a chord on a MIDI keyboard and the model renders it in the current style. It needs a representation of "which notes are sounding right now" that lives on the 40 ms grid. That representation is a pianoroll.

Picture graph paper. The vertical axis has one row per MIDI pitch, 128 channels, from the lowest rumble to the highest ping. The horizontal axis is time, one column per frame. A cell is lit when its pitch sounds in that frame. That grid is c_notes.

Where do training pianorolls come from? MRT2 was trained on roughly 71,000 hours of instrumental stock music, but as audio, not MIDI. So the team ran a transcription model, MT3, over all of it to infer the notes. The model learns "these pitches, in this style, make this audio" from 71k hours of automatically transcribed examples.

A lit cell isn't enough information, though. Striking a note and holding one that's already ringing sound very different. So each pitch row, each frame, carries one of four tokens:

• Note offThis pitch is silent in this frame.
• Generic note onSounding, attack timing unspecified: "it's on, you decide the articulation."
• Note onsetA fresh attack starts in this frame: strike it now.
• Note continuationAlready sounding; let it ring, don't re-strike.

This vocabulary unlocks the signature note feature, Auto-Strum. Its two modes differ in who decides the attacks:

• Auto-Strum on: you specify only which pitches are active (generic note on). The model decides when each note attacks: it strums, arpeggiates, re-voices. You hold the harmony; it performs it.
• Auto-Strum off: you specify exact onsets. The model attacks precisely when you say; you get note-level rhythm, it fills in timbre and style.

The model learns the first mode through onset-masking augmentation. During training, some onset and continuation tokens are randomly replaced with a special onset-mask token, hiding when notes were struck. Forced to produce plausible audio anyway, the model learns to invent musical attacks when onset information is missing. At inference, feed it masked onsets and it strums; feed it explicit onsets and it obeys. One trained behaviour, two playing styles.

The style lane, c_style, answers "what should it sound like?": genre, mood, instrumentation. You might specify style by text ("warm analog synthwave") or by an audio reference. The model needs one internal notion of style that both can map into. That shared space comes from MusicCoCa.

MusicCoCa embeds music and text about music into the same vector space, a contrastive captioning model for audio in the CLIP family. Train it so a clip of bossa nova and the words "relaxed bossa nova guitar" land near each other, and proximity means stylistic similarity whether a point came from audio or text. MRT2 uses MusicCoCa's quantised (RVQ) style embeddings as c_style and keeps MusicCoCa frozen: its weights are not trained further inside MRT2.

Freezing exposes a real problem: the train/test modality mismatch. During training, style usually comes from audio. At inference, users often give text. Even in a shared space, text and audio don't land in the same places: one prompt like "lo-fi piano" corresponds to many valid audio renditions, scattered across a region. Feed the generator a lone text point and it may sit slightly off the distribution of audio embeddings it trained on.

MRT2's fix is a small bridge network, a Pixel Mean Flow (pMF) mapper. It takes a text embedding and maps it into the distribution of plausible audio embeddings, learning that one-to-many relationship explicitly. Mean Flow is a generative technique, a cousin of flow matching and diffusion, tuned here for high-quality one-step inference: it jumps from the text point to a valid audio-style point in a single shot, essential inside a 40 ms budget. It's trained on short tags ("techno") and long captions ("driving four-on-the-floor techno with a detuned acid bassline") alike.

Three controls ride the bus: style, notes, and the one we deferred, c_drums. This chapter explains drums, then shows how MRT2 lets you balance all three like channels on a desk, and leave channels deliberately blank.

You might expect drums to work like the pianoroll, with exact hits. But a drum hit is instantaneous, and the end-to-end reaction (the ~200 ms we keep meeting) is too slow to place individual kicks and snares tightly. So MRT2 transcribes drum stems from training audio with OaF Drums (Onsets and Frames for drums) and learns a frame-wise drum-presence signal. At inference c_drums is mainly a switch: generate with drums, or a drumless texture. "Drop the drums for this section, bring them back for the chorus."

How does one model both honour your controls and let you dial their strength, or switch one off? Classifier-free guidance (CFG), extended to several signals. Train the model to generate both with a condition and without it, by sometimes dropping the condition in training. At generation time you have two predictions, conditional and unconditional, and you extrapolate: push the output in the direction the condition pulls it, by an adjustable amount. A weight near 1 follows the control about as much as training did; higher exaggerates adherence; toward 0, it barely heeds it. MRT2 does this independently for each of the three signals, so you can crank note adherence while loosening style. Three guidance knobs, one stream.

That machinery is also how the drums switch works: drumless means running the drum signal's guidance toward its negative direction, steering away from drums rather than toward them.

That seg control is the last idea, and the most creative: masking as a paintbrush. During training, MRT2 randomly masks contiguous regions of each control signal, sometimes a stretch of time, sometimes a band of pitches. The model learns to follow controls where they're given and improvise where they're blank. At inference that robustness becomes expressive:

• Mask the pitch dimension, holding only the notes you press, and the model is free to add its own harmonies around them. You play a melody; it voices the chords.
• Constrain to only your notes, specifying the pianoroll tightly, and it plays only what you gave it.
• Condition partially in time, the next bar but not the one after, and it improvises a continuation in your style.

Masking turns missing input from a bug into a dial for how much freedom you grant the model: full control at one end, open-ended jamming at the other.

We've built every piece. Now let's account for the 200 ms. Here's the puzzle: the model generates a 40 ms frame at a time, so why is the end-to-end reaction ~200 ms and not ~40? Because reaction time is a chain, and the model's own step is only one link. Follow a single keypress through it:

1. Input buffering. Your MIDI or audio doesn't arrive instantly; the OS and plugin collect it into a small buffer first.
2. Temporal step (θ). The temporal Transformer attends over the window and makes the frame's context vector.
3. Depth decode (φ). The depth Transformer unrolls the 12 RVQ tokens.
4. Codec decode. SpectroStream turns 12 codes back into 40 ms of stereo waveform.
5. Output buffering. The audio driver keeps samples queued ahead to avoid glitches.
6. Host overhead. The DAW or app passing data around adds its own slice.

Add them up and you land near 200 ms, about 5× the frame. The model's compute (the three warm links) is the minority. The buffering at the edges is the real tax, the price of running glitch-free on a general-purpose laptop inside real audio software.

The compute link stays small on a laptop because of the deployment stack. MRT2 ships as a C++ inference engine built on MLX, Apple's array framework for Apple Silicon, which compiles the model into a .mlxfn container that DAW plugins and standalone apps can load. The 2.4B-parameter base model streams in real time on an M3 Pro or M2 Max; a smaller 230M variant runs on any MacBook Air. (v1 needed a TPU or GPU in the cloud.) The architecture made 40 ms frames possible; MLX made them fast enough on hardware you own.

You can now read the Magenta RealTime 2 write-up and explain every line: why frames instead of chunks, why two Transformers, why attention sinks, why MusicCoCa is frozen, why the latency is 200 and not 40. Let's lock it in.

The one-paragraph summary

MRT2 turns 48 kHz stereo into 300 codec tokens per second with SpectroStream's 12-deep RVQ. A decoder-only model generates one 40 ms frame at a time: a temporal Transformer summarises the windowed past once per frame, a small depth Transformer unrolls the frame's 12 tokens. Sliding-window attention with a learnable attention sink (and NoPE) lets it stream forever at constant cost. Three controls ride a 25 Hz bus: style through frozen MusicCoCa plus a one-step Mean Flow mapper, notes through a 128-channel four-token pianoroll with Auto-Strum, and drums on or off. Each is weighted by its own classifier-free guidance and can be masked for creative freedom. It runs on Apple Silicon through an MLX C++ engine, reacting in ~200 ms: fast enough to jam.

The numbers that matter

One frame of generation, as code

python# MRT2 generation loop (one frame), in pseudo-Python
def generate_frame(history, control, guidance, masks):
    # 1. CONTROL: sample each signal at this frame (25 Hz bus)
    c_style = mean_flow(musiccoca_text(control.prompt))           # frozen encoder + 1-step mapper
    c_notes = pianoroll_tokens(control.midi, auto_strum=control.strum)  # 128 ch x 4-token vocab
    c_drums = control.drums_on
    c = apply_masks(concat(c_style, c_notes, c_drums), masks)     # blank regions = freedom

    # 2. TEMPORAL: summarise the windowed past ONCE (SWA + sink + NoPE)
    ctx = temporal_theta(history[-W:], sink=learned_sink, cross_attend=c)  # cost set by W, not len(history)

    # 3. DEPTH: unroll 12 RVQ tokens, coarse to fine, each sees the coarser ones
    frame = []
    for j in range(12):
        logits = cfg_mix(depth_phi(frame, ctx), guidance)  # independent weight per signal
        frame.append(sample(softmax(logits)))

    # 4. CODEC: 12 codes -> 40 ms of 48 kHz stereo audio
    return spectrostream_decode(frame)                  # a' = Dec(x')

Where this sits in the field

• Lineage. The codec language model recipe (audio → RVQ tokens → predict tokens) traces to SoundStream and EnCodec, then AudioLM and MusicLM. The temporal × depth split is the RQ-Transformer idea. MRT2's contribution is bending all of it toward real-time, controllable, on-device use.
• Attention sinks and NoPE. From the streaming-LLM literature: the tricks that let a chat model run an endless conversation let MRT2 run an endless jam.
• Classifier-free guidance and Mean Flow. From the diffusion world, repurposed for tokens and for one-step style mapping under a real-time budget.
• What's next, per the authors. Fine-tuning on your own datasets, and performance tools built with musician Manaswi Mishra.

Keep going

• Neural audio codecs: SoundStream, EnCodec, and the RVQ ladder in depth.
• Music generation: the offline models MRT2 is reacting against.
• The Transformer and positional encoding: the substrate, and why NoPE is surprising.
• Attention sinks: the streaming-LLM finding behind Chapter 4.
• Score matching and guidance and flow matching: where CFG and Mean Flow come from.
• Contrastive learning and CLIP: the recipe MusicCoCa follows.

Now press Present or Teach and explain the 200 ms budget back, out loud, from memory. If you can account for every link, you own this model. Then go back to the pads and play with the MRT2 partner one more time.