Live Music Models:
Magenta RealTime

Chapter 1: The Key Insight

How do you generate infinite audio in real time with a Transformer, when Transformers were trained on fixed-length clips and slow down as context grows? Magenta RT's answer is three ideas stacked together — each one removing a different blocker.

The three moves

The pipeline (Figure 1, in words)

Three trained components, chained, looping every 2 seconds:

We will build each one, in the order the data flows: tokens (Ch 2–3), then the streaming trick (Ch 4), then the style control (Ch 5), then the LM that ties them together (Ch 6).

Chapter 2: Audio as Tokens (SpectroStream)

A Transformer predicts discrete tokens. But audio is a continuous waveform — at 48 kHz, that's 48,000 floating-point samples per second per channel. You cannot ask a language model to predict raw samples. So step one of any "codec language model" is to turn audio into something token-shaped.

That job belongs to a neural audio codec: a learned encoder/decoder pair. The encoder squeezes the waveform into a short sequence of integers; the decoder reconstructs the waveform from those integers with minimal perceptible loss. Think of it as a learned, lossy "JPEG for sound" — but the compressed form is a grid of codebook indices, which is exactly what an LM can model.

The shapes, made concrete

Formally, the encoder maps a stereo waveform a ∈ R^{(T·f_s) × 2} into a token tensor of shape V_c^{(T·f_k) × d_c}. Decode each symbol:

• T — duration in seconds.
• f_s — sampling rate of the waveform: 48 kHz (full-band, both channels).
• f_k — token frame rate: 25 Hz. So 1 second of audio becomes 25 "frames."
• d_c — RVQ depth: how many integers describe each frame. SpectroStream's full depth is 64.
• V_c — codebook size: 1024. Each integer is one of 1024 possible symbols.

Why a spectrogram-domain codec, and the bitrate math

SpectroStream operates in the spectral domain (hence the name) and is built on residual vector quantization (RVQ) — the next chapter's whole topic. Full SpectroStream runs at 16 kbps (f_k=25, d_c=64, |V_c|=1024). Sanity-check that number: each token carries log₂(1024) = 10 bits. Tokens per second = 25 frames × 64 depth = 1600. So 1600 × 10 = 16,000 bits/s = 16 kbps. The math closes.

That's beautiful fidelity but far too many tokens for real time. The fix, previewed: keep only the first 16 of the 64 RVQ levels → 25 × 16 = 400 tokens/sec → 4 kbps. That "live throughput target of 400 tokens per second" is the number the whole system is engineered around.

Drag the frame rate. Low f_k → few frames → the reconstruction can't track the wiggles (audible smearing). High f_k → faithful, but more tokens to predict. SpectroStream sits at 25 Hz — a deliberate quality/throughput trade.

Chapter 3: RVQ Depth — the dial that buys real time

We keep saying "the first 4 of 16 levels" as if dropping levels were free. To understand why coarse tokens are good enough for the history but not for the output, you have to understand what an RVQ level actually is.

One codebook can't be precise. So stack them.

Vector quantization replaces a continuous vector with the nearest entry in a learned codebook of, say, 1024 prototypes. The problem: 1024 prototypes is a coarse grid — the error (the leftover residual) can be large. You could grow the codebook to a billion entries, but the LM's vocabulary would explode.

Residual vector quantization is the elegant fix. Quantize the vector with codebook 1; you get a coarse approximation and a leftover residual. Quantize that residual with codebook 2. Quantize its residual with codebook 3. Each level is one integer; each level shrinks the error. This is exactly the move behind SoundStream and Encodec.

where e_k is the k-th codebook and i_k is the chosen index at level k. The first few levels carry the loudest, most salient structure (the gist of the sound); later levels add fine texture and air.

Worked numbers: the cost of remembering

The model conditions on H = 5 chunks (10 s) of history. Compare the two choices for representing that history:

Add the 12 style tokens (next chapter) and the encoder input is exactly 1000 + 12 = 1012 tokens — the number the paper states. Every design choice is in service of that token count.

The green curve is the target signal; the orange curve is the RVQ reconstruction using only the chosen number of levels. Notice how the shape is mostly captured by level 4, and later levels only refine fine wiggles. That is precisely why 4 levels suffice for the memory but not the output.

Chapter 4: Chunked Streaming — SHOWCASE

Now the centerpiece. We have tokens (Ch 2) and a cheap way to represent history (Ch 3). The remaining problem: a Transformer trained on fixed clips behaves unpredictably when you ask it to extrapolate to sequences longer than it ever saw. Generating a 10-minute song token-by-token from position 0 means the model is soon operating far outside its training distribution.

The fix: a sliding Markov window of chunks

Operate on chunks of length C = 2 s. Under a Markov assumption, predict chunk i from only the last H = 5 chunks (10 s of history) and the current style vector. Slide the window forward 2 s, predict the next chunk, repeat — forever.

The formal modeling objective the paper writes is:

Reading it: predict the full-depth tokens of chunk i, given the coarse (4-level) tokens of the previous H chunks and the style vector c_i. Two distinct token resolutions in one equation — that is the whole trick on a single line.

What degrades when you change the knobs

This is the intuition the sim below makes physical:

• Tiny window (H=1): the model forgets the groove — beats and harmony lose continuity between chunks.
• Big chunk (C large): control delay D grows — your prompt change takes longer to "land."
• Tiny chunk (C small): snappy control, but more boundaries per second → more risk of audible seams and more overhead.

Magenta RT's C=2 s, H=5 is the sweet spot the team found through play-testing with musicians: responsive enough to feel like an instrument, long enough to keep the music coherent.

Watch the orange playhead advance one chunk at a time. The shaded green band is the context window the model "sees" to predict the next chunk. Hit Change prompt now mid-stream: the new style takes effect only at the next chunk boundary — that delay is D. Shrink C and the change lands sooner; grow H and the band stretches further back (more memory, more tokens).

Chapter 5: Style Embeddings (MusicCoCa)

We've been calling c_i "the style vector." Where does it come from, and why is a single 768-dim vector enough to mean "techno flute, but moodier"?

The control signal is produced by MusicCoCa — a joint audio–text embedding model. It has two towers that map into the same 768-dim space: an audio tower (a 12-layer Vision Transformer over log-mel spectrograms) and a text tower (a 12-layer Transformer over tokenized text). It's built on MuLan and CoCa (a "contrastive captioner").

From embedding to tokens (the data flow)

The continuous 768-dim embedding is then quantized into 12 discrete tokens (codebook size |V_m| = 1024) so it can sit in the LM encoder's input alongside the audio tokens. That's the "12 style tokens" we kept adding to the 1000 audio tokens to reach 1012.

Symbol decode for c_i in the objective:

where M_A is the MusicCoCa audio tower and the subscript picks the most-recent 10-s MusicCoCa embedding for chunk i. At training time the style comes from the target audio itself (the model sees what it should produce); at inference the user supplies it via text/audio prompts.

The dot is the resulting style vector c, a weighted average of two prompt embeddings. Slide the weight and watch c travel along the line between "techno" and "flute" — that geometry is exactly what the model conditions on.

Chapter 6: The Encoder–Decoder LM

Everything funnels into one Transformer. Magenta RT uses a T5-style encoder–decoder language model (released at T5 Base = 220M and Large = 770M). Let's trace exactly what enters each half.

The encoder: digest history + style

The bidirectional encoder reads the concatenation:

That's the 1000 coarse audio tokens (4 levels × 25 fps × 2 s × 5 chunks = 1000) plus 12 style tokens = 1012 tokens. Crucially the vocabulary is unified: V = {<S>, <P>} ∪ V_c ∪ V_m — special tokens, codec tokens, and quantized style tokens all share one embedding table.

The decoder: the depth trick that buys RTF ≥ 1×

Here is where naive approaches die. The output chunk is 50 frames × 16 levels = 800 tokens. A vanilla autoregressive decoder would predict 800 tokens one at a time — far too slow. MusicLM's answer was a hierarchical cascade of multiple LMs; MusicGen's was a delay pattern. Neither hits RTF ≥ 1× at full bandwidth.

Magenta RT instead uses a two-module decoder (following the "between-frame / within-frame" method, like Moshi/UniAudio):

Working Python: the chunked streaming loop

The core insight, runnable. This is the inference loop — the exact thing the showcase sim animates:

Read the two load-bearing lines: c = style_vector(user_prompts()) is re-evaluated every loop, which is why the music is live; history = history[1:] + [new_chunk] is the sliding Markov window, which is why it's infinite and stateless.

Chapter 7: Prompt Blending & Transitions

Live control isn't just "pick a style." It's moving between styles smoothly while the music keeps playing. Two mechanisms enable this: weighted prompt mixing (the algebra) and conditioning interpolation (the time dimension).

The weighted-average formula

With N prompts (text or audio), the target conditioning vector is:

Each w_i is a slider on prompt i's influence; M is the relevant MusicCoCa tower. Set "techno"=0.8, "flute"=0.2 and you get techno that's faintly flute-tinted. This is the embedding arithmetic from Ch 5, now with a normalization so weights act like proportions.

Transitions: interpolate the conditioning over time

To cross-fade from prompt A to prompt B, linearly interpolate between their embeddings across the performance — the paper does it over 60 s in 6 steps (10 s each). At each chunk the style vector is a little more B and a little less A, so the music morphs continuously.

The dashed line is the conditioning we command (a clean A→B ramp). The solid orange line is the audio's measured similarity to B. Raise "context inertia" (the pull of the 10-s history) and watch the response lag and dip mid-transition — exactly the smooth, slightly-behind blending the paper observes.

Chapter 8: Audio Injection

Text and style vectors steer the mood. But what if you want to feed the model live sound — hum a melody, tap a rhythm, play a chord on a real instrument — and have the AI react to it? That's audio injection, and it's almost free given the architecture we've built.

The mechanism (and the clever non-obvious bit)

At each generation step: mix the user's input audio into the model's most recent output, tokenize the mixture, and feed it as the context for the next chunk. That's it. No new training, no new module — it rides on the existing "condition on the last 10 s of audio" loop.

Data flow, and what degrades without it

Normally history = [past model chunks]. With injection, the most recent slot becomes tokenize(mix(model_out, user_audio)). If the user audio is silent, the mix is just the model's own output → identical to vanilla generation. If the user audio dominates the mix, the model leans hard on it and may echo it. The mix ratio is the dial between "ignore me" and "follow me."

Blue = your injected audio (a steady melody). Teal = the model's continuation. At mix 0 the model does its own thing; raise the mix and the teal output starts tracking the blue melody's contour — because that contour now lives in the context it's continuing from.

Chapter 9: Experiments & Sampling

Does this live model actually sound good — or did real-time come at the cost of quality? The headline result: Magenta RT beats bigger offline open models on standard metrics, while being the only one that's live.

The benchmark table (Song Describer Dataset, 47 s clips)

Read the columns: FD_openl3 (Fréchet distance, lower = output distribution matches real music) and KL_passt (lower = semantically closer to reference) — Magenta RT wins both, with 38% fewer params than Stable Audio Open and 77% fewer than MusicGen Large. CLAP (higher = text adherence) — it lands in the middle; the paper notes Stable Audio Open trains on CLAP embeddings, an apples-to-oranges advantage, while Magenta RT trains on MusicCoCa.

The sampling recipe (the knobs that matter at inference)

Generation uses classifier-free guidance (CFG) with temperature 1.3, top-K 40, CFG weight 5.0. CFG pushes the sample toward the prompt by extrapolating away from the unconditional prediction:

where s is the CFG weight. At s=1 you get plain conditional sampling; s=5 strongly amplifies the prompt's influence (sharper adherence, but too high can over-saturate and hurt diversity). Temperature 1.3 and top-K 40 keep the samples varied without going incoherent.

Each dot is a sampled output around the prompt (the cross). Raise CFG and the cloud is yanked toward the prompt (tighter adherence). Raise temperature and the cloud spreads (more diversity). The paper's chosen point — s=5, T=1.3 — is the balance that felt best in play-testing.

Chapter 10: Connections & Cheat Sheet

Magenta RT sits at the intersection of three lineages: neural audio codecs (SoundStream → Encodec → SpectroStream), codec language models for audio (AudioLM → MusicLM → MusicGen), and joint audio-text embeddings (MuLan → CoCa → MusicCoCa). Its novelty is the live reframing: chunked autoregression + coarse context + streaming decoder.

How it relates to its ancestors

What the paper doesn't solve (read the fine print)

• Control delay floor. D ≈ 2 s (one chunk). Direct MIDI/note-level control would need far lower D — explicitly named as future work.
• Single-stem. Output is a single mixed stream. Multi-stem (separate drums/bass/lead) would enable true "jamming partner" accompaniment — also future work.
• Mostly instrumental data. ~190k hours of largely instrumental stock music; vocals are out of scope here.

Cheat sheet — every symbol in one place