MusicLM: Generating Music
From Text

Chapter 1: The Key Insight

MusicLM solves all three hardnesses with one architectural move: cast text-to-music as a hierarchical sequence-to-sequence problem over discrete tokens, using three independently-pretrained-and-frozen models to produce those tokens. No single network learns audio + semantics + text alignment all at once. Each job gets its own specialist.

The three frozen specialists

Each defeats one of the three hardnesses from Chapter 0.

Why "frozen" is doing real work here

All three models are pretrained separately and then frozen — their weights never update while MusicLM trains. This is a deliberate engineering decision, not laziness:

• Each can train on its own ideal data. SoundStream and w2v-BERT learn from unlabeled audio. MuLan learns from weakly-paired music-text. None needs the others.
• The generative model becomes a pure token translator. MusicLM's three Transformers only ever see integer token IDs going in and out. They never touch a waveform or a text string directly. This is exactly why it can borrow AudioLM wholesale.
• Reproducibility of the conditioning. Because MuLan is frozen, the same quantizer maps both audio and text into tokens — which is the linchpin of the train/infer swap in Chapter 7.

Chapter 2: Background — AudioLM & Quantization

MusicLM is, in the authors' words, AudioLM extended with text conditioning. So we need two prior ideas firmly in hand: quantization (turning continuous audio into discrete tokens) and AudioLM's coarse-to-fine hierarchy.

Why quantize at all?

Transformers predict from a finite vocabulary via softmax — the same machinery that picks the next word. To reuse that machinery for audio, you must turn audio into a sequence of integers drawn from a fixed codebook. That's quantization. A VQ-VAE learns a codebook of K vectors; an encoder maps each chunk of audio to its nearest codebook vector and emits that vector's index. The decoder turns indices back into audio.

AudioLM's two token types

AudioLM's key realization (which MusicLM inherits) is that one token type can't do everything, so it uses two, each from a different model:

The generation order is coarse-to-fine: first generate the slow semantic tokens (they fix the "what" — the structure), then generate the fast acoustic tokens conditioned on the semantics (they fill in the "how it sounds"). This is the move that resolves Hardness #2: structure lives in a short, learnable sequence; detail lives in a longer sequence that only needs local context because the structure is already pinned down.

Chapter 3: SoundStream & Residual Vector Quantization

The acoustic tokens come from SoundStream, a neural audio codec. Its encoder strides over 24 kHz audio with a factor of 480, producing one embedding every 480 samples → 50 embeddings per second. Each of those embeddings then has to be turned into discrete tokens. The naive way fails badly — and fixing it is the heart of this chapter.

Why a single codebook can't do high fidelity

Suppose you want 6 kbps of audio quality. The bitrate of one codebook is rate × log₂K bits/sec. To hit 6 kbps at 50 Hz you'd need log₂K = 120 bits per embedding → a codebook of K = 2¹²⁰ vectors. That's astronomically, unstorably large. A single codebook hits an exponential blowup as you demand more fidelity.

The arithmetic, worked

MusicLM's SoundStream uses 12 quantizers, each with a vocabulary of 1024. So the cost is linear in fidelity:

Each embedding becomes 12 token IDs (one per quantizer level). The bitrate:

And the token count that the generative model must produce:

That 600/sec figure is the number to remember — it's the firehose from Hardness #1, now compressed from 24,000 samples/sec down to 600 tokens/sec, a 40× reduction, while still reconstructing crisp 24 kHz music.

The orange dot is the target embedding we want to encode. Drag RVQ levels up. Each level snaps the remaining residual to its nearest small-codebook vector (the teal dots) and adds it to the running reconstruction (blue). Watch the reconstruction crawl toward the target and the residual shrink — fidelity grows by adding levels, never by one huge codebook.

# Residual vector quantization: the core insight in ~10 lines
import torch

def rvq_encode(x, codebooks):
    """x: [D] target embedding.  codebooks: list of [K, D] tensors."""
    residual = x.clone()
    tokens, recon = [], torch.zeros_like(x)
    for cb in codebooks:                 # 12 levels, each K=1024
        d = ((cb - residual) ** 2).sum(-1)  # dist to every codeword
        idx = d.argmin()                  # nearest codeword index = the TOKEN
        tokens.append(idx.item())
        recon += cb[idx]                   # reconstruction = SUM of chosen codewords
        residual = x - recon            # quantize the LEFTOVER next
    return tokens, recon              # 12 ints; recon -> SoundStream decoder -> audio

Chapter 4: Semantic Tokens from w2v-BERT

Acoustic tokens are detail-rich but structure-poor: 600/sec, mostly local. To carry long-term musical structure — the melody, the groove, where the song is going — MusicLM uses a second, much slower token type: semantic tokens, denoted S, derived from w2v-BERT.

What w2v-BERT is, and why a mid-layer

w2v-BERT is a 600M-parameter model pretrained with masked language modeling (MLM) on audio — it learns to fill in masked chunks of an audio sequence, with no labels. Exactly like BERT for text, its intermediate layers learn abstract, structure-bearing representations. MusicLM extracts embeddings from the 7th layer specifically.

Quantizing with k-means, not RVQ

Unlike SoundStream's RVQ, the semantic embeddings are quantized with a single k-means codebook: cluster the layer-7 embeddings into 1024 centroids, and each embedding becomes the ID of its nearest centroid. One token per frame — no residual stack — because we want a compact structural sketch, not high-fidelity reconstruction.

The rate — and the worked count

Semantic tokens come out at 25 Hz: just 25 tokens per second of audio. Contrast that with 600 acoustic tokens/sec. The ratio matters:

So a 10-second clip is just 250 semantic tokens but 6,000 acoustic tokens. The semantic sequence is 24× shorter — short enough for a Transformer to plan a coherent minute of music over. That length asymmetry IS the resolution of the coherence-vs-fidelity dilemma: plan in the short sequence, render in the long one.

Each gray dot is a w2v-BERT layer-7 embedding (shown in 2-D). The big colored dots are the k-means centroids (the codebook). A line connects each embedding to its nearest centroid — that centroid's index is the semantic token. Increase K to see finer structure captured; the stream of token IDs along the bottom is exactly what the semantic Transformer learns to model.

Chapter 5: MuLan — The Trick That Removes Paired Data

Now the masterstroke. We have great audio tokens (acoustic + semantic) but they all come from audio. How do we steer generation with text, given we have almost no paired data (Hardness #3)? Answer: MuLan, a joint music-text embedding model.

What MuLan is

MuLan has two encoders — two "towers." A text tower (a BERT) maps a caption to a 128-d vector. An audio tower (a ResNet-50) maps a music clip to a 128-d vector. They are trained with a contrastive loss: pull the embeddings of a matching (music, text) pair together, push mismatched pairs apart. The result is a single shared 128-d space where "a song about it" and the actual song sit close.

Symbol by symbol: a is the audio embedding, t₊ its true caption's embedding, t_j ranges over all captions in the batch (the true one plus mismatched "negatives"), sim is cosine similarity, and τ is a temperature. Minimizing this makes the matching pair the most similar — i.e., it aligns the two towers into one space. (Same recipe as CLIP for images.)

Why this beats the DALL·E 2 "prior"

DALL·E 2 also uses a CLIP joint space — but it trains a separate "prior" model to map text embeddings → image embeddings before decoding. MusicLM omits that prior entirely. Because MuLan's two towers are already so well aligned, the text embedding can be used directly as a stand-in for the audio embedding. Simpler pipeline, and it lets the decoder train on audio-only data.

Quantizing M into tokens

The 128-d MuLan vector is continuous, but MusicLM wants everything as tokens for a homogeneous interface. So it applies another RVQ — 12 quantizers × 1024 — to the MuLan embedding, yielding 12 MuLan tokens. Since MuLan operates on 10-second windows, longer clips are handled by sliding a 10-s window with 1-s stride and averaging the embeddings before quantizing. Same RVQ is used for audio-derived M_A (training) and text-derived M_T (inference) — critical, so the token distributions match.

Each pair of dots (a music note ♪ and its caption ✎) starts scattered. Hit Run contrastive training: matching pairs are pulled together, mismatched ones repelled, until each caption sits on top of its song. Then hit Swap: we generate using the music embedding (training), then replace it with the nearby text embedding (inference) — and because they coincide, the generator can't tell the difference. That coincidence is the whole paired-data-free trick.

Chapter 6: The 3-Stage Hierarchy (the showcase)

Now we assemble everything. MusicLM generates music as a cascade of three autoregressive stages, each a separate decoder-only Transformer (24 layers, 16 heads, 430M params each). Each stage predicts one token type, conditioned on the tokens above it. This is the reconstruction of the paper's Figure 2 — and the payoff of the whole lesson.

The three stages and their conditional distributions

The token-count budget — worked end to end

Let's trace exactly how many tokens flow through each stage for 1 second of generated audio. This is the single most clarifying calculation in the paper:

Read the progression: the conditioning gets richer and the output rate gets higher as we descend. The model plans with 25 cheap tokens, then commits to 200, then renders 400 more — a coarse-to-fine waterfall. Each stage only has to model a sequence short enough to stay coherent given what's above it.

The four rows are MuLan → Semantic → Coarse acoustic → Fine acoustic. Each row's token count scales with the seconds slider so you can see the budget grow. Step a stage generates the next row, conditioned (dashed arrows) on the rows above — watch the per-second counts (12, 25, 200, 400) light up. Run full cascade animates the whole waterfall, ending at the SoundStream decoder. This IS MusicLM, end to end.

Chapter 7: Train vs Infer — The M_A→M_T Swap

Everything hinges on one swap. At training time MusicLM conditions on M_A (MuLan tokens from the audio). At inference time it conditions on M_T (MuLan tokens from the text prompt). Nothing else changes. Let's make the asymmetry precise because it's easy to misread.

Training (audio-only)

1. Take an unlabeled music clip from the 280k-hour corpus. No caption involved.
2. Run the three frozen tokenizers on it: SoundStream → A, w2v-BERT→k-means → S, MuLan audio tower + RVQ → M_A (12 tokens).
3. Train Stage 1 to predict S from M_A; Stage 2 to predict coarse A from (M_A, S); Stage 3 to predict fine A from coarse A. Standard next-token cross-entropy, teacher-forced — fully parallel over the sequence.

Inference (text-prompted)

1. Take the user's text: "Berlin 90s techno with a low bass and strong kick."
2. Run MuLan's text tower + the same RVQ → M_T (12 tokens).
3. Feed M_T where M_A went. Sample Stage 1 → S; Stage 2 → coarse A; Stage 3 → fine A. SoundStream-decode A → waveform.

Sampling temperatures — a real engineering knob

The paper uses different temperatures per stage, tuned by listening: 1.0 for semantic, 0.95 for coarse acoustic, 0.4 for fine acoustic. Why the descending schedule?

The paper's own probe experiment. Fix text only → re-sample semantics + acoustics: samples vary a lot in melody and rhythm, all still on-genre (high diversity). Fix text + semantics → re-sample only acoustics: samples share melody/rhythm but differ in reverb, distortion, instrument timbre (low diversity, same structure). Each bar shows how much that musical attribute varies — proving semantic tokens own structure and acoustic tokens own detail.

Chapter 8: Experiments & Results

How do you even measure "good text-to-music"? You need two axes: audio quality (does it sound like real music?) and faithfulness (does it match the words?). MusicLM uses three metrics, evaluated on MusicCaps — a new 5.5k-example dataset of expert musician captions the authors released.

The three metrics, demystified

• FAD (Fréchet Audio Distance) ↓: distance between the distribution of generated audio features and real audio features. Low = plausible-sounding. Reference-free (doesn't compare to a specific target). Reported with two backbones: Trill (speech-trained) and VGG (general audio).
• KLD (KL divergence) ↓: run an audio classifier on both generated and reference clips; compare the class-probability distributions. Low = the generated music has similar acoustic characteristics to the reference for that caption → a proxy for text faithfulness.
• MCC (MuLan Cycle Consistency) ↑: cosine similarity between the caption's MuLan embedding and the generated audio's MuLan embedding. High = the audio "cycles back" to the text. A clever use of MuLan as its own judge.

The headline table

Read it: MusicLM wins on FAD_VGG by a wide margin (4.0 vs 9.6), ties Mubert on FAD_Trill, and dominates on both faithfulness metrics (KLD 1.01, MCC 0.51). In human A/B listening it was preferred 312 times vs 158 (Riffusion) and 97 (Mubert) — though the real reference music still won more (472), so a gap to ground truth remains. Notably, MusicLM matches Mubert on quality despite Mubert stitching pre-recorded sounds made by humans.

Toggle the metric. Bars rescale; the arrow tells you whether lower or higher is better, and the winning bar is highlighted. Flip between quality (FAD) and faithfulness (KLD/MCC/Wins) to see that MusicLM's biggest edge is faithfulness — it captures the text best.

The crucial ablation: do semantic tokens earn their keep?

The authors trained a stripped model that predicts coarse acoustic tokens directly from MuLan tokens — skipping the semantic stage: p(A_t | A_<t, M_A). Result:

Failure modes the paper is honest about

When humans preferred the real music over MusicLM, three patterns recurred — worth knowing as the system's edges:

• Very detailed captions (5+ instruments, or non-musical asides like "wind, people talking") overwhelm it.
• Temporal ordering ("first a piano, then drums kick in") — MuLan embeddings are sequence-wide, so order is hard to honor.
• Negations ("no drums") are poorly captured — contrastive embeddings don't represent "absence" well.

Chapter 9: Extensions & Responsible Release

Two parts of the paper round out the story: an extension that adds melody conditioning, and a memorization study driven by the risks of generative music.

Melody conditioning — hum it, then restyle it

Words can't capture a melody you have in your head. So MusicLM adds a second conditioning signal: you hum, sing, or whistle a tune, and the model renders it in the style your text describes ("...as an electronic synth lead").

The mechanism reuses the joint-embedding playbook: build a melody embedding model trained so that two clips with the same melody but different acoustics (covers, instrumentals, vocals, hummed versions) land close together. The synthetic training pairs are made from different versions of the same song. Quantize the melody embedding with RVQ, and concatenate its tokens with the MuLan text tokens as a richer conditioning. Now the semantic stage sees both "what melody" and "what style."

Long generation & story mode

Because the semantic sequence is short, MusicLM can generate up to 5-minute clips. It can also change the MuLan conditioning every 15 seconds to produce smooth, tempo-consistent transitions between described sections — a primitive "story mode."

Memorization — the safety study

Generative music raises a real risk: regurgitating copyrighted training audio. The authors adapt the LLM memorization methodology to the semantic stage. They prompt with M_A plus the first T semantic tokens of a real clip (T from 0 to 250 = up to 10 s), greedily generate a 5-s continuation, and check for matches.

• Exact matches stay below 0.2% even with a 10-s prompt — the model rarely copies verbatim.
• Approximate matches (via optimal-transport distance over token histograms, threshold τ=0.85) are higher and grow with prompt length — but the matched sequences have very low token entropy (1.0 bits vs 4.6 bits average), i.e. they're repetitive, low-information segments where coincidental matches are expected.

Chapter 10: Connections & Cheat Sheet

MusicLM is a synthesis paper: it takes AudioLM's hierarchy, SoundStream's codec, w2v-BERT's semantics, and MuLan's joint space, and shows the combination unlocks text-to-music with no paired data. Let's pin down the whiteboard summary and the symbol table.

The one-paragraph whiteboard summary

MusicLM generates music as a coarse-to-fine token cascade. Three frozen tokenizers turn audio into tokens: SoundStream's RVQ gives 600 acoustic tokens/sec (fine detail), w2v-BERT + k-means gives 25 semantic tokens/sec (long-term structure), and MuLan's contrastively-trained joint space gives 12 conditioning tokens that are interchangeable between audio (M_A, training) and text (M_T, inference). Three autoregressive Transformers then model p(S | M_A), p(A^coarse | S, M_A), and p(A^fine | A^coarse) in sequence; SoundStream decodes the acoustic tokens to 24 kHz audio. Because MuLan aligns audio and text, the model trains on audio-only data and is prompted with text — sidestepping the scarcity of paired music-text data, which is the paper's central contribution.

Cheat sheet — every key symbol

Where it sits in the lineage

• Borrows from AudioLM the semantic/acoustic, coarse/fine token hierarchy.
• Borrows from SoundStream the RVQ neural codec (acoustic tokens).
• Borrows from MuLan/CLIP the contrastive joint embedding (the paired-data trick).
• Mirrors DALL·E 2's use of a CLIP space — but drops the prior, using the text embedding directly.
• Inspired follow-ups like MusicGen (single-stage with EnCodec + delay pattern) and the broader "tokens-everything" audio-LM line.