AudioLM: Audio as a Language

Chapter 1: Audio as a Language

Chapter 0 ended on a slogan: cast audio generation as a language modeling task. Slogans are cheap. This chapter makes it mechanical — what the objects are, what shapes they have, what gets frozen, and what the loss function is actually summing over.

The framework has exactly three components. The paper introduces them in a single paragraph and then never adds a fourth, which is unusual restraint and worth honoring by reading that paragraph slowly.

The three components, formally

Start with the input. A single-channel audio sequence is a vector of real numbers:

x is the waveform. T is the number of samples — 16,000 per second at the sample rate used throughout this paper. Each entry is one amplitude measurement, typically in [−1, 1]. That is the whole input format. No spectrogram, no framing, no windowing at this level of the description.

Component 1 — the tokenizer. A frozen encoder that maps the waveform to a short sequence of symbols drawn from a finite vocabulary:

Every symbol: enc is the encoder (in practice either SoundStream's convolutional encoder plus its quantizer, or w2v-BERT plus k-means). h_t is one token — an integer index into a codebook, not a vector. T′ is the token count. The double-less-than sign is not decoration; the paper says T′ is "typically 2–3 orders of magnitude smaller than T," and Chapter 3 will show the ratios are 640 for semantic tokens and 320 for acoustic frames.

Component 2 — the language model. A decoder-only Transformer over those tokens, trained on the ordinary maximum-likelihood objective:

Read the product as: the probability the model assigns to the whole token sequence is the product of the probability it assigns to each token given everything before it. h_<t means the tokens at positions 1 through t−1. This is GPT's objective with the word "word" crossed out and "audio token" written in. Nothing about it is audio-specific.

Component 3 — the detokenizer. A frozen decoder that turns predicted tokens back into a waveform:

where the hats mark quantities the model produced rather than measured. In AudioLM, dec is always SoundStream's convolutional decoder — the semantic tokenizer has no decoder at all, which is exactly the property Chapter 2 measures and complains about.

Every symbol, once, with an analogy

The paper's notation is compact and it never repeats a definition. Here is the complete key; refer back to it whenever a superscript surprises you.

Two notational traps worth pointing at directly. First, Y is a matrix, z is a vector — the acoustic representation has a second axis (the quantizer index) that the semantic representation does not. Almost every confusing sentence in Section III is confusing because of that asymmetry. Second, superscripts on y are quantizer indices, not exponents. y^Q′_t is "the Q′-th quantizer's code at time t," never "y to the power Q′."

Where the product objective comes from

The product in Component 2 is not a modeling choice; it is an identity. Any joint distribution over a sequence factorizes by the chain rule of probability, exactly, with no assumptions:

Nothing has been lost. What the architecture chooses is how to compute each conditional: with a causal Transformer that attends over all previous positions. A model that attends over only the previous 20 tokens is making an approximation; a full-context causal Transformer is not.

This matters for audio in a way it does not for text. In text, a 512-token window covers a paragraph and most dependencies are local. In audio, a limited window is precisely the WaveNet failure — the receptive field ends before the word does. The chain rule says nothing is lost in principle; the receptive field says everything is lost in practice. Tokenization is what makes the principle achievable.

One consequence to hold: at generation time, every token is drawn from the model's own distribution conditioned on tokens it itself produced. There is no ground truth to fall back on. This is why sampling temperature is not a cosmetic knob — it shifts the distribution the entire remaining generation is conditioned on.

Sampling versus argmax

Why sample at all? Take the greedy path — always emit the highest-probability token — and the generation is deterministic given the prompt. Two problems, one aesthetic and one fatal.

The aesthetic problem: a single prompt yields a single continuation, forever. The acoustic-generation experiment in Chapter 7 depends on running the same semantic tokens through stage 2 repeatedly and getting different speakers. Greedy decoding makes that experiment impossible.

The fatal problem: greedy decoding on an autoregressive model over a high-entropy stream falls into loops. The most likely next token given a stretch of near-silence is more near-silence; the most likely token after that is more of the same. Text LMs show this as repeated phrases; audio LMs show it as a held tone or a stuck hum. Temperature sampling with T < 1 keeps most of the sharpening benefit while retaining enough entropy to escape.

The Transformer itself, and a parameter count that does not quite add up

The paper uses "identical decoder-only Transformers in all stages." The configuration, verbatim from Section IV-B:

Do the arithmetic yourself rather than accepting the 0.3B. Per Transformer block, the attention projections (query, key, value, output) are four d×d matrices:

The feed-forward block is two matrices, d×4096 and 4096×d:

Per block: 4,194,304 + 8,388,608 = 12,582,912, plus a few thousand for the two layer norms. Times 12 blocks:

Add the token embedding table — even a generous 5,120-entry vocabulary at d = 1024 is only 5.2M — and the relative-position bias tables, which are tiny. The stated dimensions account for roughly 0.15B, not 0.3B.

One genuinely interesting choice in that table is relative positional embeddings. Absolute position embeddings tie the model to the crop length it was trained on; a model trained on 750-position crops has never seen position 900. Relative embeddings encode "how far back" rather than "where," which is what you want for a signal with no canonical origin. Audio has no sentence-initial token. Every position is the middle of something.

Why the tokenizers are frozen — and what it costs

The paper flags this as one of two "aspects to emphasize": the tokenizer and detokenizer "are pre-trained and frozen ahead of training the language model, which decouples the tokenizers and the language model and simplifies the training setup."

Decoupling is a real engineering benefit and it is easy to underrate. SoundStream is trained with reconstruction losses plus adversarial losses — a GAN, with all the instability that implies. w2v-BERT is trained with a masked-language-modeling loss plus a contrastive loss on 0.6B parameters. The language models are trained with plain cross-entropy. Joining those three optimization problems into one would mean tuning a GAN, a contrastive objective, and an autoregressive objective simultaneously, with gradients from the LM pulling the codebook around while the discriminator pulls it somewhere else. Nobody wants that.

Freezing also means the tokenizers can be trained once and reused. All three stages share one SoundStream and one w2v-BERT. Change the LM architecture and you do not retokenize your 60,000 hours of audio.

But be honest about the cost, because the paper is not. A frozen codebook is optimized for reconstruction, not for predictability. SoundStream's RVQ minimizes waveform distortion at a given bitrate; nothing in its objective encourages the resulting token sequence to be easy for a language model to predict. It is entirely possible that a slightly worse codec would produce a much more learnable token stream. AudioLM does not test this, and the question stayed open for years.

What a "token" even is here

This is where audio language modeling diverges from text in a way that trips people up, so let us be blunt about it.

In text, token 4,281 is " the", and there is a sense in which the model inherits meaning from the token's form — subword pieces share characters, and embeddings for related pieces end up related partly because they co-occur with related contexts.

In audio, token 511 is the 511th centroid of a k-means fit, or the 511th entry in a learned codebook. The index is arbitrary. Index 511 and index 512 have no relationship whatsoever; they were assigned in whatever order the initialization produced. There is no ordering, no arithmetic, no similarity readable from the number.

The consequence: the Transformer's embedding table must learn the entire geometry of the codebook from co-occurrence statistics alone. It sees only "index 511 tends to follow index 88 and precede index 903." From that it must reconstruct the fact that 511 and 512 are acoustically adjacent — or fail to, and pay for it in likelihood.

This is not a flaw, it is just the setting. And it explains a design choice that would otherwise look strange: the per-layer offsets in the flattening scheme (Chapter 4). When you flatten a T_A×Q matrix of RVQ codes into one sequence, index 7 from quantizer 1 and index 7 from quantizer 2 mean totally different things. The offsets keep them as distinct vocabulary entries rather than collapsing them.

The loss, with actual numbers

Maximizing a product of probabilities is minimizing a sum of negative log probabilities — cross-entropy, one term per token. Watch it happen on a toy vocabulary of four semantic tokens.

Suppose at some position the model emits logits over four candidate tokens:

Step 1 — exponentiate. e^2.0 = 7.389056, e^0.5 = 1.648721, e^−1.0 = 0.367879, e^0.3 = 1.349859.

Step 2 — sum. 7.389056 + 1.648721 + 0.367879 + 1.349859 = 10.755515.

Step 3 — divide. p(A) = 7.389056 / 10.755515 = 0.68700. p(B) = 1.648721 / 10.755515 = 0.15329. p(C) = 0.367879 / 10.755515 = 0.03421. p(D) = 1.349859 / 10.755515 = 0.12550. They sum to 1.00000, as they must.

Step 4 — score the truth. Say the ground-truth next token was D. The loss at this position is

Three bits of surprise. For calibration: a uniform guess over 1024 semantic tokens costs log₂1024 = 10 bits. A trained model averaging 3 bits per semantic token is compressing the stream by a factor of more than three beyond the tokenizer's own 250 bps — which is a concrete way of saying "the model has learned that speech is predictable."

Temperature, and why 0.6

At inference AudioLM uses temperature sampling with temperatures of 0.6, 0.8 and 0.6 for the three stages. Temperature divides the logits before the softmax. Run the same four numbers at T = 0.6:

Step 1 — divide. [2.0, 0.5, −1.0, 0.3] / 0.6 = [3.3333, 0.8333, −1.6667, 0.5].

Step 2 — exponentiate. e^3.3333 = 28.0316, e^0.8333 = 2.3009, e^−1.6667 = 0.18888, e^0.5 = 1.64872. Sum = 32.1701.

Step 3 — divide. p(A) = 0.87135, p(B) = 0.07152, p(C) = 0.00587, p(D) = 0.05125.

The top candidate went from 0.687 to 0.871; the tail candidate C fell from 3.4% to 0.6%. Temperature below 1 sharpens. The paper's justification is one sentence: "we found that these temperature values provide a good trade-off between diversity and semantic consistency of the generated speech."

Why decoder-only, and how conditioning gets in

Stages 2 and 3 are conditional models — stage 2 needs the whole semantic sequence, stage 3 needs the coarse acoustic tokens. The textbook architecture for conditional sequence modeling is an encoder–decoder with cross-attention. The paper uses neither. It uses a decoder-only Transformer and puts the conditioning in the sequence, as a prefix.

Concretely, stage 2's training sequence is written out in full in Section III-C, and it is worth transcribing because it is the least intuitive object in the paper:

All the semantic tokens first, in order. Then the coarse acoustic tokens, flattened. The paper adds one clarifying detail: "with y¹₁ being the first token predicted during training" — meaning the loss is not applied to the semantic prefix at all in stage 2. Those positions are conditioning, not targets.

Why prefix rather than cross-attention? Three reasons, in descending order of how much the paper cares. It keeps all three stages architecturally identical, so one implementation serves everything. It lets the conditioning participate in the same relative-position scheme as the targets. And it sidesteps the alignment question entirely: a prefix does not need to be time-synchronized with what follows, which — recall the deduplication wrinkle from Chapter 0 — is exactly the property this system needs.

The cost is sequence length. Cross-attention would let stage 2 attend to 250 semantic tokens without spending 250 positions of its own context. Prefixing spends them. At 2,250 total positions, 11% of stage 2's context is conditioning. That is affordable; at longer crops it would not be.

How much does tokenization actually buy?

Time to make the compression concrete. The sim below is a calculator you can feel: drag the duration and watch every derived quantity move — sample count, token counts per stream, the flattened sequence length each stage actually sees, and the attention cost of each option on a log scale.

Push the slider to 30 seconds and read the top two bars. Raw samples: 480,000. Semantic tokens: 750. The attention costs differ by a factor of 410,000. That is the entire justification for the tokenizer, drawn to scale.

Now push to 3 seconds and look at the stage-3 row. 1,800 tokens — comfortably inside the "up to 10³–10⁴" regime the paper calls acceptable, and small enough that the stage can be run on independent chunks in parallel batches. Chapter 4 explains why that independence is legitimate.

The training loop, in code

There is nothing audio-specific left once tokenization is done. Here is stage 1 in full — if you have written a character-level language model, you have written this:

python — stage 1 training, complete
# z: int tensor (B, Ts) of semantic tokens, values in [0, 1023]
# 30-second crops => Ts = 750 before dedup

logits = model(z[:, :-1])                 # (B, Ts-1, 1024)
loss   = cross_entropy(
             logits.reshape(-1, 1024),
             z[:, 1:].reshape(-1))       # next-token, teacher forced
loss.backward()

# that is the entire objective. No audio loss, no spectrogram loss,
# no adversarial term — those all live inside the frozen SoundStream.

Notice what is absent. No mel-spectrogram L1 term. No multi-scale STFT loss. No discriminator. All of the perceptual machinery that makes neural audio sound good has been pushed inside the frozen codec, where it was trained once and forgotten. The language models never touch a waveform.

That is the deepest structural point of the paper, and it is easy to miss because it takes the form of an absence: AudioLM contains no audio-specific loss function. It is three text-style language models in a trench coat, and the trench coat is SoundStream.

Where "teacher forcing" bites later

The training loop above conditions each prediction on ground-truth previous tokens. The paper is explicit: each stage is "trained for predicting next tokens given all previous ground-truth tokens in the corresponding stage."

At inference, stages 2 and 3 are conditioned on tokens the previous stage generated, not on ground truth. This is exposure bias, and in a cascade it compounds: a slightly off-distribution semantic sequence from stage 1 becomes the conditioning for stage 2, whose output becomes conditioning for stage 3.

Does it hurt? Chapter 7 gives the measurement. When stage 2 is fed ground-truth semantic tokens, the resulting audio transcribes at 6.0% word error rate versus 2.6% for a straight SoundStream reconstruction. So the semantic→acoustic mapping alone costs about 3.4 points of WER, before stage 1 has contributed any error at all. The paper says as much: "most of the errors are coming from the mapping of semantic to acoustic tokens."

The vocabulary budget, stage by stage

Because tokens from different sources must not collide, each stage works over a composite vocabulary. Laying it out removes a lot of later confusion:

Stage 3 has the largest vocabulary and the shortest sequences; stage 1 has the smallest vocabulary and the longest sequences. That inverse relationship is not an accident — it falls straight out of the bitrate ladder. Coarse information is cheap per symbol and must span long times. Fine information is expensive per symbol and only needs to span a moment.

One more consequence, easy to overlook: because each stage is a separate model with a separate vocabulary, the three stages could in principle have different architectures. The paper chooses identical ones — "we use identical decoder-only Transformers in all stages" — which is a simplicity decision, not a necessity. Later work in this lineage (notably the RQ-Transformer style designs) exploits exactly that freedom.

Inline check

(a) Why does the LM's vocabulary size for the acoustic stages exceed 1024 even though each RVQ codebook has exactly 1024 entries?
Because flattening adds per-layer offsets: quantizer q's codes occupy a disjoint block of the vocabulary. With Q′ = 4 coarse layers the stage-2 acoustic vocabulary is 4 × 1024 = 4096 entries, plus the semantic conditioning vocabulary.

(b) If you doubled the SoundStream frame rate from 50 Hz to 100 Hz at fixed Q, what happens to reconstruction quality and to stage-2 attention cost?
Quality improves (more bits); attention cost quadruples (sequence doubles, cost is quadratic). This is constraint 1 versus constraint 2 from Chapter 0, felt in a single knob.

(c) The paper removes consecutive repetitions of semantic tokens in the first two stages. What does that do to the effective token rate, and why would you want it?
It compresses runs: a phoneme spanning three 40 ms frames that lands in the same cluster three times becomes one token. Effective rate drops below 25/s, so a 30-second crop fits comfortably. It also removes a trivially predictable pattern — predicting "same again" is free likelihood that teaches the model nothing.

One closing observation about component 3, the detokenizer. It is the only part of the system that ever touches a waveform at inference, and it is completely deterministic given the tokens. All the creativity, all the sampling, all the uncertainty lives upstream in the discrete world. The audio is merely what falls out.

The cascade's three failure surfaces

Before moving to the tokenizers, name the places a three-stage system can break. You will meet all three again with numbers attached.

Every one of those has a number, and every number comes from a different measurement instrument: a phonetic probe, an ASR system, a perceptual quality proxy. That is what a well-instrumented systems paper looks like — one metric per interface, not one metric for the whole pipeline.

Chapter 2 puts the two candidate tokenizations on a bench and measures them. Everything after that is consequence.

Read it with one question in mind: if these two codes were not complementary, what would AudioLM be? The answer is: an unnecessarily complicated way to do what GSLM already did. The complementarity is the load-bearing empirical fact.

Chapter 2: SHOWCASE — Two Tokens, One Trade-off

This is the chapter the paper is built on. Everything before it is motivation; everything after it is consequence. If the two tokenizations did not genuinely complement each other, AudioLM would be a needlessly complicated way to do something simpler.

So: measure them. Take the same speech, encode it two ways, and ask two questions of each encoding. Question one: can you still hear the difference between "bit" and "bet"? Question two: can you rebuild the waveform? The answers are startlingly lopsided in opposite directions, and Table I of the paper is one of the cleanest complementarity results you will find anywhere.

The two questions, made into instruments

You cannot measure "does it carry meaning" directly. You need a probe that is sensitive to linguistic content and insensitive to everything else, and a second probe that is sensitive to waveform fidelity and indifferent to meaning. The paper picks one of each.

Probe 1 — ABX error rate, for phonetic discriminability. ABX is a distance-based metric with no training and no classifier. You take a set of phoneme trigrams that differ only in the central phoneme — the canonical pair is "bit" versus "bet." Then:

Two variants matter enormously here. In the within-speaker condition, A, B and X come from the same speaker. In the across-speaker condition, A and B share a speaker and X comes from a different one. The gap between the two numbers is a direct readout of how much speaker information contaminates the representation — and that gap turns out to be the sharpest evidence in the entire table.

The paper computes ABX "using scripts published with the Libri-Light dataset with the default settings" and reports on LibriSpeech dev-clean. Standard tooling, standard split; no room for a thumb on the scale.

Probe 2 — ViSQOL, for reconstruction quality. ViSQOL is "a computational proxy for perceived similarity between a reference audio and its reconstruction," run in "speech" mode on 16 kHz signals. It outputs a number on a roughly 1–5 mean-opinion-score scale: 1 means "this does not resemble the reference," 5 means "perceptually identical." Higher is better.

And here is the crucial methodological move, easy to skim past: to measure reconstruction quality from semantic tokens at all, the authors train a SoundStream decoder to reconstruct audio from those tokens. The semantic tokenizer has no native decoder. They built the fairest possible one — the same decoder family that makes SoundStream sound good — and pointed it at w2v-BERT-derived codes. If the semantic tokens contained the waveform information, this decoder would find it.

ABX by hand, on five numbers

ABX is one of those metrics that sounds abstract until you compute one. Do it in two dimensions.

Pretend each utterance has been reduced to a 2D point where the first coordinate captures something speaker-ish (vocal tract length, say) and the second captures something vowel-ish (the height of the vowel). Take a within-speaker triplet from speaker S1:

Step 1 — distance from X to A. Differences: 0.55 − 0.40 = 0.15 and 0.75 − 0.90 = −0.15. Squares: 0.0225 and 0.0225. Sum: 0.0450. Square root: d(X, A) = 0.2121.

Step 2 — distance from X to B. Differences: 0.55 − 0.85 = −0.30 and 0.75 − 0.30 = 0.45. Squares: 0.0900 and 0.2025. Sum: 0.2925. Square root: d(X, B) = 0.5408.

Step 3 — score. 0.2121 < 0.5408, so X is nearer the other "bit". Not an error. Within-speaker, this representation works.

Now the across-speaker condition. Speaker S2 has a systematic offset in this space — a longer vocal tract shifts the first coordinate up, a different recording chain shifts the second down. Say the offset is (+0.35, −0.30). Then S2's "bit" lands at:

Step 4 — distance from X′ to A. Differences: 0.90 − 0.40 = 0.50 and 0.45 − 0.90 = −0.45. Squares: 0.2500 and 0.2025. Sum: 0.4525. Root: d(X′, A) = 0.6727.

Step 5 — distance from X′ to B. Differences: 0.90 − 0.85 = 0.05 and 0.45 − 0.30 = 0.15. Squares: 0.0025 and 0.0225. Sum: 0.0250. Root: d(X′, B) = 0.1581.

Step 6 — score. 0.1581 < 0.6727: X′ is nearer "bet". Error. The speaker offset — which has nothing to do with the vowel — was large enough to swamp the phonetic difference entirely.

That single worked triplet is the within/across gap. A representation that encodes speaker identity strongly will pass the within-speaker test and fail the across-speaker one. A representation that has discarded speaker identity will score nearly the same on both. Hold that thought for exactly one table.

From one triplet to an error rate

One triplet gives a bit. An ABX score aggregates over thousands. The procedure: enumerate every valid (A, B, X) triplet in the evaluation set, score each, average.

Make the paper's headline numbers concrete by putting them back into counts. Suppose you evaluate 1,000 within-speaker triplets:

The acoustic representation gets one in five wrong. That is not a subtle degradation; it is a representation in which "bit" and "bet" are frequently not distinguishable by distance. Chance on this task is 50%, so 22.4% is far better than random — the information has not vanished, it has been buried under variation the metric cannot ignore.

Here is the whole metric in code, which fits in ten lines and is worth reading because it removes any remaining mystery:

python — ABX error rate, from scratch
import numpy as np

def abx_error(triplets):
    """triplets: list of (A, B, X) arrays; A and X are the SAME trigram."""
    errors = 0
    for A, B, X in triplets:
        dA = np.linalg.norm(X - A)          # distance to the right anchor
        dB = np.linalg.norm(X - B)          # distance to the wrong anchor
        errors += int(dB < dA)               # nearer the wrong one => error
    return 100.0 * errors / len(triplets)

# real ABX uses DTW-aligned frame sequences, not single vectors,
# and averages over speaker/context conditions — but this IS the idea.

Two refinements the real implementation adds and this sketch omits. Utterances are variable-length frame sequences, so the distance is a dynamic-time-warping alignment cost rather than a Euclidean norm. And the averaging is stratified by speaker and phonetic context so that a few prolific speakers cannot dominate. Neither changes the logic: nearer the right anchor, or not.

A brief aside on scale, since the numbers in the table below are about to be compared across metrics with opposite polarities: ABX is an error rate, so lower is better; ViSQOL is a quality score, so higher is better. Getting the arrows the wrong way round on Table I inverts the entire argument, and it is an easy mistake to make at a glance.

Calibrating ViSQOL

"ViSQOL 1.1" means nothing until you know the scale. The metric targets a mean-opinion-score-like range, and it is worth carrying a rough conversion:

Now the third stage's existence has a number attached. It moves the system from 3.3 to 3.9 — from "clearly good" to "close to transparent" — and it costs 4000 of the 6000 total bits per second to do it. That is the price of the last increment of perceptual quality, and it is the reason stage 3 was given its own model, its own vocabulary, and its own chunking scheme.

Table I, the result the paper turns on

Read it three times, each time looking for something different.

First reading — the columns are anti-correlated. The rows that are good at ABX are bad at ViSQOL and vice versa, with no overlap and no middle. Semantic tokens err on 6.7% of within-speaker triplets; acoustic tokens err on 22.4%, more than three times as often. Semantic tokens reconstruct at ViSQOL 1.1; acoustic tokens at 3.3, which is the difference between "unusable" and "good."

Second reading — bitrate does not convert one into the other. Compare the two 6000 bps rows, which are matched bit for bit. Semantic: ABX 5.6/6.2, ViSQOL 1.4. Acoustic: ABX 17.8/26.6, ViSQOL 3.9. At identical information budgets the two representations remain in completely different regimes. Raising the semantic bitrate 24-fold bought 1.1 points of ABX and 0.3 points of ViSQOL. It did not turn semantic tokens into acoustic tokens. This is the negative result promised in Chapter 0, and it is what rules out the "just pick a middle bitrate" solution.

Third reading — the within/across gap. Compute it yourself; the paper never does, and it is the most revealing number on the page:

SHOWCASE — reconstruct from each, side by side

Now stop reading numbers and look at them. The sim below runs the same source clip down two paths simultaneously. Left column: the semantic path — w2v-BERT features, standardized, k-means-quantized, then decoded. Right column: the acoustic path — SoundStream encoder, RVQ, decoder. The top row shows what survives in the representation; the bottom row shows the reconstruction against the original.

Watch the phoneme track on the left survive the k-means bottleneck almost intact while the speaker/room track collapses to a flat gray — that is ViSQOL 1.1 drawn as a picture. On the right the speaker/room track comes through in full colour while the phoneme boundaries smear — that is ABX 22.4. Switch to matched 6000 bps and notice how little moves. Both sides get slightly sharper. Neither side crosses over.

Play with ABX directly

The second sim lets you build triplets yourself. Drag X around and watch the decision flip; toggle the representation between "semantic-like" (speaker axis compressed) and "acoustic-like" (speaker axis expanded) and see the same triplet score differently.

Two things to try. First, put X exactly halfway between A and B and nudge it — the verdict is knife-edge, which is what a hard triplet looks like. Second, set the representation to acoustic-like and push the speaker offset to maximum: the verdict flips even though you never moved X along the vowel axis. That is the 6.3-point across-speaker penalty, generated by your own hand.

The experiment that proves the negative half

Tables are one kind of evidence; a broken model is another. The paper does not merely argue that acoustic tokens alone are insufficient — it trains that model and reports what it sounds like.

The setup: flatten the acoustic token matrix Y in row-major order into a single sequence, train a decoder-only Transformer on it, prompt with 4 seconds of speech, and sample a continuation. The verdict, verbatim: "While both the recording conditions and the speaker identity from the prompt are preserved, the linguistic content is inconsistent, and often akin to babbling."

Every clause of that sentence is a result. Recording conditions preserved → the acoustic tokens carry room and channel. Speaker identity preserved → they carry voice. Linguistic content inconsistent → they do not carry meaning, or the LM cannot find it at that timescale. This is Chapter 0's babble problem, reproduced under controlled conditions with the paper's own components.

And note what they explicitly did not do: train the mirror-image model on semantic tokens alone. The reason is given in one clause — "we perform this on the acoustic tokens, since the semantic tokens only allow for poor audio synthesis." You cannot listen to a semantic-only model, because there is nothing to listen to. ViSQOL 1.1 is not a quality complaint; it is a statement that the audio is not there.

What each representation is actually optimized to keep

Step back and ask why the result comes out this way. It is not mysterious once you look at the training objectives.

The middle row of that table is the whole explanation. Masked prediction over long spans is an invariance-inducing objective: information that varies within a speaker's utterance but does not help predict the future is a liability, so the encoder learns to throw it away. Rate–distortion with an adversary is a fidelity-inducing objective: every bit that makes the output more convincing is worth keeping, and speaker identity is extremely audible.

These are not two settings of one dial. They are two different questions asked of the same signal, and the answers happen to be complementary. That complementarity is a gift of the objectives, not a design achievement — which is exactly why the authors had to measure it before building on it.

Why not just interleave the two streams?

Given two complementary token types, the most obvious combination is to interleave them into one sequence — z₁, y₁, z₂, y₂, … — and train a single language model. One model instead of three. Why did the paper not do that?

Section III-C gives two reasons, and the second is the practical one.

Reason one is statistical. The hierarchy "reflects the conditional independence assumption that semantic tokens are expected to be conditionally independent from past acoustic tokens given past semantic tokens." Written out: p(z_t | z_<t, y_<t) ≈ p(z_t | z_<t). If that is true, the interleaved model is spending its capacity attending to tokens that carry no information about the prediction it is making. Worse, they are the numerous tokens — acoustic tokens outnumber semantic ones 24 to 1 — so the semantic signal is diluted in a sea of irrelevant context.

Reason two is arithmetic. "The token sequence per stage is reduced compared to alternatives such as modeling the interleaved sequence of semantic and acoustic tokens, allowing for computationally more efficient training and inference." Do the numbers for ten seconds. Interleaved, all 12 quantizers: 250 semantic + 6,000 acoustic = 6,250 positions, and the model must handle all of it in one context. Split into stages: 2,250 for stage 2 and 1,800 for stage 3 (on 3-second chunks, batched). Attention cost of the interleaved model is 6,250² = 39.1 million pairs; the hierarchical version pays 5.06 million for stage 2 and 3.24 million per chunk for stage 3.

Nearly five times cheaper, and the saving grows with duration because the quadratic term dominates. The hierarchy is not merely an inductive bias; it is a compute decision that the inductive bias happens to justify.

Chapter 4 takes this apart properly, including the choice to split acoustic modeling into two stages rather than one. But you now have the shape of the argument: every structural decision in AudioLM is a sequence-length decision wearing a probabilistic costume.

How you would reproduce Table I

The measurement is more approachable than it looks. Sketch of the pipeline, end to end:

python — reproducing Table I, sketch
# --- 1. build both token streams for LibriSpeech dev-clean ---
Z = {u: semantic_tokens(wav) for u, wav in devclean}   # (Ts,)   25 Hz
Y = {u: acoustic_tokens(wav) for u, wav in devclean}   # (Ta, Q) 50 Hz

# --- 2. map each frame back to its quantized EMBEDDING (not the index) ---
Ez = {u: centroids[z]            for u, z in Z.items()}   # centroid per frame
Ey = {u: rvq.dequantize(y)      for u, y in Y.items()}   # sum of Q codewords

# --- 3. ABX on the embeddings, Libri-Light scripts, default settings ---
abx_within, abx_across = libri_light_abx(Ez)      # -> 6.7 / 7.6

# --- 4. ViSQOL needs audio, so train a decoder FROM the tokens ---
dec_z = train_soundstream_decoder(inputs=Ez, targets=wavs)
visqol_z = visqol(dec_z(Ez), wavs, mode="speech")      # -> 1.1

Step 2 is the one people get wrong. ABX is a distance metric, so it needs vectors, not integers — the codebook index 511 has no geometry. You must map each frame back to the embedding its token represents: the k-means centroid on the semantic side, the sum of chosen codewords on the acoustic side. That is what the paper means by "each frame is represented by its corresponding centroid for w2v-BERT or by the output of a SoundStream quantizer."

Step 4 is the expensive one: a whole decoder has to be trained just to make the semantic side measurable. That is not a formality — it is the fairest possible attempt to extract audio from a code that does not want to give any.

Inline check

(a) Semantic tokens at 6000 bps have better ABX than at 250 bps. Why does AudioLM use the 250 bps version anyway?
Because the improvement is 1.1 ABX points for 24× the sequence length. Stage 1 would go from 750 positions to 18,000 for a 30-second crop, and attention cost would rise 576-fold, to buy phonetic discriminability the downstream stages do not consume directly.

(b) If someone reports a new tokenizer with ABX 6.0 and ViSQOL 3.5, what have they achieved?
They have collapsed AudioLM's premise: one code with both properties, which would make the whole hierarchy unnecessary. Be suspicious, then check the bitrate and the evaluation split — and check whether ABX was computed across speakers.

(c) The paper trains an LM on acoustic tokens alone and reports babbling. Why is that experiment necessary rather than obvious?
Because Table I measures the representation, not the model. It is logically possible that a big enough Transformer could recover linguistic structure from acoustic tokens despite their poor ABX — the information is degraded, not absent (22.4% error is far below chance). The babbling result closes that gap empirically: at this scale, it does not.

A note on where "semantic" is and is not the right word

The paper calls one token type semantic, and the word does real work in the argument. It also overstates the case, and being precise about that will save you confusion later.

What the measurements actually establish is that these tokens are strongly phonetic (ABX 6.7) and largely speaker-invariant (across−within gap 0.9). Phonetic and speaker-invariant is not the same as semantic. A token stream that perfectly encoded phoneme sequences while understanding nothing would score exactly the same on both metrics.

The justification for the stronger word comes later, from Chapter 8's probes: a language model over these tokens prefers real words to non-words 71.5% of the time and grammatical sentences to ungrammatical ones 64.7% of the time. That is evidence about lexicon and syntax — which is closer to semantics, though still not identical to it.

The most defensible reading: semantic tokens are a phoneme-like code whose sequence statistics carry linguistic structure. The structure is in the sequence, not in the individual token. Token 511 does not mean anything; the transition from 511 to 88 to 903 does. That is exactly how text tokens work too, and it is why the analogy holds.

Keep the distinction because it predicts where the framework will strain. Anything that is linguistically meaningful but not phonetically marked — irony, reference, discourse structure — is not obviously carried by these tokens, and the paper never claims it is.

What would falsify the complementarity claim

A claim worth believing is one you can say how to break. Three results would force a retreat from this chapter's thesis; none has been observed in the AudioLM setting.

Keep the list. It is also a research agenda: the second row in particular has been probed repeatedly by later work, and the answer has stayed "representation, not capacity" at the scales anyone has tried.

Concept and realization: what actually flows

Close the chapter by tracing the data, since a table of metrics is not yet an implementation.

Notice the asymmetry in the middle steps. The semantic path replaces each frame with one centroid — a single 1024-dimensional vector chosen from 1024 options. The acoustic path replaces each frame with a sum of twelve centroids, each chosen from its own 1024-entry codebook. That is 1024 possible frame values versus 1024¹². The representational capacity per frame differs by a factor beyond astronomical, and the frame rate differs by 2× on top of it.

Said that way, ViSQOL 1.1 versus 3.9 stops being surprising and starts being arithmetic. What is surprising — genuinely, and it is the paper's real find — is that the 1024-option-per-frame code is the one that knows what the sentence means.

Chapter 3: Building the Two Tokenizers

Chapter 2 measured the two token types. This chapter builds them. By the end you will have hand-computed a residual vector quantization from raw numbers, derived every rate in the paper from the sample rate, and seen exactly why a single line of preprocessing — subtract the mean, divide by the standard deviation, per dimension — is described as "significantly" improving phonetic discriminability.

Take the acoustic side first, because it is the one where the arithmetic is unforgiving.

SoundStream, from sample rate to token rate

SoundStream's encoder is a stack of strided convolutional blocks. The paper gives the configuration exactly: "4 convolutional blocks having strides (2, 4, 5, 8)." Each stride divides the temporal resolution. Multiply them:

So the encoder emits one embedding per 320 input samples. At 16 kHz:

That is the whole derivation, and it is worth doing once by hand because every other number in the acoustic pipeline hangs off it. Ten seconds of audio → 160,000 samples → 500 embeddings. Three seconds → 48,000 samples → 150 embeddings.

The paper calls this "a 16000 / 50 = 320-fold reduction in the sampling rate." Notice it is a reduction in rate, not in information: each of those 50 embeddings per second is a real-valued vector of substantial dimension. The information reduction happens in the next step.

Why one codebook cannot possibly work

Vector quantization replaces a continuous vector with the index of its nearest entry in a learned codebook. Simple, and it has a fatal scaling property.

To hit 6000 bits per second at 50 frames per second, each frame must carry

A single codebook carrying 120 bits per frame needs 2¹²⁰ entries. Write that out: approximately 1.3 × 10³⁶ codewords. There is not enough matter in the solar system to store the codebook, let alone data to train it. Even the modest 2000 bps configuration needs 40 bits per frame — 2⁴⁰ ≈ 1.1 × 10¹² entries. Still impossible.

Residual vector quantization, the algorithm

Four lines, and they are the entire method:

The consequence that matters for AudioLM: the layers are ordered by importance. Layer 1 captures the bulk of the vector's energy; layer 12 captures a whisper. Truncating after layer 4 gives a valid, coarser reconstruction — which is precisely what stage 2 predicts and stage 3 refines. The paper states the interpretation directly: "tokens from the coarse quantizers recover acoustic properties like speaker identity and recording conditions, while leaving only the fine acoustic details to the fine quantizer tokens."

Hand-worked RVQ: every intermediate step

Two dimensions, three quantizers, four codewords each. Small enough to do on paper, structurally identical to the real thing.

The input vector (think of it as one 20 ms SoundStream frame, radically simplified):

The three codebooks (learned; here just given):

Notice the codebooks shrink in scale: coarse entries have magnitude ~0.5–0.9, mid ~0.1, fine ~0.03. That is not a coincidence — it is what training produces, because each codebook is fit to the residual distribution left by the previous one, and residuals get smaller.

Layer 1 — quantize x. Squared distance to each C¹ entry, every arithmetic step:

Winner: index 0, at squared distance 0.0369, i.e. distance √0.0369 = 0.1921.

Residual after layer 1: r¹ = x − C¹₀ = (0.62 − 0.50, −0.35 + 0.50) = (0.12, 0.15). Its norm is √(0.0144 + 0.0225) = √0.0369 = 0.1921, which is the same number — the residual norm is the quantization error. That identity is worth pausing on: in RVQ, the error you make at one layer becomes the input to the next.

Layer 2 — quantize r¹ = (0.12, 0.15).

Winner: index 0, d² = 0.0029, d = √0.0029 = 0.0539.

Residual after layer 2: r² = r¹ − C²₀ = (0.12 − 0.10, 0.15 − 0.10) = (0.02, 0.05).

Layer 3 — quantize r² = (0.02, 0.05).

Winner: index 2, d² = 0.0001, d = 0.0100.

Residual after layer 3: r³ = (0.02 − 0.01, 0.05 − 0.05) = (0.01, 0.00).

The reconstruction. Sum the three chosen codewords:

The tokens. Three integers: (0, 0, 2). Six bits total, at log₂4 = 2 bits per layer. From a 2D real vector to six bits, with 1.4% relative error.

Watching the error collapse

Roughly 11 dB per layer, which is a useful rule of thumb: each residual quantizer buys about two bits of effective precision. It also makes the coarse/fine split legible. The first layer alone removes 73% of the vector's magnitude. In the real codec, Q′ = 4 layers remove enough that what remains is inaudible as structure — you can hear that it is compressed, but you can identify the speaker, the room, and the words. That is why stage 2 stops at 4.

Flattening, with the same three numbers

Chapter 4 does this properly, but the arithmetic belongs here while the tokens are fresh. The three tokens (0, 0, 2) come from three different codebooks, and "0 from codebook 1" is a completely different object from "0 from codebook 2." To put them in one sequence for a language model, the paper adds an offset:

With Q = 3 layers and N = 4 codewords, the offsets for the three positions are 0·4 = 0, 1·4 = 4, 2·4 = 8. Applying them:

In the real system, Q′ = 4 coarse layers with N = 1024 gives a coarse acoustic vocabulary of 4096 symbols, and the fine stage's 8 layers give 8192. Same trick, bigger numbers.

The same computation, three ways

Form one was arithmetic on paper. Form two is the loop, written so every line maps to a step above:

python — RVQ from scratch, step by step
import numpy as np

x = np.array([0.62, -0.35])

C = [np.array([[ 0.50, -0.50], [ 0.80, 0.10], [-0.60, 0.40], [ 0.00, 0.90]]),
     np.array([[ 0.10,  0.10], [-0.10, 0.05], [ 0.05,-0.15], [-0.05,-0.05]]),
     np.array([[ 0.02,  0.02], [-0.02, 0.04], [ 0.01, 0.05], [-0.01,-0.03]])]

r, codes = x.copy(), []
for q, Cq in enumerate(C):
    d2 = ((Cq - r) ** 2).sum(axis=1)      # squared distance to every codeword
    k  = int(d2.argmin())                   # greedy nearest
    codes.append(k)
    r  = r - Cq[k]                          # the residual becomes the next input
    print(f"layer {q+1}: code={k}  residual={r}  norm={np.linalg.norm(r):.4f}")

# layer 1: code=0  residual=[0.12 0.15]  norm=0.1921
# layer 2: code=0  residual=[0.02 0.05]  norm=0.0539
# layer 3: code=2  residual=[0.01 0.00]  norm=0.0100

xhat = sum(Cq[k] for Cq, k in zip(C, codes))    # [0.61 -0.35]
Q, N = len(C), len(C[0])
flat = [k + (i % Q) * N for i, k in enumerate(codes)]  # [0, 4, 10]

Form three is the library call, which does all of the above plus codebook learning, exponential-moving-average updates, and dead-code restarts:

python — the one-liner
from vector_quantize_pytorch import ResidualVQ

rvq = ResidualVQ(dim=2, num_quantizers=3, codebook_size=4)
quantized, indices, commit_loss = rvq(x)     # indices -> tensor([0, 0, 2])

# SoundStream's real configuration, for comparison:
#   ResidualVQ(dim=D, num_quantizers=12, codebook_size=1024)
#   at 50 Hz that is 50 * 12 * log2(1024) = 6000 bits per second

Three forms, one computation. If the third one ever surprises you, drop back to the first.

Feel it: the residual staircase

Drag the target far from every coarse codeword and watch the first residual stay large — the later layers cannot fully recover, because they were trained for small residuals. That is quantization error you can see, and it is why codebook coverage matters more at layer 1 than anywhere else.

Every rate in the paper, derived

Every cell is the same formula: rate × Q × log₂(codebook). The piano row is the interesting one — the authors traded quantizer depth for codebook width, using 3 layers of 16,384 entries instead of 12 layers of 1,024. The paper says this "already provides high reconstruction quality," so they skip stage 3 entirely for music. Chapter 8 returns to why piano tolerates that and speech does not.

Now the semantic side: w2v-BERT

The other tokenizer is a 0.6-billion-parameter Conformer trained with two self-supervised objectives at once: a masked language modeling loss (predict discretized targets for masked spans) and a contrastive loss (pull true continuations closer than distractors). No transcripts. No labels.

Its output geometry: "w2v-BERT performs downsampling along the temporal dimension, so that real-valued 1024-dimensional feature vectors are computed at a sampling rate of 25 Hz (one every 40 ms)." Hence

Twice the temporal stride of SoundStream, which is where the "factor of 2" in Figure 2's caption comes from — for every semantic token there are two acoustic frames, hence 2Q′ coarse acoustic tokens.

Why an intermediate layer, and which one

Here is a choice that looks arbitrary and is not: the paper takes activations from the 7th layer of the MLM module, not from the final layer.

The intuition comes from layer-wise analyses of self-supervised speech models. Early layers stay close to the acoustics — they still encode speaker and channel. Late layers specialize toward the pre-training objective's own targets, which are not necessarily phonetic. The phoneme-like abstraction peaks somewhere in the middle. Figure 3 (left) of the paper plots ABX for layers 6 through 9 and shows exactly this shape: a minimum, with worse scores on either side.

The selection procedure is stated honestly and is refreshingly unglamorous: "we adopt a set of heuristics for choosing the intermediate layer to quantize and the number of k-means clusters K. Namely, we inspect ABX, sWUGGY and sBLIMP scores computed for different layers… In addition, we performed a small subjective evaluation test by listening to a few continuations." Three quantitative probes plus listening. The winner: layer 7, K = 1024.

The one line of preprocessing that changes everything

"We found that normalizing w2v-BERT embeddings such that each dimension has zero mean and unit variance before clustering significantly improves their phonetic discriminability."

That sentence is easy to skim. It should not be. k-means uses Euclidean distance, and Euclidean distance is dominated by whichever dimensions have the largest scale. If one dimension of the w2v-BERT feature encodes something loud and non-phonetic — overall gain, a speaker-correlated bias — it will dominate every distance computation, and the clusters will partition speakers rather than phonemes.

Watch it happen in three dimensions reduced to two. Let dimension 1 be a large-scale, speaker-correlated feature and dimension 2 be a small-scale, phoneme-correlated one. Three frames:

Distances before normalization. d(a, b): differences (0.4, −2.0), squares 0.16 and 4.00, sum 4.16, root 2.0396. d(a, c): differences (8.0, 0.1), squares 64.00 and 0.01, sum 64.01, root 8.0006.

So a is nearly four times closer to b — a different phoneme from the same speaker — than to c, the same phoneme from a different speaker. Run k-means with K = 2 on this data and you get one cluster per speaker. Your "semantic" tokens are speaker IDs.

Now standardize. Say over the corpus dimension 1 has mean 104 and standard deviation 5, while dimension 2 has mean 0 and standard deviation 1:

Distances after normalization. d(a′, b′): differences (−0.08, 2.00), squares 0.0064 and 4.0000, sum 4.0064, root 2.0016. d(a′, c′): differences (−1.60, −0.10), squares 2.5600 and 0.0100, sum 2.5700, root 1.6031.

The verdict has flipped. a is now closer to c — same phoneme, different speaker — than to b. k-means will cluster by phoneme. One preprocessing step, one changed answer, and the difference between a semantic tokenizer and an expensive speaker-ID system.

Turn the nuisance scale to maximum on the raw view and cluster: the boundary is vertical, splitting speakers. Switch to the standardized view and cluster again: the boundary rotates to horizontal, splitting phonemes. Same data, same algorithm, different metric — because standardization is a change of metric.

k-means, and what "the token" finally is

python — the complete semantic tokenizer
import numpy as np
from sklearn.cluster import MiniBatchKMeans

# 1. features from an INTERMEDIATE layer, not the last
F = w2v_bert.forward(wav, return_layer=7)       # (Ts, 1024) float, 25 Hz

# 2. per-dimension standardization — mu/sigma fit on the CORPUS, not the clip
F = (F - mu) / sigma                            # (Ts, 1024)

# 3. k-means fit once, offline, on a large sample of frames
km = MiniBatchKMeans(n_clusters=1024).fit(F_corpus)

# 4. the semantic tokens ARE the centroid indices
z = km.predict(F)                                # (Ts,) int in [0, 1023]

# 5. dedup: collapse consecutive repeats (stages 1 and 2 only)
z = z[np.insert(np.diff(z) != 0, 0, True)]

Step 2 deserves one emphasis. The mean and standard deviation are corpus statistics, computed once and frozen. Standardizing per-clip would be a subtly different — and worse — operation: it would remove exactly the between-clip variation you might want, and it would make the tokenizer non-causal within a clip.

And now the answer to "what is a semantic token, really": it is the index of the nearest of 1,024 centroids, in a standardized 1,024-dimensional space, of the 7th-layer activation of a masked-language model, computed every 40 ms. Nothing more mystical than that. The paper notes the lineage plainly: "our proposal for the extraction of semantic tokens from w2v-BERT resembles the token extraction from HuBERT in prior works."

Inline check

(a) A 3-second stage-3 chunk: how many acoustic frames, and how many fine tokens?
48,000 samples / 320 = 150 frames. Fine layers Q − Q′ = 8, so 150 × 8 = 1,200 fine tokens, plus 150 × 4 = 600 coarse tokens as conditioning: 1,800 positions.

(b) Why is codebook 3 in the hand-worked example so much smaller in magnitude than codebook 1?
Because it is fit to the residual distribution left after two layers, and residuals shrink by roughly 11 dB per layer. A fine codebook with coarse-sized entries would overshoot every residual it was asked to represent.

Chapter 4: The Hierarchy

We have two token streams and we know what each one carries. The remaining question is the one every reader asks at this point and the paper answers in a single dense subsection: why three models? Two would seem enough — one for semantics, one for sound. Why is the acoustic half split in two?

The answer is sequence length, twice over, dressed as two conditional-independence assumptions. This chapter takes the dressing off.

The three factors, with every symbol named

Section III-C writes down three conditional distributions. Here they are, followed by a full symbol key, because the superscripts and subscripts do a lot of work.

Three observations that unlock the notation. First, z appears without a time subscript in stage 2: it is a complete prefix, not a running condition. Stage 2 knows the whole sentence before it renders the first 20 ms. Second, y^<q_t means generation is ordered within a frame: quantizer 1's token is chosen, then quantizer 2 conditioned on it, and so on. Coarse before fine, at every timestep. Third, stage 3's conditioning contains no z at all. That absence is the paper's second big assumption, and it is worth two sentences of its own: "considering that fine acoustic tokens are conditionally independent from semantic tokens when conditioned on coarse acoustic tokens, the third stage can ignore the semantic tokens, which reduces the total sequence length."

Flattening a matrix into a sentence

A language model consumes a 1D sequence. The acoustic representation is a 2D matrix: T_A frames by Q quantizers. Something has to give, and the paper takes "the simple approach of flattening the acoustic tokens in a row-major order."

Row-major means: all quantizers of frame 1, then all quantizers of frame 2, and so on. Make it concrete with three frames and four coarse quantizers (Q′ = 4). Suppose the RVQ produced:

Row-major flattening gives the raw sequence

and now the problem from Chapter 1 bites. The 17 at position 1 came from quantizer 1; the 17 at position 5 also came from quantizer 1 (fine, same meaning) — but the 512 at position 4 came from quantizer 4 and the 512 at position 8 also from quantizer 4. What if a quantizer-2 code happened to be 17? The model would see the same symbol for two unrelated things.

The offsets fix it. The paper defines

where i is the position in the flattened sequence (1-indexed), Q the number of quantizers being flattened, N the codebook size. With Q = 4 and N = 1024 the offsets cycle 0, 1024, 2048, 3072, 0, 1024, 2048, 3072, …

Apply them to the twelve positions above, one at a time:

Now every symbol is unambiguous. Position 1's 17 and position 5's 17 are the same vocabulary entry, correctly, because they are both quantizer-1 codes. Position 3's 44 and position 11's 44 both map to 2092, also correctly. And nothing from quantizer 2 can ever collide with anything from quantizer 1, because their blocks are disjoint: [0, 1023] versus [1024, 2047].

The paper adds a parenthetical worth honoring: "In the following, we omit the offsets from the notation and assume proper offsetting implicitly." Every y^q_t you see afterwards secretly carries its offset.

The factor of two

SoundStream runs at 50 Hz; w2v-BERT at 25 Hz. Two acoustic frames per semantic token. Figure 2's caption states the consequence exactly: "for every semantic token there are 2Q′ acoustic tokens in the second stage and 2(Q − Q′) tokens in the third stage."

So the token budget per 40 ms of audio is: 1 semantic, 8 coarse, 16 fine. Twenty-five tokens to describe forty milliseconds. Twenty-four of them are about how it sounds; one is about what it says.

Now the arithmetic that justifies three stages

The paper's argument for splitting acoustic modeling in two is stated in one sentence — "we adopt the solution with two separate stages to limit the sequence length that the model has to process at once" — and then supported by two assumptions. Put numbers on it.

The merged alternative. One acoustic stage predicting all 12 quantizers, conditioned on semantics, on a 10-second crop:

The split version. Stage 2 on a 10-second crop, stage 3 on 3-second chunks:

Two models whose largest context is 2,250, versus one model at 6,250. And the split has a second, larger benefit that the pair-count comparison understates: because stage 3's chunks are independent, they can be batched. Generating 30 seconds of fine detail is ten independent 3-second problems solved in parallel, not one 18,000-position problem solved serially.

The paper spells out that scaling property: performing stage 3 on non-overlapping 3-second chunks allows "us to scale this stage independently of the target audio sequence length as well as to use more residual quantization layers Q to achieve higher quality." Read the second clause again. The chunking is not only a compute saving — it is what makes Q = 12 affordable at all. A merged stage at Q = 12 on 30-second crops would be an 18,750-position sequence.

See the tape

Slide Q′ to 1 and stage 2 becomes tiny while stage 3 balloons; slide it to 11 and the reverse. Q′ = 4 is where both stay inside a few thousand positions — and, per Chapter 2, where the coarse reconstruction is already at ViSQOL 3.3, meaning speaker and room are fully captured before the split.

The two assumptions, and what would break them

That second failure mode is not hypothetical. Non-overlapping 3-second chunks mean the fine detail at t = 2.99 s and t = 3.01 s are generated by different forward passes with no shared context. Any artifact at a chunk boundary is a direct consequence. The paper does not report boundary artifacts, and the piano configuration sidesteps the issue entirely by having no stage 3 — but it is the obvious place to look if you were reproducing this and heard a periodic click every three seconds.

Building the three sequences, in code

Everything above is twenty lines of tensor manipulation. Reading it removes any remaining ambiguity about what each model actually consumes.

python — constructing the training sequences for all three stages
import numpy as np

N, Q, Qp = 1024, 12, 4          # codebook size, total quantizers, coarse split

def flatten_with_offsets(Y):    # Y: (Ta, q_count) int
    Ta, qc = Y.shape
    off = (np.arange(qc) * N)[None, :]    # (1, qc): 0, 1024, 2048, ...
    return (Y + off).reshape(-1)             # row-major -> (Ta*qc,)

# ---- stage 1: semantic only ----------------------------------------
seq1 = z_dedup                                # (~750,) for a 30 s crop

# ---- stage 2: full semantic prefix, then flattened coarse ----------
coarse = flatten_with_offsets(Y[:, :Qp])   # (Ta*4,) = (2000,) for 10 s
seq2   = np.concatenate([z_dedup, coarse + SEM_VOCAB])
# loss is applied ONLY from the first coarse position onward
mask2  = np.arange(len(seq2)) >= len(z_dedup)

# ---- stage 3: coarse conditioning, then flattened fine -------------
# run on independent NON-OVERLAPPING 3-second chunks (Ta_chunk = 150)
c_chunk = flatten_with_offsets(Y[s:s+150, :Qp])   # (600,)
f_chunk = flatten_with_offsets(Y[s:s+150, Qp:])   # (1200,)
seq3    = np.concatenate([c_chunk, f_chunk + COARSE_VOCAB])
mask3   = np.arange(len(seq3)) >= len(c_chunk)

# no semantic tokens appear in seq3 at all — that is the whole point

Three things to notice in that code. The + SEM_VOCAB and + COARSE_VOCAB shifts are the same offset trick applied at a higher level, keeping conditioning symbols disjoint from target symbols. The loss masks encode "conditioning is not a target" — the paper's "with y¹₁ being the first token predicted during training." And seq3 is built from a slice Y[s:s+150], with no reference to anything outside that window.

Generation order, drawn

Within stage 2, the model walks the tape left to right. Because the tape is row-major, that walk has a specific rhythm worth internalizing:

The within-frame ordering is doing real work. Recall from Chapter 3 that RVQ layers are hierarchically dependent by construction — layer 2's codebook was fit to layer 1's residuals. Predicting q2 without knowing q1 would be predicting a residual without knowing what it is a residual of. Row-major flattening makes that dependency the shortest possible attention hop.

What this hierarchy is not

Two nearby ideas it is easy to conflate with AudioLM's cascade. Distinguishing them sharpens what is actually being claimed.

The third row is the one to keep. AudioLM's hierarchy has a seam in it. Below the seam (stages 2 and 3) is a genuine residual hierarchy where each level refines the previous in the same vector space. Above the seam (stage 1) is a different representation entirely, connected to the rest only by a learned mapping in stage 2. The paper's contribution is the seam.

The deduplication wrinkle, faced honestly

Chapter 0 flagged this; here is the full version.

Figure 2's caption says there are exactly 2Q′ acoustic tokens per semantic token, which presupposes a fixed 1:2 time alignment. Section IV-B says: "in the first two stages, we follow the previously proposed practice of removing consecutive repetitions of the semantic tokens."

These cannot both be literally true of the sequences the model sees. If a phoneme spans five 40 ms frames and lands in the same cluster all five times, deduplication turns five semantic tokens into one — while the corresponding 200 ms of audio still has ten acoustic frames and forty coarse tokens. The ratio is now 40:1, not 8:1, and it varies from segment to segment.

The consistent reading: the semantic tokens function as an unaligned prefix, not a synchronized track. Stage 2 gets the sequence of distinct semantic units and must work out the timing itself, from the acoustic history and from whatever prosodic information the semantic sequence implicitly carries. The 2Q′ statement describes the underlying frame rates, not the post-deduplication sequence.

Why not condition stage 3 on the semantic tokens anyway?

Suppose you disbelieved the conditional-independence assumption and wanted stage 3 to see z. What exactly would it cost?

The semantic prefix for a 3-second chunk is 75 tokens (fewer after dedup), so the chunk sequence grows from 1,800 to about 1,875 — a 4% increase. That sounds cheap. But it is not the cost that matters; it is the dependency.

Conditioning stage 3 on z means stage 3 can no longer be run on arbitrary coarse token sequences in isolation. You would need to carry the aligned semantic segment for every chunk, which reintroduces the alignment problem deduplication destroyed (Chapter 0's wrinkle), and you would couple stage 3's correctness to stage 1's output quality. Right now a stage-3 model is a pure function of coarse acoustic tokens; it can be retrained, replaced, or applied to coarse tokens from any source — including real audio — without touching the rest of the system.

That modularity is worth more than 4% of sequence length. It is also what made the piano configuration trivial: drop stage 3 entirely and nothing else changes.

Counting a real generation

Put it all together for the paper's headline task: a 3-second prompt continued for 7 seconds.

Four thousand three hundred and seventy-five autoregressive steps for seven seconds of audio. Compare: 112,000 steps if you generated waveform samples. Compare again: about 1,700 tokens for seven seconds of text at typical rates. AudioLM sits between text and waveform, roughly 2.5× the token cost of speaking the same content as text — which is a remarkably tight bound on "how much more expensive is sound than symbols."

Before the audit, one correction to a natural but wrong mental image: the three stages are not three resolutions of the same thing. Stages 2 and 3 are, but stage 1 lives in a different space entirely. Keep the seam visible.

A sanity check you can do in your head

Whenever you meet a hierarchical token system, run this three-line audit. It catches most misunderstandings.

Line 1 — what is the frame rate of each level, and what is their ratio? Here: 25 Hz and 50 Hz, ratio 2. That ratio is where every "factor of 2" in the paper comes from.

Line 2 — how many symbols per frame at each level, and from how many codebooks? Here: 1 symbol from 1 codebook of 1024 (semantic); 12 symbols from 12 codebooks of 1024 (acoustic). That asymmetry is why one is a vector and the other a matrix.

Line 3 — which levels are conditioning and which are targets, per stage? Here: stage 1 targets semantics with no conditioning; stage 2 conditions on semantics plus coarse history and targets coarse; stage 3 conditions on coarse and targets fine. Write that down and the three probability expressions reconstruct themselves.

The hierarchy as a set of interfaces

One more way to hold Chapter 4, which is the way you would hold it if you were building this rather than reading about it. Each stage boundary is an API, and each API has a contract:

Every successor in Chapter 9's lineage table is a substitution at exactly one of these four interfaces. That is why the architecture proved so durable despite every one of its components being replaced within two years.

Inline check

(a) Stage 2 sees all of z but only y_<t. Why the asymmetry?
Because z is conditioning — produced entirely by stage 1 before stage 2 runs — while y is what stage 2 is generating, so causality forbids seeing the future. Prefix conditioning is non-causal by construction; target modeling is causal by necessity.

(b) You set Q′ = 12 (no stage 3). What survives and what breaks?
Quality survives at full 6000 bps — but stage 2's 10-second sequence becomes 250 + 6,000 = 6,250 positions, you lose the chunked parallelism, and you can no longer scale Q without scaling the crop. This is the merged alternative, priced out above.

(c) Why must the offsets cycle with period Q rather than being applied per-frame?
Because the ambiguity being resolved is "which quantizer produced this code," and that is a property of the column, not the row. Frame index is already encoded by position in the sequence; quantizer index is not, so it has to go into the symbol.

The hierarchy as a summary table

Every column of that table has been derived from first principles in this chapter and the last. If you can reconstruct it from the sample rate, the stride product, Q, Q′, and the two conditional-independence assumptions, you understand AudioLM's architecture completely. What remains is watching it run.

Chapter 5: SHOWCASE — Watch Three Stages Run

Everything is assembled. Two tokenizers, three models, one flattening scheme, two conditional-independence assumptions. This chapter runs it — slowly, with the token streams visible — so that the sentence "AudioLM performs three subsequent stages" becomes something you have watched rather than something you have read.

Start with the exact procedure, because the paper's description of continuation is precise and every clause matters.

The continuation procedure, clause by clause

Setup. A prompt x of 3 seconds. Two indices name where the prompt ends: t_s in semantic frames and t_a in acoustic frames. For a 3-second prompt at 16 kHz:

Step 0 — tokenize the prompt. "We first map the prompt x to the corresponding semantic tokens z_≤ts and to the coarse acoustic tokens y^≤Q′_≤ta." So we extract 75 semantic tokens (fewer after dedup) and 150 × 4 = 600 coarse acoustic tokens. Note what we do not extract: the prompt's fine acoustic tokens are never used as conditioning anywhere.

Step 1 — semantic continuation. "The first stage generates ẑ_>ts, the continuation of semantic tokens autoregressively based on the conditioning z_≤ts." Stage 1 sees only the prompt's semantic tokens and extends them. It has no idea who is speaking or what room they are in; it does not need to.

Step 2 — coarse acoustic continuation. This is the step that carries the voice, and its conditioning list is longer than you might expect. "We concatenate the entire semantic token sequence (z_≤ts, ẑ_>ts) along with the coarse acoustic tokens of the prompt y^≤Q′_≤ta and feed it as conditioning to the coarse acoustic model, which then samples the continuations of the corresponding acoustic tokens."

Read that carefully. Stage 2's input contains three things: the prompt's semantic tokens, the generated semantic tokens, and the prompt's coarse acoustic tokens. The full semantic plan — past and future — plus 3 seconds of the speaker's actual voice.

Step 3 — fine acoustic. "In the third stage, we process the coarse acoustic tokens with the fine acoustic model." No prompt-specific handling is described, because stage 3 does not distinguish prompt from continuation. It chunks whatever coarse sequence it is given and fills in fine detail everywhere.

Step 4 — decode. "Finally, we feed both the prompt and the sampled acoustic tokens to the SoundStream decoder to reconstruct a waveform x̂." The output includes the prompt, re-synthesized through the codec — which is exactly why the subjective evaluation in Chapter 8 compresses the ground-truth samples through SoundStream too. Otherwise the codec artifacts in the first 3 seconds would give the game away.

The three inference modes, side by side

The same three trained models produce three behaviors depending on what you clamp. This is worth its own table because the modes get conflated constantly.

The middle column is the scientific instrument. By feeding real semantic tokens and resampling everything else, the authors isolate exactly one question: what information did the semantic tokens carry? Chapter 7 reads the answer off two probes.

SHOWCASE — the pipeline, running

Now watch it. The sim below is the paper's Figure 2 turned into a machine you can step through. The three token tracks are stacked; the prompt region is shaded; press Play and each stage fills in its continuation left to right, in the row-major rhythm from Chapter 4. The waveform at the bottom builds as the acoustic tokens arrive.

Four things to do with that sim, in order.

One. Play it once at default settings and watch the ordering. Stage 1 completes the entire semantic track before a single acoustic token appears. That is the "concatenate the entire semantic token sequence" clause made visual — stage 2 cannot start until stage 1 has finished, because it conditions on the whole plan.

Two. Step through and pause after stage 2. The waveform is already there, and already in the prompt's voice — but it is the 2000 bps version. Stage 3's contribution is the difference between that and the final trace, which is the ViSQOL 3.3 → 3.9 gap from Chapter 2.

Three. Turn the temperature up past 1.2 and replay. The semantic track loses its runs and its phrase-like grouping; the meaning blocks stop aligning. This is the diversity/consistency trade-off the paper's 0.6 was chosen to avoid, and it is why stage 1 gets the coldest temperature of the three.

Four. Toggle Ablate semantics. Stage 1 is disconnected and stage 2 runs on the acoustic prompt alone. The voice track — speaker colour, room shading — continues perfectly. The meaning track dissolves into unstructured fragments. That is the babbling result from Section III-B, reproduced in a picture: "both the recording conditions and the speaker identity from the prompt are preserved, the linguistic content is inconsistent, and often akin to babbling."

The same procedure, in code

Here is continuation written out with every conditioning list explicit. Compare it line by line against the four clauses above.

python — AudioLM continuation, with all conditioning explicit
def continuation(prompt_wav, seconds_out):
    # ---- step 0: tokenize the prompt ------------------------------
    z_p = semantic_tokens(prompt_wav)          # (ts,)      ts = 75 for 3 s
    Y_p = acoustic_tokens(prompt_wav)          # (ta, 12)   ta = 150
    coarse_p = Y_p[:, :4]                       # (150, 4) — the ONLY voice carrier

    n_new_sem = int(seconds_out * 25)           # 175 for 7 s
    n_new_ac  = int(seconds_out * 50)           # 350 frames

    # ---- step 1: semantic continuation, T = 0.6 -------------------
    z_hat = LM1.sample(prefix=z_p, n=n_new_sem, temperature=0.6)
    z_all = concat(z_p, z_hat)                  # (250,)

    # ---- step 2: coarse acoustic, T = 0.8 ------------------------
    # conditioning = ENTIRE semantic sequence + the prompt's coarse tokens
    cond2 = concat(z_all, flatten(coarse_p))     # 250 + 600 = 850 positions
    coarse_hat = LM2.sample(prefix=cond2,
                              n=n_new_ac * 4,       # 1400 tokens
                              temperature=0.8)
    coarse_all = concat(coarse_p, unflatten(coarse_hat, q=4))   # (500, 4)

    # ---- step 3: fine acoustic, T = 0.6, INDEPENDENT 3 s chunks ---
    fine_all = []
    for s in range(0, len(coarse_all), 150):        # 150 frames = 3 s
        chunk = coarse_all[s:s+150]                 # (150, 4)
        fine_all.append(LM3.sample(prefix=flatten(chunk),
                                  n=len(chunk) * 8,
                                  temperature=0.6))     # no z anywhere
    fine_all = unflatten(concat(*fine_all), q=8)     # (500, 8)

    # ---- step 4: decode prompt AND continuation together ---------
    Y_out = concat_cols(coarse_all, fine_all)        # (500, 12)
    return soundstream_decoder(Y_out)                # (160000,) waveform

Three lines deserve a second look. cond2 is 850 positions of pure conditioning before stage 2 predicts anything — 250 semantic plus 600 coarse. The stage-3 loop has no z in scope at all, which is the conditional-independence assumption enforced by code structure rather than by an argument. And the final decode takes the prompt's frames along with the generated ones, so the output waveform's first three seconds are a SoundStream re-synthesis of the original, not the original itself.

What stage 2 actually has to learn

It is tempting to think of stage 2 as a "vocoder with extra steps." It is not, and the distinction matters.

A vocoder maps a deterministic conditioning signal (mel-spectrogram, say) to a waveform, and the mapping is close to a function — the same mel gives essentially the same audio. Stage 2's mapping is one-to-many in the most extreme way: the same semantic sequence is compatible with every speaker in the world, every room, every microphone, every noise floor. The paper measures exactly this: resampling stage 2 on fixed semantic tokens produces "a wide variety of speakers and recording conditions."

So stage 2 is not learning a mapping; it is learning a conditional distribution whose entropy is enormous. Its job is to sample a coherent point from that distribution — pick a voice, and then stay with it for ten seconds. Consistency is the hard part, not selection.

And when a prompt is present, the task changes shape again: the coarse acoustic prefix collapses most of that entropy. The model is no longer choosing a voice; it is recognizing one from 600 tokens and extending it. Chapter 7's two numbers — 3.2% speaker accuracy without a prompt, 92.6% with one — are the same model doing these two very different jobs.

Unconditional mode, and why diversity counts as evidence

Run the cascade with no prompt at all and you get the mode the paper describes first: "we sample unconditionally all semantic tokens ẑ, which we then use as conditioning for acoustic modeling."

The reported behavior is worth quoting in full because each clause is a separate claim: the model "generates diverse, syntactically and semantically consistent linguistic content, with varying speaker identity, prosody, acoustic conditions."

Diversity here is not an aesthetic bonus. It is evidence about what was learned. A model that collapsed to one speaker would be telling you that the acoustic stage had memorized a mode rather than learned a distribution — which is exactly GSLM's limitation, imposed by its architecture. That AudioLM's unconditional samples vary in speaker and room and prosody says the coarse acoustic model represents those as genuinely free variables.

Meanwhile the linguistic content stays consistent within a sample. Free variation across samples, coherence within a sample: that pair is the signature of a correctly factorized model, and it is what Chapter 8's probes will quantify.

Walking the indices

To make the sim's bookkeeping concrete, here is the same 3-second-prompt, 7-second-continuation generation written as index ranges.

Two subtleties fall out of that table. First, the prompt's fine tokens are regenerated by stage 3, not copied — stage 3 is handed the coarse sequence and fills in fine detail across the whole thing, prompt included. Second, stage 3's chunk boundaries land at frames 150, 300, 450 for 3-second chunks, which means the first chunk boundary coincides exactly with the prompt/continuation boundary. Whether that is deliberate or a happy accident of the 3-second prompt length, the paper does not say.

What can go wrong, and what it sounds like

A cascade has characteristic failure signatures. Knowing them is most of debugging.

That last row is a genuinely informative failure. Why proper nouns specifically? Because they are low-frequency, high-entropy, and phonetically arbitrary — exactly the case where a 1024-cluster semantic vocabulary has the least support. A common word has thousands of training instances to pin down its semantic-token trajectory. A rare surname has a handful. The failure mode of a discrete bottleneck is always the tail.

The generation cost, honestly

Autoregressive means serial. Count the forward passes for a 7-second continuation, using Chapter 4's totals:

Two and a half thousand sequential forward passes through 0.3B-parameter models, for seven seconds of audio. This is not real-time, and the paper never claims it is. Real-time audio language modeling arrives later in this lineage, and it arrives by attacking exactly this number — which is why the flattening scheme was the first thing subsequent work replaced.

What you would see in a real implementation log

To make the abstraction concrete one last time, here is what a 7-second continuation looks like as a trace — the kind of output you would print while debugging.

trace — one continuation, annotated
[tokenize]  prompt 3.00 s -> wav (48000,) float32
[tokenize]  w2v-BERT layer7 -> (75, 1024) -> standardize -> kmeans -> z_p (75,)
[tokenize]  dedup            -> z_p (61,)          # 14 consecutive repeats collapsed
[tokenize]  soundstream enc  -> (150, D) -> rvq -> Y_p (150, 12) int16

[stage 1]   prefix 61 tok, sampling 175 tok, T=0.60
[stage 1]   done in 175 steps -> z_hat (175,), z_all (236,)

[stage 2]   cond = z_all (236) + flatten(Y_p[:, :4]) (600) = 836 positions
[stage 2]   sampling 1400 tok, T=0.80
[stage 2]   ctx grows 836 -> 2236; done -> coarse_all (500, 4)

[stage 3]   chunking 500 frames -> 4 chunks of 150 (last = 50)
[stage 3]   chunk 0: 600 cond + 1200 tgt   T=0.60   [parallel]
[stage 3]   chunk 1: 600 cond + 1200 tgt   T=0.60   [parallel]
[stage 3]   chunk 2: 600 cond + 1200 tgt   T=0.60   [parallel]
[stage 3]   chunk 3: 200 cond +  400 tgt   T=0.60   [parallel]
[stage 3]   done -> fine_all (500, 8)

[decode]    Y_out (500, 12) -> soundstream dec -> x_hat (160000,) float32
[decode]    3.00 s resynthesized prompt + 7.00 s generated
[total]     4375 tokens generated, ~2508 serial steps

Three things this trace makes visible that the prose does not. Deduplication removed 14 of 75 semantic tokens from the prompt — nearly 19%, which is typical and which is why the "2Q′ per semantic token" statement cannot be taken literally. Stage 2's context grows from 836 to 2,236 positions during generation, so its cost is not fixed. And the last stage-3 chunk is short (50 frames), because 500 does not divide evenly by 150 — a mundane implementation detail that becomes a bug if you assume uniform chunks.

Reading Figure 2 correctly

The paper's Figure 2 is a three-panel diagram, and it is the image most people remember from AudioLM. It is also slightly misleading if read casually, so here is what each panel actually asserts.

And the caption carries the one number the diagram cannot show: "the factor of 2 comes from the fact that the sampling rate of SoundStream embeddings is twice as that of the w2v-BERT embeddings." Every time you look at that figure, mentally attach 25 Hz to the top row and 50 Hz to the other two.

Inline check

(a) In acoustic-generation mode, why does the transcript stay fixed while the speaker changes?
Because z is clamped to ground truth (fixing content) while stage 2 samples the coarse acoustic tokens fresh with no acoustic prompt (freeing voice and room). The two are separable precisely because Chapter 2's measurements say they live in different codes.

(b) Could you preserve the speaker while replacing the content, using only these three models?
Yes — that is continuation with a substituted semantic sequence: feed the prompt's coarse acoustic tokens plus a semantic sequence taken from a different utterance. It is voice conversion, and it falls out of the architecture for free. The paper does not run this experiment, but the machinery is entirely present.

(c) The paper regenerates the prompt's fine tokens rather than reusing the originals. Does that matter?
It matters for evaluation: it means the first 3 seconds of an AudioLM output are not bit-identical to the source, so the human raters in Chapter 8 hear a re-synthesis throughout. It also means any stage-3 artifact appears in the prompt region too, which removes an obvious cue ("the audio gets worse at 3 seconds").

Concept and realization: the full data flow, one more time

Six objects, five transformations, two of them frozen. If you can redraw that diagram from memory with the shapes attached, you can implement AudioLM.

One caveat on the sim before we leave it: the token tracks are schematic. Real semantic tokens do not have a "height," and the bars are standing in for how structured versus random the sequence is. What the sim gets right is the ordering, the prompt boundary, the conditioning dependencies, and what disappears when stage 1 is removed.

Where the entropy is

A last quantitative way to see why the stages behave so differently: count the entropy each one is responsible for.

Note the inversion. The stage with the fewest bits carries the most uncertainty that matters, and the stage with the most bits carries the least. Stage 3 emits two thirds of the total bitrate while making almost no consequential decisions — which is exactly why it can be run on independent chunks, at a cold temperature, without semantics.

This also explains a practical asymmetry you would hit immediately in implementation: stage 1 is where sampling hyperparameters matter enormously and stage 3 is where they barely matter at all. Change stage 1's temperature from 0.6 to 1.0 and the output becomes incoherent. Change stage 3's and almost nothing audible happens.

And it reframes the 0.6 / 0.8 / 0.6 schedule one final time. The hottest temperature sits on the stage whose uncertainty you want to exercise — the one that chooses a voice. The two cold temperatures sit on the stage that must not wander (structure) and the stage that has nothing to wander about (detail).

A final note on cost. Nothing in this chapter's procedure requires the stages to run on the same machine, in the same process, or even in the same week. Stage 1 could run on a phone; stage 3 could run as a batch job. The cascade is a pipeline in the Unix sense, and that is a deployment property most monolithic generative models do not have.

It also means you can inspect the intermediate. Print the semantic tokens and you have a machine-readable transcript-of-sorts of what the model decided to say, before any audio exists. No other point in the system offers that.

The one thing to take from this chapter

If Chapter 2 is the paper's empirical core, this chapter is its mechanical core, and it reduces to a single asymmetry.

Structure is generated first and completely; sound is generated second and conditionally. Stage 1 finishes the entire semantic sequence before stage 2 emits a single acoustic token. That ordering is not an implementation convenience — it is the reason the output has long-horizon coherence at all. A model that decided what to say and how to say it simultaneously, token by token, would drift, because each local acoustic choice would constrain the remaining plan.

Separating them means the plan is fixed before rendering begins, and rendering cannot corrupt it. Every coherent generative system in any modality does some version of this, and AudioLM's version is unusually legible because the two stages are literally different models with different vocabularies.

Next: the training recipe that produced these three models, the data that made it robust, and the two tasks the whole framework is judged on.

Chapter 6: Training and the Continuation Task

An architecture is a hypothesis; a training recipe is what makes it true. This chapter covers what AudioLM was trained on, how, for how long, and how the two evaluation tasks are defined. It also does an arithmetic exercise the paper does not: counting the tokens each stage actually sees, which turns out to say something surprising about how over-trained these models are.

The data choice, and why it is a result

Everything — SoundStream, w2v-BERT, the k-means quantizer, and all three Transformers — is trained on the unlab-60k train split of Libri-Light: 60,000 hours of English speech, unlabeled, derived from public-domain audiobooks.

The paper immediately draws a contrast that is easy to read as a footnote and is actually a finding: "While previous works use the 6k-hour clean subset of Libri-Light for training the language model, AudioLM shows strong performance when trained on the more diverse and noisy unlab-60k subset."

Then the sentence that states why anyone should care: "The increased robustness to the quality of the training data reduces the data preparation effort needed to apply our framework."

That is a claim about deployability, not about scores. Data cleaning is the dominant cost of applying a speech system to a new domain, and "you can skip it" is worth more than a point of WER. But it is worth asking why AudioLM tolerates noisy data when GSLM does not — the paper does not say, and the answer follows from the architecture.

Model selection: the two sweeps

Two hyperparameters define the semantic tokenizer, and both were chosen empirically. Figure 3 of the paper reports both sweeps.

Sweep 1 — which layer. ABX scores (within and across speaker) for the unquantized embeddings of MLM-module layers 6 through 9, on LibriSpeech dev-clean with scaled embeddings. The curve has a minimum; layer 7 wins.

Sweep 2 — how many clusters. sWUGGY and sBLIMP development-set scores for K ∈ {256, 512, 1024, 2048} at layer 7. K = 1024 is selected.

Note which metrics are used for which sweep. The layer is chosen with a phonetic probe (ABX), the vocabulary size with lexical and syntactic probes (sWUGGY, sBLIMP). That is a sensible division: the layer determines what kind of information the features carry, while K determines how finely that information is discretized — and over-fine discretization hurts language modeling, not phonetics.

Then the honest coda: "In addition, we performed a small subjective evaluation test by listening to a few continuations produced by the different choices." Three automated probes and a listening test. That is how these decisions are actually made, and it is good that the paper says so.

The training recipe, complete

Notice what the table does not contain: optimizer, learning rate, schedule, warmup, weight decay, gradient clipping. None of these appear in the paper. For a reproduction attempt that is a significant gap — 1M steps at batch 256 is a long run to guess a learning rate for.

Counting the tokens: an exercise the paper skips

Batch 256 for 1M steps is 256 million sequences per stage. Multiply by sequence length:

A trillion tokens of training, spread over three 0.3B-parameter models. Put that next to the compute-optimal heuristic of roughly 20 tokens per parameter and these models are trained 30–100× past that point.

Is that wasteful? Not here, and the reason is instructive. Compute-optimal scaling assumes data is the scarce resource and you are choosing how to split a fixed budget between model size and tokens. In this setting the constraint is inverted: audio data is effectively unlimited (60,000 hours re-cropped at random offsets is an enormous number of distinct sequences), and the model size is capped by the need to run three of them in a cascade at generation time. When data is free and inference cost is the binding constraint, over-training a small model is exactly right — you are buying quality per generated token, which is the thing you pay for forever.

How many times does the data get seen? 60,000 hours is 2.16 × 10⁸ seconds:

Random offsets mean these are not literal epochs — the same 30 seconds of audio cropped at a different offset is a genuinely different training sequence — but the order of magnitude is right. Stage 1 sees the corpus a few dozen times; stage 3 barely three.

Push the crop scale up and stage 2's attention bar runs away first — it has both the longest sequence and no chunking escape hatch. That is why 10 seconds, not 30, and it is the single tightest constraint in the whole training setup.

The data pipeline, in code

What actually happens between "60,000 hours of mp3-ish audiobook audio" and "a batch of 256 integer sequences" is worth writing out, because three of the steps are where the paper's choices live.

python — one training batch for stage 2
import numpy as np

CROP_S = 10                     # stage 2; 30 for stage 1, 3 for stage 3
SR     = 16000

def make_batch(clips, B=256):
    seqs = []
    for _ in range(B):
        clip  = random_choice(clips)                 # from unlab-60k
        start = np.random.randint(0, len(clip) - CROP_S * SR)
        wav   = clip[start : start + CROP_S * SR]      # RANDOM crop, fresh each epoch

        # --- frozen tokenizers, no gradient ---
        z = kmeans.predict((w2v.layer7(wav) - mu) / sigma)   # (250,)
        Y = ss_rvq.encode(ss_enc(wav))                       # (500, 12)

        # --- dedup: stages 1 and 2 only ---
        z = z[np.insert(np.diff(z) != 0, 0, True)]              # (~150-220,)

        # --- flatten coarse with offsets, concatenate ---
        coarse = flatten_with_offsets(Y[:, :4])              # (2000,)
        seqs.append(np.concatenate([z, coarse + SEM_VOCAB]))

    return pad_to_max(seqs)          # dedup makes lengths ragged

Three lines to dwell on. The random_crop is why "epochs" are fuzzy — a 10-second window starting at second 3.7 and one starting at 3.9 are different sequences. The tokenizers run inside the data pipeline with no gradient, which in practice means they are usually run once offline and cached, turning 60k hours of audio into a few tens of gigabytes of integers. And the dedup line makes sequence lengths ragged, which is why real implementations pad — a detail the paper never mentions but every reimplementation hits within an hour.

Could the three stages be trained jointly?

The paper trains them separately and never discusses the alternative. Worth thinking through, since it is the first question a reviewer would ask.

Mechanically, no. The stages are connected by sampled discrete tokens. Sampling is not differentiable, so there is no gradient path from stage 2's loss back into stage 1's parameters. You would need a straight-through estimator, Gumbel-softmax, or REINFORCE — each of which introduces variance or bias into a 1M-step run, for a benefit nobody has demonstrated.

Statistically, it would not help much. Each stage's training targets are ground truth extracted from real audio, not from the previous stage's output. Stage 2 learns p(coarse | semantics) from true semantic tokens, which is the correct conditional. Joint training would optimize the composite, which is only better if the composite has a different optimum — and under the conditional-independence assumptions, it does not.

Practically, it would be a disaster. Separate training means three independent 16-TPU jobs that can run in parallel, restart independently, and be swapped out one at a time. Joint training means one job holding 0.9B parameters plus two frozen encoders, with a sampling step in the middle. The engineering argument is decisive on its own.

The two tasks, defined

AudioLM is evaluated on exactly two tasks, chosen "in order to showcase the general applicability of the framework… from different audio domains."

Both task definitions have the same two-part structure: preserve something from the prompt, invent something new that is consistent with it. And in both, the preserved part maps to acoustic tokens and the invented part to semantic tokens — which is why one framework serves both without modification.

The generalization requirement is stated explicitly and is stronger than it looks: "As the speech and piano prompts we use for evaluation are respectively from unseen speakers and unseen performances, generating consistent continuations requires AudioLM to generalize beyond training data." Unseen speakers means the 92.6% speaker-preservation result of Chapter 7 cannot be memorization — there is no embedding for that speaker anywhere in the model.

How a prompt is made

Mundane and worth stating, because it is where a reproduction usually goes wrong: "For generating the prompts, we truncate samples to the desired prompt length, extract the corresponding w2v-BERT and SoundStream tokens and use them as conditioning."

Truncate the waveform, then tokenize. Not: tokenize the whole clip and truncate the tokens. The difference matters at the boundary — SoundStream's convolutional encoder has receptive field, so the last few frames of a truncated waveform are computed from a partially-empty context, whereas frames from a full clip would have seen the future. Truncating first is the honest choice; it is what you would have at inference time with a live prompt.

What is missing from the recipe

A short, honest list of things you would need and would not find:

None of this makes the paper less good; systems papers from industrial labs routinely omit these. It does mean that "reproduce AudioLM" is a research project rather than an engineering task, which is worth knowing before you start.

Two framing notes before the recipe details. First, "training AudioLM" is really five training jobs, not three: SoundStream, w2v-BERT, the k-means quantizer, and then the three Transformers. The paper describes the last three in detail and cites the first two. Second, all five see the same corpus, which is a deliberate choice and the subject of the next-but-one section.

Keep that five-job picture in mind whenever the paper says "we train AudioLM." The phrase covers a lot of ground.

What "60,000 hours" actually is

Numbers like "60k hours" go by too fast. Sit with it.

That last line is worth its own moment. The tokenizer turns 13.8 terabytes of waveform into about 20 gigabytes of integers — a 690-fold reduction — and the language models never see anything else. Tokenization is not only a modeling decision; it is what turns an infrastructure problem into a laptop-sized array.

Semantic tokens alone are smaller still: 2.16 × 10⁸ s × 25 × 10 bits ≈ 6.8 GB, before deduplication. Six years of human speech, reduced to a file you could hold in RAM. That is the compression Chapter 0 promised, applied to a corpus.

Libri-Light, and what its shape implies

Libri-Light is derived from LibriVox — public-domain audiobooks, read by volunteers. Three properties of that source shape everything downstream, and none of them are neutral.

The first row explains something that otherwise looks like a limitation of ambition. AudioLM does not do dialogue not because dialogue is hard for the architecture but because there is no dialogue in the training data. The same three models trained on conversational audio would be a different system with the same equations.

Inline check

(a) Why is stage 1 trained on 30-second crops when generation only ever needs 10?
Because long-horizon structure is stage 1's entire job. Training on 30 s exposes the model to discourse-level dependencies it would never see at 10 s — and with relative position embeddings, a model trained long generalizes down to short contexts for free.

(b) The k-means quantizer is trained on unlab-60k, not on clean speech. What would go wrong if it were fit on clean data and applied to noisy data?
Distribution shift in the quantizer: noisy frames would land far from every centroid, assignments would become unstable, and the semantic stream would carry channel artifacts as spurious token changes. Fitting on the deployment distribution is the whole reason the noisy-data robustness holds.

(c) Stage 3 sees the corpus only ~3.6 times while stage 1 sees it ~36 times. Is that a problem?
Probably not, and for a structural reason: stage 3's task is local. Fine acoustic detail given coarse detail is close to a per-frame regression, so the number of effectively independent training examples is the number of frames (2.16×10⁸ × 50 = 10¹⁰), not the number of 3-second windows. Long-horizon tasks need many passes over long windows; local tasks do not.

Two more observations before that. First, the recipe is notable for what it does not include: no curriculum, no scheduled sampling, no auxiliary losses, no distillation. Three plain cross-entropy runs. Second, the three runs are independent, so wall-clock time is one run, not three, if you have the hardware — which is part of why the separate-training decision is so comfortable.

The absence of tricks is itself informative. When a system works with the plainest possible training recipe, the credit belongs to the representation rather than to the optimization. That is the honest reading of AudioLM: the hard thinking went into what to predict, not into how to fit it.

What you would need to change to train this on your own data

Suppose you had 5,000 hours of some other audio domain — podcasts, bird song, industrial machinery — and wanted an AudioLM for it. What actually has to change?

The second row is where the project either works or does not, and the paper's piano experiment is the only evidence that it transfers. Piano is a favorable case: it has a discrete symbolic layer (notes) that a self-supervised model can plausibly discover, exactly as speech has phonemes. Domains without such a layer — ambient noise, machinery — may have no semantic level to find, in which case the hierarchy degenerates to an acoustic-only model and you are back to babbling.

Which suggests a diagnostic to run before committing: fit k-means to your candidate semantic features, then measure whether an autoregressive model over those tokens has substantially lower perplexity than a unigram model. If it does not, the tokens carry no structure worth cascading, and the extra stage is pure cost.

Concept and realization: what a training step costs

Make one gradient step concrete, for stage 2, so the hardware numbers stop being abstract.

Roughly 10²¹ floating-point operations for one of three stages. The 6×N×T rule of thumb (two FLOPs per parameter for the forward multiply-accumulate, doubled again for the backward pass) is worth memorizing — it turns any stated (params, tokens) pair into a compute estimate in one line, and it is how you sanity-check whether a reported training run is plausible on the reported hardware.

Note also what the estimate does not include: the attention term, which at 2,250 positions is not negligible, and the cost of running the frozen tokenizers over 60,000 hours, which is a one-time preprocessing job that in practice takes a meaningful fraction of the total.

Setup comparison: AudioLM versus its closest ancestor

Read the last two rows together and something interesting appears. On resynthesis accuracy the two systems are essentially tied — GSLM is slightly better on characters, AudioLM slightly better on words. On linguistic knowledge AudioLM is far ahead, especially on syntax (64.7 versus 57.1). And on everything acoustic, they are not comparable, because GSLM has one voice.

The honest summary: AudioLM did not beat GSLM at GSLM's own task by much. It made GSLM's task a subproblem of a larger one, and solved the larger one. That is usually how progress looks.

One more thing the data section quietly settles: the tokenizers are trained on the same corpus as the language models. That is not automatic — you could tokenize with an off-the-shelf encoder trained elsewhere — and it matters, because a k-means quantizer fit on clean speech and applied to noisy speech would produce unstable assignments. Fitting everything on unlab-60k is what makes the robustness claim hold rather than merely being asserted.

It also means the tokenizers have seen every hour the language models have seen. There is no held-out-data story here for the tokenizers; the evaluation sets (LibriSpeech dev-clean, test-clean) are separate corpora, but the tokenizers are as adapted to the training distribution as anything can be.

What the recipe implies about reproduction difficulty

Ranked from easiest to hardest, if you set out to rebuild this today:

Easy: the three Transformers. Standard decoder-only stacks with published dimensions; any modern framework does this in an afternoon.

Moderate: the acoustic tokenizer. Open RVQ codecs exist and are well documented; retraining one on 60k hours is a compute problem, not a research problem.

Hard: the semantic tokenizer. You need a 0.6B self-supervised speech model trained on the same corpus, the right intermediate layer, corpus-level standardization statistics, and a k-means fit whose details are unstated. Every one of these is a lever with no documented setting.

Impossible: the piano results. The dataset is internal.

That ordering is typical of industrial systems papers and worth internalizing: the parts that look hardest (the models) are usually the easiest to reproduce, and the parts that look like footnotes (preprocessing statistics, clustering details, data splits) are where reproduction actually fails.

Chapter 7: What Each Token Type Carries

Chapter 2 measured the token types with generic probes: phonetic discriminability and reconstruction quality. Useful, but indirect. This chapter runs the direct experiments — the ones that take generated audio and ask two specific, falsifiable questions: are these the same words? and is this the same person?

The design is beautiful in its simplicity. Both experiments use the same generation mode (acoustic generation from ground-truth semantic tokens), the same generated audio, and two off-the-shelf classifiers pointed at it. Same stimulus, two probes, opposite answers.

Experiment 1: do the words survive?

The hypothesis under test, stated by the paper: "when modeling speech, the linguistic content is mostly captured by the semantic tokens, while speaker identity and recording conditions are captured by the acoustic tokens."

The setup. Take real speech. Extract its ground-truth semantic tokens. Throw the audio away. Run stages 2 and 3 to synthesize new audio conditioned only on those tokens. Then transcribe the result with an ASR system and compare against the original transcript.

If the semantic tokens carried the content, the transcript comes back. If they did not, it will not.

Table II, the result:

Now decompose it, because the four columns form an error budget that the paper describes but does not tabulate.

Two conclusions fall straight out, and the paper states both.

First: "the semantic content is fully captured by the semantic tokens, as the transcripts obtained from the output of acoustic generation closely follow the original transcripts." A 6.0% word error rate means 94 of every 100 words came back. From a 250 bps code, with the actual audio discarded and re-invented in a different voice. That is a strong statement about what those 10 bits per 40 ms contain.

Second: the codec is not the problem. "The error rates of the SoundStream reconstruction are comparable to those of the original audio, suggesting that most of the errors are coming from the mapping of semantic to acoustic tokens." 0.1 points versus 3.4 points. If you wanted to improve AudioLM's intelligibility, the codec is the wrong place to look.

Where those 3.4 points actually go

The paper names three error sources by inspection, and each one teaches something different.

That third one is a genuinely subtle measurement artifact and deserves a beat. The whole point of the acoustic stage is that it produces varied, realistic recording conditions — including noisy ones. The ASR system was not asked to be robust to that; it just transcribes what it hears, badly, when there is background noise. So part of the 6.0% WER is not AudioLM failing at content; it is AudioLM succeeding at acoustic diversity in a way that degrades the measuring instrument.

Both metrics are about to appear as bare numbers, so it is worth building them from scratch first. Neither is complicated; both are routinely misread.

WER and CER by hand — and why their ratio is a clue

Both metrics are edit distances, normalized differently. Compute one of each on a real LibriSpeech sentence, using exactly the failure mode the paper names.

Reference (test-clean, an actual utterance): "mister quilter is the apostle of the middle classes" — 9 words, 43 letters, 8 spaces, 51 characters.

Hypothesis (a plausible proper-noun failure): "mister quilt her is the apostle of the middle classes".

Step 1 — word-level alignment. Line them up:

Step 2 — word error rate. WER = (S + D + I) / N, where N is the number of reference words:

Step 3 — character error rate. At the character level the two strings differ by a single inserted space:

Step 4 — read the ratio. WER / CER = 22.2 / 2.0 = 11.3. One misplaced space cost eleven times more in words than in characters, because word-level metrics have no partial credit: quilt is as wrong as zebra.

Now look at AudioLM's actual ratio: 6.0 / 3.4 = 1.76. And the original audio's: 2.5 / 0.8 = 3.13. AudioLM's ratio is lower than the clean-audio baseline, which tells you its extra errors are spread more evenly across characters rather than concentrated in word-shredding boundary mistakes. Whatever is going wrong in the semantic→acoustic mapping degrades phonemes broadly rather than destroying occasional words — consistent with "a bit of everything is slightly off," not "some words are catastrophically wrong."

python — WER and CER, from scratch
def edit_distance(ref, hyp):
    # classic Levenshtein DP over token lists
    d = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)]
    for i in range(len(ref) + 1): d[i][0] = i
    for j in range(len(hyp) + 1): d[0][j] = j
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            cost = 0 if ref[i-1] == hyp[j-1] else 1
            d[i][j] = min(d[i-1][j] + 1,       # deletion
                          d[i][j-1] + 1,       # insertion
                          d[i-1][j-1] + cost)  # substitution
    return d[-1][-1]

ref = "mister quilter is the apostle of the middle classes"
hyp = "mister quilt her is the apostle of the middle classes"

wer = edit_distance(ref.split(), hyp.split()) / len(ref.split())   # 2/9  = 0.2222
cer = edit_distance(list(ref),  list(hyp))  / len(ref)          # 1/51 = 0.0196

# library one-liner:  import jiwer;  jiwer.wer(ref, hyp), jiwer.cer(ref, hyp)

The ASR probe's own blind spots

Before trusting Table II completely, note what a WER probe cannot see.

It cannot see prosody. A continuation that says the right words with entirely wrong stress and phrasing scores identically to one that sounds natural. Nothing in Table II rewards intonation.

It cannot see semantic plausibility beyond the reference. In acoustic-generation mode there is a reference transcript, so this is fine. But WER is useless for evaluating continuations, where there is no ground truth for the invented part — which is exactly why Chapter 8 needs entirely different instruments.

And it inherits the ASR system's own biases. Conformer Transducer-L was trained on a particular distribution; audio that is unusual in ways unrelated to intelligibility (odd room, unusual voice) will transcribe worse. The paper acknowledges this for background noise; the same argument applies to any acoustic condition the ASR system finds unfamiliar.

Experiment 2: does the speaker survive? (And should it?)

Now the mirror image. Same generated audio, different probe.

The classifier. The paper builds a speaker classifier from scratch and describes it in unusual detail, which is welcome because it is reused for the safety classifier in Chapter 9.

Two architecture choices worth noticing. The separable 3×1 and 1×3 kernels factorize a 3×3 convolution into a time pass and a frequency pass — cheaper, and it lets the network treat the two axes differently, which is right for spectrograms where time and frequency are not interchangeable. The 1-second crop with 250 ms hop aggregation means the classifier never needs a long-context model: speaker identity is a local property, decidable from a second of audio, and voting over windows handles the rest.

Table III, the result:

Read the middle row against chance. With 291 speakers, random guessing gives 100/291 = 0.34%. The paper is precise about this: "while higher than chance (3.2% compared to 100 / 291 = 0.3%), the speaker classification accuracy remains low."

So 3.2% is not zero. It is roughly 9× chance. The semantic tokens carry a small but real amount of speaker information — which is exactly what Chapter 2's across−within ABX gap of 0.9 points predicted. Not zero, just small. The paper's conclusion is calibrated accordingly: "the semantic tokens carry little information about the speaker identity, which is instead mostly determined by the acoustic tokens."

The continuation experiment, in detail

The 92.6% figure comes from its own protocol, described in Section IV-F.

"Excluding the prompts" is load-bearing. If the classifier were run on the whole 10 seconds it would see 3 seconds of essentially-real audio and could score well on that alone. Running it only on the generated 7 seconds means the number reflects generation, not copying.

And remember the generalization constraint from Chapter 6: the prompts come from test-clean, whose speakers are not in the training set of the language models. There is no stored speaker embedding to retrieve. The system is reconstructing a voice it has never heard, from 600 coarse acoustic tokens, and holding it for seven seconds.

Three attributes, three verdicts

The paper's experiments plus one qualitative observation give a complete assignment of attributes to token types. This table is the chapter's takeaway:

The prosody row is the interesting one, and it is the only qualitative entry — the paper reaches it "based on a subjective assessment done by comparing the synthesized samples generated from the same semantic tokens."

Think about why prosody would land on the semantic side. Semantic tokens are extracted at 25 Hz with deduplication. Deduplication destroys absolute duration but preserves relative ordering; and the k-means clusters, being fit to contextualized w2v-BERT features, are themselves sensitive to stress and position. So the semantic stream encodes something like "which syllables are prominent and in what order," which is most of what we hear as intonation, while leaving the fine timing to the acoustic stages. It is a split nobody designed and it happens to be roughly correct.

Run acoustic generation twenty times and the speaker counter hovers near 3%; the transcript strip barely moves. Switch to continuation and the speaker counter climbs past 90 within a handful of draws. You are watching a factorization work.

Before the pattern, one detail from the ASR setup worth flagging: the acoustic generation is repeated three times per sample, and the reported CER/WER average over all of them. That means Table II's numbers already incorporate the variability of the acoustic stage, rather than reporting a single lucky draw. Small choice; it is the difference between a number and an estimate.

The two probes also differ in an underappreciated way: one has a ground-truth reference (the transcript) and the other has a ground-truth label (the speaker). Both are available only because the generation was conditioned on real audio. In pure continuation mode neither reference exists for the generated portion, which is why Chapter 8 needs entirely different instruments.

Two experiments, one design pattern

Step back from the numbers and look at the shape of what was done, because it is reusable.

Step 3 is the part most often done badly. The temptation is to build a bespoke probe — train a classifier on the representation and report its accuracy. That measures decodability, which is a property of the probe as much as of the representation. Using an independent, pre-existing system that was never trained on your representation removes that degree of freedom entirely. Conformer Transducer-L does not know AudioLM exists; it just transcribes audio.

And step 2 is the part that requires the architecture to cooperate. You can only clamp one factor and free the other if the model exposes them separately — which is exactly what the hybrid tokenization provides. A single-code system offers no such handle, which is why nobody runs this experiment on a monolithic model: there is nothing to hold fixed.

What this does and does not prove

Be precise about the logical status of these results, because they are stronger than typical representation analyses and it is worth knowing why.

What is proved. These are interventional experiments, not correlational ones. The paper does not train a probe on semantic tokens and report that speaker identity is hard to decode — that would only show the information is not linearly accessible. Instead it resamples the acoustic tokens and observes the speaker change. Manipulating a variable and observing the downstream effect is causal evidence.

What is not proved. That the semantic tokens contain no speaker information — 3.2% is nine times chance. That the split is clean for other attributes: emotion, accent, and speaking rate are never measured, and the paper's own broader-impact section worries that "generated speech continuations might not be consistent with the prompt in terms of accent and dialect for underrepresented groups." That is an admission that the factorization is imperfect precisely where it matters most.

What is untested entirely. Whether any of this holds for music. There is no piano analogue of Table II or Table III. The claim that semantic tokens capture "melody and rhythm for music" is supported only by a preference test (Chapter 8), not by a probe.

Inline check

(a) Why does the SoundStream reconstruction score 100.0% on speaker classification but only 0.9 CER — why is the codec transparent to one probe and not the other?
It is transparent to both, roughly. 0.9 CER versus 0.8 for the original is a 0.1-point degradation — the codec is nearly lossless for ASR too. The 100.0% is what "nearly lossless" looks like when the metric saturates.

(b) Acoustic generation is run three times per sample. Why three?
Because a single sample from a high-entropy conditional is a poor estimate of that conditional's behavior. Three draws per source clip average out the luck of any individual generation — and for the speaker experiment, they demonstrate the variety directly, since three draws usually give three different speakers.

Before that, one number to keep in perspective: 291 speakers is a small closed set. The classifier is not doing speaker verification in the open world; it is choosing among 291 known identities. That makes 92.6% impressive but not directly comparable to a speaker-verification equal-error rate, and it is why the paper cites ASVspoof separately when discussing biometric spoofing rather than claiming its own numbers bear on it.

Reading Table III's first row

The row nobody discusses is the most important control in the chapter: SoundStream reconstruction → 100.0% speaker accuracy.

Without it, the 3.2% result is ambiguous. Maybe the classifier simply fails on codec-processed audio; maybe passing anything through SoundStream destroys speaker identity, and AudioLM's generation has nothing to do with it. The control rules that out completely: real audio, compressed through the same codec at the same bitrate, is still classified perfectly.

So the drop from 100.0% to 3.2% is attributable entirely to resampling the acoustic tokens, which is the intervention under study. And the recovery from 3.2% to 92.6% is attributable entirely to supplying an acoustic prompt. Two clean attributions, made possible by one control row.

The result in one sentence each

Table II: regenerate audio from nothing but the semantic tokens and an ASR system still recovers 94% of the words, so the semantic tokens carry the content.

Table III: do the same and a speaker classifier recovers the original speaker 3.2% of the time — barely above chance — so the semantic tokens do not carry the voice; add three seconds of coarse acoustic tokens and it recovers 92.6%, so they do.

Two sentences, two tables, one factorization confirmed from both sides. Everything else in this chapter is the careful work that makes those two sentences trustworthy: the codec baseline that shows compression is not the confound, the 291-speaker chance level that calibrates 3.2%, the "excluding the prompt" clause that stops the 92.6% from being copying, and the honest note that ASR is degraded by the model's own acoustic diversity.

A quick note on why these two probes and not others. An ASR system and a speaker classifier are the two most mature, most standardized, most widely-trusted audio classifiers in existence. Choosing them is not laziness — it is choosing instruments whose behavior the reader already understands, so the result does not depend on trusting a probe the authors built.

If you wanted to extend this analysis, the same logic tells you where to look next: emotion recognition and accent classification are the two other reasonably mature off-the-shelf probes, and both would test attributes the paper explicitly worries about and never measures.

The one experiment that would tie it all together

Chapter 5's challenge box sketched voice conversion; here it earns a second mention because it is the natural completion of this chapter's argument.

Take speaker A's coarse acoustic prompt and speaker B's semantic tokens. Generate. Then run both probes: the ASR system against speaker B's transcript, and the speaker classifier against speaker A. Clean factorization predicts roughly 6% WER and roughly 92% speaker accuracy, simultaneously, from a single generated clip.

That would upgrade the claim from "each token type mostly carries X" to "the two are independently controllable" — which is the property every downstream application (TTS, voice conversion, dubbing) actually needs. Every component required to run it is described in the paper. The result is not there, and every successor system in Chapter 9's lineage table is, in effect, a demonstration that it works.

Chapter 8: Does It Know English? (And Piano)

Chapter 7 established that semantic tokens carry linguistic content. This chapter asks a harder question: does the language model over those tokens know anything about English — lexicon, syntax — or has it merely learned to produce plausible token sequences?

The distinction matters. A model could reproduce transcripts faithfully in acoustic-generation mode (where ground-truth semantics are supplied) while being incapable of generating a grammatical sentence on its own. Table II tests the mapping; this chapter tests the mind.

Two zero-shot probes from the Zero Resource Challenge

Both metrics come from the ZeroResource Speech Challenge 2021, and both work the same way: present the model with a pair, one good and one bad, and check which one it assigns higher probability. No training, no fine-tuning, no classifier head. Pure likelihood comparison.

The datasets, exactly: 10,000 pairs for sWUGGY and 6,300 pairs for sBLIMP, from the challenge development sets, "each synthesized using four voices." Four voices matters — a model that scored well on one voice and badly on others would be measuring something acoustic rather than linguistic.

sWUGGY is reported twice: on all pairs, and on an in-vocab subset "pre-filtered to contain words that occur in the LibriSpeech data." The gap between them tells you how much of the score depends on having actually heard the word during training, versus general phonotactic plausibility.

One practical note before the mechanics: both probes are evaluated on synthesized audio, not on recordings. The challenge organizers render each text stimulus with four TTS voices, so the model is being asked to score speech it has never heard a human produce. That is a deliberate design choice — it guarantees the two members of each pair differ only in the intended contrast, with no confound from different recording sessions, speakers, or noise.

It also means a residual risk the paper does not discuss: if the model's semantic tokenizer behaves oddly on synthetic speech, all four voices share that oddity, and the scores would be systematically affected. Nothing suggests this happened, but it is the kind of assumption worth naming.

The normalization problem, worked out

Here is a subtle methodological point that the paper handles correctly and that costs it a headline number — which is the mark of a paper worth trusting.

To score a pair you compute the model's log-likelihood of each sequence and pick the higher. But: "positive examples in the sBLIMP data are on average shorter than their negative counterparts, which can implicitly bias scores towards higher success rates."

Why does length bias the comparison? Because a log-likelihood is a sum of per-token log probabilities, every one of which is negative. A longer sequence has more negative terms, so it accumulates a lower total — regardless of how good it is. Longer is penalized, mechanically.

Watch it produce a wrong answer. Suppose the grammatical sentence "the dogs sleep" becomes 12 semantic tokens and the ungrammatical "the dog sleep" becomes 10, and the model — which does know grammar — assigns:

Unnormalized comparison: −27.0 > −30.0, so the model "prefers" the ungrammatical sentence. Marked wrong. But look at the per-token numbers: the model finds every token of the correct sentence more probable on average. It knew the answer; the metric asked the wrong question.

Normalized comparison: −2.50 > −2.70, so the correct sentence wins. Marked right.

The paper's decision: "we normalize the log-likelihood returned by the model by the sequence length in all experiments." Applied uniformly, to every model, including their own.

Table IV, complete

Three readings, again.

First — AudioLM wins every no-text-supervision column. sWUGGY all: 71.5 versus 70.9 for the best prior (HuBERT-only). In-vocab: 83.7 versus 80.0 (CPC-BERT). sBLIMP: 64.7 versus 59.9. The lexical margins are modest; the syntactic margin is not.

Second — the sBLIMP jump. The paper quantifies it as "improving by 8% relative over the previous state-of-the-art (CPC-BERT)." Check the arithmetic:

And it clears the forced alignment topline at 63.7 — a BERT model trained on force-aligned ground-truth phonetic transcriptions. A model that has never seen text outscores a model trained on aligned phonemes, at judging English grammar. That is the sentence to remember from this table.

Third — the causal/non-causal split. This is the reading that is easiest to miss and most important. "Unlike AudioLM, the aforementioned models are not causal, so they are not well suited for speech generation." BERT and RoBERTa variants see the whole sequence bidirectionally; they can score a sentence but cannot generate one left to right. AudioLM beats them anyway, while being architecturally constrained in a way they are not. Compare only within the causal block and the margins widen further: 64.7 versus GSLM's 57.1 on sBLIMP.

Cycle through the sBLIMP rounds with normalization off and watch pairs where the model clearly prefers the grammatical sentence per token, yet loses on the total. Those are the rounds the footnote is about.

The human evaluation: 51.2%

Automated probes measure the linguistic half. For the whole thing — content, acoustics, and the absence of artifacts, judged together — the paper runs a listening test, and its design is worth studying because it closes every obvious loophole.

The result: "the rate of success for assigning the correct label (original vs. synthesized) is 51.2%, which, according to a binomial test, is not statistically significantly different (p = 0.23) from assigning labels uniformly at random (50% success rate)."

Note the direction of the statistics. The authors are not claiming a positive result; they are failing to reject the null that raters were guessing. With 1,000 ratings, a binomial test at 51.2% gives p = 0.23 — comfortably inside noise. The task tests three things at once, as the paper lists them: semantic and syntactic correctness of the content, acoustic coherence with the prompt, and absence of generation artifacts. Failing to detect any of the three, under those controls, is a strong composite result.

And the paper's own next sentence is the one that leads to Chapter 9: "Since human raters struggle to differentiate short speech samples synthesized by AudioLM from real speech samples in an unpaired setup, the responsible model development practices call for addressing this aspect systematically."

Checking the p-value yourself

"p = 0.23" is the kind of number worth recomputing, both to confirm it and to learn what test was used.

A 51.2% success rate over 1,000 ratings is 512 successes. Under the null hypothesis of pure guessing, the count is Binomial(n = 1000, p = 0.5), with

Step 1 — the z-score.

Step 2 — the tail probability. For a standard normal, P(Z > 0.759) ≈ 0.224. That is the one-sided p-value, and it rounds to the paper's 0.23.

Step 3 — note which test that is. A two-sided test would double it to ≈ 0.45. The paper reports the one-sided value, which is the right choice here: the alternative hypothesis of interest is "raters do better than chance," not "raters differ from chance in either direction." Nobody expected systematically sub-chance performance. Either way, both numbers are nowhere near any conventional threshold — you would need roughly 526 successes for one-sided significance at p = 0.05.

Which gives a useful sense of the experiment's resolution: with 1,000 ratings, this test could have detected a true detection rate of about 53% or higher. It cannot rule out a small real effect below that. "Indistinguishable" here means "indistinguishable at the resolution of 1,000 ratings," and the paper's careful phrasing — "not statistically significantly different" rather than "identical" — respects that.

Scoring a pair, in code

python — sWUGGY / sBLIMP scoring, exactly as described
import numpy as np

def seq_logprob(model, wav):
    """Log-likelihood of the SEMANTIC token sequence for one audio clip."""
    z = semantic_tokens(wav)                 # frozen tokenizer, then dedup
    lp = 0.0
    for t in range(1, len(z)):
        logits = model(z[:t])                  # stage-1 LM only
        lp += log_softmax(logits)[z[t]]        # each term is negative
    return lp, len(z)

def judge_pair(model, wav_good, wav_bad, normalize=True):
    lp_g, n_g = seq_logprob(model, wav_good)
    lp_b, n_b = seq_logprob(model, wav_bad)
    if normalize:                              # the paper's choice, everywhere
        lp_g, lp_b = lp_g / n_g, lp_b / n_b
    return lp_g > lp_b                         # True = scored correct

# sWUGGY: 10,000 pairs (word vs non-word), 4 voices each
# sBLIMP:  6,300 pairs (grammatical vs not), 4 voices each
score = np.mean([judge_pair(model, g, b) for g, b in pairs]) * 100

Two details this makes explicit. Only the stage-1 model is involved — sWUGGY and sBLIMP never touch the acoustic stages, so they are measuring the semantic language model in isolation. And the audio is synthesized from text by the challenge organizers using four voices, so a model with residual speaker sensitivity would score inconsistently across voices; averaging over the four is itself a small robustness check.

What these probes cannot see

Both metrics are pairwise likelihood comparisons on short stimuli. Three things they are blind to.

Long-range coherence. sBLIMP sentences are a few words long. Nothing here tests whether the model stays on topic across 30 seconds — which was Chapter 0's entire motivating problem. The evidence for long-horizon coherence in this paper is the human evaluation and the piano preference test, not these probes.

Semantics beyond syntax. "The dogs sleep" versus "the dog sleep" is a grammatical contrast. Neither probe asks whether the model prefers meaningful sentences to grammatical nonsense. The paper's claim of "semantically plausible speech continuations" rests on listening, not on measurement.

Anything acoustic. Both probes score semantic token sequences. A model with perfect sWUGGY and sBLIMP could still produce unlistenable audio — which is precisely GSLM's situation, differently. The two halves of the paper's claim are measured by entirely disjoint instruments, and that is a feature.

Beyond speech: piano

The framework was built for speech; the interesting claim is that it transfers. "We retrain all components of AudioLM on an internal dataset of 40k hours of piano music that includes players from beginner to expert level, and exhibits a wide range of different acoustic conditions, with content ranging from piano scale exercises to famous pieces."

What changed: exactly one thing. "The model hyperparameters are identical to the speech continuation setup, except for the acoustic generation stage: we found that a codec with 3 layers of quantization and a larger codebook size of 2¹⁴ per layer already provides high reconstruction quality, so the experiments on piano continuation ignore the third stage and directly predict the 3 levels of acoustic tokens in the second stage."

Read that table as a claim about the two signals. Piano needs less than half the bitrate of speech for comparable perceived quality, and reaches it with three wide codebooks instead of twelve narrow ones. Solo piano is spectrally sparser and more stationary than speech: a struck note is a decaying sum of harmonics, well described by a few large codewords, whereas speech is a rapidly-switching sequence of very different spectral shapes needing many fine corrections.

The piano result

The evaluation is a preference test, because there is no piano analogue of ABX or WER. Setup: 10 raters, 15 pairs of 20-second continuations, each pair being the same prompt continued by (a) a model trained on acoustic tokens only and (b) full AudioLM.

Result: raters preferred AudioLM in 83.3% of pairs.

And the qualitative observation that gives the number meaning: "While both are of equally high audio quality, analogously to the speech continuation experiments, only the latter display consistent melody and temporal structure."

Equal audio quality, different structure. That is Chapter 0's fidelity/coherence split, reproduced in a domain with no phonemes, no words, no syntax — and it is why the paper's conclusion generalizes the claim: hierarchical modeling "not only benefits speech generation by separating linguistic content from speaker identity, but more generally improves audio generation by explicitly disentangling the long-term structure and local acoustic details."

Flip between the two several times on the same prompt. The acoustic-only continuation is never ugly — every note sounds like a real piano — it is simply about nothing. That is the most precise available description of what long-term structure is: the difference between sounds and a piece.

Before ranking the evidence, one more piece of context on the human evaluation: the raters were told the first three seconds were real. That instruction makes the task harder for the model, not easier — it focuses attention exactly where the generation is, and it removes any chance of the prompt carrying the judgement. It is the kind of design choice that costs the authors nothing to omit and that they included anyway.

Reading the numbers as a hierarchy of evidence

This chapter has produced five different kinds of number, and they are not equally strong. Ranking them is a useful exercise.

None of these are dishonest. All of them are labelled correctly in the paper — the qualitative ones say "we observe" and "subjective assessment," the quantitative ones give protocols and sample sizes. The skill being practised here is reading those labels rather than flattening every number into "the paper showed."

And notice the pattern in what is strong: the strongest evidence comes from metrics the authors did not design, on datasets they did not build, compared against a leaderboard they did not curate. The weakest comes from listening to their own samples. That ordering is not a coincidence, and it generalizes to every empirical paper you will read.

What is missing from the music evaluation

Three honest gaps, worth naming because they bound the claim.

No Table I for music. There is no measurement of what piano semantic tokens carry versus piano acoustic tokens — no ABX analogue, no reconstruction comparison. The complementarity is assumed to transfer.

No absolute quality baseline. The comparison is against an ablation of itself, not against Jukebox, Perceiver AR, or a symbolic music model. "Better than our own ablation in 83.3% of pairs" is a statement about the hierarchy, not about the state of the art.

Solo piano only, from an unreleased dataset. The conclusion lists "polyphonic music" as future work, which in context means multi-instrument. And 40k hours of internal piano recordings cannot be reproduced by anyone outside Google.

Inline check

(a) Why is sWUGGY reported both on all pairs and on an in-vocab subset?
Because "all" mixes two abilities: recognizing words actually heard in training, and rejecting non-words on phonotactic grounds alone. The in-vocab subset isolates the first. AudioLM's larger margin in-vocab (83.7 vs 80.0) than overall (71.5 vs 70.9) suggests its advantage is more lexical memory than phonotactic generalization.

(b) The piano configuration has no third stage. What did it give up?
Nothing measurable, per the paper — 3 layers at 2¹⁴ "already provides high reconstruction quality." It also gave up the parallel-chunk trick, but with only 2 stages and 150 tokens/s the sequences were never the bottleneck.

(c) Why is the piano prompt 4 seconds rather than speech's 3?
The paper does not say. A plausible reason: musical structure has a longer minimum unit. Three seconds of speech contains several words — enough to establish voice and register. Three seconds of piano might not contain a full bar at a slow tempo, leaving metre and key underdetermined. Four seconds buys roughly two bars at moderate tempo.

Also worth noting what the four-voice synthesis buys. Each sWUGGY and sBLIMP stimulus is rendered by four different voices, and the score averages over them. A model whose semantic tokens retained speaker information would score inconsistently across the four — high for voices resembling its training distribution, low for others — and the average would drag it down. Voice-averaging is therefore a quiet second test of the speaker-invariance that Chapter 2 measured directly.

Why zero-shot probes are the right instrument here

One methodological point deserves emphasis because it is easy to take for granted.

sWUGGY and sBLIMP require no training whatsoever on the model being evaluated. No probe classifier, no fine-tuning head, no held-out split of the model's own outputs. You compute two likelihoods and compare them. That property is what makes the leaderboard comparison in Table IV meaningful across architectures as different as an LSTM on CPC features, a RoBERTa on visually-grounded representations, and a causal Transformer on k-means tokens.

Contrast this with the alternative that a less careful paper would have used: train a linear probe on the semantic tokens to predict phonemes or words, and report its accuracy. That number would confound three things — how much information is present, how accessible it is to a linear map, and how much data the probe got. Every one of those is a property of your probing setup rather than of the model.

The zero-shot design removes all three degrees of freedom. What is left is a statement about the model's own probability distribution, which is the thing you actually wanted to know.

The cost is that you can only ask questions expressible as a likelihood comparison between two stimuli. That is a real constraint — it is why there is no zero-shot probe for coherence, prosody, or acoustic quality in this paper, and why those had to be evaluated by ear.

The chapter in one line

A model that has never seen a letter of text judges English grammar better than a model trained on force-aligned phonetic transcriptions — and the authors report the smaller of their two possible numbers for it.

That is the sentence. The rest is protocol: 10,000 lexical pairs, 6,300 syntactic pairs, four voices each, length-normalized log-likelihoods, causal architecture competing against non-causal baselines. And on the music side, one preference test showing that the same hierarchy separates structure from surface in a domain with no words at all.

A closing observation about the piano configuration that is easy to skip. Dropping stage 3 for music is not a compromise — it is a demonstration that the architecture is modular in the way Chapter 4 claimed. The interface between stages 2 and 3 is a coarse token matrix; if your codec's coarse tokens are already good enough, stage 3 is simply unnecessary and nothing else changes. Two stages, one fewer model, identical everything else.

That single configuration change does more to support the interface argument than any amount of prose about conditional independence.

Connecting the two halves of the paper

It is worth stating explicitly how Chapters 7 and 8 fit together, because they are answering nested questions.

Chapter 7 asked: do the semantic tokens carry linguistic content? It answered yes, by supplying real semantic tokens and recovering the transcript. That is a statement about the representation.

Chapter 8 asked: does a language model over those tokens know English? It answered yes, by comparing likelihoods on word/non-word and grammatical/ungrammatical pairs. That is a statement about the model.

Both are needed. A representation that carries content but supports no learnable structure would pass Chapter 7 and fail Chapter 8 — you could resynthesize faithfully and never generate a sentence. A model with good likelihood statistics over a representation that carried nothing would pass Chapter 8 in some degenerate sense and fail Chapter 7. Only both together license the claim in the abstract: syntactically and semantically plausible speech continuations, with no transcript.

Chapter 9: Detection, Risk, and What Came Next

Two numbers from this paper sit next to each other and produce the most interesting question in it.

The same audio. One judge cannot tell at all; the other is nearly perfect. Understanding why is worth more than either number alone, and it generalizes far beyond this paper.

The detector, exactly as built

The paper's approach is deliberately unimpressive, which is the point: "we train a convolutional network with the same architecture as the one described in Section IV-D, but for the binary classification task of differentiating between original samples and continuations generated by AudioLM (excluding the prompt)."

Same six-block CNN from Chapter 7 — log-mel input, 25 ms window, 10 ms hop, 64 mel bins, channels [64, 128, 256, 256, 512, 512], 3×1 and 1×3 kernels. Only the output head changes: 291 speakers becomes 2 classes.

The handicap that makes the experiment honest

One design decision here is more instructive than the result. Why compress the real audio through SoundStream before training the detector? The paper explains:

"We compare continuations to original samples compressed through SoundStream rather than uncompressed audio, since otherwise i) the task is trivial (the model quickly converges to 100% accuracy) and ii) eventual compression artifacts would become a confounding factor that would prevent evaluating the generative abilities of AudioLM."

Read that as a chain of reasoning. An uncompressed-versus-generated detector reaches 100% instantly, because it learns to detect the codec, not the generation. That detector is useless: it would flag any SoundStream-compressed real recording as fake. To measure whether AudioLM's generation is detectable, both classes must go through the codec.

Why the machine wins and the human does not

The paper does not explain the gap. Here is the reasoning, which is worth having explicitly.

The two judges are answering different questions. The human is asked: does this sound like a person talking? That is a judgement about naturalness, integrated over seven seconds, using a perceptual system tuned by a lifetime of listening to real speech — and tuned specifically to what matters: intelligibility, speaker identity, emotional tone. The CNN is asked: does this one-second spectrogram come from distribution A or distribution B? That is a statistical question, and it does not care whether the difference is perceptible.

And there is a difference to find. Trace it. Real audio produces a sequence of RVQ codes that is a quantization of a real waveform. Generated audio produces a sequence of RVQ codes sampled from a language model at temperature 0.8 and 0.6. Those two distributions over code sequences are not the same, even when their decoded waveforms are perceptually equivalent. Temperature sampling below 1 systematically over-represents high-probability codes; the model's learned transition statistics are an approximation of the true ones. Decode both and the difference survives as a faint statistical texture in the spectrogram — inaudible, but perfectly learnable.

Human perception discards exactly that information. Our auditory system is a lossy, task-oriented encoder. It is superb at speaker identity and phoneme discrimination and largely blind to the fine statistics of spectral texture, because those statistics have never mattered for survival. A CNN trained on log-mel spectrograms has no such priorities: every bin is equally interesting.

Turn the codec equalization off and watch the detector's counter shoot to 100% within a few draws while the human counter does not move. That is the trivial-task failure mode the paper explicitly designed around, reproduced in ten seconds.

What the detector does not prove

98.6% is a strong result under specific conditions. Be precise about which.

The base-rate point deserves the arithmetic, because it is the one people skip. Suppose 1 in 10,000 clips in some stream is generated, and you run a detector with 98.6% accuracy on both classes across 1,000,000 clips:

Under 1% of flagged clips would actually be generated. That is not a criticism of the paper — the balanced setting is the right one for measuring whether a signal exists — but it is the difference between a scientific result and a deployable system, and the gap is four orders of magnitude of base rate.

One framing note before the comparison. The paper never uses the word "watermark," and it is worth being precise about what it did build: a classifier trained after the fact on outputs, not a signal embedded during generation. The distinction shapes every property in the table below.

Detector versus watermark: the road not taken

AudioLM ships a post-hoc detector: a classifier that examines audio and guesses its origin. The alternative — which arrived later in this lineage — is a proactive watermark: deliberately perturbing the generation so that an inaudible, robust signal is embedded in every output.

The last row of that table is the one that ends every optimistic conversation about provenance. Neither approach detects a generator you did not anticipate. A watermark tells you "this came from our system"; it says nothing about audio from someone else's. A learned detector generalizes a little further but not much, and degrades as generators improve.

Which makes the accidental-fingerprint result more interesting, not less. AudioLM's 98.6% comes from a fingerprint nobody designed — a byproduct of sampling from an imperfect model. As models get better, that fingerprint gets fainter. The 98.6% is a snapshot of a moving quantity, and the direction of travel is known.

One more consequence of the accidental-fingerprint framing: the detector is a moving target in a way a designed signal would not be. Every improvement to the generator erodes it, and no improvement to the generator erodes a properly designed watermark. That asymmetry is why the field moved toward embedded signals within a couple of years.

The arms-race arithmetic

Put a shape on that movement. A detector works when the generated distribution differs measurably from the real one within the detector's window. Two levers close that gap:

That second lever is an uncomfortable observation: the temperature schedule that makes AudioLM sound good is part of what makes it detectable. Sampling at T = 1 would produce a less skewed code distribution and a harder detection problem — at the cost of the "diversity versus semantic consistency" trade-off Chapter 1 described. Quality and detectability are coupled through the same knob, in the same direction. The paper does not run the detector at varying temperatures; it would be a two-hour experiment and a genuinely informative plot.

The broader impact, as the authors wrote it

Section VI is short and specific. Its structure is worth copying.

It names the upside first, concretely: "use-cases ranging from helping people with speech impediments to assisting in composing music."

It names inherited risks: "AudioLM inherits all concerns about language models for text, such as reflecting the societal biases in the underlying data."

It names a risk specific to this system: "the generated speech continuations might not be consistent with the prompt in terms of accent and dialect for underrepresented groups in the training data." This one is sharper than it looks. The whole selling point is that continuations preserve the prompt's characteristics. If that preservation degrades for speakers underrepresented in Libri-Light — which skews toward a particular set of English accents — then the system works better for some people than others in a way that is invisible to every metric in the paper. Table III's 92.6% is an average over test-clean speakers; no per-group breakdown is reported.

It names the misuse cases with citations: "spoofing biometric identification" (citing the ASVspoof challenge) and "impersonating a specific speaker" (citing YourTTS).

And it points to its own mitigation: "As an important step towards this direction, in Section IV-H we provide a model for accurately detecting audio synthesized by AudioLM."

What AudioLM became

The conclusion lists its own future work: "multilingual speech, polyphonic music, and audio events… as well as integrating AudioLM into an encoder-decoder framework for conditioned tasks such as text-to-speech or speech-to-speech translation." Nearly all of it happened, mostly within a year, and mostly by keeping this exact hierarchy and adding a conditioning signal on top.

Notice what every one of those inherits: audio tokens are a vocabulary, and generation is next-token prediction. And notice what most of them dropped: flattening. Row-major flattening multiplies sequence length by Q, and that is the first thing every successor attacked — VALL-E with parallel fine prediction, SoundStorm with masked parallel decoding, RQ-Transformer designs with a nested depth transformer. AudioLM's flattening was "the simple approach," as the paper says, and simple approaches are the ones that get replaced.

Where to go from here on this site

Five things to carry out of this paper

If a year from now you remember five things, make them these.

1. Complementary tokenizations beat compromise tokenizations. Table I's four rows show that matching the bitrate does not convert one representation into the other. When two objectives pull in opposite directions, look for two codes rather than a middle setting.

2. Bitrate is sequence length is compute. Every architectural decision in AudioLM — the coarse/fine split, the 3-second chunks, the crop lengths, even Q′ = 4 — is a sequence-length decision. Learn to read systems papers this way and half their choices become predictable.

3. Factorization buys robustness for free. Training on 60,000 hours of noisy audio works because the noise has somewhere to go. The same structure that made the model good made the data problem easy.

4. Interventional evidence beats probing. 3.2% versus 92.6% is a causal claim about where speaker identity lives, obtained by changing one thing and measuring the effect. It is worth more than any number of decoder probes.

5. Perceptual and statistical indistinguishability are not the same. 51.2% and 98.6%, on the same audio. This will keep being true, and it is the single most useful thing to know about detecting generated media.

Inline check

(a) Why does the detector train on 1-second crops when the samples are 7 seconds long?
Because the discriminative signal is a local spectral texture, not a long-range structure — one second contains plenty of it. Short crops also multiply the effective training set size and let the same window-and-aggregate inference trick from the speaker classifier apply unchanged.

(b) Suppose you improved AudioLM until its detector dropped to 60% accuracy. What would that tell you?
That the model's output distribution had moved substantially closer to the data distribution — a much stronger statement about model quality than any perceptual test could give. Detector accuracy is, inconveniently, one of the better available measures of generative fidelity.

Before the lineage, one honest note about scope: the successors below are a selection, not a survey. They are chosen because each one substitutes at a different interface, which makes the set useful for understanding what AudioLM was actually contributing.

What the successors reveal about AudioLM's real contribution

A good way to identify a paper's load-bearing idea is to look at what everyone kept and what everyone threw away.

Kept by everyone: audio as discrete tokens from a neural codec; a coarse/fine split with a language model per level; a short acoustic prompt as the carrier of voice identity; frozen tokenizers decoupled from the sequence model.

Thrown away by nearly everyone: row-major flattening; the semantic-token stage itself, whenever a better conditioning signal exists.

That second item is the interesting one. VALL-E replaces semantic tokens with text. MusicLM replaces them with a text-music embedding. AudioPaLM replaces them with a shared text-audio vocabulary. In every case, the semantic stage was a stand-in for a conditioning signal that was not available. AudioLM had no text, so it manufactured a text-like intermediate out of self-supervised features.

Which reframes the contribution. AudioLM is not primarily "here is how to use semantic tokens." It is "here is what the layer above the codec must provide, and here is proof that you can build it from audio alone if you have nothing else." The moment someone had something else — a transcript, a caption, a joint embedding — they slotted it into the same hole.

The cheat sheet

Every number and symbol worth carrying, in one place.

One structural observation first. The detector reuses the speaker classifier's architecture verbatim — same six blocks, same channels, same 1-second crops, same window-and-aggregate inference. Only the head changes. That is a small piece of engineering economy with a real methodological benefit: a network already shown to read speaker identity robustly from generated audio is a credible instrument for reading generation artifacts too, and reusing it removes one more free parameter from the experiment.

The safety result, restated for people who will cite it

This number gets quoted a lot, usually incorrectly. Here is the precise claim and its precise scope.

The two useful takeaways for anyone building in this space: perceptual indistinguishability arrived earlier than most people expected, and it does not imply statistical indistinguishability. Both halves matter, and quoting either without the other misrepresents the result.

A closing thought about the shape of the whole thing

AudioLM is, in the end, a paper about interfaces. It found that two independently-trained models — one built for speech recognition, one built for compression — produce representations that happen to partition audio almost exactly along the seam that matters. Neither was designed for the other. Nobody trained them jointly. The complementarity was discovered by measurement and then exploited by architecture.

That is a more interesting kind of result than "we scaled it up." It says that the objectives we already use — masked prediction and rate–distortion — carve nature at joints that are useful for generation, and that a substantial amount of progress is available to anyone willing to measure what existing components actually carry rather than assuming.

The Feynman test for this lesson is simple. Can you draw the three stages with their sequence lengths, name what each token type carries with a number attached, hand-compute one RVQ layer and its flattening offset, and explain both the 3.2/92.6 pair and the 51.2/98.6 pair to someone who has not read the paper? If yes, you can rebuild it.

And one closing note on how to read the successors. Every paper in the lineage table can be summarized as "AudioLM, with substitution X." Knowing which of the four interfaces from Chapter 4 was substituted tells you almost everything about what the paper does and what it inherits unexamined. It is a fast and surprisingly reliable way to read a whole subfield.

Reading list, with what each one adds

1. Borsos, Z. et al. "AudioLM: a Language Modeling Approach to Audio Generation." arXiv:2209.03143, 2022 — the paper. Read Section III-C twice; it is the densest half-page.
2. Zeghidour, N. et al. "SoundStream: An End-to-End Neural Audio Codec." IEEE/ACM TASLP, 2022 — where the acoustic tokens come from, including the losses AudioLM inherits without thinking about.
3. Chung, Y. et al. "w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-training." ASRU, 2021 — the semantic half.
4. Lakhotia, K. et al. "On Generative Spoken Language Modeling from Raw Audio." TACL, 2021 — GSLM, the closest ancestor and the sharpest contrast.
5. Dunbar, E. et al. "The Zero Resource Speech Challenge 2021." Interspeech, 2021 — sWUGGY, sBLIMP, and the leaderboard Table IV compares against.
6. Schatz, T. et al. "Evaluating Speech Features with the Minimal-Pair ABX Task." Interspeech, 2013 — where the metric of Chapter 2 comes from.
7. Kahn, J. et al. "Libri-Light." ICASSP, 2020 — the 60k hours, and the ABX scripts.
8. Hawthorne, C. et al. "General-purpose, Long-context Autoregressive Modeling with Perceiver AR." ICML, 2022 — the high-bitrate-tokens-only approach whose limitation motivated the hierarchy.