VALL-E: Speech as a Language

Chapter 1: The Reframe — Speech as Next-Token Prediction

Chapter 0 ended on an ambition: make speech synthesis a language modelling problem so that scale and prompting come along for free. Ambition is cheap. This chapter asks what it would actually take, and discovers that the obvious attempts fail for two different reasons — which is exactly why the solution needs the specific machinery it does.

A language model needs two things from its data: a finite vocabulary it can put a softmax over, and a sequence short enough that a transformer can attend across it. Text hands you both for free: 50,000 subwords, a few thousand of them per document. Speech hands you neither.

Attempt 1: model the raw waveform

Audio is stored as a stream of 16-bit integers. The paper opens its background section with exactly this observation: "a generative model is required to output 2¹⁶ = 65,536 probabilities per timestep to synthesize the raw audio."

Sixty-five thousand output classes is uncomfortable but not fatal — text models routinely carry vocabularies that size. The fatal part is the second sentence: "the audio sample rate exceeding ten thousand leads to an extraordinarily long sequence length."

Put numbers on it. At 24 kHz, ten seconds of speech is

Two hundred forty thousand positions for one short sentence. And self-attention costs O(T²): the attention matrix alone would have 240,000² = 5.76 × 10¹⁰ entries — 57.6 billion pairwise scores, per head, per layer, for one utterance. Nothing about this is negotiable with a bigger GPU.

Before the attempts, one orienting sentence: each of the three below fixes exactly one of the two axes, and the winner is the first representation that fixes both without destroying the speaker.

Read them as a search, not a history. The field was not confused; it was working through a genuinely constrained space, one axis at a time.

Attempt 2: µ-law companding (WaveNet's answer)

The classical fix for the vocabulary axis is µ-law companding, a nonlinear amplitude warp borrowed from telephony. It exploits the fact that human loudness perception is roughly logarithmic: differences among quiet samples matter more than differences among loud ones. So you compress the amplitude scale logarithmically, then round to 8 bits.

The paper's summary: "µ-law transformation can quantize each timestep to 256 values and reconstruct high-quality raw audio. It is widely used in speech generative models, such as WaveNet, but the inference speed is still slow since the sequence length is not reduced."

That is the whole story in one sentence. Vocabulary: 65,536 → 256, a 256× win. Sequence length: 240,000 → 240,000, no win at all. WaveNet generates one 8-bit sample at a time, autoregressively, 24,000 times per second of audio. Even with every engineering trick, that is a generation loop with a quarter of a million serial steps per utterance.

Attempt 3: self-supervised units (and the speaker they throw away)

The modern fix for the length axis comes from self-supervised speech models. vq-wav2vec, HuBERT, and w2v-BERT all run a convolutional front end that downsamples aggressively — typically to 50 frames per second — then discretize the resulting features, either with a learned codebook or by k-means over the continuous representations. Ten seconds becomes 500 tokens over a few hundred units. Both axes solved at once.

GSLM did precisely this and built a "textless" spoken language model on top. The paper credits it and then delivers the objection: "the speaker identity has been discarded and the reconstruction quality is low."

Why discarded? Because of what the pre-training objective rewards. HuBERT is trained to predict masked cluster assignments — a task about phonetic content. Two people saying the same word should land on the same unit; that is what makes the units useful for recognition. Speaker identity is, from that objective's point of view, a nuisance variable to be quotiented away, and the model obligingly quotients it away.

You cannot recover from downstream what the tokenizer deleted upstream. Chapter 9's table makes the loss quantitative: GSLM scores 0.126 speaker similarity where ground truth scores 0.754. The words survive; the person does not.

Before the table, name the criterion it is scored against, because a list of representations without a criterion is trivia. We need a representation that is short, discrete, speaker-preserving, and invertible — all four, simultaneously. Each row below fails or passes on exactly those four counts.

The requirement, stated exactly

Three attempts, three partial answers. Assemble them into a specification.

The paper lists the same three advantages of an audio codec in prose, and it is worth reading them as a checklist that no other row satisfies: (1) it "contains abundant speaker information and acoustic information, which could maintain speaker identity in reconstruction compared to HuBERT codes"; (2) "there is an off-the-shelf codec decoder to convert discrete tokens into a waveform, without the additional efforts on vocoder training"; (3) "it could reduce the length of time steps for efficiency to address the problem in µ-law transformation."

Notice the deep reason a compression codec is the right tool. A codec's training objective is reconstruction of the waveform. It is graded on getting the audio back. It therefore cannot discard speaker identity, room reverberation, or emotional colouring — those are precisely the things reconstruction error would punish it for losing. Where HuBERT is rewarded for forgetting, EnCodec is punished for it. The paper reaches for a compressor not because compression is elegant, but because compression is a forgetting-proof objective.

The length win, priced out

How much does 320× downsampling actually buy? Do it in full, because the number is larger than intuition suggests.

A hundred thousand times less attention arithmetic. That is not an optimization; it is the difference between "impossible" and "twelve layers on sixteen V100s." And the sequential-decoding win is the linear factor: 240,000 serial generation steps become 750 — before Chapter 5's second trick cuts it further.

What changes when the loss becomes cross-entropy

Chapter 0 derived the over-smoothing pathology of squared error on one ambiguous frame. Now watch what the replacement objective gives you that regression structurally cannot: a knob.

A language model's output at each step is a vector of logits, turned into a distribution by a softmax. Insert a temperature T and you can dial the distribution's sharpness continuously:

Work it out by hand on four candidate tokens with logits z = [3.0, 2.0, 1.0, 0.0]. At T = 1 exponentiate each logit:

Now T = 0.5. Divide every logit by 0.5 first, giving [6, 4, 2, 0]:

And T = 2, logits [1.5, 1.0, 0.5, 0]:

The top token's probability slides 0.8650 → 0.6439 → 0.4550 as T goes 0.5 → 1 → 2, and the tail lifts correspondingly. As T → 0 the distribution collapses onto the argmax and the model becomes deterministic; as T grows it approaches uniform. One scalar, at inference time, with no retraining, moves the system continuously between "one canonical rendition" and "wildly varied delivery."

A mel-regression model has no such parameter, and could not be given one. Its output is a point estimate; there is no distribution to sharpen.

python — the temperature knob, three forms
import numpy as np
z = np.array([3.0, 2.0, 1.0, 0.0])

# 1) by hand, T = 1 — exactly the arithmetic above
e   = [20.0855, 7.3891, 2.7183, 1.0000]
s   = 31.1929
p1  = [v / s for v in e]              # [0.6439, 0.2369, 0.0871, 0.0321]

# 2) step by step, any T
def softmax_T(z, T):
    zt = z / T
    zt = zt - zt.max()                 # numerical stability; shifts cancel
    ex = np.exp(zt)
    return ex / ex.sum()

# 3) the one-liner you would actually ship
# torch.softmax(logits / T, dim=-1)  then  torch.multinomial(p, 1)

The paper uses this knob asymmetrically, and the asymmetry is a real design decision you should be able to justify: sampling for the AR model, greedy for the NAR model. It also reports a concrete failure — "we observe that beam search may lead the LM into an infinity loop" — which is why sampling rather than beam search. Chapter 6 explains why the split makes sense; Chapter 10 explains what it costs.

A note on what "diversity" is worth

It is tempting to file output diversity under "nice to have." Three places where it is load-bearing, so the temperature knob is not mistaken for decoration.

See the difference: one text, two renditions

Two things to notice while you play with it. First, at T = 0 the two runs of the language model become identical — the diversity is not magic, it is sampling, and turning sampling off turns diversity off. Second, at high T the renditions do not merely differ, they start to degrade: durations get erratic, emphasis lands in strange places. Diversity and reliability trade off against each other along one axis, and Chapter 10's robustness failures live at the far end of it.

What the reframe inherits for free

Calling something "a language model" is not a metaphor if the object really is a sequence of discrete symbols with a next-symbol objective. It means an entire toolbox transfers without adaptation, and the transfer is a large part of why this line of work moved so fast after 2023.

Compare that list against what the mel-regression world had to build itself: duration predictors, monotonic attention mechanisms, speaker encoders, vocoders, and adaptation recipes — each a speech-specific research programme. Changing the representation did not just improve a metric; it moved the field onto infrastructure that thousands of people were already improving for other reasons.

Three questions any reframe must answer

Generalizing from this one case, "represent X as a sequence of discrete tokens and language-model it" is a move you will see attempted for images, video, actions, molecules, and time series. It succeeds when three questions have good answers, and Chapter 2 exists because two of them are hard here.

1. Is the sequence short enough? Attention is quadratic and generation is serial. Raw audio fails this by a factor of 320; codec frames pass.
2. Does the tokenizer keep what you need? HuBERT units fail this — they discard the speaker. EnCodec passes, because reconstruction forbids forgetting.
3. Can you get back to the original medium? A tokenizer with no decoder means training a second model. EnCodec passes trivially: the decoder ships with it.

Every representation in Chapter 1's comparison table fails at least one. The reason 2022 was the year this became possible is that neural audio codecs were the first speech representation to pass all three at once.

Sampling has more than one knob

Temperature is the knob the paper names, but the decision "use sampling, not beam search" opens a small family of choices, and they behave differently on acoustic tokens than on text. Worth knowing which lever does what.

Why does the degeneracy hit harder here than in text? Because acoustic token sequences are far more locally predictable. In a sustained vowel, consecutive frames are nearly identical, so the model's top continuation is "more of the same" with very high confidence, over and over. Any decoding rule that follows local likelihood gets trapped. Text has the same failure mode — it produces repetition loops — but words change often enough that the trap is shallower.

The over-smoothing story, one level deeper

Chapter 0 derived the pathology on a single ambiguous frame. It is worth seeing why the damage is worse than that one number suggests, because it explains the specific texture of regression-TTS artifacts.

Ambiguity in speech is not scattered randomly. It concentrates at boundaries — the onset of a vowel, the release of a stop, the transition between two words — because that is where small timing differences between takes produce large spectral differences. Steady-state regions (the middle of a long vowel) are highly reproducible and suffer almost no averaging.

So the damage lands disproportionately on exactly the fast, low-energy, high-information events that carry intelligibility. That is why regression TTS is described as "mumbly" rather than "quiet" — the loud, slow parts are fine and the fast, informative parts are not. It is also why an energy-weighted metric would have understated the problem, which is the same measurement trap Chapter 2 flagged about codebook 1's 84%.

Why not discretize the mel spectrogram instead?

An obvious middle road: keep the mel spectrogram everyone understands, but quantize it into tokens so the objective can be classification. The paper cites work in this direction — VQTTS, Du et al., which uses self-supervised VQ acoustic features — so the idea was live.

Two things go wrong, and both are instructive.

• You still need a vocoder. A quantized mel spectrogram is not audio. Something must be trained to invert it, and that vocoder becomes a second learned component with its own failure modes and its own training data requirements. The paper lists this as an explicit advantage of codecs: "there is an off-the-shelf codec decoder… without the additional efforts on vocoder training like VQ-based methods that operated on spectrum."
• The mel is already lossy in the wrong direction. A mel spectrogram discards phase and compresses frequency resolution logarithmically. Those choices were made for recognition, where phase is a nuisance. For generation, the vocoder has to hallucinate phase back, and that hallucination is where a great deal of the classic "vocoded" quality comes from. A waveform codec never threw phase away, because it was graded on reconstructing the waveform.

The pattern is the same one from Chapter 0's history table. Every intermediate representation encodes a set of decisions about what is safe to discard, and those decisions were almost always made with a different task in mind. A compression codec is the one intermediate whose discard policy was set by the ear.

A cost table for one second of speech

Abstract axes become concrete when you price a single second in every representation. Do the arithmetic once and the design space stops being a list of names.

Read the last two rows side by side. HuBERT is cheaper than EnCodec on every axis — fewer steps, fewer bits, less attention. It is also unusable here, because those 450 bits per second are the wrong 450 bits: they encode what was said and deliberately not who said it. The extra 5,550 bits per second that EnCodec spends are exactly the speaker, the room, and the emotion. Chapter 6 shows that the model needs only 2,250 of those bits (three seconds of one codebook) to clone a voice.

This is the moment to notice that "compression" and "representation learning" pull in opposite directions and VALL-E wants the compression side. A good representation discards nuisance variables. A good codec discards nothing that a listener can hear. For generation, the listener's ear is the specification.

One caution about that table: cheapness is only a virtue conditional on sufficiency. HuBERT wins every column and loses the only comparison that matters. Efficiency arguments are always downstream of an adequacy argument, and reversing the order is how a design ends up fast and useless.

Three reframes that did not work, and one that did

Chronology is useful here because it shows the idea converging from several directions at once.

Reading down the "what was missing" column is a good way to internalize what the paper actually added. Almost every ingredient existed. The assembly — a compression codec as the vocabulary, text as a prefix, and a hundred times more data than anyone had used — did not.

Why text conditioning matters (VALL-E vs AudioLM)

AudioLM got there first with the token substrate, so the paper is careful to draw the line. AudioLM is speech-to-speech: give it audio, it continues the audio. Its inputs are latent codes, not phonemes, so there is no channel through which you can specify what should be said. The paper's phrasing: "AudioLM is a speech-to-speech model, whereas VALL-E is a TTS model, so we can explicitly control the content in speech synthesis."

The consequence shows up in the robustness numbers. Because VALL-E conditions on pseudo-phonemes derived from a forced aligner, its generated audio is measurably more faithful to the intended words: the paper attributes its WER advantage over GSLM and AudioLM to being "trained with pseudo-phoneme instead of HuBERT/w2v-BERT codes, which enjoys better alignment quality with the input text."

So the architecture in one line: phonemes carry the content, acoustic tokens carry the voice, and the language model learns the joint distribution over both. Chapter 3 makes that sentence formal, and Chapter 6 shows the ablation that proves each half is doing the job assigned to it.

Chapter 2: EnCodec Tokens as the Vocabulary

Chapter 1 wrote the specification: a discrete alphabet, a short sequence, speaker identity preserved, and a decoder that already exists. VALL-E does not build such a thing. It picks one off the shelf — EnCodec, Défossez et al. 2022 — freezes it, and never trains it again.

This chapter is about what that frozen tokenizer actually does, because every design decision in the rest of the paper is downstream of its output shape. If you have already read the EnCodec Veanor or the neural audio codecs Gleam, treat this as the compressed re-derivation with VALL-E's numbers attached. If you have not, everything is built from zero here.

The shape of the tokenizer

EnCodec is a convolutional autoencoder with a quantizer wedged in the middle. Input: 24 kHz waveform. Output: the same 24 kHz waveform. In between, three things happen.

Fix the conversion table now and refer back to it whenever a number appears. Every quantity in this paper is derived from these five rows.

From those five rows, everything: T = seconds × 75 frames, the code matrix is T × 8 integers, the bitrate is 75 × 8 × 10 = 6,000 bits per second, and the paper's own worked case follows — "given a 10-second waveform, the discrete representation is a matrix with 750 × 8 entries, where 750 = 24,000 × 10 / 320."

The compression ratio, for scale: raw 24 kHz 16-bit audio is 24,000 × 16 = 384,000 bits per second. At 6 kbps the codec is throwing away 384,000 / 6,000 = 64× the data and reconstructing something a listener accepts. That is the technology that had to mature before this paper could exist.

Why one codebook cannot work — the number that forces RVQ

Here is the question that motivates the whole residual construction, and it is worth being frustrated by it before seeing the answer. We need 80 bits of information per frame (that is what 6 kbps at 75 Hz means: 6,000 / 75 = 80). Why not just use one enormous codebook?

Because a codebook with 80 bits of index has 2⁸⁰ entries:

And quantizing one frame would require a nearest-neighbour search over 10²⁴ candidates. This is not "expensive." It is arithmetically impossible, at any budget, forever.

Residual vector quantization is the escape. Instead of one 80-bit codebook, use eight 10-bit codebooks applied in sequence, where each one quantizes what the previous ones failed to capture. The cost becomes 8 × 1,024 = 8,192 comparisons and 8 × 1,024 × 128 × 4 ≈ 4.2 MB of storage. From 10²⁴ to 8,192.

Worked by hand: quantizing one frame

Nothing about RVQ becomes real until you do it with your fingers. So: three dimensions instead of 128, two codebooks instead of eight, four entries instead of 1,024. Every arithmetic step shown.

The encoder hands us one frame embedding:

Codebook 1 (four entries, learned during EnCodec's training — here just given):

Stage 1. Find the nearest entry by squared Euclidean distance. Subtract, square each component, add. All four, no shortcuts:

The winner is index 3. That integer is the frame's first-codebook token — in the real system, an integer in 0…1023, and the only thing the AR model of Chapter 4 will ever predict.

Now the move that makes it residual. We do not stop; we compute what we got wrong:

(The residual norm equals the winning distance, as it must — they are the same quantity written twice.)

Codebook 2 is a different codebook, trained on the distribution of first-stage residuals. Note its entries are all small: residuals live near the origin.

Stage 2. Same procedure, but the input is r₁, not v:

Winner: index 1. The frame's token stack is now [3, 1] — two small integers standing in for three floats.

The reconstruction. The decoder does not need v. It looks up and sums:

Reading the error ledger — and the hierarchy it reveals

The signal energy is ‖v‖² = 0.81 + 0.16 + 0.04 = 1.01. Track the error as a fraction of it:

Codebook 1 removed 83.17 percentage points of the error. Codebook 2 removed 15.34. The first quantizer did 84.4% of the total work (83.17 / 98.51). This is not an accident of my chosen numbers — it is structural. Every stage operates on the leftovers of the last, and leftovers are by construction smaller than what they are left over from.

The paper says exactly this in its Figure 2 caption: "Because RVQ is employed, the first quantizer plays the most important role in reconstruction, and the impact from others gradually decreases." And in the method section: "tokens from previous quantizers recover acoustic properties like speaker identity, while the consecutive quantizers learn fine acoustic details."

From hand arithmetic to numpy to the one-liner

Same computation, three levels of abstraction. Level two reproduces the hand numbers exactly — run it and check.

python — RVQ, step by step (reproduces the arithmetic above)
import numpy as np

v  = np.array([0.90, -0.40, 0.20])
CB = [
    np.array([[1.00, 0.00, 0.00],      # codebook 1
              [0.00, 1.00, 0.00],
              [0.60, -0.60, 0.00],
              [-0.50, 0.20, 0.80]]),
    np.array([[0.25, 0.25, 0.10],      # codebook 2 (residual-trained)
              [-0.30, 0.10, 0.20],
              [0.10, -0.20, 0.30],
              [0.35, 0.05, -0.15]]),
]

def rvq_encode(v, CB):
    residual, codes, recon = v.copy(), [], np.zeros_like(v)
    for stage, C in enumerate(CB):
        d = ((C - residual) ** 2).sum(axis=1)   # distance to every entry
        k = int(d.argmin())                       # the winning index
        codes.append(k)
        recon    = recon + C[k]                    # decoder SUMS the entries
        residual = residual - C[k]                 # what stage k+1 must fix
        print(f"stage {stage+1}: idx={k}  err={(residual**2).sum():.4f}")
    return codes, recon

codes, recon = rvq_encode(v, CB)
# stage 1: idx=2  err=0.1700     (0-based idx 2 == the 3rd entry)
# stage 2: idx=0  err=0.0150
# codes = [2, 0]   recon = [0.85, -0.35, 0.10]

python — the library one-liner VALL-E actually calls
from encodec import EncodecModel
import torchaudio, torch

model = EncodecModel.encodec_model_24khz()
model.set_target_bandwidth(6.0)          # 6 kbps  ->  8 quantizers

wav, sr = torchaudio.load("enrolled_3s.wav")  # resample to 24 kHz first
with torch.no_grad():
    codes = model.encode(wav[None])[0][0]     # -> LongTensor [1, 8, T]

# codes[0].T is the paper's matrix C : shape (T, 8), values in 0..1023
# 3 s of audio  ->  T = 3 * 75 = 225 frames  ->  225 x 8 = 1800 integers
wav_back = model.decode([(codes[None], None)])   # and back to 24 kHz audio

Walk the residual yourself

Two readings to take away from that curve. First, the diminishing-returns shape is exactly what justifies a variable bitrate: EnCodec lets you truncate the stack and still decode, just with more error. The paper notes the alternative operating point — "if we choose EnCodec at 12K bitrates, there are 16 quantizers… the 10-second waveform corresponds to a matrix with 750 × 16 entries."

Second, and more important for what follows: the curve's steepness at stage 1 means that getting codebook 1 wrong is unrecoverable. No amount of later refinement fixes a bad first token, because later codebooks only encode residuals relative to whatever the first one chose. That asymmetry in error sensitivity is why VALL-E is willing to pay for slow, careful, autoregressive generation of codebook 1 and not of the others.

One frame, in words

Say what a single row of C is, physically, because the abstraction can float free. Frame t of a 10-second utterance is 13.33 ms of sound — roughly the length of a stop-consonant burst, or a fiftieth of a syllable. It is described by eight integers between 0 and 1,023, and the codec's decoder turns those eight numbers, plus its neighbours, into 320 waveform samples.

Eight small integers for a sliver of a syllable, from which a listener can identify the speaker, the room, and the mood. That is the compression achievement the language model is standing on, and it is why "the tokenizer is not the contribution" understates how much the contribution depended on it.

Where the codebooks came from

Chapter 2's hand-worked example took the codebooks as given. They are not given — they were learned when EnCodec was trained, and knowing roughly how changes what you expect of them.

Each codebook is learned as an online clustering of the vectors it is asked to quantize. Entries are typically updated by an exponential moving average toward the mean of the inputs assigned to them — k-means, run continuously during training — while a commitment loss pushes the encoder's outputs to stay near whichever entry they selected, so the encoder and the codebook converge on each other rather than drifting apart.

Two consequences that matter for reading VALL-E:

• The codebooks are stage-specific by construction. Codebook 2 was fitted to the distribution of first-stage residuals, which is a tighter, origin-centred cloud than the raw embedding distribution. That is why its entries in the worked example were all small, and why Chapter 5's eight separate embedding tables in the NAR model are a type requirement rather than a luxury.
• Codebook usage is never perfectly uniform. Some entries are used far more than others, and poorly-trained codebooks suffer dead entries that no input ever selects. A 1,024-entry codebook may have an effective size well below 1,024, which means the real per-frame entropy is below the nominal 80 bits. The language model does not care — it learns whatever distribution it is shown — but it does mean the "60,000 bits per ten seconds" figure of Chapter 3 is an upper bound on the information actually present.

None of this is trained or modified by VALL-E. It inherits whatever EnCodec learned, dead entries and all. That is the price of a frozen tokenizer, and the paper judges it worth paying to avoid training a codec.

The bandwidth ladder

EnCodec is trained so that any prefix of the quantizer stack is decodable. That is a design choice with a name — quantizer dropout during codec training — and it is what makes the bitrate a runtime knob rather than a build-time constant. The ladder, for 24 kHz audio at 75 Hz:

The paper picks the 8-quantizer rung and says the choice is not sacred: "It is fine to choose other bitrate settings. A larger bitrate corresponds to more quantizers and better reconstruction quality." Chapter 5's cost ledger explains why the choice is cheap — adding quantizers adds NAR calls, and NAR calls are the inexpensive kind of step.

Sanity-check the residual walk against the paper

Before leaving the simulation, confirm that what it shows matches the paper's Figure 2 claim rather than an artifact of the toy numbers. Two invariants should hold no matter what codebooks you use.

• Error is non-increasing. Each stage picks the entry closest to the current residual, and one of the options is always "an entry near zero," so a stage can never make things much worse. In a well-trained codec it strictly improves.
• Improvements shrink. Stage k+1 operates on a residual that is smaller than stage k's by construction. Its codebook was fitted to that smaller scale, so its corrections are smaller. The curve must bend.

Those two properties together are the entire justification for treating codebook 1 differently — and they are structural, not empirical. Any residual quantizer, on any signal, produces this shape. Which means the AR/NAR split is not tuned to speech; it is tuned to residual quantization, and would transfer to any domain tokenized the same way.

A property VALL-E does not use: streamability

EnCodec is a convolutional model, and its causal variant is designed to run streaming — encode and decode incrementally, with bounded latency, because that is what a transmission codec is for. Frames can be produced and consumed as they arrive.

VALL-E inherits none of that benefit, and it is worth being clear about why, because the reason is architectural rather than a limitation of the codec. Chapter 7 traces the dependency: the NAR model conditions on the entire first-codebook layer, which does not exist until the AR model has emitted ⟨EOS⟩. A streamable tokenizer wrapped in a non-streamable language model is a non-streamable system.

The ceiling nobody mentions

Here is a consequence of freezing the tokenizer that is easy to miss and important for reading Chapter 9's numbers honestly.

VALL-E's output is decoded from 8 codebooks at 6 kbps. Even if the language model were perfect — if it predicted exactly the token matrix that EnCodec would have produced from the true target waveform — the audio you hear is EnCodec's 6 kbps reconstruction of that waveform, not the waveform. Quantization error is not something the language model can fix; it is baked into the vocabulary.

So every naturalness number in this paper is measured against a ceiling that the codec sets. The paper's ground-truth rows are real recordings, not codec round-trips — which means part of the remaining CMOS gap to ground truth is not the language model's fault at all. A cleaner ablation would report a "codec round-trip" row: pass the ground-truth audio through EnCodec and back, and score it. That row is absent, and it would have separated "the model made a mistake" from "6 kbps is 6 kbps."

Data-flow summary — the shapes you must carry forward

Those last two rows are the entire zero-shot TTS problem in twelve integers of description: given 1,800 integers of a stranger's voice and a phoneme sequence, produce 6,000 integers that a frozen decoder will turn into that stranger saying those words. Chapter 3 writes it as a probability.

Chapter 3: The Formulation — Conditional Codec Language Modelling

We now have an alphabet. This chapter writes down what we want the model to do with it, defines every symbol the rest of the paper uses, and — the part most summaries skip — asks why the eight codebooks get factorized the way they do rather than one of the three other obvious ways.

One promise about this chapter: everything in it is bookkeeping, and bookkeeping done carefully now makes Chapters 4 through 7 read as consequences rather than as new material. If a symbol here feels arbitrary, it is because its purpose arrives two chapters later — the drill at the end of this chapter is the place to confirm it stuck.

The objects

A dataset D = {x_i, y_i} of paired examples. For each pair:

The objective, in one line

The paper's statement: "we regard zero-shot TTS as a conditional codec language modeling task. We train a neural language model to generate an acoustic code matrix C conditioned on a phoneme sequence x and an acoustic prompt matrix C̃^T′×8 with the optimization objective of max p(C | x, C̃)."

Three symbols, and each carries a distinct job:

The paper's hope, stated plainly: "We expect the neural language model learns to extract the content and speaker information from the phoneme sequence and the acoustic prompt, respectively." Note the word expect — nothing in the architecture enforces the split. There is no disentanglement loss, no adversarial speaker classifier, no bottleneck. The division of labour is supposed to emerge because it is the easiest way to fit the data. Chapter 6 shows the ablation that checks whether it did.

One habit that pays off in the next three chapters: whenever an equation appears, immediately ask which of the two models it belongs to and which axis its product runs over. Every equation in this paper is identified by those two facts, and confusing them is the only real difficulty in the method section.

Unpacking the probability — what is actually being summed

"Maximize p(C | x, C̃)" is compact to the point of hiding the work. Expand it. C has T × 8 entries, each a categorical variable over 1,024 classes. The log-likelihood of one utterance is a sum of that many cross-entropy terms:

For a 10-second utterance that is 750 × 8 = 6,000 categorical predictions, each over 1,024 options — 60,000 bits of decision if the model were maximally uncertain. Compare against the input: the same sentence is perhaps 150 characters of text, roughly 120 phonemes. The model is expanding roughly 120 discrete symbols into 6,000, an expansion factor of 50×, and everything in that gap — duration, pitch, timbre, breath, room — is what it must invent or copy from the prompt.

The phrase "whatever we condition on" is doing all the remaining work, and choosing it is the architecture. That is the next section.

A last word on the objective's modesty. It contains no term for naturalness, no perceptual loss, no adversarial discriminator, and no speaker-similarity objective. It is next-token likelihood, and every capability in this lesson is downstream of it. When you find yourself reaching for an auxiliary loss to encourage a behaviour, it is worth asking first whether a representation change would make the behaviour the likelihood's own idea.

What is frozen and what is trained

Composed systems get confusing fast unless you keep a ledger of which parts learn. For VALL-E it is short, and the shortness is the point.

Three frozen components, two trained ones, and no joint fine-tuning anywhere. That has a consequence worth carrying into Chapter 9: gradients never flow from a perceptual objective back into the tokenizer, so the model is optimizing token likelihood, not audio quality. Those coincide only to the extent that EnCodec's tokens are perceptually well-organized — which is another way of stating the ceiling argument from Chapter 2.

The design space: four ways to factorize 6,000 tokens

You have a T × 8 grid to fill and an autoregressive machine that fills things one step at a time. There are four defensible choices, and the paper picks the fourth. Understanding why requires seeing the first three fail.

Scheme 2 is the instructive failure, so make its incoherence concrete. Return to Chapter 2's hand example. The frame's codes were [3, 1]: entry 3 of codebook 1, then entry 1 of codebook 2. But entry 1 of codebook 2 was chosen because the residual after e₃ was [0.30, 0.20, 0.20]. Had the first stage picked e₁ instead, the residual would have been [−0.10, −0.40, 0.20], and the best second-stage entry would have been a completely different one. The pair "3 then 1" is coherent; the pair "1 then 1" is nonsense — it reconstructs to [1.25, 0.25, 0.10] against a target of [0.90, −0.40, 0.20], worse than using no second codebook at all. Independent sampling produces exactly these mismatched pairs.

The factorization below is the paper's Equation 3, and it is the sentence the next two chapters implement. Read it slowly; every symbol has already been defined above.

If you can reconstruct it from the four-scheme table alone, you have understood the design rather than memorized it.

The factorization, written out

Scheme 4 in symbols — this is the paper's Equation 3, and it is the whole model:

Read it left to right as a generation procedure, and notice four things the equation quietly tells you:

1. There are two separate parameter sets, θ_AR and θ_NAR. Two models, trained separately, not one model in two modes.
2. The AR term conditions on C̃_:,1 only — the first codebook layer of the prompt, 225 integers. It never sees the prompt's other seven layers.
3. Every NAR term conditions on all of C̃ — all eight layers, 1,800 integers. The NAR model gets a richer view of the speaker than the AR model does. Chapter 5 explains why that asymmetry is necessary rather than sloppy.
4. The product over j is ordered. Codebook 3 waits for codebook 2's actual sampled values. Seven sequential calls, not seven parallel ones. The paper: "the NAR decoder will be called seven times to generate codes in seven quantizers."

Where the phonemes come from — and why they are "pseudo"

x is described as "its corresponding phoneme transcription," which sounds like it was written by a linguist. It was not. LibriLight is 60,000 hours of unlabelled audiobook audio — the paper says so directly: "The original data is audio-only, so we employ a speech recognition model to generate the transcriptions."

The pipeline, exactly as specified in the experimental setup:

Do the arithmetic on step 3, because it reveals what the model has to learn. Ten seconds at a 30 ms frameshift is 10 / 0.03 = 333 alignment frames. English speech runs roughly 10–13 phonemes per second, so after collapsing repeats you are left with on the order of 100–130 phoneme symbols. (That rate is a standard estimate, not a figure the paper states.) Against 750 acoustic frames, that means:

Six frames per phoneme on average — but a stressed diphthong might take eighteen and a flap might take two. Nothing tells the model which. There is no duration predictor, no attention-monotonicity constraint, no alignment supervision at all at generation time. The autoregressive model decides, implicitly, how many acoustic frames each phoneme deserves, by deciding when to stop producing frames that sound like it.

One more consequence of pseudo-labelling worth naming, because it is a genuine strength rather than a cost. The transcriptions are wrong sometimes — the paper concedes its data "contain more noisy speech and inaccurate transcriptions" than LibriTTS. Under a regression objective, systematically wrong targets are poison. Under a language-model objective at this scale, they are noise that averages out: for any given phoneme context, the majority of the 60,000 hours agrees, and the errors are uncorrelated. Scale is what converts a label-quality problem into a signal-to-noise problem.

The generative story, told in words

Before more symbols, say Equation 3 out loud as a procedure. If you can narrate this without looking, you have the model.

Every subsequent chapter is one of those five boxes, expanded. If a later derivation feels disconnected, come back and locate it here first.

If step 3 of that story felt like a sleight of hand — "start the sequence with someone else's audio and just keep going" — good. That is the trick, and Chapter 4 shows why the model has no way to object to it.

A notation drill

Subscript confusion causes more misreadings of this paper than any conceptual difficulty. Answer these before continuing; the answers are in the right column, so cover it.

If any of those was uncomfortable, the interactive matrix below is the fastest fix — switch between the highlight modes until each row of that table is obvious by inspection.

The objective is a compression bound — and that is useful

Cross-entropy has units. A negative log-likelihood in bits is exactly the number of bits an optimal arithmetic coder would need to transmit the data given the model. That gives a way to read the training loss that is more informative than "it went down."

The uninformed baseline first. If the model knew nothing, every token would be uniform over 1,024 options:

Now suppose the trained model reaches an average of 4 bits per token — a plausible figure for a well-fit codec LM, though the paper reports no perplexity. Then the same utterance costs 24,000 bits, and the model has effectively compressed the EnCodec stream by 2.5×. Two readings follow, both worth carrying:

• Learning is compression. Every bit the model saves is a regularity it found — that vowels persist, that this speaker's pitch sits here, that the phoneme sequence implies this ordering. Perplexity is a measure of how much structure it has absorbed.
• Conditioning should show up as bits. Adding the acoustic prompt should measurably reduce the bits-per-token of the target, because the prompt tells the model which speaker's statistics apply. That is the information-theoretic shadow of Chapter 6's ablations, and it is a clean experiment the paper does not run: report perplexity with and without each conditioning signal.

Joint, conditional, and what is being modelled

One clarification that prevents a common confusion later. p(C | x, C̃) is a conditional distribution. Nothing here models p(x) — the model has no opinion about which sentences are likely, and could not generate text if you asked it to. The phoneme sequence is always given.

That has a practical consequence worth stating: VALL-E cannot detect that a phoneme sequence is implausible. Feed it a string of phonemes that no English word produces and it will dutifully synthesize them, because "is this sayable?" is not a question the objective ever asks. Compare a text LM, which would assign low probability to nonsense and could refuse to continue it.

It also clarifies what the ⟨EOS⟩ token is doing. It does not mean "the sentence is over" in any semantic sense — the model has no access to sentence semantics. It means "the acoustic realization of the phoneme sequence I was given appears to be complete." Those coincide when alignment is healthy and come apart exactly when it is not, which is the failure mode of Chapter 10.

The phoneme side, in more detail

x gets less attention than C in most readings of this paper, which is a mistake — Chapter 6's ablation shows it is worth 16.6 WER points. Three practical facts about it.

The inventory is small. A Kaldi-style English phone set is a few dozen base phones, typically expanded with stress markers and context to a few hundred symbols. Against a 1,024-entry acoustic vocabulary, the phoneme embedding table is a rounding error in the parameter count — and yet it carries all the content.

Training and inference use different producers. At training, x comes from forced alignment of the audio (the aligner heard the speech). At inference, x comes from grapheme-to-phoneme conversion of text (nothing was heard). These two processes can disagree systematically — an aligner labels what was actually said, including reductions and elisions, while G2P produces the canonical pronunciation. The model trains on the former and is asked to perform on the latter.

Repetition removal changes what the sequence means. Before dedup, the aligner's output is a per-frame label track where duration is explicit — a long vowel is simply many repeated frames. After dedup, duration information is gone; the sequence carries identity and order only.

This is a deliberate and consequential choice. Keeping the repetitions would have handed the model a duration signal for free — and would have made x nearly as long as C, roughly tripling sequence length while forcing the model to copy durations rather than infer them from the speaker's style. Removing them buys short sequences and prosodic freedom, and it is the third place in this lesson where the same trade appears: expressiveness bought with a guarantee. Chapter 10 is where the bill arrives.

How much training signal is in one utterance?

Compare the supervision density against the era's TTS practice, because this is a quiet reason the scale bet works.

Note the NAR training detail hiding in that last row: only one stage is trained per example. The model does not compute all seven residual layers during a training step — it samples i uniformly and predicts only codebook i. So over 800,000 steps each stage receives roughly 114,000 steps of gradient, and the sharing of the trunk across stages is what makes that sufficient. This is exactly the trade AdaLN buys, quantified: seven separate models would each see one-seventh of the data; one AdaLN-conditioned model sees all of it and specializes through the modulation parameters.

What this objective cannot express

Be precise about the shape of the probability model, because three natural requests are outside it.

Every "no" in that table is a missing conditioning variable, not a missing capability of the architecture. Adding a control means adding a channel to the conditioning set and finding data that varies along it — which is precisely what the instruction-tuned speech models of Chapter 10 did.

Explore the matrix

A training batch, with shapes

python — what one training example looks like (pseudo-code, shapes are real)
# --- offline, once over the corpus ---
wav      = load_24khz(utt)                 # crop to a random 10-20 s window
C        = encodec.encode(wav)             # int64 [T, 8],  T = 75 * seconds
phn      = asr_align(wav)                  # 30 ms frames -> phoneme ids
x        = drop_consecutive_repeats(phn)   # int64 [L]

# --- AR example: no prompt is extracted at all ---
ar_input  = cat([emb_phn(x), EOS, emb_a1(C[:, 0]), EOS])   # [L+1+T+1, 1024]
ar_target = C[:, 0]                                       # [T]  — shifted by one
# every prefix of C[:,0] is implicitly a prompt for its own suffix

# --- NAR example: a prompt IS extracted, and a stage is sampled ---
i        = np.random.randint(2, 9)                     # which codebook to predict
prompt   = random_3s_segment(C)                          # int64 [225, 8]
e_prompt = sum(emb_a[j](prompt[:, j]) for j in range(8))    # [225, 1024]
e_curr   = sum(emb_a[j](C[:, j])      for j in range(i-1))  # [T, 1024]
nar_in   = cat([emb_phn(x), e_prompt, e_curr])           # [L+225+T, 1024]
nar_tgt  = C[:, i-1]                                       # [T]  — all positions at once

Two details in that sketch are easy to skim past and are the subject of the next two chapters. The AR example has no explicit prompt — the paper: "In the AR model, we do not explicitly extract an audio clip as the prompt in training. The training process is pure causal language model training." The NAR example samples a stage index i uniformly and a random 3-second segment from the same utterance as its prompt. Different training regimes, deliberately.

Chapter 4: The AR Decoder — One Column, Left to Right

Chapter 2 measured it: the first quantizer carried 84% of the reconstruction. Chapter 3 formalized it: c_:,1 gets its own model with its own parameters θ_AR. This chapter builds that model, traces a tensor through it, and answers the question that decides everything downstream — how does a system with no duration predictor know when to stop?

The equation

This is Equation 1 of the paper and it is, deliberately, the least surprising equation in the field. It is GPT. The product runs over time; each factor conditions on everything to its left plus two fixed conditioning blocks. If you can write a decoder-only transformer, you can write this.

The interesting content is in what the symbols are attached to. Take them one at a time.

The four components

The paper: "It comprises a phoneme embedding W_x, an acoustic embedding W_a, a transformer decoder, and a prediction layer."

Pause on that last box, because weight tying is one of those choices that looks like a parameter-saving hack and is actually a representational commitment. Untied, the model has two independent 1024 × 1024 matrices: one mapping token id → vector on the way in, another mapping vector → token score on the way out. Tied, there is one matrix, used in both directions. The logit for token k becomes the dot product of the hidden state with token k's own embedding.

The consequence: "predict token k" and "make the hidden state point along token k's embedding direction" become the same instruction. Codebook entries that sound alike will have been given similar embeddings (because they occur in similar contexts), so they get similar logits automatically — a smoothness prior you would otherwise have to hope for. And it saves 1024 × 1024 = 1,048,576 parameters, which is the least interesting benefit.

Building the input sequence

The paper: "the model input is the concatenation of x and c_:,1, and two special <EOS> tokens are appended after each of them."

Read carefully: two special end-of-sequence tokens, one appended after the phoneme block and one after the acoustic block. They are distinct special tokens — the phoneme-EOS says "the text is finished, start speaking"; the acoustic-EOS says "the utterance is finished, stop." Laid out:

And a detail that is easy to skim and matters: "We compute sinuous position embedding separately for prompt and input tokens." Sinusoidal positions are computed independently for the phoneme block and the acoustic block. The acoustic block's position counter restarts at zero rather than continuing from L+1.

Why? Because L varies wildly — a short sentence and a long one produce very different phoneme counts — and you do not want the acoustic tokens' positional signal to be shifted by an amount that depends on how much text preceded them. Restarting makes "acoustic frame 40" mean the same thing regardless of whether the text was five words or fifty. Positional information within each modality is preserved; spurious coupling across modalities is removed.

Where the phoneme block sits

One consequence of putting text at the front of a causally-masked sequence: the phoneme block cannot see the audio. Phoneme x₅₀ attends to x_≤50 and nothing else, ever. Only the acoustic positions get a two-way view of the pair.

That is strictly less than an encoder-decoder would give — a bidirectional text encoder lets every phoneme see the whole sentence. The paper accepts the loss, and it is probably small: what matters is that each acoustic token can see all of the text, which it can, since the text is entirely to its left.

Positions, concretely

"Sinusoidal position embeddings computed separately for prompt and input tokens" deserves a picture, because the alternative is the natural thing to do and it is subtly wrong.

Two properties fall out. Acoustic position 40 means "40 frames into the audio" no matter how long the text was — the model is never asked to learn that the same acoustic moment appears at wildly different absolute indices depending on sentence length. And the prompt and the generated audio share one continuous acoustic counter, which is what makes them a single sequence rather than two: the model has no positional cue that would let it distinguish the seam.

The mask, and what each token may see

"For the causal transformer model, each token c_t,1 can attend to (x, c_≤t,1)." That is the left panel of the paper's Figure 3. Three regions of the attention matrix, with different behaviour:

Trace it yourself in the simulation below, then come back for the training story, which is where the design gets genuinely clever.

Training: the prompt that is never extracted

Here is the passage that most repays a slow read: "In the AR model, we do not explicitly extract an audio clip as the prompt in training. The training process is pure casual language model training. In this way, any prefix sequence c_<t,1 is treated as a prompt for the latter part of the sequence c_≥t,1."

There is no prompt-construction code in AR training. No sampling of a 3-second window, no special separator, no marking of what is prompt and what is target. The model is trained on complete 10–20 second utterances with a plain next-token loss, and that is all.

The in-context learning falls out of the arithmetic. A 750-token sequence contains 750 prefix/suffix splits, and the causal loss trains all of them simultaneously: predicting token 200 given tokens 1–199 is exactly the task "continue this voice, you have 199 frames of it." Predicting token 700 given 1–699 is the same task with a longer prompt. Every training example is, silently, hundreds of voice-continuation examples at every prompt length from one frame to T−1.

Inference then just uses the machinery honestly. The paper: "During inference, given an enrolled recording, we should concatenate the phoneme sequence of the enrolled recording and the phoneme sequence for synthesis together. Meanwhile, the acoustic token sequence of the enrolled recording is used as the prefix in AR decoding."

So at inference the sequence is:

Both phoneme blocks are concatenated into one x. Both correspond to speech: the first part to audio the model is given, the second to audio it must produce. The model has spent 800,000 training steps learning that phonemes and acoustic tokens line up in order, so it continues the pattern.

How it knows when to stop — the implicit duration model

Everything in Chapter 3 said there is no duration predictor. So what ends the utterance?

⟨EOS_ac⟩. It is one more entry in the output distribution, and every training example ended with it. At each step the model assigns some probability to stopping, and when that token is sampled, generation halts. T is not a parameter of the system; it is a random variable the model draws.

This is a genuinely elegant answer to the problem the paper raises against fully-NAR designs: "the rate of the generated speech should be consistent with the enrolled recording, and it is hard to train a length predictor for different speakers since their speaking speed may be very diverse. In this case, the AR model is a more natural choice with its flexibility for acoustic sequence length prediction."

And notice it solves a harder problem than a length predictor would. A duration predictor outputs a number up front. The AR model makes the decision continuously, in context: having seen 225 frames of a fast talker, it is already in a fast-talking regime, so it emits fewer frames per phoneme and reaches ⟨EOS⟩ sooner. Speaking rate is inherited from the prompt through exactly the same mechanism as timbre. One machine, two behaviours, zero extra code.

Decoding: sampling, and the infinite loop

"For the AR model, we use sampling-based decoding conditioned on the prompts since we observe that beam search may lead the LM into an infinity loop. Furthermore, the sampling-based method could significantly increase the diversity of the output."

Two claims, and the first is a bug report. Beam search maximizes sequence likelihood. In an acoustic token stream, the highest-likelihood continuation of a stretch of near-silence is more near-silence; of a sustained vowel, more sustained vowel. There is always a locally-more-probable way to keep going than to emit ⟨EOS⟩, so the search never terminates. This is the same degeneracy that makes beam search produce repetitive text, but worse, because acoustic token sequences are far more locally predictable than words.

Sampling breaks the loop by construction: even a 3% probability of leaving the vowel becomes near-certain over enough steps. The paper's second sentence then reframes the necessity as a feature — the same stochasticity produces the diversity documented in Figure 4, which Chapter 9 examines.

KV caching, precisely

The naive reading of "run the model again with the new token appended" is that each step costs a full forward pass over the whole sequence — which would make the AR phase quadratic in T and put ten seconds of audio out of reach. It does not, and the reason is worth stating because it is what makes the 750-step loop affordable.

In a causal transformer, position t's keys and values never change when position t+1 arrives — causality guarantees that nothing later can influence anything earlier. So they are computed once and cached. Each new step computes a query for the single new position and attends against the cache.

This is standard text-LM infrastructure, and inheriting it for free is one of the unglamorous benefits of Chapter 4's decoder-only choice. An encoder-decoder TTS model with cross-attention needs its own equivalent machinery; a decoder-only codec LM runs on whatever the text-LM ecosystem already built.

One practical note for anyone implementing it: the acoustic prefix is 225 tokens that will never change, so the cache for them can be computed in a single batched pass before the loop starts. Only the generated tokens need step-by-step treatment. The 225 given tokens cost one pass, not 225.

Data flow with real numbers

One inference, end to end. Enrolled clip 3 s; target sentence ten seconds long.

At the last step the sequence holds 157 + 225 + 750 = 1,132 positions. Two costs follow. The attention matrix at that point is 1,132² ≈ 1.28 million entries per head per layer — trivial. The KV cache is the real memory:

Fifty-three megabytes per concurrent stream. Compare that to the 240,000-step raw-waveform world of Chapter 1, where the same cache would have been 240,000/1,132 ≈ 212 times larger — about 11 GB, for one sentence, on one stream.

The tied projection, worked with numbers

Weight tying is abstract until you compute a logit with it. Toy dimensions — a 3-dimensional hidden state and a 4-entry codebook — but the operation is identical at 1024 and 1024.

The transformer's last position produces a hidden state, and the acoustic embedding table is:

Because the output layer is W_a, the logit for token k is just h · (row k). Four dot products, every term shown:

Softmax at temperature 1:

Two things to take from this. First, the tied form means "the model wants to say token 3" and "the hidden state points along token 3's embedding" are literally the same statement — the logit is a similarity score in embedding space, so tokens whose embeddings are close get close logits, which is the smoothness prior described above.

Second, look at the distribution: the winner has only 31.9% probability and three tokens sit between 24% and 32%. If you always took the argmax you would get one fixed rendition. Sampling from this shape is where the diversity of Chapter 9's Figure 4 comes from — and it is also, honestly, where a wrong token can enter and start the drift of Chapter 10.

What the AR model is actually learning

"Predict the next codebook-1 token" sounds like one task. Decompose it and it is at least five, all learned from the same loss, none of them supervised separately.

They are not independent. Bad alignment produces bad duration; bad duration destabilizes prosody; unstable prosody makes stopping miscalibrated. That coupling is why the AR model's failures tend to be catastrophic-at-the-utterance-level rather than uniformly distributed — once the model loses its place, everything downstream is conditioned on the mistake.

A one-line summary of the AR model

If you had to compress this chapter into a sentence for someone who knows transformers: it is a 150 M-parameter GPT whose vocabulary is EnCodec's first codebook, whose prompt is a phoneme sequence followed by 225 acoustic tokens of a stranger's voice, and whose stop token is the only duration model in the system.

Every clause in that sentence is load-bearing, and each one is a chapter reference: the vocabulary is Chapter 2, the phoneme prompt is Chapter 3, the acoustic prefix is Chapter 6, and the stop token is the paragraph above.

Exposure bias, named

The classic autoregressive pathology applies here in full, and the paper does not name it, so name it. During training the model always sees ground-truth previous tokens (teacher forcing). During inference it sees its own samples. The moment it emits a slightly-off token, it is conditioning on a prefix from a distribution it never trained on, and errors compound.

Two features of this setting make it worse than in text. Sequences are longer — 750 steps for one sentence, against maybe 30 word tokens. And there is no discrete anchor to recover to: in text, a wrong word still leaves you inside a grammatical sentence that the model has seen many of. In acoustic tokens there is no grammar to snap back to, only a continuous trajectory that can drift arbitrarily far.

Why decoder-only rather than encoder-decoder?

TTS had used encoder-decoder architectures for years — encode the phonemes, cross-attend from the decoder. VALL-E does not. The comparison is worth making explicit because the choice is not obviously right.

The decisive column is prompting. In an encoder-decoder design there is no position where "here is three seconds of the target voice" is a natural input — you would bolt on a speaker encoder, which is exactly the design Chapter 0 is trying to escape. Flattening everything into one sequence makes audio conditioning free, and pays for it by giving up the monotonic-alignment guarantee. That is the same trade, seen from a third angle.

Counting the parameters

The paper gives the hyperparameters but not a parameter count, so derive it. Per transformer layer, with d = 1024 and d_ff = 4096:

(Layer-norm and bias terms add a few hundred thousand more; heads split d_model, so 16 heads × 64 dims = 1024 and the attention count is unaffected by head count.) A hundred and fifty million parameters — in January 2023, when GPT-3 was 175 billion, this is a small model. The paper's bet was never on model scale. It was on data scale, and Chapter 8 is about that.

Chapter 5: The NAR Decoder — Seven Layers in Seven Steps

The AR model produced c_:,1: 750 integers, generated one at a time, carrying 84% of the reconstruction. Seven codebook layers remain. If we handed them to the same machinery, the bill would be:

Seven times the cost of the part that mattered, spent on the part that does not. This chapter is the refusal to pay it — and it is a more interesting refusal than "just do it in parallel," because doing it in parallel naively is exactly the incoherent scheme Chapter 3 ruled out.

The equation

Compare it against Equation 1 from Chapter 4 and notice precisely one thing has moved: the product index. The AR equation's product ran over t — sequential in time. This one runs over j — sequential in depth. Within a given j, every one of the 750 positions is produced in a single forward pass, simultaneously.

The independence assumption being made, stated honestly: given the phoneme sequence, the full acoustic prompt, and every shallower codebook layer, the tokens of layer j at different times are treated as conditionally independent. That is false in the strict sense — adjacent frames' residuals are correlated. It is defensible because the conditioning set is enormous: layer 1 already fixed the phonetic identity, pitch, and energy of every frame, so what remains for layer j at frame t is largely a local refinement determined by frame t's own shallower codes.

Eight embedding tables, one trunk

"The NAR model has a similar architecture to the AR model, except that it contains eight separate acoustic embedding layers." Same 12 layers, 16 heads, d = 1024, d_ff = 4096. Different embedding front end.

Why eight tables? Because token 517 in codebook 3 and token 517 in codebook 6 are unrelated objects. Codebook 3's entries live in the space of residuals-after-two-stages; codebook 6's live in the much smaller space of residuals-after-five-stages. Sharing one table would force the model to represent both with the same vector — a type error. Eight tables cost 8 × 1024 × 1024 ≈ 8.4 M parameters, which is 5% of the model, and buys type safety.

How the shallower layers enter: Equations 4 and 5

The paper writes the input construction as two steps. First, look up each code in its own table (⊙ denotes index selection):

Then — and this is the load-bearing step — add them up:

One vector per frame, of dimension 1024, regardless of how many codebooks have been filled in so far. When predicting codebook 2 the sum has one term; when predicting codebook 8 it has seven. Same shape either way.

Why summing, not concatenating — and why it is the right operation

The obvious alternative is concatenation: stack the i−1 embeddings into a vector of size 1024(i−1). Two things break immediately.

Mechanically: the input dimension would change with the stage, from 1024 (predicting codebook 2) to 7168 (predicting codebook 8). One shared transformer cannot accept seven different input widths, so you would need seven input projections — or seven models.

Conceptually — and this is the better reason — summation is exactly what the codec's own decoder does. Return to Chapter 2's hand-worked frame: the reconstruction was v̂ = e₃ + f₁. Not concat(e₃, f₁). RVQ reconstructs by adding codebook vectors, because each stage encodes an additive residual. Summing the embeddings gives the transformer a representation with the same algebra as the thing it is modelling: "the audio so far, to depth i−1," as a single point in a vector space, exactly as EnCodec would compute it.

The prompt: all eight layers, and why that asymmetry is necessary

For the enrolled recording, the paper sums over all eight codebooks:

Contrast with Chapter 4, where the AR model received only c̃_:,1. Why the difference? It is not an oversight; it is forced.

The AR model's job is to produce codebook-1 tokens. Its input and output live in the same space, and a decoder-only LM is a sequence continuer: whatever you put in the prefix must be the same kind of thing as what comes out. Feeding it a full 8-deep summary would put objects in the prefix that it can never emit — a type mismatch that buys nothing, since the AR model's output has no depth to condition.

The NAR model's job is to fill in depth. It is not continuing the prompt; it is referring to it. And it is fully known — all 1,800 integers of a 3-second enrolment were computed by EnCodec before inference began. Throwing seven-eighths of a known conditioning signal away would be free information destroyed. So the NAR model takes all of it, and Chapter 6's ablation shows it converts that extra information into a speaker-similarity jump from 0.541 to 0.732.

The full input is a three-segment concatenation, (e_x, e_c̃, e_c:,<i), with positional embeddings again computed separately for the prompts and the acoustic sequence:

And the mask, unlike Chapter 4's: "Unlike AR, the NAR model allows each token to attend to all the input tokens in the self-attention layer." Fully bidirectional. Frame 12 sees frame 700. There is no causality to preserve because nothing is being generated in time order.

AdaLN: one trunk, seven behaviours

A problem the equations hide. Seven different prediction tasks — "predict codebook 2" through "predict codebook 8" — share one set of transformer weights. But they are genuinely different tasks: codebook 2 sees a large first-stage residual and must make a coarse correction; codebook 8 sees a tiny residual and must make a delicate one. How does one trunk know which job it is doing?

The paper: "The current stage i is injected into the network with Adaptive Layer Normalization operator, i.e. AdaLN(h, i) = a_i LayerNorm(h) + b_i, where h is the intermediate activations, a_i and b_i are obtained from a linear projection of the stage embedding."

Unpack it. Ordinary LayerNorm standardizes activations and then applies a learned per-dimension scale γ and shift β — the same γ, β for every input. AdaLN replaces those constants with functions of the stage:

The stage index becomes a small embedding; a linear layer turns it into a scale vector and a shift vector for every normalization site in the network. The trunk's computation is modulated — the same weights, re-tuned per stage. This is FiLM-style conditioning, and it is the same operator that conditions diffusion transformers on the noise timestep, for the same reason: one network, a family of related tasks indexed by a scalar.

Count the cost: a stage embedding of 8 × 1024 plus a projection to 2 × 1024 per normalization site — on the order of a million parameters total, against the alternatives:

The weight tying is also stated with an index shift worth decoding: "the weights of the j-th prediction layer are the same as the (j+1)-th acoustic embedding layer." There are eight embedding tables but only seven prediction layers, because codebook 1 is never predicted here. Prediction layer j predicts codebook j+1, and shares weights with embedding table j+1 — that is, tokens are scored against their own table, exactly as in Chapter 4. The off-by-one is bookkeeping, not a subtlety.

The two models, side by side

Everything that differs between them, in one table, because the differences are all consequences of one decision — which axis each model is sequential along.

The parameter budget of the NAR model

Chapter 4 counted 152 M for the AR model. Redo it for the NAR model, since the differences are exactly the design choices of this chapter.

Two lines in that table are the chapter in miniature. The seven prediction layers cost nothing because of tying — predicting codebook j scores against codebook j's own table. And AdaLN costs about a million parameters where seven separate models would have cost 1.1 billion. Both are cases of the same instinct: find the structure already present in the problem and let it carry the weight instead of adding parameters.

One NAR training step, in detail

"In each training step, we randomly sample a training stage i ∈ [2, 8]." Seven tasks, one uniformly sampled per example. Walk through what that means concretely.

python — one NAR training step (shapes are the real ones)
i        = np.random.randint(2, 9)              # uniform over the 7 stages
prompt   = random_3s_segment(C)                # int[225, 8] — SAME utterance

e_x      = emb_phn(x)                          # [L, 1024]
e_prompt = sum(emb_a[j](prompt[:, j]) for j in range(8))     # [225, 1024] — all 8
e_curr   = sum(emb_a[j](C[:, j])      for j in range(i-1))   # [T, 1024] — GROUND TRUTH

h        = trunk(cat([e_x, e_prompt, e_curr]), adaln_stage=i)  # bidirectional
logits   = h[-T:] @ emb_a[i].weight.T          # tied — [T, 1024]
loss     = cross_entropy(logits, C[:, i-1])     # ALL T positions at once

Three observations that are easy to miss.

• The shallower layers are ground truth, always. e_curr is built from the real C, never from the model's own predictions. This is teacher forcing across depth, and it creates the same exposure-bias exposure that Chapter 4 named across time: at inference, layer 5 conditions on the model's own layers 1–4. The mitigating factor is that the chain is 7 long, not 750.
• Each stage gets one-seventh of the steps. Over 800,000 steps that is roughly 114,000 gradient updates per stage — and it is enough only because all seven stages share the trunk. AdaLN is what converts "one-seventh of the data each" into "all of the data, seven specializations."
• Loss is computed on all T positions simultaneously. A single NAR training step yields 750 categorical predictions of supervision, matching the AR model's density per example.

What lives in which codebook?

The paper states the hierarchy qualitatively — "tokens from previous quantizers recover acoustic properties like speaker identity, while the consecutive quantizers learn fine acoustic details" — without assigning specific content to specific layers. The honest position is that the split is a gradient, not a partition, but the evidence in this paper does constrain it:

Which explains a result that otherwise reads as a contradiction. Chapter 2 said layer 1 carries 84% of the reconstruction; Chapter 6 will show that a NAR model with no prompts produces 19.6% WER given a perfect layer 1. Both are true, because they measure different things: 84% is squared error in the codec's embedding space, and intelligibility is a perceptual property that depends heavily on the remaining 16%. Consonants are low-energy and high-information. Energy-weighted metrics systematically understate them.

The independence assumption, stress-tested

Chapter 3 argued that dropping time-conditioning within a layer is defensible. Interrogate that claim by asking what would break if it were badly false.

That last row is the strongest evidence the assumption holds. Given a ground-truth first codebook, the parallel NAR stack reconstructs speech at 2.8% WER — better than the complete system, whose 5.9 is dominated by AR errors. If parallel residual generation were the weak link, the ceiling would show up here. It does not. The bottleneck is upstream, in the sequential part, which is exactly where the paper's Limitations section says it is.

Decoding: greedy, and why that is fine here

"For the NAR model, we use greedy decoding to choose the token with the highest probability." No sampling, no temperature.

Justify it from the division of labour. Sampling exists to inject diversity — different phrasings, different rhythms. All of that was already decided by the AR model when it chose codebook 1 and, implicitly, T. The NAR model's job is fidelity: given those decisions, add the residual detail that reconstructs cleanly. There is no diversity left to express, and sampling would only add reconstruction error. Greedy is the right call, and it also removes 5,250 random draws from the inference path.

Why not train seven NAR models and be done?

It is worth asking directly, since it is the simplest design and the paper rejects it implicitly by choosing AdaLN. Three reasons, in increasing order of importance.

The third row is the argument. The seven tasks are highly related: "predict the residual after k stages" for k = 1 through 7 are variations on one skill. A shared trunk lets evidence from stage 2 improve stage 7. Seven isolated models throw that away, and each is trained on a seventh of the examples. AdaLN is how you get specialization without paying for isolation — roughly a million parameters to make one network behave as seven.

What bidirectional attention buys

The mask change from Chapter 4 is not a technicality. Dropping causality lets every position see the entire utterance, and there are concrete things the NAR model can only do with that view.

And the thing it gives up: the NAR model cannot decide length, cannot stop early, and cannot be run before the sequence exists. Those are exactly the jobs assigned to the AR model. The two masks are complementary, and each model is given the mask that suits the decision it owns.

The diffusion analogy, and where it breaks

If AdaLN-with-a-stage-index feels familiar, it is: diffusion transformers condition on the denoising timestep with the same operator, for structurally the same reason. Both are "one network, a family of related tasks indexed by a scalar." The analogy is worth pushing because it clarifies both sides — and worth stopping at the point where it fails.

The analogy is genuinely useful for intuition — "the NAR stack is a seven-step, discrete, coarse-to-fine refiner" is an accurate sentence. It breaks on the last two rows. A diffusion model's step count is a hyperparameter you may choose at inference; VALL-E's is a property of the frozen tokenizer. And a diffusion step subtracts a continuous estimate, whereas a NAR step commits to one of 1,024 discrete entries, which cannot be partially undone by later stages.

The inference-cost ledger, derived in full

Now price the design. T = 750 (ten seconds), J = 8 codebooks.

Sequential steps. A "step" is one forward pass that must finish before the next can start — the quantity that determines latency.

Isolate the part the NAR model is responsible for, because that is where the paper's O(T) → O(1) claim lives:

The paper's sentence, now fully cashed out: "for the consecutive stages, as the number of output slots follows the sequence length of the first stage, NAR can reduce the time complexity from O(T) to O(1)." The clause before the comma is the enabling condition — the NAR model knows how many slots to fill only because the AR model already decided T. The two halves of the design are not independent optimizations; the second is licensed by the first.

The honest counter-accounting: arithmetic is not latency

A parallel pass is not free. It processes all 1,132 positions, where an AR step with a KV cache processes one. Count floating-point operations, using the rule of thumb that a forward pass costs about 2 × parameters FLOPs per token, with 152 M (AR) and 160 M (NAR) parameters:

That is the trade, and it is the right one on any parallel hardware: GPUs are throughput machines starved for work, and an AR step that touches one token leaves the device almost entirely idle. Spending 60% more FLOPs to eliminate 5,243 round trips is not a compromise; it is what the hardware wants.

A rough wall-clock picture, with an illustrative 20 ms per AR step and 60 ms per NAR pass (these latency figures are not from the paper — they are plausible numbers for a 150 M model, used only to show the shape):

Chapter 6: Three Seconds — Prompting as Voice Cloning

Chapters 4 and 5 built two decoders that both accept an acoustic prompt. This chapter asks the question that makes VALL-E interesting rather than merely efficient: why does 225 frames of a stranger reproduce that stranger?

The paper's framing is deliberately borrowed from text. "In-context learning is a surprising ability of the text-based language model, which is able to predict labels for unseen inputs without additional parameter updates. For TTS, if the model can synthesize high-quality speech for unseen speakers without fine-tuning, the model is believed to have in-context learning capability."

And the honest assessment of what came before: "the in-context learning capability of existing TTS systems is not strong, because they either require additional fine-tuning or degrade dramatically for unseen speakers." Both failure modes were catalogued in Chapter 0. This chapter shows the mechanism that avoids both, and then the ablations that prove the mechanism is what is actually doing the work.

A framing worth holding through this chapter: prompting is not how you use the model, it is what the model does. Everything below is an unpacking of that sentence, with numbers.

The recipe, exactly

Inference construction, spelled out for both models. Nothing here is learned at inference time; it is all sequence assembly.

The structural difference is worth naming precisely. In the AR model the prompt is a prefix: it occupies the same positions, in the same stream, as the tokens being generated. The model literally cannot tell where the enrolment stops and its own output begins — the paper: "the concatenation of c̃_:,1 and c_:,1 is a whole sequence, and we do not distinguish them or insert a specific token in training." In the NAR model the prompt is a separate conditioning segment with its own positional embeddings, which the model attends to but does not continue.

Prefix versus reference. Continue-this versus consult-this. Both are in-context learning; only the first is the thing that makes GPT-3 analogies apt.

Notice how much of this chapter is about a structural property rather than a trained one. The prefix mechanism is not learned separately, not tuned, and not present in the code as a distinguishable feature. It is a consequence of sequence layout, which is why it costs nothing and why it cannot be removed without changing what the model is.

The information budget of a voice

Count the bits. Three seconds at 75 Hz is 225 frames; eight codebooks of 10 bits each:

Two hundred eighty-one bytes. That is the entire acoustic evidence the autoregressive model — the one that determines prosody, rate, timbre trajectory, and 84% of the reconstruction — receives about who is speaking. It is less than this paragraph.

Compare with what a speaker-encoder system passes around: a 256-dimensional float embedding is 256 × 32 = 8,192 bits, more than three times as much. Yet Chapter 9's numbers show VALL-E beating YourTTS decisively on speaker similarity. More bits, worse result — because the encoder's bits are a compressed identity claim produced by a network trained to discard everything but identity, while VALL-E's bits are raw acoustic evidence that a model trained on 60,000 hours can interpret however it needs to.

The two settings below exist because "clone a voice" and "finish a sentence" are different tasks that share machinery, and the paper reports both rather than folding them together. Which number you quote depends on which task you mean.

Two inference settings

The paper distinguishes two ways the prompt can relate to the target, and reports both in every table — so knowing which is which matters when reading Chapter 9.

The paper's explanation for the WER gap: "the word error rate can be further reduced in VALL-E-continual setting, because the acoustic tokens for the first 3 seconds are extracted from the ground truth." The first three seconds of the evaluated audio are not synthesized at all — they are real. Fewer generated frames, fewer chances to drop a word.

The speaker-similarity direction flips, which is initially puzzling and has a clean explanation. Similarity is measured between the prompt and the synthesized speech. In the plain VALL-E setting, prompt and target come from two different recordings of the same person, and the metric compares like with like across a natural within-speaker gap. In the continual setting the prompt is the head of the very same recording, so the comparison is against a much narrower reference — and any drift over the remaining seven seconds is measured against that single, specific instance. Different denominators, not a contradiction.

The ablations — a clean double dissociation

This is the most quietly rigorous part of the paper, and it is the part that turns "we expect the model learns to extract content and speaker information respectively" from a hope into a finding.

Experiment 1: what does the NAR model need? Three NAR models trained with different conditioning, all evaluated with ground-truth first-codebook tokens as input, so the AR model's quality cannot confound the result.

Read down the two numeric columns separately and the dissociation is stark. Adding phonemes: WER 19.6 → 3.0 (a 16.6-point improvement), SPK 0.518 → 0.541 (0.023). Adding the acoustic prompt on top: WER 3.0 → 2.8 (0.2), SPK 0.541 → 0.732 (0.191). Each conditioning signal moves its own metric by an order of magnitude more than it moves the other one.

The paper's summary: "It shows the phoneme prompt mainly contributes to the content of the generation. In the NAR-2 prompts, the model can learn speaker information from the acoustic token prompt and thus improve the speaker evaluation quality."

The first row also deserves its own moment. WER 19.6 with ground-truth codebook-1 tokens means: even with the coarse structure handed to you for free, filling the residual layers wrongly makes speech unintelligible. Chapter 2's "84% of reconstruction energy" was about squared error, not intelligibility. The remaining 16% is where consonant detail lives.

Experiment 2: what does the AR model need? Now hold the NAR model fixed at its best configuration and remove the AR model's acoustic prefix.

Look at that WER column. Identical. 5.9 and 5.9. Removing the speaker evidence from the autoregressive model changed intelligibility by exactly nothing, while speaker similarity collapsed by 60%, from 0.585 to 0.236 — below even the "no prompt" NAR ablation.

This is a textbook double dissociation, of the kind cognitive scientists spend careers hunting for: intervention A moves metric 1 and not metric 2; intervention B moves metric 2 and not metric 1. It licenses the causal claim the architecture assumed — the phoneme channel carries content, the acoustic channel carries identity, and they are genuinely separable in the trained model even though nothing forced them apart.

A useful reframing before the details: prompting here is not a user-interface convenience layered on top of a model. It is the model's native mode of operation, and text-conditioned synthesis without a prompt is the special case — the one Chapter 6's ablation shows scoring 0.236.

Where "three seconds" came from

The number is not derived from anything in VALL-E. It is inherited, and knowing from where explains why the paper reports it as the headline rather than the ten-second result that scores better.

The speaker-encoding literature established three seconds as the enrolment budget: the paper notes that in Jia et al. and Arik et al., "the experiments show that the model is able to generate high-quality outputs with 3 seconds enrolled recordings for in-domain speakers." Adopting the same budget makes the comparison against that line of work like-for-like rather than a matter of who was allowed more audio.

Which means the framing of Chapter 9's VCTK table is deliberate: three seconds is the hard setting, chosen for comparability, and the paper reports 5 s and 10 s alongside it to show the trend rather than to pick a flattering operating point. If you are building on this, three seconds is a floor, not a target — 0.382 versus 0.484 is a large difference for seven more seconds of audio you very likely have.

What exactly gets copied from the prompt?

"It clones the voice" is too coarse. The prompt is a sample of a recording, and a recording is a convolution of several independent things. Sort them, because the paper's qualitative findings are claims about specific rows here.

The pattern: everything the codec preserves and the corpus varies over is copied; everything else is not. Accent is the instructive exception — EnCodec preserves it perfectly (it is just audio) but LibriLight barely varies over it, so the language model has not learned to model it as a dimension it can continue. Preservation by the tokenizer is necessary; variation in the corpus is also necessary.

The prefix and the reference, side by side

The AR model's prompt and the NAR model's prompt are both "the acoustic prompt," and treating them as the same object is the most common misreading of this architecture. Lay them out.

Read the last row against the row above it. The NAR model's prompt is explicitly constructed and contributes 0.191 of speaker similarity. The AR model's prompt is never constructed at all — it is an accident of causal training — and contributes 0.349. The unengineered mechanism is doing nearly twice the work of the engineered one.

Why in-context learning is available here and was not before

Make the argument structurally, because it generalizes past speech.

In-context learning requires that the conditioning information and the thing being generated share a type. A text LM can learn from examples in its prompt because examples are text and output is text — the same next-token machinery reads both. There is no separate "example encoder."

A mel-regression TTS system generates spectrogram frames from phonemes. Its input type is phoneme ids; its output type is real-valued frames. An enrolment recording is neither — it is audio. There is no slot in the sequence where audio is the natural next thing, so audio must be converted into a third type (a speaker embedding) by a third module, trained with a third objective. Every one of those steps loses information and adds a distribution the system can fall outside of.

VALL-E collapses the types. Enrolment audio, generated audio, and the model's output vocabulary are all codec tokens. The prompt is not converted into anything; it is prepended. That is the whole mechanism.

A Bayesian reading of the prompt

Another angle that some readers find clarifying. Think of the 60,000-hour model as having learned a prior over "ways speech can be" — a distribution p(C | x) marginalized over all seven thousand speakers, all rooms, all moods. Sampling from that prior gives you generic speech in an averaged voice.

The acoustic prompt is evidence. Conditioning on C̃ slices that prior down to the region compatible with the observation: this vocal tract, this room, this rate. The model is not learning a speaker; it is performing inference in a space it already knows, using 281 bytes of observation.

Two predictions fall out of this reading, and both match the data:

• More evidence should sharpen the posterior. Longer prompts should improve similarity — which is precisely the 0.382 → 0.423 → 0.484 trend at 3, 5, and 10 seconds.
• Evidence outside the prior's support should not help much. A voice unlike anything in the corpus — a strong accent — cannot be reached by slicing a prior that has no mass there. Which is exactly the VCTK result, and it is a limitation no prompt length fixes.

Practical prompt hygiene

The paper evaluates on clean, single-speaker, correctly-transcribed enrolment clips. Real deployments do not have that luxury. The following predictions follow from the mechanism; the paper does not test them, and they are labelled as reasoning rather than results.

The general shape: because the prompt is raw evidence rather than a filtered summary, the system is exactly as robust as the enrolment audio is clean. A speaker encoder's bottleneck was a liability for fidelity and an asset for robustness. VALL-E gives up the asset along with the liability.

How long should the prompt be?

Three seconds is a convention inherited from the speaker-encoding literature, not an optimum. The VCTK evaluation varies it, and the trend on the full 108-speaker set is clean:

From 3 s to 10 s, VALL-E gains 0.102 while the baseline gains 0.037 — VALL-E extracts nearly three times as much value from the extra evidence. The paper: "By comparing different lengths of the prompt, we can see our model is able to generate more similar speech when the prompt becomes longer, which is consistent with our intuition."

Why should it be? Three mechanisms, all of which are just "more evidence":

• Phonetic coverage. Three seconds is about 35 phonemes. English has around 44. A 3-second clip simply may not contain this speaker's /ʒ/ or their /θ/, so the model must extrapolate those from the ones it saw. Ten seconds covers most of the inventory.
• Prosodic evidence. Speaking rate, pitch range, and pause behaviour are statistics. Estimating a range from 225 frames is noisy; from 750 frames much less so.
• Longer prefix, stronger conditioning. In the AR model the prompt competes with the model's own generated tokens for attention. A 750-token prefix followed by 750 generated tokens keeps the speaker evidence at 50% of the context; a 225-token prefix drops it to 23% and falling.

One honest wrinkle the paper reports without comment: on the 11-speaker fully-unseen subset, VALL-E scores 0.389 at 3 s and 0.380 at 5 s — a slight decrease — before rising to 0.414 at 10 s. With eleven speakers, that non-monotonicity is comfortably inside sampling noise; it is a reminder that the 11-speaker column is a small-sample column and should be read as a direction, not a measurement.

The claim, stated as strongly as the evidence allows

Putting the chapter's results into one careful sentence: a 150 M-parameter model, given 281 bytes of a stranger's first-codebook tokens and a correct phonemization of what they said, produces speech that human raters find nearly as similar to that stranger as a second real recording of them — on read English audiobook speech, and with no weights changed.

Every qualifier in there is doing work, and dropping any of them turns a defensible claim into an overclaim. "On read English audiobook speech" is Chapter 9's VCTK gap. "A correct phonemization" is the enrolment-transcript requirement. "Nearly as similar" is 4.38 against 4.5 with overlapping intervals. Being able to state the claim with its qualifiers intact is the difference between understanding the result and repeating a headline.

Prompt or fine-tune? A decision table

The paper's contribution is that prompting became viable, not that fine-tuning became useless. If you were choosing between them for a real product, the trade is legible.

The last row is the one Chapter 10 returns to. Prompting removed the artifact that could be deleted, and with it the natural control point. Everything that made the capability easy also made it uncontainable.

The "continual" setting, read skeptically

VALL-E-continual scores the best WER in the paper (3.8), and it is worth being clear about why that number should not be quoted as the system's robustness.

In that setting the first three seconds of the evaluated utterance are ground-truth audio — not generated. So:

Both effects push WER down, and neither reflects the setting a user cares about, which is "read me this new sentence in this voice." The paper is transparent about the mechanism — "because the acoustic tokens for the first 3 seconds are extracted from the ground truth" — and reports both settings side by side. Read 5.9 as the system's number and 3.8 as an upper bound on what better alignment could buy.

Reproducing the ablations

The two ablation tables use different protocols, and mixing them is the most common way to misquote this paper. Keep them separate:

That derived number is the sharpest statement of where VALL-E's robustness problem lives: of the 3.7-point gap between VALL-E and ground truth, about 3.1 points are attributable to the autoregressive stage. Chapter 10's failure modes are not a side issue; they are essentially the entire quality gap.

The requirement everyone forgets

Read the AR recipe once more: "we should concatenate the phoneme sequence of the enrolled recording and the phoneme sequence for synthesis together."

VALL-E needs the transcript of the enrolled clip, not just the audio. Three seconds of a stranger is not sufficient input; you need three seconds plus a correct phonemization of what they said. In the paper's evaluations this is free — LibriSpeech and VCTK ship transcripts — but in the wild it means running ASR on the enrolment and inheriting its errors.

Why is it needed at all? Because the acoustic prefix and the phoneme sequence must stay in registration. The model learned that phonemes and acoustic frames advance together through the sequence. If you gave it 225 acoustic frames of speech whose phonemes were absent from x, the alignment it has learned to expect would be broken at exactly the moment it starts generating — the model would be somewhere in the middle of the audio while being at the start of the text.

(That last paragraph is a mechanistic prediction, not a reported experiment — the paper does not ablate transcript quality. It is the reasoning the design implies, and it is worth flagging as an untested corner rather than a result.)

Chapter 7: Showcase — Filling the Token Grid

Every piece is on the table. The alphabet (Chapter 2), the objective (Chapter 3), the sequential decoder (Chapter 4), the parallel one (Chapter 5), the prompt (Chapter 6). This chapter runs them, in one picture, with the controls in your hands.

The picture is the paper's Figure 3 turned into a machine you can operate. The object being built is the acoustic code matrix C: time across, codebook depth down. Watching it fill is watching VALL-E synthesize speech.

Everything, on one page

Before operating the machine, one consolidated statement of what it is doing, with the chapter each piece came from.

One instruction for using the panel well: run it once without touching the controls, then decide what you expect each control to do, then use it. Simulations teach when they confirm or refute a prediction; they decorate when you just watch them.

The generation script

Three phases, in strict order. Nothing in phase 2 can begin before phase 1 finishes, for the reason Chapter 5 derived: the NAR model needs to know how many slots exist.

What the animation deliberately distorts

A simulation that is honest about its lies teaches better than one that is not. Three ways this panel differs from the real thing, each chosen to make a specific point legible.

The counters, by contrast, are not distorted: "sequential passes 383 / 757" and "tokens placed 2,250 / 6,000" are the real numbers for a ten-second utterance, scaled from the animation's position. Trust the counters over the pixels.

The panel is the paper's Figure 3 with a clock attached. Figure 3 shows which tokens attend to which; this shows the same structure unfolding in time, which is the part a static diagram cannot convey.

Five things to look for

1. The seam is invisible. When the AR phase starts, the first generated cell sits immediately beside the last prompt cell in the same layer, with nothing marking the boundary. That is not a rendering shortcut — it is the paper's design: "we do not distinguish them or insert a specific token in training." The model is doing one thing (continue this sequence), and prompting is not a special case of it.
2. The top layer costs everything. Count how long phase 1 takes versus phase 2. Ratio 750 : 7. All the latency, all the sampling, all the risk of dropped words — concentrated in one of eight layers, the one that Chapter 2 measured at 84% of the reconstruction.
3. Depth is ordered, time is not. Inside phase 2, layer 5 waits for layer 4 — you will never see them fill together. But within layer 5, frame 3 and frame 700 arrive in the same instant. That is exactly the independence structure of Equation 2.
4. Temperature only touches the top layer. Drag it and watch: the AR row's token values scatter, the seven NAR layers do not change their behaviour at all. Greedy decoding below, sampling above — Chapter 5's argument, visible.
5. The prompt fraction moves. At 3 s the given region is 225 of 975 columns (23%); at 10 s it is 750 of 1,500 (50%). More of the AR model's context is speaker evidence and less is its own output — the third mechanism from Chapter 6's prompt-length discussion, drawn to scale.

Compute the prompt fractions yourself so the slider means something. Target utterance ten seconds = 750 frames in every case:

And the total token accounting for the standard case — 3-second prompt, ten-second output:

Before running it: predict what you expect the latency bar at the bottom of the panel to look like when generation finishes. If your prediction is "roughly half AR and half NAR," the panel is about to correct you, and the correction is the single most important number in this chapter.

Ten seconds, narrated

A walkthrough in words, with counters, so the animation has a script to follow. Target text: "Your appointment is confirmed for Thursday." Enrolment: three seconds of a stranger.

The row worth staring at is pass 750. Every one of the seven cheap passes is blocked behind all 750 expensive ones, and the first sample of audio is blocked behind all 757. That single dependency is the difference between a batch synthesizer and a conversational one.

What changes for a short utterance?

Not the structure, only the ratio — and the ratio is why short-form synthesis is a much better fit for this design.

Efficiency creeps upward with length — the fixed seven passes amortize — but latency grows linearly and so does the exposure-bias chain of Chapter 4. Short utterances are cheaper and more reliable, which is consistent with the paper's own observation that VCTK's shorter sentences produced better naturalness scores than LibriSpeech's longer ones.

The grid is a clock — reading columns as time

Because the frame rate is fixed, the horizontal axis has a hard physical interpretation. Every column is exactly 1/75 s = 13.33 ms of audio, and 320 waveform samples. So:

That last row is the one to remember. Human conversational turn-taking runs on gaps of roughly 200 ms. VALL-E's AR phase alone needs 750 sequential forward passes before the first sample of audio can be decoded. Fifteen columns of latency budget against 750 columns of serial work is not a tuning problem; it is a structural one, and it is why Chapter 10's successors changed the frame rate rather than the decoder.

Why this grid cannot be streamed

An obvious optimization suggests itself: why wait for the whole utterance? Emit audio for the first frames as soon as they are ready.

Follow the dependencies and watch it fail. To decode frame 1 into audio you need all eight of its codes. Codes 2–8 come from the NAR model. The NAR model runs on the entire layer at once and is conditioned on the full first-codebook layer, which does not exist until the AR model has emitted ⟨EOS⟩. So frame 1's audio is not available until frame 750's coarse token has been generated.

You could chunk it — generate a few seconds, run the NAR stack on that block, emit, continue — and later systems do exactly this. But it is a modification, not a property of the design, and it costs you the global conditioning that makes the NAR stage cheap. The honest statement is that VALL-E as published is a batch synthesizer: text in, complete utterance out.

The same grid at other operating points

Everything in this chapter is parameterized by three numbers — frame rate, codebook count, and utterance length. Vary them and the ledger moves in instructive ways.

Read the fourth row carefully, because it is the whole strategic point. Dropping the frame rate 6× cuts sequential passes 5.7× — a bigger win than the entire AR/NAR split delivered — without changing the decoder at all. Adding codebooks is nearly free; adding frames is brutally expensive. Any future work on latency in this family attacks the frame rate first.

The inference loop, as code

Everything in the animation, written out. The shapes are the ones from Chapters 4 and 5; run your eye down the two loops and notice that one has 750 iterations and the other has 7.

python — VALL-E inference, end to end (pseudo-code with real shapes)
# ---------- inputs ----------
enrolled_wav   = load_24k("stranger_3s.wav")       # 72,000 samples
enrolled_text  = "and so the evening passed"          # its transcript — REQUIRED
target_text    = "your appointment is confirmed"

# ---------- tokenize ----------
C_tilde = encodec.encode(enrolled_wav)               # int[225, 8]
x = g2p(enrolled_text) + g2p(target_text) + [EOS]   # int[L]  — ONE sequence

# ---------- phase 1: AR, 750 sequential steps ----------
c1 = list(C_tilde[:, 0])                            # the acoustic PREFIX, 225 tokens
while True:
    h      = ar_model(emb_phn(x), emb_a(c1))          # causal; KV-cached in practice
    logits = h[-1] @ W_a.T                              # tied weights -> [1025]
    tok    = sample(softmax(logits / T))                 # NOT beam search (infinite loops)
    if tok == EOS_AC: break                            # the model decides the length
    c1.append(tok)
c1 = np.array(c1[225:])                              # drop the prefix -> int[T], T ~= 750

# ---------- phase 2: NAR, 7 parallel passes ----------
C = np.zeros((len(c1), 8), dtype=int); C[:, 0] = c1
e_prompt = sum(emb_a[j](C_tilde[:, j]) for j in range(8))    # ALL eight layers
for i in range(2, 9):                                  # seven calls, ordered
    e_curr = sum(emb_a[j](C[:, j]) for j in range(i - 1))  # SUM, not concat
    h      = nar_model(emb_phn(x), e_prompt, e_curr, stage=i)  # AdaLN(h, i)
    C[:, i - 1] = (h @ W_a[i].T).argmax(-1)          # GREEDY, all T at once

# ---------- decode ----------
wav = encodec.decode(C)                               # T*320 samples at 24 kHz
# 750 frames -> 240,000 samples -> exactly 10.0 seconds

Two lines in there are the entire paper. c1 = list(C_tilde[:, 0]) — voice cloning, implemented as list initialization. And C[:, i-1] = (...).argmax(-1) — an entire codebook layer assigned in one statement, which is what O(T) → O(1) looks like when you write it down.

From grid to waveform — the last step, in detail

The animation ends with a filled matrix, and it is easy to treat "hand it to the decoder" as a black box. Open it, because it is the same arithmetic as Chapter 2's hand example, run 750 times.

Two things follow from step 2 that are worth holding. The decoder sums — which is why Chapter 5's NAR model also sums its input embeddings rather than concatenating them, matching the algebra of the object. And the decoder has no idea which codes were generated and which came from the prompt; it sees one matrix. The seam is invisible to the decoder for the same reason it is invisible to the AR model.

Three reading exercises for the grid

Run these deliberately. Each one is a claim from an earlier chapter, made checkable.

The latency bar at the bottom of the panel is the summary of all three: the AR phase occupies 750 / 757 = 99.1% of the sequential budget while producing 750 / 6,000 = 12.5% of the tokens. That inversion — almost all the time spent on one-eighth of the output — is the shape of the entire design, and it is what the next generation of systems set out to fix.

Break it: four experiments the grid invites

The simulation is a teaching device, but each control corresponds to a real degradation you could measure. Predict the outcome before you drag.

Only the first two are things the paper reports on. The third and fourth are predictions the architecture makes; a reader with a GPU could check them in an afternoon, which is a good sign that the model has been explained rather than described.

Chapter 8: The Scale Bet — 60,000 Hours

Chapter 4 counted the parameters: about 152 million. In January 2023 that is a small model — GPT-3 was 175 billion, a thousand times larger, and it was two and a half years old. Twelve layers with d = 1024 was an unremarkable configuration in 2019.

So VALL-E is not a bet on model scale. The bet is entirely on the other axis, and the paper states it as a contribution: "We build a generalized TTS system in the speaker dimension by leveraging a huge amount of semi-supervised data, suggesting that simple scaling up semi-supervised data has been underestimated for TTS."

This chapter prices that bet — how much data, where it came from, what it cost to train on, and, at the end, exactly what it did and did not buy.

Two axes, and which one everyone reached for

A model has two obvious scaling axes and a fixed budget. The field's instinct in 2022 was to spend on parameters, because that is what had worked for text. VALL-E spends on data. The comparison is worth making explicit before the numbers.

Frame it as diagnosis. Pre-VALL-E TTS was not failing because 30 M-parameter acoustic models lacked capacity to fit 600 hours — they fit it well and sounded good on the speakers they knew. It was failing on speakers it had never encountered, which is a coverage failure. Coverage failures are cured by data and not by parameters, and reading the diagnosis correctly is why a 150 M-parameter model beat a decade of architectural work.

Keep one distinction sharp through this chapter: hours is a measure of audio, frames is a measure of model timesteps, and tokens is a measure of integers. All three appear below, they differ by factors of 270,000 and 8, and mixing them is the fastest way to get an epoch count wrong by an order of magnitude.

This chapter is the one where the paper's contribution actually lives, which is worth saying plainly given how little of it is architecture.

The corpus

Set that against the prior art. LibriTTS — the standard multi-speaker TTS corpus of the era — is a filtered, cleaned, segmented derivative of LibriSpeech, and the paper's summary row says existing systems trained on "≤ 600 hours." The ratio:

And the paper is candid about what the extra 59,400 hours are like: "our data contain more noisy speech and inaccurate transcriptions but provide diverse speakers and prosodies. We believe the proposed approach is robust to the noise and generalize well by leveraging large data."

That sentence is the whole thesis restated as a wager. Chapter 0 showed why noisy data poisons a regression objective. The claim here is that a language-model objective converts the same noise from poison into coverage — because a token sequence extracted from a reverberant recording is a perfectly valid token sequence, and the model that learns it gains the ability to produce reverberant speech when prompted with reverberant speech. Chapter 9's acoustic-environment finding is that wager paying off.

Where the 60,000 hours came from

LibriLight is not a TTS corpus and was never meant to be one. It was assembled as an ASR benchmark — a testbed for limited-supervision and unsupervised speech recognition — from LibriVox, the volunteer public-domain audiobook project. Volunteers read out-of-copyright books at home, with whatever microphone they own, in whatever room they have.

Three consequences follow directly from that provenance, and every one of them shows up later in this lesson:

• No transcripts. The corpus was built for the setting where labels are scarce, so audio-only is the point. Hence Chapter 3's pseudo-labelling pipeline.
• Uncontrolled acoustics. Home recordings mean varied microphones and rooms — which is the "noisy speech" the paper concedes, and also the source of the environment-transfer capability of Chapter 9.
• Read prose, mostly American. Volunteers read books aloud; they do not converse, interrupt, or argue. That is the style limitation, and the accent gap that VCTK exposes.

A corpus built for one purpose, repurposed for another, carrying its origins into every result — strengths and gaps alike. Worth asking of any dataset before trusting what a model trained on it can do.

Everything below is standard, and that is the observation. Read the configuration looking for the clever part; there isn't one.

The training configuration, in full

Two of those rows encode design decisions worth dwelling on.

Random 10–20 s crops. The window length varies from example to example, so the model never learns "utterances are N frames long." It sees short ones and long ones, and the ⟨EOS⟩ decision has to be driven by content and prosody rather than by a memorized position. Given that ⟨EOS⟩ is the duration model (Chapter 4), randomizing the crop is the closest thing to explicit duration training in the entire system.

NAR prompt from the same utterance. A three-second window is drawn from the same recording as the target. Same speaker, same microphone, same room, same session. This teaches the NAR model "copy the acoustic character of the reference," which is precisely the job it will do at inference. It also means the model was never trained on the harder case where the prompt comes from a different session — and yet that is exactly what the plain VALL-E evaluation setting does. A small train/test mismatch that the results suggest it survives.

How much data did it actually see? A derivation with an ambiguity

The paper does not report an epoch count, but everything needed to compute one is given. Start with the corpus size in the model's own units.

Now the training budget:

Units of what? "6k acoustic tokens per GPU" is ambiguous — a token could mean one integer (so a frame contributes eight) or one frame. The two readings give very different answers, so resolve it with a sanity check rather than a guess.

Crops average about 15 seconds, which is 15 × 75 = 1,125 frames.

Reading B it is. So:

Under five passes over 60,000 hours of audio. Put that beside the text-model numbers the paper's introduction cites — GPT-3 saw roughly 300 billion text tokens — and VALL-E's 76.8 billion next-token predictions land within a factor of four of a frontier language model of the era. Measured in training signal rather than parameters, this is not a small run.

What it cost

The paper reports no wall-clock time, so estimate it with the standard rule that transformer training costs about 6 × parameters × tokens FLOPs (two for the forward pass, four for the backward):

(That is a derived estimate, not a paper figure. Real runs with data loading, checkpointing, and imperfect overlap typically land at two to four times the idealized number, so call it a week for both models on sixteen V100s.) Either way the headline is the same: the expensive resource was the 60,000 hours, not the GPUs. A modern research group could reproduce this compute budget on a handful of contemporary accelerators; what they could not trivially reproduce is the corpus and the willingness to trust it.

A caveat on the epoch estimate before spending it: it assumes every hour of LibriLight is used and that the random 10–20 second crops sample the corpus uniformly. Neither is stated. Treat 4.74 as an order-of-magnitude statement — "a handful of passes, not hundreds and not a fraction" — rather than a precise figure.

Noise as coverage — a concrete example

"Noisy data helps" is counterintuitive enough to deserve a worked case rather than an assertion.

Suppose 5% of LibriLight recordings have noticeable room reverberation. Under the two objectives:

The mechanism is that a conditional distribution can represent a mixture, and a point estimate cannot. Under regression, "5% of the data is reverberant" becomes "everything is 5% reverberant." Under a language model, it becomes "there is a reverberant mode, reachable by conditioning" — and the acoustic prompt is exactly the conditioning that reaches it. Chapter 9's acoustic-environment finding is this table, observed.

Which yields a clean rule: noise hurts point estimators and informs distribution estimators. The same corpus is poison to one and a feature to the other. That is a much stronger statement than "the model is robust to noise," and it is the actual reason the scale bet worked.

Why the 60-second average utterance forces cropping

A detail that sounds like housekeeping and is not. LibriLight's average utterance is 60 seconds — a chapter-length chunk of an audiobook. At 75 Hz that is 4,500 frames, and:

So cropping to 10–20 seconds is partly an economics decision. But it also shapes what the model learns, in two ways that matter at inference:

• It sets the length distribution the model expects. Trained on 10–20 second windows, the model's ⟨EOS⟩ calibration is tuned for that range. Synthesizing a two-second phrase or a forty-second paragraph is extrapolation, and the paper does not report how it behaves there.
• It caps how much context prosody can span. Discourse-level prosody — the arc of a paragraph, the setup and payoff of a sentence — lives on timescales longer than twenty seconds. The model never sees them, so it cannot model them. That is a quieter version of the "diversity of speaking styles is not enough" limitation, and it is caused by the crop rather than the corpus.

What "semi-supervised" means here, exactly

The paper describes its data as "semi-supervised," which in speech usually means something specific and is worth unpacking against the alternatives.

The distinction between the last two rows is more than terminology. Whisper's labels are human-written and therefore wrong in human ways — paraphrases, missing disfluencies, translated rather than transcribed. VALL-E's labels are machine-written and wrong in model ways — systematic confusions between similar phones, boundary errors at 30 ms resolution. Different error distributions demand different tolerances, and VALL-E's is arguably the easier one because its aligner is trained on the very domain it will label.

The same bet, pointed the other way

It is illuminating to put VALL-E next to Whisper, published two months earlier. The two papers make structurally identical arguments in opposite directions.

Reading them together is the best argument that neither result is a fluke of speech. Both are instances of a general move: find the place where your pipeline demands clean data, ask what that demand is protecting, and check whether a different objective makes the protection unnecessary. Where it does, two orders of magnitude of data are usually waiting.

Hours or speakers — which axis is doing the work?

"60,000 hours" and "7,000 speakers" are quoted together, but they are different resources and it matters which one the capability depends on. Divide:

Now run the thought experiment the paper does not. Imagine two corpora, both 60,000 hours.

So the capability is a function of speaker count, and audio volume is what makes each speaker learnable. This is the direct analogue of few-shot prompting in text needing task diversity, not merely token volume: you cannot learn "adapt from the prefix" from a training distribution where the prefix never varies in the relevant way.

It also predicts the shape of the returns. Going from 10 to 1,000 speakers should be transformative; from 7,000 to 70,000 should be incremental, except along axes the first 7,000 did not cover — accents, languages, speaking styles. Which is exactly what the paper's own limitation section asks for.

The pseudo-label error budget

The phoneme sequences are machine-generated by a hybrid DNN-HMM system trained on 960 hours. That system is not perfect, so the model is learning from a corrupted content channel. How bad is it?

The paper reports no alignment accuracy, so reason about the structure of the errors instead:

• The errors are unbiased with respect to the acoustics. When the aligner mislabels a frame, the audio is still whatever it was. The audio side of every training pair is exactly right; only the label is noisy. This is the opposite of the studio-TTS situation, where the audio was pristine and the text was correct but the pairing was scarce.
• Errors are diluted by agreement. A given phoneme context appears millions of times across 60,000 hours. If the aligner is wrong 5% of the time and wrong in uncorrelated ways, the majority signal dominates and the noise contributes gradient variance rather than gradient bias.
• Systematic errors do not dilute. If the aligner consistently mislabels a particular phone in a particular context — which hybrid systems do, especially for reduced vowels and flaps — then the model learns the mistake. That is a real, unmeasured risk in this pipeline.

Note also the choice of a hybrid DNN-HMM rather than a modern end-to-end recognizer. It looks dated until you remember what is needed: not a transcript but a frame-level alignment, at a 30 ms frameshift. HMM-based forced alignment produces exactly that. CTC and attention-based systems produce transcripts and only approximate alignments. The old technology was the right tool for the job.

Batch, memory, and why 800,000 steps

Work out what one GPU is holding, since the constraint shows in the hyperparameters. Under reading B, 6,000 frames per GPU at an average crop of 1,125 frames is about five utterances per batch. Each AR sequence is roughly L + T ≈ 120 + 1,125 ≈ 1,245 positions.

The batch is small by language-model standards (96,000 frames globally, against millions of tokens for text pre-training), which is consistent with a 5 × 10⁻⁴ peak learning rate — small batches want smaller steps. And 800,000 steps at that batch is how they reached ~4.7 epochs: with a bigger batch and fewer steps the same data would have been consumed, but small-batch training is the safer choice when your labels are noisy, because more, noisier updates average out label error better than fewer, sharper ones.

Warmup is 32,000 steps — exactly 4% of training — then linear decay to zero. Nothing exotic. The point of listing these is that nothing in the recipe is exotic: this is a standard transformer training configuration from 2020, applied to a novel data source. Again, the innovation is not here.

What a replication would cost today

Useful for calibrating whether this is a frontier-lab result or a lab-scale one.

What the scale bought

Three capabilities, none of which was designed in, all of which the paper attributes to data volume and diversity.

• In-context learning. With 7,000 speakers, "continue this voice" is a task the model encountered in seven thousand variants. With one speaker it is not a task at all — there is nothing to condition on. The capability requires speaker diversity in the same way few-shot prompting requires task diversity.
• Acoustic-environment fidelity. "When the acoustic prompt has reverberation, VALL-E could synthesize speech with reverberation as well, whereas the baseline outputs clean speech." The paper's explanation is purely about data: "VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline, so VALL-E could learn the acoustic consistency instead of a clean environment only during training." A model trained only on studio audio has never observed the variable, so it cannot represent it.
• Emotion transfer. Prompted with clips from EmoV-DB, a five-emotion database, "VALL-E is able to keep the same emotion of the prompt in speech synthesis, even if the model is not fine-tuned on an emotional TTS dataset." No emotion labels, no emotion embedding, no emotional training data. Sixty thousand hours of audiobook narration contains a great deal of acted emotion, and the codec preserved it.

One more thing scale did not fix, and this is the important one: robustness. The dropped and duplicated words of Chapter 10 are not a data-coverage failure. They are a consequence of unconstrained autoregressive alignment, and more data does not remove them — the paper's own future-work sentence proposes architectural fixes ("applying non-autoregressive models or modifying the attention mechanism"), not more hours. Knowing which failures scale cures and which it does not is the difference between a research plan and a hope.

Chapter 9: Results — What the Numbers Actually Say

Eight chapters of machinery. Now the evidence. This chapter reports every number the paper publishes, and then does the thing summaries skip: normalizes them against the ground-truth ceiling, so that "better than the baseline" becomes "closed 58% of the gap to a real human recording."

What would have counted as failure

Before the numbers, decide what would have falsified the paper's claims. Doing this first is the difference between reading results and being persuaded by them.

Four claims survive their falsification tests and one is partially refuted, by the paper's own evidence, and named as a limitation. That is the profile of an honest empirical paper, and it is worth registering before reading the numbers that follow.

How everything was measured

Read the instruments before the readings, because two of the four are subjective and one is not strictly comparable across systems.

The LibriSpeech protocol: samples from test-clean between 4 and 10 seconds long, giving a 2.2-hour subset, with no speaker overlap against LibriLight. For each synthesis, "VALL-E randomly choose another utterance of the same speaker and crop a 3-seconds speech segment as the enrolled speech." Each experiment was run three times and averaged. The baseline is YourTTS, the SOTA zero-shot TTS system, from its released checkpoint.

Why two datasets and not one

The choice of LibriSpeech and VCTK is not padding, and understanding why makes the two results readable as a pair rather than a repetition.

Reporting only LibriSpeech would have been the flattering choice and would have supported every headline claim. Including VCTK is what makes the data-coverage limitation visible — and the paper does draw that conclusion from it, in its own limitations section, rather than leaving it for a critic.

LibriSpeech, objective

Two asterisks on that table before reading it. AudioLM's WER of 6.0 is the number reported in its own paper, obtained with a Conformer Transducer ASR model, not the HuBERT-CTC model used for every other row — so it is indicative, not a like-for-like comparison. And AudioLM's speaker score is blank because "AudioLM* is not open-source, we cannot evaluate its speaker score with our tool."

Now the readings, with the gap-to-ceiling arithmetic done explicitly.

GSLM's row is the argument of Chapter 1 made numerical. Speaker similarity 0.126 — barely above chance on a [−1, 1] scale — because HuBERT codes were trained to discard exactly the attribute being measured. And WER 12.4, the worst in the table, which the paper attributes to VALL-E being "trained with pseudo-phoneme instead of HuBERT/w2v-BERT codes, which enjoys better alignment quality with the input text." Text conditioning is worth roughly 6.5 WER points against a system that has to infer content from audio latents alone.

Objective metrics done; now the human studies, which are the ones the abstract's headline numbers come from.

LibriSpeech, human

Forty test cases — one randomly sampled utterance per speaker in test-clean.

The SMOS result is the strongest number in the paper, and normalizing makes it obvious:

Human raters, listening to a synthesized voice they have never heard, judged it nearly as similar to the target speaker as an actual recording of that speaker — from three seconds of enrolment, with no fine-tuning. On naturalness the picture is honest rather than triumphant: ground truth is still preferred by +0.17 CMOS, so listeners can tell.

LibriSpeech is the in-domain test. VCTK is the out-of-domain one, and the difference between the two is where this chapter's most useful finding lives.

Read the two datasets as a pair; either alone gives a misleading picture of the capability.

VCTK — the harder test, and the deliberately unfair one

VCTK has 108 speakers, none seen by VALL-E. YourTTS, however, trained on VCTK and saw 97 of them. So the paper reports two splits: all 108 speakers (where the baseline has a large advantage) and the 11 speakers YourTTS also never saw (the fair comparison).

The paper's reading: "VALL-E outperforms the baseline even if the baseline has seen 97 speakers in training… When we compare with the baseline in a fair setting (11 speakers), the performance gap becomes larger, especially when only 3s prompts are available."

Now normalize, and the story becomes considerably more interesting than "we win."

On LibriSpeech, a three-second prompt closes 58% of the gap. On VCTK, three seconds closes 13% and it takes ten seconds to reach 40%. Something about VCTK is materially harder, and the paper names it: "In terms of speaker similarity, VCTK is more challenging as it contains speakers with various accents while the training data and LibriSpeech test data do not contain various accent speakers."

That is Chapter 8's data-coverage limitation appearing as a measurement. LibriLight is LibriVox: predominantly American-accented volunteer readers. VCTK is a British corpus spanning Scottish, Irish, Northern English, and other accents. Sixty thousand hours of one accent distribution does not teach you another one, and no amount of clever prompting recovers coverage the corpus never had.

The headline here is the last cell. Ground truth scores −0.04 against VALL-E — listeners rated real human recordings very slightly less natural than the synthesis, a difference well inside noise. The paper's phrasing is appropriately restrained: "demonstrating no statistically significant difference from human recordings on this dataset."

And it immediately supplies the deflationary explanation rather than leaving the claim to inflate: "Compared to the evaluation results on LibriSpeech, VALL-E shows a better CMOS score in the comparison with ground truth, which is mainly because the average sentence length is shorter and some of the ground truth utterances also have noisy environments in VCTK." Shorter sentences give the autoregressive model fewer opportunities to drift, and some of VCTK's real recordings are themselves imperfect. This is the paper grading its own best result down, which is the behaviour you want to see.

Note also the SMOS normalization: (3.81 − 3.70) / (4.29 − 3.70) = 0.11 / 0.59 = 18.6% of the gap closed, against 88.6% on LibriSpeech. Naturalness matched human recordings on VCTK; similarity did not come close. Cloning an accent you have never heard is a different problem from producing a natural-sounding voice.

Two conventions before the human-evaluation tables, because both are counterintuitive. CMOS is reported relative to VALL-E, so VALL-E's own row is always exactly 0.00 and a negative number means the other system was judged worse. SMOS is absolute on a 1–5 scale, so higher is better and the numbers are comparable across rows.

How to read a CMOS number without over-reading it

CMOS causes more misreadings than any other number in speech papers, so slow down on it. The scale runs from −3 ("the new system is much worse than baseline") to +3 ("much better"), in intervals of 1. A rater cannot express 0.12. They pick an integer, and 0.12 is the average of many integers.

So a CMOS of +0.12 means something like: most raters said "about the same" (0), and a modest surplus said "slightly better" (+1) rather than "slightly worse" (−1). With 12 raters, that is a real but subtle preference. Concretely, if every rater scored 0 or ±1, an average of +0.12 is consistent with roughly one net rater in eight preferring VALL-E.

The paper gets this right — it says "no statistically significant difference," not "surpasses human recordings" — and it volunteers the reason VCTK flatters the model. That restraint is worth noticing precisely because the number would have made a much louder headline if stated carelessly.

SMOS deserves a matching caution. Six raters, on a 1–5 scale in 0.5 steps, with reported intervals of ±0.09–0.10. VALL-E's 4.38 ± 0.10 and ground truth's 4.5 ± 0.10 have overlapping intervals: the honest statement is "close to indistinguishable," not "equal."

WER is a robustness proxy, and a lossy one

The paper uses word error rate as its robustness measure. It is the right family of metric and the wrong resolution, for three reasons worth understanding before treating 5.9 as a quality score.

• It confounds error types. WER = (substitutions + deletions + insertions) / reference words. A model that drops one word in every twentieth utterance and a model that lightly slurs consonants everywhere can post the same number, and they are very different products. The Limitations section names deletions and duplications specifically; the metric never separates them.
• It confounds the synthesizer with the recognizer. Ground truth scores 2.2, not 0 — that 2.2 is the ASR model's own error on real audio. So VALL-E's 5.9 is 2.2 points of recognizer error plus roughly 3.7 points of synthesis error, and the two do not simply add (a synthesis error can also make the recognizer fail on neighbouring words).
• No language-model fusion cuts both ways. Using CTC without an LM is the honest choice — an LM would hallucinate dropped words back into the transcript and hide exactly the failures being measured — but it inflates all absolute numbers relative to what a production ASR stack would report.

The comparison that is not fair, and the one that is

Two asymmetries run in opposite directions and largely cancel, but you should track them separately rather than trusting the cancellation.

None of this is misconduct — every asymmetry is disclosed in the paper's own text. But "VALL-E beats YourTTS" is a claim about two complete systems, and the interesting question, "would the VALL-E architecture beat the YourTTS architecture on equal data?", is not answered anywhere. Given that the paper's thesis is the data is the contribution, that unanswered question is arguably the point rather than an oversight.

Three qualitative findings that no metric captures

Diversity. Because generation is sampling, the same input produces different outputs. The paper synthesizes each utterance twice with different random seeds and plots the waveforms (Figure 4). On the LibriSpeech sample "After early nightfall, the yellow lamp would light up here and there the squalid quarter of the brothels," the two takes "have different lengths and phrase durations, where the first has a faster speech rate." On the VCTK sample "I must do something about it," the accents differ: "The second output emphasizes the word 'must' with a larger amplitude whereas the first output does not."

The paper then makes an argument that is easy to miss and genuinely useful: this makes VALL-E a good pseudo-data generator for speech recognition. ASR training benefits from varied speakers and acoustic conditions, which a one-to-one regression TTS cannot supply. "Considering the diversity feature of VALL-E, it is an ideal candidate to generate pseudo-data for speech recognition." Chapter 1's temperature knob, cashed out as a data-augmentation pipeline.

Acoustic environment maintenance. "When the acoustic prompt has reverberation, VALL-E could synthesize speech with reverberation as well, whereas the baseline outputs clean speech." No reverberation parameter exists anywhere in the system. The room came in through the prompt's tokens and left through the decoder, because nothing in the pipeline was ever instructed to remove it.

Emotion maintenance. Prompts drawn from EmoV-DB, a database of five emotions: "VALL-E is able to keep the same emotion of the prompt in speech synthesis, even if the model is not fine-tuned on an emotional TTS dataset." Compare the classical approach the paper cites — train on a labelled emotional TTS corpus with an emotion label per utterance. VALL-E has no emotion labels, no emotion embedding, and no emotional fine-tuning, and transfers emotion zero-shot.

One structural observation about the results section as a whole: every headline claim is supported by both an objective metric and a human study, and where the two disagree the paper reports both rather than choosing. That is worth noting because it is rarer than it should be, and it is why the normalized comparisons in this chapter are possible at all.

Reading the diversity figure carefully

Figure 4 is two waveform plots per sentence, one per random seed, and it is easy to under-read as "the outputs differ, how nice." The paper points at two specific differences, and each is a distinct capability.

The second row is the more interesting claim. Different emphasis is a different reading of the sentence — "I must do something" versus "I must do something" — which is a semantic-adjacent choice that a regression model, outputting the conditional mean, must average away into flatness. Chapter 0's over-smoothing derivation predicted exactly this, and here it is on the other side.

Pseudo-data for ASR — is the argument any good?

The paper claims VALL-E "is an ideal candidate to generate pseudo-data for speech recognition." Take it seriously for a moment, because it is a nontrivial argument with a real weakness.

The case for: ASR training benefits from variation in speaker, rate, environment, and prosody. VALL-E supplies all four — unlimited speakers via prompting, sampled rate and prosody, and environment inherited from prompts. A regression TTS system supplies none of them; synthesizing a million utterances gives you a million near-identical deliveries of different sentences.

The case against, which the paper does not raise: the pseudo-data is generated by a model trained on machine-generated labels from an ASR system. There is a loop — ASR labels the audio, the TTS model learns from those labels, and the TTS output is then used to train ASR. Any systematic error in the original aligner is at risk of being reinforced rather than corrected, because it now appears in the audio itself rather than only in the labels.

What does 0.580 actually mean?

Speaker-similarity scores are the least intuitive numbers in the paper because the scale has no natural anchors. Build some.

WavLM-TDNN outputs a cosine similarity in [−1, 1] between speaker embeddings. Two things fix the scale for this evaluation:

The ceiling being 0.754 rather than 1.0 is the crucial anchor and the reason raw scores mislead. A naive reading of "0.580 out of 1.0" says 58% of the way there. The correct reading uses 0.754 as the top and 0.337 as the incumbent, which is why every comparison in this chapter is normalized. Absolute cosine similarities are meaningless without the same-speaker ceiling reported alongside them — and the paper does report it, which is good practice.

One methodological detail supports the numbers: "Each experiment runs three times and the average score is reported." Three runs average out sampling variance, which matters because VALL-E's output is stochastic by design. It does not control for the choice of enrolment clip, which is randomly drawn per sample and is plausibly a larger source of variance — a three-second window containing a rich phoneme inventory is worth more than three seconds of one drawn-out vowel.

Restating the results for a colleague

If you had sixty seconds to convey what was measured, this is the shape of it — and being able to produce this summary from memory is a decent test of the chapter.

• On read English audiobook speech (LibriSpeech), it works well. Three seconds of an unseen speaker, no fine-tuning, and human raters put similarity within noise of a real recording (4.38 vs 4.5). Objectively it closes 58% of the gap from the previous best to the ceiling.
• On accented speech (VCTK), it works considerably less well. 13% of the gap closed at three seconds, rising to 40% at ten. It still beats a baseline that trained on most of those speakers, which says more about the baseline than about VALL-E.
• Naturalness is a smaller win than similarity. +0.12 and +0.23 CMOS are real but modest; ground truth is still preferred on LibriSpeech.
• Intelligibility is the weak spot. WER 5.9 against 2.2 for real audio, and Chapter 6's ablation arithmetic attributes roughly 3.1 of those 3.7 points to the autoregressive stage.
• Three capabilities came free. Output diversity, acoustic-environment transfer, and emotion transfer — none designed, none quantified.

Last framing note before the summary table. This chapter has separated three things that headlines usually merge: what was measured, what it means once normalized against a ceiling, and how strongly the evidence supports the claim attached to it. The table below keeps them separate.

The honest scoreboard

Notice the row that is missing: there is no robustness metric beyond aggregate WER, and no measurement at all of how often words are dropped or duplicated. Chapter 10 opens with the failure mode the paper concedes in prose but never counts.

Chapter 10: Limits, Safety, and What Came Next

The paper's final section is titled "Conclusion, Limitations, and Future Work," and it is unusually direct. Three technical limitations, one paragraph on broader impacts. This chapter takes all four seriously — the failure modes first, because they follow mechanically from decisions we have already derived, then the safety question, which is the reason this paper is remembered outside the speech community.

The limitations, ranked by how much they mattered

The paper lists three. With hindsight they did not age equally, and ranking them is a useful exercise before the details.

The interesting entry is the one that is not on the paper's list. Papers are good at naming the limitations that show up in their own metrics and less good at naming the ones their evaluation protocol never measures. Latency was never measured here, so it was never listed — and it turned out to be the axis the field moved along.

Limitation 1: synthesis robustness

The paper: "We observe that some words may be unclear, missed, or duplicated in speech synthesis. It is mainly because the phoneme-to-acoustic language part is an autoregressive model, in which disordered attention alignments exist and no constraints to solving the issue."

Nothing in that sentence should surprise you, because Chapters 3 and 4 built the cause.

The classical TTS pipelines had defences. FastSpeech has an explicit duration predictor and a length regulator: each phoneme is told how many frames it gets, and the decoder cannot disobey. Tacotron 2 has location-sensitive attention that is biased toward advancing monotonically. Both trade expressiveness for a guarantee. VALL-E has neither, and the paper's future-work sentence proposes importing them back: "which was addressed by applying non-autoregressive models or modifying the attention mechanism in modeling. In the future, we would like to leverage these techniques to solve the issue."

Note what the paper does not do: it never counts the failures. There is no deletion rate, no duplication rate, no per-utterance failure analysis — only aggregate WER, which folds all three error types plus ASR error into one number. A WER of 5.9 against ground truth's 2.2 tells you 3.7 points of something went wrong; it does not tell you whether that is one catastrophic utterance in twenty or a light dusting of slurred consonants everywhere. For a system being proposed for production use, that is a real gap in the evaluation.

One structural detail the lab makes visible: the failures are not uniformly distributed. They cluster where the path is already unstable — long utterances, unusual phoneme sequences, and high temperature. That is why VALL-E-continual scores 3.8 WER against plain VALL-E's 5.9: three of its seconds are ground-truth audio, so the model has fewer frames in which to lose its place, and it starts from a state that is by construction correctly aligned.

Why the failures cluster

One more property of the robustness limitation, visible in the lab above and easy to miss: errors are not sprinkled uniformly. They arrive in bursts, and the reason is compounding.

Once the AR model's implicit alignment slips — say it has produced the acoustics for "confirmed" but its internal sense of position still points at "is" — every subsequent token is generated from a state that encodes the error. The model is not confused about one frame; it is confused about where it is in the sentence, and that confusion persists until something re-anchors it.

This is why Chapter 9 argued that a per-utterance failure rate is the metric this system actually needed. Bursty, compounding errors and a smooth aggregate average are a bad match, and the aggregate is the only thing published.

Limitation 2: data coverage

"Even if we use 60K hours of data for training, it still cannot cover everyone's voice, especially accent speakers. The worse result on VCTK than LibriSpeech also implies insufficient coverage of accent speakers. Moreover, the diversity of speaking styles is not enough, as LibriLight is an audiobook dataset, in which most utterances are in reading style."

Chapter 9 quantified it: 58.3% of the similarity gap closed on LibriSpeech, 13.2% on VCTK at the same prompt length. The paper's response is more of the same medicine — "we will further scale up the training data" — and then a genuinely bold prediction: "We believe the zero-shot TTS task could be almost solved through our approach with model and data scale-up."

Worth pausing on how that prediction aged. On the narrow reading — read-aloud English TTS with high speaker similarity — it was broadly right; successors reported human-parity results within eighteen months. On the broad reading — speech is solved — it was not, because the frontier moved to problems scale does not touch: latency, interruption handling, and simultaneous listening-and-speaking. The last section of this chapter is about that.

Limitation 3: model structure

"Now, we use two models to predict codes of different quantizers. A promising direction is to predict them with a large universal model. Another interesting direction is using full NAR models to speed up model inference in the framework."

Two separate wishes. Unifying the models is an elegance argument: two transformers with separate parameters, separate training runs, and separate embedding conventions is a lot of machinery for one distribution. Going fully non-autoregressive is the latency argument — Chapter 5's ledger showed VALL-E at roughly 1.5× slower than real time, with 99.1% of the latency in the AR phase.

And Chapter 5 also showed why the second wish is hard: fully-NAR requires knowing T in advance, which requires a duration predictor, which the paper rejected because "it is hard to train a length predictor for different speakers." You cannot remove the AR model without re-solving the problem it was introduced to solve. Later systems escape by changing the frame rate rather than the decoder — if the codec runs at 12.5 Hz instead of 75 Hz, ten seconds is 125 steps rather than 750, and the AR bottleneck shrinks by 6× without giving up implicit duration modelling.

The safety chapter

The paper's Broader Impacts paragraph, in full: "Since VALL-E could synthesize speech that maintains speaker identity, it may carry potential risks in misuse of the model, such as spoofing voice identification or impersonating a specific speaker. To mitigate such risks, it is possible to build a detection model to discriminate whether an audio clip was synthesized by VALL-E. We will also put Microsoft AI Principles into practice when further developing the models."

Four sentences. Take them seriously, because the capability described in this lesson is genuinely different from what came before, and the difference is worth stating precisely.

The mitigation offered — "it is possible to build a detection model" — is worth examining rather than accepting, because it is the standard proposal and it has known weaknesses.

• It is an arms race with an asymmetry. A detector must generalize to synthesis systems that do not exist yet. A generator only has to defeat detectors that already do. The defender's problem is strictly harder.
• Post-processing is cheap. Codec artifacts are the detector's signal; re-encoding through a phone codec, adding room noise, or resampling attacks exactly that signal — and the transmission channels where impersonation matters (telephony, voice messages) apply those transforms anyway.
• Detection is downstream of harm. By the time a clip is analysed, the fraudulent call has happened or the recording has spread. Detection helps forensics; it does not help the person on the phone.
• It places the burden on the victim. Everyone whose voice can be cloned would need access to detection for every clip they are alleged to have spoken. That is not a deployable model of protection.

What would a stronger mitigation stack look like? Not one thing, and not detection alone:

The general lesson generalizes past speech. When a capability's cost falls by three orders of magnitude — hours of clean audio and a training run, to three seconds and an API call — controls that relied on the cost being high stop working, and controls have to be rebuilt around consent and provenance instead. That transition is the interesting engineering problem, and it is not a machine-learning problem.

Limitation 4, unlisted: latency

The paper's own list has three entries. Add a fourth that it does not name, because with hindsight it turned out to be the one that mattered most.

Chapter 7's dependency analysis showed that no audio can be decoded until the AR phase has finished, because the NAR passes need the complete first-codebook layer. Time-to-first-audio therefore equals total generation time, which Chapter 5 estimated at roughly 1.5× real time.

The last row is not a matter of degree. A duplex system must model its own output and the user's speech simultaneously, and must be able to abandon a partially-spoken utterance mid-word. VALL-E generates one complete utterance from a fixed text; there is no point at which it is listening, and no mechanism for stopping early that is not "throw away what you generated."

That gap is why the voice-interface line of work after 2023 did not simply scale VALL-E. It rebuilt the substrate: a lower-frame-rate codec so the AR chain is short enough to run live, a semantically-distilled first codebook so the coarse layer carries content as well as acoustics, and parallel streams so listening and speaking coexist. VALL-E proved speech is a language; the next generation had to make that language fast enough to have a conversation in.

Limitation 5, unlisted: the evaluation

Chapter 9 catalogued the gaps; collect them here as a single honest ledger, because "what was not measured" is part of understanding a paper.

Notice that the AR/NAR trade — the paper's central architectural claim — is argued entirely from complexity analysis (O(T) → O(1)) and never measured in seconds. Chapter 5's ledger had to derive the numbers because the paper does not report them.

The deployment questions this capability forces

If you were reviewing a product built on this method, the technical questions are not the hard ones. These are:

• Where did the enrolment audio come from? Live-recorded challenge phrase, or uploaded file? Everything about the risk profile turns on this one answer, and it is a product decision, not a model decision.
• Can the voice owner revoke? The "model" for a voice is 1,800 integers held by whoever has the audio. There is no key to rotate and no weight to delete.
• Is generated output marked? At the audio level (watermark), the file level (provenance metadata), and the interaction level (disclosure to the listener) — three different layers, each defeatable alone.
• What happens on the phone network? Telephony strips watermarks and metadata as a matter of course. The channel where impersonation pays best is the channel where technical mitigations survive worst.
• Who is liable when it works exactly as designed? Nothing in this system malfunctions when it clones a voice without consent. The failure is entirely in the deployment surface.

That last point is the one to carry out of this chapter. This is not a safety problem caused by a model behaving unexpectedly. It is a capability whose ordinary, correct operation is the hazard — which means the controls cannot live in the model, and a detection classifier is not a control at all, only a forensic aid.

The lineage — what VALL-E started

The reframe in Chapter 1 turned out to be the durable contribution. Once speech is tokens, everything the language-model ecosystem knows applies: prompting, scaling, instruction tuning, multimodal fusion, streaming. The systems that followed all inherit the substrate and attack a different one of VALL-E's limitations.

Three inheritances, three attacks:

• Cross-lingual (VALL-E X). If the prompt carries the voice and the phoneme sequence carries the content, the two can come from different languages — speak English in the voice of a Mandarin prompt. The factorization Chapter 6 measured as a double dissociation is precisely what makes this possible.
• Quality and robustness (VALL-E 2). Direct attacks on the Limitation-1 failure modes and on the sampling degeneracy of Chapter 4, with human-parity claims on LibriSpeech and VCTK. The architecture is recognizably the same two-decoder design.
• Latency and duplex interaction (Moshi and the omni models). The most consequential direction, and the one that had to change the substrate rather than the decoder. VALL-E's 75 Hz frame rate makes 750 AR steps for ten seconds unavoidable; dropping the codec to 12.5 Hz makes it 125. Add a semantically-distilled first codebook, a depth transformer over the residuals, and two simultaneous streams (the model's own voice and the user's) and you get a system that can be interrupted — which is the capability VALL-E has no path to, because it must know the whole sentence before it starts.

Follow the arc here: EnCodec built the vocabulary · AudioLM proved a language model over it could generate speech · VALL-E added text control and prompting · Moshi made it conversational in real time. On the Gleam side, TTS architectures covers the cascaded systems this paper displaced, neural audio codecs derives RVQ from zero, and Whisper is the mirror image — the same weak-supervision-at-scale bet, pointed at recognition instead of synthesis.

The arc, on one timeline

Placing VALL-E among its neighbours makes the pace of the period visible — and makes clear how little separated the pieces from each other.

Six months separate AudioLM from VALL-E, and one component — a phoneme channel — separates them architecturally. That is a useful thing to notice about research: the decisive addition is often small, and it is decisive because of what it makes possible (control, prompting, evaluation against text) rather than because of what it adds computationally.

Open questions this paper leaves

A good measure of whether a paper has been understood is whether you can state what it did not settle. Six, ordered from most tractable to most open.

Notice how many of these are cheap. That is characteristic of a paper whose contribution is a reframing rather than an optimization — the reframing is the hard part, and the measurement backlog it creates is large and mostly straightforward.

How to check this lesson's derived claims

Several numbers here were derived rather than quoted, and you should be able to reconstruct each in a minute. Nothing below requires a GPU.

Anything not in that list and not attributed to the paper is flagged in the text as reasoning or estimation — the latency figures in Chapter 5, the FLOP estimate in Chapter 8, the prompt-hygiene predictions in Chapter 6, and the per-codebook content assignment in Chapter 5. Keep the boundary between measured and inferred sharp; it is the difference between knowing the paper and having a feeling about it.

Cheat sheet — every symbol, every number

One closing thought about why this paper matters more than its metrics. The contribution is not the AR/NAR split — that is a good engineering trade worth 7.93×. The contribution is the demonstration that a whole subfield's central constraint was self-imposed by its loss function. Text-to-speech required clean studio data because regression required clean targets. Change the objective, and sixty thousand hours of noisy audiobooks — free, already recorded, sitting on a server — become the best training set anyone had. That move is available in more places than speech, and looking for it is a better habit than memorizing this architecture.