Moshi: Speech as a Language, Conversation as a Duet

Chapter 1: Speech as Tokens — and the Sequence-Length Wall

We ended Chapter 0 with a commitment: one model, audio in, audio out, no text transport. Now the practical question. A transformer language model is a machine for predicting the next element of a discrete sequence. Speech is a continuous pressure wave. Before anything else can happen, somebody has to turn one into the other.

The naive answer is to model the waveform samples directly. Moshi's audio runs at 24 kHz, so one second of speech is 24,000 numbers. Predicting them one at a time autoregressively means 24,000 forward passes per second of audio. For a 7B model that is not slow; it is roughly four orders of magnitude beyond feasible. The waveform is the wrong unit.

So we compress. Not to save disk space — to make the sequence short enough to model. This is the single most important reframing in the whole audio-language-model literature: a neural audio codec is not a compression tool, it is a tokenizer. Its bitrate matters far less than its frame rate, because frame rate is what shows up in the transformer's sequence length.

Why text tokenizers do not transfer

It is tempting to reach for the text playbook — byte-pair encoding merges frequent symbol pairs into longer units, so why not do that to audio? The reason is worth stating because it explains why audio tokenization is a learned problem while text tokenization is a counting problem.

The second row is the fatal one. BPE works because the same symbol sequence recurs exactly; audio never repeats exactly, so any discretization must first learn a space in which "the same sound" means "nearby vector". That learning is the codec. And because the mapping is lossy, the choice of what to discard becomes an objective — which is precisely the fight between reconstruction and phonetics that Chapter 3 has to referee.

One more inherited constraint: since frames are fixed-length, timing is not compressible. A text model can spend one token on a common word and five on a rare one. An audio model spends 12.5 frames per second regardless of what is being said, including silence. That is why the padding fraction in Chapter 6's text stream is ~65% — the text stream is forced to run on the audio's clock, not its own.

Two species of audio token

The literature before Moshi had converged on two very different kinds of discrete audio unit, and understanding why both existed is the setup for Mimi's central trick.

Acoustic tokens come from a neural codec — SoundStream, EnCodec — trained as an autoencoder with a discrete bottleneck and a reconstruction objective. Their job is fidelity: decode them and you get the waveform back, with the speaker's timbre, the room, the background noise, everything. Because they are trained to reconstruct, they spend their capacity on whatever is perceptually loud, not on whatever is linguistically meaningful.

Semantic tokens come from quantizing the internal representations of a self-supervised speech model — HuBERT, wav2vec 2.0, w2v-BERT, WavLM. Their job is linguistic structure: they correlate strongly with phonetic content, which makes them predictable by a language model. But you cannot reconstruct good audio from them, because the SSL model was trained to discard speaker and channel information as nuisance.

AudioLM's answer was to use both, in a hierarchy: first generate semantic tokens (which carry the content), then condition acoustic-token generation on them (which supplies the sound). It worked — it is the reason unconditioned speech generation stopped being babble — and Moshi inherits the hierarchical idea wholesale.

But look at the last row of that table again, because it is the row that kills the approach for dialogue. Semantic tokens come from non-causal encoders. A non-causal encoder needs the future to compute the present. You cannot run it in a live conversation. And even if you could, you are now running two encoders on every incoming frame — the paper calls this "a non-negligible computational burden", which is a polite way of saying you have doubled the cost of the cheap part of the system.

The sequence-length wall, in numbers you can do by hand

Before the codec, one more budget. Suppose we have our tokenizer. How long is the sequence?

Let the codec produce Q tokens per frame at f_r frames per second. Token rate is the product:

For Mimi's final configuration, Q = 8 and f_r = 12.5 Hz. Work it out step by step, no shortcuts:

arithmetic — the token budget, done by hand
Step 1. Tokens per frame                Q  = 8
Step 2. Frames per second               fr = 12.5
Step 3. Tokens per second               8 x 12.5 = 100
Step 4. Seconds in a 5-minute context   5 x 60   = 300
Step 5. Frames in that context          300 x 12.5 = 3,750
Step 6. Tokens in that context          3,750 x 8  = 30,000
Step 7. Text tokens for the same 5 min  300 x 3.5  = 1,050   (English speech is ~3-4 text tok/s)
Step 8. Ratio                           30,000 / 1,050 = 28.6x

Those are the paper's own numbers: "with Q = 8 codebooks at a frame rate of 12.5 Hz, one would require a sequence length of 100 steps per second of audio to generate. To model 5 minutes of audio, this would amount to 30,000 timesteps."

Two separate disasters hide in step 6, and it is worth separating them because they are fixed by different tricks:

Play with the budget. The simulation lets you move frame rate and codebook count independently and watch three quantities respond: token rate, sequence length for a 5-minute conversation, and bitrate. Try to find a setting that gets the sequence under 5,000 tokens without dropping below 1 kbps — and notice which knob actually does the work.

Here is what the sim should have taught you: frame rate and codebook count are not interchangeable, even though they multiply into the same token rate. Halving the frame rate halves the number of timesteps; halving Q leaves the timestep count untouched and merely reduces how much is predicted per timestep. Since the expensive model runs once per timestep, frame rate is the knob that buys real-time inference. Q is the knob that buys audio quality. Moshi pushes frame rate as low as it can (12.5 Hz — the paper notes this is "crucial to achieve the low latency of Moshi, since generating one temporal frame of audio tokens with Moshi requires a full forward pass through the Temporal Transformer") and then recovers quality by stacking Q = 8.

What a token physically is, and what the sequence costs in memory

It is easy to say "audio becomes tokens" and never picture the object. So picture it. One Mimi token is an integer between 1 and 2048. Eight of them describe 80 milliseconds of sound. That is the entire representation — no floats, no spectrogram, no waveform. Eight small integers, eighty times a second.

Written out, one second of speech is a 12 × 8 grid of integers:

what one second of audio actually looks like to the model
frame:      0     1     2     3     4     5     6     7     8     9    10    11    12
k=1 sem  [ 1443  1443   902   902   331   331   331  1780  1780   445   445  1102  1102 ]
k=2      [  774   201  1655   88   1290  1290   640   19    902   1444  77    1655  208 ]
k=3      [ 1001  1902   345  1120   88    640  1655  1301   77    902  1443   201  990 ]
  ...        (five more rows of the same shape)
k=8      [  512   933  1204  1888   64    712   399  1522   840   66   1477   303  911 ]

Notice the semantic row repeats values across neighbouring frames — phones last
longer than 80 ms, so the same centroid recurs. The acoustic rows do not: they
carry fine detail that changes every frame. That difference is exactly what makes
row 1 easy for a language model and rows 2-8 nearly noise.

That last observation is the whole reason Chapter 3 exists, and it is worth predicting now: if row 1 is smooth and predictable while rows 2–8 are close to noise, then a loss that treats all eight equally will spend seven eighths of its effort on the unpredictable part. Chapter 7 fixes it with a weight of 100.

Now the memory cost, which nobody mentions until it bites. An autoregressive transformer caches keys and values for every position it has seen. For the Temporal Transformer — 32 layers, dimension 4096, two tensors per layer, 2 bytes per element in bfloat16:

arithmetic — the KV cache for a 5-minute conversation
per position, per layer:  2 (K and V) x 4096 dims x 2 bytes = 16,384 bytes = 16 KB
per position, 32 layers:  16 KB x 32                        = 512 KB

RQ-Transformer (positions = frames):
  5 min = 3,750 positions   ->  3,750 x 512 KB  = 1.92 GB

flattened model (positions = tokens, K = 17):
  5 min = 63,750 positions  ->  63,750 x 512 KB = 32.6 GB

Difference: 17x the cache, for the same conversation.
(The Depth Transformer's cache is negligible: it is at most K = 17 positions
 long and 4x narrower, and it is discarded at the end of every frame.)

So the factorization of Chapter 4 is not only a compute trick. It is what keeps a five-minute conversation inside the memory of a single accelerator alongside a 7B model's weights. Sequence length is a cost that shows up in three places at once — forward passes, attention area, and cache — and frame rate is the knob that controls all three.

The same computation, three ways

Following the house rule that every calculation should appear as arithmetic, as explicit code, and as a one-liner:

python — step by step, then compact
# 1. explicit, one line per quantity
frame_rate  = 12.5          # Hz, Mimi
codebooks   = 8             # Q
card        = 2048          # N_A, centroids per codebook
duration_s  = 300           # 5 minutes

bits_per_token = math.log2(card)                 # 11.0
tokens_per_s   = codebooks * frame_rate          # 100.0
n_frames       = int(duration_s * frame_rate)  # 3750
n_tokens       = n_frames * codebooks            # 30000
bitrate_bps    = tokens_per_s * bits_per_token   # 1100.0  -> 1.1 kbps

# 2. the same thing as a numpy sweep over configurations
frs = np.array([12.5, 25.0, 50.0, 75.0])
qs  = np.arange(1, 17)
tok_rate = qs[:, None] * frs[None, :]        # (16, 4) grid
seq_len  = tok_rate * duration_s                # tokens for the whole conversation
steps    = frs * duration_s                     # temporal steps — independent of Q!

# 3. the one-liner you would actually write in a notebook
bitrate = lambda q, fr, card=2048: q * fr * math.log2(card)
bitrate(8, 12.5)   # 1100.0 bits/s

Note the line steps = frs * duration_s. It has no q in it. That absence is the RQ-Transformer's entire reason for existing, and we will meet it formally in Chapter 4.

Numbers to carry forward

Inline check before you move on

Three quick ones. Answer them in your head; the answers are one line below each, so cover the page if you want the test to be real.

(a) A codec runs at 25 Hz with Q = 4 and codebooks of size 1024. What is its bitrate, and how many temporal steps does a 3-minute conversation need?
log₂(1024) = 10 bits; 10 × 4 × 25 = 1,000 bps = 1.0 kbps. Steps = 25 × 180 = 4,500 — note this does not depend on Q at all.

(b) Why can a semantic tokenizer from a self-supervised model not simply be made causal by masking its attention?
You could, but the model was trained bidirectionally; masking at inference puts it out of distribution. Retraining it causally costs exactly the future context that made its representations phonetic. Chapter 3's answer — keep the teacher non-causal, keep the student causal, and only connect them during training — is the way around this.

(c) Two configurations have the same token rate: Q = 8 at 12.5 Hz and Q = 4 at 25 Hz. Which has the lower latency floor?
The 25 Hz one — its frame is 40 ms, so the acquisition term halves. Which is why Chapter 8's ledger and Chapter 1's compute budget pull in opposite directions, and 12.5 Hz is a negotiated settlement rather than an optimum.

Chapter 2: Mimi I — the Ladder Down to 12.5 Hz

Chapter 1 left us with a shopping list: a causal encoder that turns 24 kHz audio into 12.5 frames per second, a discrete bottleneck that keeps enough information to reconstruct convincing speech, and a decoder that runs in a stream. This chapter builds the skeleton. Chapter 3 adds the strange organ that makes it work for language modelling.

Mimi's ancestry is explicit and worth knowing, because almost every design decision is either inherited or a deliberate break: a SeaNet convolutional autoencoder with a Residual Vector Quantizer, the same recipe as SoundStream and EnCodec. If you have read our EnCodec lesson, you already know the frame: conv encoder down, quantize, conv decoder up, trained with reconstruction plus adversarial losses. Mimi changes four things — the frame rate, the bottleneck, the loss recipe, and the quantizer topology — and each change is a response to a dialogue-specific requirement.

The stride ladder

The encoder is a stack of residual convolutional blocks that interleave dilated convolutions (which widen the receptive field without downsampling) and strided convolutions (which downsample), with ELU nonlinearities and weight normalization. Every convolution is causal — padded only on the left — so no output ever depends on a future sample.

The downsampling schedule is the part you should memorize: four blocks with strides (4, 5, 6, 8), then a final 1-D convolution with stride 2. Multiply them:

arithmetic — where 12.5 Hz comes from
Step 1. Block strides                4 x 5 = 20
Step 2.                              20 x 6 = 120
Step 3.                              120 x 8 = 960
Step 4. Final stride-2 convolution   960 x 2 = 1,920 samples per latent frame
Step 5. Input sample rate            24,000 samples per second
Step 6. Frame rate                   24,000 / 1,920 = 12.5 frames per second
Step 7. Frame duration               1 / 12.5 = 0.080 s = 80 ms

Each latent frame is a vector of dimension D = 512. So the encoder is a map

where L is the number of waveform samples. Twelve seconds of training audio, the window Mimi is trained on, is 288,000 samples in and 150 frames out. The decoder mirrors the encoder with transposed convolutions, mapping R^{S × D} back to 24 kHz.

Explore the ladder. Each rung shows the sample rate after that stage, the samples-per-frame accumulated so far, and the resulting frame duration. Toggle the final stride-2 conv off to see what the frame rate would have been without it — and what that would have done to the language model's workload.

What "causal" costs, concretely

"All convolutions are causal" is one clause in the paper and a real constraint in practice, so unpack it. A convolution with kernel size k normally centres its window on the output position, using (k−1)/2 samples on each side. Causal convolution shifts the window entirely to the left: output t sees inputs t−k+1 through t, and the layer is padded on the left only.

The cost is receptive field. A non-causal stack of the same depth sees the same total span but centred; a causal stack sees the same span entirely in the past. To have equivalent context about the current instant, a causal model needs twice the depth — or it accepts less context. Mimi accepts less, and compensates with the transformer bottleneck, whose 250-frame window is a much cheaper way to buy context than more convolution layers.

The benefit is that streaming becomes trivial rather than an afterthought. Each layer keeps a small ring buffer of its last k−1 inputs; a new sample arrives, each layer advances one step, and no output ever needs revising. Compare that to a non-causal codec, where the first output cannot be computed until enough of the future has arrived — which is a latency term, in the ledger, forever.

Two transformers in the bottleneck

A purely convolutional encoder has a fixed, local receptive field. Mimi adds a small Transformer immediately before quantization and another immediately after (on the decoder side), each with:

The ablation (Table 3 of the paper) shows both transformers earn their place perceptually, and the encoder-side one does something extra: it makes the distillation of Chapter 3 work much better. The authors' explanation is worth internalizing because it generalizes: "distilling a large Transformer based encoder into a purely convolutional one is challenging, while increasing the capacity and receptive field of the encoder helps." When you distil, the student needs an architecture capable of representing what the teacher represents.

Vector quantization, by hand

Now the discrete bottleneck. Vector quantization is the simplest idea in the whole system, and doing it once by hand removes all the mystery.

A codebook is a list of N_A vectors called centroids. To quantize a vector, find the nearest centroid and emit its index. That index is the token. Reconstruction means looking the index back up.

One codebook is not enough — with 2048 centroids you get 11 bits for an entire 512-dimensional frame, which is hopeless. Residual vector quantization fixes this by iterating: quantize, subtract, quantize the leftover, subtract again. Let us do it in two dimensions with tiny codebooks so every number is checkable.

Take the vector to encode:

Level 1. Codebook C₁ = { c₁=(1.0, 0.0), c₂=(0.0, 1.0), c₃=(−1.0, 0.0), c₄=(0.0, −1.0) }. Squared distances, every term written out:

level 1 — nearest centroid search
d(x, c1) = (0.90 - 1.0)^2 + (-0.40 - 0.0)^2 = (-0.10)^2 + (-0.40)^2 = 0.01 + 0.16 = 0.17   <-- min
d(x, c2) = (0.90 - 0.0)^2 + (-0.40 - 1.0)^2 = 0.81 + 1.96 = 2.77
d(x, c3) = (0.90 + 1.0)^2 + (-0.40 - 0.0)^2 = 3.61 + 0.16 = 3.77
d(x, c4) = (0.90 - 0.0)^2 + (-0.40 + 1.0)^2 = 0.81 + 0.36 = 1.17

token_1 = 1        (index of c1)
residual r1 = x - c1 = (0.90 - 1.00, -0.40 - 0.00) = (-0.10, -0.40)
error norm  ||r1|| = sqrt(0.01 + 0.16) = sqrt(0.17) = 0.4123

Level 2. A different codebook C₂ = { d₁=(0.0, −0.5), d₂=(−0.25, −0.25), d₃=(0.3, 0.2), d₄=(0.0, 0.0) }, applied not to x but to the residual r₁:

level 2 — quantize the leftover
d(r1, d1) = (-0.10 - 0.00)^2 + (-0.40 + 0.50)^2 = 0.01 + 0.01 = 0.02    <-- min
d(r1, d2) = (-0.10 + 0.25)^2 + (-0.40 + 0.25)^2 = 0.0225 + 0.0225 = 0.045
d(r1, d3) = (-0.10 - 0.30)^2 + (-0.40 - 0.20)^2 = 0.16 + 0.36 = 0.52
d(r1, d4) = (-0.10)^2 + (-0.40)^2 = 0.17

token_2 = 1        (index of d1)
residual r2 = r1 - d1 = (-0.10 - 0.00, -0.40 + 0.50) = (-0.10, 0.10)
error norm  ||r2|| = sqrt(0.01 + 0.01) = 0.1414       (was 0.4123 — a 2.9x reduction)

Level 3. C₃ = { e₁=(−0.1, 0.1), e₂=(0.1, −0.1), e₃=(0, 0), e₄=(−0.2, −0.2) }:

level 3 — and the reconstruction
d(r2, e1) = (-0.10 + 0.10)^2 + (0.10 - 0.10)^2 = 0 + 0 = 0.00   <-- exact hit
token_3 = 1,   residual r3 = (0, 0)

reconstruction = c1 + d1 + e1
               = (1.0, 0.0) + (0.0, -0.5) + (-0.1, 0.1)
               = (1.0 + 0.0 - 0.1,  0.0 - 0.5 + 0.1)
               = (0.90, -0.40)  =  x     exactly recovered with 3 tokens

error curve: 0.4123  ->  0.1414  ->  0.0000

Three properties fall straight out of that arithmetic, and all three matter later:

• The levels are ordered. Level 1 carries the coarsest, largest-magnitude structure; each subsequent level carries strictly less energy. This is why "keep the first k codebooks" is a valid way to lower the bitrate — the trick called quantizer dropout, which Mimi uses so one trained codec serves several bitrates.
• Later levels depend on earlier ones. Codebook 2 never sees x, only what codebook 1 failed to capture. Change level 1 and every subsequent residual changes. Hold on to this; it is the exact reason the split RVQ in Chapter 3 has to exist.
• Errors are monotone but not necessarily fast. Nothing guarantees a big drop per level. In real codecs the marginal value of level 8 is small — which is why the language model can weight it less (Chapter 7).

python — RVQ three ways
# 1. step-by-step, mirroring the arithmetic above
import numpy as np
x  = np.array([0.90, -0.40])
C1 = np.array([[1,0],[0,1],[-1,0],[0,-1]], dtype=float)
r  = x.copy(); tokens = []
for C in (C1, C2, C3):
    d = ((C - r) ** 2).sum(axis=1)   # squared distance to every centroid
    k = int(d.argmin())               # nearest
    tokens.append(k)
    r = r - C[k]                    # subtract, then quantize the leftover
# tokens == [0, 0, 0]  (0-indexed);  r == [0., 0.]

# 2. as a reusable function over a whole batch of frames
def rvq(X, books):                      # X: (T, D)   books: list of (N, D)
    R, out = X.copy(), []
    for C in books:
        idx = ((R[:, None, :] - C[None]) ** 2).sum(-1).argmin(1)
        out.append(idx); R = R - C[idx]
    return np.stack(out, 1), R      # (T, Q) tokens, plus what is still unexplained

# 3. the library one-liner you would actually ship
# from moshi.quantization import SplitResidualVectorQuantizer
# codes = quantizer.encode(latent)      # (B, Q, T) int64

The quantizer's parameters, and the bitrate

Mimi's real configuration: Q = 8 quantizers, each with N_A = 2048 centroids, applied not to the 512-dimensional latent but to a 256-dimensional projection of it (and projected back to 512 before the decoder — a lower-dimensional space makes nearest-neighbour search better conditioned and codebooks easier to keep alive).

arithmetic — Mimi's 1.1 kbps
bits per token   = log2(2048) = 11
tokens per frame = Q = 8
frames per second = 12.5
bitrate = 11 x 8 x 12.5 = 1,100 bits/s = 1.1 kbps
discrete space per second = {1..2048}^(8 x 12.5) — 100 symbols/s from a 2048-way alphabet

For calibration: telephone-grade speech codecs live around 8–13 kbps; Opus at its lowest usable speech setting is ~6 kbps. Mimi is running an order of magnitude below that and carrying linguistic structure, which is the subject of the next chapter.

Two training decisions that look like mistakes

Quantize only half the time. During training, with probability 0.5 per sequence, Mimi skips quantization entirely and passes the unquantized embeddings to the decoder. Note the difference from prior work: DAC/RVQGAN's version passes embeddings quantized with all quantizers; Mimi passes them raw. This significantly improves objective quality (VisQOL), and the authors note the counter-intuitive detail that the gain grows as the bitrate drops. The intuition: the decoder gets to learn its job in a clean setting instead of only ever seeing quantization noise, and the encoder is pushed to produce a latent that is good in itself rather than good-after-rounding.

Alongside it, quantizer dropout: during training a random prefix of the codebooks is used and the rest are dropped, so the codec learns to be decodable from the first k levels for any k. That is what gives one trained model a whole bitrate ladder:

The ladder exists because RVQ levels are ordered, which we proved by hand two paragraphs ago. Nothing about quantizer dropout would work with the split topology if the semantic branch were not level 1 — another dependency to keep in mind for Chapter 3.

Throw away the reconstruction loss. The EnCodec recipe is multi-scale mel-spectrogram reconstruction plus a multi-scale STFT discriminator. Mimi's headline configuration keeps only the adversarial terms — feature-matching loss and discriminator loss. Objectively this looks like vandalism: VisQOL collapses from 2.82 to 1.84. Subjectively it is the single biggest quality win in the paper:

The optimization details, for completeness: AdamW with weight decay 5×10⁻² applied only to the transformer parameters (the convolutional stack keeps plain Adam-style updates), learning rate 8×10⁻⁴, momentum decay 0.5 and squared-gradient decay 0.9 (both unusually low), an exponential moving average of weights with decay 0.99, batch size 128, random 12-second windows, and 4 million steps. The selective weight decay is the tell: adding transformers to a conv codec required regularization the conv codec never needed.

Chapter 3: Mimi II — Split RVQ and the Semantic Graft

The codec from Chapter 2 works. Feed it speech, get eight tokens per 80 ms, decode them, hear speech. And if you hand those tokens to a language model and ask it to continue a conversation, you will get babble.

Chapter 1 told us why: acoustic tokens are optimized for reconstruction, so they spend their capacity on whatever is perceptually loud rather than whatever is linguistically structured. The paper measures exactly this. The metric is ABX, and it deserves a paragraph because it is the workhorse of the whole audio-LM field.

ABX: how to measure "is this token space phonetic?"

Take a triphone — three consecutive phones, say beg. Take a second recording of the same triphone, and a third recording of a minimally different triphone: bag. Now embed all three in the representation you are testing, and ask: is the distance between the two begs smaller than the distance from beg to bag? If your space is phonetic, yes, nearly always. The ABX error rate is how often it gets this backwards. Moshi reports within-speaker ABX (all three utterances from the same speaker, so speaker identity cannot be used as a shortcut) on LibriSpeech dev-clean.

Why care? Because ABX error rate has been shown to predict whether a downstream audio language model can produce coherent speech. It is a cheap proxy for "can a next-token predictor find structure here". And Mimi's undistilled first codebook scores 23.3% — comparable, the paper notes, to the acoustic tokens of previous work, which is another way of saying "not usable as a language".

Let us make ABX concrete with a tiny worked case, because "error rate on a discrimination task" is the kind of phrase that slides past unexamined. Suppose the representation embeds three utterances as 2-D vectors:

arithmetic — one ABX trial, by hand
A = "beg" spoken once      -> ( 0.80,  0.10)
X = "beg" spoken again     -> ( 0.72,  0.22)     same triphone, same speaker
B = "bag" (minimal pair)   -> ( 0.30,  0.55)     one vowel different

d(A, X) = (0.80-0.72)^2 + (0.10-0.22)^2 = 0.0064 + 0.0144 = 0.0208
d(A, B) = (0.80-0.30)^2 + (0.10-0.55)^2 = 0.2500 + 0.2025 = 0.4525

d(A,X) < d(A,B)  ->  trial CORRECT   (the two "beg"s really are closer)

ABX error rate = fraction of such trials that come out backwards,
averaged over many triphone pairs on LibriSpeech dev-clean.
  8.1%  = 1 in 12 minimal-pair judgements inverted   (Mimi, shipped)
 23.3%  = 1 in 4 inverted                            (Mimi, no distillation)
 42.2%  = barely better than a coin flip             (RVQGAN, purely acoustic)

Note what makes the within-speaker variant strict: if the three utterances came from different speakers, a representation could score well simply by encoding voice identity, since two clips from the same speaker would land near each other regardless of content. Holding the speaker fixed removes that shortcut and forces the distances to be about phonetics. Notice also that ABX asks nothing about reconstruction — a representation can ace it and be useless for audio, which is exactly the semantic-token situation from Chapter 1.

The graft: distil WavLM into codebook 1

The idea, borrowed from SpeechTokenizer and extended: we cannot run a self-supervised model at inference time, but we can force our causal encoder to imitate one during training. Knowledge distillation, with a twist — the target is not a distribution over classes but a continuous embedding, and the student is one specific quantizer level.

The teacher is WavLM-large, frozen. Trace the shapes carefully, because the resampling is where the subtlety lives:

Three details in that table are worth stopping on, because each is a real engineering decision rather than a formality.

Kernel 8 with stride 4. Stride 4 turns 50 Hz into 12.5 Hz, as required. But the kernel is 8, so each pooled target averages 8 teacher frames — the windows overlap, and each target looks 4 frames into the future. The paper is emphatic: "we observed that it was critical for performance to perform this average pooling in a non-causal way". A causal pooling would have thrown away the teacher's forward-looking context, which is precisely the thing the causal student cannot compute for itself and therefore the thing most worth teaching.

And it is still streaming-compatible. This is the elegant part, and the point most often missed on a first read: the non-causality lives entirely in the target. WavLM, the resampling, and the pooling all run only during training. At inference Mimi is exactly the causal conv-plus-transformer stack from Chapter 2, with a codebook whose centroids happen to have been shaped by a non-causal teacher. You get the benefit of bidirectional context without paying for it at inference — the same trick as distilling a large teacher into a small student, applied along the time axis instead of the parameter axis.

The projection is parallel, not in series. The distillation head is a separate linear map out of quantizer 1; it is not on the path to the decoder. So the reconstruction gradient and the distillation gradient meet at the codebook, not downstream of it. That is what makes the next section's conflict a genuine tug-of-war over one shared resource.

Two design questions you should be asking

Why cosine distance rather than mean squared error? Because we want the codebook to reproduce WavLM's directions, not its magnitudes. WavLM's embeddings live in a 1024-dimensional space with its own arbitrary scale, and the quantizer's outputs live in Mimi's 256-dimensional latent space with a completely different one. Cosine distance is invariant to the linear projection's overall gain, so the loss never fights the reconstruction path over how big vectors should be — it only argues about where they point. Phonetic identity is a direction; loudness is not.

Why distil into one level rather than all of them? Try the alternative in your head. If every codebook were pulled toward WavLM, the codec would become a quantized WavLM — excellent ABX, unlistenable audio, since WavLM discards exactly the speaker and channel information the decoder needs. The design intent is a division of labour: one codebook is a phonetician, seven are recording engineers. Concentrating the semantic objective in one place is what makes the division possible, and it is also what makes downstream weighting possible — Chapter 7's α = 100 only makes sense if "the semantic token" is a single, identifiable row.

And a third question the paper answers implicitly: why not just use WavLM's tokens directly and a separate codec for acoustics, as AudioLM did? Two encoders, one of them non-causal, on every frame. That is the design Chapter 1 ruled out. The graft exists to collapse two encoders into one causal pass.

What distillation costs

It works, spectacularly, on the metric it targets — and it damages the thing the codec was for:

ABX drops from 23.3% to 6.5% — a 3.6× reduction in phonetic confusions. And audio quality falls by 8 MUSHRA points. Why?

Go back to the hand-worked RVQ in Chapter 2 and re-read the second bullet: later levels quantize the residual of earlier levels. Codebook 2 never sees the signal; it only sees what codebook 1 left behind. Now force codebook 1 to be phonetic. It stops choosing centroids that capture the biggest chunk of acoustic energy and starts choosing centroids that discriminate beg from bag. The residual handed to codebooks 2–8 is now the difference between the real frame and a phonetic sketch of it — a strange, high-energy, structurally unfamiliar signal. Seven acoustic quantizers are asked to clean up after a quantizer that was optimizing something else entirely.

The paper puts it in one sentence: "As higher-order quantizers operate on the residual of the first one, the latter needs to trade audio quality for phonetic discriminability."

The fix: stop making them share a residual

Split RVQ. Instead of one 8-level residual chain, run two quantizers in parallel: a plain single-level VQ that receives the distillation loss (the semantic quantizer), and a 7-level RVQ that quantizes the same latent independently (the acoustic quantizers). Sum their outputs to reconstruct.

Read the topology change as a statement about information: in the serial version, the acoustic branch is only allowed to encode "everything except what the semantic quantizer captured". In the split version, the acoustic branch may encode whatever it likes — including re-encoding information the semantic branch also has. The cost is redundancy; the benefit is that neither objective is expressed as a constraint on the other's leftovers. Trading a little bitrate efficiency for a lot of objective independence is a pattern worth stealing.

The trade in numbers: MUSHRA recovers from 57.8 to 64.0 while ABX degrades only from 6.5% to 8.1%. That is the configuration Moshi ships — so the placeholder from the top of the chapter reads 8.1% ABX, down from 23.3%, at a cost of 1.9 MUSHRA points versus not distilling at all.

Where Mimi lands against the field

With the split RVQ and the adversarial-only recipe from Chapter 2, here is the comparison table that matters — note especially the frame-rate column, which is the one Chapter 1 taught us to read first:

Read the last row against the others. Mimi beats every baseline on human-rated quality while running at one quarter the frame rate of the 50 Hz codecs, at the lowest bitrate in the table, and it is the only causal entry. SemantiCodec and SpeechTokenizer have better ABX — 3.3% versus 8.1% — and that is a real advantage the paper does not hide; Mimi's answer is that the extra phonetic precision is not worth 4× the language model's compute, especially once Inner Monologue (Chapter 6) supplies linguistic structure from the text side anyway.

Two honest wrinkles the table does not show. First, the ABX advantage of SemantiCodec and SpeechTokenizer is partly a frame-rate effect: at 50 Hz each token covers 20 ms, which is close to a single phone, whereas Mimi's 80 ms token may straddle two. Finer time resolution makes phonetic discrimination easier and language modelling harder — the same trade as everywhere else in this paper, viewed from the metric's side. Second, Mimi's MUSHRA advantage is measured against baselines evaluated at their own native sample rates; the paper includes a 16 kHz-downsampled Mimi in the same listening study for fairness, and it still scores 77.7 ± 1.4, above every baseline.

python — the distillation loss, as you would actually write it
import torch, torch.nn.functional as F

# frozen teacher, training-time only — never runs at inference
with torch.no_grad():
    x16 = torchaudio.functional.resample(x24, 24000, 16000)
    e   = wavlm(x16).last_hidden_state              # (B, T50, 1024) at 50 Hz
    # kernel 8, stride 4 -> 12.5 Hz, windows OVERLAP and look ahead
    tgt = F.avg_pool1d(e.transpose(1, 2), kernel_size=8, stride=4).transpose(1, 2)

# student: a projection of quantizer 1's output, parallel to the decoder path
q1   = semantic_vq(z)                                # (B, S, 256)
pred = distill_proj(q1)                              # (B, S, 1024)

# cosine distance — direction only, so the codebook keeps its own scale
loss_distill = (1 - F.cosine_similarity(pred, tgt[:, :pred.shape[1]], dim=-1)).mean()

# the decoder never sees `pred`; it sees q1 + acoustic_rvq(z), summed
recon = decoder(q1 + acoustic_rvq(z))

Read the last line once more. q1 + acoustic_rvq(z) — a sum, with z passed to both branches. That single line is the split RVQ. In the serial version it would have been acoustic_rvq(z - q1), and the difference between those two expressions is 6.2 MUSHRA points.

Mimi, assembled — the complete data flow

Eight tokens per frame leave at step 5, and those eight integers are the entire interface between Chapters 2–3 and everything that follows. From here on, Mimi is a black box that converts sound to {1..2048}⁸ every 80 ms and back.

Inline check before you move on

(a) WavLM emits 50 Hz embeddings and Mimi needs 12.5 Hz. Average pooling with stride 4 achieves that. Why is the kernel 8 rather than 4?
Kernel 4 with stride 4 gives disjoint windows — each target sees only its own 80 ms. Kernel 8 overlaps into the next window, so each target carries 4 frames of future teacher context. That look-ahead is the part the causal student cannot compute for itself, and therefore the part most worth distilling.

(b) The distillation head is a linear projection to 1024 dimensions that the decoder never uses. What would change if it were placed in series, i.e. if the decoder read from it?
The reconstruction gradient would then flow through the distillation projection, coupling the two objectives twice over — once at the codebook and once at the projection. Keeping it parallel means the only shared resource is the codebook itself, which is precisely the tug-of-war the split RVQ is designed to referee.

(c) Mimi's ABX is 8.1% versus SpeechTokenizer's 3.3%. Why is that not a straightforward loss?
SpeechTokenizer runs at 50 Hz — four times the frames, four times the language model's compute per second of audio, and no causality. Mimi trades phonetic precision for a frame rate that makes real-time dialogue possible, and Inner Monologue supplies linguistic structure from the text side anyway.

Chapter 4: The RQ-Transformer — Time × Depth

Mimi hands us, every 80 ms, a little column of tokens: eight of them for one audio stream, and by the end of Chapter 6 we will be carrying seventeen. A language model predicts one thing at a time. What, exactly, is "one thing"?

Start from the standard object. An autoregressive model of a discrete sequence U ∈ {1,…,N}^S estimates the joint distribution by factorizing it into conditionals:

with U₀ = 0 a fixed start token. GPT does this. Helium does this. Nothing here is new. What is new is that we do not have one sequence — we have K of them, stacked. Write V_s,k for the value of sub-sequence k at timestep s, each drawn from its own vocabulary of size N_k, and write V_s = (V_s,1,…,V_s,K) for the whole column.

Three ways to model a column, and why two of them fail

Option A — flatten. Read the columns out in raster order: V_1,1, V_1,2, …, V_1,K, V_2,1, …. Now it is one sequence again and ordinary machinery applies. This is correct and it is unaffordable: the sequence is K · S long, so the big model runs K times per frame.

Option B — independent heads. Keep the sequence at length S, and at each step predict all K tokens in parallel from the same hidden state with K separate classification heads. Cheap, but it asserts something false: that the K tokens of a column are conditionally independent given the past. They are emphatically not — codebook 3 of a frame is a function of the residual left by codebooks 1 and 2 of that same frame.

Option C — factorize time and depth. Keep the expensive model on the time axis, and put a second, much smaller autoregressive model on the depth axis, run inside each timestep. This is the RQ-Transformer, borrowed from residual-quantized image generation and hierarchical byte modelling, and it is exactly the right shape for our problem.

Formally, two functions. The Temporal Transformer — Helium-sized, 32 layers, dimension 4096 — consumes all previous columns and produces one context vector:

The Depth Transformer — 6 layers, dimension 1024, 16 heads — consumes that context vector plus whichever tokens of the current column have already been decided, and produces logits for the next one:

with the first token of each column handled by a plain linear map, l_s,1 = Lin(z_s). Train all three pieces so that:

Look at the second line closely: nothing has been approximated. This is still an exact autoregressive factorization of the joint distribution over the whole grid — it is simply an ordering (time-major, then depth) plus a decision about which model computes which conditional. Option B approximates; Option C reorganizes. That distinction is the whole reason this works.

Two plumbing details the paper states in one sentence each, both of which you would have to guess otherwise:

• Temporal input: at step s the Temporal Transformer receives the sum of K learnt embedding tables applied to the previous column V_s−1. One vector in, carrying all seventeen tokens additively — not a concatenation, not K separate positions.
• Depth input: at depth k the Depth Transformer receives z_s plus a learnt embedding of V_s,k−1. So the temporal context is injected additively at every depth step, and the depth model's own "sequence" is at most K long — it never attends across time.

And one genuine contribution over prior RQ-Transformers: depthwise parametrization. The linear layers, projections and fully-connected weights inside the Depth Transformer are different for each index k. The reasoning is that sub-sequence 3 (a mid-level acoustic residual) and sub-sequence 1 (a semantic token) require genuinely different transformations, and forcing them to share weights makes them compete. Because the Depth Transformer is tiny, per-index weights cost nothing measurable in time; the ablation later in this chapter shows they buy real quality.

The cost ledger, worked out

Let us make the saving concrete with parameter counts. These are our estimates from the published hyper-parameters (32 layers / d = 4096 / MLP 11264 for the Temporal Transformer, 6 layers / d = 1024 / MLP 4096 for the Depth Transformer), not figures the paper prints — but they are the right order of magnitude and they settle the argument.

arithmetic — why the depth model is nearly free
Temporal Transformer, per layer:
  attention  4 x d^2         = 4 x 4096^2        = 67,108,864
  gated MLP  3 x d x d_ff    = 3 x 4096 x 11264  = 138,412,032
  total per layer                                 = 205,520,896
  x 32 layers                                     ~ 6.58 x 10^9   (the "7B")

Depth Transformer, per layer:
  attention  4 x 1024^2      = 4,194,304
  gated MLP  3 x 1024 x 4096 = 12,582,912
  total per layer            = 16,777,216
  x 6 layers                 ~ 1.01 x 10^8

Ratio                        6.58e9 / 1.01e8 = 65x

Per audio frame with K = 17:
  flattened:   17 passes through the 6.58e9 model  = 17.00 "big-model units"
  RQ-T:         1 pass through 6.58e9              =  1.00
              + 17 passes through 1.01e8           = 17/65 = 0.26
              total                                 =  1.26 big-model units

Speed-up on the dominant term:  17.00 / 1.26 = 13.5x
Real-time requirement:  12.5 frames/s x 1.26 = 15.8 big-model passes/s
                        (vs 212.5/s for the flattened model)

And the context saving is separate and multiplicative: the Temporal Transformer's attention runs over 3,750 positions for a five-minute conversation, not 63,750. Since attention is quadratic, that is a factor of 289 in attention work.

The acoustic delay — buying independence with time

One more idea belongs in this chapter, because it interacts with the RQ-Transformer in a way the ablations make vivid.

Rather than putting all of a frame's tokens in the same column, Moshi delays the acoustic codebooks by τ steps relative to the semantic one:

Why would shifting tokens sideways help? Because it moves a dependency from within a column (where only the small Depth Transformer can model it) to across columns (where the big Temporal Transformer can). The acoustic tokens of frame t now appear in the same column as the semantic token of frame t+τ, which the model has already seen — so the hard joint structure gets modelled by the powerful component. Prior work makes the same point from two directions: delays reduce the statistical dependence between sub-sequences at a given step, and the mutual information between sub-sequences at a step predicts how powerful a model you need to estimate them.

The cost is equally simple: every step of delay is 80 milliseconds you cannot get back, because the acoustic detail for the audio you are playing now was decided τ frames ago. The full latency formula, which we will re-derive properly in Chapter 8:

The ablation that proves the RQ-Transformer is necessary

Here is the result that makes this chapter's argument airtight, and it is subtler than "the new component helps".

With the generous staircase delay, the RQ-Transformer buys almost nothing — 42.2 to 40.3. Every token in a column comes from a different audio frame, so the conditional-independence assumption of Option B is nearly true, and cheap independent heads suffice. That is exactly the regime MusicGen operates in, and it explains why MusicGen never needed a depth model.

But the staircase costs 640 ms of latency — disqualifying for dialogue. Compress the delay to a flat 2 and the columns fill up with tokens from nearly the same frame, mutual information within a column explodes, and independent heads fall apart: perplexity 135.4, more than three times worse. The RQ-Transformer holds at 36.8, better than anything in the top row.

The paper's own summary is one line: "the RQ-Transformer becomes a critical component of generative models of RVQ tokens under strict latency constraints." And the delay ablations (measured with transcript quality rather than perplexity, since perplexities are not comparable across delay patterns) fill in the rest of the trade:

Note the shape: the first step of delay is worth a lot (4.36 → 4.12), the second almost nothing (4.12 → 4.09). Diminishing returns arrive immediately. This is why Moshi pretrains with τ = 2 — take the easier learning problem while nobody is waiting for an answer — and fine-tunes with τ = 1, landing at 160 ms of theoretical latency with essentially no quality left on the table. Transcript length, incidentally, is used as a quality proxy because weak audio models collapse into silence; a model that keeps talking is a model that has something to say.

Chapter 5: Showcase — Two Streams, No Turns

This is the chapter the paper is remembered for, and the idea is so simple that the first reaction is usually suspicion.

We have an architecture that models K sub-sequences per timestep. So far K = 8: Moshi's own audio codebooks. The question that produces full-duplex dialogue is: what if one of the sub-sequences were the user's voice?

Not "the user's transcript". Not "a flag saying the user is speaking". The user's actual audio, tokenized by the same Mimi codec, modelled as an ordinary part of the same joint distribution, at the same 12.5 Hz, in the same column.

The joint sequence, in full

Let A be Mimi's tokens for Moshi's audio and A′ its tokens for the user's audio. Let W be the aligned text stream we build in Chapter 6 (take it on faith for now). The complete object Moshi models, the paper's Equation 6, is:

Count the rows: one text stream, plus Q = 8 for Moshi, plus Q = 8 for the user.

Seventeen tokens per 80 ms column, one column every 80 ms, one Temporal Transformer pass per column and seventeen tiny Depth Transformer passes inside it. That is the entire runtime shape of Moshi. Everything else is training.

What actually happens at inference

Training and inference differ in exactly one respect, and it is worth being precise because this is where readers usually get confused.

At training time, all 17 rows are ground truth and the model is trained to predict all of them — including the user's. At inference time the model still produces a distribution over all 17, but:

So the model spends capacity predicting something it will never be allowed to use. Why?

Reason one: prediction is how it learns the dynamics. To predict what the user does next, the model must internalize turn-taking — that a question tends to be followed by the other person speaking, that a trailing intonation invites a response, that a backchannel is not a bid for the floor. Those predictions are what let it place its own speech correctly. Modelling the user is how Moshi learns when to talk.

Reason two: self-play. Because both sides are generated by the same model, you can simply not feed a microphone and let it hallucinate an entire two-person conversation. That is what makes the offline dialogue evaluation of Chapter 9 possible — 1,000 generated conversations with measurable pause, gap and overlap statistics, no humans required.

And one detail that reveals how thoroughly the turn abstraction is gone: when the user is speaking and Moshi is not, Moshi's audio tokens do not become some fixed silence symbol. They decode to what the paper calls "natural silence" — an actual near-silent waveform, with room tone, breath, whatever the model thinks belongs there. Silence is generated, not defaulted. Meanwhile Moshi's text stream fills with PAD tokens.

The showcase

Enough prose. Below is the two-stream timeline. Trigger the three conversational phenomena from Chapter 0 — the ones that broke the cascade — and watch what the representation does in each case. Scrub the playhead to inspect the 17-token column at any moment.

Work through what each scenario shows:

One frame of inference, in code

The entire runtime is a loop over 80-millisecond frames. Written out, with shapes, it is shorter than most people expect:

python — the Moshi inference loop, one frame
Q, K = 8, 17                       # 1 text + 8 Moshi + 8 user
prev = torch.zeros(K, dtype=torch.long)   # V_0 — the deterministic start column

while streaming:
    # --- 1. one pass through the 7B model, on the PREVIOUS column ---
    z = temporal(prev)                    # (d,) = (4096,)   KV-cached across frames

    # --- 2. K steps through the small model, inside this frame ---
    col = []
    for k in range(K):
        logits = depth(z, col, index=k)   # per-index weights: depthwise parametrization
        col.append(sample(logits, temp))  # text first, then semantic, then acoustic

    # --- 3. overwrite the user rows with what the microphone actually said ---
    user_frame = mimi.encode(mic.read(1920))   # 1920 samples = 80 ms at 24 kHz -> (8,)
    col[1 + Q:] = user_frame                     # rows 10..17 discarded and replaced

    # --- 4. play Moshi's audio for this frame ---
    speaker.write(mimi.decode(col[1:1 + Q]))     # -> 1920 samples of output

    prev = torch.tensor(col)              # becomes the input of the next frame
    # budget for the whole body: 80 ms.

Three things to notice in that loop. The microphone read and the speaker write are the same size, 1920 samples — the system consumes and produces at exactly the same rate, which is what "full duplex" means mechanically. The user rows are overwritten after sampling, not skipped, so the model always sees a complete column. And there is no branch anywhere on who is speaking.

How this compares to the only prior full-duplex system

Moshi is not the first attempt. dGSLM modelled user and system speech as separate token streams with a Siamese architecture, and it deserves the credit for the idea. The paper is direct about why it remained a proof of concept, and each limitation maps to something Moshi fixes:

The measured consequence, previewing Chapter 9: on generated dialogues scored by an external language model, dGSLM's non-cascaded model reaches perplexity 195.9 while Moshi reaches 41.9 — and Moshi's turn-taking statistics (pause, gap, overlap durations) sit close to the ground-truth human distribution rather than collapsing.

The engineering that makes two streams behave like two people

One honest caveat before we leave. A model trained on cleanly separated channels would fall apart on a real phone call, where Moshi's own voice leaks back into the user's microphone and the room is noisy. The instruction-tuning stage therefore attacks the user stream with a specific menu of corruptions — we will meet the full list in Chapter 7, but the most telling one is echo simulation: a scaled copy of Moshi's own audio (factor uniform in [0, 0.2]) is added to the user stream with a delay uniform in [100 ms, 500 ms].

Think about what that trains. Without it, Moshi would hear its own voice returning through the user stream and have no idea it was its own — a model that talks to itself, or worse, interrupts itself. The augmentation teaches a distinction humans make effortlessly and machines do not get for free: that sound is me. The user's stream is also randomly gained between −24 and +15 dB, mixed with noise 30% of the time, and given reverberation — and, pointedly, alternated with silent stretches of up to 30 seconds "so that the model can handle the audio condition going from noisy to silent, and vice versa."

Chapter 6: The Inner Monologue

Moshi as built so far is a pure audio model with a text-pretrained backbone. It works. It also, in the paper's own ablations, produces speech with a transcript negative log-likelihood of 3.65 and an average length of 602 characters — meaning it says things, but not for long and not always coherently.

Turn on one more component and those numbers become 2.77 and 1,920. Negative log-likelihood down by a quarter; length up by more than 3×. It is by a wide margin the largest single quality improvement in the paper. That component is the Inner Monologue.

The idea, and the two ways it is not done

The idea: make Moshi also model the text of its own speech, time-aligned to the audio frames, as the first sub-sequence of every column — so that within a timestep the Depth Transformer decides the text token before the semantic token, and the semantic token before the acoustic ones.

That is the AudioLM semantic→acoustic hierarchy with one more rung bolted on top. And its position in the ordering is the entire trick, so contrast it with the two alternatives the literature had tried:

One deliberate asymmetry: only Moshi's own text is modelled, never the user's. Two reasons given: transcribing the incoming stream in real time would be challenging, and depending on an external ASR would contradict the end-to-end speech-to-speech premise of the whole system. So the user's side stays purely acoustic — which also means paralinguistic information from the user is never squeezed through text, which was the Chapter 0 complaint.

Aligning text to 12.5 Hz — the hand-worked build

Text tokens and audio frames run at wildly different rates: roughly 3–4 text tokens per second versus 12.5 frames per second. To put text in a column, we need exactly one text token per frame. The construction uses two special symbols that never appear inside a word:

• PAD — "nothing is being said in this frame."
• EPAD — "end of padding": inserted immediately before a word begins. Strictly unnecessary, but the authors observed it helps by splitting one hard decision (should I start talking, and if so with which word?) into two easy ones (start talking now, then this word).

The recipe (the paper's Equation 5): initialize the whole stream to PAD; then for each word i with start frame t_i and tokens w_i,1…w_i,ni, write EPAD at t_i−1 and the word's tokens from t_i onward.

Let us build one by hand, completely. Take Moshi saying "Hello, I'm Moshi." with these Whisper word timestamps:

step 1 — from timestamps to frame indices
frame rate fr = 12.5 Hz   ->   one frame = 1/12.5 = 0.08 s

word 1  "Hello,"  starts at 0.32 s   ->  t1 = 0.32 / 0.08 = 4
word 2  "I'm"     starts at 1.04 s   ->  t2 = 1.04 / 0.08 = 13
word 3  "Moshi."  starts at 1.44 s   ->  t3 = 1.44 / 0.08 = 18

SentencePiece tokenization (_ marks a word boundary):
  "Hello,"  -> ["_Hello", ","]        n1 = 2
  "I'm"     -> ["_I", "'", "m"]       n2 = 3
  "Moshi."  -> ["_M", "oshi", "."]    n3 = 3

Now the stream itself, 24 frames long (1.92 seconds), built by applying the rule three times:

step 2 — the build, frame by frame
Initialise:  W[1..24] = PAD  everywhere

Word 1 (t1 = 4, n1 = 2):
  W[3]  <- EPAD          (t1 - 1)
  W[4]  <- "_Hello"
  W[5]  <- ","

Word 2 (t2 = 13, n2 = 3):
  W[12] <- EPAD
  W[13] <- "_I"
  W[14] <- "'"
  W[15] <- "m"

Word 3 (t3 = 18, n3 = 3):
  W[17] <- EPAD
  W[18] <- "_M"
  W[19] <- "oshi"
  W[20] <- "."

Final stream:
  frame:  1    2    3     4        5   6    7    8    9    10   11   12
  token:  PAD  PAD  EPAD  _Hello   ,   PAD  PAD  PAD  PAD  PAD  PAD  EPAD
  frame:  13   14   15   16   17    18   19    20   21   22   23   24
  token:  _I   '    m    PAD  EPAD  _M   oshi  .    PAD  PAD  PAD  PAD

Accounting:
  real text tokens  = 2 + 3 + 3 = 8      (33.3% of frames)
  EPAD              = 3                  (12.5%)
  PAD               = 24 - 8 - 3 = 13    (54.2%)
  padding of any kind = 16 / 24 = 66.7%   (paper: ~65% for English conversational speech)

The edge cases, which are where implementations break. Suppose word 3 had started at 1.28 s instead, so t₃ = 16. Then the EPAD would belong at frame 15 — which already holds "m" from word 2. The rule says: do not insert an EPAD if it would overwrite a text token from a previous word. Similarly, if a word starts at frame 1 there is no frame 0 to hold the EPAD, so EPAD goes at index 1 and the word's tokens shift right by one. Both rules exist for the same reason: the stream must have exactly one symbol per frame, and real text is more important than the hint.

python — the alignment three ways
# 1. step by step, exactly mirroring the hand-built stream above
PAD, EPAD = "<pad>", "<epad>"
fr, T = 12.5, 24
W = [PAD] * (T + 1)                     # 1-indexed for readability

words = [(0.32, ["_Hello", ","]),
         (1.04, ["_I", "'", "m"]),
         (1.44, ["_M", "oshi", "."])]

for start_s, toks in words:
    t = max(1, round(start_s * fr))          # 0.32 * 12.5 = 4
    if t > 1 and W[t - 1] in (PAD,):        # never overwrite real text
        W[t - 1] = EPAD
    for j, tok in enumerate(toks):
        if t + j <= T:
            W[t + j] = tok

# 2. the same as arrays, for a whole batch of utterances
import numpy as np
starts = np.array([0.32, 1.04, 1.44])
idx    = np.round(starts * fr).astype(int)      # array([ 4, 13, 18])
stream = np.full(T + 1, PAD_ID, dtype=np.int64)
# scatter EPADs first, then words, so words always win the collision
stream[np.clip(idx - 1, 1, T)] = EPAD_ID
for t, toks in zip(idx, tok_ids):
    stream[t:t + len(toks)] = toks

# 3. what you actually call in practice
# segments = whisper_timestamped.transcribe(model, wav)["segments"]
# stream   = align_to_frames(segments, frame_rate=12.5)   # one token per 80 ms
padding_fraction = (stream == PAD_ID).mean()      # ~0.65 on English conversation

Two consequences of that 65% figure, which you should predict before reading on. First, PAD dominates the text stream, so an unweighted cross-entropy would be mostly a PAD-prediction loss — which is why training halves the weight of padding tokens in the loss (Chapter 7). Second, PAD is a legitimate output: sampling PAD is Moshi choosing to stay silent this frame. Silence is generated, not defaulted — the same principle we met in Chapter 5.

One hyper-parameter, three systems

Now the part that turns a design detail into a result. Nothing forces the text stream to be aligned to exactly the same frame as the audio. Introduce a delay between W and A, and the direction of that delay decides which modality makes the decision:

The paper's phrasing is worth quoting because the claim is stronger than it first appears: "a single delay hyper-parameter allows for switching from an ASR to a TTS model with no changes in the loss, architecture, or training data." Not a shared encoder. Not multi-task fine-tuning. The same objective on the same data, with one number changed.

The zero-shot TTS trick deserves its own note, because "teacher-force the text" hides a problem: the text you want to speak is not padded. Moshi's solution is to let the model sample PAD and EPAD freely, and the moment it tries to sample anything else, inject the next real word instead. The model chooses the timing; you choose the content. Speaking rate can then be steered by keeping a running average of the padding fraction and adding a small bonus to PAD logits whenever it falls below target — a beautifully simple controller for something that is usually a separate duration model.

python — zero-shot padded TTS, including the rate controller
target_pad_ratio = 0.65          # English conversational speech
window, bonus    = 50, 1.5          # frames of running average, logit bonus
words            = iter(text_to_speak)
recent           = []

for frame in stream:
    logits = model.text_logits(state)
    ratio  = sum(recent[-window:]) / max(1, len(recent[-window:]))
    if ratio < target_pad_ratio:                 # speaking too fast — encourage silence
        logits[PAD] += bonus
    tok = sample(logits)
    if tok not in (PAD, EPAD):                    # the model wants to say something
        tok = next(words)                          # we choose WHAT; it chose WHEN
    recent.append(1 if tok in (PAD, EPAD) else 0)
    state = model.step(text=tok)                 # audio rows sampled normally

Read the line tok = next(words): this is the whole zero-shot padding trick. The model is never shown a padded transcript; it improvises the timing and we substitute the content at the moment it commits to speaking. And the rate controller is three lines — a running average and a logit bonus — replacing what is usually a separately trained duration predictor.

During pretraining the delay is randomized between −0.6 and +0.6 seconds, so the model never assumes one regime; post-training fixes it to 0 for dialogue. This is why one checkpoint can be repurposed at inference.

Chapter 7: The Curriculum — and the Loss That Steers It

We now have every architectural piece. None of it converses. Getting from "a model that can represent dialogue" to "a model that holds one" takes five training stages, and the ordering is not arbitrary — each stage teaches a capability that the next one presupposes.

Stage 1 — Helium, and why the data pipeline is a first-class contribution

Helium is a 7B decoder-only transformer with the standard modern recipe: RMS normalization at the input of attention blocks, feed-forward blocks and the output linear layer; RoPE positional embeddings; a 4,096-token context; FlashAttention; and gated linear units with SiLU gating in the MLP. Dimension 4096, MLP dimension 11264, 32 heads, 32 layers. The tokenizer is a 32,000-piece SentencePiece unigram model, English-focused, with all numbers split into single digits and byte-backoff so nothing is unrepresentable.

Training: 500k steps, batch of 4.2M tokens, AdamW, constant learning rate 3×10⁻⁴ followed by cosine decay, on 2.1T tokens.

The interesting part is the data. 12.5% comes from curated sources — Wikipedia (five different dumps from 2017, 2018, 2019, 2021 and 2022, rather than five passes over one dump, which is a quiet anti-memorization choice), Wikibooks, Wikisource, Wikinews, StackExchange, and the peS2o scientific corpus. The other 87.5% is CommonCrawl put through three filters:

It works: Helium reaches MMLU 54.3, ARC-easy 79.6, ARC-challenge 55.9, HellaSwag 76.3, PIQA 79.4 — competitive with or better than MPT, Falcon, Llama 2 and OLMo at similar compute, and close to Mistral and Gemma on several tasks despite up to 3× less training compute.

Stage 2 — audio pretraining on 7 million hours

Now initialize the Temporal Transformer from Helium and the Depth Transformer randomly, and train on 7M hours of readily available audio, mostly English speech, transcribed with Whisper large-v3, resampled to 24 kHz mono. Single stream — all speakers mixed into one audio channel, all words in one text stream. One million steps, batches covering 16 hours of audio, each item a 5-minute sequence.

Five deliberate perturbations in this stage, each fixing a specific failure:

• Half the batches are pure text, from Helium's own dataset, with a separate optimizer state. This is the anti-forgetting mechanism, and the separate optimizer state matters: shared Adam moments would let the much larger audio gradients set the scale for the text updates.
• Text tokens masked 30% of the time, so the model cannot become dependent on a text stream that will sometimes be absent or wrong.
• Text-audio delay randomized in [−0.6, +0.6] s — this is what makes the ASR/TTS repurposing of Chapter 6 possible from one checkpoint.
• Text embedding and text output layer get 0.75× the learning rate when operating on the text stream of an audio batch. A gentle brake on drifting away from Helium's text competence.
• Padding tokens are down-weighted 50% in the cross-entropy. Recall from Chapter 6 that PAD is ~65% of the text stream; without this the text loss would mostly measure the model's ability to say nothing.

Learning rates: 3×10⁻⁵ for the Temporal Transformer (small — it is already trained) and 2×10⁻⁴ for the Depth Transformer (large — it starts from noise). A different learning rate per component, because they are at different points in their lives.

How much data is 7 million hours, really?

Numbers at this scale stop meaning anything, so convert them into the units the model actually consumes.

arithmetic — the audio corpus in model-native units
Step 1. Corpus duration            7,000,000 hours
Step 2. In seconds                 7e6 x 3600 = 2.52 x 10^10 s
Step 3. Frames at 12.5 Hz          2.52e10 x 12.5 = 3.15 x 10^11 frames
Step 4. Audio tokens (Q = 8)       3.15e11 x 8 = 2.52 x 10^12 tokens
Step 5. For comparison, Helium’s text corpus         2.1 x 10^12 tokens
        -> the audio corpus is comparable in TOKENS, though it is
           2.9 million years of sound.

One training batch:
Step 6. Batch covers               16 hours of audio
Step 7. Each item is               5 minutes = 300 s
Step 8. Items per batch            16 x 60 / 5 = 192
Step 9. Frames per item            300 x 12.5 = 3,750
Step 10. Frames per batch          192 x 3,750 = 720,000
Step 11. Tokens per batch (K = 9 in single-stream: 1 text + 8 audio)
                                   720,000 x 9 = 6.48 x 10^6 tokens

Whole run:
Step 12. Steps                     1,000,000
Step 13. Audio seen                1e6 x 16 h = 1.6 x 10^7 hours
        -> about 2.3 passes over the 7M-hour corpus
           (and half the batches are text, so ~1.1 effective epochs of audio
            per million steps of wall clock)

Two things fall out of that ledger. First, the audio pretraining run is roughly the same token scale as the text pretraining run — this is not a fine-tuning stage bolted onto a text model, it is a second pretraining of comparable magnitude. Second, at only ~2 epochs there is limited pressure to memorize, which is relevant to the regurgitation analysis in Chapter 10: what memorization does occur is driven by duplicated segments, not by many passes over the corpus.

Stage 3 — simulated multi-stream, from diarization

The model can now speak, but only into one channel. To learn two streams we need two-channel data, and 7M hours of it does not exist. So the paper manufactures it: run PyAnnote speaker diarization over the unsupervised audio, pick one speaker at random as "main", build a binary mask over the waveform (1 where the main speaker is active, 0 elsewhere), and split the audio into two waveforms — the masked speaker, and the residual (which may contain several other speakers). Encode each separately as the two streams. The text stream carries only the main speaker's words, and the text-audio delay is fixed to 0.

100k steps, batch of 8 hours, learning rates 3×10⁻⁶ / 5×10⁻⁵, still 10% pure-text batches.

The paper is refreshingly clear about what this stage cannot teach: "it contains no overlap, and the stream of an inactive speaker is perfectly silent." A mask-based split is a partition — by construction the two streams are never active at the same time, and the silent one is digital zero rather than room tone. Both of those are exactly the properties real conversation violates.

Stage 4 — Fisher, where duplex actually happens

The Fisher corpus is 2,000 hours of telephone conversations between randomly paired strangers given a topic, and its decisive property is that each side is recorded on a separate channel. That is genuine ground-truth stream separation, including all the overlap, interruption and backchannel that Stage 3 could not synthesize.

The engineering: Fisher is 8 kHz, so it is upsampled to 24 kHz with AudioSR — a super-resolution model, not naive interpolation, because Mimi has never seen telephone-band audio and would otherwise learn that duplex conversations always sound muffled. Timestamps come from whisper-timestamped with the medium model, chosen for reliability across the long silences each channel contains.

Only 10k steps, batch of 40 minutes, learning rates 2×10⁻⁶ / 4×10⁻⁶, and no text batches at all. A thousandth of the pretraining budget — the full-duplex capability is a thin veneer over an enormous amount of single-stream competence.

Stage 5 — instruction tuning, and where Moshi's voice comes from

Fisher teaches conversational dynamics but not assistant behaviour, and the paper found that off-the-shelf text instruct datasets are actively wrong for speech: URLs cannot be spoken, and bullet points and long enumerations are not an oral register. So the instruct data is manufactured end to end:

The transcript generation also covers deliberate awkwardness: questions with misspellings so Moshi learns to ask for clarification, questions containing false premises ("Is the Eiffel Tower in Beijing?") so it learns to say no, basic arithmetic and trivia because "Moshi was initially not performing well on simple factual tasks like adding numbers", roleplay prompts for emotional styles, and safety conversations where the user asks unethical questions and Moshi refuses.

30k steps, batch of 2.7 hours, learning rate 2×10⁻⁶ for both transformers, plus the user-stream augmentations from Chapter 5 (random gain −24 to +15 dB half the time; DNS-challenge noise 30% of the time at −30 to +6 dB; echo of Moshi's own stream scaled by [0, 0.2] with 100–500 ms delay; reverb; echo and reverb together with probability 0.3; silent stretches up to 30 s with probability 0.5).

The whole curriculum on one card

For reference — and because seeing the learning rates fall by two orders of magnitude across the stages tells its own story about how much is being learned versus refined:

Three patterns to read off it. The Depth Transformer always gets the higher learning rate until the very end, because it starts from random initialization while the Temporal Transformer starts from a trained language model. The text-batch fraction decays 100% → 50% → 10% → 0%, a controlled release of the text crutch. And the batch size shrinks monotonically after pretraining — 16 h to 40 minutes — because the later datasets are small and precious, and the goal shifts from learning a distribution to nudging a behaviour.

The loss, worked out on real numbers

All five stages optimize one objective. Given ground-truth tokens V_s,k and predicted logits l_s,k:

Two design decisions hide in that formula, and both are consequential.

First: the text token stands alone. It is one term; all sixteen audio tokens together are the other term. The paper says it plainly — "we give the same importance to the text token, and the combined audio tokens." Text is 1/17th of the predictions and half of the loss.

Second: α_k = 100 for semantic tokens, 1 for acoustic tokens. Let us see what that actually does, with numbers. Take a multi-stream column, K = 17, and suppose the per-token cross-entropies at some step are:

arithmetic — one step of the Moshi loss, every term
k=1   text                 CE = 0.90     alpha = (its own term)
k=2   Moshi semantic       CE = 2.10     alpha = 100
k=3..9  Moshi acoustic x7  CE = 5.50     alpha = 1 each
k=10  user semantic        CE = 2.60     alpha = 100
k=11..17 user acoustic x7  CE = 5.80     alpha = 1 each

Normaliser:  sum of alpha for k >= 2
  = 100 + (7 x 1) + 100 + (7 x 1) = 100 + 7 + 100 + 7 = 214

Weighted audio sum:
  100 x 2.10 = 210.0
    7 x 5.50 =  38.5
  100 x 2.60 = 260.0
    7 x 5.80 =  40.6
  total      = 549.1

Audio term  = 549.1 / 214 = 2.5659
Total loss  = 0.90 + 2.5659 = 3.4659

Effective share of the audio term:
  each semantic token   100 / 214 = 46.7%
  each acoustic token     1 / 214 =  0.47%
  two semantic tokens together   = 93.5% of the audio gradient

Counterfactual with all alpha = 1 (normaliser 16):
  audio term = (2.10 + 38.5 + 2.60 + 40.6) / 16 = 83.8 / 16 = 5.2375
  semantic share = (2.10 + 2.60) / 83.8 = 5.6%

So the weighting moves the semantic tokens from 5.6% of the audio gradient to 93.5% of it — a 17× reallocation. The justification is a diagnosis the authors report from early experiments: "the individual losses per RVQ level were conflicting with one another, despite each level being more important in the final intelligibility and audio quality than the next one." Codebook 8 is worth far less than codebook 1, but an unweighted loss treats them identically, so the model spends most of its capacity chasing residual detail nobody can hear.

The measured effect, from the delay-pattern ablation table: transcript NLL improves 4.09 → 3.75 from the semantic weight alone, and 3.75 → 3.65 from adding depthwise parametrization on top — before Inner Monologue takes it to 2.77.

What each stage is actually buying

The fourth column is the one worth studying. Each stage has a blind spot that only the next stage's data can fill, which is why the ordering is forced rather than conventional — and why the last column of the last row is a limitation that ships, because there is no sixth stage.

Inline check before you move on

(a) Why does Stage 2 use two separate optimizer states for text batches and audio batches, rather than one?
Adam normalizes by a running estimate of gradient magnitude. Audio batches produce far more tokens and different gradient scales; sharing the moments would let audio statistics set the effective step size for text updates, quietly undoing the anti-forgetting mechanism the text batches exist to provide.

(b) Stage 3 builds two streams by masking one diarized speaker out of a mixed recording. Name the two properties of the result that Stage 4 has to repair.
A mask is a partition, so (i) the two streams are never simultaneously active — no overlap — and (ii) the inactive stream is digital zero rather than room tone. Fisher's separately recorded channels have both.

(c) Instruct fine-tuning is 30k steps out of more than 1.6M total. Why does such a short stage determine so much of the user-visible behaviour?
Because everything else was already learned; the last stage only has to select among behaviours the model can already produce. That is also why it is fragile — Chapter 10 notes that fine-tuned suppression of regurgitation "could potentially be over-ridden", precisely because it is a selection, not a removal.

One structural remark before the loss. Notice that every stage after the first is shorter and gentler than the one before it, and that each introduces exactly one new thing: audio, then a second stream, then real duplex dynamics, then persona. When a stage introduces two new things at once, failures become unattributable — you cannot tell whether Fisher taught overlap or whether it taught telephone-band acoustics. Upsampling Fisher to 24 kHz with AudioSR is precisely the move that keeps that stage single-variable.

Chapter 8: The Latency Ledger — Deriving 160 ms

Chapter 0 opened with a number we were told to distrust until it was derived. Here is the derivation. It takes four lines, and every term traces to a design decision you have already met.

First, a definition, because "latency" is used sloppily in this field. The quantity we want is: the delay between a piece of acoustic evidence arriving at the microphone and the earliest moment a response conditioned on it can begin leaving the speaker. Not time-to-full-answer. Not average response time. The minimum possible reaction delay, imposed by the representation, with an infinitely fast computer.

Term one: you must wait for a frame

Mimi cannot produce a latent from half a frame. Its initial frame size and its overall stride are both 80 ms, so the first latent appears only after 80 ms of audio has arrived. There is no partial credit: a sample that lands 1 ms after a frame boundary waits 79 ms for its column.

Notice this term is set by the frame rate we chose in Chapter 2 for compute reasons. Lowering the frame rate further — say to 6.25 Hz to halve the language model's work again — would raise this term to 160 ms on its own. The frame rate sits at the intersection of two opposing pressures, and 12.5 Hz is where they balance.

Term two: the acoustic delay

From Chapter 4: acoustic codebooks are shifted τ steps later than the semantic one, so the acoustic detail of frame t is emitted in column t+τ. You cannot play audio you have not finished describing, so every step of delay is one more frame of waiting:

Moshi is pretrained with τ = 2 and fine-tuned with τ = 1, so the shipped model pays one step:

arithmetic — the theoretical latency ledger
frame acquisition   1 / 12.5 Hz          =  80 ms
acoustic delay      tau = 1  ->  1 x 80  =  80 ms
                                            ------
theoretical latency                       = 160 ms

general form:   latency = (1 + tau) / frame_rate

the other configurations in the paper:
  tau = 0            (1 + 0) x 80 =  80 ms   — minimum possible with Mimi
  tau = 1  SHIPPED   (1 + 1) x 80 = 160 ms
  tau = 2  pretrain  (1 + 2) x 80 = 240 ms
  staircase [0..7]   (1 + 7) x 80 = 640 ms   — MusicGen-style, disqualifying

measured in practice                      = 200 ms
  => real-world overhead = 200 - 160       =  40 ms
     (forward-pass compute, audio I/O buffering, transport)

human turn-taking gap, mean over 10 languages = 230 ms
  => Moshi's practical latency is 30 ms UNDER the human average

Play with the frame-rate slider and something instructive happens: raising it lowers latency, monotonically. So why not run Mimi at 50 Hz and get a 40 ms ledger? Because Chapter 1's other budget bites: at 50 Hz the Temporal Transformer must complete a forward pass every 20 ms — fifty 7B-model passes per second — and the 40 ms of real-world overhead in the ledger would balloon past the theoretical saving. The two constraints pull in opposite directions and 12.5 Hz is the compromise; the paper says as much when it calls the low frame rate "crucial to achieve the low latency of Moshi, since generating one temporal frame of audio tokens with Moshi requires a full forward pass through the Temporal Transformer."

The compute budget that the ledger implies

A latency figure is only real if the model can keep up. The requirement is exact:

arithmetic — the real-time budget
columns per second            = 12.5
wall-clock budget per column  = 1 / 12.5 = 80 ms

work per column:
  1 x Temporal Transformer forward pass  (7B params, KV-cached)
 17 x Depth Transformer forward passes   (~0.1B params, K-long sequence)

from Chapter 4's ratio: 17 depth passes ~ 0.26 temporal passes
  => ~1.26 "big-model units" must complete in 80 ms
  => the 7B model has roughly 63 ms per token-column, minus decode overheads

That is a demanding but achievable target for a 7B model with KV caching on a modern accelerator — and it explains the paper's interest in quantization (Chapter 9), where the whole point is fitting this budget on smaller hardware.

Where each term was purchased

Every entry in the ledger traces to a decision made for an entirely different reason, several chapters ago. Tracing them backwards is the best summary of the architecture we have:

The two ledgers, side by side

Put Chapter 0's cascade next to Chapter 8's Moshi and sort every term into the three buckets from the callout below. This table is the paper's central claim in one place:

The row that matters most is the first one, and it is the only row where Moshi's entry is not "faster" but "absent". Every other saving is an ordinary engineering win of the kind a determined team could approach with enough optimization; the endpointing row can only be removed by changing what the model represents.

What the ledger looks like for other systems

To calibrate, apply the same three-bucket accounting to designs you will meet in the wild. The point is not to rank them — each is right for some deployment — but to see that the buckets predict which ones can ever be fast:

Row two is the interesting one, because it is what most production voice agents actually are in 2026: everything streamed, everything optimized, and still gated by a turn hypothesis. That row is the argument for why the architectural change matters even after the engineering is excellent.

The two systems that fall out for free

Chapter 6 showed that the text-audio delay is a dial. Set it to about ±2 seconds and Moshi becomes a streaming ASR or a streaming TTS, with a fixed 2-second latency by construction. Both are evaluated on LibriSpeech test-clean, which the models never saw in training:

Notice the shape of both comparisons: Moshi loses to the specialist that is allowed more information, and beats the specialist held to the same constraint. Against NaturalSpeech 3, which sees the entire sequence, Moshi's 4.7% versus 1.81% is not a fair fight and the paper does not pretend otherwise; against Vall-E, also non-streaming, Moshi wins outright at 4.7% versus 5.9%. On the ASR side the streaming FastConformer with comparable lookahead is genuinely better (3.6% versus 5.7%) — and it is a purpose-built ASR model, whereas Moshi got there by changing one hyper-parameter on a dialogue checkpoint.

The ASR result also comes with word-level alignments accurate to 80 ms — the Temporal Transformer's frame period, which is the finest granularity the representation can express. Alignment for free is a genuinely useful byproduct; it is what makes the training-data pipeline of Chapter 7 self-hosting, since the multi-stream TTS that generated the instruct data was itself derived this way.

The paper is careful not to oversell: "This limited experimentation is not intended to compete with state-of-the-art systems (in particular for ASR), but is rather designed to illustrate how Inner Monologue is flexible enough to cast several tasks into the same framework." It also notes that LibriSpeech — read speech, 4–10 second clips — is a poor showcase for a TTS whose distinguishing feature is generating expressive two-speaker dialogue across five minutes.

Why the context is five minutes, and what happens after

Latency and context length are usually discussed separately. In a streaming model they are the same conversation, because the model runs forever and its cache does not.

Moshi is trained on 5-minute sequences: 3,750 frames. That number was not chosen for elegance — it is what fits, given the KV-cache arithmetic from Chapter 1 (roughly 1.9 GB at 3,750 positions for the Temporal Transformer) alongside 7B parameters of weights. Push the context to 20 minutes and the cache alone approaches 8 GB, on top of a model that must also complete a forward pass every 80 ms.

arithmetic — cache growth during a live conversation
cache per frame (Temporal, 32 layers, d=4096, bf16)  = 512 KB

after  1 minute    750 frames  x 512 KB  = 0.38 GB
after  5 minutes  3,750 frames x 512 KB  = 1.92 GB   <- training horizon
after 10 minutes  7,500 frames x 512 KB  = 3.84 GB
after 30 minutes 22,500 frames x 512 KB  = 11.5 GB   <- weights + cache no longer fit

growth rate = 12.5 frames/s x 512 KB = 6.4 MB per second of conversation
           = 384 MB per minute, whether anyone is talking or not.

That last line deserves emphasis, because it is a property of full duplex specifically: silence costs exactly as much as speech. A turn-based system consumes nothing while waiting; Moshi emits a column every 80 ms regardless, so a two-minute pause is 750 MB of cache. The elegance of "always generating" has a bill, and it is paid in memory that grows linearly with wall-clock time.

The paper reports no solution beyond the 5-minute horizon — it evaluates "several minutes of context (5 min in our experiments)" and stops there. In production the options are the familiar ones (sliding window with eviction, attention sinks, periodic summarization into the text stream), each with a cost the paper does not measure. It is an honest limitation, and it is the natural next constraint after latency: Moshi solved how fast, not how long.

A latency the ledger cannot see

One more honest note before the chapter closes, because "160 ms" is a claim about reaction and users experience something slightly different.

The ledger measures how quickly Moshi can begin responding to acoustic evidence. It does not measure how quickly Moshi decides to. Those are different quantities: the model may correctly wait several hundred milliseconds because it has learned that a speaker who trails off mid-clause is not finished. That waiting is a modelling decision, not an architectural cost, and it is exactly what the turn-taking statistics of Chapter 9 measure — Moshi's gap distribution at temperature 1.0 is 4.5 s per conversation against a human 4.2 s, which is a behavioural match, not a latency figure.

Keep the two apart when reading any voice-agent claim. "Time to first audio token given that the model has decided to speak" is an architecture number. "Time from when a human would have replied to when the system does" is a behaviour number. A system can be excellent on the first and unbearable on the second, and most demos quote only the first.

the three numbers, disentangled
architectural floor   160 ms   (1 + tau) / frame_rate — cannot be beaten by any policy
measured latency      200 ms   floor + compute + I/O — what an engineer can optimize
observed gap          varies   when the model CHOOSES to speak — a learned behaviour,
                               tuned by sampling temperature, measured against humans

Inline check before you move on

(a) A team reports 90 ms latency for their speech model at 25 Hz frame rate with τ = 1. Check them.
(1 + 1) / 25 = 80 ms theoretical, so 90 ms measured implies only 10 ms of compute and I/O — implausible unless the model is tiny. Ask whether the 90 ms includes audio buffering, and whether the frame is really the first unit they can act on.

(b) Why does the Depth Transformer's 17 passes not appear as a term in the theoretical ledger?
Because they happen inside the 80 ms frame budget, not in addition to it. They are a constraint on whether the model can keep up (the real-time factor), not on how long you must wait before it may act. Compute shows up in the measured 200 ms, not the theoretical 160 ms.

(c) If you halved Mimi's frame rate to 6.25 Hz to save compute, what happens to the ledger and what would you have to change to compensate?
Acquisition becomes 160 ms, and τ = 1 adds another 160 ms — 320 ms theoretical, worse than a human. You would have to drop to τ = 0, which the ablations show costs real quality (NLL 4.36 vs 4.12), and you would still be at 160 ms plus overhead. This is why the frame rate is not a free compute dial.

A last framing for the ledger, since it is the number this paper is quoted for. 160 ms is not impressive because it is small; plenty of DSP runs in microseconds. It is impressive because it is a 7B language model's reaction time, achieved without giving up knowledge, voice, or the ability to hear who is talking. Every term in the ledger was bought with an architectural decision from an earlier chapter, and every one of those decisions had a measured price. That is what a systems paper looks like when it is done properly: no free lunches claimed, and a receipt for each one taken.

Chapter 9: Evaluation — Four Axes and an Honest Anomaly

How do you test a model that listens and speaks at the same time? There is no single number. The paper measures along four axes, and the interesting reading is not "Moshi wins" — it mostly does — but where the axes disagree with each other.

Here is the map of the four axes, and — more useful — what each one would miss on its own:

Keep the "blind to" column visible while reading. Two of the paper's most interesting findings — the sWUGGY regression and the temperature trade — are only visible because two axes disagreed.

Axis 1 — textless NLP: does the token stream contain a language?

These benchmarks test an audio language model without ever transcribing it, by comparing the likelihood it assigns to a good and a bad audio example:

Sequences are scored by negative log-likelihood normalized by length. One methodological wrinkle worth noting: because Moshi emits several tokens per timestep, the paper sums a timestep's tokens using the training weights — 100 for semantic, 1 for acoustic — and excludes the Inner Monologue text tokens entirely, since the benchmarks are designed for untranscribed audio.

Moshi leads its category almost everywhere. From a cold start (no text pretraining) it reaches sWUGGY 74.8 / sBLIMP 59.9 / sTopic-StoryCloze 80.9 / sStoryCloze 56.9, against AudioLM's 71.5 / 64.7 and GSLM's 64.8 / 54.2. Warm-started from Helium it is at or above TWIST-13B and Spirit-LM on most metrics, and its MMLU of 49.7 is 12.8 points above Spirit-LM's 36.9 — the clearest evidence that the text backbone survived the audio training.

The full picture, across the three evaluation settings the literature uses (all numbers from the paper's Table 7; ∅ marks an unsupported modality):

Read the columns against each other and the anomaly sharpens: from single-stream pretraining to multi-stream instruct, sWUGGY falls 9.6 points while Spoken StoryCloze rises from 60.8 to 62.7. Lexical judgement gets worse, commonsense reasoning gets better. Those are supposed to move together, and they do not — which is the empirical basis for the authors' claim that these benchmarks do not guide dialogue-model development well. One more structural note worth flagging: Moshi is the only model in the table that puts semantic and acoustic tokens in a single generative model. AudioLM uses three separate stages; VoxtLM, TWIST and Spirit-LM model semantic tokens only and hand off to an external vocoder, which is why none of them can produce an arbitrary voice.

Axis 2 — spoken question answering: does it know things?

Three benchmarks, all spoken: Spoken Web Questions, LlaMA-Questions, and a TTS-synthesized Audio TriviaQA. The question audio is inserted into the user stream — simulating a real interaction — and a final EPAD token is forced into the text stream to trigger an immediate answer. (That is the control hook from Chapter 5, used as an evaluation harness.)

Three readings, in order of importance.

Inner Monologue nearly triples every column (9.2→26.6, 21.0→62.3, 7.3→22.8) for the 6% inference cost computed in Chapter 6. That is the single best cost-benefit ratio in the paper.

Moshi beats Spectron and SpeechGPT while being the only streaming entry. Those two use Chain-of-Modality: they must produce a complete text answer before speaking. Moshi produces text and speech in the same 80 ms frame, and still wins on accuracy.

The Helium gap is real and mostly honest. 26.6 vs 32.3 on Web Questions is modest — the price of spending parameters on audio. But 22.8 vs 56.4 on TriviaQA is a chasm, and the authors went and looked at the errors rather than shrugging. The failures cluster on multi-sentence questions and unusual syntax — their example is "The Terror of the Monster was an early title for a best-selling novel which inspired one of the highest-grossing movies of the mid-70's. Under what name did it eventually terrify the reading and film going public?" — because instruct tuning taught Moshi an oral register in which nobody speaks that way. It is a data-distribution failure, not a knowledge failure, and their proposed fix is to cover more syntactic scenarios in fine-tuning.

A methodological note on the harness, since it is reusable. Inserting the question into the user stream rather than as a prompt prefix means the model is evaluated in its deployment configuration — two streams, real acoustics, the same code path a user would hit. Many multimodal evaluations quietly test a different configuration from the one that ships (text prompts to a model that will receive audio, or single-turn to a model that will run multi-turn), and the gap between the two is invisible in the results table. If you build one of these systems, make the evaluation harness call the same entry point as the product.

Each row is a controlled comparison, and the last row is the one to keep: this architecture is not free. It buys duplex, prosody and 200 ms at a measurable price in factual recall, and the paper reports that price in the same table as its wins.

Axis 3 — turn-taking statistics: does it converse like a person?

This axis exists only because of the dual-stream design. Since Moshi models both sides, it can generate complete two-person conversations unprompted, and those can be measured against real ones. Four quantities, all defined on inter-pausal units (IPUs — continuous stretches of speech bounded by at least 0.2 s of silence on each side):

1,000 ten-second Fisher prompts, 32 continuations each, at three temperatures, scored for linguistic quality with DialoGPT:

Look at the cascaded topline's overlap column: exactly 0.0 s. Not "small" — structurally impossible, because a cascade produces one speaker at a time by construction. Its pause column is 0.0 s for the same reason. Two of the four turn-taking statistics are not measurements of a cascaded system; they are consequences of its architecture. That single row justifies the entire dual-stream premise better than any prose argument.

And now the trade that the temperature sweep exposes. At 0.8, Moshi's language is better than the humans' by DialoGPT perplexity (41.9 vs 59.6 — the paper attributes this to both models being trained on data closer to DialoGPT's distribution than Fisher is) but the conversational dynamics are wrong: 1.2 s of overlap where humans produce 3.3 s, and a 12.5 s gap where humans take 4.2 s. It is speaking beautifully and too politely. At temperature 1.0 the dynamics land almost exactly on the human distribution — overlap 4.1 vs 3.3, gap 4.5 vs 4.2, pause 7.0 vs 6.4 — and the linguistic quality degrades to 79.3.

One more reading of the temperature sweep, because it is the most practically useful table in the paper for anyone building a product. Track all four dynamics columns as temperature rises from 0.8 to 1.0: IPU 35.1 → 50.8 (human 51.1), pause 13.2 → 7.0 (human 6.4), gap 12.5 → 4.5 (human 4.2), overlap 1.2 → 4.1 (human 3.3). All four converge on the human values simultaneously. That is not four coincidences; it is one underlying quantity — how readily the model commits to speaking — expressed four ways. Sampling temperature is functionally a conversational assertiveness dial, and it happens to be entangled with linguistic quality because both flow from the same distribution.

Which suggests the obvious product move the paper does not make: decouple them. Sample the text row at a low temperature (for quality) and the audio rows at a higher one (for dynamics). The architecture permits it — they are different rows of the same column, sampled by different Depth Transformer steps with per-index weights. Nothing in the paper reports trying this, and it is the first experiment this lesson would run.

Axis 4 — compression: what breaks first?

Moshi has to run on real hardware, so the paper studies post-training quantization: activations dynamically quantized to 8 bits (symmetric, AbsMax) at every linear input, weights quantized asymmetrically at several bitwidths and block sizes, with embeddings, RMSNorms and Mimi itself left in full precision.

Helium survives 4-bit weights within 2 MMLU points at 3.43× compression — a format nearly identical to llama.cpp's Q4_0. Moshi does not: the same recipe costs 5 to 10 points. The online demo therefore runs 8-bit, accepting a 2-point drop for roughly half the size.

Meanwhile the audio holds up far better than the reasoning. Measured as the share of generated windows free of artifacts: unquantized 95.8%, W4A8 block-32 95.7% — statistically nothing. At 3 bits it falls to 80.7% (block 32) or 62.7% (block 256), and at 2 bits it collapses to 45.4% and 5.9%.

Read the failure-mode column as the degradation ordering: text repetition appears before acoustic noise. At 3 bits the model is still producing clean-sounding speech while saying the same thing over and over — the exact failure MOSNet is blind to, and the exact reason the entropy diagnostic below had to be invented.

What this evaluation suite teaches you about evaluating your own system

Strip away the specific benchmarks and a transferable checklist remains. Any conversational speech system needs measurements on all four axes, because each one hides the others' failures:

And two methodological habits this paper is worth copying for:

Always report the reference distribution, not just the score. Turn-taking is meaningless as "overlap = 2.2 s"; it is meaningful as "2.2 s against a human 3.3 s". Half the numbers in Table 9 only became interpretable when the ground-truth row was measured in the same units on the same corpus.

When a metric and your ears disagree, instrument the model. Three times in this paper an objective metric pointed the wrong way — VisQOL on adversarial-only training, sWUGGY after instruct tuning, MOSNet on quantization artifacts — and each time the resolution came from either a human study or from measuring the generator's internal statistics (token entropy over a sliding window) rather than the output signal. That second move is cheap, general, and underused: your model's own output distribution is a diagnostic channel that no external metric has access to.

Inline check before you move on

(a) Moshi scores 22.8 on Audio TriviaQA against Helium's 56.4, but 26.6 against 32.3 on Web Questions. Why is one gap four times the other?
TriviaQA questions are written-register: multi-sentence, subordinate clauses, quiz-show syntax. Instruct tuning taught Moshi an oral register in which nobody speaks that way. It is a distribution mismatch in the question, not missing knowledge — the same weights answer the Web Questions version far better.

(b) Textless-NLP scores sum a timestep's tokens using the training weights (100 semantic, 1 acoustic). What would change if they were summed unweighted?
The score would be dominated by the seven acoustic tokens, which are close to noise — the metric would mostly measure how predictable the room tone is. Weighting keeps the comparison on the linguistic content, which is what sWUGGY and sBLIMP are asking about.

(c) Audio quality is essentially untouched at 4-bit weights while MMLU drops 7.5 points. Give the one-sentence reason.
An audio token is a choice among 2048 centroids whose neighbours sound similar, so small logit perturbations produce small perceptual errors; a factual answer is a single discrete commitment where a small perturbation flips right to wrong.

One more habit, easy to state and rarely followed: report the intermediate checkpoints. The most informative row in the paper's Table 7 is not the final model — it is "Moshi after single-stream pretraining", which shows a capability (sWUGGY 72.6) that the shipped model does not have (63.0). Without that row, the instruct-tuning regression would have been invisible, and the hypothesis about the corrupted user stream could never have been formed. A results table that reports only the final system hides every trade the training pipeline made.

Chapter 10: Safety, Limits, and What Came Next

A model that speaks in a chosen voice, in real time, from a 7B language model, raises questions that text models do not. The paper asks four of them explicitly, and answers three. The fourth answer is a negative result, and it is the most valuable page in the appendix.

Question 1 — does it say toxic things?

Measured with the ALERT red-teaming benchmark across hate, self-harm, weapons, crime, sex and substance categories. Because there is no established protocol for audio toxicity, the analysis is restricted to the text Moshi produces — that is, the Inner Monologue.

Mid-table, and the paper says so: "the industry models perform the best, which is expected considering the massive amount of private annotation, red-teaming and feedback loop from which these models have benefited." The safety data Moshi has is the synthetic refusal conversations from Chapter 7's instruct set — a few prompt templates, not an alignment programme.

The deeper caveat is methodological, and the paper flags it up front: comparing audio and text models on toxicity is not apples to apples, because "multiple meanings are conveyed by non-verbal signal (irony, tone, etc.)". A model that can say a safe sentence in a menacing voice is not evaluated by any benchmark in this table. The whole point of Chapter 0 was that prosody carries meaning; the safety evaluation quietly reintroduces the text bottleneck it argued against, because nothing better exists.

Question 2 — does it regurgitate training audio?

Regurgitation — reproducing a training sequence verbatim — is a familiar text problem with an extra edge in audio: what gets reproduced is not only words but a voice, its pitch and timbre, and any background music. That converts a memorization issue into a likeness-rights issue.

The protocol is careful. Build an audio fingerprinting system (Appendix B: a Shazam-style constellation map — mel-spectrogram keypoints filtered by energy, time and frequency maxima, hashed into triples with time offsets), use it to find the most frequent segment in the entire training set that is at least 16 seconds long, then measure how often it appears in 100,000 generations.

Question 3 — does it keep its own voice?

A speech-to-speech model hears a voice on the user stream at every timestep. Nothing in the architecture forbids copying it. The evaluation: generate 100 hours of conversation between Moshi and a synthetic second speaker, extract WavLM-large speaker embeddings per segment, and for each of Moshi's segments ask whether it is closer to Moshi's own first segment or to the other speaker's first segment (excluding anything starting before 15 s, so the reference turn is not counted).

Result: 10,249 segments (98.7%) closer to Moshi's own reference, 133 (1.3%) closer to the user's. And no drift — by 5-second buckets from 20 s out to 45 s, the consistency runs 98.4%, 99.2%, 99.1%, 99.2%, 99.3%, if anything improving.

What produced this? Not a module. Recall Chapter 7: every Moshi-side example in instruct fine-tuning is the same voice actor, and every user-side example is a random voice. The model learns "my stream sounds like this; the other stream sounds like anything" as a property of the two streams, and that is enough. The paper's own summary: "simply using a consistent voice for Moshi during instruction tuning is enough to guarantee almost surely that it does not use another voice, without further control during inference."

Question 4 — can we tell that Moshi generated an audio clip? (No.)

This is the negative result, and it is worth the whole section.

The obvious approach is signal watermarking. The paper evaluates AudioSeal, a strong open-source method, by measuring detection scores under several attacks:

Compare the last row with the first. After a single pass through Mimi — encode, decode — the watermarked audio scores lower than unwatermarked audio. The mark is not weakened; it is gone, below the noise floor. And the mechanism is not adversarial: "the two auto-encoders that we consider are low bitrate and therefore discard anything not related to the signal reconstruction." A watermark is by definition perceptually irrelevant information. A perceptual codec is by definition a machine for deleting perceptually irrelevant information. They are in direct opposition, and the codec wins.

So the authors tried the other family: generative watermarking, biasing the sampling distribution with a hash-keyed pattern, as done for text. To detect it you must re-encode the audio into tokens and look for the bias. That requires the codec to be idempotent — decode tokens to audio, re-encode, get the same tokens back. Mimi is not:

Even with no attack at all, only 40% of level-8 indices survive a round trip. Shift the audio by 40 ms — half a frame, an operation any audio editor performs invisibly — and the semantic token survives barely half the time. The hash context that a sampling watermark depends on is destroyed. Shortening the context makes the hash more stable but pushes the generation toward degeneracy, which is a familiar trap from text decoding.

To their credit the authors publish the failure and sketch four repairs: mark only the first RQ levels (measurably more stable); add an explicit idempotence loss in the discrete latent space so tokens are stable through auto-encoding; make that stability robust to small time shifts, as image watermarking does for geometric transforms; or watermark the text stream instead (low capacity, and detection would need reliable transcription). They close with the sharpest observation in the section: for one popular open-weights image model, removing the watermark required commenting out a single line of code. Any watermark shipped with open weights is a request, not a control.

Limitations, stated plainly

Where Moshi sits in the lineage

Upstream. The codec is a direct descendant of SoundStream and EnCodec — SeaNet autoencoder, residual vector quantization, adversarial training — with the frame rate pushed 4× lower and the split-RVQ semantic graft added. The hierarchical semantic→acoustic generation is AudioLM's, extended upward with text. The delay-pattern idea comes from MusicGen; the two-transformer factorization from RQ-Transformer and MegaByte; the distillation of a self-supervised teacher into codebook 1 from SpeechTokenizer. Almost nothing here is unprecedented in isolation. The contribution is the composition, plus two genuinely new pieces: multi-stream modelling and Inner Monologue.

Downstream. Moshi became the reference architecture for real-time speech-to-speech, and the field promptly measured where it falls short. On FullDuplexBench, PersonaPlex (NVIDIA, 2026) reports 100% barge-in success against Moshi's 60.6% and Gemini Live's 43.9%, and a task-adherence score of 4.34 against Moshi's 1.26 — while adding voice and role control that Moshi's single-actor fine-tuning cannot express. Sesame's CSM took the same dual-transformer shape in a different direction, chasing "voice presence" through contextual, conversation-conditioned TTS. Qwen2.5-Omni's Thinker–Talker split is Inner Monologue's idea — a text brain feeding a speech mouth — scaled to full multimodality.

Read the barge-in numbers charitably in both directions. 60.6% is not a good barge-in success rate for a product, and Moshi's own paper never claimed one — the ability to represent interruption is not the same as reliably acting on it, and the Fisher fine-tuning stage that teaches duplex behaviour is 10k steps out of more than 1.6M. Moshi proved the architecture; the follow-ups are proving the behaviour.

Cheat sheet — every symbol and number

Further reading

1. Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., Zeghidour, N. "Moshi: a speech-text foundation model for real-time dialogue." Kyutai, 2024. arXiv:2410.00037
2. Zeghidour, N. et al. "SoundStream: An end-to-end neural audio codec." 2022 — the RVQ codec Mimi descends from.
3. Défossez, A. et al. "High fidelity neural audio compression" (EnCodec), 2023 — the loss recipe Mimi starts from and then discards.
4. Borsos, Z. et al. "AudioLM: a language modeling approach to audio generation," 2022 — the semantic→acoustic hierarchy Inner Monologue extends.
5. Zhang, X. et al. "SpeechTokenizer," 2024 — distillation into the first RVQ level, which the split RVQ repairs.
6. Chen, S. et al. "WavLM," 2022 — the frozen teacher, and the speaker-verification model used for the voice-consistency study.
7. Copet, J. et al. "Simple and controllable music generation" (MusicGen), 2023 — delay patterns between codebook levels.
8. Lee, D. et al. "Autoregressive image generation using residual quantization," 2022; Yu, L. et al. "MegaByte," 2024 — the time×depth factorization.
9. Nguyen, T.A. et al. "Generative spoken dialogue language modeling" (dGSLM), 2023 — the first full-duplex system, and the turn-taking metrics.
10. Stivers, T. et al. "Universals and cultural variation in turn-taking in conversation," PNAS 2009 — where the 230 ms comes from.
11. San Roman, R. et al. "Proactive detection of voice cloning with localized watermarking" (AudioSeal), 2024 — the watermark that Mimi erases.

If you reread only three things from the paper itself, make them Figure 4 (the joint-sequence diagram — the whole architecture on one page), Table 5 (the RQ-Transformer ablation, whose meaning depends entirely on the delay column), and Section 6.4 (the watermarking negative result, which is more useful than most positive ones).