Qwen2.5-Omni: the Thinker and the Talker

Chapter 1: The Thinker and the Talker

Chapter 0 ended with a coupling problem: one model must emit two different kinds of output, text and speech, and training either one damages the other. This chapter is about the paper's answer, which is so simple it is easy to underrate — and about why the obvious alternative fails.

Start with the failure, because the need has to be manufactured before the solution means anything.

Why not just add speech tokens to the vocabulary?

Here is the design any of us would try first. A language model already emits tokens from a vocabulary. Speech can be turned into tokens too — that is what a neural audio codec does. So: concatenate the vocabularies. Give the model 151,643 text tokens plus, say, 4,096 speech codes, and let one softmax head choose among all of them. Train on interleaved text-and-speech data. Done.

Now count what goes wrong.

Statistical mismatch. Text tokens are slow and information-dense: a 7B model emits maybe 30–60 of them per second of speech-equivalent content, and each carries a large amount of meaning. Speech codes are fast and information-sparse: you need dozens per second per codebook, and any individual one carries almost nothing — it is a fragment of a fragment of a phoneme. Ask one softmax to model both distributions and you have asked it to be simultaneously sharp (over meanings) and smooth (over acoustic detail). The gradients pull the shared trunk in different directions.

Sequence-length mismatch. The same reply, in text, is maybe 40 tokens. In speech codes it is hundreds. If they share one stream, the speech codes dominate the sequence and therefore the loss, and the model's language ability — the thing you spent trillions of tokens building — gets diluted by acoustic bookkeeping.

Interference in the residual stream. This is the subtle one and the one the paper names directly: "it is essential to manage potential interference among outputs from different modalities, ensuring that the training processes for outputs such as text and voice tokens do not disrupt each other." The last few layers of a decoder are specialized for the output head — they arrange the residual stream into a shape that the unembedding matrix can read. If the head must serve two very different vocabularies, those last layers must serve two masters, and the compromise costs both.

The split

Thinker is a Transformer decoder with an audio encoder and a vision encoder attached. It processes text, audio, images and video; it produces high-level representations; it generates text. It is, functionally, a normal multimodal LLM. If you deleted Talker, you would still have a competent model — it would just be mute.

Talker is a dual-track autoregressive Transformer decoder. It takes in Thinker's high-dimensional representations and the text tokens Thinker sampled, and it emits discrete speech codes. It has no encoders of its own and no independent understanding of the world. Its entire job is: given what the brain is currently thinking and currently saying, produce the sound.

And — this is the part that makes it "end-to-end" rather than "a TTS bolted on" — Talker shares all of Thinker's historical context information. It is not handed a sentence in isolation. It sees the conversation. It sees, in representation form, that the user sounded worried and that the video showed a burnt component.

What "division of labour" buys, precisely

The abstract makes a claim that is easy to skim past: the block-wise encoder strategy "effectively decouples the handling of long sequences of multimodal data, assigning the perceptual responsibilities to the multimodal encoder and entrusting the modeling of extended sequences to a large language model. Such a division of labor enhances the fusion of different modalities via the shared attention mechanism."

Unpack it. Two jobs are being separated:

• Perception — turning raw signal into meaningful features. This is local. Whether a 40 ms slice of audio is a fricative depends on its neighbours within a few hundred milliseconds, not on what happened thirty seconds ago. So the encoder only needs local attention, which means it can work in blocks, which means it can stream.
• Long-range modelling — relating things far apart in time, across modalities, across turns. This is global, and it is exactly what a big transformer decoder with full attention is for.

Giving the encoder only the local job is what makes streaming possible and what improves fusion: because the encoder does not consume the long-range budget, all cross-modal integration happens inside the LLM's shared attention, where a video token and an audio token and a text token are all just vectors in one sequence that can attend to each other freely.

Explore the architecture

Click through the components below. Each shows what goes in, what comes out, and what would break if you removed it. The Interference toggle switches to the single-head design we rejected above, so you can see where the conflict physically lives.

Contrast: Moshi's single stream

Thinker-Talker is not the only way to make a model talk in real time, and the most interesting alternative is worth holding beside it. Moshi (Kyutai, 2024) takes the opposite bet: one stream, with text and audio interleaved inside a single autoregressive model, plus a second parallel stream for the user's audio so the model can listen and speak at the same time.

Which is better is genuinely unsettled, and the honest answer is "different products". Moshi optimizes for conversational dynamics — interruption, overlap, backchannels. Qwen2.5-Omni optimizes for breadth — it also sees, and it keeps a full-strength LLM intact behind the voice. Note which capability each one sacrificed to get the other.

Code: the shape of the handoff

Here is what the Thinker-Talker interface looks like in pseudocode, at three levels of resolution. First the naive single-head design we rejected:

python — the single-head design (rejected)
# One vocabulary, one head. Simple, and it fights itself.
V_text  = 151643          # Qwen byte-level BPE, regular tokens
V_codec = 4096            # illustrative speech-code vocabulary
head = Linear(d_model, V_text + V_codec)

for step in range(T):
    h = trunk(seq)                       # the SAME residual stream
    logits = head(h[-1])                 # must be sharp over meanings AND
    tok = sample(logits)                 # smooth over acoustics. Pick one.
    seq.append(tok)
# Failure: hundreds of codec steps per ~40 text tokens, so the loss
# is dominated by acoustics and language ability erodes.

Now the Thinker-Talker version. Notice that the loop is the same loop — the handoff happens inside it, per step, which is what makes the speech streaming rather than post-hoc:

python — Thinker-Talker, streaming handoff
def omni_turn(inputs):
    ctx = thinker.prefill(inputs)          # chunked prefill (Ch 6)
    talker.share_history(ctx)             # Talker sees ALL of Thinker's context

    while not done:
        h, logits = thinker.step()         # h: (B, 1, d) hidden state
        text_tok  = sample(logits)         # (B, 1) discrete
        emit_text(text_tok)

        # THE HANDOFF: continuous state + discrete token, both
        cond = talker.fuse(h, embed(text_tok))    # (B, 1, d_talker)
        for code in talker.step(cond):        # several codes per text token
            code_buf.append(code)
            if len(code_buf) >= block + lookahead:   # Ch 6
                mel  = dit.decode_block(code_buf)   # flow matching
                wav  = bigvgan.chunk(mel)
                play(wav)                          # audio is already out

Read the innermost if again. That single condition is the entire streaming story: audio leaves the machine as soon as one block of codes plus one block of lookahead exists — not when the sentence ends, not when the reply ends. Chapter 6 derives why the lookahead is needed and Chapter 7 turns it into a latency ledger.

What each half can and cannot do alone

That table is the argument for the design, stated as four things that break. Keep it: in Chapter 5 we will justify rows 2 and 3 from first principles, and in Chapter 9 we will see the benchmark that row 1 makes possible — the comparison against Qwen2.5-VL that shows the price of adding a mouth.

Three more things before the quiz: the object graph you would actually build, the exact tensor that interference happens in, and a fair hearing for the strongest objection to this design. Take them in that order, because each makes the next one easier to argue about.

The system, as code you could serve

One more pass over the architecture, this time as the object graph you would actually instantiate. Every shape is derived from a rule established in the chapters that follow; nothing here is invented.

python — the whole system, sketched
class Qwen25Omni:
    def __init__(self):
        # --- perception (Ch 2) ---
        self.audio_enc  = WhisperEncoder(mels=128, block_sec=2.0)  # 1 frame / 40 ms
        self.vision_enc = QwenViT(patch=14, merge=2, params=675e6)   # 2x2 -> 1
        self.a_adapter  = Linear(d_audio,  d_model)
        self.v_adapter  = Linear(d_vision, d_model)

        # --- the brain (Ch 1) ---
        self.thinker = Qwen25Decoder(vocab=151643, rope="tmrope")

        # --- the mouth (Ch 5) ---
        self.talker  = DualTrackDecoder(codec="qwen-tts-tokenizer")

        # --- codes to sound (Ch 6) ---
        self.dit     = FlowMatchingDiT(lookback=2, lookahead=1)     # 4 blocks
        self.vocoder = BigVGAN(chunked=True)

    def build_prompt(self, text, audio, frames, fps):
        # Ch 2: encode each modality into the LLM's space
        a = self.a_adapter(self.audio_enc(audio))     # (25*secs,  d_model)
        v = self.v_adapter(self.vision_enc(frames))   # (F*(px/28)^2, d_model)
        t = self.thinker.embed(text)                  # (n_text,   d_model)
        # Ch 3 + Ch 4: interleave in 2 s chunks, assign (t,h,w) ids by wall clock
        seq, pos = interleave_and_position(t, v, a, fps, chunk=2.0)
        return seq, pos                             # (T, d_model), (T, 3)

Notice what the constructor makes obvious: seven components, two of which are new. Talker and the codec are this paper's inventions. Everything else is a checkpoint plus a modification. The value is concentrated entirely in build_prompt and in the handoff — that is, in the wiring.

And notice what build_prompt returns: not one tensor but two. The sequence and its positions are separate objects, computed by separate rules, and Chapters 3 and 4 are about why that separation is the point.

Where the interference physically lives

"Interference" is a vague word until you can point at the tensor it happens in. Let us do that, because it turns an intuition into an argument.

A decoder's final layers are not general-purpose. Over training they specialize into a readout stage: they arrange the residual stream so that multiplying by the unembedding matrix produces well-calibrated logits. Concretely, for a token to be predicted with high probability, the residual vector must have a large component along that token's unembedding row.

Now put two vocabularies in there.

• Text tokens live in a space where semantic neighbours cluster: the rows for cat and kitten point in similar directions, because contexts that predict one predict the other.
• Speech codes live in a space where acoustic neighbours cluster: two codes are similar if they sound similar, which has nothing to do with meaning.

A single residual stream, feeding a single unembedding, must simultaneously be positioned correctly in both geometries at every step. The optimizer resolves this the way optimizers always do: by compromise. And because the speech codes vastly outnumber the text tokens in any training batch, the compromise is not symmetric — it settles closer to the acoustic geometry.

That ratio is the whole problem in one number. The gradient signal reaching the shared trunk is dominated by the acoustic task, so language modelling — the capability that took trillions of tokens to build — is the thing that degrades.

Separating the heads removes the compromise by construction. Thinker's last layers only ever serve text. Talker's only ever serve codes. Neither has to be two things at once.

Before the objection, one framing to carry forward: Thinker-Talker is a claim that generation can be modular even when understanding must be unified. All the modalities meet in one attention on the way in; only on the way out do they separate. That asymmetry is deliberate.

Two organs, one nervous system — how far the analogy goes

The authors reach for biology, and analogies deserve to be tested rather than admired.

Three rows hold, one fails, one is partial. That is a good analogy: it earns its keep on the parts that motivated the design and it fails exactly where the system's real limitations are.

An objection worth taking seriously

Here is the strongest argument against Thinker-Talker, and it deserves a fair hearing because it is the argument Moshi's authors implicitly made.

The objection: separating thinking from talking recreates, in miniature, the very seam the cascade suffered from. Talker sees Thinker's states but Thinker does not see Talker's. So the speech generator can never influence what is said — it can only render it. A human, choosing words, is partly choosing them because of how they will sound; a Thinker that cannot hear its own mouth cannot do that.

The reply: true, and the paper does not claim otherwise. What it claims is narrower and defensible — that the seam is continuous rather than lexical. The cascade's seam destroyed information (a string is all that crosses). This seam preserves it (hidden states cross). The influence is one-way, but the bandwidth is high.

What it costs: anything requiring speech to feed back into content. Choosing a shorter word because the sentence is running long. Rephrasing because a tongue-twister emerged. Adjusting content mid-utterance because the user started talking — which is the full-duplex problem again, from a different direction.

Five architectures for a talking model, side by side

Do not read these sequentially; read the columns. The interesting comparison is which property each design gets for free and which it must engineer.

Nobody is winning on every column, and that is the honest state of the field. Each row picked a property to optimize and paid for it somewhere else. The skill is reading which column your product actually needs.

What Mini-Omni contributed

The report names its influence precisely: Talker is "a dual-track autoregressive Transformer Decoder architecture, motivated by Mini-Omni (Xie & Wu, 2024)". Worth understanding what that means, because it is the one piece of Talker's design with a clear published lineage.

Mini-Omni's observation: if you ask a model to generate hundreds of acoustic tokens in a row with only acoustic history to condition on, it drifts. It repeats, it slurs, it wanders into babble — failure modes anyone who has trained an audio LM recognizes. The fix is to carry a text token stream in parallel with the audio stream, so every step is anchored to a symbol.

Why that works: text is a low-entropy, highly structured signal that the model is already extremely good at predicting. Predicting it alongside the acoustics regularizes the whole trajectory — the text track cannot drift without the language model noticing, and the audio track is tied to the text track.

What the boundary actually carries

One more level of concreteness. Per generated text token, the handoff transmits:

Notice that the first two are the same shape and the same type, and that they are added rather than concatenated in the sketch above. That is a design choice this lesson is inferring rather than quoting — the report says Talker "receives both high-level representations and embeddings of the text tokens sampled by Thinker" without specifying the fusion operator. Addition, concatenation-then-projection, and cross-attention are all plausible; the argument for why both signals are needed does not depend on which.

Chapter 2: Perceivation — Waves and Pixels Become Tokens

Chapter 1 drew boxes. This chapter fills them with numbers, because the next two chapters are about aligning audio tokens with video tokens on a timeline, and you cannot align things whose rates you do not know.

By the end of this chapter you will be able to answer, for any clip, exactly how many tokens the model sees and how much real time each one covers. That is the arithmetic TMRoPE operates on.

The audio path, step by step

The report gives the audio front end in one dense sentence: "we resample it to a frequency of 16kHz and transform the raw waveform into a 128-channel mel-spectrogram with a window size of 25ms and a hop size of 10ms. We adopt the audio encoder from Qwen2-Audio, to make each frame of audio representation roughly corresponds to a 40ms segment of the original audio signal."

Five numbers: 16 kHz, 128 channels, 25 ms window, 10 ms hop, 40 ms per output frame. Let us derive the last one from the others, because the gap between 10 ms and 40 ms is where all the downsampling hides.

Step 1 — resample. Whatever came in (44.1 kHz from a phone, 8 kHz from a phone line) is converted to 16,000 samples per second. Why 16 kHz? Because by the Nyquist theorem it represents frequencies up to 8 kHz, and essentially all speech information lives below that. Doubling the rate would double the compute for content that is mostly breath noise.

Step 2 — frame it. A 25 ms window at 16 kHz is

And a 10 ms hop is

Windows therefore overlap by 400 − 160 = 240 samples, or 60%. Overlap matters: a phoneme boundary that fell exactly on a window edge would be smeared, and overlapping windows guarantee every instant is well inside at least one window.

Step 3 — count mel frames. The hop determines the frame rate. One frame every 10 ms is

Each frame is a vector of 128 numbers — energy in 128 mel-spaced frequency bands. So one second of audio is a 100 × 128 array. This is the "picture of sound" the encoder actually sees.

Step 4 — the encoder downsamples. The paper says the encoder is Qwen2-Audio's, which is itself Whisper's, and that the result is roughly 40 ms per frame. Whisper's stem is two 1-D convolutions, the second with stride 2, taking 100 frames/s down to 50 frames/s (20 ms each). Qwen2-Audio adds a pooling layer with stride 2 on top, which halves it again:

There it is. Every audio token the LLM sees covers 40 ms of real time. Remember that number — the whole of TMRoPE is built on it.

Worked example: how many tokens is a 4-second clip?

Take a concrete 4-second recording. Every step, no skipping.

Note the two ways of counting mel frames — 398 versus 400 — and that they differ by exactly the boundary convention. Real implementations centre-pad, which is why "100 frames per second" is exactly true rather than nearly true. This is the kind of off-by-two that ruins alignment code, so it is worth having seen once.

Compression ratio, end to end: 64,000 numbers in, 100 tokens out. Each token carries the information of 640 raw samples. The encoder's job is to make sure those 640 samples' worth of meaning survives.

python — the audio front end, from scratch
import numpy as np

SR, N_FFT, HOP, N_MELS = 16000, 400, 160, 128   # 25 ms / 10 ms / 128 bands

def frames(x, win=N_FFT, hop=HOP):
    # centre-pad so frame k is centred at sample k*hop
    x = np.pad(x, win // 2, mode="reflect")
    n = 1 + (len(x) - win) // hop
    return np.stack([x[k*hop : k*hop+win] for k in range(n)])   # (n, 400)

def log_mel(x):
    F   = frames(x) * np.hanning(N_FFT)          # (n, 400)
    S   = np.abs(np.fft.rfft(F, axis=-1)) ** 2     # (n, 201) power
    M   = S @ mel_filterbank(N_MELS, N_FFT, SR).T   # (n, 128)
    return np.log10(np.maximum(M, 1e-10))

x   = np.zeros(4 * SR)          # 4 seconds -> 64000 samples
mel = log_mel(x)               # (400, 128)   == 100 frames/s
# encoder stem: conv stride 2 -> (200, d); pool stride 2 -> (100, d)
print(mel.shape[0] // 4)      # 100 audio tokens == 4 s / 40 ms

And the library one-liner you would actually run, which produces the identical array:

python — the one-liner
from transformers import AutoProcessor
proc = AutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B")
feats = proc(audio=x, sampling_rate=16000, return_tensors="pt")
# feats["input_features"] -> (1, 128, 400)  == (batch, mels, frames)
# after the encoder: (1, 100, d)            == one token per 40 ms

The vision path, step by step

The vision encoder is Qwen2.5-VL's ViT, about 675 million parameters, trained on a mix of image and video data so it is competent at both. Three published specifics matter for us:

• Patch size 14. The image is cut into non-overlapping 14×14-pixel squares; each becomes one initial token. Patch 14 is the standard ViT-L/14 choice inherited from CLIP-era vision transformers.
• A simple MLP merges adjacent 2×2 tokens into one. Four neighbouring patch tokens are concatenated and projected down to a single token, cutting the token count by 4×.
• Native-resolution packing. Because patches are a fixed pixel size rather than a fixed grid, images of different resolutions can be packed into one sequence — a big image simply produces more tokens.

Worked example. A 448 × 448 video frame:

A smaller 224 × 224 frame: 224/14 = 16 per side, 256 patches, merged to 8 × 8 = 64 tokens. A 112 × 112 frame: 112/14 = 8 per side, 64 patches, merged to 4 × 4 = 16 tokens. We will use the 16-token frame in Chapter 3's position arithmetic because it is small enough to write out entirely.

The one strange rule: an image is two frames

"For consistency, each image is treated as two identical frames." Read that twice, because it looks like a typo and is not.

Why would a still image be duplicated? Because the vision encoder was trained on video, and video processing in this family of models operates on frame pairs — temporal patches spanning two consecutive frames. If you feed a single frame to a stack expecting pairs, you either need a special code path or you pad. Duplicating the image gives you a legal two-frame clip in which nothing moves. The temporal component of the position encoding stays constant across the pair (Chapter 3), so the model correctly reads it as "a still".

This is a small detail with a large lesson: uniformity beats special-casing. Every place where images and video would need different handling is a place where a bug can hide and a training-inference mismatch can grow. Paying a 2× token cost on still images to have exactly one code path is a trade the authors clearly considered worth it.

Explore the two front ends

Play with the frame-rate slider and watch the visual token count relative to the audio token count. At 448 px and 2 fps you are spending 512 visual tokens per second against 25 audio tokens — a 20:1 ratio. Video is, by a wide margin, the expensive modality, and every design decision about video in this paper is downstream of that ratio.

Text, for completeness

Text uses Qwen's tokenizer: byte-level byte-pair encoding with a vocabulary of 151,643 regular tokens. Byte-level BPE means there is no out-of-vocabulary failure mode — any byte sequence, in any language or encoding, is representable, at worst one token per byte. The word "regular" is doing quiet work: on top of the 151,643 sit the special control tokens you saw in the ChatML example (<|im_start|>, <|vision_start|>, and so on), which mark modality boundaries in the sequence.

The counts are settled. What remains is to make one frame's values concrete, and then to justify the two choices — the mel warp and the number 128 — that the counts took for granted.

One mel frame, computed entirely by hand

Everything above was arithmetic about counts. Do one frame's actual values once, on a toy, so nothing about the front end remains a black box.

Shrink the problem: window of 8 samples instead of 400, sample rate 8 Hz instead of 16 kHz, and 3 mel bands instead of 128. The structure is identical; only the sizes shrink.

Step 1 — the windowed samples. Take a pure tone at 2 Hz, sampled 8 times per second, for 8 samples. That is x[n] = sin(2π · 2 · n/8) = sin(πn/2):

Check n = 1: sin(π/2) = 1. ✓   n = 2: sin(π) = 0. ✓   n = 3: sin(3π/2) = −1. ✓

Step 2 — the DFT magnitudes. For a real 8-point signal the FFT gives 8/2 + 1 = 5 useful bins, at frequencies 0, 1, 2, 3 and 4 Hz. A pure 2 Hz sinusoid puts all its energy in bin 2. Computing bin 2 explicitly:

Term by term, using x = [0, 1, 0, −1, 0, 1, 0, −1]:

Every other bin sums to zero by the same cancellation. So the power spectrum is:

Step 3 — the mel filterbank. Three triangular filters, overlapping, covering the five bins. A plausible small bank:

Step 4 — multiply. Each mel value is the dot product of a filter row with P:

Step 5 — log compression. Clamp at a floor and take the log, exactly as the real pipeline does:

That three-number vector is one column of the spectrogram. Scale it up — 400 samples, 201 bins, 128 bands, one column every 10 ms — and it is exactly what Qwen2.5-Omni's audio encoder receives. The arithmetic does not change; only the sizes do.

The log matters, by the way, and for the same reason the mel warp does: loudness perception is logarithmic. A sound ten times more powerful is heard as roughly one step louder, not ten. The log makes the numeric scale match the perceptual one, which is what lets a neural network treat differences in the array as differences in what a listener would notice.

With the counts and one full frame's values in hand, one question remains about the audio path: why these particular frequency bands?

Why "mel", and why 128 of them

We have used the word "mel" three times without defining it. Fix that now, because the choice of 128 is one of the paper's inherited-but-consequential decisions.

Human hearing is not linear in frequency. The difference between 100 Hz and 200 Hz is enormous perceptually — an octave, a completely different pitch. The difference between 8,000 Hz and 8,100 Hz is inaudible. The mel scale is a warping of frequency designed so that equal steps in mel correspond to equal steps in perceived pitch. The standard formula:

Work it by hand for the endpoints of our band, so the compression is visible:

Read the last column. The first 200 Hz of real frequency occupies 283 mel — ten percent of the whole scale. The last 4,000 Hz (from 4 kHz to 8 kHz) occupies only 694 mel. Low frequencies, where vowels and pitch live, get vastly more resolution than high frequencies, where only sibilance and noise live. That is exactly how your cochlea allocates hair cells.

So what does a mel filterbank do? It takes the linear-frequency power spectrum from the FFT — for a 400-sample window that is 201 frequency bins — and sums it into 128 overlapping triangular bands whose centres are equally spaced in mel. In matrix terms:

Compression from 201 numbers to 128, allocated where hearing actually is.

And why 128 rather than 80? Because Whisper-large-v3 uses 128, and this encoder is initialized from Whisper-large-v3. Earlier Whisper versions used 80, the long-standing default in speech recognition. The v3 bump to 128 buys finer spectral resolution, which matters more once the model is asked to do things beyond transcription — identifying instruments, classifying vocal sounds, judging musical tempo. Qwen2.5-Omni is asked to do all of those (Chapter 9 has the VocalSound and GiantSteps numbers), so it inherits the higher-resolution front end and benefits from it.

Frozen or trained? Reading the encoder stack

A question worth asking of any composed system, and the answer is staged (full detail in Chapter 8):

The row that teaches the most is the first. The mel transform has no parameters, and yet it is doing an enormous amount of work — a 640× compression with a perceptually motivated frequency warp, entirely by hand-designed signal processing. Even in 2025, the very first stage of the most modern omni model is a piece of 1930s psychoacoustics.

Putting the rates together: a one-minute call

Make it concrete with a realistic input, because this is the arithmetic that decides whether a design is deployable.

One minute of 448-pixel video at two frames per second essentially fills the entire 32k context. That is not a coincidence and it is not a comfortable margin — it is the design point. It also explains the dynamic frame rate as something other than a nicety: drop to 1 fps and you halve the cost; drop the resolution to 224 px and you quarter it again. Those knobs are what keep the model usable on real video, and Chapter 7 lets you turn them.

Chapter 3: TMRoPE — Putting Everything on One Clock

We now have three streams of tokens with three different rates. Audio ticks 25 times a second. Video ticks at whatever frame rate we chose. Text does not tick at all. They are about to be poured into one sequence and fed to one attention mechanism, and the attention mechanism has no idea what time it is.

That is the problem TMRoPE solves. It is the paper's one genuinely novel piece of mathematics, and it is small enough to derive completely.

Step back: what does a position embedding actually do?

Self-attention is permutation invariant. Shuffle the tokens and the attention scores shuffle with them — the mechanism itself cannot tell "the cat sat" from "sat the cat". Order has to be injected.

Rotary Position Embedding (RoPE) injects it by rotating. Split each query and key vector into consecutive pairs of coordinates. Treat each pair as a point in a 2-D plane. Then, for a token at position m, rotate pair i by the angle m · θ_i, where θ_i is a fixed frequency assigned to that pair.

Why rotation? Because of one beautiful property. The attention score between a query at position m and a key at position n is a dot product, and rotating both by their own angles makes that dot product depend only on m − n:

Absolute positions go in; a relative position comes out. The model never has to learn "position 4,127" as a special thing; it only ever perceives distances. That is why RoPE extrapolates gracefully and why it took over the field.

From 1-D to 3-D: M-RoPE

Plain RoPE assumes position is a single number. For an image that is wrong — a patch has a row and a column, and flattening them into one index destroys the two-dimensional structure. Qwen2-VL's answer, M-RoPE, is to split the rotation budget three ways.

Instead of one position m, every token carries a triple: (t, h, w) — temporal, height, width. The head dimension is partitioned into three groups of frequency pairs; group T is rotated by t · θ, group H by h · θ, group W by w · θ. The relative-position property still holds, independently, on each axis.

That last row is TMRoPE. The report defines it as "Multimodal Rotary Position Embedding (M-RoPE) with absolute temporal positions". Everything else was already in Qwen2-VL. The contribution is: stop letting the temporal index mean "the n-th thing in the sequence", and make it mean "the moment at time n × 40 ms".

Dynamic frame rate: how video gets onto the 40 ms grid

Video frame rates are not fixed and generally are not multiples of 25 fps. The report handles this directly: "Since the frame rate in video is not fixed, we dynamically adjust the temporal IDs between frames based on the actual time corresponding to each frame to ensure that one temporal ID corresponds to 40ms."

So the rule for a video frame displayed at time τ seconds after the start of the visual block is:

The temporal IDs of successive frames are therefore not consecutive integers. At 2.5 fps they step by 10. At 1 fps they step by 25. At 25 fps they step by 1. The gaps encode the frame rate, which means the model can tell a slideshow from a fast pan without being told.

Modality chaining: where each block starts

One more rule, quoted exactly: "In scenarios where the model's input encompasses multiple modalities, position numbering for each modality is initialized by incrementing the maximum position ID of the preceding modality by one."

Note carefully: the maximum, across all three components, of everything that came before — plus one. Not the count of tokens. This matters because a modality's IDs can be sparse (a 4-second video occupies 100 temporal IDs but may contribute only 10 frames) and because the h and w components can run ahead of t for a tall image.

The fully worked position-ID example

Now we compute a complete assignment by hand. Nothing is skipped. Take the input:

Part A — the text prefix. Text uses identical components. Token i gets (i, i, i):

Maximum position ID so far = 2. Therefore the video-with-audio block starts at st = 2 + 1 = 3.

Part B — how many tokens of each kind? From Chapter 2's arithmetic:

• Audio: 4 s ÷ 0.040 s = 100 audio tokens, indexed j = 0 … 99.
• Video: 2.5 fps × 4 s = 10 frames, indexed k = 0 … 9, at times τ_k = 0.4k seconds.
• Each frame: 112/14 = 8 patches per side → 8×8 = 64 patches → 2×2 merge → 4×4 = 16 tokens per frame, at grid positions (r, c) with r, c in {0,1,2,3}.
• Total visual tokens: 10 × 16 = 160. Total in the block: 160 + 100 = 260 tokens.

Part C — audio temporal IDs. Audio token j covers the interval [0.040j, 0.040(j+1)) seconds. Identical components, anchored:

So j = 0 gives (3, 3, 3); j = 30 gives (33, 33, 33); j = 99 gives (102, 102, 102). Maximum audio ID = 102.

Part D — video temporal IDs. Frame k sits at τ_k = 0.4k seconds. Convert to temporal ID:

Why floor rather than round? Because floor means "the 40 ms tick that contains this frame". That makes the alignment exact at every frame rate: whichever audio token covers the same instant carries the same number, with no off-by-one at rates that do not divide 25 evenly. At 2.5 fps the two conventions agree anyway — 0.4k / 0.04 is exactly 10k — which is why the example below is clean either way.

Part E — the alignment check, which is the whole point. Look at frame k = 3: temporal ID 33. Now find the audio token covering 1.2 s: j = 1.2 / 0.04 = 30, giving temporal ID 3 + 30 = 33. Identical.

Two tokens of completely different modalities, arriving at completely different points in the flattened sequence, carry the same temporal position ID because they describe the same 40 ms of the world. When attention computes their relative temporal rotation it gets zero. They are, to the model, simultaneous.

Part F — spatial components for video. Frame k, grid cell (r, c):

So the token at frame 3, row 2, column 1 is (33, 5, 4). Its neighbour to the right is (33, 5, 5); the one below is (33, 6, 4); the same cell one frame later is (43, 5, 4). Every relationship the model might want — adjacency in space, adjacency in time — is a small difference in exactly one component.

Part G — the trailing text token. Maximum position ID anywhere in the block: audio reached 102; video reached t = 93 and h = w = 6. So max = 102, and the next text token starts at 103, receiving (103, 103, 103).

Now the rotation itself, with actual numbers

Position IDs are only half the story; they have to become rotations. Take a toy head dimension of d = 8, giving 4 frequency pairs. The frequencies:

M-RoPE partitions those 4 pairs among the three axes. Real configurations give temporal the low-frequency (long-range) pairs; for our toy, allocate pairs {0, 1} to t, pair {2} to h, pair {3} to w.

Take our token from Part F: (t, h, w) = (33, 5, 4). The four rotation angles are:

The 33-radian angle is worth checking by hand, because it shows the wrap-around that makes the fast pairs work as fine-grained hands: 33 − 10π = 33 − 31.415927 = 1.584073 rad, which is just past a quarter turn — hence cos near zero and sin near one.

Apply them to a query vector q = (1, 0,  1, 0,  1, 0,  1, 0), pair by pair, using the rotation formulas above (each pair has x_2i = 1, x_2i+1 = 0, so x' = cosα and x'' = sinα):

Now the payoff computation. Take the audio token j = 30, whose triple is (33, 33, 33), and give it the same raw vector. Its angles: pair 0 → 33 × 1 = 33 (same as the video token), pair 1 → 3.3 (same), pair 2 → 33 × 0.01 = 0.33, pair 3 → 33 × 0.001 = 0.033.

The relative angles between the video token and the audio token are therefore:

The attention mechanism sees these two tokens as temporally coincident, differing only along the spatial axes (which is correct: audio has no meaningful spatial position, and it sits at a spatial "origin" defined by the block start). That is the entire mechanism, in numbers you just checked.

Compare with what plain sequence-index RoPE would have produced. The video token is the 3×16 + 2×4 + 1 = 57th visual token in the block, the audio token is the 160 + 30 = 190th token in the block — a relative distance of 133 positions. The model would perceive the clap and the hands meeting as separated by 133 tokens' worth of phase. TMRoPE turns 133 into 0.

Build the position table yourself

Code: the assignment, three ways

First, by hand for one token — the arithmetic you just did:

python — one token, by hand
st   = 3            # max position id of the text prefix, plus one
k, r, c = 3, 2, 1    # video frame 3, grid row 2, column 1
fps  = 2.5
tau  = k / fps                      # 1.2 seconds
t_id = st + int(tau / 0.040)       # 3 + 30 = 33  (floor: the tick containing tau)
h_id = st + r                       # 5
w_id = st + c                       # 4
print(t_id, h_id, w_id)              # 33 5 4

Then the whole block, step by step in numpy — this is the function you would actually write:

python — the full TMRoPE position table
import numpy as np
TICK = 0.040                   # one temporal ID = 40 ms. The whole paper.

def tmrope_block(st, secs, fps, grid, n_audio_tok):
    """Return (N,3) int array of (t,h,w) ids, and the new max id."""
    ids = []
    # --- visual tokens: t from wall-clock time, h/w from the merged grid
    n_frames = int(round(secs * fps))
    for k in range(n_frames):
        t = st + int((k / fps) / TICK)
        for r in range(grid):
            for c in range(grid):
                ids.append((t, st + r, st + c))
    # --- audio tokens: identical components, one id per 40 ms
    for j in range(n_audio_tok):
        ids.append((st + j, st + j, st + j))
    ids = np.array(ids, dtype=np.int64)
    return ids, int(ids.max()) + 1      # next modality starts here

ids, nxt = tmrope_block(st=3, secs=4, fps=2.5, grid=4, n_audio_tok=100)
print(ids.shape, nxt)               # (260, 3) 103   <- matches Part G
print(ids[3*16 + 2*4 + 1])          # [33  5  4]   <- matches Part F
print(ids[160 + 30])                # [33 33 33]   <- same instant!

One simplification to flag: that function emits all visual tokens and then all audio tokens. The real arrangement chops the block into 2-second chunks and alternates — which changes the sequence order but not a single position ID. Chapter 4 does exactly that, and you will be able to check that the IDs are unchanged.

And the rotation, applied with the three-way split. This is the shape every M-RoPE implementation has, including the one in transformers:

python — applying the three-way rotary split
def mrope(x, ids, mrope_section):
    """x: (N, d) queries or keys. ids: (N,3). mrope_section: pairs per axis."""
    d = x.shape[-1]
    i = np.arange(d // 2)
    theta = 10000.0 ** (-2 * i / d)          # (d/2,)  the frequency ladder

    axis = np.repeat([0, 1, 2], mrope_section)  # which id feeds which pair
    pos   = ids[:, axis]                        # (N, d/2)
    ang   = pos * theta                         # (N, d/2)  ID x frequency

    a, b = x[:, 0::2], x[:, 1::2]
    out = np.empty_like(x)
    out[:, 0::2] = a * np.cos(ang) - b * np.sin(ang)
    out[:, 1::2] = a * np.sin(ang) + b * np.cos(ang)
    return out

q  = np.array([[1,0,1,0,1,0,1,0]], dtype=float)
print(mrope(q, np.array([[33,5,4]]), mrope_section=[2,1,1]))
# [[-0.013277 0.999912 -0.987480 -0.157746 0.998750 0.049979 0.999992 0.004]]
# exactly the table above — you can check every entry by hand.

And the one-liner, for completeness — in practice you never write this yourself:

python — the library call
from transformers import Qwen2_5OmniForConditionalGeneration, AutoProcessor
m = Qwen2_5OmniForConditionalGeneration.from_pretrained("Qwen/Qwen2.5-Omni-7B")
# the processor builds the (3, B, T) position_ids itself from the
# interleaved inputs; rope_scaling["mrope_section"] holds the split.
out = m.generate(**proc(text=prompt, audio=wav, videos=frames, return_tensors="pt"))

Five ways to get TMRoPE wrong

Position bookkeeping is the kind of code that fails silently: nothing crashes, the loss goes down, and the model just never learns to relate sound to sight. Here are the failure modes, each with the symptom it produces.

The unifying diagnosis: every one of these comes from thinking of position as "how far along the sequence am I" rather than "when did this happen". Once the second framing is internalized, the rules stop needing to be memorized — you can re-derive each one by asking what time the token describes.

Chapter 4: The Two-Second Interleave

TMRoPE told the model when each token happened. It did not tell the model in what order to receive them. Those are different problems, and the second one only exists because of a constraint we have not yet made explicit: a causal decoder can only attend backwards.

This chapter is short in the paper — four sentences — and consequential out of all proportion to its length.

Manufacturing the need

Suppose you take the naive route from Chapter 3's code: put all 160 visual tokens first, then all 100 audio tokens. Now ask what the model can do.

The audio token at 0.2 seconds sits at sequence index 165. It can attend backwards to every video frame, including frames from 3.6 seconds — the future. Fine for offline understanding, fatal for streaming: the model cannot be run until the entire clip has been seen, because the very first audio token's context requires the last video frame to exist.

Flip it — audio first, then video — and the symmetric problem appears: the video frame at 0.0 s can attend to audio from 3.96 s, and again nothing can be computed until the clip ends.

What about strict time order — sort every token by its timestamp and interleave at the finest granularity? Now the ordering is causal and streaming works. But you have created a different problem: within each 40 ms tick you must interleave 16 visual tokens with 1 audio token, and you have shattered each video frame's spatial block into fragments separated by audio. Attention within a frame now has to reach across interruptions, and the KV-cache layout becomes a mess. Worse, at high frame rates the pattern changes shape entirely.

The rule

Quoting exactly: "we have a special design for video with audio called the time-interleaving method, which segments the representation in the video with audio into chunks every 2 seconds according to the actual time. We then arrange the visual representation at the front and the audio representation at the back within the 2 seconds, interleaving the representations of the video with audio."

Three decisions in one sentence:

1. Chunk by real time, not by token count. Two seconds of clock, whatever number of tokens that turns out to be. At 2.5 fps it is 5 frames; at 10 fps it is 20. The chunk boundary is a wall-clock boundary, so it stays aligned with the audio encoder's 2-second attention blocks (Chapter 6). These two "2 second"s are the same 2 seconds.
2. Visual first, audio second, within each chunk. Not alphabetical, not arbitrary — and Chapter 6 explains the payoff.
3. Alternate chunks. V₁ A₁ V₂ A₂ V₃ A₃ …

Applied to our 4-second worked example, the sequence order becomes:

Check the invariant promised at the end of Chapter 3: not one temporal ID moved. Frame 3 is still 33; audio token 30 is still 33. The positions encode time; the order encodes causality. Separating those two jobs is the reason both can be satisfied at once, and it is worth pausing on because most sequence models conflate them completely.

The layout is fixed. Now the two questions any reader should be asking: what does the chunk size cost, and why is the audio placed after the video rather than before it?

What does the 2 seconds actually cost?

Define staleness: for a given token, how far into the future (in real time) does some token that precedes it in the sequence reach?

Inside chunk 1, the audio token at 0.00 s sits after the video frame at 1.6 s. So that audio token can attend to visual evidence from 1.6 seconds in its own future. That is an acausality of up to one chunk — bounded, and bounded by exactly the chunk size.

Both terms are the chunk size, which is why the knob is a straight trade. Shrink to 0.5 s and you get a 0.5 s streaming delay and only 0.5 s of cross-modal lookahead; grow to 8 s and you get rich context and an 8 s delay. The paper's 2 seconds is a claim that two seconds of audio-visual context is roughly what you need to relate a gesture to a word, and that two seconds of buffering is roughly what a conversational system can absorb before it feels sluggish.

Notice, too, that within a chunk the acausality is not a bug that has to be hidden. It is exactly the property that lets a spoken "this" be interpreted using a gesture that occurred slightly later — something humans do constantly. A strictly causal ordering would forbid it.

Why visual first?

The paper does not say. Here is the reasoning that makes it the obviously right choice, offered as reconstruction rather than quotation.

Because audio is the modality that must be answered. In a voice interaction the instruction arrives as speech. The tokens closest to the model's generation point — the ones with the shortest attention path and, empirically, the strongest influence — should be the ones carrying the request. Putting audio at the back of each chunk means the last thing before the prompt continues is what was said.

Because audio should see the visuals of its own chunk. A causal decoder lets audio attend backwards to the video in the same chunk, but not vice versa. If you want "what does this refer to" resolved inside the audio representations, audio must come after the pixels. Reversing the order would force the resolution to happen later, higher in the stack, with a longer path.

Because visual tokens are the bulky, contiguous ones. Keeping all of a chunk's frames adjacent preserves the block structure the vision encoder produced and keeps the KV-cache tidy for the flash-attention paths.

Before the numbers, feel the mechanism. The sim below shows both strips at once — the world's timeline and the model's sequence — and lets you drag the boundary between them.

Build the interleave

Push the chunk size to its minimum and watch the sequence become almost perfectly time-ordered — and watch the video frames get chopped into isolated islands. Push it to maximum and you recover the all-video-then-all-audio layout with its four-second delay. Two seconds sits deliberately in the middle.

One thing left to make fully concrete: the code, and the assertion that proves the invariant.

Code: the chunked arrangement

python — time-interleaving, and the ID invariant
import numpy as np
TICK, CHUNK = 0.040, 2.0          # 40 ms per temporal id; 2 s per chunk

def interleave(st, secs, fps, grid, chunk=CHUNK):
    seq, ids = [], []
    n_chunks = int(np.ceil(secs / chunk))
    for ci in range(n_chunks):
        t0, t1 = ci * chunk, min((ci + 1) * chunk, secs)
        # ---- visual first ----
        k = int(np.ceil(t0 * fps))
        while k / fps < t1:
            t = st + int((k / fps) / TICK)             # SAME formula as Ch 3
            for r in range(grid):
                for c in range(grid):
                    seq.append(("V", k, r, c)); ids.append((t, st + r, st + c))
            k += 1
        # ---- audio second ----
        for j in range(int(t0 / TICK), int(t1 / TICK)):
            seq.append(("A", j)); ids.append((st + j, st + j, st + j))
    return seq, np.array(ids)

seq, ids = interleave(st=3, secs=4, fps=2.5, grid=4)
print(len(seq), ids.max())                       # 260 102  — unchanged

# the invariant: frame 3 and audio token 30 both carry temporal id 33,
# even though they now sit in different halves of the sequence.
i_v = next(i for i,s in enumerate(seq) if s == ("V", 3, 2, 1))
i_a = next(i for i,s in enumerate(seq) if s == ("A", 30))
print(i_v, i_a, ids[i_v][0], ids[i_a][0])       # 57 110 33 33

Run that mentally: the sequence indices changed from 57 and 190 to 57 and 110, because audio token 30 moved from the back of the whole block to the back of chunk 1. The temporal IDs did not move at all. Order and position, doing separate jobs.

What the ChatML wrapper looks like

All of this lives inside an ordinary chat template. The report gives the format, and it is worth reading because it shows how modality boundaries are marked and how video-with-audio is described in training data:

ChatML — the paper's own dataset format example
<|im_start|>user
<|vision_start|>Video.mp4 [Two people are talking in the video]<|vision_end|>What are the
people in the video saying?<|im_end|>
<|im_start|>assistant
Both pictures are of SpongeBob SquarePants. The person in the red clothes says, "Hello,
how's the weather today?" The person in the black clothes responds, "Hello, the weather
is quite nice today."<|im_end|>

Notice: the special tokens delimit the visual span, the video's audio content is part of what the assistant is expected to report, and the whole thing is just a conversation. Modality plumbing is invisible at this layer — which is the point of doing the position bookkeeping properly underneath.

Before the second example, restate the invariant one more time, because everything below is a test of it: chunking changes the order of tokens and nothing else. Position IDs come from wall-clock time; sequence indices come from the chunking. Any implementation where the two are entangled will pass the easy case and fail the ragged one.

So the exercise that follows is deliberately awkward: a clip length that is not a multiple of the chunk size, and a frame rate that does not divide the 40 ms tick rate. If the rule survives both, it is the right rule.

A second worked example, end to end

The 4-second clip divided suspiciously evenly. Do a messier one, because the messy cases are where an implementation reveals whether it understood the rule.

Counts first.

Now the chunk table. Frames land at k/3 seconds for k = 0..20. Audio token j covers [0.04j, 0.04(j+1)).

Check the frame counts. Chunk 1 holds frames with k/3 < 2, i.e. k < 6, so k = 0..5 — six frames. ✓ Chunk 4 holds 6 ≤ k/3 < 7, i.e. 18 ≤ k < 21 — three frames. ✓ Totals: 6+6+6+3 = 21 frames ✓ and 50+50+50+25 = 175 audio tokens ✓.

Sequence order.

And the position IDs, which the chunking did not touch. Frame k: t = 5 + floor((k/3) / 0.040). Spot-check a few:

Every row matches, at a frame rate (3 fps) that does not divide evenly into 25 — which is exactly the case the floor convention was chosen to handle. Had we rounded, frame 1 at 8.33 would still floor and round to 8, but a frame at, say, 0.6667 s would give 16.67, rounding to 17 while the containing audio token is 16. One tick of drift, silently, on some frames and not others.

Maximum ID and what comes next. Audio reaches 5 + 174 = 179; video reaches t = 171 and h = w = 5 + 7 = 12. So max = 179, and the next text token starts at 180. Sanity: 7 seconds ÷ 0.040 = 175 temporal ticks, occupying IDs 5 through 179. ✓

The rule is stated and tested. Now price it — first as a curve over chunk sizes, then as a memory-layout argument, then as the coupling that makes the whole pipeline work.

What the chunk size costs, at four settings

The knob has one setting in the paper and a whole curve behind it. Here is the curve, computed for our 4-second clip at 2.5 fps with 16 tokens per frame.

Notice that the delay column and the lookahead column are the same number, because they are the same quantity read from two directions. You cannot buy more cross-modal context without buying more waiting. That is the trade in its purest form, and it is why picking a chunk size is a product decision rather than a research one.

Two seconds is a defensible middle. Human conversational turn-taking gaps average roughly 200 ms, so two seconds of buffering would be fatal if it were on every turn — but it is not. It is the granularity at which input chunks become available, and the pipeline overlaps that with everything downstream (Chapter 7). Meanwhile two seconds comfortably contains a pointing gesture, a head turn, or a full spoken clause — the units of meaning that need to be related across modalities.

Three consequences remain: what the sequence layout does to the KV cache, how the chunk boundary locks to the encoder's, and how all of this is wrapped in an ordinary chat template. The template comes first, because it is where the training signal for video-with-audio instruction following actually lives.

Reading the ChatML wrapper as structure

The template in the previous section is not decoration; the control tokens are load-bearing. Three jobs are being done at once.

That third row is easy to miss and quietly important. In the paper's own example the training item contains [A person in the video is saying, "Please describe the person in front of you."] — the instruction arrives inside the video's audio, and the assistant is expected to obey it. That is the training signal for video-with-audio instruction following, and it is what makes the OmniBench result in Chapter 9 possible.

The KV-cache view: why contiguity is not an aesthetic preference

An argument that only becomes visible once you think about implementation.

A frame's 16 (or 256) visual tokens are produced together by the vision encoder and they attend to each other heavily — that is what spatial structure means. When they sit contiguously in the sequence, their keys and values sit contiguously in the KV cache, and the attention over them is one dense tile that flash attention handles at peak efficiency.

Shatter them — interleave one audio token every 40 ms, as strict time ordering would — and at 2.5 fps you get roughly this pattern:

Now every frame's self-attention is a strided gather rather than a contiguous tile. Correctness is unaffected; throughput is not. And at 448 px, where a frame is 256 tokens, the interleave pattern changes shape entirely with frame rate, so no single kernel configuration is right.

Layout is settled. What remains is the coupling: why the interleave chunk and the encoder block are the same two seconds, and what breaks if you decouple them.

How this locks together with the rest of the stack

The 2 seconds appears three times in this paper, and they are the same 2 seconds. That is worth making explicit, because it is the kind of coupling that looks like a coincidence until you try to change one of them.

Now imagine breaking the coupling: encoder blocks of 2 s but interleave chunks of 1 s. The first 1-second chunk would need audio tokens from a block that is only half-encoded. You would either stall (losing the benefit) or encode a partial block and re-encode it later (paying twice and risking a train-inference mismatch). Aligned boundaries are what make the pipeline a pipeline.

What happens at the edges

Two edge cases the report does not discuss but any implementation must:

A clip that is not a multiple of the chunk size. Our 4-second clip divides evenly into two chunks. A 4.7-second clip gives two full chunks and a 0.7-second remainder: 0.7 / 0.04 = 17 audio tokens and however many frames fall in that window. The remainder chunk is just shorter; nothing about the position rule changes, since IDs come from wall-clock time rather than from counting.

A chunk with no video frames. At 0.5 fps with 1-second chunks, half the chunks contain no frame at all. The visual segment is empty and the chunk is audio-only. Again nothing breaks — and this is precisely the case that would break a scheme based on fixed per-chunk token counts. Deriving structure from time rather than from counts is what makes the degenerate cases degenerate gracefully.

Chapter 5: Inside the Talker

Thinker is a normal LLM and you already understand it. Talker is the strange one. This chapter answers three questions: why it takes two different kinds of input, what it emits, and what "dual-track" means.

Why two inputs? The paper's own argument, unpacked

The report gives the reasoning in two sentences that repay slow reading. Here is the first:

"As a streaming algorithm, voice generation must anticipate the content's tone and attitude before the entire text is fully generated. The high-dimensional representations provided by Thinker implicitly convey this information, enabling a more natural streaming generation process."

Make it concrete. Thinker is about to produce: "Unfortunately, the capacitor is definitely blown." A streaming voice generator receiving only text sees the word "Unfortunately" and nothing else. How should it be said? Sympathetically? Briskly? The correct prosodic contour for the whole sentence depends on the whole sentence — which does not exist yet.

But Thinker's hidden state at the moment it emitted "Unfortunately" already encodes where the sentence is going, because that is how autoregressive planning works: the model computes a rich representation and then collapses it to one token. The token is the lossy part. The hidden state is the plan.

Now the second sentence, which explains why hidden states are not enough:

"Furthermore, Thinker's representations primarily express semantic similarity in the representational space rather than phonetic similarity. Consequently, even phonetically distinct words may have very similar high-level representations, necessitating the input of sampled discrete tokens to eliminate such uncertainty."

This is a precise and slightly counterintuitive claim, so let us make it painfully concrete.

The claim needs examples before it will stick, and the examples are more surprising than the claim.

The semantic-phonetic mismatch, with examples

An LLM's representation space is organized by meaning. Words that mean similar things sit close together. But how a word sounds has essentially nothing to do with what it means — that is the arbitrariness of the sign, one of the founding observations of linguistics.

So the representation space is organized along an axis that is nearly orthogonal to the one the voice needs.

Row 1 is the killer argument. If Talker had only the hidden state, then at the moment Thinker's state says "small domestic pet", Talker faces a cloud of candidates — cat, dog, kitten, puppy — that are near-identical semantically and completely different acoustically. It cannot choose. The sampled discrete token collapses the cloud to a point.

Row 3 is the mirror image, and shows why you cannot solve this by handing Talker only text: "read" is one token with two pronunciations, and only the surrounding meaning tells you which. So each input covers the other's blind spot:

The claim is now precise enough to test visually: two spaces, organized by different things, and a word that is unambiguous in one and ambiguous in the other.

Explore the two-input argument

qwen-tts-tokenizer: the vocabulary Talker writes in

Talker does not emit a waveform. It emits discrete codes from a purpose-built speech codec the paper calls qwen-tts-tokenizer, described as follows: it "efficiently represents key information of speech and can be decoded to speech streamingly through a causal audio decoder".

Two properties, both essential:

• It represents "key information". Not every acoustic detail — enough to reconstruct convincing speech. A codec that tried to encode every sample would produce token sequences too long to model autoregressively.
• It decodes causally and streamingly. This is the non-negotiable one. A codec whose decoder needs the whole sequence — a bidirectional one, or a diffusion model with global attention — forces you to wait for the utterance to finish. The causal constraint is what makes the first sound come out early.

The report does not publish the codec's frame rate, codebook count, or vocabulary size. For calibration, the neighbouring systems of the era operate roughly in the range of 12.5–75 codes per second per codebook with 1–8 residual codebooks; we will treat the rate as a slider in Chapter 7 rather than pretend to a number.

Which leaves one phrase from the report still undefined, and it is the phrase readers most often misunderstand.

What "dual-track" means

Talker is described as "a dual-track autoregressive Transformer Decoder architecture, motivated by Mini-Omni". And elsewhere: "After receiving the information, Talker starts to autoregressively generate audio tokens and text tokens."

So Talker maintains two parallel token tracks in one autoregressive process: a text track and an audio-code track. Mini-Omni's contribution (Xie & Wu, 2024) was exactly this idea — keeping a text stream alongside the audio stream so that the language-modelling signal keeps the audio generation coherent.

The intuition: generating hundreds of acoustic codes with nothing but acoustic history is a recipe for drift — the model wanders off, repeats, or trails into babble. Carrying a text token alongside gives every step an anchor in symbol space. It is the same insight as Moshi's "Inner Monologue", arrived at independently and used in a different architectural position.

All of which reduces, in code, to one small object with two inputs and two heads.

Code: the Talker step

python — one Talker step, from scratch
class Talker:
    """Dual-track AR decoder. Consumes Thinker's state + sampled token,
       emits (text_track, code_track) autoregressively."""

    def fuse(self, h_think, text_tok):
        # h_think: (B, 1, d_think)  continuous  -> tone, attitude, plan
        # text_tok: (B, 1)          discrete    -> exact lexical identity
        c = self.proj_state(h_think) + self.embed_text(text_tok)
        return c                          # (B, 1, d_talker)

    def step(self, c):
        # shares Thinker's KV history: cross-turn, cross-modal context
        s = self.decoder(self.cache, c)      # (B, 1, d_talker)
        code = sample(self.head_code(s))    # qwen-tts-tokenizer id
        txt  = sample(self.head_text(s))    # the anchoring text track
        self.cache.append(code, txt)
        return code, txt

# ABLATIONS you can reason about:
#   drop proj_state(h_think)  -> flat, context-free prosody (a TTS)
#   drop embed_text(text_tok) -> "cat" may come out as "dog"
#   drop shared cache         -> no memory of the video or the last turn

The three ablation comments are the chapter in three lines. Each input is load-bearing, and each failure mode is different in kind — which is the strongest possible evidence that the design is not over-engineered.

The argument for two inputs is made. What follows makes it precise — first as a probability model, then as the concrete constraints a speech codec has to satisfy, then as the prosody phenomenon that motivates the whole design.

Start with the probability model, because everything else in this chapter is a consequence of one term in it.

Talker as a conditional language model — the factorization

Strip away the words and Talker is a probability model. Writing down which probability makes the design decisions look like modelling decisions, which is what they are.

Let c_1..N be the speech codes for a reply and let y_1..M be the text tokens Thinker sampled. Talker models:

Read the conditioning set left to right, because each element is a design decision you can now justify:

The interesting object is m(n) — the map from "which code am I emitting" to "how far has the text got". Classical TTS fixes m by forced alignment: code n corresponds to a known millisecond, hence a known phoneme, hence a known word. Talker does not. All it is required to respect is that m is monotonic non-decreasing: as codes advance, the text position never goes backwards.

That is a strictly weaker constraint, and the paper says so in its own words: Talker "learns to establish a monotonic mapping from semantic representation to speech", and "the generation of speech does not require word-level and timestamp-level alignment with the text."

Residual codebooks, and why "dual track" is not the same thing

One clarification that trips people up, because two different kinds of "multiple streams" live in this area.

Residual quantization is a codec technique. A single continuous vector is quantized by a first codebook; the error is quantized by a second; that error by a third, and so on. Each additional codebook refines the reconstruction. So one instant of audio is represented by several codes stacked in depth, not spread in time.

More codebooks means higher bitrate and better fidelity. Truncating the stack after the first few gives you a lower bitrate for free — which is why codec-based systems can offer a bandwidth dial.

Dual-track is something else entirely: two parallel token types generated by the same autoregressive process — text and speech codes — where the text acts as a symbolic anchor for the acoustics.

The last row of the last column is the one to remember. Every time you see "dual" in this literature, ask which axis. Getting it wrong is how people come away believing Qwen2.5-Omni can be interrupted.

The factorization tells us Talker emits discrete codes, one after another, conditioned on the past. It does not tell us what those codes are. That is the codec's job, and although the paper withholds its specifications, the constraints it must satisfy are fully derivable from the streaming requirement.

Deriving them is worth doing, because it shows why a purpose-built codec was needed rather than an off-the-shelf one.

What a speech codec has to give you, quantitatively

The paper does not publish qwen-tts-tokenizer's specifications, but the constraints it must satisfy are derivable, and deriving them tells you why the component exists.

Start from the target: 24 kHz output audio, say. That is 24,000 numbers per second. An autoregressive model cannot generate 24,000 tokens per second of speech — at even 10 ms per forward pass you would need 240 seconds of compute per second of audio. So the codec must compress hard.

How hard? Work backwards from the compute budget:

Contemporary systems land between 12.5 and 75 codes per second per codebook, usually with several residual codebooks stacked. That is the window qwen-tts-tokenizer must live in, and it is why the chapter after next treats the code rate as a slider rather than asserting a number.

Compression ratio, for scale: at 25 codes per second against 24,000 samples per second, one code carries 960 samples' worth of sound. Compare Chapter 2's input side, where one audio token carried 640 samples. Roughly the same order — the input and output bottlenecks are balanced, which is not an accident.

Two loose ends remain about Talker: how the two input signals are combined, and what "dual track" does and does not mean. Neither changes the argument, but both are places where readers routinely form wrong beliefs, so they are worth closing explicitly.

Take the fusion question first. The report is silent on it, and silence in a technical report is data: it usually means the choice was not load-bearing enough to defend. That turns out to be exactly right here.

Where the two signals could be fused, and why it does not much matter

The report says Talker "receives both high-level representations and embeddings of the text tokens sampled by Thinker" without saying how the two are combined. Three options are standard; it is instructive that the argument for needing both survives all three.

Whichever is used, the load-bearing claim is unchanged: two different kinds of signal are required, because they carry complementary information. Semantic representation cannot pick phonemes; a token cannot pick a delivery. That argument is about information content, not about matrix shapes, which is why this lesson can teach it confidently while flagging the fusion operator as unstated.

The constraints are derived. Now the phenomenon they exist to serve — and it is worth feeling before it is analysed, because the argument only lands once you have heard the ambiguity in your own head.

The prosody problem, made concrete

"Prosody depends on the whole sentence" is easy to assert and easy to under-feel. Here is a sentence with two readings.

Stress any one of the seven words and the meaning changes completely: I didn't say it (someone else did); I didn't say it (I implied it); she took the money (not the jewellery). Seven readings, one string.

Now run a streaming voice generator over it. At the moment it must produce the sound for "I", the correct stress pattern depends on which of the seven meanings the speaker intends — and none of the disambiguating words have been generated yet.

A text-only TTS is stuck. It picks a default contour and hopes. A Thinker-Talker system is not stuck, because Thinker's hidden state at the "I" token already encodes which reading it is producing — the model decided that before it emitted a single word.

Two consequences of that phenomenon remain: what it means for the training data, and how you would test whether a system actually has the capability. The data question first.

Monotonic versus aligned — and what forced alignment costs

Chapter 5 already flagged that Talker learns a monotonic mapping and needs no timestamp alignment. Make the distinction sharp, because it is the difference between two eras of speech synthesis.

The weaker constraint is the better one here, and the reason is subtle. Timing is not a fact about the text; it is a choice about delivery. A forced alignment pins the model to one particular speaker's timing on one particular recording. Requiring only monotonicity lets the model choose its own pacing — which is precisely the freedom you need if the pacing is supposed to depend on the emotional context that Thinker's hidden states carry.

So the alignment-free design is not only cheaper. It is the design that makes contextual prosody possible. Removing a constraint removed a ceiling.

The mechanism is established and the data implications are drawn. One practical question closes the chapter: given a black box, how would you tell whether it has any of this?

Four tests that distinguish Talker from a TTS

If someone hands you a black box and claims it is a Thinker-Talker system rather than an LLM with a TTS glued on, here is how you would tell.

Rows 1–3 are what the architecture buys. Row 4 is the honest boundary: the handoff makes the voice context-aware, not interruptible. Those are different problems, and Moshi solved the second one while Qwen2.5-Omni solved the first.

Chapter 6: Streaming Everything

Here is the observation that organizes this chapter: a pipeline streams only if every stage streams. One blocking stage anywhere and the whole thing becomes batch. If the audio encoder needs the full clip, it does not matter how clever the decoder is.

So Qwen2.5-Omni modifies every component. This chapter walks the stack from microphone to speaker and, at each stage, asks the same two questions: what does this stage need to see before it can emit, and how was that requirement made finite?

What "streamable" means, formally

Define, for any stage, its lookahead L: the amount of future input required before output for time t can be produced. Then:

Two consequences worth stating explicitly, because they are the whole design logic:

• Full attention over a whole sequence has L = ∞. Not large — infinite, in the sense that it grows with the input. That is why every attention in the streaming path must be restricted.
• Lookahead is additive across stages but quality is not. Doubling one stage's lookahead adds delay linearly; whether it buys quality depends entirely on whether that stage's decisions actually depend on the future. This is why the paper spends lookahead budget on the DiT (where future context genuinely helps continuity) and not, say, on BigVGAN (where it does not).

Stage 1 — the audio encoder: from full attention to 2-second blocks

The report: "the audio encoder is changed from full attention over the entire audio to performing attention in blocks of 2 seconds each."

Whisper's encoder, which this one is initialized from, does bidirectional full attention over a fixed 30-second window. Every frame sees every other frame. For transcription of a recorded file that is ideal. For live audio it is a wall: nothing can be encoded until 30 seconds exist, and every new sample invalidates the whole computation.

Block-wise attention partitions the time axis into consecutive blocks and confines attention to within a block. With 40 ms frames, a 2-second block holds

Cost accounting is instructive. Full attention over an N-frame clip costs O(N²). Block attention with block length B costs O((N/B) · B²) = O(N · B) — linear in the clip length. For a 60-second clip: N = 1,500 frames, so full attention touches 2,250,000 pairs while block attention touches 1,500 × 50 = 75,000. A 30× reduction, and it is the difference between a cost that explodes with call duration and one that does not.

Stage 2 — the vision encoder

The report is briefer here: "The vision encoder utilizes flash attention for efficient training and inference with a simple MLP layer that merges adjacent 2×2 tokens into a single token. The patch size is set to 14, which allows images of different resolutions to be packed into a sequence."

Three mechanisms, each doing a different job:

Video streams naturally: a frame is a self-contained unit, so frames can be encoded as they arrive. The relevant knob is not lookahead but rate — and Chapter 2's dynamic frame rate is what turns that knob.

Stage 3 — chunked prefill in the LLM

"Chunked-prefills is a mechanism widely used in modern inference framework. To support it in modalities interation, we modified the audio and visual encoders to support block-wise attention along the temporal dimension."

Prefill is the forward pass over the prompt before any token is generated. It is one big parallel matrix operation, and for long multimodal prompts it dominates time-to-first-token. Chunked prefill splits it: process 512 tokens, then the next 512, and so on, interleaving with decoding work from other requests to keep the GPU saturated.

Here is the dependency that this chapter exists to expose. Chunked prefill requires that a prefix of the input be meaningful on its own. If the encoders used full attention over the entire clip, then no prefix of the token sequence is final — every token's value depends on audio that has not arrived. Chunking the prefill would be computing on values that are about to change.

That chain is the answer to "why 2 seconds, in both places?" — the encoder block size and the interleave chunk size are the same number because they are the same boundary. A different pair of numbers would create a partial-block state that neither mechanism knows how to represent.

Stage 4 — the sliding-window DiT, the paper's prettiest piece of engineering

Talker emits codes. Something must turn codes into a waveform, and that something is a two-step decoder: a flow-matching DiT (Diffusion Transformer) that produces a mel-spectrogram from the codes, then a modified BigVGAN that turns mel into samples.

Flow matching, in one paragraph, because you need the shape of it: instead of learning to denoise across many steps, you learn a velocity field that transports a simple noise distribution to the data distribution along straight-ish paths. At inference you integrate that field over a small number of steps. It is faster than classical diffusion and produces high-quality mel, which is why it dominated TTS around 2024–2025 (F5-TTS, CosyVoice 2, E2 TTS all use it).

The problem: a DiT is a transformer, and its default attention is global over the whole mel sequence. That is L = ∞. So the paper restricts it.

"We propose a sliding window block attention mechanism that restricts the current token's access to a limited context… we group adjacent codes into blocks and use these for our attention mask. We limit the DiT's receptive field to 4 blocks, including a lookback of 2 blocks and a lookahead of 1 block."

Count the four: 2 lookback + 1 current + 1 lookahead = 4. The current block is included in the receptive field, which is how the arithmetic closes.

Why any lookahead at all? Because speech production is anticipatory. Say "true" and "tea" out loud and feel where your lips are during the /t/: rounded for "true", spread for "tea". The consonant is physically shaped by the vowel that has not happened yet. A decoder with zero lookahead must guess, and it will guess wrong at exactly the block boundaries — producing the audible discontinuities that plague naive streaming vocoders.

One stage remains in the chain from microphone to speaker.

Stage 5 — BigVGAN, chunked

"a modified BigVGAN to reconstruct the generated mel-spectrogram back into the waveform… We also use this chunk-by-chunk method for BigVGAN's fixed receptive field to facilitate streaming waveform generation."

BigVGAN is a convolutional GAN vocoder. Convolutions have a fixed receptive field determined by kernel sizes and dilations — a known, finite number of mel frames on each side. That is already the good case: no attention, so no unbounded dependency. The only work needed is bookkeeping — feed it overlapping chunks with enough context on each side that the output is identical to what the full-sequence pass would have produced, then discard the overlap.

This is the same trick used in overlap-add convolution and in streaming CNN inference generally. It costs a little redundant compute at the boundaries and nothing in quality.

Four mechanisms, four masks. Seeing them side by side is the fastest way to internalize which ones can stream and why.

See the masks

Set lookahead to 0 and the mask becomes strictly block-causal — zero added delay, and the boundary artifacts we just discussed. Set it to 3 and the mask gets a comfortable future window at three times the delay. The paper's setting, 2 / 1, is the point where the curve of "quality per millisecond of delay" is steepest.

Finally, the masks as code — three functions, one of which is not streamable, and the arithmetic that turns a mask into milliseconds.

Code: the three masks, and why only two of them stream

python — building the masks from scratch
import numpy as np

def full_mask(n):
    return np.ones((n, n), bool)                  # lookahead = infinity

def block_mask(n, B):
    # attention confined WITHIN each block of B frames
    b = np.arange(n) // B
    return b[:, None] == b[None, :]              # lookahead <= B-1 frames

def sliding_block_mask(n, B, lookback, lookahead):
    b  = np.arange(n) // B
    db = b[None, :] - b[:, None]                  # key block - query block
    return (db >= -lookback) & (db <= lookahead)

n, B = 24, 4
m = sliding_block_mask(n, B, lookback=2, lookahead=1)   # the paper's DiT
print(m[12].sum())        # 16 = 4 blocks x 4 frames -> receptive field
print(int(np.argmax(m[12][::-1] == False)))   # future frames visible

# delay accounting, in real time:
CODE_HZ = 25                       # codes per second (illustrative)
print((B * 1) / CODE_HZ)          # 0.16 s of lookahead delay at B=4, la=1

The last two lines are the point. Lookahead is not an abstract hyperparameter; multiply blocks by block size, divide by the code rate, and you get milliseconds you can feel. Chapter 7 assembles all such terms into one ledger.

Two more things to settle before the cost arithmetic: what the block approximation genuinely gives up, and how to place any streaming system on a common set of axes. Both are the kind of framing that outlives the specific paper.

What block-wise attention actually loses

Chapter 1 argued that perception is local, so confining the encoder to 2-second blocks costs nothing. That argument is mostly right, and being precise about the "mostly" is worth a section.

Here are three real acoustic dependencies that reach further than two seconds:

The first two rows are the reason the approximation is safe: the properties that genuinely span a whole recording are also stationary, so any two-second window contains enough evidence to estimate them. Locality hurts when a property is both long-range and non-stationary, and most acoustic properties are not both.

The third row is a real, if modest, loss — and it is exactly the case the architecture recovers at the next level up. The LLM sees all the blocks in one sequence with full attention, so a prosodic contour spanning a boundary is still recoverable from the token sequence; it just is not resolved inside the encoder. That is the division of labour from Chapter 1, doing its job.

A taxonomy of streaming, for placing any system

Since "streaming" gets used for four different things, here is the vocabulary, with where Qwen2.5-Omni sits on each axis.

The third row is the one people forget. Some streaming ASR systems emit hypotheses and revise them; you have seen text change on screen as you keep speaking. Audio cannot do that — once a sample is played, it is committed. That irreversibility is why lookahead exists at all: it is the only way to be a little bit right about the future before committing.

Enough framing. The arithmetic is what makes the block-attention argument concrete, and it is short.

The cost table, worked out

"O(N·B) instead of O(N²)" is the kind of statement that slides past. Put numbers on it and it becomes a reason.

The first row is the important one: at exactly the block size, the two schemes are identical. Block attention costs nothing until the input is longer than a block — and then the saving grows without bound. That is the signature of a good approximation: free where it does not matter, decisive where it does.

The second thing to read off the table is that the block-attention column grows linearly. A ten-minute call costs twice a five-minute call, not four times. For a live system that is the difference between a cost model you can serve and one you cannot.

Three things remain in the streaming story: the asymmetry of the DiT's window, what flow matching is doing inside it, and how the vocoder gets chunked. The asymmetry first, because it is the most portable idea here.

Why lookback is nearly free and lookahead never is

The asymmetry deserves an argument rather than an assertion, because it is the single most reusable idea in this chapter.

Consider decoding block n. The lookback blocks — n−2 and n−1 — were decoded already. Their codes exist; their keys and values are sitting in a cache. Attending to them costs some arithmetic and no waiting whatsoever. The information was free; only the FLOPs are charged.

Now the lookahead block n+1. Its codes do not exist yet. Talker has to be stepped further before block n can be decoded at all. Every code of lookahead is a code the system waits for, and that wait lands directly on time-to-first-audio:

So the cost functions have completely different shapes: lookback is priced in FLOPs, lookahead in milliseconds. A rational designer therefore buys lookback until compute complains and lookahead only until the artifacts stop. Two blocks and one block is what that looks like.

The asymmetry explains the shape of the window. What it does not explain is what the DiT is doing inside it, and that turns out to matter for why a transformer is affordable here at all.

Flow matching, in five lines

The DiT is a flow-matching model, and we have used the term without unpacking it. Here is the whole idea, because it explains why the paper could afford a transformer in the vocoder path at all.

The goal is to turn a sample from a simple distribution (Gaussian noise) into a sample from a complicated one (a mel-spectrogram consistent with these codes). Classical diffusion does this with many small denoising steps along a curved, learned path. Flow matching instead learns a velocity field and defines an almost-straight path:

Work one Euler step by hand on a scalar, so the loop is concrete. Say x₀ = 0.0 (noise), and the network predicts velocity 2.4 at t = 0 with 8 integration steps, so Δt = 0.125:

Eight forward passes, not a thousand. That is the entire practical appeal: because the path is nearly straight, few steps suffice, and a transformer becomes affordable inside a real-time audio path. Every serious 2024–2025 TTS system in the comparison table — F5-TTS, E2 TTS, CosyVoice 2 — made the same bet.

One stage left in the chain, and it is the easy one — easy for a reason that is itself the lesson of this chapter.

BigVGAN, chunked — the arithmetic

The vocoder is the easy case and it is worth seeing why, because it shows what "streamable" looks like when a component was already well behaved.

BigVGAN is a stack of transposed convolutions and residual blocks with dilations. Its receptive field is fixed: a given output sample depends on a bounded window of mel frames, say R frames on each side. To produce output for a chunk of C mel frames, you feed the chunk plus R frames of context on each side and keep only the middle:

With, say, R = 16 and C = 64 that is 32 / 96 = 33% redundant compute — and output that is bit-identical to a full-sequence pass. Double the chunk to C = 128 and the waste falls to 20%. This is overlap-save convolution, a technique older than any of the models here.

The reason this component needed only bookkeeping and not an architectural change: convolutions have finite receptive fields by construction. Attention does not. That single distinction explains why the audio encoder and the DiT both needed new masks while the vocoder needed only a loop.

Chapter 7: SHOWCASE — The Handoff and the TTFA Ledger

Everything so far has been a component. This chapter assembles them and puts a stopwatch on the result.

The quantity we are measuring is TTFA — time to first audio. Not time to complete answer; time until the first sample of the reply reaches the speaker. It is the number a listener actually experiences as "responsiveness", and it is what the paper calls initial packet latency.

Start where every latency analysis should start: with a concrete request, and a list of everything that has to happen before the first sound.

Trace one turn, end to end

The user speaks a 4-second question while pointing at something on camera. Follow the data.

Two structural facts jump out of that table.

First, stages 3–6 overlap. While block 2 of the audio is being encoded, block 1 is already being prefilled. That is what chunked prefill buys: those costs are not additive, they are pipelined. Only the last chunk's encode-and-prefill sits on the critical path.

Second, the critical path is short. After the user stops speaking, what remains is: encode the final partial block, prefill it, run Thinker once, run Talker enough times to fill one block plus lookahead, run the DiT once, run BigVGAN once. Everything else already happened while the user was still talking.

With the stages enumerated and the overlaps identified, the ledger can be written down precisely.

The ledger, written out

Let us define the terms precisely, so the sim below is not a black box.

And the two structural terms that dominate when the knobs are set badly:

Work an example by hand. Suppose the codec runs at 25 codes per second, block size B = 8 codes, lookahead = 1 block. Then before the DiT can decode block 0 it needs blocks 0 and 1, i.e. 16 codes:

Now halve the block to B = 4: it needs 8 codes = 0.32 s. Halve again to B = 2: 4 codes = 0.16 s. The delay is linear in block size, which is why block size is the single most powerful knob in the whole ledger — and why the DiT's quality at small block sizes is the thing that ultimately bounds how responsive the system can be.

Everything is now in place to build the instrument. The sim below is this lesson's showcase: the full pipeline with live shapes, and the ledger those shapes determine.

The showcase

Now use it. The five experiments below are ordered so that each one isolates a different term of the ledger, and together they cover every claim the streaming design makes.

Five experiments to run in the sim

1. Turn streaming off. The ledger collapses into one enormous bar: you must generate the whole reply, then synthesize the whole waveform. TTFA becomes proportional to reply length — a 200-token answer is a multi-second silence. This is the baseline every streaming decision is measured against.
2. Set DiT lookahead to 0. Watch that term vanish from the bar. Then remember Chapter 6: this is the setting that produces audible clicks at block boundaries. Free latency, paid for in quality.
3. Push the encoder block from 2 s to 8 s. The first term grows. Beyond about 2 seconds, blocking on the encoder starts to dominate everything the decoder side achieved — which is precisely why the paper chose 2.
4. Raise the codec rate. At a fixed block size in codes, a faster codec means each block covers less time, so the lookahead delay shrinks — but the Talker must run more steps per second of speech, so its compute term grows. The two effects fight. Find where the total is minimized.
5. Raise video resolution and frame rate together. Watch the visual token count explode and the prefill term with it. This is the moment the dynamic frame rate stops being a nicety and becomes the mechanism that keeps the system usable.

The ledger is built and the shapes are derived. Two things remain: what each knob costs in quality, and the honest boundary of what the sim models at all.

What each knob costs in quality, not just milliseconds

The ledger prices everything in time. Every knob has a second price, and a design that ignores it produces a fast system nobody wants to listen to.

Two of these rows interact in a way worth noticing. Lowering the codec rate makes each code cover more time, which shrinks the lookahead delay at fixed B — but it also coarsens the audio. So the codec rate is not a latency knob at all; it is a quality knob whose latency effect is a side effect. Reading it as a latency knob leads you to a system that is fast and sounds bad.

Where the shapes come from

The shape badges are not decoration; every one is computed from a rule you have already derived. Collected here so you can verify the sim:

What the sim is not modelling

Honesty about the instrument, so you do not over-trust it.

• Batching. A real server runs many requests concurrently; your request's Thinker step may wait behind others. That is a queueing term, invisible here.
• Memory bandwidth. Per-step decode cost on an LLM is usually bandwidth-bound, not FLOP-bound, so it scales with parameter count and KV-cache size rather than with sequence length in the way the sim's linear prefill term implies.
• Cold starts. The first request to a freshly scheduled replica pays weight loading and graph compilation — often seconds.
• The tail. The sim computes one deterministic number. Real TTFA is a distribution, and the p95 is what users judge.

What the sim does model correctly is the part the paper is actually about: which terms exist, what each one depends on, and how the structural waits dominate. That is the reasoning the architecture encodes, and it is reasoning that survives all four caveats above.

Three lessons close the chapter, and none of them is specific to speech.

Reading the ledger like an engineer

Three lessons that generalize past this paper.

Latency lives in the boundaries, not the layers. Almost every term above is a waiting term — waiting for a block, a chunk, a lookahead. Only two terms (prefill and the forward passes) are genuinely compute. When people optimize a slow voice pipeline they usually reach for a smaller model; the ledger says they should first look at what everything is waiting for.

Pipelining converts sums into maxima. The reason overlapping stages 3–6 matters so much is arithmetic: serial stages add, pipelined stages take the maximum. A pipeline is only as slow as its slowest stage plus the drain, not the sum of all stages. Every "block-wise" change in this paper exists to make a stage pipelineable.

Every knob is a quality knob too. There is no free latency in this system. Encoder block size trades against acoustic context. DiT block size trades against continuity. Frame rate trades against visual detail. The engineering skill is knowing which trade is cheapest at the current operating point — which is exactly what the sim lets you feel.

Before the ledger, one visualization that makes the entire streaming argument obvious in a way the table cannot: put the stages on wall-clock time and see how much of the work happens while the user is still talking.

The turn as a Gantt chart

The table above lists stages. What it does not show is when each one runs, and that is where the whole design lives. Lay the four-second utterance on a timeline and mark what the machine is doing during it.

Read the third column of rows 3 and 4. While the user is still talking, chunk 1 is already prefilled. Its 2 seconds of encoding and its prefill cost are paid inside the user's own speaking time and never appear in TTFA at all.

Only two things remain after the user stops: the tail block (a partial one, so cheaper than a full block) and its prefill. Everything else on the critical path is generation, and generation could not have started earlier under any architecture — the model has to hear the question before it can answer it.

One consequence worth drawing out: a longer question does not mean a slower answer, up to the context limit. That is the opposite of the naive intuition, and it is only true because every input-side stage was made incremental. In a batch pipeline, TTFA scales with the length of the input; here it scales with the length of the last chunk, which is bounded by the chunk size.

Two settings, worked by hand

Do the ledger arithmetic yourself once, so the sim stops being a black box. Assume a codec at 25 codes per second and these illustrative per-stage compute costs (the shapes are the paper's; the milliseconds are ours):

Setting A — the paper-shaped defaults. Encoder block 2.0 s, DiT block B = 8 codes, lookahead 1, prompt T = 2,156 tokens.

Setting B — tuned for responsiveness. Encoder block 1.0 s, B = 4, lookahead 1, everything else identical.

A 40% reduction, and look at where it came from: 320 ms from halving the wait for codes, 125 ms from halving the encoder block. Both are waiting terms. Not one millisecond came from making anything compute faster.

Two numbers is enough to see the shape. What follows is the honest widening: what a deployed system pays that this ledger does not count.

What a real deployment adds that this ledger does not show

Be honest about the gap between a ledger and a product. Everything above is what happens inside the model. A deployed voice agent also pays:

The endpointing row often dwarfs everything the model does. A system that waits 700 ms of silence before deciding your turn ended has spent more time on that one heuristic than on the entire forward pass. This is why "our model is fast" and "the assistant feels fast" are different claims, and why turn-taking is its own research area.

It also reframes the paper's contribution correctly. Qwen2.5-Omni removes the model's contribution to latency as a bottleneck. It does not, and does not claim to, solve conversational responsiveness end to end.

Which brings us to the number the paper never gives. If you wanted to produce it honestly, here is what that would take.

A checklist for measuring TTFA yourself

Since the paper gives none, here is how you would produce the number it is missing.

1. Define the start event precisely. Last sample of user audio received? Or the endpointer's decision? These differ by hundreds of milliseconds and papers quietly choose whichever flatters them. State it.
2. Define the end event precisely. First audio sample produced, or first sample played? The playback buffer sits between them.
3. Report the distribution, not the mean. Latency distributions are heavy-tailed. The p95 is what users perceive as "this thing is unreliable"; the mean hides it.
4. Sweep the prompt length. Prefill scales with T, and T is dominated by video. A number measured on a text-only prompt tells you almost nothing about the video case.
5. Report the knobs. Block sizes, lookahead, code rate, frame rate, resolution. Without them the number is unreproducible — as this chapter's two settings demonstrate, the same model can be 680 ms or 1,143 ms depending only on configuration.
6. Separate structural from compute. Report the two sums separately. The first tells you about the design; the second tells you about the hardware. Conflating them makes both uninterpretable.

Chapter 8: The Curriculum

An architecture is a hypothesis. Training is where you find out. This chapter covers both halves of the regime: three pre-training stages for the whole model, and a separate three-stage path for Talker — which is where the most interesting technique in the paper hides.

Pre-training, stage by stage

The report: "Qwen2.5-Omni consists of three training stages. In the first stage, we lock the LLM parameters and focus exclusively on training the vision encoder and audio encoder, utilizing a vast corpus of audio-text and image-text pairs to enhance semantic understanding within the LLM. In the second stage, we unfreeze all parameters and train with a wider range of multimodal data for more comprehensive learning. In the final stage, we use data with a sequence length of 32k to enhance the model's ability to understand complex long-sequence data."

Why freeze the LLM first?

This ordering is standard and the reason is worth stating properly, because it explains what an "adapter" is for.

At initialization, the audio encoder (Whisper-large-v3) produces vectors in Whisper's representation space. The LLM (Qwen2.5) expects vectors in Qwen's embedding space. These are unrelated coordinate systems. Feed one to the other and the LLM sees noise — and the gradients it produces are also noise.

If everything were trainable at that moment, those noisy gradients would flow into the LLM and start degrading a model that cost enormous amounts to train. The freeze is a protective measure: the only thing that can change is the mapping, so the only thing that can be learned is the mapping.

Within stage 1 there is a further ordering: "both initially focusing on training their respective adapters before training the encoders". So it is a three-level thaw — adapter, then encoder, then everything.

Why 8,192 first, then 32,768?

Attention cost is quadratic in sequence length. Quadrupling the context multiplies attention cost by sixteen. Training the bulk of the run at 8,192 and only extending at the end is a straightforward compute allocation: you buy most of your learning cheaply and spend the expensive long-context budget only on the capability that requires it.

And 32,768 is not arbitrary either. Put Chapter 2's rates against it: at 448 px and 2 fps, one second of video-with-audio is 512 + 25 = 537 tokens. So 32,768 tokens is about 61 seconds of video-with-audio. A minute. That is the real meaning of the context extension — it is what lets the model hold a whole short clip.

Natural language prompts instead of tag hierarchies

A small change with large consequences: "We replace the hierarchical tags with the natural language prompts following Qwen2-Audio, which can improve better generalization ability and better instruction following ability."

The older convention (Qwen-Audio) marked tasks with structured tags — a hierarchy of special tokens saying "this is transcription, in English, with timestamps". Qwen2-Audio replaced them with plain instructions: "Transcribe the English speech." The consequence is that task specification stops being a closed vocabulary the model was trained on and becomes ordinary language the model already understands. Unseen tasks become expressible, which is exactly what "generalization" means here.

Post-training the Thinker

Brief and conventional: instruction fine-tuning in ChatML format, on "pure text-based dialogue data, visual-modality conversation data, audio-modality conversation data and mix-modality conversation data".

Four data types, and the fourth is the one that matters — conversations that mix modalities within a single exchange are the only data that teaches the model to relate what it heard to what it saw at instruction-following time.

Thinker's post-training is conventional. Talker's is not, and it is where the most transferable technique in the paper lives.

Post-training the Talker — three stages of a different kind

Here the paper gets specific and interesting. "We introduced a three-stage training process for Talker… In the first stage, we train Talker to learn context continuation. The second stage utilized DPO to enhance the stability of speech generation. In the third stage, we applied multi-speaker instruction fine-tuning to improve the naturalness and controllability of the speech responses."

Stage A — in-context learning (context continuation)

"we perform a speech continuation task through next-token prediction, leveraging an extensive dataset of dialogues that incorporate multimodal contexts and spoken responses. Talker learns to establish a monotonic mapping from semantic representation to speech, while also acquiring the ability to express speech with diverse attributes that are contextually appropriate, such as prosody, emotion, and accent."

Two things being learned at once, and they are worth separating:

• A monotonic mapping from semantic representation to speech. Monotonic means order-preserving: the n-th piece of meaning becomes the n-th stretch of sound. Not aligned to timestamps — just in order. This is the weakest possible constraint that still yields correct speech, and it is why no forced alignment is needed (Chapter 5).
• Contextually appropriate attributes — prosody, emotion, accent. Learned from context, not from labels. The training data is dialogues; the model infers how something should be said from what is being said and what came before.

Plus one defensive technique: "we implement timbre disentanglement techniques to prevent the model from associating specific voices with infrequent textual patterns." Without it, a model trained on many speakers will pick up spurious correlations — if the only person in the corpus who said "quokka" had a Scottish accent, the model learns that "quokka" is Scottish. Disentangling timbre from content breaks the shortcut.

Stage B — DPO for stability

"To broaden the coverage of speakers and scenarios, the pretraining data inevitably contains label noise and pronunciation errors, leading to model hallucinations. To mitigate this issue, we introduce a reinforcement learning phase to improve the stability of speech generation."

The paper writes the DPO objective explicitly — equation (1). Here it is with every symbol defined:

Read the objective in words. For each pair, compute how much more likely the model makes the good sample relative to the reference, and how much more likely it makes the bad one. Subtract. Push that difference up. Nothing asks the model to make the good sample likely in absolute terms — only more likely than the bad one, relative to where it started. That relative framing is why DPO does not collapse the distribution the way naive fine-tuning on "good" samples does.

Does it work? Chapter 9 has the number, and it is the cleanest ablation in the paper: on seed-tts-eval the ICL model scores 1.70 / 2.72 / 7.97 WER on test-zh / test-en / test-hard, and after the DPO stage it scores 1.42 / 2.33 / 6.54. Every set improves, and the hardest set improves most in absolute terms — which is exactly what a stability intervention should do.

Stage C — multi-speaker instruction fine-tuning

"Lastly, we performed speaker fine-tuning on the aforementioned base model, enabling Talker to adopt specific voices and improve its naturalness."

The order matters. Stability is trained before specialization, so the fine-tuned speakers inherit the robustness rather than having to relearn it. Chapter 9's single-speaker table confirms this: the speaker-tuned models keep the base model's low error rates while gaining naturalness.

Six stages across two curricula, each with its own frozen set and its own data. The sim below lets you step through them.

Explore the curriculum

The objective, the reward, and the ordering are all established. What remains is to write the loss the way you would actually run it — by hand on one pair, batched, and as a library call.

Code: the DPO loss, three ways

By hand on one pair, so you can see the numbers move:

python — one preference pair, by hand
import math
beta = 0.1
# log-probabilities of the two candidate speech sequences
lp_w, lp_w_ref = -20.0, -21.0     # winner: model likes it 1.0 nat more
lp_l, lp_l_ref = -25.0, -24.0     # loser:  model likes it 1.0 nat less

r_w = beta * (lp_w - lp_w_ref)      # 0.1 * ( 1.0) =  0.10
r_l = beta * (lp_l - lp_l_ref)      # 0.1 * (-1.0) = -0.10
margin = r_w - r_l                  # 0.20
loss = -math.log(1 / (1 + math.exp(-margin)))
print(round(margin, 3), round(loss, 4))     # 0.2 0.5981

# margin 0 -> loss = -log(0.5) = 0.6931  (no preference learned yet)
# margin 4 -> loss = 0.0181              (comfortably ordered, gradient tiny)

Then batched, as you would actually implement it:

python — batched DPO
import torch, torch.nn.functional as F

def dpo_loss(lp_w, lp_l, lp_w_ref, lp_l_ref, beta=0.1):
    """All args: (B,) summed log-probs of the full speech-code sequence."""
    margin = beta * ((lp_w - lp_w_ref) - (lp_l - lp_l_ref))
    return -F.logsigmoid(margin).mean()

# building D: no human raters needed
def make_pairs(prompts, model, asr, n=8):
    for x in prompts:
        cands = [model.sample_speech(x) for _ in range(n)]
        scored = [(wer(asr(c), x.text) + pause_err(c, x.text), c) for c in cands]
        scored.sort()
        yield x, scored[0][1], scored[-1][1]     # (x, y_w, y_l)

And the library version:

python — the one-liner
from trl import DPOTrainer
DPOTrainer(model=talker, ref_model=talker_icl, beta=0.1,
           train_dataset=pref_ds).train()

Stage 3: what actually changes when the context quadruples

"Train at 32,768 instead of 8,192" sounds like changing a constant. It is not, and the reason connects directly back to Chapter 3.

Recall RoPE's frequency ladder: pair i rotates at θ_i = 10000^−2i/d per position. The slowest pairs are the ones that encode coarse, document-scale position — and at a training length of 8,192 those pairs have only ever been rotated through a limited range of angles. Positions beyond 8,192 present the model with rotation angles it has literally never seen.

So extending the context is not a matter of allowing a longer buffer. It is a matter of teaching the model what the far end of the frequency ladder means. That requires actual training data at that length, which is why stage 3 exists as a separate stage with its own corpus of "long audio and long video data".

Three things change together:

The third row is specific to this model. In a text-only LLM, extending context stretches one position axis. Here it stretches the temporal axis of a three-axis scheme, while the h and w axes stay bounded by image dimensions. The axes are being asked to generalize by very different amounts, and only the temporal one is under real pressure.

Three questions remain about the curriculum: why an adapter is sufficient to bridge two representation spaces, why DPO rather than the alternatives, and what a preference pair actually looks like when the reward is automatic.

Why an adapter works at all — the projection argument

"Train a small adapter to align the spaces" is a sentence everyone nods at. It deserves an actual argument, because the reason it works is not obvious and the reason it sometimes fails is instructive.

The audio encoder emits vectors of some dimension d_a; the LLM consumes vectors of dimension d_m. The adapter is a learned map between them — in the simplest case a matrix W of shape (d_m, d_a) plus a bias.

Why should any linear map suffice? Because of what the two spaces have in common. The encoder's output already separates the things that matter — different phonemes, different speakers, different sound events land in different regions. The LLM's embedding space also separates the things that matter. Neither space is arbitrary; both are organized by the same underlying structure of the world. What differs is the coordinate system: which direction means what.

Changing coordinate systems is exactly what a linear map does. So the adapter's job is a rotation-and-rescale, not a re-derivation of meaning, and rotations are cheap to learn.

Where it breaks. When the encoder does not already separate what the LLM needs. Whisper's encoder was trained for transcription, so it separates phonemes beautifully — and has comparatively little reason to separate, say, a violin from a cello, or a sarcastic tone from a sincere one. No linear map can create a distinction the input does not contain.

That is exactly why stage 1 unfreezes the encoder after the adapter: to let the encoder start representing things it was never asked to represent. And it is why stage 2 unfreezes everything: to let the LLM reorganize around what it is now receiving.

Two questions remain about the Talker curriculum: why these three stages, and why in this order. The second is the more interesting one, because the ordering is doing real work.

The order of the Talker stages is an argument

ICL, then DPO, then speaker fine-tuning. Any other order would be worse, and saying why makes the design legible.

Chapter 9's single-speaker table is the evidence: the four fine-tuned speakers keep content-consistency error rates near the RL base model's while gaining naturalness. Stability was not re-earned per voice; it was inherited.

Why DPO and not something else

The stability problem is: the pretraining data has label noise and pronunciation errors, and the model hallucinates. Three families of fixes were available. Compare them on this specific problem.

The deciding factor is the fourth column of the DPO row: the reward is automatic. When you can score a sample without a human and without a learned model, the expensive parts of preference learning evaporate. DPO then reduces to a supervised loss over pairs.

Scoring a pair by hand

Make the reward concrete. Target text: "The meeting starts at four, not five." The model generates two candidates; we run an ASR system on each.

The repeated word is the classic autoregressive audio failure — the model loops. WER catches it; a naturalness metric might not, because a fluent repetition can sound perfectly natural. Pairing the two reward terms covers two different failure modes: WER catches what was said, pause-error rate catches how it was timed.

Now feed that pair into the loss from the section above. If the model currently assigns the loser a higher probability than the winner, the margin is negative, the sigmoid is below 0.5, and the loss is above 0.693 — a large gradient pushing the ordering the right way. Once the winner is comfortably preferred, the margin is large, the sigmoid saturates, and the gradient goes quietly to zero. The objective stops caring about pairs it has already fixed, which is precisely the behaviour you want from a stability intervention.

Chapter 9: Results, Limits, and What Comes Next

Chapter 0 wrote down four claims and their falsification tests. Time to collect the evidence — and then to be honest about what the paper does not show.

Claim 1: adding a mouth did not cost the eyes

The comparison is against Qwen2.5-VL-7B, the dedicated vision-language model that shares this model's vision encoder. If omni-training degraded vision, it would show here.

Mostly within a point, four benchmarks improved, worst case two points on ChartQA. And on video the picture is better still — Video-MME with subtitles 72.4 against 71.6, MVBench 70.3 against 69.6, EgoSchema 68.6 against 65.0, with only Video-MME without subtitles slightly behind (64.3 against 65.1). Grounding likewise: Refcoco val 90.5 against 90.0, and open-vocabulary detection on ODinW 42.2 against 37.3.

Claim 1 survives. Notice which benchmarks improved: RealWorldQA, MME-RealWorld, EgoSchema — the ones closest to embodied, egocentric, real-scene understanding. Plausibly the audio-visual training data helped exactly where the visual world is messy and temporal.

Claim 2: it beats the audio model it descends from

The MMAU row is the striking one. Audio reasoning — not recognition — jumps from 49.20 to 65.60 average, beating Gemini-Pro-V1.5's 54.90. Recognition improved incrementally; reasoning improved by a third. That is the signature of a better language model attached to the same ears, which is exactly what an omni model is.

Claim 3: the headline — speech instructions work like text instructions

Take pure-text benchmarks. Convert the instructions to speech. Re-run. The gap between speech-in and text-in is the price of talking to the model instead of typing.

Look at the Qwen2-Audio column first, because it is the control. 33.2 on MMLU. 18.4 on GSM8K. Speaking to that model destroyed roughly three quarters of its reasoning ability. Every "audio LLM" before this had a version of that column.

Now Qwen2.5-Omni: 65.6 and 85.4. On GSM8K the spoken version beats the text baseline it is compared against. The gap has gone from catastrophic to, on several benchmarks, small.

Why does GSM8K improve at all? A reasonable hypothesis: grade-school word problems are narrative, exactly the register speech handles well, and the 2025-era model simply has better math than the 2024-era text baseline. The comparison is across model generations as well as across modalities, and the paper does not disentangle them.

Claim 4: streaming speech output did not cost quality

Zero-shot TTS on seed-tts-eval. WER measures content consistency (did it say the right words); SIM measures speaker similarity to the prompt voice.

Three readings.

The DPO stage works. ICL → RL improves every column: 1.70→1.42, 2.72→2.33, 7.97→6.54. That last one is 1.43 points of absolute WER on the hardest set, from a preference phase built entirely on automatic rewards.

It beats the streaming competitor on the hard set. Against CosyVoice 2-S — the streaming variant, the fair comparison — 6.54 against 8.08 on test-hard. Against the non-streaming CosyVoice 2, 6.54 against 6.83. A streaming system beating a non-streaming one on the hardest set is the result this claim needed.

Seed-TTS still wins overall. 1.00 / 1.94 / 6.42, and higher speaker similarity everywhere. Qwen2.5-Omni is not the best TTS in the world; it is a very good TTS that is also a full multimodal LLM and streams. The paper says "outperforms most existing streaming and non-streaming alternatives", and "most" is doing honest work.

The single-speaker table closes the loop: after speaker fine-tuning, content consistency stays at human-adjacent levels (human recordings score 1.25 zh / 2.14 en; the four fine-tuned speakers score 1.28–1.37 zh and 1.83–2.13 en), and naturalness (NMOS) lands in the 4.48–4.62 range against human recordings at 4.51 zh / 4.46 en.

The multimodal headline: OmniBench

OmniBench requires reasoning over image, audio and text simultaneously — the capability that only exists if the alignment machinery of Chapters 3 and 4 actually works.

56.13% against a previous best of 42.91%. More than thirteen points, and against Gemini-1.5-Pro. This is the number that most directly validates TMRoPE — the benchmark is precisely "can you use two modalities at once", and the mechanism for using two modalities at once is the one thing this paper invented for perception.

The bill: what omni-training cost the text model

The paper reports this openly, which is to its credit. Qwen2.5-Omni-7B "generally falls between Qwen2-7B and Qwen2.5-7B".

Beats Qwen2-7B on most rows; loses to Qwen2.5-7B on all of them. MMLU-Pro is 9.3 points behind its text-only sibling. That is the omni tax, stated plainly. Whether it is worth paying depends entirely on whether you need the model to hear and speak.

Explore all the results

Limitations — what the paper does not show

Cheat sheet — everything worth memorizing

The cheat sheet closes the technical content. What remains is placement: where this model came from, what it points at, and which lessons on this site sit on either side of it.

Where this sits in the lineage

Connections

The last word

Strip away the benchmarks and one idea remains. For a decade, systems that spoke were assembled from stages, and every stage boundary was a place where information was thrown away and time was spent. Qwen2.5-Omni's argument is that both losses have the same cure: make the boundaries continuous and make them small. Continuous, so the plan survives the crossing — hidden states, not strings. Small, so nothing has to wait for a whole utterance — two-second blocks, four-block windows, forty-millisecond ticks.

That is why TMRoPE and the Thinker-Talker split belong in the same paper even though one is about perception and the other about generation. They are the same move, applied at the two ends of the machine.

Four claims, four verdicts. What follows is the residue — the results that did not fit the abstract but change how you read it.

Numbers that did not make the headline

A few results worth pulling out of the tables, because each says something the abstract does not.

The last row deserves a moment. It would have been easy to omit — three other CoVoST2 directions improved. Including a regression in a table you control is a small signal of good faith, and it is the sort of thing to look for when calibrating how much to trust the rest.

If you wanted to replicate or extend this

Five experiments the paper makes possible and does not run. Each is a plausible follow-up.

1. Measure TTFA properly, using Chapter 7's checklist, and publish the sweep over block sizes and lookahead. This is the missing number and it is cheap to produce.
2. Ablate the hidden-state handoff. Train a Talker on sampled tokens alone and compare on a contextual prosody benchmark — which you would first have to build, since none exists. That absence is itself the finding.
3. Sweep the interleave chunk size. The paper asserts 2 seconds without an ablation. Does OmniBench move at 1 s or 4 s? If not, the choice is free and the delay could be halved.
4. Test the TMRoPE claim directly. Construct clips where the answer depends on precise audio-visual synchrony — which sound accompanied which gesture — and compare against sequence-index positions. AV-Odyssey is a start; a synthetic, controllable version would be sharper.
5. Add duplex. Take the Thinker-Talker split and give Talker a second input stream carrying the user's live audio. Whether the split survives the addition is a genuinely open architectural question, and it is where the next generation lives.