PersonaPlex: Voice and Role Control for Full-Duplex Speech

Chapter 1: Two Knobs Nobody Could Turn

Chapter 0 asserted that role control and voice control are two independent knobs and that no system had both. That is the right picture, but it is still a picture. This chapter turns it into mechanism: what a knob physically is in each architecture family, why the duplex family has neither socket, and what it costs to add one.

Start with the word. Duplex is borrowed from telecommunications and it means exactly what it means there. A walkie-talkie is half duplex: one channel, one direction at a time, and a protocol token ("over") to hand the channel back. A telephone is full duplex: two independent channels, both live continuously, and you can talk over each other — which is not a bug, it is most of how humans actually coordinate a conversation.

Almost every voice assistant ever shipped is a walkie-talkie wearing a telephone costume. It waits for silence, decides you are done, thinks, and speaks. The "over" is implicit — a voice activity detector guessing at it — but it is there, and you can feel it. Full-duplex speech models remove it.

Four stacks, drawn honestly

The paper's related-work families differ in exactly one structural question: where does the conversation stop being audio? Follow the signal through each and both knobs will place themselves.

Stack A — cascaded ASR → LLM → TTS. Audio becomes text at hop one, stays text through the reasoning, and becomes audio again at hop three. Every knob you could want is available, because the middle of this pipeline is a text LLM with a system prompt and the end is a TTS with a speaker argument. This is why cascaded stacks still dominate enterprise deployments. The price is stated in the paper's first related-work sentence: these systems "inevitably lose paralinguistic information." The transcript is a lossy projection. Hesitation, sarcasm, a laugh, an audible sigh, the rising uncertainty in "…the Premium plan?" — none of it survives ASR, so none of it can influence the response, and none of it can be echoed back.

Stack B — streaming TTS. Not a dialogue system at all; a component upgrade. It attacks the LLM→TTS hop by starting synthesis before the text is finished. The paper cites it because it is the reason cascaded latency stopped being hopeless. It does not touch the ASR loss or the turn-taking problem.

Stack C — half-duplex speech language models (Mini-Omni, Qwen-2.5-Omni, LLaMA-Omni2, GLM-4-Voice, Freeze-Omni). The audio never becomes text: speech tokens go straight into the language model and speech tokens come out. Paralinguistics survive, because the representation is acoustic. But there is still exactly one live direction at a time, and something outside the model must decide when to switch. The paper's words: they "remain reliant on external turn-taking mechanisms, do not listen while speaking, and are limited to fixed voices." Note the paper's own footnote that its Qwen-2.5-Omni evaluation had to borrow Freeze-Omni's voice activity detector, "as none was originally provided" — the turn manager is genuinely a separate part you must supply.

Stack D — full duplex (Moshi, OmniFlatten, SyncLLM, SALM-duplex). Two audio streams, one clock, no turn manager. This is the row with the property you cannot buy elsewhere, and the row with no conditioning surface.

Why this is a taxonomy and not a spectrum

One more framing before the case study. It is tempting to line the four families up as increasing sophistication. They are better understood as answers to a single structural question: where does the conversation stop being audio, and how many directions are live?

The two columns that matter are the last two, and they move independently. Half duplex fixed the transcript bottleneck without fixing the turn manager; a cascaded stack with a very good endpointer fixes neither. Only the last row changes both, and it changes them together because they are the same change: if the model is producing output on every frame while consuming input on every frame, there is nothing left for an external component to decide.

One utterance, four stacks

Abstractions settle when you follow a single concrete input through each design. The customer says, with a hesitation in the middle:

Cascaded. The VAD hears speech, then 400 ms of silence at "about…". Depending on your threshold it either fires — and the agent interrupts with an answer to half a question — or it waits, and the customer finishes. Suppose it waits. ASR produces "Hi, I'm calling about the charge on my card from Tuesday": the hesitation is gone, the rising uncertainty on "Tuesday?" is gone, the fact that the customer sounded annoyed is gone. The LLM sees clean text and answers confidently. TTS renders it in an even, cheerful voice. Total 1.06 s. The reply is correct and slightly wrong-footed, because it is responding to a transcript rather than to a person.

Half-duplex. Same VAD decision, same interruption risk. But now the speech tokens go straight into the model, so the hesitation and the uncertainty are in the representation and can shape the answer — the model can hedge back, or slow down. What it still cannot do is say "mm-hm" during the pause to signal that it is listening, because it has one active direction. The customer gets silence and wonders if the line dropped.

Full duplex (Moshi). Every 80 ms the model consumes a user frame and emits its own. During "about… um," it can emit a short acknowledgement without claiming the floor. When the customer's pitch falls on "Tuesday?", it can begin its reply within a frame. Nothing external decided any of that. But ask it which bank this is and it does not know, because there was nowhere to tell it — and it answers in the one voice it has.

PersonaPlex. Identical to the above, except that 187 frames before the call began, someone wrote a voice sample and a role description into the agent's own channels. So the same reply comes back in the brand voice, from an agent who knows the dispute window is five business days and will not promise anything faster.

The commercial systems, and what their numbers reveal

Two closed systems sit in the paper's related work: gpt-realtime and Gemini Live. Only Gemini is benchmarked, and its numbers are a diagnostic of a design you cannot otherwise inspect.

Put together, this is the profile of a system tuned hard for responsiveness to completed turns and not at all for the collaborative behaviours that fill the gaps between them. It answers quickly and well; it does not listen in the way a person does. Whether that is the right product decision is a matter of taste, but it is clearly a decision, and Table 2 makes an otherwise opaque system legible.

The knob checklist, applied

Run each family against the three-property test from above and the landscape resolves into a single table you can hold in your head.

Read the first and last rows together. Cascaded stacks have always had every knob; that was never the problem. The problem was the four rows of Chapter 0's checklist they could not tick — latency, native interruption, paralinguistics, and the fact that a transcript is not a conversation. What PersonaPlex does is get the knobs back without giving those four up, which is the first time both halves have been available in one system.

The latency ledger, added up by hand

"Cascaded is slow" is folklore until you add it up. Do it once, with a stopwatch model of a single turn, so that every later latency number has somewhere to sit. These are typical component figures, not this paper's measurements — we mark them as ours.

Take the midpoint: roughly a second. A second of dead air after every single thing you say. Human conversational gaps cluster around 200 ms, and cross-linguistic studies put the modal gap near zero — which is only possible because listeners predict the end of your turn rather than detect it.

Now put the paper's measured number next to that. PersonaPlex's smooth-turn-taking latency in Table 2 is 0.070 s. Seventy milliseconds. That is not an improvement on the ledger; it is the deletion of four of its five rows. There is no endpointing term because there is no endpointer. There is no ASR flush because there is no ASR. There is no TTS time-to-first-audio because the model's output is audio tokens, produced one frame per frame.

What humans actually do, and why endpointing is a policy

The ledger above says the dominant cost is t_endpoint. That deserves unpacking, because it is not an engineering inefficiency you can optimise away — it is a decision problem with no good answer, and full-duplex models dissolve it rather than solving it.

Here is what a voice activity detector must do. Audio arrives; some of it is speech, some is not. After the last detected speech frame, the system starts a timer. If the timer reaches a threshold before more speech arrives, it declares the turn over. That threshold is your entire turn-taking policy compressed into one number:

There is no threshold that is right, because the correct wait depends on what was just said. "My account number is four seven…" should be given four seconds. "Thanks, bye" should be given none. A silence-duration threshold cannot know the difference, because it does not have the words — and by the time the words are transcribed, you have already paid the wait.

Humans solve this by prediction rather than detection. Conversation-analytic studies of turn-taking across many languages find modal between-turn gaps clustered near 200 ms and a long tail of overlaps — people start before you finish. That is only possible if listeners are continuously forecasting the end of your turn from syntax, prosody and content, and launching their own production in advance. Detection cannot beat 200 ms; prediction can go negative.

Write the two loops side by side and the structural difference is unmissable:

python — the cascaded turn loop
while call_active:
    audio = record_until_silence(threshold_ms=500)   # ← the policy, and the cost
    text  = asr.transcribe(audio)                    # paralinguistics die here
    reply = llm.chat(system=persona, user=text)      # knobs live here
    speaker.play(tts.synth(reply, voice=brand))    # and here
    # nothing above is listening. an interruption is lost.

python — the duplex loop
while call_active:                       # every 80 ms, forever
    u = mimi.encode(mic.read(80))         # user frame — ALWAYS consumed
    x, a = model.step(u)                  # agent text token + audio codes
    speaker.play(mimi.decode(a))          # agent frame — ALWAYS produced
    # silence is a value of `a`, not an absence of one.
    # there is no branch, no threshold, no turn variable.

Count the decisions in each. The first loop makes an explicit turn decision every iteration, in code you wrote, using a number you guessed. The second makes no turn decision at all — or rather, it makes one 12.5 times a second, inside the weights, conditioned on everything that has been said.

Knob one: what "role control" actually requires

It is tempting to think role control means "the model has some text in its context." It does not. Three properties have to hold together, and a system can have the first without the others:

1. Delivery. There must be a channel through which non-conversational text reaches the model. This is what Moshi lacks entirely.
2. Persistence. The specification must remain in force for the whole session, resisting conversational pressure. If the user says "actually you work for a different company now," a role-conditioned agent does not comply. This is why speaking the instructions fails: spoken text arrives as dialogue and is negotiable.
3. Precedence. When the role context and the model's generic priors conflict — a customer asks about appliance repair and the model knows a lot about appliance repair, but Brody Murphy works at a health insurer — the role must win. Probe Q6 in Chapter 7 exists precisely to test this.

Notice that all three are properties of training, not architecture. A channel can be carved out of the existing streams (Chapter 3). Persistence and precedence are learned by showing the model 105,410 dialogues in which the prompt was in force and the agent behaved accordingly (Chapter 5). The architecture never changes; the reading of the input does.

The three properties a "knob" must have

Before separating the two knobs, name what makes something a knob at all. A conditioning signal is only useful if all three of these hold; systems fail at different ones.

The Hybrid System Prompt satisfies all three: arbitrary text and arbitrary audio (expressive), pure prefix so zero marginal cost (cheap), and 4.48 versus 1.75 on the same architecture (binding). The third is the one that needed 2,250 hours of data; the first two came free from the design.

The paralinguistic budget

"Cascaded systems lose paralinguistic information" is the paper's one-line indictment. Make it concrete, because the size of the loss is what justifies the whole full-duplex research programme.

Speech carries at least four channels of information simultaneously. A transcript preserves one of them:

Two of the four are gone entirely, and one of the two is turn-taking intent — which is precisely the information the endpointing threshold was guessing at. The cascaded stack destroys the evidence it needs, then estimates the answer with a timer. Stated that way, the architecture's problem stops looking like a latency issue and starts looking like an information-flow bug.

Why half-duplex is not almost-full-duplex

It is tempting to see the taxonomy as a spectrum with half-duplex most of the way there. It is not; there is a discontinuity, and it has a name: double talk.

A half-duplex speech LM has a single active direction. Even a very good one, with a very good VAD, is architecturally incapable of representing the state "I am speaking and you are speaking." That state is not rare — in natural conversation, overlap is routine: backchannels land inside the other person's utterance by definition, and interruptions begin before the current speaker stops.

Count what a half-duplex model must throw away:

Now look back at Table 2 with that in mind. Qwen-2.5-Omni and Freeze-Omni are the half-duplex entries, and both have backchannel frequencies of 0.001 and 0.012 per second — that is, essentially zero. This is not a tuning failure. It is the architecture: the event they are being scored on cannot be represented in their state space. The paper's evaluation footnote is the tell — Qwen's evaluation had to borrow Freeze-Omni's voice activity detector "as none was originally provided," because a half-duplex model does not come with turn-taking; you bring your own.

Knob two: what "voice control" actually requires

The voice knob has its own three-way distinction, and papers are often sloppy about which one they deliver. The difference is entirely about what happens at enrolment time.

The third row is the one that inherits an idea from language modelling. If a model is trained to continue an audio stream, and you hand it a prefix in someone's voice, then continuing in that voice is the likeliest continuation. Voice cloning becomes a special case of in-context learning; you get it for free from a strong enough autoregressive audio model, provided the training distribution taught it that voice identity is a property that persists across a conversation. The paper's phrase for the outcome: "subsequent agent utterances are generated in the same voice, enabling zero-shot voice cloning."

And "zero-shot" is a claim about generalisation that has to be earned with data hygiene. The paper earns it: of 26,296 single-speaker voice samples drawn from VoxCeleb, Libriheavy, LibriTTS, CommonAccent and Fisher, 2,630 are reserved and used only for the speaker-similarity measurement. The 0.57 is on speakers the model never heard.

What full duplex still cannot do

Balance requires saying where the duplex family loses, and it loses in places that matter commercially. Four, and none of them are fixed by this paper.

These are the same four rows that will reappear as the "still favours cascaded" side of Chapter 10's decision table. It is a coherent picture: the transcript boundary that costs you paralinguistics is also what buys you modularity, inspection, and unbounded context. Full duplex trades all of that for timing.

Why "just fine-tune per customer" is not the answer

An engineer with a GPU budget will now object: if fine-tuning takes six hours on eight A100s, why not fine-tune one duplex model per deployment and bake the persona in? Run the arithmetic, because it is short and it decides the design.

That is for fifty roles, which is the size of the paper's evaluation set, not of a business. A contact-centre platform has thousands of tenants, each wanting their own agent name, their own product facts, their own refusal policy, updated whenever a price changes. Baking the persona into weights means a training run per price change. Conditioning at inference means a string.

There is also a deeper reason, and it is the one that makes this a research contribution rather than an ops preference. A per-persona fine-tune cannot generalise to a persona it was not trained on. A conditioned model learns the mapping from specification to behaviour, which means an unseen specification produces sensible behaviour — that is the entire point of criterion 4 from Chapter 0, and it is why the paper insists that evaluation scenarios are disjoint from training scenarios.

A glossary for the taxonomy

What has to survive

One last piece of framing before the mechanism. Fine-tuning is destructive. Moshi's conversational reflexes — the sub-100 ms turn-taking, the backchannels, the willingness to stop mid-word when interrupted — were learned from a training distribution that PersonaPlex is about to overwrite with 2,250 hours of mostly synthetic dialogue. Nothing guarantees they survive.

This is the risk that Chapter 5's data design and Chapter 4's loss design are both really about, and it is why Table 2 exists in the paper at all. A paper that only wanted to show "role adherence went up" would print Table 4 and stop. Printing the full conversational-dynamics table is a commitment to being caught if the fine-tune broke something — and, as Chapter 8 will show, it did move some numbers in the wrong direction, and the paper prints those too.

Chapter 2: Three Streams, One Clock

The plan is to write a system prompt into a conversation. Before that can mean anything, you need to know precisely what a conversation is to this model — not metaphorically, but as an array with a shape and a dtype. This chapter builds that array from the waveform up.

Everything here is inherited. PersonaPlex "follows the Moshi architecture by receiving three input streams: user audio, agent text, and agent audio," and its weights are initialised from Moshi's. So this chapter is really a compressed Moshi lesson, kept to exactly what Chapter 3 needs. If you want the full treatment — Mimi's semantic distillation, the RQ-Transformer, the inner monologue delay schedule — that is the Moshi veanor. Come back when the three streams feel solid.

Step 1: audio becomes tokens

A transformer predicts elements of a discrete sequence. Audio is a continuous pressure signal sampled tens of thousands of times a second. Something must bridge those, and that something is a neural audio codec. Moshi's is called Mimi, and the paper's Figure 1 shows it sitting underneath everything, converting in both directions.

The contract is simple to state and worth stating in shapes:

Read every symbol. T is duration in seconds and f_s the sample rate, so T·f_s is just "how many numbers are in the recording." N is the number of frames the codec produces — far fewer than samples, because each frame summarises a slice of time. Q is how many discrete codes describe one frame, and V is the codebook size, so each code is an integer index into a learned dictionary of vectors.

Why Q codes per frame instead of one? Because one index into one dictionary cannot carry enough information about 80 ms of speech. The standard answer is residual vector quantisation: quantise the frame's vector with codebook 1, take the leftover error, quantise that with codebook 2, and so on. Each successive code is a correction to the previous approximation. The first code carries the coarsest, most linguistically loaded content; later codes carry progressively finer acoustic texture.

Provenance note: the paper never prints Q. It writes only "we downweight the loss on non-semantic audio tokens" — a phrase that presupposes exactly this one-semantic-plus-rest structure. Moshi's Mimi uses Q = 8 (one semantic + seven acoustic), and since PersonaPlex initialises from Moshi's weights and leaves the architecture unchanged, we use Q = 8 throughout this lesson and flag it wherever it enters a calculation.

Why a codec at all

Before the mechanics, the motivation, because "audio must become tokens" deserves one paragraph of justification rather than being assumed.

You could, in principle, have a model predict raw waveform samples. People did: early neural vocoders modelled audio sample by sample at 16–24 kHz. The arithmetic kills it for conversation — at 24 kHz a model needs 24,000 forward passes per second of audio, and a real-time system has one second per second. Even at enormous engineering cost this leaves nothing for reasoning.

You could also have a model predict continuous features — mel spectrogram frames, say — with a regression loss. That is cheaper, and it is what classical TTS did. But regression on a multimodal distribution produces the average of the plausible outputs, which for speech sounds like a muffled blur. Discrete tokens with a cross-entropy loss let the model represent "it could be this or that" honestly, and sample from it.

So the codec earns its place twice: it compresses the rate by three orders of magnitude, and it converts a continuous generation problem into the categorical one that transformers are good at. Everything downstream — the depth transformer, the semantic/acoustic split, Chapter 4's loss weights — is a consequence of that second choice.

Residual quantisation, worked by hand

"Each successive code is a correction to the previous approximation" is easy to say and worth doing once with numbers, because the semantic/acoustic split that Chapter 4 depends on is a direct consequence of the ordering.

Take a two-dimensional toy frame vector (the real thing is hundreds of dimensions; nothing about the mechanism changes):

Codebook 1 is coarse — four widely-spaced entries:

Step 1 — find the nearest entry. Squared distances, computed term by term:

Winner: index 0. Our reconstruction so far is (1, 0), and the error is 0.412 — which is a lot.

Step 2 — quantise the residual. That residual is exactly the second row of the table: r = v − (1, 0) = (−0.1, −0.4). Codebook 2 is fine-grained, with entries near the origin:

Winner: index 1. Final reconstruction:

Two small integers — (0, 1) — now describe a continuous vector to four times the precision one integer gave. That is the whole idea, and three properties of it matter downstream:

• The codes are ordered and dependent. Code 2 is meaningless without code 1, because it names a correction to a specific point. This is why the depth transformer generates them autoregressively.
• Information is front-loaded. Code 1 captured 76% of the magnitude; code 2 the rest. Later codebooks carry ever finer detail — which is why "down-weight the non-semantic codes" is a coherent instruction rather than an arbitrary one.
• Truncation degrades gracefully. Keep the first k codes and you get a lower-fidelity but valid reconstruction. Codecs use this for variable bitrate; PersonaPlex's loss uses it as a priority ordering.

Now the bitrate, which is the number that makes the whole architecture feasible. With Q = 8 codebooks of V = 2048 entries each, at 12.5 frames per second:

Compare: raw 24 kHz 16-bit PCM is 384,000 bits per second. The codec compresses by roughly 350×, and what survives is still enough to reconstruct a recognisable, specific human voice. That compression is the reason a 7B transformer can run a conversation in real time.

Step 2: derive the clock from the paper's own numbers

Now a small piece of detective work, because the paper hands you the frame rate without ever printing it. From the experiments section: training uses "a maximum sequence length of 2048 tokens which corresponds to 163.84 seconds."

"Tokens" here means timesteps of the temporal transformer — that is, frames. So:

Check the division rather than trusting it: 12.5 × 163.84 = 12.5 × 163 + 12.5 × 0.84 = 2037.5 + 10.5 = 2048. Exact. And 163.84 itself is a suspiciously round number in disguise — 163.84 = 2048/12.5 = 2048 × 0.08, and 0.08 s at 24 kHz is 1920 samples per frame. The numbers are internally consistent with Mimi's published 12.5 Hz rate, which is a nice confirmation that we are reading "tokens" correctly.

Why 12.5 Hz and not 25, or 50

The frame rate is the single most consequential number in the architecture, so it is worth asking what would happen at other values. Every column of this table is a real trade someone had to make.

Two constraints squeeze from opposite sides. From above: the temporal transformer's forward pass must fit in one frame period on real hardware, so a 7B model cannot go much faster than 12.5 Hz. From below: turn-taking resolution. A latency of 0.070 s is a fraction of an 80 ms frame; at 6.25 Hz the frame itself is 160 ms and sub-100 ms responsiveness stops being expressible.

So 12.5 Hz is not arbitrary — it is roughly the slowest rate at which human-grade turn-taking is representable, and roughly the fastest at which a large model can keep up. The reason residual quantisation is essential is that it lets you buy back the fidelity you lose at a low frame rate by spending more codes per frame rather than more frames per second: Q = 8 codes at 12.5 Hz costs one big-model pass, where 8× the frame rate would cost eight.

Step 3: three streams, stacked

Now assemble the frame. At each of the N timesteps, the model handles three things at once. The paper's Figure 1 labels them under "Input Channels":

So one frame is a small integer array. With Q = 8:

Seventeen integers per 80 ms. That is the entire state of a conversation as far as this model is concerned. Everything Chapter 3 does is deciding what to put in those seventeen slots for the first couple of hundred frames.

Which of the seventeen does the model predict? The agent's, plus — in Moshi — the user's, which is what lets it model both sides of a conversation and is why it can anticipate turn ends. The paper's loss discussion mentions only text tokens and audio tokens with a semantic/non-semantic split, so for our purposes the targets that matter are the agent's text token and the agent's eight audio codes: nine predictions per frame.

Step 4: two transformers, because one would be too slow

Here is a problem the frame layout creates. Nine or seventeen tokens per frame, and they are not independent — the eighth acoustic code depends on the first seven, since RVQ codes are corrections to each other. A naive fix is to flatten everything into one long sequence and let a single transformer chew through it. But then your sequence length multiplies by 17, and your big model runs 17 times per 80 ms. Real time dies.

Moshi's answer, visible in the paper's figure as two stacked boxes, is a factorisation:

The economics: the expensive model runs at 12.5 Hz; the cheap model runs at 12.5×Q Hz but is small enough that this is fine. You get autoregressive dependence within the frame without paying for it in the big model.

This factorisation also explains a detail of PersonaPlex's training recipe that would otherwise look arbitrary. The paper uses two different learning rates: "the depth transformer's learning rate is 4e-6 and the temporal transformer is 2e-6." The depth transformer is small and is being asked to adapt to new voices, so it gets twice the step size; the temporal transformer holds the conversational competence you are trying not to destroy, so it moves half as fast. Differential learning rates as a form of catastrophic-forgetting insurance.

What a training batch looks like

One more level of concreteness, because "17 integers per frame" becomes a tensor the moment you train on it, and the shapes are what you will actually debug:

python — one training batch, with shapes (PersonaPlex settings)
B, N, Q = 32, 2048, 8          # batch, frames, codebooks

batch = {
    "user_audio":  # (32, 2048, 8) int64  — what the microphone heard
    "agent_text":  # (32, 2048)    int64  — word or PAD, one per frame
    "agent_audio": # (32, 2048, 8) int64  — what the agent said
    "loss_mask":   # (32, 2048)    float  — 0 on prompt frames, 1 on dialogue
}
# wall-clock represented: 32 × 163.84 s = 87.4 minutes of conversation per step
# prediction targets per step: 32 × 2048 × 9 = 589,824 cross-entropy terms
# … of which 32 × 2048 × 7 = 458,752 are non-semantic audio codes,
   carrying 0.02 weight each — see Chapter 4.

Two things to notice. First, one optimiser step sees nearly an hour and a half of conversation — which is why 24,576 steps can be sixteen passes over a 2,250-hour corpus. Second, over half a million cross-entropy terms per step, three quarters of them acoustic. When Chapter 4 says the reweighting changes the objective's shape, it is talking about redistributing those 589,824 terms, and the redistribution is why it works.

The loss_mask row is the entire interface between Chapter 3's mechanism and Chapter 4's objective. It is a vector of zeros and ones, and it is what makes the prompt a specification instead of a target.

Step 5: the inner monologue, and why it matters here

The agent-text channel deserves its own paragraph because it is the strangest part of the design and the part PersonaPlex exploits hardest.

In a cascaded stack, text is upstream of audio: you decide the words, then you synthesise them. In Moshi, text and audio are the same timeline. Frame 400 might carry the text token "Premium" and, in the same frame, the eight audio codes for the acoustic realisation of the beginning of that word. The model writes the word and speaks it together.

Two consequences follow, and both are load-bearing.

Consequence A — reasoning gets a scaffold. Predicting audio codes directly is hard; predicting a word and then predicting audio consistent with that word is easier. The text channel acts as a low-bandwidth, high-semantic-density rail that keeps the audio coherent. This is why duplex models with an inner monologue are more linguistically competent than ones without.

Consequence B — text on that channel is a commitment to speak. This is the trap Chapter 0's speculative discovery story ran into. If you write role-description tokens onto the agent-text channel and let the model behave normally, the coupling does what it always does: the model produces audio realising those words. Your system prompt gets read aloud to the customer. The fix is to break the coupling by force during the prompt segment — write the text, and hold the audio channel at silence — which is precisely the paper's sentence, and precisely Chapter 3.

Frozen versus trained — what PersonaPlex actually moves

For any composed system, the question "which parts are learning?" tells you what the model can and cannot adapt. Here the answer is unusually clean:

The 2× learning-rate split now reads as a thesis about where the new capability has to live. Role conditioning is a change in how the temporal transformer reads its prefix — a small change to a big model. Voice cloning is a change in how the depth transformer realises a frame — a bigger change to a small model. The optimiser settings encode that belief.

What happens when the inputs degrade

Edge cases reveal the design. Four, traced through the loop:

That last row is worth sitting with. "User audio" is a single channel, so the model's world contains exactly two participants. Three-way calls, a colleague in the background, a supervisor joining — none of these have a representation. It is a scoping assumption so deep in the architecture that the paper never mentions it.

Putting the data flow together

python — one inference step, as pseudocode with shapes
# state: KV cache of the temporal transformer over frames 0..n-1

def step(user_pcm_80ms, state):
    u = mimi.encode(user_pcm_80ms)        # (Q,)   int64  — user audio codes
    prev = state.last_agent_frame          # (Q+1,) int64  — text + agent codes

    h = temporal(embed(u, prev), state)     # (d_model,)  one big-model pass

    x = sample(head_text(h))              # ()     int64  — this frame's word or PAD
    a = []
    for q in range(Q):                     # Q tiny depth-transformer passes
        a.append(sample(depth(h, x, a)))    # code q conditioned on codes < q

    pcm = mimi.decode(a)                   # 80 ms of agent audio to the speaker
    return pcm, state.advance(u, x, a)

Trace the loop once more with the deployment question in mind. Where would you put a system prompt? There is no argument to step other than 80 ms of microphone audio. The only mutable thing is state — the KV cache built from previous frames. So the only way in is to have been some previous frames. That is not a hack; given this loop, it is the only door in the building.

A five-line summary of the substrate

Everything from this chapter, compressed to what Chapter 3 needs:

1. Audio becomes Q discrete codes per frame via a frozen residual-quantising codec; code 1 is semantic, 2…Q are acoustic detail.
2. One frame is 80 milliseconds; the model ticks at 12.5 Hz, derived from 2048 frames over 163.84 seconds.
3. Each frame carries three channels: user audio (Q codes), agent text (1 token, usually PAD), agent audio (Q codes).
4. A large temporal transformer runs once per frame; a small depth transformer runs Q times inside it. That factorisation is what makes real time possible.
5. The agent's text and audio are time-aligned, so text on the agent channel is a commitment to be speaking those words right now — the fact Chapter 3's mechanism has to work around.

The context budget, in seconds

One more consequence of the 12.5 Hz clock, and it will matter in Chapter 3. Context length in a duplex model is measured in time, not tokens, and the exchange rate is brutal:

Under three minutes. A customer-service call is ten. That constraint is not discussed in the paper, and it is one of the more interesting unstated limits of the whole approach — we will return to it in Chapter 10 when we cost out a deployment.

Chapter 3: The Hybrid System Prompt — showcase

This is the paper. Everything before it is setup and everything after it is evidence. Four sentences in section 3.1 describe a mechanism that adds no parameters, changes no shapes, and turns a fixed-persona model into a configurable one. We are going to spend a chapter on those four sentences.

Here they are, quoted in full, because you should have the primary text in front of you before anyone explains it:

Read it again with Chapter 2's frame layout in your head. Every noun in it is a channel you now know: agent text channel, agent audio channel, user audio channel. Every verb is an assignment into those channels for a range of frames. The mechanism is a rectangle of the token tape, filled in by hand.

Why this could not have been done before Moshi

One reason the mechanism looks obvious in retrospect and was not available earlier: it needs an architecture with all three of these, and full-duplex models are the first family to have them together.

• A text channel that is not the output. In a TTS model, text is the input and audio is the output — there is no place to put text that is about the agent rather than to be spoken. The inner monologue creates one.
• An audio channel the model produces. In an ASR model, audio is only ever input. You cannot prompt a model with a voice sample on a channel it does not generate.
• A user channel distinct from both. Without a separate user stream there is nowhere to put the 440 Hz marker, and no way to distinguish "the prompt region" from "a very quiet conversation".

Three channels, all present, all attended to, all generatable or writable as appropriate. That configuration exists in exactly one place in the 2025 landscape, which is why this paper is built on Moshi and not on anything else.

The rectangle, drawn

Lay the tape out as a grid: rows are channels, columns are frames, time runs left to right. Four regions:

Now notice the elegance, which is the reason this works at all. The two conditioning signals never collide, because they occupy different channels at different times. During the voice segment the text channel is inert; during the text segment the audio channel is inert. Neither has to compress itself into the other's representation. There is no fusion module because there is nothing to fuse — the model's own attention does the integration, exactly as it integrates any other part of its context.

Why a 440 Hz sine wave, of all things

This is the detail that looks like a joke and is actually the most instructive line in the section. The user audio channel during the prompt is not silence, and is not the real microphone. It is a pure tone at concert-pitch A.

Ask why silence would fail, and the answer falls out of Chapter 1. In a duplex model, silence on the user channel is a meaningful signal — it is the single strongest cue that it might be the agent's turn to speak. That is what the model spent its entire pretraining learning. Fill 125 frames of user channel with silence and you have written, in the model's native language, "the user has said nothing for ten seconds; you should almost certainly be talking by now." You would be fighting the conditioning mechanism with the model's own turn-taking prior.

So the filler must be (a) unmistakably not-speech, so it never gets transcribed or responded to; (b) stationary, so it carries no turn-taking dynamics — no onsets, no offsets, nothing that looks like the beginning or end of an utterance; and (c) far from anything in the training distribution of real conversations, so it can be learned as a marker rather than confused with content. A constant sine tone satisfies all three. It is the acoustic equivalent of a special token.

The paper says this filler is used "for stable conditioning" — a compressed phrase whose most likely meaning is that without it, conditioning was unstable: models talking over the prompt, starting to speak before the delimiter, or letting the prompt bleed into the dialogue. We cannot verify that from the text; we flag it as the natural reading of an ablation the page limit ate.

Delimiters: where the specification stops and the world starts

"Custom text/audio delimiters mark the boundary." Both channels get one, which is the right design and worth a sentence on why.

The model needs an unambiguous, learnable signal for "everything up to here was configuration; everything after here is a real conversation with a real person." A text-only delimiter would be invisible to the audio pathway; an audio-only delimiter would be invisible to the text pathway. Both channels are attended to, both channels carry state, and both channels change meaning at the boundary — so both channels get a marker. In the paper's figure these appear as the <system> tokens bracketing the prompt region on the text row.

This is the same job that <|im_start|>system does in a chat-tuned text LLM, and it works for the same reason: not because the token is special, but because the training data was consistent about what follows it.

Three signals for one boundary

Count how many independent cues tell the model where the specification ends and the conversation begins:

Three redundant runtime cues, on three different channels, all changing at the same frame. That redundancy is the design being careful: any one of them could be ambiguous in some conversation, and the conjunction of all three occurs exactly once per session, at a moment the training data always agreed on.

It is also why the mechanism is robust enough to work at all. The model is not being asked to infer "this text is configuration" from the text's content — a fragile, semantic judgement. It is being asked to notice a simultaneous, unambiguous, multi-channel state change. That is the easiest kind of thing for a transformer to learn.

Ordering, prefill, and the one latency insight

The paper reports something mildly surprising and then does something clever with it:

Two claims. Take them separately.

Order does not matter for quality. This is a genuine result and a slightly deflating one — you might have expected the segment closest to the dialogue to dominate, via recency. It does not. Both segments are fully attendable from every subsequent frame, so the transformer does not care which came first. That is evidence that the mechanism is really doing conditioning, not exploiting a positional artefact.

Order matters a lot for engineering. Since quality is order-invariant, you get to choose the order that serves the serving stack, and the paper chooses voice-first. Here is the reasoning, which the paper compresses into one clause and we will spell out.

The prompt region is a prefix. A transformer prefix can be turned into a KV cache once and reused, as long as nothing before it changes. Put the voice segment first, and the voice segment's cache depends on nothing — so for a deployment that uses one fixed brand voice for every call, that cache is computed once, at server start, and every session begins by loading it. Only the role text, which genuinely varies per tenant or per call, must be processed at connect time. If you had put the text first, the voice segment's cache would depend on the preceding text and could never be shared across sessions with different roles.

The paper's own phrasing — "when zero-shot voice cloning is not required" — matches this exactly: if you are not cloning a new voice, the voice segment is a constant, and constants get cached.

Reading the paper's Figure 1, label by label

The figure is dense and its labels repay a slow pass. Everything in it is now nameable:

One detail is easy to miss and is the figure's best teaching moment: the tokens after the boundary are the same kind of thing as the tokens before it. There is no mode switch, no separate encoder path, no conditioning vector entering from the side. The prompt and the conversation are the same tape.

The mechanism in one sentence

If you had to write it on a whiteboard from memory: prepend to the conversation a voice sample on the agent-audio channel with the text channel padded, then the role description on the agent-text channel with the audio channel forced silent, with a 440 Hz tone on the user channel throughout and delimiters on both agent channels at the boundary — and mask the loss over all of it.

Thirteen clauses, no new parameters, one paper.

How prompts fail

Since cloning is in-context learning, its failure modes are in-context-learning failure modes. Knowing them saves you a week of confused debugging:

Worked example: what does a persona cost?

Time to put real numbers on the rectangle. Use the paper's own example role context from Table 3 — the National Health Coverage scenario — and a ten-second reference clip.

Step 1 — the voice segment. A 10-second clip at 12.5 frames per second:

Step 2 — the text segment. Count the role context. Table 3's context is 37 words. Subword tokenisers expand text by roughly 1.3–1.6× on prose with numbers and currency in it — "076-65-0542" and "($450/month)" are each several tokens. Take 60 tokens as a fair estimate, and remember that the text channel carries one token per frame:

Step 3 — delimiters. Call it 2 frames.

Step 4 — total, and the conversion everyone forgets.

Sanity-check the middle line by hand: 12.5 × 14.96 = 12.5×14 + 12.5×0.96 = 175 + 12 = 187. Correct.

Push it to the failure point, because that is where you learn the shape of the constraint. The released checkpoint's appendix shows a "highly detailed" prompt of about 55 words — call it 90 tokens. Add a 20-second voice clip for better cloning:

Still fine. But a 500-token knowledge base in the prompt would be 40 seconds of frames — a quarter of the whole context — and this is why a duplex agent cannot simply be handed a product catalogue the way a text agent can. Retrieval and tool use, which the paper lists as future work, are not conveniences here; they are the only way past a hard wall.

Three designs they did not choose

The strongest way to appreciate a design is to price its alternatives. Every one of these would have worked in some sense; each fails a constraint the duplex setting imposes.

Notice that the winning design is the least clever of the four. That is a recurring shape in systems research: the mechanism that adds nothing usually beats the mechanism that adds something, provided the substrate already had the capacity. The intellectual work went into noticing the capacity was there.

Building the tape, in code

The whole mechanism is about twenty lines. Writing them out makes the abstraction concrete and exposes exactly which decisions are yours:

python — constructing the Hybrid System Prompt (reconstructed from §3.1)
import numpy as np

FPS = 12.5
Q   = 8

def hybrid_prompt(voice_wav, role_text, tok, mimi, voice_first=True):
    # --- voice segment -------------------------------------------------
    a_voice = mimi.encode(voice_wav)               # (Nv, Q) int
    Nv      = len(a_voice)
    x_voice = np.full(Nv, PAD)                     # text channel padded

    # --- text segment --------------------------------------------------
    x_text  = tok.encode(role_text)                 # (Nt,) int, ONE PER FRAME
    Nt      = len(x_text)
    a_text  = np.tile(SILENCE_CODES, (Nt, 1))      # audio FORCED silent

    # --- order (quality-invariant; voice first is prefill-friendly) -----
    if voice_first:
        x = np.concatenate([x_voice, x_text])
        a = np.concatenate([a_voice, a_text])
    else:
        x = np.concatenate([x_text, x_voice])
        a = np.concatenate([a_text, a_voice])

    # --- user channel: 440 Hz sine, NOT silence ------------------------
    n   = Nv + Nt
    t   = np.arange(int(n / FPS * SR)) / SR
    u   = mimi.encode(0.1 * np.sin(2 * np.pi * 440 * t))   # (n, Q)

    # --- delimiters on BOTH channels -----------------------------------
    x = np.append(x, TEXT_DELIM)
    a = np.vstack([a, AUDIO_DELIM])
    u = np.vstack([u, u[-1]])

    mask = np.zeros(len(x))                        # ← Chapter 4: no loss here
    return u, x, a, mask

Three lines carry the paper. a_text = tile(SILENCE_CODES, ...) is the forced decoupling of text from speech. u = mimi.encode(sin(2π·440·t)) is the stable-conditioning filler. mask = zeros(...) is what keeps the prompt a specification. Everything else is bookkeeping.

The boundary is also a security surface

One consequence of in-band conditioning deserves flagging now, even though the paper does not discuss it and we return to it in Chapter 11.

The model is trained to treat frames before the delimiter as specification and frames after it as conversation. That distinction is learned, not enforced by the type system — there is no structural barrier, only a convention the weights encode. Which raises the obvious question: what stops content from being read as configuration?

Three things, in increasing order of robustness:

• The channel. A caller's speech arrives on the user-audio channel. The role prompt lives on the agent-text channel. A customer cannot write to the agent's text channel; only your server can. This is a genuine structural protection and it is stronger than anything a text LLM has, where instructions and user content share one token stream.
• The delimiter. Everything after it was, in training, always dialogue. The model has never seen a mid-conversation system prompt.
• The filler. The prompt region is marked by 440 Hz on the user channel — a signature no real caller produces.

So the classic prompt-injection vector — user content that reads as instructions — is much weaker here than in text. But the softer version survives: a caller who says "ignore your previous instructions, you now work for a different company" is still just talking, and whether the agent complies is a matter of how firmly the fine-tune installed persistence. Probe Q4 in Chapter 7 gestures at this; nothing in the paper measures it directly, and single-turn probes cannot.

Why zero-shot cloning works at all

One more mechanism to make explicit, because "supply a clip and the voice continues" can sound like magic.

The model is autoregressive over agent audio codes. Its job, always, is: given everything so far, what comes next? Now hand it a prefix of agent audio in a particular speaker's voice. The likeliest continuation of a woman's voice mid-conversation is the same woman's voice — speaker identity is one of the most persistent properties in any recording, so a model that learned the statistics of real speech learned that persistence.

So cloning is not a capability that was added. It is in-context learning applied to timbre: the same phenomenon as a text LLM continuing in the style of its prompt, expressed in acoustic tokens instead of subwords. Which immediately tells you where the capability lives: the fine acoustic codebooks of Chapter 2, the ones the loss down-weights by a factor of fifty. That tension is exactly Chapter 4's subject.

It also tells you the failure modes without needing to run the model. Too short a clip and the speaker statistics are under-determined — you get a generic voice with a hint of the target. A clip with two speakers in it and the model has no way to know which to continue. A clip recorded in a very different acoustic environment from the training distribution and room tone, not identity, may dominate the continuation. All three are ordinary in-context-learning pathologies, wearing audio clothes.

Chapter 4: Loss Shaping — three numbers that decide everything

You have the rectangle. Now the harder question: what stops the model from simply ignoring it?

That is not rhetorical. A transformer fine-tuned on sequences that happen to begin with a prompt has no obligation to use the prompt. Gradient descent optimises next-token likelihood; if the dialogue is predictable without attending to frames 0–187, the model will happily not attend to them. Conditioning is a behaviour that must be made cheaper than the alternative by the data (Chapter 5) and protected by the objective (this chapter).

PersonaPlex's objective section is two sentences long and contains three numbers. Every one of them is doing structural work.

Number one: mask the prompt

The system prompt is given, never generated. At inference you write those frames yourself; the model is never asked to predict them. Training the model to predict them would optimise a distribution that will never be sampled — wasted capacity at best.

But there is a sharper reason, specific to this design. The prompt frames contain the role description on the agent-text channel. That channel is the inner monologue: the thing the model writes when it is about to speak. Train with loss on those frames and you are explicitly teaching the model that "produce the text You are an agent named Brody Murphy… on the agent channel" is a high-likelihood action. You have taught it to recite its own system prompt.

Masking is therefore not an efficiency tweak. It is the thing that keeps the prompt a specification rather than a template for output. Formally, with m_n = 0 on prompt frames and 1 on dialogue frames:

Every symbol: n indexes frames; m_n is the prompt mask; x_n is the true agent text token at frame n and x̂_n the model's predicted distribution over it; a_n,q is the true agent audio code in codebook q; CE is ordinary cross-entropy; Z is whatever normaliser your implementation uses. The two remaining sentences of the paper set w_text and w_q.

Numbers two and three: the imbalance

Count the prediction targets in one dialogue frame with Q = 8:

Leave this alone and more than three-quarters of your gradient is spent on acoustic residuals. Those codes are also, individually, the hardest to predict — they are the leftover error after seven previous approximations, closer to noise than to structure — so they contribute large, persistently non-decreasing losses. The result is a model that spends its capacity chasing acoustic dither while under-training the one token per frame that carries meaning.

The second imbalance is the text channel's sparsity. At 12.5 frames per second and roughly 3 words per second of speech, a majority of frames carry PAD. Predicting PAD is nearly free and teaches nearly nothing, but at full weight it is a large fraction of the text-channel gradient.

Hence the two weights:

Where the objective comes from

Attribution first, because it changes what counts as this paper's contribution. The paper writes: "Following Moshi, we also adjust the training objective to account for token imbalance." So the reweighting is inherited — it is Moshi's recipe for the same imbalance, carried across unchanged.

What is new here is the sentence before it: "During training, we mask out loss backpropagation to the system prompt." There is no system prompt in Moshi, so there is nothing to mask; the mask exists only because the Hybrid System Prompt exists.

That split matters when you are deciding what to reuse. If you are building a conditioning mechanism on a different duplex backbone, the weights you should take from that backbone's recipe — they are calibrated to its codebook count and its text sparsity. The mask you should take from this paper, because it belongs to the mechanism rather than to the substrate.

The two failure modes an objective must prevent

Frame the chapter before deriving it. A fine-tune of this kind can fail in two opposite ways, and the three numbers in the paper's objective section address them in turn.

Failure A — the prompt becomes output. The model learns that "produce the role description on the agent-text channel" is a high-likelihood action, and greets your customer by reciting its configuration. Prevented by the mask.

Failure B — the prompt becomes invisible. The model's gradient is dominated by targets that have nothing to do with the prompt — specifically the seven acoustic codes per frame and the mostly-PAD text channel — so the capacity that would have learned to read the prefix is spent elsewhere. Prevented by the two weights.

Notice that the two failures pull in opposite directions on the same channel. Too much attention to the agent-text channel during the prompt gives you A; too little attention to it during the dialogue gives you B. The objective has to be selective about where the text channel matters, which is exactly what a mask plus a per-token weight expresses.

What one cross-entropy term actually is

Before weighting terms, be sure what a term is. Take one prediction — say the semantic audio code of a frame — with a toy vocabulary of four:

Step 1 — exponentiate:

Step 2 — normalise:

Step 3 — the loss is the negative log of the probability assigned to the truth:

Now apply the paper's weights. If this were an acoustic code rather than the semantic one, its contribution to the total loss would be 0.02 × 0.5110 = 0.0102. Same prediction, same error, one fiftieth of the gradient. And if it were a PAD text token: 0.3 × 0.5110 = 0.1533.

That is all "weighting" means — a scalar multiplying each term before summation, which by linearity of differentiation multiplies that term's gradient contribution by the same scalar. Nothing subtler is happening. What is subtle is what those scalars do when you sum a few hundred million terms, which is the next section.

The hand-worked example — every intermediate step

Rather than trusting "it rebalances things," compute the rebalancing. We will do it in three passes: a single speaking frame, a single padded frame, then a realistic 2048-frame sequence. Only arithmetic — no calculator needed.

Pass 1 — one frame in which the agent is saying a word. Sum the weights that multiply the nine cross-entropy terms.

Now the shares. Divide each part by 2.14:

Check the long division once: 1.00/2.14 — 2.14 × 0.46 = 0.9844, remainder 0.0156; 0.0156/2.14 ≈ 0.0073; so 0.4673. And 0.4673 + 0.4673 + 0.0654 = 1.0000. Good.

Compare to unweighted: 11.1% / 11.1% / 77.8%. The acoustic share fell from 77.78% to 6.54%, a factor of 77.78 / 6.54 = 11.9×.

Pass 2 — one frame in which the agent is between words (text = PAD).

Note what happened: on a silent-ish frame the semantic audio code becomes the dominant learning signal, which is exactly right — the interesting thing about a frame with no word in it is still what sound is happening.

Pass 3 — a whole 2048-frame training sequence. Suppose 320 of the 2048 frames carry a real text token and the remaining 1728 carry PAD. (That is a ~16% word-bearing rate, consistent with an agent that speaks for part of a two-sided conversation at a few words per second; we choose it to make the arithmetic clean, and the interactive sim below lets you vary it.)

Total weighted mass, term by term:

Verify 1728 × 1.44 by hand: 1728 × 1 = 1728; 1728 × 0.4 = 691.2; 1728 × 0.04 = 69.12; total 1728 + 691.2 + 69.12 = 2488.32. Correct.

Now split that mass by target type:

Against the unweighted baseline of 11.11% / 11.11% / 77.78%. The whole objective has been turned inside out: the model now spends nearly two-thirds of its learning on the semantic code and a quarter on text, and 9% on the seven acoustic codebooks combined.

Pass 4 — the counterfactual that shows why 0.3 exists. Keep the acoustic weight at 0.02 but set the PAD text weight back to 1.0. Every frame now weighs 2.14:

Thirty-nine percent of the entire training signal spent learning to emit "no word right now." The 0.3 pulls that down to 518.4 / 3173.12 = 16.34% — still substantial, because knowing when to be silent genuinely matters in a duplex model, but no longer dominant. The choice of 0.3 rather than 0 is itself a statement: silence timing is content.

The objective, assembled

All three numbers in one place, with every symbol already defined:

Three lines of definitions under one standard cross-entropy sum. That is the entire training-objective contribution of the paper, and two of the three lines are inherited from Moshi. The new one is the mask — which is not a weight at all, but the thing that turns a prefix into a specification.

What you would watch during the run

The weights change what the total loss means, so a single scalar loss curve becomes nearly uninformative — it is dominated by whichever term carries most mass. Track the components separately, and here is what each one tells you:

The fourth row is the one that causes false alarms. An acoustic cross-entropy that stops improving looks like a broken training run to anyone used to text models. It is not: codebook 8 encodes the residual after seven approximations, which is close to white. Its irreducible entropy is high, and the loss floors there. Judge those heads by speaker similarity on held-out voices, not by their loss.

From arithmetic to numpy

Same computation, mechanically, so you can check the hand numbers:

python — step by step, no shortcuts
import numpy as np

Q          = 8
N          = 2048
n_speaking = 320
n_pad      = N - n_speaking          # 1728

w_audio = np.array([1.0] + [0.02] * (Q - 1))   # semantic, then 7 acoustic
w_word  = 1.0
w_pad   = 0.3

# --- per-frame weight totals -------------------------------
frame_speak = w_word + w_audio.sum()      # 1.0 + 1.14 = 2.14
frame_pad   = w_pad  + w_audio.sum()      # 0.3 + 1.14 = 1.44

# --- mass by target type ------------------------------------
m_text = n_speaking * w_word + n_pad * w_pad     # 838.4
m_sem  = N * w_audio[0]                          # 2048.0
m_ac   = N * w_audio[1:].sum()                   # 286.72
Z      = m_text + m_sem + m_ac                   # 3173.12

print(m_text / Z, m_sem / Z, m_ac / Z)
# 0.26422…  0.64542…  0.09036…   ← matches the hand computation

And the same thing as one expression, which is how you would actually write it inside a training step — a per-target weight vector broadcast against a per-target cross-entropy tensor:

python — the one-liner, and the real training-step form
shares = np.array([m_text, m_sem, m_ac]) / (m_text + m_sem + m_ac)

# in the actual loop: ce_text (N,), ce_audio (N, Q), mask (N,)
w_t  = np.where(is_pad, 0.3, 1.0)                       # (N,)
loss = (mask * (w_t * ce_text + (ce_audio * w_audio).sum(-1))).sum() / mask.sum()

Three forms of the same idea: pencil, explicit numpy, vectorised one-liner. If the third looks obvious now, the first two did their job.

If you could only keep one weight

A useful way to check that you understand an objective: rank its terms by how much you would miss them.

The prompt mask is not optional. Without it the model learns to speak its own system prompt, which is not a degradation but a total failure of the mechanism. Keep this first.

The 0.02 is next. Without it, 77.8% of the gradient goes to acoustic residuals and the text channel — where the entire role capability lives — is starved. You would get a model that sounds plausible and follows no instructions: Moshi, essentially, at greater expense.

The 0.3 is the one you could most nearly live without. Setting it to 1.0 wastes 39% of the objective on predicting PAD; the model still learns, just less efficiently per step. Setting it to 0 is worse than setting it to 1, because silence timing is genuine content in a duplex model — knowing when not to speak is half of turn-taking. That asymmetry is why the paper chose 0.3 rather than 0: it is a down-weight, not a deletion.

All three come from Moshi's recipe rather than being invented here — the paper says "following Moshi" explicitly. That is worth noting as a fact about the contribution: the objective is inherited, the mask and the mechanism it protects are new.

The training run, costed

The paper gives an unusually complete recipe. Adam with cosine annealing; depth-transformer learning rate 4e-6, temporal 2e-6; 24,576 steps; batch size 32; sequence length 2048 frames (163.84 s); six hours on eight A100s. Derive what that means.

Sixteen passes over the corpus. That is a lot for a fine-tune, and it is worth being honest about the caveat: mean dialogue length is 1,840×3600/105,410 = 62.8 s for service dialogues and 410×3600/39,322 = 37.5 s for QA — both well under the 163.84 s window. So sequences are almost certainly packed or padded, and the true number of distinct audio-seconds per step is lower than the window implies. The order of magnitude stands: this is many-epoch fine-tuning on a modest corpus, not a single-pass sweep.

Forty-eight A100-hours. At 2026 spot prices that is roughly the cost of a nice dinner. The reason it is so cheap is precisely the paper's thesis: nothing is being learned from scratch. Moshi already knows how to converse; the fine-tune is teaching it a new reading of a prefix.

Why cosine annealing, and why such small steps

Two optimiser choices, both stated without justification in the paper, both explicable from what the run is trying to preserve.

Cosine annealing decays the learning rate smoothly to near zero by the end of training. In a from-scratch run this mostly buys convergence stability. In a fine-tune whose main risk is forgetting, it buys something more specific: the model takes its largest steps early — when it is learning the new reading of the prefix — and its smallest steps late, when it is polishing without further disturbing the pretrained behaviour. Late-stage large steps are how you destroy conversational reflexes and never notice, because your role-adherence metric keeps improving while your turn-taking quietly rots.

4e-6 and 2e-6 are roughly three orders of magnitude below a typical pretraining rate. Combined with the ~16 effective epochs computed above, this is a deliberate profile: many gentle passes rather than few aggressive ones. The intent is unmistakable when you put the two facts side by side — the authors wanted the model to see each example many times while moving very little per example.

What each weight predicts about the results tables

Before seeing Chapter 8, you can already predict the failure directions. This is the payoff of doing the arithmetic.

The paper runs none of these ablations — four pages. But the reasoning chain from objective to metric is tight enough that you should feel confident about the directions, and it is exactly the chain you would need to debug your own version.

Chapter 5: Manufacturing 2,250 Hours of Conversation

The mechanism is free. Writing tokens into channels costs nothing; you could have done it in an afternoon. What costs is teaching the model that those tokens mean something — and that requires a corpus of a kind that does not exist anywhere in the world.

Be precise about what is needed. Not "conversations." Not "role-played conversations." You need paired data: a role specification, and a two-speaker conversation in which the agent demonstrably behaves according to that specification, recorded with realistic full-duplex timing, in a known voice, at scale, with the roles diverse enough that the model learns the mapping rather than memorising the roles.

Nothing like that is collectable. Real call-centre recordings exist, but they come without the system prompt that produced them, are legally encumbered, and their agents follow scripts you cannot reconstruct. So the paper builds the corpus. That construction is section 3.2, and it is half the work of the paper.

Why not just record real calls?

The obvious objection to 2,250 hours of synthesis is that call centres record everything. Four reasons that corpus is unusable here, and they are worth having ready because someone will ask.

The third row is the one people miss and it is fatal on its own. Stereo call recordings with per-speaker channels exist, but they are the exception. And notably, the corpus the paper eventually reaches for — Fisher, in the released checkpoint — is exactly a corpus of real conversation recorded with the speakers separable.

Half one: the transcripts, generated hierarchically

All dialogue transcripts come from two open large language models: Qwen3-32B and GPT-OSS-120B. The service half is generated top-down through a sampling hierarchy:

Why hierarchical rather than "ask the model for 100,000 customer-service dialogues"? Because a single flat prompt collapses. Ask any LLM for a hundred thousand samples of anything and you get a few dozen archetypes with the names changed — the entropy is in the prompt, and a fixed prompt has none. Sampling a domain, then a scenario, then a grounding description injects fresh entropy at three levels before generation begins. It is stratified sampling, applied to synthetic data.

The second half of the corpus is different by design: question-answering assistant scenarios, two-turn, across many topics, with an explicit taxonomy of what the second question does (topic change, follow-up, and so on). And crucially these use a fixed role:

Pause on that choice — it is not laziness. If every training example had a rich, unique role context, the model could learn "the prompt is always long and specific." A large block of data with one constant, generic role teaches that the prompt is a variable, including the case where it says almost nothing. That is what lets the released checkpoint's "Minimal" prompt tier work at all. And it keeps a reservoir of general assistant competence alive through a fine-tune otherwise dominated by customer service.

Derive the mean lengths yourself, because they say something. 1,840 h × 3,600 = 6,624,000 s, divided by 105,410 dialogues = 62.84 s. And 410 × 3,600 = 1,476,000 s over 39,322 = 37.54 s. Both comfortably inside the 163.84 s training window, which is presumably why that window was chosen: it fits a whole dialogue with room for the Hybrid System Prompt in front of it.

And note the number that should raise an eyebrow: 105,410 distinct role contexts, each an LLM-authored specification like Table 3's. That is the actual scale of the conditioning supervision. Chapter 9's ablation — cutting this to 50% and 25% — is measuring exactly how many role contexts you need to learn "follow role contexts."

Half two: transcripts become speech

Now the transcripts must be spoken, by two different people, with realistic timing, in voices you can also supply as prompts. Three sub-problems, and the paper solves them differently for the two splits.

The voice pool. 26,296 single-speaker voice samples drawn from five corpora chosen for complementary coverage:

Of these, 2,630 samples are reserved as a test set for the speaker-similarity measurement. Roughly a 10% held-out split. Without it, "zero-shot cloning" would be an unfalsifiable claim.

Service dialogue audio: Dia. The paper chose a multispeaker TTS model that "can receive two speaker samples and generate audio continuation following a transcript while cloning each voice." The reason given is precise and important: a jointly-generating model "better captur[es] timing, interruptions, and room tone."

That phrase is the whole argument for joint generation. If you synthesise speaker A and speaker B independently and lay them side by side, you get two monologues in a shared timeline. What you do not get is the interaction: the way a listener's "yeah" lands in the trough of the speaker's phrase, the way an interruption begins on a stressed syllable, the shared acoustic space that makes two voices sound like they are in one room. A duplex model trained on independently-synthesised turns would learn a caricature of conversation.

QA dialogue audio: Chatterbox, plus stitching. The QA split uses a single-speaker zero-shot cloning TTS, with a randomly chosen voice sample per role. Being single-speaker, it produces each turn separately — so an "additional audio stitching step is required" to assemble the two-sided timeline.

The negative-silence trick

Stitching sounds like a chore. It is actually where one of the paper's best ideas lives, and it is one sentence long:

Make it concrete. Let the user's turn end at time t_u and the agent's turn be placed at t_u + g, where g is the inserted gap:

One scalar, swept from positive to negative, generates the entire spectrum of turn-taking behaviour including the hardest one. And it costs nothing: no interruption-specific data collection, no annotation, no separate model. You just move a number below zero.

Do the frame arithmetic once so the overlap is not abstract. At 12.5 frames per second, g = −0.8 s means 10 frames in which the user-audio channel carries speech and the agent-audio channel carries speech. Those 10 frames are the only place a duplex model can learn what to do when it is talked over. A half-duplex model literally cannot represent them.

The paper adds one line of validation — "prior work validates our methodology" — pointing at SALM-duplex, which used the same construction. Worth knowing: this is a technique with independent support, not a one-off.

The paired-data principle

Step back from speech for a paragraph, because the shape of this chapter recurs everywhere and recognising it is worth more than the details.

Any time you want a model to obey a new kind of control signal, you need examples in which the signal was in force and the behaviour complied. Not examples of the signal. Not examples of good behaviour. Examples of the pair, consistent with each other. The mechanism for delivering the signal is usually trivial; manufacturing the pairs is the work.

In every row the "aha" looks like an architecture insight and turns out to be a data insight. That is why the paper spends half its length on section 3.2, and why a reader who skims the data section has missed most of the work.

It also predicts where the next capability comes from. Want a duplex model that calls tools? You will need dialogues in which a tool was available, was called at the right moment, and the agent said something sensible during the round-trip. Nobody has that corpus either — and manufacturing it is a harder version of exactly what Chapter 5 describes, because now the timing of the tool call is part of what must be right.

Where the diversity actually comes from

105,410 distinct role contexts is a big number for LLM-generated data, and it is worth asking whether they are 105,410 different things or one thing wearing 105,410 hats. The hierarchy is what decides that, so count its branching.

Suppose the generator samples from D domains, and within each domain from S scenario types, and then writes a grounding description before generating. The description step is where the entropy really enters, because it invents the specifics: names, account numbers, prices, policy windows. With D and S modest — say 40 domains and 12 scenario types — the structured part gives only 480 combinations, but each is expanded into hundreds of distinct groundings:

(D and S are our illustration; the paper names examples but no counts.) The structure matters more than the arithmetic: the hierarchy guarantees coverage across domains, and the free-form grounding step supplies variation within them. Flat prompting gives you neither guarantee — you get whatever the model's mode collapses to, usually a handful of archetypes.

python — the generation pipeline, as pseudocode (§3.2.1)
for _ in range(N_DIALOGS):
    domain   = sample(DOMAINS)                   # restaurant, bank, …
    scenario = sample(SCENARIOS[domain])         # refund, info request, …
    desc     = llm.gen(f"one concrete {scenario} situation at a {domain}")

    # prompt and dialogue are generated TOGETHER — this is the pairing
    role, transcript = llm.gen(f"""Write an agent role context and a
        two-speaker transcript for: {desc}. The role context must state
        the agent name, employer, verifiable facts, and policy limits.
        The agent must behave consistently with it.""")

    # two speakers, jointly synthesised → timing and room tone survive
    audio = dia.generate(transcript, spk_a=sample(VOICES), spk_b=sample(VOICES))
    emit(prompt=(role, spk_b_clip), dialog=audio)

The single most important line is the joint generation of role, transcript. Generate them separately and you get pairs that are merely plausible together; generate them in one pass and the transcript is conditioned on the role, which is the property the model must learn. Consistency by construction, not by filtering.

The second-question taxonomy

The QA half deserves one more look, because it contains a quiet piece of design. The paper says these are two-turn dialogues "across various topics and second-question scenarios (topic change, follow up etc.)."

Why enumerate what the second question does? Because the second turn is where conversational state either survives or dies, and different second-question types stress different machinery:

A model trained only on single-turn exchanges learns none of these; a model trained on undifferentiated multi-turn data learns them unevenly, because natural corpora are dominated by follow-ups. Enumerating the types is stratified sampling again, applied one level down.

The voice pool, and what each corpus is for

26,296 samples from five corpora is not a convenience sample — the five cover complementary axes of variation, and the coverage is what makes zero-shot cloning generalise rather than memorise.

Note the held-out arithmetic: 2,630 of 26,296 is exactly 10%, reserved for measuring speaker similarity. So the 0.57 is on speakers the model never trained on — which is what makes "zero-shot" a claim rather than a slogan. If those samples had been in training, 0.57 would measure memorisation and would tell you nothing about your brand voice.

And the appendix's pivot is instructive precisely because it changed this table. Replacing all of it with TortoiseTTS voices, pitch- and formant-augmented in Praat, trades natural speaker variation for synthesised speaker variation. That could easily have hurt — synthetic timbres might not span the space real voices occupy. It did not; speaker similarity rose to 0.65. The most likely explanation is the other change made in the same breath: unifying on Chatterbox for "superior speaker consistency" meant the training targets were more internally consistent, and consistency of the target mattered more than realism of the voice pool.

Is synthetic data a cheat?

Worth confronting directly, because "trained on LLM-generated dialogues voiced by TTS models" invites suspicion, and some of that suspicion is earned.

The case for. You get the one thing real data cannot give you: the pairing. Every dialogue comes with the specification that produced it, guaranteed consistent because they were generated together. You also get controllable coverage — you decide how many domains, how many refusal scenarios, how many barge-ins — and you get privacy by construction. And it evidently works: role adherence goes from 1.75 to 4.48.

The case against, honestly. Three risks, none of which the paper measures.

1. Distributional narrowness. LLM-written customer-service dialogue is tidier than the real thing. Real callers mumble, change their minds mid-sentence, have children in the background, and produce disfluent repairs that no language model spontaneously writes.
2. TTS artefacts become learning targets. The model is trained on Dia's and Chatterbox's output. Any systematic quirk of those synthesisers — prosodic flatness, characteristic breath placement — is signal as far as the loss is concerned.
3. Judge-generator alignment. Training transcripts come from Qwen3-32B and GPT-OSS-120B; role adherence is scored by GPT-4o. These are different models, which helps, but they are all trained on overlapping web text and share notions of what a "good" service reply looks like. Some of the 4.48 is agreement about style, not correctness.

The paper does not rebut these. But the appendix does something better than a rebuttal: it shows the authors hit the first one and fixed it. The released checkpoint adds 7,303 real conversations (1,217 hours) from Fisher — genuine, disfluent, unscripted telephone speech — specifically "to improve natural backchanneling, expressions, and emotional responses." That is a direct admission that the synthetic corpus was too clean, and Chapter 9 shows the measured consequences.

What the released checkpoint changed about this chapter

Everything above describes the experimental corpus. The shipped one is different in ways that reverse two of this chapter's design decisions, and the reversals are instructive.

Read as a lesson rather than a changelog: the first version of the corpus optimised for realism of the generation process (joint multispeaker synthesis) and the second optimised for consistency of the targets plus realism of a real corpus. The second decomposition is better. If you need timing realism, take it from recordings of humans; if you need clean supervision, synthesise it. Trying to get both from one synthesiser was the compromise they eventually abandoned.

Two numbers worth deriving yourself

The corpus statistics are given as totals, and totals hide the shape. Two quick derivations:

Voice reuse. 26,296 voice samples minus 2,630 held out leaves 23,666 for training, spread across 144,732 dialogues, each needing two speakers:

Twelve times each, in twelve different roles and scenarios. That is exactly the structure you want: the model sees each timbre paired with many different contents, so it cannot bind a voice to a script — it has to learn that voice identity is an independent, prompt-controlled property.

Corpus balance. Service is 81.8% of hours but only 72.8% of dialogues (105,410 of 144,732), because service dialogues are longer (62.8 s versus 37.5 s). So the QA half is over-represented per example and under-represented per second. Since the loss is summed over frames, the effective weighting is the hours figure — the fine-tune is four-fifths customer service, which is the most likely explanation for Chapter 7's small negative grounding gap.

What the corpus does not contain

A corpus is defined as much by its absences, and three of these explain results you will meet later.

• No tool calls. Not one dialogue in which the agent looks something up. This is why tool use is future work rather than a fine-tuning exercise — the behaviour has no supervision anywhere in 144,732 examples.
• No conversations longer than the window. Mean dialogue length is 62.8 s against a 163.84 s window, so the model has never seen what happens when context runs out. Chapter 10's ten-minute call is entirely off-distribution.
• No adversarial users, beyond rudeness. The service dialogues are LLM-written and cooperative. Nobody tries to talk the agent out of its role, which is why persistence is untested and why Chapter 7 lists it as the natural next benchmark extension.

The stitching algorithm, written out

The negative-silence trick is four lines, and writing them makes the frame-level consequence unmissable:

python — stitching two single-speaker turns into a duplex timeline
FPS, SR = 12.5, 24000

def stitch(user_wav, agent_wav, gap_s):
    # gap_s > 0 → silence between turns; gap_s < 0 → OVERLAP
    start = len(user_wav) + int(gap_s * SR)
    n     = max(len(user_wav), start + len(agent_wav))

    u = np.zeros(n); u[:len(user_wav)]              = user_wav
    a = np.zeros(n); a[start:start + len(agent_wav)] = agent_wav
    # ← the two channels stay SEPARATE. never mix them down.
    return u, a

# at gap_s = -0.8 the two channels are simultaneously non-zero for
#   0.8 * 12.5 = 10 frames — the only place barge-in can be learned.

The comment that matters is "never mix them down." A duplex model needs the user and agent streams as separate channels, because the whole point is that it observes one while producing the other. Summing them into a single waveform — the obvious thing to do if you are thinking about audio files rather than about the model — destroys exactly the information the architecture exists to use.

Detecting synthetic-data collapse before it costs you a run

The risks listed above are not hypothetical, and none of them announce themselves in the training loss. Four cheap checks, worth running on any synthetic corpus:

That last one is the sharpest, because it connects directly to Chapter 9. The released checkpoint's headline improvement is pause handling, and the reason 1,217 hours of real telephone speech fixed it is that real gap distributions have a shape synthetic ones do not.

And the privacy pivot

One more appendix decision that belongs in this chapter. The experimental model's voices came from real people — VoxCeleb celebrities, LibriTTS readers, Fisher callers. The released model's do not: "for data privacy, all synthetic dialogs use synthetic voices sampled from TortoiseTTS rather than real voice datasets," pitch- and formant-augmented with Praat "to cover a wide variety of timbres."

Two things follow. First, this is a real-world compliance constraint appearing inside a research paper, and it is refreshingly explicit — shipping a voice-cloning model trained on identifiable people's voices is a liability, and they chose not to. Second, it improved the result: switching everything to Chatterbox for "superior speaker consistency" raised speaker similarity from 0.57 to 0.65. The privacy-preserving choice was also the better engineering choice, which is not the usual shape of that trade.

Chapter 6: How Do You Grade a Conversation?

Suppose someone hands you two duplex speech models and asks which is better. What do you measure?

Word error rate tells you nothing — neither model is transcribing. Perplexity is not comparable across different token vocabularies. You could ask people, but human evaluation is slow and expensive and you need it for the final claim, not for every experiment. And the thing you actually care about — does talking to it feel like talking to a person — is not a property of any single utterance. It is a property of timing.

This is the gap Full-Duplex-Bench fills, and PersonaPlex leans on it hard enough that you cannot read the results without understanding it. This chapter builds it from scratch.

Why the existing benchmarks could not be used

The paper lists four predecessors — VoiceBench, VoxDialogue, VoxEval, URO-Bench — and dismisses them in one clause: they "focus on single-turn responses and enforce a half-duplex turn-by-turn interaction."

Sit with why that is fatal rather than merely limiting. A half-duplex evaluation harness plays the user's audio, waits, then records the model's answer. That protocol cannot represent the events that distinguish full-duplex models: it never plays audio while the model is speaking, so there are no interruptions; it never pauses mid-utterance, so there is no pause handling; it decides turn boundaries for the model, so turn-taking is not being tested at all. Evaluating Moshi with such a harness measures its question answering, which is not why anyone built it.

Full-Duplex-Bench changes the protocol first and the metrics second: "user input audio is streamed in and the model's corresponding generated audio is captured and evaluated." The model runs in real time against a scripted audio stream, and everything it emits — including silence, including talking over the user — is on the record.

The four situations

Full-Duplex-Bench scripts four conversational situations. In each, there is an obviously right behaviour and an obviously wrong one, and the metrics simply count which happened.

Two of these use the same statistic with opposite desired directions, which is the single most important thing to internalise about this benchmark.

TOR, defined and computed

Takeover rate is the fraction of trials in which the model took the conversational floor — began producing speech (not silence, not a short backchannel) in the window the situation defines.

Compute one by hand. Imagine 25 pause-handling trials. In 14 of them the model started talking during the user's pause:

That is the whole computation. Its simplicity is a feature: no judge, no embedding model, no scoring rubric, nothing to argue about. A binary event, counted.

And now the trap. Here is PersonaPlex on all three TOR columns of Table 2:

Backchannel: frequency and timing

Backchannelling is the hardest of the four to measure, and the benchmark uses three statistics because no one of them suffices.

TOR (↓) catches the gross failure: mistaking "I should acknowledge" for "I should take over." Gemini's 1.000 here is spectacular — it takes the floor in every single backchannel trial.

Frequency (↑) counts backchannels per unit time. Look at the scale: PersonaPlex 0.025, Moshi 0.015, Freeze-Omni 0.012, Gemini and Qwen 0.001. These are events per second, so 0.025 is one "mm-hm" every 40 seconds, and Gemini's 0.001 is one every ~17 minutes. Effectively: three of the five models do not backchannel at all.

JSD (↓) is the subtle one. Frequency alone is gameable — a model that says "mm-hm" every two seconds would score wonderfully and be unbearable. So the benchmark also compares the distribution over when backchannels occur against the human distribution, using Jensen–Shannon divergence.

Jensen–Shannon divergence, derived and hand-computed

Build it rather than quoting it. You have two distributions over the same bins: p, when the model backchannels, and q, when humans do. You want one number for "how different."

The natural first try is Kullback–Leibler divergence:

Two flaws make it unusable here. It is asymmetric — KL(p‖q) ≠ KL(q‖p), so "how different are these two models' timings" would depend on which you wrote first. And it is unbounded: if the human distribution has zero mass in a bin where the model has some, you divide by zero and get infinity. With sparse events over discrete time bins, empty bins are guaranteed.

The fix is to compare each distribution not to the other but to their average:

Both problems evaporate. Symmetric by construction, since swapping p and q swaps two terms of a sum. And finite always: m_i is zero only if both p_i and q_i are, in which case the term is zero by convention. With log base 2, JSD lies in [0, 1] — 0 for identical distributions, 1 for distributions with disjoint support.

Hand-worked, every step. Four bins of position-within-the-user's-utterance: early, early-mid, late-mid, late.

Step 1 — the mixture, bin by bin:

Step 2 — KL(p ‖ m), term by term. Ratios first, then logs, then products.

Where do those logs come from without a calculator? log₂(0.4) = log₂(4) − log₂(10) = 2 − 3.321928 = −1.321928. log₂(0.8) = log₂(8) − log₂(10) = 3 − 3.321928 = −0.321928. log₂(1.75) = log₂(7) − log₂(4) = 2.807355 − 2 = 0.807355. All three from the two facts log₂10 = 3.321928 and log₂7 = 2.807355.

Step 3 — KL(q ‖ m):

Step 4 — average them:

Interpret it. 0.359 on a [0,1] scale is a real mismatch — this model backchannels far too early — but nothing like the 0.997 of a model that does not backchannel at all. PersonaPlex's actual 0.649 sits between: it backchannels, at somewhat human-like times, imperfectly. And it is the best of the five, which tells you how far the field has to go.

python — the same computation, three ways
import numpy as np

p = np.array([0.7, 0.2, 0.1, 0.0])
q = np.array([0.1, 0.3, 0.4, 0.2])

# --- 1. explicit, mirroring the tables above -----------------
m  = (p + q) / 2
def kl(a, b):
    t = 0.0
    for ai, bi in zip(a, b):
        if ai > 0:                       # 0·log0 = 0
            t += ai * np.log2(ai / bi)
    return t
jsd = 0.5 * kl(p, m) + 0.5 * kl(q, m)   # 0.35935…

# --- 2. vectorised ------------------------------------------
def kl_v(a, b):
    return np.sum(np.where(a > 0, a * np.log2(np.where(a > 0, a, 1) / b), 0.0))
jsd = 0.5 * kl_v(p, m) + 0.5 * kl_v(q, m)

# --- 3. the library one-liner (note: returns the DISTANCE) ---
from scipy.spatial.distance import jensenshannon
jsd = jensenshannon(p, q, base=2) ** 2       # ← square it: scipy gives sqrt(JSD)

That last gotcha costs people an afternoon. scipy.spatial.distance.jensenshannon returns the Jensen–Shannon distance, which is the square root of the divergence. Forget the square and you will report 0.599 where the answer is 0.359.

Four situations, one question

Before the metric definitions, notice what unifies the four categories. Each presents the model with a moment of silence or overlap and asks the same underlying question — is it my turn? — with a different correct answer:

Four cells, four different answers, and the acoustic evidence in two of them is nearly identical: a pause mid-utterance and a pause after a completed turn are both silence. Distinguishing them requires understanding what was said and how it was said — syntactic completeness, falling versus level pitch, whether the last clause resolved. That is the competence the benchmark is really probing, and it is why a silence-threshold VAD can never do better than guess.

Why pause handling is measured twice

Table 2 has two pause columns: "Pause (Synthetic)" and "Pause (Candor)". Two datasets for one behaviour is a design choice worth reading.

Synthetic pauses are constructed: take an utterance, insert a silence of controlled duration at a controlled point. That gives you a clean, parameterised probe — you know exactly how long the pause is and exactly where it falls.

Candor pauses are real: taken from a corpus of recorded human conversation. Messier, uncontrolled, and distributed the way pauses actually are — clustered at clause boundaries, varying with speaking rate, often filled rather than silent.

Now compare a model across the two and the difference is diagnostic:

PersonaPlex's +0.078 is small but it points the same direction as everything in Chapter 9: a model fine-tuned on synthetic timing is slightly out of distribution on human timing, and adding real conversation is what closes the gap. Having both columns is what lets you see that at all.

Latency, and the interruption you can trigger

Latency in Full-Duplex-Bench is the delay from the moment the model should respond to the moment it produces speech. Two situations use it and they mean subtly different things: after a completed user turn (smooth turn taking) and after a barge-in (user interruption). The second is harder, because the model must first stop.

That stopping is the part a cascaded stack cannot do at all and a half-duplex model does with a bolted-on VAD. A full-duplex model does it natively: the user's audio is arriving on a channel the model attends to, every frame, including the frames in which it is speaking. It can notice mid-word.

Play with Moshi in that sim and you will hit the paper's most instructive oddity: Moshi's interruption latency is 0.257 s, the fastest in the table — faster than PersonaPlex's 0.400 s. And Moshi's GPT-4o content score after the interruption is 0.765 against PersonaPlex's 4.210. Moshi answers in a quarter of a second, and what it says is worth almost nothing.

How precise are these numbers, really?

Before comparing models on a TOR column, ask what resolution that column has. TOR is a proportion estimated from a finite number of trials, so it carries binomial error, and you can bound it without knowing the exact trial count.

The standard error of a proportion is:

Take PersonaPlex's pause TOR of 0.584. Full-Duplex-Bench has 400 questions spread over four categories, so on the order of 100 trials per category:

So PersonaPlex's pause TOR is something like [0.49, 0.68]. Three consequences, all of which change how you read Table 2:

• Differences of less than about 0.1 in a TOR column are not distinguishable. PersonaPlex 0.584 versus Qwen 0.642 is a coin flip; PersonaPlex 0.584 versus Freeze-Omni 0.255 is real.
• Values near 0 or 1 have smaller error — p(1−p) shrinks at the ends. Gemini's 0.985 and PersonaPlex's 1.000 are tightly determined; the mid-range numbers are the mushy ones.
• The released checkpoint's 0.584 → 0.358 improvement (Chapter 9) is 0.226, well beyond the ±0.097 band. That one is safe.

The paper prints no error bars on Table 2. That is normal for the venue and it is not a scandal, but it does mean the reader has to supply the resolution — and it is why Chapter 8 builds a derived statistic out of a difference of 0.4 rather than arguing about gaps of 0.05.

How each metric can be gamed

A benchmark is only as good as its resistance to cheap wins. Full-Duplex-Bench is unusually well constructed on this axis, and seeing why teaches you the metric set:

Every metric in the set has at least one partner that punishes its degenerate strategy. That is the mark of a benchmark designed by people who expected it to be optimised against.

What Full-Duplex-Bench does not measure

Four blind spots, worth holding while reading Table 2:

• Long-horizon behaviour. Every category is a short scripted situation. Drift, fatigue and context-window effects are invisible.
• Recovery from its own errors. There is no category for "the model said something wrong and the user corrected it."
• Adversarial users. All the scripted users are cooperative. Nobody is trying to break the agent.
• Voice quality. Timing metrics say nothing about whether the audio sounds good. That is what DMOS is for, and DMOS is expensive, which is why it covers only two categories.

The content score, and its one weakness

The interruption category's GPT-4o metric works by transcribing the model's post-interruption response and asking GPT-4o to rate it 1–5 against the situation. It is the only place in Table 2 where what was said matters.

Be aware of what it inherits. It runs on a transcript, so all the paralinguistic information the duplex models worked so hard to preserve is discarded before scoring. A model that yields gracefully with a soft "sorry, go ahead" and one that stops dead mid-word score identically if their subsequent words match. The benchmark knows this — it is why the human DMOS evaluation in Table 1 exists, and why the paper polls 152 evaluators on this exact category.

Scale check: Full-Duplex-Bench contributes 400 questions, to which PersonaPlex's authors add 350 of their own. That is Chapter 7.

Chapter 7: Service-Duplex-Bench — measuring the thing you built

Full-Duplex-Bench is excellent and it cannot evaluate this paper. Every one of its 400 questions is asked of a generic assistant. There is one role, it is implicit, and no question depends on it. Run PersonaPlex on it and you measure whether the fine-tune broke anything — genuinely useful, and it is Table 2 — but you learn nothing about the capability the paper exists to add.

So the authors built the missing half. The paper's framing: "Since Full-Duplex-Bench is limited to a single assistant role, we propose Service-Duplex-Bench, an extension that covers real-world multi-role customer service scenarios."

The design decision that makes it work

Read this sentence carefully, because it is the one that determines whether the benchmark measures anything: "Unlike the multi-turn conversational training data, each evaluation question is a single-turn probe designed to test specific capabilities."

Training data is multi-turn; evaluation is single-turn. That mismatch is deliberate and correct. In a multi-turn evaluation, a role failure at turn one contaminates turns two through eight — you cannot tell whether the model failed six times or once, six times over. A single-turn probe isolates the capability: one role context, one question, one answer, one score. Independent trials, and the mean is interpretable.

The second design decision is stated even more plainly: "all training scenarios are distinct from those used in our Service-Duplex-Bench evaluation, ensuring the model is tested on unseen service contexts." This is criterion 4 from Chapter 0, satisfied explicitly. Without it, 4.48 would be a memorisation score.

350 questions, and why the number is right

Two sanity checks on the size of the extension before looking at its content.

Relative to the host benchmark. 350 added to 400 makes role conditioning 47% of the combined suite — a co-equal half rather than an appendix. That proportion is a statement: the authors are claiming role conditioning is as central to duplex evaluation as turn-taking is, not a specialisation of it.

Relative to what it must resolve. With 50 scenarios per probe column and a rating spread near 0.8, a per-probe mean has a standard error of about 0.11 — enough to separate models that differ by half a point (PersonaPlex 4.6 versus Freeze-Omni 3.9 on Q0) and not enough to separate 4.6 from 4.5. That is the right resolution for a benchmark meant to distinguish kinds of failure rather than to rank near-identical systems.

And one more design detail hiding in plain sight: seven probes and 50 scenarios means every scenario is exercised seven ways, so a badly-written scenario shows up as an outlier across a whole row rather than corrupting one number. That is a quiet form of quality control you get for free from the grid structure.

The seven probes, one at a time

Each scenario supplies a role context in the style of Table 3, then asks seven questions with fixed tags. This is the paper's own worked example, reproduced:

Notice the arc. Q0 tests delivery — did the text arrive? Q1–Q2 test fidelity — is it being used precisely? Q3 tests whether constraints bind. Q4 tests persistence under pressure. Q5–Q6 test precedence over the model's own priors. Those are exactly the three requirements Chapter 1 laid out for role control, plus fidelity, one probe at a time.

The results, read as profiles rather than a ranking

Here is Table 4 in full. Resist reading only the last column.

Three profiles, and each says something different.

Moshi is flat and low. 1.4 to 2.1 across every probe, with no structure. A flat profile is the signature of a model answering from priors alone — the probe categories do not differ for it because none of them are reaching a role context. It is not bad at customer service; it is not doing customer service.

Qwen-2.5-Omni has a dramatic slope. 1.3 and 1.6 on the context-grounded probes, rising to 3.4–3.6 on the generic-behaviour ones. Read that shape: this model is competent and polite — it declines unfulfillable requests, it handles rudeness, it deflects out-of-domain questions — and it cannot recall a single fact from the role context. Those are exactly the skills a general instruction-tuned LLM has without any role conditioning at all.

Make it a number. Define, for our own diagnostic use (this is not in the paper):

A large positive gap means "polite but blind": generic conversational skill without access to the specification. Compute all five, with the arithmetic shown for one:

The judge, and what it inherits

Every number in Table 4 is a GPT-4o rating of a transcript. That is a defensible choice — it is how essentially all open-ended dialogue evaluation is done in 2026 — but "defensible" is not "neutral," and you should know the three biases you are importing.

None of these are reasons to distrust the ordering. Moshi's 1.75 versus PersonaPlex's 4.48 is far outside anything verbosity could explain — Moshi is not losing because it is terse, it is losing because it names the wrong employer. But they are reasons to distrust small gaps, which brings us to a useful arithmetic check.

Running the same evaluation yourself is not hard, and the shape of the harness is worth having in mind because it makes the probe design concrete:

python — the Service-Duplex-Bench evaluation loop, sketched
scores = []
for scenario in SCENARIOS:                      # 50 of them
    prompt = hybrid_prompt(brand_voice, scenario.role_context)
    for probe in scenario.probes:              # 7 each → 350 total
        session = model.new(); session.prefill(prompt)

        # stream the probe audio in real time, capture what comes out
        out = []
        for frame in stream_80ms(probe.audio, tail_silence_s=8):
            out.append(session.step(frame))

        reply = asr.transcribe(mimi.decode(out))    # ← paralinguistics drop here
        scores.append(judge.rate(scenario.role_context, probe.text, reply))

# report per-probe means, then the grand mean over unrounded scores

Two details in that sketch carry real weight. session.new() per probe is what makes the trials independent — no state leaks between questions. And tail_silence_s=8 is why a slow model is not simply scored as silent: the harness keeps the clock running after the probe ends, so latency is measured rather than truncated.

How much can 350 questions resolve?

Before comparing means of 4.48 and 4.73, ask what resolution 350 single-turn probes gives you. Judge scores on a 1–5 scale with most mass at the top typically have a standard deviation around 0.8 for a competent model:

So the Gemini–PersonaPlex gap of 0.25 is around three standard errors — real. And the Freeze-Omni–PersonaPlex gap of 0.46 is comfortably real. But a gap of 0.05 between two hypothetical models would be noise, which is the resolution to keep in mind when tuning your own prompt against this benchmark: differences below about a tenth of a point are not differences.

Note also why per-probe numbers are noisier still. Each probe column is 50 scenarios, not 350:

Which means individual cells in Table 4 should be read as approximate. PersonaPlex's 4.6 on Q0 and 4.3 on Q5 are not reliably different from each other. What is reliable is the shape across many probes — which is exactly why the grounding gap, built from six cells rather than one, is a sturdier statistic than any single column.

A rounding puzzle worth solving

Add up PersonaPlex's seven printed scores and divide:

Do the same for Moshi: 12.3/7 = 1.7571 → 1.76, printed 1.75. And for Freeze-Omni: 28.1/7 = 4.0143 → 4.01, printed 4.02. Meanwhile Gemini (33.1/7 = 4.7286 → 4.73) and Qwen (19.3/7 = 2.7571 → 2.76) match exactly.

Nothing is wrong. The means are computed from the unrounded per-probe scores and then rounded once, which is the correct procedure; the per-probe column is rounded for display. Three of five agree with the naive recomputation and two disagree in the last digit, exactly as you would expect from independent rounding.

Two reasons this small check is worth doing on any paper. It confirms you have read the table's orientation correctly — if you had transposed rows and columns, none of the five means would reproduce. And it tells you the true precision: differences below about 0.02 in this column are display noise, so PersonaPlex's 4.48 and a hypothetical 4.47 are the same result.

The three requirements, tested one probe at a time

Chapter 1 claimed role control needs delivery, persistence and precedence. Here is the mapping, which is the cleanest evidence that the benchmark was designed against a theory rather than assembled from convenient questions:

Five requirements, seven probes, no wasted questions. If you are designing an evaluation for any conditioned system, this is the shape to copy: start from what the capability requires, then write one probe per requirement, then check that a plausible failure mode maps to each.

Why 50 × 7 and not 350 × 1

The same 350 items could have been 350 one-off questions across 350 roles. The chosen structure — 50 roles, 7 probes each — is better for two reasons worth stating.

Within-scenario comparison. Because all seven probes share a role context, differences between them cannot be explained by the context being easier or harder. When PersonaPlex scores 4.6 on Q0 and 4.3 on Q5 in the same scenario, the difference is about the probe type, not about the prompt. This is what makes the grounding-gap statistic meaningful at all.

Authoring economics. Writing a good role context is the expensive part — it needs a name, an employer, exact values, an enumerated option set, a policy limit, and deliberate gaps. Amortising that over seven probes gets you seven times the measurement per unit of authoring effort. This is why the recipe in the next section is a day of work rather than a week.

The cost is statistical: 350 items across 50 contexts are not fully independent, so the effective sample size is somewhere below 350. With seven probes deliberately testing different capabilities, the within-context correlation is low, and the trade is clearly worth it.

Why an extension rather than a new benchmark

The authors could have built a standalone role-conditioning benchmark. Extending Full-Duplex-Bench instead has three consequences that are easy to overlook.

The baselines come for free. Every model already evaluated on Full-Duplex-Bench can be run on the extension with the same harness, the same streaming protocol, and the same judge. Building a fresh benchmark would have meant re-implementing five baselines' evaluation from scratch, with all the fairness disputes that invites.

The conversational metrics remain applicable. Because the protocol is unchanged — user audio streamed in, generated audio captured — you can in principle score turn-taking behaviour on service scenarios too, not just answer quality. The paper does not report that, but the door is open.

It composes rather than competes. 400 + 350 = 750 questions in one framework, with the role-conditioning half clearly separable. A community that adopts it does not have to choose between two benchmarks, which is usually how benchmarks die.

The cost is inheritance: every limitation of Full-Duplex-Bench — single-turn framing, transcript-based judging, cooperative users — is inherited wholesale. Chapter 6's blind-spot list applies unchanged to the extension.

What makes a good scenario

Fifty scenarios sounds like a lot until you try to write them and discover that thirty of yours are the same scenario. The paper's Table 3 example implicitly defines what a usable one contains, and it is worth extracting as a specification.

That last row is the counterintuitive one. The instinct when writing an evaluation context is to make it complete. Do the opposite: a good scenario is deliberately partial, because the most dangerous behaviour of a service agent is confidently filling in a detail nobody gave it. Table 3's context lists three plan prices and says nothing about what any of them covers — and then Q2 asks which covers dental and vision. That omission is the test.

Grading your own agent

Once you have 350 items, the reporting that is worth doing is not the mean:

1. Per-probe means, plotted as a profile. Flat and low means the prompt is not arriving; steeply rising means polite-but-blind.
2. The grounding gap, mean(Q3–Q6) − mean(Q0, Q1). Positive and large is the alarm.
3. The Q1 confirmation rate specifically — how often does your agent confirm a value that does not match? Report it as a rate, not a 1–5 score; this is the one with legal exposure.
4. The worst decile, read by a human. Means hide the failures that will end up in a complaint.

Porting the benchmark to your own domain

The probe taxonomy is the transferable part of this chapter, and it is domain-independent. Here it is as a recipe, with the health-insurance example alongside a second domain so the pattern is visible rather than memorised:

Fifty scenarios × seven probes = 350 items, which is one focused day of authoring. Then score with a judge and compute the grounding gap. The probes that will find bugs in your prompt are Q1 (does the agent confirm things that are false?), Q3 (does your policy actually bind?) and Q6 (does the role take precedence over the model's general knowledge?).

What a 1 and a 5 look like

The paper does not print its judge rubric. But the probe design implies one, and writing it out makes the scores interpretable rather than magical. For Q0, the proper-noun probe:

Two things follow. Moshi's 1.5 means it is consistently giving a generic non-answer, not occasionally getting it right — a mean near the floor with seven probes leaves little room for variance. And the ceiling of 4.6 rather than 5.0 for the best models suggests the judge reserves 5 for something more than correctness: fluency, tone, or completeness. Nobody is getting full marks, which is a sign of a rubric with headroom rather than a saturated benchmark.

What the benchmark still cannot see

Three honest limits, none of which the paper claims otherwise.

Single-turn probes cannot test persistence. The scariest role failures are drift and jailbreak — the agent that is Brody Murphy at turn one and something else by turn twelve, or that revises its policy when a customer pushes hard three times. A one-question probe cannot see any of that. It is the natural next extension of this benchmark.

The judge scores a transcript. Same limitation as Chapter 6's content metric: tone, warmth and hesitation are outside the score. A model can be textually perfect and audibly robotic. That is what Table 1's human DMOS is for, and Service-Duplex-Bench DMOS is where PersonaPlex's lead is largest — 3.59 against Gemini's 3.22.

All 50 scenarios are customer service. The paper is explicit about this scope. Role conditioning for a tutor, a therapist, a game character, or a technical-support engineer with a decision tree is not measured. The mechanism has no reason to be domain-specific, but the evidence is.

The authors say they "plan to release this dataset to provide an evaluation framework for future models" — which, given how much of the value here is in the probe taxonomy rather than the model, may end up being the paper's most durable contribution.

Chapter 8: The Results, Read Adversarially

Everything is now in place: the mechanism, the objective, the data, and both benchmarks. This chapter is the evidence. We are going to read it the way a reviewer would — looking first for where it fails.

Chapter 0 committed to five acceptance criteria before seeing any numbers. Grade against those, in order.

How to read a results section adversarially

The procedure this chapter follows, stated once so it transfers to the next paper you read:

1. Write your acceptance criteria before looking. Done in Chapter 0. Without this you will accept whatever is shown.
2. Find the losses first. Every table has them; papers rarely point at them. In Table 2 there are four.
3. For each loss, ask whether it is evidence the metric is real. A method that wins everything is usually being measured by metrics it chose.
4. Look for columns that are secretly one axis. When two metrics trade off mechanically, the difference is more informative than either.
5. Recover the statistics the paper omitted. Sample sizes from evaluator counts, spreads from confidence intervals, effect sizes from both.
6. Separate the controlled comparison from the context. Five-way tables are context; the one clean A/B is the experiment.

Applied here, steps 2 through 5 produced everything in this chapter that is not simply a restatement of the paper: the eagerness axis, the floor-discrimination statistic, the naturalness/correctness orthogonality, the recovered rating spread of ~1.15, and the finding that one of the two headline naturalness leads is not statistically established.

Criterion 1 — the role knob works

Service-Duplex-Bench mean GPT-4o score: Moshi 1.75, PersonaPlex 4.48, Gemini 4.73. The knob works: from "not reading the role at all" to a quarter-point behind the best commercial system in the paper, entirely from a six-hour fine-tune of the model that scored 1.75.

Chapter 7's per-probe analysis strengthens this beyond the mean. PersonaPlex is the only model in the table whose context-grounded probes outscore its generic ones. The improvement is specifically in reading the prompt, which is what was claimed.

The honest deduction. Gemini wins this column, and the paper says so plainly: PersonaPlex "outperforms all models except Gemini Live." That gap is real and it is worth understanding rather than explaining away. Gemini Live sits on a very large general model; PersonaPlex is a 7B duplex model specialised on customer service. On the two probes that ask for general-world reasoning outside the prompt (Q5 unspecified, Q6 unrelated), Gemini leads by 0.4. That is the size of the model, not the mechanism.

Criterion 2 — the voice knob works

Between five and eleven times the next-best value, measured on 2,630 held-out voice samples with a standard speaker-verification model. This is the cleanest result in the paper: the baselines are not slightly worse, they are at the "ignored the prompt" floor, which is what a missing capability looks like in a metric.

The honest deduction. 0.57 is not 0.95. Dedicated zero-shot cloning TTS systems reach higher speaker similarity than this, and they should — they are not simultaneously holding a conversation, obeying a role, and running in real time at 12.5 Hz. What 0.57 buys you is "recognisably the target speaker." What it does not buy you is an impersonation that would fool the target's family, which — see Chapter 10 — is arguably a feature.

Criterion 3 — nothing conversational broke

This is where a careless read goes wrong, so take it slowly. Full Table 2:

PersonaPlex takes six of the ten columns. It also loses four, and those are the interesting ones:

• Both pause-handling columns, badly. Freeze-Omni 0.255 / 0.310 against PersonaPlex 0.584 / 0.662. PersonaPlex talks into more than half of the user's mid-sentence pauses.
• Backchannel TOR, to Freeze-Omni again (0.091 vs 0.327).
• Interruption GPT-4o content, to Qwen-2.5-Omni (4.590 vs 4.210).
• Interruption latency, to Moshi (0.257 s vs 0.400 s).

Take the pause losses seriously first, then explain them — in that order, because the reverse is how people fool themselves.

They are real. A model that starts talking during 58–66% of your mid-sentence breaths is genuinely interrupting you, and any deployment will feel it. The released checkpoint (Chapter 9) improves this to 0.358 / 0.431, which is both an improvement and an admission that the experimental model had a problem.

Now the explanation, which is Chapter 6's eagerness axis made quantitative. Freeze-Omni wins the pause columns because it barely talks: its smooth-turn-taking TOR is 0.655, meaning it fails to respond to a third of completed turns, and its turn-taking latency is 1.301 s, the worst in the table. It is not better at telling pauses from turn-ends; it is reluctant, and reluctance scores well on half the columns.

So build the statistic that separates disposition from discrimination. Ours, not the paper's:

How much more likely is the model to take the floor when it should than when it should not? A model with no discrimination scores 0 no matter how eager it is.

The other two losses are smaller and both explained by Chapter 6. Qwen's higher interruption content score comes with 2.740 s of latency — it is a half-duplex model that thinks carefully and answers late. Moshi's faster interruption response comes with a content score of 0.765 — reflex without comprehension.

Criterion 5 — humans agree

Table 1, the human evaluation. 354 Mechanical Turk evaluators, 2,832 rated samples, 1–5 dialogue mean opinion score.

Two things jump out before any statistics.

First, Moshi outranks Qwen and Freeze-Omni on Service-Duplex-Bench naturalness (2.83 vs 2.37 and 2.38) while scoring 1.75 on role adherence against their 2.76 and 4.02. Listeners rate it more pleasant to converse with while it says things that are wrong for the scenario. Naturalness and correctness are close to orthogonal, which is exactly why the paper reports both and why a voice-agent team that tracks only one will ship the wrong thing.

Second, every model scores lower on Service-Duplex-Bench than on Full-Duplex-Bench. Role-driven service conversation is simply harder to make sound natural than generic assistant chat.

Are those differences real? Do the statistics.

The paper reports 95% confidence intervals and leaves it there. Push a little further, because the answer differs between the two columns.

Step 1 — recover the sample sizes. For Full-Duplex-Bench User Interruption: 152 evaluators × 8 samples = 1,216 ratings, spread over 5 models, so about 243 per model. For Service-Duplex-Bench: 202 × 8 = 1,616 ratings, about 323 per model.

Step 2 — back out the rating spread. A 95% interval half-width is 1.96 · s / √n, so s = half-width · √n / 1.96:

Both land near 1.1–1.2 on a 1–5 scale. That consistency is the check: two independently reported intervals, from two different evaluator pools and sample counts, implying nearly the same per-rating standard deviation. Our reconstruction of the design is almost certainly right, and a spread of ~1.15 is very typical for crowd-sourced MOS.

Step 3 — test PersonaPlex against Gemini. For a difference of independent means, standard errors add in quadrature:

Not significant at the conventional 5% level. Now the same test on Service-Duplex-Bench:

Baseline fairness — three caveats the paper flags or implies

Comparisons across five systems of very different kinds are never perfectly fair. Three specific caveats, so you weight the table correctly.

None of these undermines the paper's central claims, because the central claims are about PersonaPlex versus Moshi — same architecture, same scale, same harness, six hours of fine-tuning apart. That comparison is as clean as this field gets. The five-way table is context, and should be read as context.

Effect sizes, because p-values are not magnitudes

A p-value tells you whether a difference is distinguishable from noise; it says nothing about whether the difference matters. For that you want an effect size, and we already recovered everything needed: the per-rating spread is about 1.19 (Full-Duplex-Bench) and 1.10 (Service-Duplex-Bench).

Read those honestly and the picture sharpens considerably. The service-scenario lead over Gemini is real (p < 0.0001, because n is large) and small (d = 0.34). Both statements are true, and reporting only the first would be misleading. Against the open baselines the effects are medium to large — which is the comparison that matters for the claim "the first open model to reach comparable naturalness as closed commercial systems."

The comparison that carries the paper

Five-way tables are context. One comparison is the actual experiment, and it is worth isolating because it is unusually clean.

Everything moved in the right direction, several by a lot, with the architecture held fixed and six hours of fine-tuning as the only intervention. Note especially the interruption content score: 0.765 to 4.210. That is not a conditioning result — it is the fine-tune making the model coherent after being interrupted, which nobody was claiming. Some of the paper's benefit is simply that 2,250 hours of clean two-speaker dialogue is better instruction data than whatever Moshi last saw.

Reading Table 1 and Table 2 together

The two tables answer different questions and the temptation is to read whichever supports your prior. Read them as a pair instead:

The third row is the uncomfortable one and the most instructive. A model that says nothing useful can still be rated more natural than models that say useful things, purely on timing. Whatever "naturalness" measures, it is heavily weighted toward the temporal channel — which is both a validation of why full duplex matters and a warning that DMOS alone is not a quality metric either.

Column by column: who wins what, and why

Three claims this paper could have made and did not

Reading what an author declined to claim is often more informative than reading what they claimed. Three available overreaches, all avoided:

Three careful hedges in a four-page paper with an obvious incentive not to hedge. That earns the rest of the numbers a degree of trust, and it is worth noticing explicitly, because the usual experience of reading a systems paper is the opposite.

A reviewer's remaining objections

If you were reviewing this for ICASSP, what would you still write in the box? Four things, in descending order of how much they would bother us:

1. One seed, no error bars on Tables 2, 4, 5. Chapter 6's binomial estimate puts roughly ±0.1 on a mid-range TOR, and Chapter 9's non-monotonic ablation column is exactly what single-seed noise looks like. The human evaluation has intervals; nothing else does.
2. No mechanism ablations. Voice-only versus text-only versus hybrid; sine versus silence; the loss weights. The order-invariance result is the only evidence offered that the two conditioning segments do not interfere, and it is indirect.
3. The judge and the data generator share a lineage. Qwen3-32B and GPT-OSS-120B write the training dialogues; GPT-4o grades. Different families, overlapping priors about what a good service reply looks like. Some of the 4.48 is stylistic agreement.
4. Scope is one domain. Fifty customer-service scenarios. The mechanism should not care, but the evidence does not extend past service.

And what would we write in the strengths box? Three, and the first two are rarer than they should be:

• The ablation's zero point is a published baseline, so the scaling curve is anchored to something real.
• The released checkpoint is evaluated separately and reported honestly, including the three metrics where it is worse.
• The benchmark extension is released, and its probe taxonomy is domain-general — likely to outlive the model.

The scorecard

Five for five, one with an asterisk, on a four-page paper. And the conclusion that matters for anyone building on this is the structural one, not the leaderboard one: conditioning was added to a duplex model without changing its architecture, for 48 A100-hours, and the conversational behaviour survived.

Chapter 9: Ablations — and the model they actually shipped

A four-page paper gets one ablation. The authors spent it on the question that most determines whether anyone else can reproduce this: how much data do you need?

They retrained at 100%, 50%, and 25% of the synthetic corpus, and reported a fourth point that is quietly the best design choice in the table.

The two trends diverge, and the paper says so

Read the columns separately, because they behave completely differently. The paper's own summary: "On Full-Duplex-Bench, strong performance is achieved with limited data, while on Service-Duplex-Bench, role adherence improves steadily with more data."

Speaker similarity saturates almost immediately. From 0.10 to 0.54 in the first quarter of the data, then 0.54 → 0.56 → 0.57 across a 4× increase. Quantify the share:

Full-Duplex-Bench interruption content saturates too — in fact it is non-monotonic: 4.44 at 25%, 4.52 at 50%, 4.21 at 100%. The 100% run is the worst of the three. With no error bars reported, the honest reading is that all three are within noise of each other and the curve is flat after 25%. But it would be dishonest not to notice that the flat curve's endpoint is the low one.

Service role adherence climbs the whole way. 1.75 → 4.20 → 4.24 → 4.48. Break the increments down:

An odd shape: a big jump, a plateau, then a second rise. If forced to interpret, the most economical story is that the first quarter teaches the mechanism — "there is a prompt; it is about you; use it" — after which more of the same scenarios add nothing, and the second doubling finally adds enough scenario diversity to improve generalisation to unseen roles. Speculative; the paper offers no explanation and there is only one seed.

The ablation in one sentence each

Three findings, stated as plainly as the data allows:

1. Almost all of the capability arrives with the first quarter of the data. 562 hours takes speaker similarity from 0.10 to 0.54 and role adherence from 1.75 to 4.20 — 93.6% and 89.7% of their respective total gains.
2. Only role adherence keeps improving after that, and it is still improving at 100%, which is what you would expect from a capability that depends on the number of distinct specifications seen rather than on hours.
3. Nothing measurably degrades with more data, though the largest run happens to hold the lowest interruption-content score, and with one seed that is indistinguishable from noise.

The practical compression: generate 560 hours, train, measure. Then decide whether the last quarter-point is worth quadrupling your generation budget.

Reading a scaling table honestly

Four rows, three columns, one seed, no error bars. Before extracting conclusions, fix what this table can and cannot support.

Three usable answers and two refusals from a four-row table is a good yield, and being explicit about which is which is the difference between reading a paper and repeating it.

The one table the paper had room for

A four-page submission can afford roughly one ablation, so which one you choose is a statement about what you think readers most need. The available candidates and what each would have answered:

They chose the one whose answer changes what other people do. That is the right instinct, and the payoff is Chapter 9's central practical finding: the capability appears almost entirely within the first quarter of the corpus, so the entry cost to this line of work is roughly 560 hours of generated dialogue and 48 A100-hours — not the full 2,250.

The cost of that choice is everything in the table above, which is why this chapter spends as much time on the missing ablations as on the present one.

What the percentages mean in dialogues

"25% of the data" is abstract. Convert it, because the reproduction question is "how many conversations do I need to generate?", and generation is the expensive part:

(The paper reports only the percentage; the dialogue counts assume proportional subsampling, which is the natural reading.)

Now the practitioner's question: what does the last doubling buy? Going from 52,705 to 105,410 service dialogues — another 52,705 LLM-generated conversations and 1,125 hours of TTS — moves Service-Duplex-Bench from 4.24 to 4.48. A quarter of a point on a five-point scale, for double the generation budget.

Whether that is worth it depends entirely on where you sit. At 4.24 versus 4.48, both are past the "deployable" line and the difference is invisible to most users. But the same 0.24 is what separates PersonaPlex from Gemini's 4.73, so at the top of the range every quarter-point is contested. This is the ordinary shape of a saturating curve, and the honest summary is: get to 25% quickly, then decide whether you are competing on the last quarter-point at all.

One more decomposition worth doing. Compare what each halving costs against what it buys:

The last row is a cliff, not a slope. Almost the entire capability is bought by the first 562 hours; everything after is refinement. If you remember one thing from this ablation, it is the shape of that cliff — it says the mechanism is easy to teach and the polish is expensive.

The released checkpoint as an intervention study

Appendix A is, structurally, a three-part intervention with a shared outcome measure. It is not a controlled experiment — all three changes were made at once — but the changes are different enough that the effects are largely attributable, and doing that attribution is good practice.

Now the outcome that is not attributable to any single change: the three regressions (turn-taking TOR, turn-taking latency, interruption TOR). Our reading is that all three are the same shift toward restraint that fixed the pause columns — one behavioural change with a good side and a bad side, not three separate problems. Chapter 8's derived statistic is what supports that: floor discrimination improved from +0.408 to +0.550 even while both of its constituent columns moved. The model did not get worse at turn-taking; it got less eager, and less eagerness is net positive on the axis that matters.

Reading Table 5 as three questions

One ablation, three separable questions. Separating them is what makes the odd shapes interpretable.

Question 1: how much data to learn the mechanism? Answer: very little. The jump from 0% to 25% carries 93.6% of the speaker-similarity gain and 89.7% of the role-adherence gain. Whatever "learning to read a hybrid prompt" costs, 562 hours pays for it.

Question 2: how much data to generalise to unseen roles? Answer: considerably more, and the curve is still rising at 100%. Service-Duplex-Bench goes 4.20 → 4.24 → 4.48, with the largest late increment in the final doubling. Since evaluation scenarios are disjoint from training ones, this column is specifically measuring transfer to roles never seen — and transfer wants diversity, which is what doubling a hierarchically-sampled corpus buys.

Question 3: does more data cost anything? Possibly. Full-Duplex-Bench interruption content peaks at 50% (4.52) and is lowest at 100% (4.21). One seed, no error bars, so the responsible reading is "flat" — but the direction is consistent with mild over-specialisation on customer service at the expense of general assistant behaviour, which is the same effect visible as PersonaPlex's small negative grounding gap in Chapter 7.

The ablations the paper did not run

Worth listing, because these are the experiments you would want before building on this, and because naming them is how you read a short paper responsibly.

One number frames everything that follows. The full corpus is 2,250 hours of generated conversation — the most expensive artefact in the paper by a wide margin, far more costly than the 48 A100-hours of training it feeds. So "how much of it did you actually need?" is not an academic question; it is the entry price for anyone who wants to build on this.

What an ablation is for

Two different jobs, and this one does the second rather than the first.

Job one: attribution. Remove a component, see what breaks, and thereby show that the component is responsible for the result. This is what "ablation" usually means, and it is what reviewers ask for. This paper does none of it — there is no run without the sine wave, no run with text-only conditioning, no loss-weight sweep.

Job two: transferability. Vary a resource and show how the result depends on it, so that someone with a different budget knows what to expect. This is what Table 5 does, and for a method paper whose main contribution is a recipe others should follow, it is arguably the more useful of the two.

Both jobs matter, and a longer paper would do both. If you only have room for one, the choice tells you who the authors imagine their reader to be. Choosing transferability says: we expect you to build this, and here is what it costs.

If you were reproducing this

The ablation is really a budget document. Read as one, it gives a staged plan:

The ordering is the finding. Stage 3 — adding real human conversation — moves the conversational-dynamics metrics far more than stage 2 moves the role metric, and stage 3 was not in the paper's main experiments at all. If you have a fixed budget, the ablation plus the appendix jointly say: get to 25% synthetic, then spend everything else on real speech.

Why the appendix belongs in the paper

One editorial observation before the details. It is common for a research artefact and a released artefact to diverge, and it is uncommon to say so. The usual pattern is that the paper reports the experimental model and the download is quietly something else — better, usually, but differently trained, so nothing you read predicts what you get.

Appendix A breaks that pattern in three ways worth naming:

• It enumerates the differences precisely: which corpus was added, which TTS replaced which, why the voice pool changed.
• It re-runs the full conversational-dynamics table on the shipped model (Table 6), rather than reporting only the metric that improved.
• It re-runs the human evaluation with a fresh annotator pool (Table 7) — and explicitly warns that the scores "are relative within this study and not directly comparable to Table 1," which is the correct caveat and the one most often omitted.

The cost is that the paper's own headline model is not the one you can download, and three of its numbers get worse when you do. The benefit is that a reader can predict their experience. For a paper whose stated purpose is to advance "duplex speech towards real-world deployment," that trade is obviously right — and it is the reason this chapter treats the appendix as a first-class result rather than as back matter.

The experiment that is missing, and how to run it

Chapter 9's two-axis observation implies a specific next experiment, and it is cheap enough that a reader with a few hundred GPU-hours could settle it. Spelling it out is the most useful thing this chapter can do.

The confound. The paper's ablation subsamples the corpus, which reduces hours and distinct role contexts together. So "role adherence improves with more data" cannot distinguish two very different explanations: more exposure to the same scenarios, or exposure to more scenarios.

The design. Hold total hours fixed at, say, 560 (the 25% point, where the mechanism is already learned) and vary only the number of distinct role contexts, resampling dialogues to keep hours constant:

The prediction. Given that evaluation scenarios are disjoint from training ones, and given that the metric is transfer to unseen roles, we would expect C > B > A, and the gap between A and C to be larger than the gap between the paper's 25% and 100% runs. If instead A ≈ C, then role conditioning is a mechanism that needs volume rather than variety, and everyone reproducing this can stop paying for scenario generation.

What it would cost. Three fine-tunes at roughly a quarter of the paper's 48 A100-hours each — call it 36 A100-hours total, plus the generation for arm C which the paper's pipeline already produces. This is a weekend, not a project.

The released checkpoint: a second paper hiding in an appendix

Now the unusual part. Appendix A describes the model the authors actually put on Hugging Face, and it is not the model in the tables. Most papers do this silently. This one prints both, with a fresh evaluation.

Three changes:

1. Real conversational data. 7,303 Fisher English conversations — 1,217 hours of genuine, unscripted, disfluent telephone speech — "to improve natural backchanneling, expressions, and emotional responses." Prompts for them were generated backwards with GPT-OSS-120B at three detail tiers (minimal / topic-specific / highly detailed) "to balance generalization capability with instruction-following precision."
2. Synthetic voices only. For data privacy, TortoiseTTS voices pitch- and formant-augmented with Praat, replacing the real-speaker datasets.
3. One TTS everywhere. Chatterbox for all synthetic dialogue, replacing the mixed Dia/Chatterbox approach, "since Chatterbox provides superior speaker consistency."

Scale check: 1,217 hours of Fisher against 2,250 synthetic. Real conversation is now 35% of the corpus by duration — 1217/3467 = 0.351. That is not a garnish; it is a third of the training signal.

Now the measured effect, Table 6 against Table 2:

Recompute the derived statistic from Chapter 8 for the released model: 0.908 − 0.358 = +0.550 floor discrimination, comfortably the best number in the paper, at 0.170 s turn-taking latency. The released checkpoint is not a marketing refresh; it is a better model on the axis that matters most.

The naturalness re-evaluation (Table 7) uses a different annotator pool and the paper explicitly warns the scores "are relative within this study and not directly comparable to Table 1." Take only the ordering:

Same ordering at the top, compressed differences, and much wider intervals — the whole pool rated everything lower and less consistently. The paper's own phrase is "maintains competitive naturalness," which is the right amount of claim for a ±0.25 interval over a 0.15 gap.

Chapter 10: Paper → Product

Everything so far has been about understanding a paper. This chapter is about the other direction: you are on a team shipping a voice agent, PersonaPlex exists, and someone has to decide what changes.

The honest headline first, before the enthusiasm: in 2026 the cascaded stack still wins most enterprise deployments, and this paper does not change that on its own. It changes the trajectory, and it removes the single most-cited reason duplex models were not deployable. Understanding exactly which objection it removes — and which four it does not — is the useful thing.

The decision, laid out honestly

Four of the ten rows still favour cascaded, and three of those four (tools, retrieval, session length) are the same underlying issue: a duplex model's context is clocked, and everything you want to put in it costs seconds.

The context clock is the real deployment ceiling

This is the constraint the paper never names, and Chapter 3's arithmetic makes it unavoidable. Take a ten-minute support call:

A ten-minute call needs nearly four times the context the model was trained on. And in a duplex model you cannot trim the way you would in a chat app: every frame of silence, every "mm-hm", every second of the customer thinking is a frame that has to exist, because time itself is the sequence axis. There is no "summarise the last hour" — the last hour is 45,000 frames.

The mitigations, in increasing order of unpleasantness:

1. Sliding window with a pinned prefix. Keep the Hybrid System Prompt's KV cache permanently and evict the oldest dialogue frames. The persona survives; the conversation's early history does not. Cheap, and probably what you would ship first.
2. Periodic re-prompting. Fold a summary of what has happened into a regenerated text prompt segment. Costs a stall and one token per frame per token of summary.
3. Length extrapolation. Train or adapt for longer sequences. Straightforward in principle, and the reason the paper's 163.84 s is a training choice rather than a law.
4. Hybrid routing. Duplex model for conversation, cascaded path for anything needing tools or long memory. Now you have two systems and a state-synchronisation problem.

Writing a persona spec that works

The paper gives you a schema for free. Table 3's context has exactly four components, and the released checkpoint's appendix gives you a detail ladder. Combine them:

Two things are conspicuously absent from that schema and their absence is the interesting part. There is no tone instruction and no refusal policy. Yet PersonaPlex scores 4.5 on the rudeness probe and 4.5 on the unrelated-request probe with neither specified — because those behaviours came from the 105,410 training dialogues, not from the prompt. The prompt supplies facts; the fine-tune supplies manners. That division is worth knowing before you write a 300-token prompt trying to specify politeness that is already there.

And the detail ladder from Appendix A tells you how much to write:

The authors' stated reason for training on all three is "to balance generalization capability with instruction-following precision" — a model shown only detailed prompts overfits to detail and flounders on a one-liner. That is a data lesson with a direct product consequence: your prompts should live in the same distribution the model was trained on, and now you know what that distribution looks like.

Where the risk actually sits

Deployment risk is rarely where the demo is impressive. Ranked by how likely each is to be the thing that stops your launch:

Five of the six are visible in this lesson's numbers before you write any code. That is the argument for reading a paper carefully before adopting it: most of your deployment surprises are already printed in its tables.

What the paper hands you for free

Three artefacts, all immediately usable, and all cheaper to adopt than to rebuild:

• The mechanism. Voice segment, text segment, sine filler, delimiters, masked loss. About twenty lines, reproduced in Chapter 3. It applies to any Moshi-style duplex model, not just this checkpoint.
• The probe taxonomy. Seven categories that decompose "holds a role" into measurable behaviours. Domain-independent, and the paper says the dataset will be released.
• The data recipe. Hierarchical scenario sampling, joint prompt-and-transcript generation, joint multispeaker TTS, and the negative-silence stitch. Chapter 5 has all four as pseudocode.

What it does not hand you is a serving stack, a session manager, a guardrail layer, or an answer to the context clock. Those are your quarter's work, and they are the subject of this chapter.

A scoring worksheet for the architecture decision

The decision table above lists the axes; here is how to actually decide. Weight each row by what your deployment needs, score each stack 0–3, and total. Two worked profiles show how differently it comes out.

Companion app: duplex 30, cascaded 19 — not close. Contact centre: cascaded 51, duplex 20 — also not close, in the other direction. The axes that flip the result are tools, retrieval, auditability and session length, and all four trace back to the same root cause: the context clock.

Which suggests the interesting third option. If your profile is "contact centre that wants to sound human", you want a hybrid: the duplex model owns the conversation and the timing; a cascaded side-path owns tools, retrieval and the audit log, driven by the inner-monologue channel as its input. The duplex model's agent-text stream is, conveniently, exactly the transcript such a side-path needs — available live, frame by frame, at no extra cost.

That design is not in the paper. It is, however, the natural reading of the paper's own future work, and it is what the "voice + action" line of research is converging on.

Serving arithmetic

One number decides whether this is deployable at all: the model must finish one temporal-transformer forward pass plus Q depth-transformer passes within 80 milliseconds, every frame, per concurrent call. Not on average — every frame, because a missed deadline is an audible glitch.

That is a hard-real-time constraint of a kind most inference stacks are not built for. It has three consequences worth naming: batching across calls helps throughput but adds jitter, so your batch window has to fit inside the frame budget; tail latency matters more than mean latency, which inverts the usual serving intuition; and a 7B model at 12.5 Hz is a very different cost profile from a 7B model serving chat, because you are paying for a forward pass every 80 ms of an entire call, including the parts where nobody is saying anything.

The paper reports no serving benchmarks. It does report that the released checkpoint is personaplex-7b-v1, which at least tells you the scale you are budgeting for.

A session manager, written out

The mitigations list is abstract until you write the loop. Here is the sliding-window design — the one you would actually ship first — with the two decisions that make or break it marked:

python — a duplex session with a pinned prompt and a sliding window
MAX_FRAMES = 2048

class Session:
    def __init__(self, role_text, voice_wav):
        u, x, a, mask = hybrid_prompt(voice_wav, role_text, ...)
        self.prompt_len = len(x)                # e.g. 187
        self.kv = model.prefill(u, x, a)         # the connect-time cost
        self.pinned = self.kv.slice(0, self.prompt_len)   # ← DECISION 1: never evict

    def step(self, mic_80ms):
        if self.kv.len() >= MAX_FRAMES:
            # ← DECISION 2: what to drop. Oldest dialogue frames, never the prompt.
            keep = MAX_FRAMES - self.prompt_len - EVICT_BLOCK
            self.kv = self.pinned + self.kv.tail(keep)
        return model.step(mic_80ms, self.kv)

    # budget: 2048 − 187 = 1861 frames = 148.9 s of rolling memory.
    # at 10 min, the caller's first 8 minutes are gone. plan for that.

Decision 1 is why voice-first ordering matters twice: the pinned prefix is not only cacheable across sessions, it is the natural eviction boundary within a session. Decision 2 is where product judgement lives — evicting in blocks rather than one frame at a time avoids recomputing the cache every 80 ms, and how large a block you choose trades memory smoothness against a periodic hitch.

What this design cannot do is remember. After nine minutes of a ten-minute call, the account number the customer read out at minute one is gone from context. In a cascaded stack it would still be in the transcript. If your use case needs that, you need mitigation 2 or 4, and you should decide which before writing any of this.

Costing a concurrent call

One more piece of arithmetic that decides feasibility. A duplex call is a continuous workload, unlike a chat turn:

and every one of those temporal passes must land inside its 80 ms window. Three consequences that invert normal serving intuition:

• Tail latency is the metric, not mean latency. A p99 of 90 ms means roughly one audible glitch every eight seconds. Nobody tunes chat serving this way.
• Batching is constrained by the frame budget. You can batch across concurrent calls — they all tick on the same 80 ms clock — but the batch-collection window plus the forward pass must fit inside 80 ms, which caps how long you may wait to fill a batch.
• You pay for silence. A cascaded stack costs nothing while the customer is thinking. A duplex model runs a full forward pass every 80 ms of the entire call regardless. For a workload with long hold times, that is a materially different cost curve.

The paper reports no serving benchmarks, so none of this is measured. It is, however, entirely determined by the architecture, and it is the first thing an infrastructure reviewer will ask.

Guardrails, and where they can go

In a cascaded stack you can inspect and block at every hop: filter the transcript, filter the LLM output, refuse to synthesise. In a duplex model there is no hop. The audio is produced frame by frame and it is already leaving the building.

What you have instead:

• The prompt. Policy constraints in the role context genuinely bind — Q3's 4.5 is the measurement of that. Put your hard limits there.
• The inner monologue. The agent-text channel is a live, time-aligned transcript of what the model is saying, available as it says it. That is a real monitoring surface: you can run a classifier on it frame by frame. It is not a complete record — tone is not in it — but it is far better than nothing, and it is a genuine advantage of the inner-monologue design over duplex architectures without one.
• Post-hoc audit. The paper's own final sentence lists "post-training alignment" as future work, which is an accurate statement of the current position: there is no RLHF or DPO stage here. What you get is what supervised fine-tuning on 144,732 synthetic dialogues produced.

The one-paragraph version for a decision meeting

"Full-duplex speech models are the only architecture that gives genuinely human turn-taking — sub-100 ms responses, native interruption, backchannelling — and until this year they could not be given a persona or a brand voice, which made them undeployable for us. That is now solved, in an open model, by a fine-tuning recipe that costs about fifty GPU-hours and changes nothing architecturally. What is still missing is tool calling, retrieval, and anything longer than about three minutes of context, all of which our current cascaded stack does well. So the recommendation is a hybrid: pilot the duplex model on the conversational layer where naturalness is the product, keep the cascaded path for anything that has to look something up, and build the evaluation first so we can tell which is actually better on our own calls."

Every clause in that paragraph is defended somewhere in this lesson, with a number attached. If you can say it and answer the follow-up questions, the lesson has done its job.

What to build first

Assume you are convinced and have a quarter to spend. The order of work is not obvious, because the tempting first step — getting the model running — is not the step that will kill the project.

1. Build the evaluation before the agent. Fifty scenarios, seven probes, a judge, and the grounding-gap computation. One focused day. Everything after this is measurable, and without it nothing is.
2. Run the released checkpoint against it, unchanged. You now know your baseline on your domain, not on health insurance. This is also where you find out whether your prompts are the problem.
3. Solve the session-length question on paper. Sliding window, re-prompting, or hybrid routing — pick one before writing serving code, because it determines your architecture.
4. Measure turn-taking, not just answers. Scripted pause and barge-in trials against your own agent. Compute floor discrimination. This is the metric your users will feel and the one nobody instruments.
5. Only then consider fine-tuning. 48 A100-hours is cheap, but Chapter 9 says the first 560 hours of data carry almost everything — and generating that data well is a bigger job than the training run.

The ordering principle: every step produces a number that tells you whether to take the next one. The failure mode this avoids is the common one — six weeks of serving infrastructure for a model that turns out to interrupt customers.

The pre-flight checklist

Twelve things to have answers for before a duplex voice agent takes a real call. Every one traces to something in this lesson.

Item 12 is the one teams skip. It is entirely possible to ship an agent that answers every question correctly and that nobody can stand talking to, because it speaks into every pause. That failure will not appear in any content metric you are likely to have set up, and it is the first thing a customer notices.

Voice cloning: the part that needs a policy, not a feature flag

A model that reproduces a target speaker from ten seconds of audio is a voice-cloning system, whatever else it is. Three things belong in your plan.

Consent, structurally. The reference clip is just tokens in a prefix; the model cannot tell an authorised brand-voice recording from a clip scraped off a podcast. Whatever prevents misuse has to live in your system, before the audio reaches the prefix, because nothing in the model will help.

The authors' own precedent. They faced this and acted: the released checkpoint drops real-speaker training data entirely in favour of TortoiseTTS synthetic voices with Praat pitch/formant augmentation, explicitly "for data privacy." That is a research team choosing not to ship a model trained on identifiable people's voices — and, usefully, the substitution also improved speaker similarity to 0.65. Worth citing internally when someone argues the privacy-preserving option is always the worse one.

The fidelity ceiling is doing some work. 0.57–0.65 speaker similarity is "recognisably this person," not "indistinguishable from this person." For brand-voice deployment that is enough. For impersonating a specific individual to their bank it is meaningfully less than a dedicated cloning TTS would give you. That is an accident of the design rather than a safety mechanism — do not rely on it — but it is worth knowing which side of the line the artefact sits on.

Absent from the paper entirely: watermarking, and any anti-spoofing discussion. Both are standard in the neighbouring TTS literature. Their absence in a four-page ICASSP submission is unsurprising and their absence in a deployment would not be.

Chapter 11: Connections and the Cheat Sheet

PersonaPlex is a small paper sitting on top of a tall stack. Almost nothing in it would work without four earlier ideas, and the reason a four-page submission can make a real contribution is that it inherits all four intact. Here is the lineage, so you know what to read next and why.

What to remember in a year

Most of the numbers in this lesson will be stale within a year; several are already superseded by the released checkpoint in the paper's own appendix. Four things will not be:

The five things this lesson derived that the paper does not print

A short inventory, so the boundary between paper and lesson stays visible right to the end.

Every one is arithmetic on numbers the paper gives you. None required information the paper withheld. That is usually true, and it is the main reason to read a results table with a pencil rather than with your eyes.

What connects to what

Before the timeline, the dependency structure — which idea you need in order to understand which.

Four years in one table

PersonaPlex is one step in a fast-moving line. Placing it helps you predict what comes next.

Read the last column down the table. Each row's "still impossible" becomes the next row's contribution, and the rightmost cell of the PersonaPlex row is precisely the paper's stated future work. That is what a healthy research line looks like from the inside.

Where to go next

The three ideas, ranked

If this paper survives, it will be for one of these. They are worth ranking, because the ranking is not obvious.

Third: the results. 4.48 and 0.57 are good numbers today and will be ordinary numbers soon. Every leaderboard entry ages this way.

Second: the benchmark extension. Service-Duplex-Bench's probe taxonomy decomposes "holds a role" into seven measurable behaviours, and that decomposition is independent of any model. The authors say they plan to release it. Benchmarks routinely outlive the systems that motivated them.

First: the mechanism. A conditioning surface carved out of an existing timeline, costing nothing at steady state, expressive over arbitrary text and arbitrary audio, learnable from paired data alone. That idea has no version number. It will be reimplemented on top of whatever replaces Moshi, and then again on whatever replaces that.

Where this sits in the site

This lesson is one node in a longer arc about audio and voice, and the arc has a shape worth knowing so you can enter it at the right point.

If any row above this one felt shaky while reading, that is the row to go and fill in; the links in the table further up point at each. If the rows below look more interesting than this one, you are in the right place at the right time — that is where the field is going next, and this paper is one of the things making it possible.

Explaining this paper in 90 seconds, 5 minutes, and an hour

A test of understanding is whether you can compress to the audience. Three versions.

Ninety seconds, to a product manager. "Speech models that talk like people — interruptible, no awkward pauses — had one fatal flaw: you couldn't tell them who to be or what to sound like. There was no system-prompt field, because the model's only input is the conversation. PersonaPlex noticed you can write the persona into the conversation before it starts: a voice sample on the agent's audio track, the role description on its text track, a marker, then the real call. Six hours of fine-tuning on a big pile of generated dialogues, no architecture change. Role-following went from 1.75 out of 5 to 4.48, voice matching from basically zero to 0.57, and it still answers in 70 milliseconds."

Five minutes, to an engineer. Add: the three channels at 12.5 Hz; the forced silence on the agent-audio channel during the text segment, and why (text and audio are time-aligned, so unforced it would read the prompt aloud); the 440 Hz sine on the user channel, and why (silence is a turn-taking cue); loss masking on the prompt; the 0.02 / 0.3 reweighting and what it does to gradient shares; and the fact that a system prompt costs seconds, not tokens.

An hour, to a researcher. Everything above, plus: Full-Duplex-Bench's metric set and its anti-gaming pairings; the eagerness axis and floor discrimination; the Service-Duplex-Bench probe taxonomy and the grounding gap; the dataset-scale cliff at 25%; the released checkpoint's restraint/eagerness trade after adding Fisher; the effect sizes behind the DMOS claims; and the five open problems.

How to read the paper itself

If you go to the PDF now — and you should — here is the efficient order, given what you know:

1. Figure 1 first. You can now read every label. Ten minutes here is worth the rest of the paper.
2. Section 3.1, the four sentences of Chapter 3. Check our reading against the primary text.
3. Table 3, the Service-Duplex-Bench example. Notice the Q1 digit mismatch yourself.
4. Table 2, and immediately compute floor discrimination for each row. The leaderboard changes under your pen.
5. Appendix A, the released checkpoint. The most honest page in the document.
6. Section 3.2 last. It reads as logistics until you know what the pairing is for.

The cheat sheet

Everything you would need to re-derive this paper on a whiteboard.

The mechanism.

The objective.

The training run.

The metrics.

Further reading, in the order that helps

If you want to go deeper than this lesson, these are the primary sources and what each one gives you that the others do not.

The last word

Papers that add a module are common. Papers that find capacity already present in a system everyone had been using are rarer, and they tend to age better, because a mechanism with no parameters cannot become obsolete in the way a module can. The Hybrid System Prompt will outlive Moshi as an architecture, in the same way that "put the instructions in a delimited prefix" outlived the specific models it was first demonstrated on.

The other durable thing here is smaller and easier to miss: the probe taxonomy. Proper noun, context detail, context reasoning, unfulfillable request, rudeness, unspecified, unrelated. Seven categories that will still be the right seven when the models have changed twice over, because they are a decomposition of what it means to hold a role rather than a property of any particular system.

If this lesson leaves you with one instinct, let it be the one from Chapter 0's callout: before adding machinery, ask whether the interface you already have is wider than everyone assumed.

Every symbol, defined

Nothing in this lesson should still be a stranger. If any row surprises you, its chapter is one tap away.

Open problems this paper leaves you

Five things a reader is now equipped to work on. The first two are the paper's own stated future work; the rest fall out of the analysis in this lesson.

1. Tool use in a clocked model. The paper's final sentence names "integration with external tools." The hard part is not the API call; it is that a tool round-trip takes hundreds of milliseconds during which the model must keep producing audio frames — so tool calling in a duplex model is inseparable from the question of what to say while waiting.
2. Post-training alignment. There is no preference-optimisation stage here. What a duplex DPO or RLHF signal even looks like — preferences over timing as well as content — is an open design question.
3. Long sessions. 163.84 s of training context against ten-minute calls. Length extrapolation for a stream where silence occupies frames is not the same problem as long-context text.
4. Persistence and jailbreak resistance. Service-Duplex-Bench is single-turn by design. Nobody has yet measured whether Brody Murphy is still Brody Murphy after twelve turns of a customer insisting otherwise.
5. Domains beyond customer service. All 50 evaluation scenarios are service. The mechanism has no reason to be domain-specific; the evidence is.

What would change our mind

In the spirit of stating falsifiers rather than only evidence — four results that would force a retreat from this lesson's reading, none of them yet observed:

1. A voice-only or text-only prompt matching the hybrid prompt's numbers on both knobs. That would mean the two segments are not independent and "hybrid" is decorative.
2. Silence performing as well as the 440 Hz sine on the user channel. That would kill Chapter 3's central engineering argument.
3. Role adherence surviving a scenario distribution far from customer service — a game character, a therapist, a technical decision tree — without new data. That would be good news, and it would mean the 105,410-dialogue corpus was mostly unnecessary.
4. The pause-handling regression persisting after the Fisher data. It did not (0.584 → 0.358), which is the strongest single piece of evidence that the problem was distributional rather than architectural.

References

1. Roy, R., Raiman, J., Lee, S., Ene, T.-D., Kirby, R., Kim, S., Kim, J., Catanzaro, B. "PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models." NVIDIA, 2026. arXiv:2602.06053
2. Lin, G.-T., Lian, J., Li, T., Wang, Q., Anumanchipalli, G., Liu, A. H., Lee, H. "Full-Duplex-Bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities," 2025 — the 400 questions and every timing metric in Chapter 6.
3. Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., Zeghidour, N. "Moshi: a speech-text foundation model for real-time dialogue," 2024 — the architecture, Mimi, and the loss-reweighting convention.
4. Hu, K., Hosseini-Asl, E., et al. "SALM-duplex: Efficient and direct duplex modeling for speech-to-speech language model," 2025 — the prior work validating the negative-silence stitching method.
5. Xu, J., et al. "Qwen2.5-Omni technical report," 2025 · Wang, X., et al. "Freeze-Omni," 2024 · Google, "Gemini Live," 2025 — the baselines.
6. Chen, S., et al. "WavLM: Large-scale self-supervised pre-training for full stack speech processing," IEEE JSTSP 16(6), 2022 — the speaker-verification model behind SSIM.
7. Nagrani, A., Chung, J. S., Zisserman, A. "VoxCeleb," 2017 · Kang, W., et al. "Libriheavy," 2024 · Zen, H., et al. "LibriTTS," 2019 · Demirsahin, I., et al. "CommonAccent," 2020 · Cieri, C., Miller, D., Walker, K. "The Fisher corpus," 2004 — the voice pool.
8. Nari Labs, "Dia-TTS," 2025 · Resemble AI, "Chatterbox-TTS," 2025 · Betker, J., "Tortoise TTS," 2023 · Boersma, P., Weenink, D., "Praat" — the speech generation stack.
9. Ribeiro, F., Florêncio, D., Zhang, C., Seltzer, M. "CrowdMOS: An approach for crowdsourcing mean opinion score studies," ICASSP 2011 — the DMOS methodology.
10. Released checkpoint: nvidia/personaplex-7b-v1