A voice assistant that answers 1,600 ms late feels broken. One that answers in about 200 ms feels like a person. Pick a design and watch where every millisecond goes.
Learn how models that hear and speak are built from audio tokens, how data, size and compute shape them, and how they answer fast enough to hold a conversation.
Press each design in turn and watch when the reply starts after you stop talking. Then we build, piece by piece, what sits underneath: the tokens, the architectures, the scaling curves, the data and the streaming tricks.
You need a feel for what a language model does (it predicts the next piece of a sequence) and a little Python for the code. We build the rest from zero.
Based on eight papers, each linked in the Field Guide: Scaling Properties of Speech Language Models, Scaling Analysis of Interleaved Speech-Text Language Models, Moshi, Spirit LM, AudioLM, Generative Spoken Language Modeling, LLaMA-Omni and Mini-Omni.
The 226 ms figure is the response latency reported in the LLaMA-Omni abstract. The 160 ms theoretical and 200 ms practical figures are from the Moshi abstract. The cascade stages (500, 300, 400, 300 and 100 ms) are illustrative round numbers for a typical three-model pipeline, not measurements.
Chapter 0
See where every millisecond of a spoken reply goes, and why 200 ms is the number a voice model has to beat
You call a friend and ask, "Which mountain is the tallest in Africa?" Your friend answers almost at once. The gap is so short that you do not notice it. Now picture the same call with a one-second pause before every answer. You say "hello?" and wonder if the line dropped.
A voice assistant lives or dies by that gap. The hard part is a single number: how long after you stop talking does the first sound of the reply begin? We call it the response latency. Latency means delay, here measured in milliseconds (ms), where 1,000 ms is one second.
We need a target. The Moshi paper reports a theoretical latency of 160 ms and a practical latency of 200 ms for its voice model. We use 200 ms as our yardstick for the whole lesson. It is a measured result from a real system, so it shows what is possible, and it sits close to what feels like a live turn.
The LLaMA-Omni paper reports a response latency as low as 226 ms. So two research systems from different teams land in the same small range. The hero at the top of this page shows both against a typical older design. The older design is far away from the yardstick.
Most voice assistants began as a cascade: a chain of three separate models, each handing its result to the next. Think of three clerks passing a note down a corridor.
There is also a fourth job that people forget. Before step 1 can finish, the system has to decide that you have stopped. This is end-of-turn detection. You may only be taking a breath, so the system waits a little to be sure. That wait is part of your latency, and nobody sees it in a model benchmark.
Add up one turn of a cascade. The numbers below are illustrative round numbers, because real systems differ. They show the shape of the problem, not a measurement.
500 ms wait + 300 ms ASR + 400 ms first words + 300 ms first sound + 100 ms network = 1,600 ms
The total is 1,600 ms, eight times the yardstick. Each stage looks reasonable alone. The trouble is that they run one after another, so every delay adds. Make each stage twice as fast and you still wait 800 ms, four times too long.
Try it yourself below. Drag each stage and watch the bar against the dashed line. Then switch on Stream between stages. Streaming lets a stage start work as soon as the stage before it has produced a small piece. It helps a lot, but the endpoint wait is still there.
The bar is one turn of a cascade, one block per stage. The dashed line is the 200 ms yardstick. Drag any stage, or press a preset, then try the streaming switch.
Illustrative: the stage times are made-up round numbers. Streaming is modelled as each of the three model stages needing only 40% of its time before it can pass its first piece on; the endpoint wait and the network are unchanged.
Even with streaming, a cascade struggles to get near 200 ms, because three separate models must each start up and each pass a piece along. A single model that takes speech in and gives speech out removes two of the hand-offs. That idea drives the rest of this lesson.
To answer in 200 ms, a model has to start speaking while it is still working out the rest of the sentence. It must also keep listening while it talks, so you can interrupt. And it must do this without becoming a worse thinker than a text model. Each of those goals costs something, and the chapters ahead count the cost.
Other lessons here go deeper on single pieces. The audio language models lesson surveys the field. The codecs lesson builds the tokenizer from zero. The streaming speech lesson and the turn-taking lesson cover the live side. This lesson ties them to one question: how do size, data and compute decide how good, and how fast, a speech model can be?
Look again at the first block, the wait to be sure you stopped. Your voice is not a clean on-off switch. You pause between words. You breathe. You think for a moment in the middle of a sentence. The system has to guess which pauses are the end of your turn.
Suppose it waits for 200 ms of silence. It will often cut in while you are still thinking, and you hear the assistant talk over you. Suppose it waits 700 ms. It rarely cuts in, but every reply now starts at least 700 ms late. There is no setting that is both safe and fast. The only way out is a model that does not wait for the silence at all, one that listens to the meaning and the rhythm of your voice and starts early when it is sure. Chapters 3 and 8 return to this.
This is why the yardstick is a hard target. The 200 ms covers the thinking, the speech and the guess about your turn, all together. A design must earn every millisecond.
Chapter 1
Turn one minute of speech into tokens and count what it costs the model to hold them
Say the word "mountain" into your phone. A microphone does not hear a word. It measures air pressure, thousands of times each second, and writes down a long list of numbers. One second of ordinary phone-quality audio is a list of 16,000 numbers. A language model cannot do much with that list as it stands.
A language model reads a short list of choices, one at a time. For text, the choices are pieces of words. For speech, we have to build a similar list of pieces. This chapter shows how, and what it costs.
The idea in plain words: slice the sound into short frames, and give each frame a name from a fixed menu. A frame is a short slice of time, often 20 to 80 ms. The menu is a codebook: a numbered list of typical sound shapes, say 1,024 of them. For each frame, a trained network picks the entry that matches best, and writes down its number. That number is a token.
The network that does this is a neural audio codec. A codec has an encoder that makes the tokens and a decoder that turns tokens back into a waveform. Our codecs lesson builds one from zero. Here we only need the shape of what comes out.
One name per frame loses too much detail. So a codec adds a second codebook that names what the first one missed, then a third that names what the second missed. This is called residual vector quantization. Think of describing a street address: first the city, then the street, then the house number. Each step adds detail.
So each frame gets not one token but a small stack, one token per codebook. If a clip has T frames and the codec uses Q codebooks, the output is an array of shape (T, Q). The model must read or write all T times Q tokens.
# one second of speech, as a codec sees it (illustrative sizes) frame_rate = 50 # frames per second codebooks = 8 # tokens stacked on each frame codes = encoder(wave) # shape (50, 8): ints from 0 to 1023 # {"frame": 0, "codes": [512, 87, 1001, 33, 640, 9, 270, 815]} tokens_per_second = frame_rate * codebooks # 400
Codec tokens keep nearly everything about a voice: who is speaking, how loud, how they sound. That is good for playing the sound back, but there are many tokens. A second kind of token keeps mostly the content. These are semantic tokens, or units: one token per frame from a model trained to capture what was said, not how.
The Generative Spoken Language Modeling paper builds a whole system from such units. It has three parts: a speech encoder that produces discrete units, a language model trained on those units, and a decoder that turns units back into a waveform. It tests three encoders (CPC, wav2vec 2.0 and HuBERT) and three unit counts (50, 100 and 200). No text or labels are needed.
AudioLM combines the two kinds. A model pretrained on audio gives semantic tokens that hold long-range structure, such as which sounds follow which. A neural codec gives acoustic tokens that make the output sound good. The language model first writes the semantic tokens, then fills in the sound.
Now count. We use illustrative sizes: 50 frames per second and 8 codebooks. That is 400 codec tokens each second. One minute is 60 times 400, or 24,000 tokens. A semantic stream at one token per frame gives 50 per second, so 3,000 per minute.
Compare text. A person speaks about 150 words a minute, and a word is about 1.3 tokens in text. That gives roughly 200 text tokens for the same minute. The codec stream is about 120 times longer than the text it carries. Drag the sliders and see.
Three bars, one per kind of token, on a log scale: each gridline is ten times longer than the last. Change the clip length, the frame rate and the number of codebooks.
Illustrative: speech at 150 words a minute and 1.3 text tokens per word, one semantic token per frame, and a 32,000-token context window. Real codecs and tokenizers differ; the arithmetic (frames times codebooks) is exact.
With the starting values, 60 seconds becomes 24,000 codec tokens. A context window, the most tokens a model can read at once, of 32,000 tokens then holds only 80 seconds of speech. The same window would hold hours of text. This is the first big reason speech models are expensive: the sequence is long before the model has learned anything.
A longer sequence costs more in two ways. First, the model does more steps to read or write it. Second, attention compares every token with earlier ones, so the work grows with the square of the length. A sequence 120 times longer costs about 14,400 times more attention work. Smart designs attack the length, the number of codebooks, or both. The next chapters show how.
The codec also runs backwards. The decoder takes the (T, Q) array of codes and writes a waveform. Count what it saves. A raw waveform at 16,000 samples a second, with 16 bits per sample, is 256,000 bits a second. A codebook with 1,024 entries needs 10 bits to name one entry. Our example codec makes 400 tokens a second, so it needs 4,000 bits a second. That is 64 times smaller.
The saving is why codecs are worth the trouble. A language model could not read 16,000 numbers a second, but it can read 400 tokens. The cost is the quality of the copy. The decoder rebuilds a close match to the sound, not the exact sound, and fewer codebooks make a rougher match.
Chapter 2
Compare cascaded, end-to-end, thinker and talker, and full-duplex designs by what each one loses and gains
You say, in a flat, tired voice: "Oh, great." Your friend hears the words and the sigh, and knows something went wrong. A transcript of the same moment reads "Oh, great." and looks happy. Which parts of a voice survive depends on how the machine is built.
There are four main designs. We go through them in the order they were invented, and for each one we ask three questions. What does it read? What does it write? What gets lost along the way?
You met this design in Chapter 0. Three models hand a note down a corridor. The speech recognizer reads sound and writes text. The language model reads that text and writes more text. The speech synthesizer reads the text and writes sound.
The data that moves between the models is plain words. So everything that is not a word is dropped at the first hand-off: the sigh, the sarcasm, a cough, a second voice in the background. The model that thinks never hears you. It reads a transcript of you.
# cascade: three models, two text hand-offs text = asr(audio) # "oh great" (tone is gone) reply = llm(text) # "Sorry to hear that." audio_out = tts(reply) # spoken in a fixed voice
The cascade has one real strength. You can pick the best text model in the world for step two. All the intelligence of a large text model comes along for free.
The second design removes the hand-offs. A speech encoder turns your audio into vectors, one per frame. A language model reads those vectors directly. A speech decoder writes the reply as sound. LLaMA-Omni builds this on top of Llama-3.1-8B-Instruct and reports a response latency as low as 226 ms. It trained on 200,000 speech instructions in under three days on four GPUs. That is a small bill, because it reuses a text model that already knows how to think.
Nothing is turned into text on the way in. The tone of voice can reach the model. The price is that someone has to teach a text model to read audio vectors, and to write speech without forgetting how to write text. Chapter 9 returns to that risk.
The third design splits the work inside one system. A thinker works in text, the form a language model knows best. A talker turns what the thinker produces into speech tokens, while the thinker is still going. Think of a speaker with a note-taker beside them, whispering each next phrase as it is written.
Mini-Omni is an example of the idea. Its title says it: language models can hear and talk while thinking, in streaming. Its method, called "Any Model Can Talk", has the model produce text and audio tokens side by side, so the speech does not wait for the full written answer. The authors report that the approach keeps the base model's language skill with little loss, and they release a dataset, VoiceAssistant-400K, to train the speaking part.
All three designs so far take turns. You talk, the system waits, then it answers. People do not talk like that. We say "mm-hm" while the other person is mid-sentence, and we cut in when we are sure what they mean.
The Moshi paper builds a full-duplex model, one that listens and speaks at the same time. It models its own speech and your speech as parallel streams. There is no separate step that decides whose turn it is. The model simply keeps writing its own stream, and sometimes that stream is silence. The paper reports 160 ms of theoretical latency and 200 ms in practice.
Try the four designs below. Watch where the arrows change colour: blue arrows carry text, warm arrows carry sound.
Choose a design. A dot carries your question through it. Blue arrows carry text, warm arrows carry sound. Read what the design keeps and what it loses.
The boxes are simplified from the papers: LLaMA-Omni (speech encoder, adaptor, language model, streaming speech decoder), Mini-Omni (text and audio tokens produced side by side) and Moshi (parallel streams for its speech and yours). The cascade is the common three-model pipeline. Model counts are exact for those descriptions.
No design wins on every count. The cascade is easy to build and keeps a strong text brain, but it is slow and deaf to tone. The single model and the thinker-talker pair are faster and hear more. The two-stream model can be interrupted and can interrupt back, and it is also the most work to train.
The symptom: the assistant answers a sarcastic "Oh, great." with "Glad you are happy!" The measurement: build pairs of recordings with the same words and opposite tone, and count how often the answer changes with the tone. A cascade scores near zero on that test, because the text is identical. The fix: let the model read audio features, not only words, as the other three designs do.
Count the data that moves. In the cascade, one second of your speech becomes about 2.5 words, which is about 3 text tokens. That is all the language model sees. In the one-model design, the speech encoder sends the model one vector for every frame. At 50 frames a second and 1,024 numbers in each vector, that is an array of shape (50, 1024) for each second. It is far more than three tokens, and it carries the tone.
The thinker and talker pair sends something in between. The thinker sends text, and often its hidden states too, so the talker knows how to say the words and not only what they are. The two-stream model sends frames in both directions all the time.
Which design should you pick? If you need the smartest text brain today and can accept a slow reply, a cascade works. If you need speed and tone, pick one of the other three. If you need natural interruptions, pick two streams. Each step toward speed asks for more speech training, which is the cost Chapter 6 counts.
Chapter 3
Build an interleaved sequence by hand, then see how two parallel streams let a model listen while it speaks
Imagine a student who has read a million books but never heard anyone talk. Another student has heard a thousand hours of talk but has read little. The first knows facts and grammar. The second knows accents, rhythm and mood. A good speech model wants to be both students at once.
The trick that helps is to put text and speech in the same sequence, so one model learns from both. This chapter shows two ways to do it. The first mixes them in time, word by word. The second keeps them in parallel lanes.
To interleave is to weave two things into one line, like the teeth of a zipper. Spirit LM is a model that does this. It starts from a pretrained text language model with 7 billion parameters and continues its training on text and on speech units. Text and speech sit in a single token stream, and the training switches between them at word boundaries. The authors call this word-level interleaving. They use a small, automatically curated set of paired speech and text.
Why switch at words? Because the pairing tells the model that a spoken word and its written form mean the same thing. If the stream goes "the cat" in text and then the speech units for "sat on the mat", the model has to continue a spoken sentence that began in writing. That forces it to line up the two worlds.
Spirit LM comes in two versions. The Base version uses HuBERT phonetic units for speech. The Expressive version adds pitch and style units, so the model can carry how something was said, not just what. Text uses ordinary subword tokens in both. The authors report that the model combines the semantic strength of text models with the expressive range of speech models. It can also learn new tasks from a few examples across both forms, such as speech recognition, speech synthesis and speech classification.
Take the sentence "the cat sat on the mat". Suppose we switch between text and speech every two words. The text words stay as they are. Each spoken word becomes a short run of speech units. We use three units per word as an illustrative figure. Two special marker tokens, written [TEXT] and [SPEECH], tell the model which form comes next.
The sentence is six words. Blue chips are text tokens, warm chips are speech units. Drag how many words come before each switch, and turn on the expressive units.
Illustrative: three speech units per spoken word, one extra pitch unit and one extra style unit per spoken word in the expressive mode, and unit numbers made up for display. Spirit LM’s abstract describes one stream of text and speech with word-level interleaving, HuBERT phonetic units in the base version, and pitch and style units in the expressive version. The marker names and every count here are a teaching toy.
At 2 words per switch, the sentence has three segments: text, then speech, then text again. Turn on the expressive units and each spoken word grows from 3 tokens to 5. That is the price of carrying tone: a longer sequence, as Chapter 1 predicted.
Interleaving still takes turns. At any moment the model is in one form, text or speech. It cannot hear you while it speaks. Moshi takes a different route. It models its own speech and your speech as parallel streams: two lanes that run side by side, frame after frame.
At every time step, the model reads the newest piece of your stream and writes the newest piece of its own. When it has nothing to say, its stream holds silence tokens. When you speak over it, your lane fills up and it can decide to stop. There is no turn manager, because the lanes never stop running.
Moshi adds a third lane for text. Before each stretch of its own audio, the model predicts the matching words as a prefix. The paper calls this an inner monologue. The text acts like a quiet draft that guides the sound, which the paper says improves the quality of the speech. The paper reports 160 ms of theoretical latency and 200 ms in practice.
# one step of a full-duplex model (shapes illustrative) user_tokens = codec.encode(mic_frame) # (Q,) codes for your newest frame state = model.step(user_tokens, prev_own_tokens) text_tok = state.text_head() # the inner monologue word piece own_tokens = state.audio_head() # (Q,) codes for its newest frame speaker.play(codec.decode(own_tokens)) # one frame later, you hear it
Look at what the loop implies. The model never waits for you to finish. It writes one frame for every frame it reads. That is how it can answer in 200 ms: the answer may start while your last word is still dying away.
Where does an interleaved example come from? You need a recording and a transcript whose words are matched to their times. A tool called a forced aligner does the matching. It tells you that the word "cat" runs from 0.31 s to 0.58 s. With those times you can cut the audio at word boundaries, and the speech units on either side of the cut line up with the words.
# words with start and end times, from a forced aligner words = [("the", 0.00, 0.18), ("cat", 0.18, 0.58), ("sat", 0.58, 0.95)] stream = [] for i, (w, t0, t1) in enumerate(words): if (i // n) % 2 == 0: # text segment stream += ["[TEXT]"] * (i % n == 0) + [w] else: # speech segment stream += ["[SPEECH]"] * (i % n == 0) + units(audio[t0:t1])
The failure mode sits at the seams. The symptom: the model babbles or repeats a word right after a switch. The measurement: compute the loss on the first few tokens after each switch and compare with the loss elsewhere. A spike at the seam means the cuts do not line up with the words. The fix: cut only on the aligner's word boundaries, and drop any example whose alignment looks wrong.
Before you move on, notice what the two ideas cost. Interleaving makes the sequence longer, by the extra tokens the speech needs. Parallel streams keep it short but make the model run two lanes at once. Neither is free, and Chapter 7 shows how the cost turns into delay.
Chapter 4
Read a scaling curve for speech and see why the same linguistic skill costs far more compute than in text
A team trains a small text model and a small speech model. Both cost the same to train. The text model writes tidy grammar. The speech model babbles. The team doubles the budget, then doubles it again. The text model improves a lot. The speech model improves a little. Why?
To answer, we need a tool that researchers call a scaling law. It is a rule that predicts how good a model gets as you spend more. The next three paragraphs teach what you need.
Compute is the total arithmetic spent training a model. We count it in FLOPs, floating-point operations, and the numbers are huge: 1021 is a 1 followed by 21 zeros. Loss is the model's average error on held-out data. Lower is better.
Plot loss against compute on log-log paper, where each gridline is ten times bigger than the last. A neat thing happens. The points fall on a nearly straight line. A straight line on log paper means a power law: every time you multiply compute by 10, loss drops by the same fixed fraction. Once you have the line, you can read off how much compute buys a target loss. Our scaling laws lesson builds this in full.
One more link makes it useful. Loss is hard to feel, so people also test models on tasks, like telling a grammatical sentence from an ungrammatical one. The Scaling Properties of Speech Language Models paper reports that the pre-training loss correlates strongly with these syntactic and semantic scores, in both speech and text models. That means loss predicts skill. If you can predict loss, you can predict skill before you spend the money.
The same paper then compares speech with text. Its finding is blunt: speech language models improve linguistically more slowly than text models, by as much as three orders of magnitude. An order of magnitude is a factor of ten, so three orders is a factor of a thousand. The authors use the trends to estimate how much scale current methods would need to reach the English skill of text models. The size of the gap is why that estimate is a very large number.
Why would speech be so much worse per unit of compute? The paper does not give one cause, but Chapter 1 shows a likely clue. A minute of speech is about 24,000 codec tokens, while the same words are about 200 text tokens. Most speech tokens describe voice, noise and timing, not meaning. The model spends its budget on detail that a grammar test does not need. The words are there, but they are spread thin.
The paper also looks at two helpers. One is synthetic data, made to boost semantic understanding. The other is coarser speech tokenization, which gives fewer tokens per second. Both are aimed at the same suspect: too little meaning per token. This lesson treats that as a likely reading, not the paper's proven claim.
The device draws two lines, one for text and one for speech. The speech line is the text line moved to the right by the gap you choose. Slide the compute marker and read the scores. Then pick a gap of 1,000 and see how far right the speech line must go to catch up.
The x axis is compute on a log scale. The y axis is a linguistic score. Pick how many times more compute speech needs, then drag your compute.
Illustrative: both lines are straight on this scale and the score is a made-up 0 to 100. Only the idea is the paper’s: its abstract says speech models improve linguistically as much as three orders of magnitude more slowly than text models, which is why the 1,000 times stop is the largest here. It does not give a compute figure.
Run the numbers once by hand. Say a text run of one week reaches a score of 40. Speech at a gap of 1,000 needs 1,000 weeks, about 19 years of the same cluster. Nobody waits 19 years. So the field looks for ways to shrink the gap, and the next chapter shows the biggest one.
A researcher fits the line on small runs and then reads off the compute for a target. The data below are made up to show the steps.
import numpy as np C = np.array([1e18, 1e19, 1e20, 1e21]) # FLOPs of four small runs L = np.array([3.20, 2.90, 2.63, 2.38]) # their final loss (illustrative) slope, icpt = np.polyfit(np.log10(C), np.log10(L), 1) need = lambda target: 10 ** ((np.log10(target) - icpt) / slope) print(slope, need(1.8)) # compute for loss 1.8
Fit on a text model and use the answer for speech, and you under-budget by up to the gap. The symptom is a run that ends with a loss far above the forecast. The measurement is to fit speech runs on their own and check that loss still predicts the benchmark score. The fix is to forecast from speech runs, never from text ones.
Suppose four small runs at 1018 to 1021 FLOPs give the losses in the code above. The slope of the fitted line comes out near −0.043, meaning each tenfold increase in compute multiplies the loss by about 0.91. To reach a loss of 1.8 you need to go from 2.38 down to 1.8, a factor of 0.76. That takes about 2.8 tenfold steps beyond 1021, so about 7 × 1023 FLOPs. The forecast is only as good as the straight line, which is why you need runs that span several powers of ten.
Coarser tokens are another way to shrink the gap. Halve the frame rate and you halve the tokens for the same sound. The sequence is shorter, each token carries more, and the model sees more speech for the same compute. The price is detail, because fewer tokens must describe the same second.
A last point about method. The paper uses scaling-law models fitted to its runs to estimate the scale a speech model would need. When you use a forecast like this, say how far you went past your data. A line that works over three powers of ten may bend at ten.
Chapter 5
Spend a fixed compute budget on model size or on tokens and see how starting from a text model moves the best split
Two cooks have to learn a new cuisine in one month. The first cook has never held a knife. The second cook is a trained chef who already knows how to cook, and only needs to learn the new spices. Who needs less practice? You do not need a stopwatch to guess.
A speech model has the same choice. It can start from nothing and learn language from sound alone. Or it can start from a trained text model and learn to link speech to what that model already knows. Chapter 4 showed that the first path is very expensive. This chapter asks how much the second path saves.
The paper Scaling Analysis of Interleaved Speech-Text Language Models starts from the earlier finding. Older scaling studies predicted that speech models need much more compute and data than text models. But modern speech models are often not trained from scratch. They are initialised from a pretrained text language model and trained on interleaved speech and text, the weave you built in Chapter 3. The authors ask whether this setup scales more efficiently than a textless model, one that sees only speech units.
Their answer is yes. They report that speech models scale more efficiently with compute under the interleaved setup. They trained several dozen models to get there. They also find that the scaling dynamics differ. The best use of a fixed budget moves toward a larger model trained on relatively fewer tokens. They study synthetic data and the choice of text model family as well. A scaled-up model reaches semantic speech performance comparable to leading models while using less compute and data, and they release models, samples and data.
To see why the split matters, we need one rule of thumb. Training a model with N parameters on D tokens costs roughly 6 times N times D FLOPs. Each token goes forward and backward through every parameter. About two operations per parameter go forward and four go back, which gives the 6.
C ≈ 6 × N × D
With C fixed, a bigger N means a smaller D. You must pick a point on that tradeoff. Too small a model cannot use the data. Too few tokens leaves a big model half-trained. The best point balances the two, and researchers call it compute-optimal.
Now a toy that shows the shape of the paper's claim. We write the loss as a floor plus a size penalty plus a data penalty: L = E + A/Na + B/Db. For the textless model we pick B large, because it must learn language from speech tokens alone. For the model that starts from text we pick B smaller, as a stand-in for the language it already knows. Nothing else changes. Drag the sliders and watch the best model size move.
Each curve is loss against model size at a fixed budget. Lower is better. The dots mark the best size for each. Change the budget, then drag your own model size.
Illustrative toy, not the paper’s fit: L = 1.7 + 400/N0.34 + B/D0.28 with D = C/(6N), and B = 410 for textless, B = 150 for the text start. Only the direction is the paper’s: its abstract says the interleaved setup scales more efficiently and suggests giving notably more of the budget to model size relative to training tokens.
At a budget of 1022 FLOPs the toy's textless optimum is a model of about 5 billion parameters trained on about 335 billion tokens. Check the arithmetic: 6 × 5 × 109 × 3.35 × 1011 is about 1022. The text-start optimum is about 26 billion parameters on about 65 billion tokens. The model is about five times larger and sees about five times fewer tokens, and its loss is lower, 1.96 against 2.14.
That mirrors the paper's advice. A model that already knows language needs fewer speech tokens, so a bigger share of the budget buys parameters. Chapter 6 asks where those tokens come from.
import numpy as np C = 1e22 N = np.logspace(8, 11.5, 300) # candidate model sizes D = C / (6 * N) # tokens the budget allows loss = 1.7 + 400 / N**0.34 + 150 / D**0.28 # toy, text start best = loss.argmin() print(N[best], D[best], loss[best]) # about 2.6e10, 6.5e10, 1.96
The failure mode here is copying a text recipe. The symptom: a speech run ends with a small model that has seen far too many tokens and stalls. The measurement: run the sweep above on your own small runs and see where the minimum sits. The fix: move the model size toward the minimum you measured, not the one a text paper used.
Drop the budget by a factor of ten, to 1021 FLOPs, and run the sweep again. In the toy the textless optimum falls to about 1.8 billion parameters on about 93 billion tokens. The text-start optimum falls to about 9 billion parameters on about 18 billion tokens. The ratio between the two best sizes is about 5 again. The shape holds as the budget changes, which is what makes the sweep a useful planning tool.
Why does the paper need several dozen models? Because every point on a scaling curve is one finished training run. To see the bend of the curve you need runs at several sizes and several token counts. You also need to repeat them for each choice you want to compare, such as the text model family you start from, or whether you add synthetic data. The authors study both choices, so a team that picks a start model without a small sweep is guessing.
For a team planning a real run, the lesson is practical. Do not copy the model size from a text recipe. Spend about one percent of the budget on small runs. Fit the curve for your tokens and your start model. Then place your big bet where the curve says.
Chapter 6
Count how many tokens an hour of speech costs, then choose a mix of speech, text, interleaved and synthetic data
Picture a pantry. One shelf holds huge sacks of flour: plentiful and cheap, but flour alone is not dinner. Another shelf holds a few jars of a rare spice. A speech model is cooked from a pantry too, and the quality of the meal depends on how much of each sack goes in the pot. This chapter opens the pantry.
Raw speech is audio with no labels. It is the flour: there are enormous amounts of it, and it costs little. Generative Spoken Language Modeling and AudioLM both learn from raw audio with no transcripts or annotations. They show that a model can pick up the structure of a language from sound alone. It teaches the model what speech sounds like. It does not directly teach what is true about the world.
Text is written language, and the web has far more of it. A model that starts from a text model, as in Chapter 5, carries all of that knowledge in its weights. Text keeps the model clever.
Paired and interleaved data link the two. Here a spoken word sits next to its written form, like the weave in Chapter 3. Pairs are rare because someone, or something, must align the audio to the words. Spirit LM uses a small, automatically curated speech-text parallel corpus for its word-level interleaving. A small set can go a long way, because its job is only to teach the link.
Synthetic data is made by a machine. A text-to-speech model reads written questions and answers aloud, which makes speech data for any text we already have. Both scaling papers study synthetic data. The first looks at synthetic data built to boost semantic understanding. The second studies its role in the interleaved setting. Two instruction sets show the idea in practice: LLaMA-Omni's InstructS2S-200K has 200,000 speech instructions, and Mini-Omni's VoiceAssistant-400K has a name that carries its size.
Here is what one training record of that kind can look like. The values are made up to show the fields.
{"id": "ex_0192", "source": "synthetic_tts",
"user_audio": "u/0192.wav", "user_text": "What is the tallest mountain in Africa?",
"reply_text": "Kilimanjaro, at about 5,895 metres.",
"reply_audio": "r/0192.wav", "reply_codes_shape": [250, 8]}
Data for text is counted in tokens, and data for speech is counted in hours. To compare, convert. At 400 codec tokens per second (the Chapter 1 example), one hour is 3,600 seconds times 400, or 1.44 million tokens. A budget of 100 billion speech tokens is then 100 billion divided by 1.44 million, about 69,444 hours of audio. That is nearly eight years of non-stop talking.
Now compare with an instruction set. If each of the 200,000 spoken instructions lasted 10 seconds, the set would hold about 556 hours, or 0.8% of that budget. It is tiny next to raw speech. Its value is not volume. It is shape: it teaches the model to answer in speech.
The device below lets you choose a mix of the four kinds and see the hours it asks for. Each slider is a weight, and the shares are the weights divided by their sum.
The bar shows your mix of a fixed token budget. Raise a weight and the others shrink. The read-out turns the speech shares into hours of audio.
Illustrative: the mixes and the budget are made up, and a codec at 400 tokens a second (50 frames times 8 codebooks) turns tokens into hours. Interleaved pairs count their speech half as speech tokens, taken here as half of that share. The papers name these kinds of data but give no fixed recipe in their abstracts.
With the starting mix, 1,000 billion tokens give 200 billion raw speech tokens, 100 billion synthetic, and 100 billion of speech inside the pairs. That is 400 billion speech tokens, or about 277,778 hours. Move the text slider up and the hours fall. Move it down and the model sees less of the language it needs.
The symptom: a model that talks beautifully in the lab and stumbles on a real caller with an accent and a noisy room. The measurement: score the model on real recorded voices and on synthetic ones separately. A big gap between the two scores is the warning. The fix: keep real speech in the mix and hold out real voices for testing. Our audio data pipelines lesson shows how to collect and clean them.
Take a ten-second recording of someone reading a book aloud. You have the audio and the book text. A forced aligner (Chapter 3) marks each word with a start and an end time. Now you can produce interleaved examples from one recording in many ways, by choosing different places to switch. That is why a small parallel corpus goes far: each recording makes many training sequences.
Synthetic data works the other way round. You start with text, which is abundant, and use a text-to-speech model to read it aloud. The text knowledge of a large language model then reaches the speech side for free. The danger is that the synthetic voice has a narrow range of accents and noise, so mix it with real recordings and test on real voices.
The mix has a second use. It is a budget you can write down and review. A line such as "50% text, 20% raw speech, 20% interleaved, 10% synthetic" is a decision, with a reason, that a teammate can challenge. Keep it in the run's config file, with the hours each line implies.
Chapter 7
Add up the stages before the first audio chunk plays, then shrink them with streaming, chunking and parallel codes
You order a coffee. The barista could wait until all the beans are ground, the water is hot and the whole cup is finished before saying a word. Or they could say "coming right up" the moment they start. Both baristas make the same coffee. Only one feels fast.
A voice model has the same choice. It can finish the whole reply, then play it. Or it can start playing the first sound while it is still working on the rest. This chapter is about the second way, and about the number that measures it: the time to first audio, the delay from the end of your question to the first sound of the answer.
1. A streaming codec. Some codecs need the whole clip before they can decode it. A streaming codec looks only at the past, never the future, so it can turn tokens into sound one small piece at a time. Without it, the first sound cannot play until the last token exists.
2. Chunked decoding. The model does not decode token by token, and it does not wait for all of them. It collects a small chunk of frames, say 4 frames, then sends the chunk to the codec decoder and plays it. A small chunk starts sooner. A large chunk wastes less effort on set-up. The best size is a trade.
3. Parallel token prediction. Chapter 1 showed that each frame carries a stack of codebook tokens. If the model writes the 8 tokens of one frame one after another, it takes 8 steps per frame. If it predicts the whole stack at once, it takes 1. Mini-Omni goes further. It generates text and audio tokens in parallel, and it uses batch-parallel inference to improve quality while keeping the speech streaming.
4. Speculative guessing. A small, quick model guesses the next few tokens, and the big model checks all the guesses in one pass. Right guesses are free. Our LLMs for builders lesson works through speculative decoding step by step.
Here is the time to first audio as plain addition. First the model reads your question: the prefill, in milliseconds. Then it writes the first chunk: chunk frames times steps per frame times the time of one step. Then the codec decoder turns the chunk into sound.
first audio = prefill + (chunk frames × steps per frame × step time) + codec decode
Take illustrative numbers: 60 ms of prefill, a chunk of 4 frames, 10 ms per step and 15 ms of codec decode. With 8 codebooks written one after another, that is 60 + 4 × 8 × 10 + 15 = 395 ms. Predict all 8 at once, and it is 60 + 4 × 1 × 10 + 15 = 115 ms. The same model gets close to the yardstick, only by changing how it writes each frame.
There is a second test, and it is easy to forget. The sound must keep coming faster than you play it. Otherwise the audio stutters. The real-time factor is compute time spent per second of audio. Below 1 the stream holds. Above 1 it falls behind. Play with all the knobs.
Top bar: the stages before the first sound, against the 200 ms line. Bottom bar: the real-time factor, against 1. Change the chunk, the codebooks and how they are written.
Illustrative: 50 frames a second, 15 ms for the codec to decode one chunk, and the real-time factor is (steps a second times step time, plus chunks a second times 15 ms) divided by 1,000. The 200 ms line is the practical latency Moshi reports in its abstract; LLaMA-Omni reports 226 ms. The stages here are a teaching model, not either system’s breakdown.
Start with sequential codebooks and the stream cannot keep up: 8 steps for each of 50 frames is 400 steps a second, which costs 4,000 ms of compute for each second of audio. The real-time factor is about 4.2. Switch to parallel and it falls to about 0.7. Now push the chunk to 1 frame: the fixed codec cost is paid 50 times a second and the factor climbs back above 1. Fast first sound and steady sound pull in opposite directions.
def first_audio_ms(prefill, chunk, steps_per_frame, step_ms, decode_ms): return prefill + chunk * steps_per_frame * step_ms + decode_ms def real_time_factor(fps, chunk, steps_per_frame, step_ms, decode_ms): work = fps * steps_per_frame * step_ms + (fps / chunk) * decode_ms return work / 1000 # must stay below 1 or the audio stutters print(first_audio_ms(60, 4, 1, 10, 15)) # 115 print(real_time_factor(50, 4, 1, 10, 15)) # 0.6875
Look for the failure in the numbers. The symptom: the first word arrives on time, then the voice breaks into gaps. The measurement: log the real-time factor over a whole reply, not just the first chunk. The fix: bigger chunks, fewer steps per frame, or a faster step.
One more cost hides inside the codec. Some streaming codecs look a few frames into the future, to make the sound smoother. Each frame of look-ahead adds one frame of delay. At 50 frames a second a frame is 20 ms, so a codec with 2 frames of look-ahead adds 40 ms before any sound can play. A strictly causal codec, which looks only backward, adds none, but its sound may be a little rougher.
Now see how speculative guessing fits in. Say a small model guesses the next 4 tokens of a frame stack and the large model checks them in one pass. If 3 guesses are right, the large model has produced 4 tokens, 3 accepted and 1 corrected, in the time of a single pass. In our sum, that cuts the number of steps per frame, so it shrinks the middle term.
All of these tricks work on the same sum. Prefill is cut by caching. The write step is cut by parallel codes and guesses. The decode step is cut by streaming and small chunks. When a team reports a latency win, ask which term of the sum it attacked.
Chapter 8
Measure response gaps and interruptions the way a listener feels them, and see how an average hides the slow turns
A restaurant says: "Our average wait for a table is 12 minutes." That sounds fine. But on Saturday nights, one party in ten waits an hour, and they never come back. The average hid them. Voice assistants fall into the same trap.
You now know how to build a fast voice model. How do you know it is fast, and fast for whom? This chapter is about measuring a conversation the way a listener feels it.
Treat a conversation as a log of events with times. For each turn, write down when the user stopped talking and when the first sound of the reply began. The difference is the response gap, also called turn-taking latency. Measure it at the speaker, not inside the server. Anything the server cannot see still reaches the listener's ears. That includes the wait to decide you finished and the trip across the network.
{"turn": 7, "user_end_ms": 12480, "reply_start_ms": 12690,
"gap_ms": 210, "barge_in": false}
{"turn": 8, "user_end_ms": 31020, "reply_start_ms": 32490,
"gap_ms": 1470, "barge_in": false}
A gap can be negative. If the model starts talking before you finish, the gap is below zero. That is either a good quick answer or an interruption of you. A full-duplex model such as Moshi makes this ordinary, because its stream never stops.
Take 40 turns. Thirty-six take 200 ms. Four take 1,500 ms. The average is (36 × 200 + 4 × 1,500) / 40 = 330 ms. That looks close to the yardstick. The median, the middle value, is 200 ms. The 95th percentile (p95), the value that 95 turns in 100 stay under, is 1,500 ms. One turn in ten is a long, awkward silence. Users remember the slow turns.
So report three numbers, not one: the median for the typical turn, the p95 for the bad turn, and the share of turns over a limit you choose. In the device below, 40 turns are drawn from a mix of quick and slow ones. Raise the slow share and watch the three numbers part ways.
Each bar counts turns whose gap falls in a 100 ms bin. The lines mark the mean, the median and the p95. Drag the share of slow turns.
Illustrative: quick turns are drawn between 150 and 250 ms and slow turns between 900 and 2,000 ms from a fixed seed, so the same slider position gives the same turns on every visit. The 200 ms line is the practical latency Moshi reports.
At 4 slow turns the mean is about 350 ms and the median about 215 ms, but the p95 is about 1,557 ms. At 8 slow turns the mean is about 484 ms while the median barely moves, near 220 ms. The median hides the slow turns too. Only the p95 and the share of turns over a limit show them.
Gaps cover the model's turn. People also cut in. Three numbers describe how a system handles that. The stop time is how long the model keeps talking after you start. The miss rate is how often you cut in and the model keeps going. The false stop rate is how often the model quits for a cough or a "mm-hm". A good system keeps all three low. You cannot get this from a transcript. You need a labelled recording of overlapping speech. Our turn-taking lesson goes deeper.
import numpy as np gaps = np.array([t["reply_start_ms"] - t["user_end_ms"] for t in turns]) report = { "median": np.percentile(gaps, 50), "p95": np.percentile(gaps, 95), "over_600": (gaps > 600).mean(), # share of turns that feel slow "talk_over": (gaps < 0).mean(), # share where it spoke first }
The symptom: a model tops a leaderboard, yet users call it slow. The measurement: check where the clock starts and stops. Many reports time only the model call, on short clips, with no network and no wait to decide you stopped. Ask for the gap at the speaker, the p95 and the setup. The fix: publish all three, with the number of turns behind them.
Here is one barge-in, written as events. The assistant is speaking. At 3,000 ms you start to talk. The system notices your voice at 3,180 ms and the assistant's audio stops at 3,320 ms. The stop time is 3,320 − 3,000 = 320 ms. During those 320 ms you heard the assistant talk over you.
{"turn": 12, "barge_in": true, "user_start_ms": 3000,
"detect_ms": 3180, "model_silent_ms": 3320, "stop_time_ms": 320}
Now the opposite case. You cough at 5,000 ms, and the assistant stops. Nothing you said needed an answer, so that is a false stop. Count it. If it happens in 6 of 50 coughs and side comments, the false stop rate is 12%. A system tuned to stop fast on any sound will have a high false stop rate. A system tuned never to stop wrongly will have a long stop time. Report both, as a pair.
To measure these numbers you need recordings with the events marked: who spoke when, and which overlaps were real interruptions. People label them by hand on a sample, then a scoring script does the rest on the whole set. Label at least a few hundred turns, because a rate computed on twenty turns will swing wildly from one run to the next.
Also test under the conditions of use. Replay the same recordings with added room noise and with a slow network. A system that holds a 200 ms median in a quiet lab and a 700 ms median on a phone call has two answers to the question "how fast is it?", and users live in the second one.
Chapter 9
Match each symptom to the measurement that catches it and the fix that repairs it
A talking model can fail in ways that a text model never does. It can sound perfect and say something silly. It can say something clever in a voice like a tired robot. It can win a contest and still feel slow in your hand. Each failure has the same three parts: a symptom you notice, a measurement that proves it, and a fix.
Symptom. You teach a strong text model to talk. Afterwards it answers a maths question with a confident wrong answer that the text version got right. This is catastrophic forgetting: new training overwrites old skill, the way learning a new route can blur an old one.
It happens because speech training pushes the weights toward sound, and nothing pulls them back toward text. The Mini-Omni paper treats this as a design goal. Its training method aims to keep the base model's language ability with minimal degradation, and its "Any Model Can Talk" method has the model produce text first to guide the speech.
Measurement. Run one fixed text benchmark on the model before and after speech training. The drop is the cost. Fix. Keep text in the training mix (Chapter 6), train the new parts first while the original weights stay frozen, and let the text lead the speech.
Slide the share of text in the mix and watch the benchmark score. The curve below is a toy that shows the shape, not a measurement of any model.
The dashed line is the text model’s score before speech training. The curve is the score after, for each share of text kept in the mix.
Illustrative: a made-up text benchmark score of 70 before speech training and a made-up retention curve, 70 × (0.55 + 0.45 × (1 − e−t/12)). The papers report that their methods limit this loss; their abstracts give no such curve.
Symptom. Every reply, happy or sad, comes out in the same even tone. This is prosody collapse. Prosody is the music of speech: pitch, rhythm, loudness and emphasis. Say "You did WHAT?" and the stress on the last word carries the meaning.
It happens when the tokens leave the music out. Spirit LM shows the cause and the cure in one pair of models. The Base version uses HuBERT phonetic units, which keep mostly what was said. The Expressive version adds pitch and style units, so the model can carry how it was said. The authors built the second version to model expressivity.
Measurement. Compare how much pitch varies across many replies against recordings of people. A flat voice has a small spread. Fix. Give the model tokens, or features, that hold pitch and style, and accept the longer sequence that costs (Chapter 3).
Chapter 8 covered this: an average, or a clock that starts too late, hides the slow turns. Add one more case, the stutter from Chapter 7, where the first sound is quick and the rest falls behind.
Six symptoms follow. For each, pick the measurement that would catch it. After you pick, you see the reasoning and the fix.
Read the symptom, then press the measurement you would run first.
The six cases are typical, written for this lesson. The mechanisms behind them come from the papers named in this chapter and in Chapters 2, 6, 7 and 8.
Work the forgetting curve by hand. With no text in the mix the toy retains 70 × 0.55 = 38.5 points of the original 70. With 10% text it retains about 56.3. With 30% text, about 67.4, and with 50% about 69.5. The first 10% of text buys back about 18 points. The next 40% buy back about 13 more. The first slice of text is the most valuable, so even a small share is worth keeping.
For the voice, the measurement is a number you can compute. Take the pitch track of each reply, in semitones, and compute its spread. Compare the spread of the model's replies with the spread of human recordings of the same kind of speech.
import numpy as np def pitch_spread(f0_hz): # f0 track of one reply, voiced frames only semis = 12 * np.log2(f0_hz / np.median(f0_hz)) return semis.std() # bigger means livelier ratio = np.mean([pitch_spread(r) for r in model_replies]) / np.mean([pitch_spread(h) for h in human_replies]) # a ratio far below 1 means the voice has gone flat
For hidden latency, run the arithmetic on a leaderboard claim. A report says 180 ms. It timed only the model call, on 2-second clips, with no network and no wait to decide you stopped. Add the 500 ms wait and 100 ms of network from Chapter 0, and the same system gives 780 ms at the speaker. The model did not get slower. The clock moved.
The three failures share a lesson. Each one passes the test you were already running. The text benchmark was not run after speech training. The words were scored and the pitch was not. The model call was timed and the speaker was not. Add the missing test, and the failure appears.
Chapter 10
Keep the numbers, the terms, the checklist, the core code and the sources on one page
Here is the whole lesson in one paragraph. A model that talks must turn sound into tokens, and speech makes far more tokens than text for the same words. That one fact makes speech models slow to improve with compute, up to three orders of magnitude more slowly than text models. Starting from a text model and interleaving text with speech makes the spending far more efficient, and shifts the best budget toward a bigger model. To answer in about 200 ms, a model must stream: small chunks, parallel codes, a codec that looks only at the past, and a design that listens while it speaks. Then you must measure the whole conversation at the speaker, with tails, interruptions and a text benchmark kept in view.
| Number | What it is | Where it comes from |
|---|---|---|
| 160 ms | Moshi, theoretical latency | Moshi abstract |
| 200 ms | Moshi, practical latency, our yardstick | Moshi abstract |
| 226 ms | LLaMA-Omni response latency | LLaMA-Omni abstract |
| 200K, 8B, under 3 days, 4 GPUs | LLaMA-Omni: speech instructions, base model size, training time, hardware | LLaMA-Omni abstract |
| 7B | Spirit LM starts from a 7 billion parameter text model | Spirit LM abstract |
| Up to 3 orders of magnitude | How much more slowly speech models improve than text models | Scaling Properties abstract |
| 50, 100, 200 | Unit counts tested with CPC, wav2vec 2.0 and HuBERT | GSLM abstract |
| 400K | Size in the name of Mini-Omni’s VoiceAssistant dataset | Mini-Omni abstract |
| 400 tokens a second | 50 frames times 8 codebooks, our example codec | Illustrative |
| 1.44 million tokens | One hour of speech at 400 tokens a second | Arithmetic on the example |
| 6 N D | Training FLOPs for N parameters and D tokens | Rule of thumb |
| Term | Plain meaning |
|---|---|
| Response gap | Time from the end of your speech to the first sound of the reply. |
| Codec token | A number naming a short slice of sound, stacked over several codebooks. |
| Semantic token | A number naming what was said in a slice, with less of how it sounded. |
| Interleaving | Weaving text and speech tokens into one stream. |
| Full duplex | Listening and speaking at the same time, in parallel streams. |
| Time to first audio | Prefill plus the first chunk plus decoding, before sound plays. |
| Real-time factor | Compute time per second of audio. Keep it below 1. |
| Prosody collapse | A voice that loses its pitch, rhythm and stress. |
import numpy as np def codec_tokens(seconds, fps=50, codebooks=8): return seconds * fps * codebooks # Chapter 1 def best_split(C, B, A=400, E=1.7): N = np.logspace(8, 11.5, 300); D = C / (6 * N) loss = E + A / N**0.34 + B / D**0.28 # Chapter 5, toy constants i = loss.argmin(); return N[i], D[i], loss[i] def first_audio_ms(prefill, chunk, steps_per_frame, step_ms, decode_ms): return prefill + chunk * steps_per_frame * step_ms + decode_ms # Chapter 7 def gap_report(turns): g = np.array([t["reply_start_ms"] - t["user_end_ms"] for t in turns]) return {"median": np.percentile(g, 50), "p95": np.percentile(g, 95), "over_600": (g > 600).mean()} # Chapter 8
This lesson joins four threads. Tokens come from neural audio codecs. Scaling comes from scaling laws. Live behaviour comes from streaming speech and voice turn-taking. Data comes from audio data pipelines. For the wider field, read audio language models. For the papers in depth, open the Veanors on Moshi, AudioLM and dMel.
Ready to test yourself? Open Teach and explain one chapter on the whiteboard, or Present it back. When you want to see the gap again, go back to the hero and try each design.