Speculative Decoding

Chapter 1: Draft, Then Verify

Chapter 0 established that a decode step has room to verify far more than one candidate token for essentially the same wall-clock cost. It did not explain how “verify” actually works without quietly changing what the model says. That's this chapter's entire job, and it's worth taking slowly, because the naive version of this idea is wrong in a way that's easy to miss.

The naive version, and why it's broken

Here's the tempting shortcut: run a small, cheap draft model to guess the next few tokens. Then run the big, accurate target model once, and for each guessed position, check “is this the token the target model would have picked as its single most likely next token (greedy)?” Accept the ones that match, reject the first one that doesn't, keep everything up to that point.

That scheme works fine if the target model always decodes greedily (always pick the single highest-probability token). But almost no production system decodes purely greedily — nearly every serving stack samples from the target model's actual probability distribution, with some temperature, to get varied, natural output. Under sampling, “the token the target model would have picked” isn't one fixed answer; it's a random draw from a distribution. Checking a draft token against greedy top-1 and calling that “verification” silently distorts the distribution the target model was supposed to be sampling from — it systematically over-favors whatever the draft model's own biases happen to be. The fix isn't a small patch to this scheme. It's a different algorithm entirely, borrowed from a much older idea in statistics: rejection sampling.

It's worth seeing exactly how badly the naive scheme fails, with real numbers, before building the fix — otherwise “silently distorts the distribution” stays an abstract worry instead of a concrete bug. Using this chapter's own worked example below, where the target's true distribution is p(A)=0.5, p(B)=0.3, p(C)=0.2, the naive scheme's “verification” step is: accept the draft's guess only if it equals A (the target's greedy top-1 token), and on any mismatch, fall back to emitting A directly rather than doing the correct residual redraw. Run that scheme and the output distribution collapses entirely onto one token:

Every single round outputs A, with total certainty — even though the target model's real distribution says B should genuinely appear 30% of the time and C 20% of the time. A verification rule built this way silently turns every sampling call into greedy decoding, permanently, no matter what temperature or top-p setting the request actually asked for. That's the bug this chapter exists to fix, and it's not a subtle statistical drift — it's a complete collapse of output diversity, hidden behind a scheme that looks reasonable on paper.

Rejection sampling, in one sentence before the proof

Rejection sampling answers a general question that has nothing to do with language models yet: given a cheap distribution q you can sample from easily, and an expensive distribution p you actually want samples from, how do you turn draws from q into exact draws from p, using p only to check (not generate) each draw? The answer: draw x from q, then accept it with probability min(1, p(x) / q(x)). If rejected, draw again from a specific residual distribution built from the leftover probability mass of p that q under-covered. Every accepted sample — whether it came straight from the first draw or from the residual redraw — turns out to be distributed exactly as p, not approximately. That's not a heuristic. It's a theorem, and the rest of this chapter proves it by hand on a toy example small enough to check every number yourself.

The three-token vocabulary, worked completely by hand

Strip the language model away entirely and work with a toy vocabulary of exactly three tokens: A, B, C. Suppose the expensive target model's true next-token distribution at this position is:

And the cheap draft model's distribution — deliberately different from p, since a real draft model never matches the target exactly — is:

Step 1 — the draft model draws a token from q. This costs almost nothing, by assumption (Chapter 3 will make this concrete). Say it draws token X.

Step 2 — the target model computes the acceptance probability for whichever token was drawn. The rule is accept(x) = min(1, p(x) / q(x)). Work out all three, one at a time:

Notice the pattern already: wherever the draft under-estimates the target (p > q, tokens A and C), the ratio exceeds 1 and gets clamped to a guaranteed accept. Wherever the draft over-estimates the target (p < q, token B), the ratio is honestly less than 1, and the draft's excess enthusiasm for B gets corrected by sometimes rejecting it.

Step 3 — build the residual distribution for when a draft token gets rejected. Only token B can ever be rejected here (A and C always accept). When B is rejected, resample from p'(x) ∝ max(0, p(x) − q(x)), the leftover probability mass p has that q didn't already cover:

Sum of the leftover mass: 0.1 + 0 + 0 = 0.1. Normalize by dividing each term by that sum:

The residual distribution is deterministic here — whenever B gets rejected, the algorithm resamples A with certainty. That's not a coincidence of these particular numbers; it's what happens whenever exactly one token is over-drafted and exactly one is under-drafted enough to soak up all the leftover mass. Real vocabularies with tens of thousands of tokens produce a smoother residual, but the mechanism is identical.

The proof: does the algorithm's actual output match p exactly?

This is the step that matters — not just running the algorithm, but checking, digit by digit, that what comes out the other end really is p, not just close to it. There are exactly two ways any given token can end up as the final output: the draft proposed it and it was accepted, or the draft proposed something else, that got rejected, and the residual redraw landed on it.

P(output = A):

Compare to the target: p(A) = 0.5. Exact match.

P(output = B):

Compare to the target: p(B) = 0.3. Exact match.

P(output = C):

Compare to the target: p(C) = 0.2. Exact match. And 0.5 + 0.3 + 0.2 = 1.0, so the three cases are exhaustive.

The overlap between p and q, and why it equals the overall acceptance rate

There's a clean geometric way to see why the accept rule min(1, p(x)/q(x)) has to take that exact shape, and it produces a number worth carrying into Chapter 2. For each token x, the quantity min(p(x), q(x)) is the probability mass p and q genuinely agree on — the “overlap” between the two distributions at that token. Summed across the whole vocabulary, that overlap is exactly the probability a single draft-and-verify round accepts the very first token it draws, averaged over everything the draft might propose:

Work it out on this chapter's own numbers:

90%. Check it the other way, directly from the per-token acceptance probabilities weighted by how often the draft actually proposes each token — the same computation, arrived at differently, as a consistency check:

Both routes land on exactly 0.9, as they must. This 90% overall acceptance probability is the very quantity Chapter 2 calls α and builds an entire speedup formula around — it's simply the total overlap area between the draft's distribution and the target's distribution, and it has an intuitive reading: the closer the cheap draft distribution sits to the expensive target distribution, the larger their overlap, and the more often a drafted guess survives verification untouched.

From one token to a whole drafted sequence

Everything above proved correctness for verifying a single drafted token against the target's distribution at one position. A real speculative decoding round drafts k tokens in a row, not one, and verifies all of them. The extension is by induction, not a new idea: apply the exact one-step proof at position 1 (conditioned on the confirmed prefix so far) to decide accept-or-reject-and-resample for the first drafted token. If it's accepted, move to position 2, now conditioning on that accepted token as part of the confirmed prefix, and apply the identical one-step proof again for the target's distribution at position 2 given that new prefix. Continue until a rejection happens (triggering the residual redraw at that position, which ends the round) or all k drafted tokens have been accepted (triggering the bonus token described at the top of this chapter). Because each individual step is an exact draw from the target's true conditional distribution at that position, the whole chain of accepted tokens is, by induction, an exact draw from the target's true joint distribution over the full sequence — not an approximation stacked k times, but an exact result stacked k times.

What actually got “verified” here

Notice what the target model's one forward pass had to compute to run this whole procedure: not a search, not a re-generation from scratch — just p(A), p(B), and p(C), the target's own probabilities at this position, evaluated once. That's the entire cost of verification: one forward pass of the target model, computing its normal output distribution, compared against whatever the draft model proposed. It's the exact same arithmetic the target model would have done anyway to produce one token on its own — the only difference is that same one pass is now being asked to evaluate the distribution at several candidate positions in the drafted sequence at once, which Chapter 0 already established is nearly free within the 156× idle-compute budget.

The one requirement this proof quietly assumes

The proof above needs exactly one thing to hold: the target model's true probabilities p(x) must be computable for every token the draft proposed, in the same forward pass. That's automatic — a forward pass produces a full probability distribution over the entire vocabulary as a side effect of computing logits, whether you use one token's worth of it or check several. Nothing about the proof needs the draft model to be good. A terrible draft model just means more rejections, which means fewer free tokens per round — a performance problem, worked out precisely in Chapter 2, never a correctness problem.

Scaling up from three tokens to a real vocabulary

Real vocabularies run 32,000 to over 150,000 tokens, not three — but nothing about the proof above used the number 3 in any essential way. The accept rule min(1,p(x)/q(x)) and the residual max(0,p(x)−q(x)), normalized, are defined identically at every one of 150,000 vocabulary entries; the three-token example was chosen purely so every number could be checked by hand, not because the mechanism needs a small vocabulary. At real scale, most tokens have both p(x) and q(x) very close to zero (the vast majority of the vocabulary is implausible at any given position), a handful of tokens carry almost all the probability mass in both distributions, and the residual distribution ends up smoothly spread instead of the clean deterministic single-token case this chapter's toy example produced — but every equation involved is the exact one derived above, run once per candidate token rather than reasoned about symbolically.

One more detail worth naming explicitly: temperature and top-p sampling, the settings that actually make a target model's output feel varied rather than deterministic, live entirely inside p(x) itself — they reshape the target's distribution before rejection sampling ever runs, not after. A request sampled at a high temperature simply has a flatter p(x) across more tokens, which Chapter 1's proof handles without any special case: whatever p(x) the target's configured sampling settings produce, verification draws exact samples from precisely that distribution, temperature and top-p included.

The limiting case: a perfect draft

It's worth checking the formula's behavior at the boundary, the same way Chapter 2 checks k=0. Suppose the draft distribution q happened to equal the target distribution p exactly, at every token — an unrealistic but instructive case. Then accept(x) = min(1, p(x)/q(x)) = min(1,1) = 1 for every single token: every drafted token is accepted with certainty, no rejections ever happen, and the residual distribution never gets used at all. The overlap-area formula from earlier confirms this the same way: ∑min(p,q) = ∑p(x) = 1, exactly 100% acceptance. This is the theoretical ceiling every real draft source in Chapter 3 is approaching, never reaching — a real draft model is always cheaper specifically because it's simpler than the target, and being simpler than the target means, by construction, it is not the target.

The opposite limiting case is worth naming too: if q and p shared no probability mass at all on any token (an extreme, unrealistic case where the draft never proposes anything the target considers plausible), the overlap sum ∑min(p,q) would be exactly 0, and every single drafted token would be rejected with certainty. The algorithm remains perfectly correct even here — the residual distribution simply does all the work, resampling directly from p every single time — but the round would produce nothing beyond its guaranteed single free token, exactly Chapter 2's break-even floor. Real draft sources sit somewhere between these two extremes, and where exactly is precisely what Chapter 2's α measures.

Why this proof, and not a weaker empirical argument

It would have been possible to write this chapter differently — run a draft-and-verify scheme on real text, measure that the output distribution looks statistically similar to the target's, and call that good enough. This lesson chose the harder route, a hand-checkable algebraic proof on a toy example, because “looks similar” is exactly the kind of claim that fails silently and expensively in production: a subtle bias that shows up as a slightly-too-repetitive assistant, or a slightly-too-cautious one, is very hard to detect from spot-checking outputs, and very easy to introduce with a plausible-looking but unproven verification rule (like the broken top-1 scheme this chapter opened with). An exact proof, checkable digit by digit on three tokens, is a stronger foundation for a technique meant to run under every request a production system serves, and it's the reason speculative decoding was adopted with real confidence rather than treated as a risky approximation.

This same standard — an exact, checkable proof rather than a plausible-sounding rule — is worth carrying forward as later chapters introduce techniques that deliberately relax it. Medusa's typical acceptance (Chapter 4) is presented as an explicit, opted-into tradeoff precisely because this chapter established what the strict baseline actually guarantees; without Chapter 1's proof as a reference point, there would be nothing concrete to measure that relaxation against.

Keep the three-token vocabulary from this chapter as a mental reference point going forward, too — whenever a later chapter introduces a new drafting mechanism, the question “what does q(x) look like here, and how much does it overlap with p(x)” is always the same question this chapter answered concretely for A, B, and C, just with q generated a different way each time.

Carry that forward into Chapter 2 as well: everything about acceptance rate, expected tokens per round, and net speedup traces back to this chapter's single overlap number, α = ∑min(p,q). The next chapter takes that as its starting input and asks what happens as it's dialed from 0 to 1.

One last framing check before that: the proof in this chapter never referenced tokens, transformers, or language at all beneath the surface — p and q were simply two probability distributions over a finite set of outcomes. That generality is precisely why the identical machinery reappears, unmodified, whenever any later chapter substitutes a different mechanism for generating q — the mathematics doesn't need to know or care where the candidate distribution came from, only that it's a genuine probability distribution the target can be checked against.

Chapter 2: The Acceptance-Rate Arithmetic

Chapter 1 proved speculative decoding never changes what comes out. It said nothing about how fast it comes out — and speed is the entire reason to use it. This chapter derives, from scratch, exactly how many tokens you should expect to harvest per verification round, and exactly where the whole scheme stops paying for itself.

Setting up the model: one acceptance probability, k drafted tokens

Simplify the bookkeeping with one clean assumption, the same one every speculative decoding paper starts from: each drafted token, independently, has some probability α (alpha) of being accepted by the target model's verification step. α isn't a fixed constant of nature — it depends on how well the draft model matches the target on this particular kind of text, which Chapter 3 explores in depth. For now, treat α as a given number between 0 and 1, and ask: if the draft proposes k tokens in a row, each independently accepted with probability α, how many tokens does one verification round produce, on average?

A verification round always produces at least 1 token, even in the worst case — if every single drafted token gets rejected, the residual-distribution redraw from Chapter 1 still yields exactly one valid token at the first rejected position, for free, as a side effect of the same forward pass. If all k drafted tokens get accepted, there's a bonus: the target model's own forward pass already computed the distribution for position k+1 too, so you get one additional token beyond the k drafted ones, sampled directly from the target. So the round produces somewhere between 1 and k+1 tokens, and the question is the expected value.

Deriving the expectation, term by term

The round survives past position i (keeps drafted token i) only if every one of the first i drafted tokens was accepted, which happens with probability αⁱ (independent acceptances multiply). The expected number of tokens produced is the sum, over every position the round could reach, of the probability of reaching it:

That's a finite geometric series, and its closed form is worth deriving in full rather than quoting, since the same trick reappears in Chapter 6. Multiply S by α and line the two sums up:

Subtract the second line from the first. Every middle term (α through α^k) appears in both sums and cancels, leaving only the first term of the first line and the last term of the second:

Sanity check at k = 0 (no drafting at all, plain autoregressive decode):

Exactly 1 token, matching plain decoding with no speedup — the formula degrades correctly to the baseline case.

The intuition behind the shrinking terms

Think of a round as a chain of coin flips, each one biased toward “keep going” with probability α. Reaching drafted position 2 requires two flips landing “keep going” in a row, which happens with probability α², strictly less likely than reaching position 1. Reaching position 3 needs three in a row, α³, less likely still. Each additional position the round could reach is strictly less probable than the one before it, which is exactly why the series α⁰ +  α¹ + α² +… shrinks term by term rather than staying flat — and why, for a good draft (α close to 1), the terms shrink slowly and a round reaches deep into its k drafted tokens on average, while for a poor draft (α close to 0), the terms collapse almost immediately and nearly every round produces just its guaranteed 1 token.

Worked example: α = 0.7, k = 4

Take a moderately good draft model that gets accepted 70% of the time (Chapter 3 will show what kind of draft achieves this), drafting 4 tokens ahead per round:

Nearly 2.77 tokens for one round of verification — not the naive “5 candidates × 70% each ≈ 3.5” a quick guess might suggest, because rejections are absorbing: the moment one drafted token is rejected, everything after it in that draft is thrown away, no matter how good the later guesses were.

From tokens/round to actual wall-clock speedup

Tokens per round alone isn't the speedup — the round itself doesn't cost zero time. Recall the fixed memory-bound decode step from Chapter 0 (7 ms). Drafting k tokens has its own cost, paid separately, before verification even starts: the draft model has to run k times, sequentially, since drafting is itself autoregressive. Call the draft model's per-token cost τ (tau), expressed as a fraction of the target's 7 ms step. A draft model roughly 10× smaller than the 7B target — 700 million parameters, a realistic size for a dedicated draft model — has:

One full round costs the drafting time (k × τ × 7ms) plus the one verification pass (≈7ms, still memory-bound, per Chapter 0):

Divide expected tokens per round by round time, and compare to the baseline of 1 token per 7 ms, and the 7ms cancels cleanly:

Plug in the worked numbers — α=0.7, k=4, τ=0.1:

Just under a 2× wall-clock speedup, entirely from a formula built out of one acceptance probability, one drafting cost ratio, and one draft depth — no benchmark required to predict it in advance, only to confirm it.

How sensitive is that number to the draft actually being good

2.7731 tokens/round and 1.98× speedup both leaned on one specific α=0.7. Hold k=4 and τ=0.1 fixed and recompute across a spread of acceptance rates, to see how much of the final speedup number is riding on that one assumption:

The relationship is convex, not linear — going from α=0.7 to α=0.9 (a 0.2 improvement) gains nearly as much speedup (0.95×) as going all the way from 0.3 to 0.7 (a 0.4 improvement, gaining 0.96×). That convexity is exactly why Chapters 4 and 5 exist: pushing α higher through a better drafting mechanism pays off disproportionately once α is already reasonably good, which is precisely the regime a well-tuned production system operates in.

Finding the best k for a fixed α, numerically

Chapter 2's speedup surface peaks at some particular k rather than climbing forever — work out where, for α=0.7 and τ=0.1, by simply evaluating the formula at several values of k:

The peak sits right around k=4 for this particular α and τ — k=2 hasn't yet captured enough of the geometric tail, and by k=8 the linear overhead has grown past what the shrinking marginal tokens are worth. This is not a universal “always use k=4” rule; it's a peak that shifts with α and τ, which is exactly what makes the interactive surface below worth exploring rather than memorizing one number.

Where it goes negative

The formula also predicts exactly when speculative decoding makes things worse. Take a poorly-matched draft — α=0.15, the kind of low acceptance rate a generic small model gets on highly unpredictable, open-ended creative text — drafted aggressively at k=8 with the same τ=0.1 draft-model cost:

A speedup of 0.65× means the “speculative” server is actually 35% slower than plain decoding. The draft model kept guessing wrong, so almost every round produced only its guaranteed 1 free token, but the server still paid the full 8 × 0.7 ms = 5.6 ms drafting cost on every single round for that privilege. This is not a hypothetical edge case — it's the exact failure mode that makes “per-request adaptivity” (Chapter 7) a real engineering requirement, not a nice-to-have: a server that speculates blindly on every request, regardless of how well the draft actually matches that request's content, will genuinely slow some fraction of its traffic down.

The shape worth memorizing

Slide k up while holding α fixed below about 0.85 in the widget above, and the speedup curve rises, peaks, then falls back down — it does not keep climbing forever, even for a very good draft. That peak-then-fall shape is the geometric-series ceiling colliding with linear overhead growth: E[tokens] saturates toward 1/(1−α) as k grows large (the tail terms αⁱ shrink to nothing), while the overhead factor (1+kτ) keeps growing without bound. Past some k*, every extra drafted token adds almost nothing to the numerator while still adding a fixed cost to the denominator. Finding that peak, for a given α and τ, is exactly what real serving systems tune k against — not by guessing, but by this same derivative.

An average hides real burstiness — what streaming output actually feels like

2.7731 is an expectation, not a guarantee for any single round. Individual rounds vary a lot: some rounds reject on the very first drafted token (producing just 1 token, the guaranteed floor), others accept every single one of the k drafted tokens plus the bonus (producing k+1 tokens at once). At α=0.7, k=4, work out the probability of hitting that maximum:

This is the concrete, user-visible consequence of speculating: instead of a smooth, metronomic one-token-every- 7ms trickle, a speculative stream arrives in irregular bursts — sometimes one token, sometimes five, timed unevenly. Client-side streaming UIs built assuming a steady per-token cadence (for smooth typewriter-style rendering, for instance) sometimes need to buffer and re-time output slightly to avoid the burstiness reading as janky, even though the total time to complete the response is genuinely shorter. The expectation this chapter derives predicts the average rate correctly; it does not, by itself, predict the rhythm.

What happens if k is tuned against the wrong α

Every k chosen so far assumed α was known accurately. In practice, α is measured (Chapter 3 shows how) and can drift as traffic mix shifts. Check how costly a misestimate actually is: suppose k=6 was chosen assuming α=0.7 (near that k's near-optimal point from the earlier table), but real traffic actually runs at α=0.5:

Compare to what k=2 (this lower α's own better-matched depth) would have achieved instead: E[tokens]=1+0.5+0.25=1.75, speedup=1.75÷1.2=1.458×. The mismatched k=6 choice still produces a real, positive speedup (1.24×) — overshooting k costs some of the available gain, but the formula degrades gracefully rather than catastrophically, as long as α itself stays well above the break-even floor from the previous section. The genuinely costly mistake is a severe overestimate of α combined with an aggressive k, which is exactly the negative-speedup scenario worked out above.

Why this chapter's formula is the load-bearing result of the whole lesson

Every remaining chapter in this lesson — four different ways to choose a draft source, two different ways to fold drafting into the target's own forward pass, and one chapter on sharing the idle budget with real traffic — plugs into exactly this chapter's speedup formula without changing its shape. What changes, technique to technique, is only the two inputs: which α a given mechanism achieves, and which τ it costs to achieve it. Keeping that structure in mind turns the rest of this lesson from six separate topics into one repeated exercise: given a new way to produce candidates, what α does it get, what τ does it cost, and where does speedup(α,k,τ) land as a result.

The formula's two failure directions, side by side

Both directions are correctable once measured, which is exactly why Chapter 3 treats measuring real α as a prerequisite step rather than an afterthought — this formula only protects a team that actually feeds it real numbers.

One further practical note: because both failure directions degrade gracefully rather than catastrophically (short of the specific low-α-and-large-k combination worked out earlier), a reasonable operational default is to start k conservative — on the low side of whatever this chapter's tables suggest for a measured α — and raise it only once real production measurements confirm the assumed α holds on live traffic, rather than launching at an aggressive k tuned against an optimistic, unmeasured estimate.

Ramp k the same way any other performance-sensitive rollout gets ramped: on a small traffic percentage first, with the acceptance-rate telemetry from the measurement methodology in the next chapter watched closely, before widening to full traffic. Because Chapter 1's proof guarantees output correctness regardless of k, this ramp is purely a performance rollout, with none of the quality-regression risk a change to model weights or sampling parameters would carry.

That's the note to carry into Chapter 3: this chapter proved how much speedup a given α, k, and τ combination buys; the next four chapters are entirely about the practical question of where a good α actually comes from, and what it costs to get there.

Nothing in the derivation above assumed greedy decoding, a particular temperature, or a particular vocabulary size — it holds for any α between 0 and 1, computed however Chapter 1's overlap formula produces it, which is exactly what makes it reusable, unchanged, across every remaining chapter of this lesson. It is, in the fullest sense, the load-bearing formula of the entire lesson.

Keep the geometric-series shape in mind above all: it is the single mathematical fact every one of this lesson's remaining chapters keeps rediscovering in a new guise, from tree-shaped candidate coverage to window-convergence probabilities.

Chapter 3: Choosing a Draft

Chapter 2's formula has three knobs: α, k, and τ. k is a serving decision (Chapters 2 and 7 cover tuning it). α and τ are both consequences of one choice this chapter is entirely about: where do the drafted candidates actually come from? There isn't one right answer — the best draft source depends heavily on what kind of text is being generated, and this chapter walks through the three real options production systems use, with the exact arithmetic from Chapter 2 applied to each.

Judge each option against three separate axes, not just the α and τ that feed directly into Chapter 2's formula: how general-purpose it is (does it need to know anything about this specific workload to work at all), how much engineering overhead it adds (a second model to train, host, version, and keep in sync with the target, versus nothing at all), and how it fails (gracefully near break-even, or with a real, costly slowdown). The four options below trade off differently along all three.

Option 1: a small, separate neural draft model

The most direct approach, and the one Chapter 2's worked example already used: train or reuse a small transformer — the same architecture family as the target, just far fewer parameters — and run it autoregressively to propose k candidate tokens. It's general-purpose: it works reasonably on any kind of text, because it's a real language model, not a lookup trick. Its downside is the cost: it's still a full forward pass per drafted token, sequential, and its acceptance rate against the target depends entirely on how well its own training matched the target's behavior on this particular kind of text. Recall the 700M-parameter example (τ=0.1) from Chapter 2, moderate acceptance α=0.6 on typical open-ended chat traffic:

This option's real cost isn't only the τ=0.1 in the formula — it's what τ=0.1 doesn't capture. The 700M draft model needs its own 1.4 GB of GPU memory, resident alongside the target's 14 GB, every hour the server runs, whether or not any given request actually benefits from it. It needs its own training or distillation pipeline, and it needs to be periodically refreshed whenever the target model itself is updated — a draft trained against an old target checkpoint quietly drifts toward a lower real-world α than whatever number was measured at launch, degrading speedup over time with no obvious symptom besides a slowly rising average latency. None of that shows up in the formula; all of it shows up on an on-call rotation.

Option 2: n-gram / prompt lookup drafting — free drafting

For an entire category of real workloads — code editing, document summarization, retrieval-augmented answers that quote source text — the model's output overlaps heavily, word for word, with text that's already sitting in the prompt. In those cases, there's no need for a neural draft model at all. Prompt lookup decoding (also called n-gram drafting) works by literal string matching: take the last few tokens the model has generated, search the prompt (and the tokens generated so far) for the most recent place that exact sequence appeared before, and propose whatever token followed it there as the draft.

This costs a hash-table lookup — call it τ ≈ 0.002, essentially a rounding error next to the 7 ms target step, because it never touches the GPU's arithmetic units or reads any model weights at all. On a code-editing task, where the model is frequently reproducing chunks of the original file with small changes, acceptance rates run high — take α=0.82 as representative:

Nearly 3.5×, and it cost nothing to draft — no extra model to host, no extra GPU memory for a second set of weights, no extra forward passes eating into the compute budget. Summarization, where the model regularly copies phrases verbatim from the source document, runs similarly well at a slightly lower α=0.75:

The catch is exactly what you'd predict: prompt lookup only helps when the output genuinely overlaps the prompt. On open-ended creative writing, where almost nothing the model says was already sitting in the input, α collapses — this is precisely the α=0.15 scenario from Chapter 2's negative-speedup example, except with prompt lookup's near-zero τ the speedup there works out closer to break-even rather than clearly negative, because there's almost no drafting cost to lose money on in the first place. Free drafting can't make a bad match actively harmful the way an expensive neural draft can — but it also can't manufacture acceptance that isn't there.

Option 3: retrieval drafts

A middle ground: maintain a cache of previously generated completions (or a retrieval index over similar past requests), and for a new request, retrieve the most similar past continuation as the draft source, instead of either running a neural model or restricting the search to the current prompt. This suits workloads with a lot of repeated structure across different requests — RAG-style systems answering variations of similar questions, or customer-support bots handling recurring issue types — where prompt lookup would find nothing (the answer wasn't in this particular prompt) but a retrieval index over past answers would. It costs a small similarity lookup plus light reranking, τ≈0.007, and typically lands at a moderate acceptance rate, α=0.5 being representative for domain-repetitive RAG traffic:

Option 4: self-speculative drafting — no second model at all

A fourth option sidesteps Option 1's operational overhead entirely: instead of a separate model, draft using an early-exit partial forward pass of the target model itself — run only the first several transformer layers, skip the rest, and read a rough next-token guess off an early exit point. This is called self-speculative decoding. There's no second set of weights to load, host, version, or retrain: the draft literally reuses a prefix of the exact same weight tensors the target model already has resident in memory.

Work out the numbers for a 32-layer version of this lesson's 7B model, early-exiting after layer 8:

That τ is more than double Option 1's dedicated 700M model (0.10), because an early-exit draft, even stopping at 1/4 of the network's depth, still has to read a meaningfully large slice of a 7B-parameter model's weights — a purpose-built small model can be far smaller than any fixed fraction of the target. Acceptance is moderate: early layers of a transformer haven't yet built up the target's full contextual reasoning, so take α=0.55 as representative, a bit below Option 1's tuned 0.60:

Barely above break-even — the worst raw speedup of the four options here. What self-speculative drafting buys instead is architectural simplicity: zero extra GPU memory footprint, zero separate training pipeline, zero drift risk between draft and target, because there is only ever one model. For teams weighing a modest, guaranteed-safe speedup against the ongoing cost of operating a second model, that tradeoff is sometimes worth making even at a smaller number — the same honest tradeoff Chapter 7 revisits when it discusses adaptivity as an operational, not just mathematical, decision.

Matching draft source to workload

Hybrid drafting: falling back instead of choosing once

A production router doesn't have to commit to exactly one draft source for an entire request. A common, higher-performing pattern chains sources with a cheap check first: attempt prompt lookup (Option 2) at essentially zero cost; if the lookup finds no matching n-gram in the prompt for the current context (a genuine possibility even on code-editing traffic, whenever the model is about to write something novel rather than reproduce existing text), fall back to the small dedicated draft model (Option 1) for that round only, rather than drafting nothing at all. The combined τ for a round where the lookup succeeds stays at Option 2's near-zero cost; a round where it falls back pays the higher τ=0.1 cost, but only on the rounds that actually need it. On a workload that's 70% coverable by prompt lookup and 30% not, the effective τ averages to roughly 0.7×0.002 + 0.3×0.10 ≈ 0.031 — noticeably cheaper than always paying the dedicated model's full τ=0.10, while still getting a real draft on the rounds where lookup alone would have found nothing.

Measuring α on real traffic, not assuming it

Every α in this chapter's table is illustrative — a representative number for its category, not a figure to trust blindly for any specific deployment. Measuring the real number is straightforward and worth doing before shipping any draft source: log, for a sample of production requests, how many of each round's k drafted tokens the verification step actually accepted, and average the fraction accepted across a large enough sample (thousands of rounds, at minimum, to average out the burstiness described in Chapter 2). Because Chapter 1's proof guarantees the accepted output is always exact regardless of α, this measurement can be run directly in production, on real traffic, with zero risk to output quality — a team can turn speculative decoding on, measure the real α it achieves on its actual workload mix, and only then decide whether the resulting speedup (via Chapter 2's formula) justifies whatever engineering cost that particular draft source adds.

Draft sources compound across a multi-turn conversation

Prompt lookup and retrieval drafting both get a further boost in a multi-turn chat setting that a single-request benchmark misses. As a conversation grows — the user's earlier questions, the model's own earlier answers, any documents pasted in along the way — the pool of text prompt lookup can match against grows with it, tending to raise α for later turns relative to the first. A user who pastes a code file in turn 1 and asks for edits across turns 2 through 5 sees prompt lookup's match rate climb turn over turn, since each new turn adds more of the model's own prior, already-accepted output back into the searchable context. This is a genuine, compounding advantage of the free drafting sources over a fixed-size dedicated model, whose α doesn't have an equivalent mechanism to improve as a single conversation lengthens — it stays tied to whatever the model was trained on, turn to turn.

A short engineering checklist before shipping any draft source

Collecting the operational threads from all four options into one pre-launch checklist, since the arithmetic alone doesn't surface every one of these on its own:

None of these questions has a universally correct answer — they're the honest operational surface behind every α/τ pair in this chapter's comparison table, and skipping them is exactly how a technique that looked great in an offline benchmark ends up disappointing, or actively harmful, once it's actually carrying production traffic.

None of these four options change the correctness guarantee

Worth restating plainly before moving to the mechanisms that fold drafting directly into the target's own forward pass: every one of this chapter's four draft sources — a dedicated small model, prompt lookup, a retrieval index, or an early-exit slice of the target itself — feeds into the exact same verification step Chapter 1 proved exact. Whatever distribution a given draft source implies over the vocabulary at a given position is simply q(x) in that proof; the mechanism generating q never enters the correctness argument at all, only the performance one. Choosing between these four options is entirely a Chapter 2 question — which α and τ a source achieves — never a question of whether the resulting output is trustworthy.

Recap: the four options, one sentence each

These four are also not a ceiling — the next three chapters introduce three more mechanisms (Chapters 4, 5, and 6) that each answer this chapter's same question (where do candidates come from, and at what α and τ) with a structurally different idea: folding drafting directly into the target's own forward pass rather than running it as a separate step beforehand. All four of this chapter's options and all three of the next three chapters' mechanisms feed into the identical Chapter 2 formula — the menu is large specifically because no single mechanism dominates every workload and every operational constraint at once.

A useful way to hold all seven mechanisms in mind at once: this chapter's four are distinguished mainly by where the drafting computation physically happens (a second model's forward pass, a hash lookup, a similarity index, or a truncated slice of the target itself), while the next three chapters' mechanisms are distinguished mainly by how tightly the drafting step is fused into the target's own single pass. Neither axis is inherently better; they're answers to different constraints, and a mature serving stack often ends up running more than one simultaneously across different request classes, exactly as the hybrid-drafting pattern above illustrated.

With that framing in place, the next chapter starts on the fused side of the split — Medusa, the simplest of the three fused mechanisms, adds nothing but a handful of extra output heads directly onto the target model's own existing forward pass.

One closing thought before that transition: every τ and α number in this chapter's tables was measured on this lesson's own 7B/A100 baseline. Rerunning any of these four options on a different model size or GPU changes the absolute numbers — a bigger target model shifts the dedicated draft model's relative size (and thus its τ), and a faster GPU shifts the absolute step time both the draft and target run against — but it never changes which structural category a given draft source falls into, or the qualitative tradeoffs this chapter derived between them.

Carry the checklist and the hybrid-fallback pattern forward as reusable design tools, independent of which specific option (or combination) a given deployment ultimately settles on — they apply just as directly to the three fused mechanisms the next three chapters introduce.

All four options measured in this chapter remain fully available even after that shift, too, as the hybrid and fallback patterns above already showed — the fused mechanisms extend the menu, they don't replace it, and a production router is free to pick from all seven mechanisms across this lesson at once, request by request, based on measured α per workload class, exactly as this chapter's own hybrid-drafting example did above.

That's the natural handoff into Chapter 4: everything from here forward stops asking “where does the draft come from” as a separate question, and starts asking how much of the drafting work the target model's own single forward pass can simply absorb for free.

Medusa answers that question first, and most simply: a handful of extra linear heads, riding on the same hidden state the target model was computing anyway.

That simplicity is deliberate, and worth appreciating on its own terms, as a genuinely useful, easy-to-add technique in its own right, before the next two chapters build further on top of it.

Chapter 4: Medusa: Heads & Trees

Every draft source in Chapter 3 shares one structural cost: drafting is sequential. Whether it's a small neural model or a hash lookup, producing the second drafted token requires knowing the first one, the third requires the second, and so on — k separate steps, paid one after another, before verification even starts. Medusa asks a different question: what if the target model itself could propose several future tokens in the same single forward pass that already produces the current one, with no separate sequential drafting step at all?

Extra heads, not extra models

A standard transformer's final layer produces one hidden-state vector per position, which a single output head (the “LM head”) turns into a probability distribution over the vocabulary for the next token. Medusa adds K extra output heads, attached to that exact same final hidden state, each trained to predict further ahead — head 1 predicts the token at position t+1 as usual, head 2 predicts the token at t+2 directly from the same hidden state (not from head 1's guess), head 3 predicts t+3, and so on. All K+1 heads run in parallel, off the same single forward pass, at essentially zero extra latency — the target model was already computing that hidden state anyway; a few extra small linear layers reading from it add negligible compute next to the 14 GB weight read that already dominates the step.

This is the structural advantage over Chapter 3's approach: no separate small model to load, no separate sequential forward passes, no drafting τ at all in the Chapter 2 sense — drafting is now free, folded entirely into the one pass the target model was going to run regardless. There's also an engineering advantage that matters just as much as the pure speed number: because the extra heads are trained alongside (or fine-tuned onto) the target model's own weights, there's no separate model lifecycle to manage — no second checkpoint to version, no separate GPU memory allocation the way Option 1 of Chapter 3 needed, and no drift risk between a draft model and a target model that have diverged after a retraining round. The heads are part of the target model, full stop.

A tiny worked example of the attention mask

Before scaling up to a full tree, see the masking idea on the smallest possible case: K=2 heads, s=2 candidates per head. The tree has a root (the last confirmed token), 2 children from head 1 (call them b₁ and b₂), and each of those has 2 children from head 2, for 4 grandchildren (b₁c₁, b₁c₂, b₂c₁, b₂c₂). That's 2 + 4 = 6 candidate positions verified in the one pass, alongside the root.

The attention mask has to keep each grandchild seeing only its own path back to the root, and nothing about the sibling branch it has nothing to do with: b₁c₁ attends to itself, b₁, and the root — never to b₂ or anything under it, since b₁c₁ represents a hypothetical continuation where the second token was b₁, and mixing in information from the b₂ branch would corrupt that hypothetical. This is exactly why a normal causal attention mask (everyone sees everyone before them, in one line) doesn't work for a tree — the tree has multiple branches occupying parallel token positions in the same pass, and only a custom, tree-shaped mask keeps each branch's hypothetical world consistent.

From heads to a tree of candidates

Each head doesn't just propose its single most likely token — it proposes its top-s most likely candidates. With K heads and s candidates per head, the full set of possible continuations forms a tree: the root is the current confirmed token, head 1 branches into s children, and each of those branches into s children from head 2, and so on. A path from root to a leaf at depth K is one full candidate continuation.

Verifying every path in this tree in a single forward pass uses tree attention: every node in the tree attends only to its own ancestors on the path back to the root, not to sibling branches it has nothing to do with, via a specially constructed attention mask. That lets the target model verify every candidate path in the tree simultaneously, in one pass, with the same single memory-bound weight read Chapter 0 already paid for — the tree just adds more token positions to that one pass, which Chapter 0 established is nearly free as long as the total stays under the 156× idle-compute budget.

The tree candidate math: why you can't just use the full tree

Count the nodes in a dense tree — every head keeping all s candidates from every parent, with no pruning. With K heads and s candidates each, the number of nodes added at depth i is sⁱ, so the total additional nodes across all K levels is:

Work it out for K=4 heads, s=4 candidates per head — a reasonable-sounding starting point:

Compare that to Chapter 0's idle-compute budget of 156 token-equivalents. 340 is more than double the budget — a dense 4-head, top-4 tree would push the verification pass past the compute-bound crossover, turning what was supposed to be a free lunch into a genuinely slower step. This is exactly why real Medusa deployments never use the dense tree.

The dense-tree node count grows explosively with both K and s — it's worth seeing exactly how fast, since that growth rate is the whole reason pruning isn't optional:

The jump from K=4,s=3 (120 nodes, still inside budget) to K=4,s=4 (340 nodes, well outside it) from a single extra candidate per head shows how steep this curve is — the dense tree's node count is exponential in K, so pruning stops being a nice-to-have and becomes structurally necessary the moment a team wants more than a handful of heads or candidates.

The fix: a calibrated sparse tree

Instead of keeping every branch, Medusa prunes the tree offline, using a calibration pass over representative text to find which specific paths are actually likely to be accepted, and keeps only those — a small, fixed, sparse subset of the dense tree's nodes, reused for every request. Published Medusa configurations converge on roughly 64 total tree nodes for a well-tuned setup, comfortably under the 156× budget:

That pruned 64-node tree still covers the overwhelming majority of the dense tree's useful acceptance probability — because most of the 340 dense nodes correspond to token sequences with near-zero probability under the target model anyway, and pruning specifically targets keeping the high-probability paths while discarding those.

The intuition for why this works is the same probability-mass falloff that makes top-s sampling itself a reasonable approximation of a full distribution. Within any one head's s candidates, the probabilities are sorted — the top candidate typically carries much more probability mass than the second, which carries more than the third, and so on. A representative illustrative spread for the top-4 candidates at one head might run 0.55, 0.22, 0.12, 0.06 — the bottom two candidates together hold under a fifth of the mass the top one holds alone. Multiply that falloff across four levels of a tree, and the vast majority of the dense tree's 340 nodes sit at the intersection of several already-unlikely branches, contributing a vanishingly small share of the tree's total covered probability. A calibration pass that keeps only the highest-mass root-to-node paths, rather than every combinatorial branch, discards nodes that were barely contributing anything in the first place.

How big are the heads themselves, in GPU memory

It's worth grounding “a few extra small linear layers” with a real number, the same way earlier chapters grounded every other claim. A naive head design would project the shared hidden state all the way to vocabulary size directly — at d=4,096 and a 128,000-token vocabulary, that single projection alone would cost 4,096 × 128,000 ≈ 524 million parameters per head, which would make four extra heads nearly as large as a dedicated small draft model. Real Medusa heads avoid this: each head is a single small residual feed-forward layer sized roughly d×d, and reuses the target model's own frozen unembedding matrix (the same weights the target's normal output head already has) to turn that transformed hidden state into vocabulary logits, rather than training a second vocabulary-sized projection from scratch:

134 MB against the target's 14 GB is under 1% additional GPU memory — genuinely negligible, and smaller than every other drafting mechanism in this lesson, including EAGLE's single full transformer layer. That's the direct benefit of tying into the target's existing unembedding rather than training a new one: Medusa's heads are cheap specifically because they reuse work the target model was already doing.

Why a tree can beat a linear chain at the same depth

Chapter 2's E[tokens] formula assumed one linear chain of guesses, each with a single fixed acceptance probability α. A tree changes the game at each level: instead of one guess, the verifier has s chances to match at each level, and the round survives to the next level if any of those s candidates is accepted, not just one specific guess. Illustrate on the small K=2, s=2 tree from earlier: suppose any one of head 1's 2 candidates has an individual match probability of 0.4, so the probability at least one of the two matches is higher than 0.4 alone — roughly 1−(1−0.4)² = 1−0.36 = 0.64 assuming rough independence between the two candidates. Continuing to head 2 conditional on head 1 surviving, say that level's best-of-2 survival probability is 0.5:

Compare that to a single linear chain of depth 2 with the same 0.4 single-candidate acceptance rate: E[tokens] = 1 + 0.4 + 0.16 = 1.56. The tree's “best of s tries per level” structure recovers meaningfully more expected tokens from the same two levels of depth, at the cost of s× more nodes verified per level — precisely the tradeoff Chapter 4's node-count table above is pricing. This is an illustrative approximation, not an exact formula (real tree acceptance correlates across sibling candidates in ways this simplified independence assumption glosses over), but it captures the right intuition: a tree buys higher effective acceptance per level by trading width (more nodes, more idle-compute budget spent) for depth-per-node-spent, which is exactly why Medusa's calibrated 64-node tree can outperform a much narrower linear draft chain of similar total budget.

Verification: typical acceptance, not strict rejection sampling

One honest caveat: Medusa's original formulation uses a slightly looser acceptance rule than Chapter 1's strict rejection-sampling proof, called typical acceptance — it accepts a drafted token whenever the target model finds it “plausible enough” (above some probability threshold), rather than the exact min(1, p/q) criterion. This trades away the exact-distribution guarantee of Chapter 1 for a somewhat higher acceptance rate in practice. Medusa also supports a strict mode using Chapter 1's exact rejection sampling, at a modest cost to acceptance rate — the choice between speed and exactness is explicit, not hidden.

What it actually takes to add Medusa heads to an existing model

Because the heads are small and separate from the target's own backbone, adding them doesn't require retraining the 7B target at all — the standard recipe freezes every one of the target's original 7 billion parameters and trains only the new heads (that 67.2 million-parameter addition from above) against a corpus of the target's own generations, teaching each head to predict further ahead from the shared hidden state. This is a comparatively cheap fine-tuning run — far cheaper than training or distilling an entire separate draft model from scratch, since the vast majority of the representational work (everything up through the shared hidden state) was already learned by the target and never needs to change. This is part of why Medusa has seen wide adoption as a first speculative-decoding technique for teams operating an existing target model: the marginal cost of trying it is low, and nothing about the target model's own behavior on non-drafted, ordinary requests changes at all, since its own original output path through the normal LM head is completely untouched.

Concretely, typical acceptance accepts a candidate token x whenever its target-model probability p(x) clears a threshold set relative to the distribution's own entropy — roughly, “is this token at least plausible, given how spread out or peaked the target's distribution is here,” rather than Chapter 1's precise min(1,p(x)/q(x)) ratio against a specific draft distribution q. On a peaked distribution (the target is very confident about one token), the threshold is effectively strict; on a flat distribution (the target is genuinely unsure among several plausible tokens), the threshold relaxes, accepting more candidates as reasonable. This tends to raise the practical acceptance rate above what strict rejection sampling would allow, at the cost of the output distribution no longer being an exact match to the target — a small, usually acceptable, quality-for-speed trade that a strict-mode toggle lets a team opt out of entirely when Chapter 1's exact guarantee genuinely matters for a given deployment.

Medusa's place in the six-technique lineup

Medusa is this lesson's first example of a mechanism that folds drafting entirely into the target's own forward pass, rather than running a separate sequential process beforehand — a structural category Chapter 6's lookahead decoding also belongs to, and one Chapter 7 singles out as degrading more gracefully under high batch occupancy than a separate-model draft does, precisely because there's no standalone drafting pass competing for GPU time before verification even starts. Chapter 5 takes this same-pass philosophy further, adding genuine autoregression back into the drafting step itself, at the cost of needing somewhat more extra parameters than Medusa's minimal heads — the next chapter picks up exactly where this one's tree-node budget left off.

A note on why calibration happens offline, not per-request

It's worth being precise about when Medusa's tree-pruning calibration actually runs, since it's easy to conflate with the per-request adaptivity Chapter 7 builds. Calibration is a one-time (or periodically refreshed) offline pass over a representative sample of the target model's own traffic, producing one fixed tree shape that every request then reuses at inference time — it is not re-run per request, and it does not react to the current batch occupancy the way Chapter 7's throttling does. The two mechanisms operate on different timescales and answer different questions: calibration decides which paths are worth verifying at all, once, in advance; occupancy-aware throttling decides whether to verify any of them right now, continuously, in production. A server needs both, but they are not the same lever.

Recalibrating the tree periodically is still worthwhile as a target model or its typical traffic mix drifts over time — a tree calibrated against last quarter's traffic distribution may no longer reflect this quarter's most common continuation patterns as well, quietly leaving some achievable α on the table without any error or alert to flag it. Treating calibration as a scheduled maintenance task, not a one-time setup step, keeps this gap from opening silently.

Chapter 5 picks up exactly this thread — EAGLE keeps Medusa's same-pass, no-separate-model philosophy, but replaces several independent heads guessing from one shared hidden state with a single, genuinely autoregressive draft over that hidden state's own evolving features.

Hold onto this chapter's node-budget arithmetic specifically — the 156-node line, the dense-vs-pruned comparison, and the probability-mass falloff argument all reappear, essentially unchanged, when EAGLE-2's dynamically grown tree gets checked against the same constraint in the next chapter.

And hold onto the GPU-memory grounding too — 134 MB for four Medusa heads, against 14 GB for the target itself — as the reference point the next chapter's larger, single-transformer-layer EAGLE draft gets compared against directly, and as evidence that this lesson's per-technique parameter counts are always grounded in a real architectural calculation, never asserted as a bare figure.

Chapter 5 keeps every idea introduced here — same-pass drafting, tree verification, the 156-node budget — and changes exactly one thing: what the extra heads are allowed to see.

That one change turns out to be worth roughly another 60% of speedup on its own, which is a striking return for a single architectural substitution, and a strong argument for reading Chapters 4 and 5 as a pair rather than in isolation — Medusa's structure, once understood, is most of what's needed to follow why EAGLE's one change matters as much as it does.

Chapter 6 closes out the drafting-mechanism half of this lesson with something structurally different again: a technique that needs no draft, no extra heads, and no q distribution to construct at all.

It's the most conceptually distinct chapter in this lesson, and the best evidence that “spend the idle compute” is a broader idea than any single one of its six concrete implementations.

Read it slowly — the Jacobi-iteration reframing is a genuinely different way of thinking about decoding itself, not just another lever on the same machinery the earlier chapters built.

Chapter 5: EAGLE: Drafting on Features

Medusa's extra heads all predict directly from one shared hidden state, with no information about what the other heads guessed. EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) makes one structural change to fix this: instead of several independent heads guessing from a fixed hidden state, EAGLE runs a small, genuinely autoregressive draft process — but autoregressive over hidden-state features, not over sampled discrete tokens. That single change turns out to matter a lot.

Why drafting on tokens throws information away

When a small neural draft model (Chapter 3) or a Medusa head (Chapter 4) samples a discrete token as its guess, it collapses an entire probability distribution — possibly dozens of plausible next tokens, each with real probability mass — down to one integer. Every subsequent drafting step then has to reconstruct context from that single collapsed token, with no memory of how confident or uncertain the model actually was, or what the runner-up candidates looked like. That's a genuine loss of information at every single step of the draft chain.

The target model's internal hidden state, at the layer just before the final output head, hasn't thrown that information away yet — it's a dense vector that implicitly encodes the full distribution, not just the sampled outcome. EAGLE's draft model consumes that richer feature vector directly, concatenated with the embedding of whatever token actually got sampled, and predicts the next feature vector — then reads a token off it using the target model's own frozen output head, reused, not retrained. Drafting one step further ahead means feeding that predicted feature back in, autoregressively, exactly the way a normal decoder would over tokens, just one representational level earlier.

What the hidden state actually holds, in numbers

It helps to put a real size on “richer than a single token id.” A discrete token id, once sampled, is a single integer — effectively log₂(vocab size) bits of information, around 17 bits for a 100,000-token vocabulary. This lesson's 7B model has a hidden dimension of roughly 4,096 — a representative size for a model this large — so its pre-output hidden state is a vector of 4,096 floating-point numbers, each carrying real information about how the target model weighed every plausible continuation, not just which one it eventually sampled. That's several orders of magnitude more raw information per position than the single sampled token id a Chapter 3 draft model has to work from, which is the concrete substance behind “a richer signal” above.

That size also grounds where EAGLE's own extra-parameter count comes from. A single transformer layer's parameter count is dominated by its attention projections and feed-forward block, and scales roughly as 12 × d² for hidden dimension d (four attention projection matrices sized d×d, plus a feed-forward block sized roughly 8d², standard for the typical 4× FFN expansion ratio used across the transformer family):

That's already close to the ~0.24 billion figure EAGLE's published draft models report — the small gap accounts for the extra projection that fuses the incoming feature vector with the sampled token's embedding before that one layer runs. One layer, reusing the target's existing output head, is genuinely all EAGLE's draft needs — not because the designers were being stingy, but because a single layer operating on an already-rich 4,096-dimensional feature is doing categorically less work than an entire multi-layer 700M model has to do reconstructing similar context from a single collapsed integer.

Why this raises α, and why it also lowers τ

Two separate advantages, both mattering to Chapter 2's speedup formula. First, α: because the draft model is working with the richer feature representation instead of a collapsed token, it captures more of the actual uncertainty and structure the target model itself would use, producing guesses that agree with the target more often — published EAGLE results report acceptance rates noticeably above equivalent token-level small-model drafts on the same target and hardware. Take α=0.80 as representative, against Chapter 3's token-level draft at α=0.65 (a well-tuned same-family distilled draft, better than the generic 0.60 used earlier):

Second, τ: EAGLE's draft model reuses the target's own frozen output head instead of training a whole separate vocabulary projection, and needs only one small transformer layer of its own on top — published EAGLE draft models run around 0.24 billion extra parameters against a 7B target, smaller than the 700M dedicated draft model used earlier:

Putting both advantages together

Both effects point the same direction — higher α raises the numerator, lower τ shrinks the denominator — and neither one alone tells the whole story. A draft technique that only improved α but kept the same heavy per-step cost, or one that only got cheaper without getting more accurate, would each capture roughly half of this gain on their own.

Why Chapter 1's exact guarantee survives this entire chapter unchanged

It's worth being explicit about what EAGLE changes and what it leaves completely alone. Everything in this chapter — the feature-level autoregression, the frozen shared output head, the higher α — concerns only how candidates get proposed. Verification is still exactly Chapter 1's rejection sampling (or, if a deployment opts in, Medusa-style typical acceptance from Chapter 4): the target model still computes its own true p(x) at each candidate position, and still accepts or rejects against that real distribution with the same min(1,p(x)/q(x)) rule, where q(x) here is simply whatever distribution EAGLE's feature-level draft implies over the vocabulary once its predicted feature is read out through the target's frozen head. EAGLE never touches the correctness side of this lesson's argument — it only ever tries to make q closer to p, which is exactly the lever Chapter 1's overlap-area derivation (∑min(p,q)) identified as the source of α in the first place.

Training the draft, briefly

Like Medusa's heads, EAGLE's single extra layer is trained with the 7B target's own backbone completely frozen — only the small feature-predicting layer learns, against a training objective of predicting the target's own next hidden-state feature (or, in EAGLE-3's variant, the next token directly) given the previous feature and sampled token embedding. Training data is generated by simply running the frozen target model and recording its own hidden states and outputs — no external labels needed, since the target model is its own teacher here. This keeps the cost of adding EAGLE to an existing deployment comparable to Medusa's: a lightweight fine-tuning run against a much larger, already-trained backbone, not a from-scratch pretraining effort the way Option 1's dedicated 700M draft model would need.

Parameter efficiency across the model-based drafts

With three model-based drafting mechanisms now derived (Option 1's dedicated model, Medusa's heads, and EAGLE's feature-level layer), it's worth comparing them on a metric none of the earlier tables isolated directly: how much speedup each buys per extra parameter spent.

Medusa's heads win on raw parameter efficiency — unsurprising, since they're the smallest addition of the three, reusing the target's own unembedding entirely. EAGLE wins on absolute speedup despite using more parameters than Medusa, because feature-level autoregression captures a genuinely richer signal than several independent linear heads reading a single shared hidden state can. Neither number alone tells a team which to choose; it depends on whether GPU memory headroom or peak wall-clock speedup is the binding constraint for a given deployment.

EAGLE-2 and EAGLE-3: pushing the same idea further

EAGLE-2 replaces Medusa's offline-calibrated, fixed tree shape (Chapter 4's 64-node example) with a dynamically grown tree: at each drafting step, expand more branches where the draft model's own predicted confidence is high, and prune low-confidence branches early, rather than committing to one tree shape in advance for every request. Since the draft model's confidence at each node is itself a byproduct of the feature-level prediction, this costs almost nothing extra to compute. The practical effect: for roughly the same total node budget as a fixed 64-node tree, EAGLE-2's tree covers more of the genuinely likely continuation paths and fewer of the unlikely ones a static tree has to keep “just in case,” which raises the effective tokens-per-round somewhat further — published results put this in the range of an additional 10–30% over EAGLE's already-improved baseline, on top of everything derived above. Apply the conservative end of that range to this chapter's own worked numbers, holding τ roughly fixed (the dynamic tree changes which nodes get spent, not how many, so it doesn't add a new sequential-drafting cost):

A meaningful further gain, from a change that costs almost nothing extra to compute — the confidence scores driving the dynamic tree are a free byproduct of the same feature-level forward pass EAGLE already runs.

EAGLE-3 goes one step further architecturally: instead of drafting from a single fixed hidden-state layer, it fuses features from multiple layers of the target model at different depths, and drops the constraint that the draft model has to predict an intermediate feature vector exactly — it trains directly against the actual next-token objective instead. Both changes loosen a training constraint that turned out to be limiting how good the feature-level draft could get, pushing α higher still on the same underlying mechanism this chapter derived. Applying a further conservative improvement of roughly 8% to EAGLE-2's already-derived 3.70 tokens/round (published EAGLE-3 results report additional gains in this range on top of EAGLE-2, from the combination of multi-layer feature fusion and the direct next-token training objective):

Three successive refinements of the same core idea — draft on features, not tokens — each adding a further increment on the same α/τ/tree-shape levers this lesson has used throughout: EAGLE improves α and τ over a token-level draft, EAGLE-2 improves tree-node efficiency, EAGLE-3 improves the feature representation itself. None required a new theory of what “speedup” means; every improvement is a better point on the same map Chapter 2 drew.

Neither EAGLE-2 nor EAGLE-3 changes anything about Chapter 1's correctness proof or Chapter 2's speedup formula — both remain draft-then-verify systems using ordinary rejection sampling (or Medusa-style typical acceptance, if a deployment opts into that trade) against the target model's real distribution. What changes across the EAGLE family is purely which candidates get proposed and how the tree covering them gets shaped — the entire family lives inside the same α/τ/tree-node-budget framework this lesson has built since Chapter 0, it just keeps finding better points inside that framework to sit at.

EAGLE-2's tree, checked against Chapter 4's budget

EAGLE-2's dynamically grown tree lives inside the exact same 156-node idle-compute budget Chapter 4 derived for Medusa — nothing about drafting on features instead of tokens changes the underlying physical constraint, only which nodes end up worth keeping. A representative EAGLE-2 tree, calibrated to a similar total node count as Medusa's 64-node example, spends that budget differently: rather than a single fixed shape reused for every request, the confidence-driven growth described earlier in this chapter routes more of those 64 nodes toward whichever branches the current request's context makes genuinely likely, and fewer toward branches a static, request-agnostic calibration pass would have had to keep “just in case.” The node count comparison against the budget is identical to Chapter 4's table; what changes is the return on each node spent, which is precisely why EAGLE-2's estimated 3.70 tokens/round beat Medusa's comparable-budget baseline earlier in this chapter's numbers.

Why EAGLE became a popular default

Collecting this chapter's numbers into one summary: EAGLE achieves the highest single-technique speedup of any model-based drafting mechanism this lesson derives (2.96× at the shared k=4 baseline, climbing toward 3.5× with EAGLE-3's refinements), at a parameter cost (0.24B) that's a fraction of a dedicated small model's (700M), by exploiting one structural insight — a hidden-state feature vector carries more information than a sampled token id — that composes cleanly with every other lever in this lesson. Nothing about EAGLE conflicts with Chapter 3's prompt lookup (a request could use lookup when a match exists and fall back to EAGLE otherwise) or Chapter 7's occupancy throttling (EAGLE's small τ makes it one of the cheaper techniques to keep running even as batch occupancy rises, per that chapter's graceful- degradation argument). That combination of a genuinely large single-technique win and easy composability with everything else in this lesson is exactly why feature-level drafting has become a common default choice for teams standing up speculative decoding for the first time.

One more sanity check: does a bigger target change any of this

Every number in this chapter used the lesson's running 7B/A100 pairing. Scale the target up to the 70B/2×H100 configuration from Chapter 0's sensitivity check, and EAGLE's relative advantage over a token-level draft holds up structurally, even though the absolute numbers shift: the target's hidden dimension grows (larger models generally use wider hidden states), which raises the absolute parameter count of both a dedicated small model and EAGLE's one-layer draft roughly in proportion — but EAGLE's structural advantages (reusing the frozen output head, needing only one layer instead of a multi-layer separate model) apply at any scale, so the qualitative story — higher α, lower τ, both compounding — is not an artifact of this lesson's specific 7B choice.

That scale-invariance is worth naming as one more reason feature-level drafting generalizes well across a serving fleet: a team running several differently-sized models doesn't need a fundamentally different drafting strategy per size class, only the same EAGLE recipe re-trained against each target's own backbone. Contrast that with a dedicated small draft model, where the right draft size for a 7B target and the right draft size for a 70B target are two genuinely separate design decisions, each needing its own from-scratch training run.

The same recipe-reuse argument extends to a fleet serving several genuinely different model families, not just different sizes of the same family — each target still trains its own EAGLE draft against its own frozen backbone, but the training procedure, infrastructure, and evaluation harness stay identical across all of them, which is a meaningfully smaller ongoing engineering surface than maintaining a separate bespoke draft-model architecture and training pipeline per target family.

With EAGLE's own numbers now derived and generalized, Chapter 6 changes direction entirely — not another way to draft candidates for verification, but a mechanism that skips the draft-and-verify split altogether, letting the target model iterate directly toward its own future tokens.

It's worth sitting with how much ground the EAGLE family covers on its own: from a token-level baseline around 1.8×, through EAGLE's 2.96×, to an EAGLE-3 estimate near 3.5×, all from three successive refinements of a single idea — draft on the richer feature, not the collapsed token — none of which required touching Chapter 1's correctness proof or Chapter 2's speedup formula even once.

That stability is the real takeaway to carry forward: three generations of real engineering improvement, all captured entirely inside the two numbers — α and τ — this lesson defined back in Chapter 2, with nothing else in the framework ever needing to change underneath them.

Chapter 6 now leaves the entire draft-then-verify family behind for a genuinely different mechanism, one that needs no drafting model, no extra heads, and no separate q distribution at all — worth reading with that structural shift in mind rather than expecting one more variation on this chapter's theme.

It borrows a numerical-methods idea instead of a machine-learning one, and it's worth reading with fresh eyes rather than trying to map it back onto α and τ before Chapter 6 has had a chance to build its own vocabulary first.

Even so, by the end of that chapter the same underlying question — how many extra tokens does one forward pass reliably buy — resolves to numbers in the same 2–3.5× neighborhood this chapter and its predecessors have each landed on, from a genuinely different starting point.

That convergence, across six structurally different mechanisms, is itself a kind of evidence: the ceiling isn't an artifact of any one technique's design, it's a property of the underlying idle-compute budget they're all drawing from.

Chapter 6: Lookahead Decoding

Every technique so far shares one structural feature: a designated draft (a separate model, extra heads, or a lookup) proposes candidates, and the target model verifies them via Chapter 1's rejection sampling. Lookahead decoding is built on a genuinely different mechanism — there is no separate draft model or heads at all. The target model guesses its own future by iterating on a whole window of positions at once, using a numerical-methods idea borrowed straight from solving systems of equations: Jacobi iteration.

The Jacobi view of autoregressive decoding

Plain autoregressive decoding is a strictly sequential process: token 5 needs token 4, which needed token 3, and so on — solve for one unknown at a time, in order, each depending on the one solved just before it. That's structurally identical to Gauss-Seidel iteration, a classical method for solving a system of coupled equations one variable at a time, using each newly-solved variable immediately in the next one's equation.

Jacobi iteration is the alternative classical method: instead of solving one variable at a time, guess values for all the unknowns at once, then update every one of them simultaneously using the previous round's guesses for all the others, and repeat until the guesses stop changing (a fixed point). Lookahead decoding applies exactly this idea to token generation: maintain a window of, say, 8 future token positions, all currently holding guessed values (initialized arbitrarily, or from a running n-gram pool — more on that below). Run the target model's forward pass once, computing what each position's next-token distribution would be given the current guesses at every other position in the window, and update every position simultaneously with a fresh guess. Repeat.

Jacobi vs. Gauss-Seidel, on two ordinary equations first

Before mapping any of this back to tokens, it's worth seeing Jacobi iteration on the kind of tiny linear system it was originally built for, since the token version is just this same procedure with a language model standing in for the equations. Take two coupled equations, x = (6 − y) ÷ 2 and y = (7 − x) ÷ 3, and start both unknowns at a guess of 0.

Gauss-Seidel (sequential, like plain decoding): solve x first using the current y=0, then immediately use that new x to solve y:

Jacobi (parallel, like lookahead decoding): solve both x and y in the SAME step, using only the previous round's values for both — here, both still 0 in round 1:

Round 2 of Jacobi now updates both again, using round 1's fresh values for both simultaneously: x = (6−2.333)÷2 = 1.834, y = (7−3.0)÷3 = 1.333. Both methods converge toward the same true solution (x≈1.571, y≈1.810) as more rounds run; Jacobi just gets there by updating everything at once from stale information rather than immediately using the freshest value available. That's the exact tradeoff lookahead decoding makes: some guesses in a Jacobi round are based on still-unconfirmed guesses elsewhere in the window, which is why not every position converges to the truth on the very first pass — but nothing about the method requires them to, only that the confirmed prefix keeps growing round over round.

The guarantee that makes this safe

This has a genuine mathematical property, not just an empirical one: on every iteration, the length of the confirmed-correct prefix — the positions that now match exactly what greedy decoding would have produced, verified by checking that the token following each guessed position is consistent with the guess before it — is guaranteed to be non-decreasing, and in every iteration that isn't already converged, it increases by at least one. In the worst case, exactly one new correct token gets confirmed per iteration; in the best case, several guesses in the window turn out right simultaneously and the confirmed prefix jumps forward by more than one token in a single pass.

That worst-case guarantee is what makes this technique provably never slower than plain decoding, given the Chapter 0 idle-compute budget: one Jacobi iteration processes a whole window of W positions in one forward pass, but (as long as W stays under the 156× idle-compute line, same as any other technique in this lesson) it still costs the same ≈7 ms as a single plain decode step, because the weight read is identical. Worst case, W iterations confirm W tokens — exactly W × 7 ms, matching plain autoregressive decoding exactly. There is no scenario, in principle, where lookahead decoding is slower.

A worked example: window of 8, converging over 3 iterations

Take a window size W=8. Suppose, on real text with the kind of local repetition and predictable phrasing typical of natural language, the first Jacobi pass confirms the first 3 positions correctly (the guesses there happened to match), the second pass — now starting from a better-informed window — confirms 3 more, and the third pass confirms the final 2:

Notice this lands in the same rough 2–3.5× range as every other technique in this lesson — not a coincidence. All of them are ultimately bounded by the same underlying question (how many tokens can one memory-bound pass reliably confirm), just approaching it through different mechanisms: propose-then-verify (Chapters 1–5) versus iterate-to-a-fixed-point (this chapter).

It's useful to model this the same way Chapter 2 modeled acceptance — treat “probability a given window position converges to the correct token on a given pass” as an implicit parameter, call it β (beta), and note the same qualitative shape reappears: more passes are needed as β falls, and the number of passes needed to fill a window of size W scales roughly like W ÷ (tokens confirmed per pass), where a higher β predictable text lets more positions converge per pass. On highly repetitive text (code with predictable syntax, common phrasing), β is effectively high and 2–3 passes fill a window of 8; on highly novel, unpredictable text, β drops toward the worst-case regime where nearly every pass confirms only its guaranteed single token, and lookahead decoding's speedup drifts toward its break-even floor of exactly 1× — never below it, per the guarantee above, but not meaningfully above it either.

Notice the worst case never gets worse than 1× regardless of window size — that's the mathematical guarantee holding at every W. The typical case improves as W grows, up to the point where W itself starts approaching the 156-unit idle-compute budget from Chapter 0, past which a wider window stops being free to evaluate in one pass and this chapter's whole argument for “never slower” needs the same headroom check every other technique in this lesson needs.

The n-gram pool: making Jacobi's guesses smarter than random

Initializing each new window with random or arbitrary guesses works, but wastes early iterations rediscovering patterns the model has already exhibited earlier in this same generation. Lookahead decoding keeps an n-gram pool: every time a sequence of positions in the window converges to a confirmed, verified n-gram, that exact (context → continuation) pair gets cached. The next time a similar context appears — a common code idiom repeating, a phrase structure recurring — the new window's initial guesses are seeded from the pool instead of starting cold, which tends to raise the fraction of positions that converge on the very first pass, directly increasing the average tokens-per-pass this chapter just derived. This is conceptually the same trick as Chapter 3's prompt lookup drafting, just applied to text the model itself generated earlier in this same response rather than to the prompt.

Work a small hit-rate example to see the effect concretely. Suppose, without an n-gram pool, cold-started windows converge in an average of 4 passes to fill W=8 (a plausible baseline for moderately repetitive text). With the pool warmed up after a few hundred tokens of generation, suppose 40% of new windows now find a full or partial seed match, cutting their convergence to 2 passes, while the remaining 60% still take the original 4:

The pool costs almost nothing to maintain (an append to a lookup table on every confirmed n-gram, a lookup on every new window) and its benefit compounds over the length of a single response — the longer a generation runs, the more the pool has seen, and the higher the pool-hit fraction tends to climb, which is why lookahead decoding's real speedup on a long response is often noticeably better than on a short one.

A practical wrinkle: windows that cross a natural stopping point

One detail every real implementation has to handle that the idealized window picture glosses over: a Jacobi window can guess past the point where the response was actually going to end. If the true continuation is only 3 more tokens before an end-of-sequence marker, but the window holds 8 guessed positions, positions 4 through 8 are guessing at text that was never going to exist. This isn't a correctness problem — those extra guessed positions simply fail to confirm against anything real and get discarded once the end-of-sequence token is confirmed at position 3 — but it does mean the effective tokens-per-pass number this chapter derived is somewhat optimistic very close to the end of a response, since part of the window's guessing capacity is being spent on text that will never be used. For long responses, this edge effect is a small fraction of the total generation and barely moves the average; for very short responses, it can meaningfully eat into the realized speedup, which is one more reason lookahead decoding (like every technique in this lesson) tends to show its best numbers on longer generations rather than one-token or few-token replies.

Real implementations typically shrink the window size dynamically as a response approaches wherever an end-of-sequence token seems likely (informed by the target model's own probability of emitting it at the current position), trading a smaller nominal window for less wasted guessing capacity right at the tail of a generation — a small refinement on top of the fixed-window picture this chapter built, not a change to any of the underlying guarantees.

Combining, rather than choosing

Nothing about lookahead decoding's Jacobi mechanism and Chapters 3–5's draft-then-verify mechanisms is mutually exclusive — several real systems run both at once. Lookahead decoding's n-gram pool can seed its window guesses from a genuine draft model's proposals rather than only its own prior convergence history, and a draft-then-verify system's rejected-token fallback can itself be treated as the start of a fresh Jacobi window rather than a plain single-token resample. The techniques in this lesson are better thought of as a shared toolkit for spending the same idle-compute budget than as a menu where exactly one item gets chosen; a production serving stack picks whichever combination best fits its workload's measured α and its appetite for the operational overhead a second model brings.

Where lookahead decoding sits among this lesson's six techniques

It's worth placing this chapter's mechanism explicitly against the propose-then-verify family from Chapters 1 through 5, now that both are on the table. Every draft-then-verify technique risks a genuinely negative speedup when α is low and τ is nonzero, as Chapter 2's negative example showed concretely. Lookahead decoding cannot produce that failure mode at all — its worst case is exactly break-even, proven from the non-decreasing confirmed-prefix property, not merely observed empirically. What it gives up in exchange is a ceiling: on highly predictable, repetitive text, a well-matched draft model or prompt-lookup source can occasionally exceed lookahead decoding's typical 2–3× range, because a genuine model of the target's distribution (Chapters 1–5's α) can, in principle, capture patterns a pure fixed-point iteration over the target's own single forward pass per round cannot see as directly. The honest summary: lookahead decoding trades a small amount of peak-case upside for the only hard safety guarantee in this entire lesson.

A deployment note: zero new infrastructure

One more practical point worth making explicit before the quiz: because lookahead decoding needs no second model, no extra trained heads, and no change to the target model's weights at all, it's the single easiest technique in this lesson to trial on an existing production deployment. Standing it up is purely a decoding-loop change — how the existing target model's forward pass gets called and how its outputs get interpreted — with nothing to train, version, or host separately, and with Chapter 1's exactness (or a documented, opt-in relaxation of it, mirroring Medusa's typical-acceptance mode) available depending on how the fixed-point convergence check is implemented. For a team wanting to validate this lesson's core promise — genuine speedup from idle compute, with a correctness guarantee — before committing to any second model's training and deployment lifecycle, lookahead decoding is often the fastest path to a first working proof of concept.

That doesn't make it the final answer for every deployment — Chapter 5's EAGLE, once trained, typically wins on peak speedup, and Chapter 3's prompt lookup is at least as cheap to add for overlap-heavy workloads specifically. But as a first experiment to confirm this lesson's central claim actually holds on a team's own model and hardware, before investing in any training pipeline at all, lookahead decoding's zero-new-infrastructure property makes it a genuinely useful starting point, not just a theoretically elegant one.

A team that starts here and later wants to layer on a trained draft mechanism doesn't have to throw this chapter's work away, either — per the “combining, rather than choosing” point above, the Jacobi window and n-gram pool built for a pure lookahead-decoding deployment become the seed and fallback path for a later EAGLE or Medusa rollout, rather than a separate, disposable prototype.

With all six drafting mechanisms now on the table — four ways to source candidates from Chapter 3, two ways to fold drafting into the target's own pass from Chapters 4 and 5, and this chapter's draft-free fixed-point alternative — Chapter 7 turns to the constraint every one of them shares: none of this arithmetic assumed a busy server, and a busy server is exactly what production traffic provides.

Recap: Jacobi iteration's three ingredients

Lookahead decoding also has to live inside the idle-compute budget

It's worth being explicit that this chapter's window width W is not exempt from Chapter 0's constraint, even though the framing above has mostly discussed it in isolation. A Jacobi iteration processes W token positions in one forward pass, exactly the same way a Medusa tree processes its node count or an EAGLE tree processes its own candidate positions — and W is subject to the identical 156-token-equivalent ceiling. Practically, this means window sizes in the range this chapter worked with (4 to 16) sit nowhere near that ceiling on their own, leaving most of the budget still available for real concurrent traffic. A team tempted to push W much larger, chasing a higher typical-case tokens-per-pass number, needs to check that expanded window against the same 156 line Chapter 4 checked Medusa's dense tree against — the mathematics doesn't distinguish between a wide Jacobi window and a wide Medusa tree; both are simply “how many token positions does this one forward pass have to evaluate.”

This also means lookahead decoding is subject to the exact same batch-occupancy interaction Chapter 7 formalizes for every other technique in this lesson: a wide window that's essentially free at low concurrency can start competing with real traffic's own share of the budget once B climbs high enough, just like a deep speculative draft chain would. Nothing about this chapter's worst-case-never-slower guarantee changes that — the guarantee is about correctness of pacing relative to plain decoding at a fixed batch occupancy, not about immunity from the shared-budget arithmetic the very next chapter builds out in full.

Read Chapter 7 as the one that applies to all six techniques in this lesson simultaneously, lookahead decoding included, rather than as a coda specific to the propose-then-verify mechanisms alone.

That closes the mechanism half of this lesson. Everything remaining is about what happens once these mechanisms meet a real, busy, multi-tenant server, rather than the idealized single-user picture every derivation so far has quietly assumed.

That transition matters more than it might first appear — a technique that looks strong in isolation and a technique that stays strong under real, shared load are not automatically the same technique, and the next chapter is where that distinction gets made precise.

Chapter 7: Speculation Meets the Batch

Every derivation so far quietly assumed one thing: that the 156× idle-compute budget from Chapter 0 is sitting there, unclaimed, waiting for speculative decoding to spend it. That assumption is true for a single user talking to an otherwise-idle GPU. It stops being true the moment a production server starts doing what Serving Engines teaches it to do: pack many concurrent users into the same batch, filling that exact same idle compute with real, revenue-generating decode work instead.

The two techniques are competing for the same budget

Recall the crossover derivation from Chapter 0: a batch of B concurrent real requests shares one weight read per step, so the step stays memory-bound (fixed 7ms) as long as B stays under the same 156-token-equivalent line — that's not a coincidence; it's the identical ratio (fixed memory time ÷ per-token compute time at peak) that gave Chapter 0 its idle-compute headroom in the first place. A server running B real concurrent decode requests has already spent B of that 156-unit budget on genuine users. Whatever remains — 156 − B token-equivalents — is the only budget actually available for speculative verification on top, without slowing anyone down.

Worked example: near-saturated batch, one speculating request

A server is running B=140 concurrent real decode requests — comfortably under 156, but not by much. Headroom remaining:

One new request wants to speculate with k=4 (per Chapter 2's worked example) — verifying k+1=5 candidate positions costs roughly 5 token-equivalents of the shared compute budget:

Now suppose, instead, 30 concurrent requests all try to speculate with k=4 simultaneously — a plausible scenario if a naive scheduler just turns speculation on for every request unconditionally:

The step is now solidly compute-bound. Its wall-clock time no longer sits at the fixed 7 ms — it grows roughly in proportion to how far total demand exceeds the 156 line, and every single request in that batch, speculating or not, now waits longer per step than it would have with speculation turned off entirely. A policy that speculates unconditionally, ignoring current batch occupancy, can make the whole server slower under exactly the high-traffic conditions where throughput matters most.

The opposite worked example: a quiet server, plenty of headroom

Run the same arithmetic at low concurrency, to see the other side of this tradeoff explicitly. A server running B=20 real concurrent requests — a realistic number for an internal tool, an off-peak overnight window, or a lightly-loaded single-tenant deployment:

Suppose every one of those 20 requests speculates at k=4 (5 token-equivalents each, this lesson's running worked example):

Every one of them speculates for free, with room to spare — the server could push considerably harder still, up to roughly 136 ÷ 5 ≈ 27 requests' worth of k=4 speculation before hitting the line, well beyond the 20 actually present. This is the regime where every technique in Chapters 1–6 delivers close to its full, uncompromised speedup number, and it's also the most common regime for a huge share of real deployments: single-tenant assistants, internal tools, and low-traffic hours on any service with a diurnal usage pattern.

Per-request adaptivity: the actual policy

The fix follows directly from the arithmetic: a scheduler should track current batch occupancy B (or an equivalent proxy — queue depth, KV cache pressure) in real time, and adjust k, or whether to speculate at all, based on how much of the 156-unit budget is still free:

Measuring B in practice

“Current batch occupancy” sounds like a number a scheduler could just read off, but a real serving stack usually only has proxies for it, not the exact token-equivalent figure derived above. The most direct proxy is simply the number of sequences currently in the decode batch, which maps to B almost exactly under this lesson's assumptions. A second, often more actionable proxy is KV-cache memory pressure — how much of the GPU's memory budget for cached keys and values is currently claimed — since that tends to rise and fall in step with real concurrency and is already instrumented on any server using paged KV allocation. A third, lagging but easy-to-monitor proxy is queue depth: a growing request queue is a leading indicator that occupancy is about to rise, useful for throttling speculation slightly ahead of the batch actually filling up rather than reacting after the fact. The Autoscaling Inference lesson in this same path builds these exact signals out in far more depth, for the related but distinct problem of deciding when to add GPU capacity rather than when to throttle a feature on existing capacity.

There's also a dollar angle worth naming briefly, developed in full in The Economics of Inference: every token-equivalent of compute spent on a speculative round that ultimately gets rejected past the compute-bound crossover isn't just a latency cost to the requests sharing that batch — it's GPU-hours being billed for arithmetic that produced zero extra tokens for anyone. A scheduler that ignores batch occupancy doesn't just risk a p99 latency regression under load; it risks quietly paying for wasted compute at exactly the traffic volumes where the GPU-hour bill is largest.

Why this doesn't apply uniformly to every technique in this lesson

The 156-unit budget applies to all of them, but not equally hard. Chapter 3's prompt-lookup drafting costs almost nothing to propose (τ≈0) — its budget hit comes entirely from the verification pass's extra positions (k+1 token-equivalents), same as everything else. Chapter 4's Medusa and Chapter 6's lookahead decoding both fold their “drafting” into the same single pass as verification, so their entire budget cost is exactly their node count or window size — no separate drafting-pass tax at all. Chapter 3's dedicated small model and Chapter 5's EAGLE both add a genuinely separate sequential drafting cost on top of verification's budget hit, which itself competes for GPU time even before the batch-occupancy question comes up. Under high load, the folded-in techniques (Medusa, lookahead, EAGLE with its tiny 0.24B draft) degrade more gracefully than a fully separate draft model does, because they're spending less of an already-shrinking budget per request.

Who gets the remaining headroom: a fairness question

Chapter 7's policy table treats remaining headroom as a single shared pool, but a real multi-tenant server usually has to decide which requests get to spend it once headroom is scarce, not just how much total headroom exists. One reasonable policy: prioritize headroom for interactive, latency-sensitive sessions (a human waiting on a chat reply) over offline batch jobs (an overnight summarization run with no one watching the clock) — the batch job's total completion time barely changes whether it speculates or not, while an interactive session's felt responsiveness depends heavily on it. Work a small example: a server at B=120 has 36 units of headroom (156−120). Ten interactive sessions requesting k=4 (5 units each, 50 units total) exceed that headroom alone; a fairness-aware scheduler grants full k=4 speculation to the 7 highest-priority interactive sessions (35 units, just under the 36-unit ceiling) and drops the remaining 3 interactive sessions and any batch jobs to k=0 (plain decoding) for that step, rather than splitting the headroom evenly and giving every session a degraded, sub-optimal k that serves nobody particularly well.

This is a genuine policy choice, not a fact the arithmetic hands you — Chapter 8 returns to this point explicitly as one of the things this lesson's formulas don't decide.

What to actually put on a dashboard

Turning this chapter's arithmetic into an operable system means instrumenting a small number of real-time signals, not just running the formula once at design time. A minimal monitoring setup tracks four numbers continuously: current batch occupancy B (or its KV-pressure proxy), the realized α per draft source per workload class (measured, not assumed, per Chapter 3), the fraction of decode steps currently running past the 156-unit line (a direct signal that speculation is costing rather than saving time right now), and p99 inter-token latency split out separately for speculating versus non-speculating requests. That last split matters specifically because Chapter 7's failure mode is silent from a request's own point of view — a request that isn't speculating still slows down when its batch-mates overload the budget, and without splitting the metric, a rising p99 during a speculation-heavy traffic period can be misread as an unrelated capacity problem rather than the direct, fixable consequence of an under-throttled speculation policy.

A worked multi-tier policy, put together end to end

Combine every threshold from this chapter into one concrete example server configuration, to see the whole policy operating together rather than one rule at a time:

None of these four rows required a new idea — every threshold traces back to a number already derived earlier in this chapter. What a real system adds on top is exactly this: a continuously updated read of where the server currently sits on that scale, and a policy that moves through these rows automatically as traffic shifts, rather than a human operator manually flipping a flag after the fact.

Relating this back to Autoscaling Inference

This chapter's headroom line and this path's Autoscaling Inference lesson's scaling triggers are solving related but genuinely different problems, worth distinguishing clearly. Autoscaling decides how many GPU replicas a fleet needs, reacting on the timescale of seconds to minutes as aggregate demand shifts — add a replica when queue depth or aggregate concurrency crosses a threshold, remove one when it's been idle long enough to justify the cost of a cold start on the way back up. Chapter 7's occupancy-aware speculation policy operates one layer beneath that, on the timescale of individual decode steps within a single already-running replica — deciding, step by step, how much of that one replica's idle-compute budget speculation gets to spend right now. A well-run fleet needs both: autoscaling keeps the number of replicas roughly matched to aggregate demand, and per-replica speculation throttling keeps each individual replica's own decode steps honest about how much idle compute genuinely remains once real traffic has claimed its share.

The two systems do share one thing worth noting: both ultimately react to the same underlying signal, real concurrent request volume, just averaged over different timescales and used to trigger different kinds of action. A team building both systems can, and generally should, source B (or its proxies) from one shared telemetry pipeline rather than maintaining two separate measurements of what is, at bottom, the same quantity.

Why this chapter had to come after, not before, Chapters 3 through 6

It's worth noting explicitly why this lesson placed the batching interaction last rather than folding it into each drafting mechanism's own chapter. Every technique in Chapters 3 through 6 is affected by occupancy in the same structural way — their token-equivalent cost per round competes with real traffic for the identical 156-unit line — but each is affected by a different amount, depending on whether it pays a separate sequential drafting cost (Options 1 and the dedicated small model, EAGLE) or folds drafting into the verification pass at near-zero extra cost (prompt lookup, Medusa, lookahead decoding). Deriving the general occupancy relationship once, here, and then applying it back across every technique this lesson already covered is more honest than repeating a partial version of this argument six times, once per mechanism, before any of the mechanisms had even been fully derived.

The single sentence this chapter adds to the lesson

Every earlier chapter answered some version of “how much idle compute is there, and how do I spend it well.” This chapter's entire contribution is one qualifier on top of all of that: the answer to “how much idle compute is there” is not a constant — it's 156 − B, and B is whatever a live server's real traffic happens to be doing at this exact moment. Every speedup number in Chapters 1 through 6 is conditional on that quantity staying positive, and this chapter is the one that makes that condition explicit and operational rather than an unstated assumption.

That single qualifier is also the difference between a technique that reliably ships value in production and one that only ever demonstrates well in a controlled benchmark environment, where B is implicitly zero and never mentioned. Any speculative decoding number reported without stating the batch occupancy it was measured at should be read as this chapter's B=0 special case, not as a number that automatically holds once real concurrent traffic is added — a distinction worth carrying into how any future benchmark or vendor claim about speculative decoding gets read.

Practically, this means the single most useful question to ask of any reported speculative-decoding speedup number is simply: at what concurrent batch occupancy was this measured, and does that occupancy resemble the traffic this deployment will actually carry? A number measured at B=0 and deployed at B=140 is answering a different question than the one that actually matters, and this chapter's entire arithmetic exists to make that gap precise rather than anecdotal.

With the full picture now assembled — how much idle compute exists, how to spend it safely, how much it's worth, six ways to actually spend it, and how much of it survives once real traffic is added — the final chapter puts every piece side by side in one live simulation.

A day in the life of a diurnal traffic pattern

Most consumer-facing services see traffic rise and fall on a roughly 24-hour cycle, and it's worth walking one full day through this chapter's policy table to see how much the right speculation setting actually moves. Overnight, from roughly 2am to 6am, batch occupancy for a typical regional service might sit around B=10 — deep in the “quiet” regime, full aggressive speculation warranted across the board. As morning traffic ramps from 7am to 10am, B climbs through the 40–80 range — the “moderate” regime, where interactive sessions keep full speculation but batch and offline work gets throttled back. Midday peak, 11am to 2pm, might push B to 130–150 — “near-saturated,” where only the cheapest draft sources stay active. And a launch-day traffic spike, or a viral moment, could genuinely push B past 156 for a sustained period — the “over budget” regime, where the correct policy is to disable speculation fleet-wide rather than let it degrade throughput during the exact window when throughput matters most.

A server running one fixed k all day, tuned for the overnight quiet hours, would be actively hurting itself during the midday peak; a server running one fixed k tuned for the midday peak would be leaving most of the overnight hours' free speedup on the table. Neither fixed setting is wrong exactly — each is simply answering the wrong hour's question. The entire point of Chapter 7's occupancy-aware policy is that it doesn't have to choose; it tracks B continuously and answers the question the traffic is actually asking, hour by hour, minute by minute if the underlying telemetry supports it, with no human operator needed in the loop.

That's the difference between a policy derived once at design time and a policy that stays correct as traffic itself changes shape, which is the only kind worth shipping to a fleet that runs continuously.

Chapter 8: The Full Race

Nine chapters have built this lesson's argument in order: an idle-compute budget (Ch. 0), a proof that spending it doesn't change the output (Ch. 1), a formula for how much it's worth spending (Ch. 2), where the candidates actually come from (Ch. 3), two ways of folding drafting into a single pass (Ch. 4, 5), a third mechanism that doesn't need a draft at all (Ch. 6), and the real-world constraint that a live server's budget is shared with genuine traffic (Ch. 7). This closing chapter puts every piece in one place: a live, side-by-side race between vanilla decoding and speculative decoding, with every knob from the lesson exposed as a control.

What the race actually simulates

Two token streams generate the same underlying response in parallel. The vanilla stream emits exactly one token every 7 ms, per Chapter 0's memory-bound floor — no shortcuts, the honest baseline. The speculative stream runs verification rounds using Chapter 2's exact formula: each round draws k drafted tokens, accepts them stochastically at the workload's acceptance rate α (color-coded green for accepted, red for the first rejection), and advances by however many tokens that round actually produced — timed against the round cost from Chapter 2 (verification pass plus drafting overhead), further slowed if Chapter 7's batch-occupancy toggle pushes total demand past the 156-unit budget.

Reading the result

Set the workload to code editing at low batch occupancy and the speculative stream should pull steadily ahead, several tokens per round, matching Chapter 3's 3.47× figure closely once the race settles. Switch to creative writing at k=8 and the gap should narrow or vanish — sometimes the vanilla stream catches up entirely, echoing Chapter 2's negative-speedup example. Drag batch occupancy up past roughly 150 on any workload and watch the speculative stream's advantage erode in real time as Chapter 7's shrinking headroom bites — the exact same numbers derived by hand two chapters ago, now animated.

A few more explorations worth running by hand before moving on. Hold the workload fixed on open chat (α=0.60) and sweep k from 1 up to 10 — the speculative stream's lead should grow, then visibly stop growing (and start narrowing again past roughly k=6–7), tracing out the same peak-then-fall shape Chapter 2 derived algebraically. Then, at any workload, push batch occupancy from 20 up past 156 in one motion and watch the transition happen live: below the line, the speculative stream's round time stays flat; above it, round time visibly starts stretching, exactly the compute-bound regime Chapter 0 first introduced as an abstract idea and Chapter 7 turned into a concrete scheduling threshold.

Choosing among six techniques for a real system

Collapsed into one practical decision path: if the workload has strong prompt-or-context overlap with its output (code editing, summarization, RAG-with-quoting), start with prompt lookup — Chapter 3 showed it wins on both raw speedup and zero deployment overhead, and it can never meaningfully hurt. If the workload is general-purpose with no reliable overlap pattern, and the team can afford to host and maintain a second model, EAGLE (Chapter 5) offers the best speedup-per-parameter of any model-based option this lesson covered. If minimizing operational surface area matters more than squeezing out the last bit of speedup — no second model to version, retrain, or drift out of sync — Medusa's extra heads (Chapter 4) or lookahead decoding's pure algorithmic reframing (Chapter 6) both avoid a separate model entirely, at some cost in peak speedup versus EAGLE. And regardless of which drafting mechanism gets chosen, Chapter 7's occupancy- aware throttling isn't optional once the server carries real concurrent traffic — it's the difference between a technique that helps in production and one that only ever looked good in a single-user benchmark.

The core chain, rebuilt from memory

Before the final comparison table, it's worth checking that the whole argument can be reconstructed from just five numbers, without looking anything up — that's the real test of whether this lesson's arithmetic stuck, rather than just its conclusions:

Everything from Chapter 3 onward is a variation on how row 3 (α) and the implicit τ get achieved — a cheaper draft source, a folded-in mechanism, a richer feature representation, an iterative fixed point — and everything in Chapter 7 is about what happens to row 2 once real concurrent traffic claims part of that 156× budget for itself.

The comparison table, all nine chapters in one place

What no chapter of this lesson decides for you

Every number in this table is derivable, and every technique's speedup is predictable in advance from the formulas built across nine chapters. What the arithmetic does not decide: whether your specific traffic mix justifies hosting a second model at all (an engineering-effort and maintenance-cost question, not a math question), whether Medusa's typical-acceptance mode's small deviation from Chapter 1's exact guarantee is acceptable for your product (a product-risk judgment), and how aggressively to speculate as batch occupancy fluctuates minute to minute in a real fleet (an operational policy question that Chapter 7 frames but a real autoscaler, covered in this path's Autoscaling Inference lesson, actually has to implement).

Back to Chapter 0's four requirements, checked off

Chapter 0 opened with a table of four things that had to be true before “guess ahead, then verify” could become a real, shippable technique. Worth closing the loop on each explicitly, now that all nine chapters are behind this lesson:

Four open requirements, four chapters (plus the two additional mechanisms of Chapters 5 and 6) that each resolve one of them with a real derivation rather than an assertion. That closure is deliberate: a lesson that opens with a list of what has to be true is making an implicit promise to come back and show exactly where each one got settled, not just to move on once the exciting parts are covered.

What this lesson's model simplified, honestly

Every formula in this lesson rested on treating each drafted token's acceptance as an independent event with a single fixed probability α. Real acceptance probabilities are not independent or uniform across positions — a draft model tends to be more reliable on the first drafted token (closest to genuinely-known context) than the fourth or fifth (compounding its own earlier guesses), so real α effectively decays slightly across a drafted sequence rather than staying flat. The formulas here are the right first-order model and the one every production system starts from, but a team tuning k precisely on real traffic should expect to measure a position-dependent acceptance curve, not a single flat number, and adjust k against the measured curve rather than a single headline α.

This lesson also treated drafting and verification as happening in strict sequence — draft, then verify, paying both costs back to back. Some real implementations pipeline the two stages across consecutive rounds (start drafting the next round's candidates while the current round's verification pass is still running), recovering a further, smaller speedup this lesson's round-time formula doesn't capture. That optimization doesn't change any of the α/k/τ relationships derived here; it shaves a further constant off the round-time denominator, on top of everything this lesson already predicts.

Where this lesson connects to the rest of the Inference Engineering path

This lesson treated the target model's decode step as a fixed 7 ms unit, spent more or less cleverly. Serving Engines, this path's sibling lesson, builds the machinery that makes that unit real in production — continuous batching, PagedAttention, and prefix caching all compete for the exact same idle-compute budget derived in Chapter 0 here, which is precisely why Chapter 7 exists: speculation and batching are not independent techniques, they're two different ways of spending one shared resource. The Economics of Inference already cited this lesson's own numbers directly — its cost-lever ranking used α=0.7, k=4, and this lesson's 2.77-tokens-per-round figure to price speculative decoding as a cost-reduction lever, ranking it ahead of distillation specifically because Chapter 1's proof makes it risk-free in a way a distilled model's changed behavior isn't. Reading these three lessons together answers a complete question a single one can't: what a decode step actually costs, how to spend its idle time well, and how those two answers change once real concurrent traffic is added back in.

Beyond this lesson: building the extra heads in at pretraining time

Every technique in this lesson bolts a drafting mechanism onto a target model that was trained the ordinary way, to predict exactly one next token. An active area this lesson deliberately leaves for later reading trains the target model itself, from the start, with an explicit multi-token prediction objective — several output heads predicting several future positions simultaneously, present during pretraining rather than fine-tuned on afterward the way Medusa's heads are. Early results on this approach report that a model trained this way natively produces noticeably better-calibrated multi-token predictions than heads fine-tuned onto an already-trained backbone, plausibly because the backbone's own internal representations adapt, during training, to make several-steps-ahead prediction genuinely easier, rather than the backbone staying frozen and only a small bolt-on head having to compensate. This doesn't replace anything in this lesson — Chapter 1's verification proof and Chapter 2's speedup arithmetic apply identically regardless of whether the drafting heads were fine-tuned on afterward or trained in from the start — it simply suggests where α itself might keep climbing, on top of every improvement this lesson has already derived.

One last worked number, to leave with

Return, one final time, to the single number this lesson opened with: 0.64% compute utilization, a single-user decode step on an idle A100. Across nine chapters, this lesson turned that one wasted number into a genuine, provable, roughly 2–3.5× wall-clock speedup — not by adding a faster GPU, not by shrinking the model, not by changing a single weight of the target model's own behavior, but purely by noticing that 99.36% of a resource already being paid for, every single step, was sitting completely idle, and building six different rigorous ways to spend it. That's the entire technique, end to end: not a trick, an accounting correction, backed by a proof that nothing about the output ever had to change to collect the gain.

If only one habit survives from this lesson, let it be this: the next time a piece of hardware looks like it's “doing nothing” on a monitoring dashboard, the right question isn't whether that's a problem to fix with faster hardware. It's what, precisely, is bottlenecking it instead, and whether that bottleneck leaves room to do something else, for free, in the meantime. Speculative decoding is one very well-worked answer to that question for one very specific workload. The question itself generalizes far beyond it.

A final self-check

Before leaving this lesson, try reconstructing the following chain purely from memory, without scrolling back: why is a single decode step memory-bound rather than compute-bound; what does that imply about how much idle arithmetic sits inside one 7 ms step; why does verifying several drafted candidates in that same step cost almost nothing extra; why does verification never change what the model would have said, no matter how the candidates were generated; and why does all of this stop being free once a server's real concurrent traffic grows large enough. Each of those five links is one chapter of this lesson, and each was built from arithmetic simple enough to check by hand. That's the whole test this lesson was written to pass: not whether the conclusion sounds right, but whether every step that produced it still holds up on a second look.

Where to go from here

This lesson's sibling lessons in the Inference Engineering path are the natural next stops, and each answers a question this one deliberately left open. Serving Engines builds the continuous-batching and paged KV-cache machinery that Chapter 7's batch occupancy B is actually a number produced by, in real detail. Autoscaling Inference covers the fleet-level decision of how many replicas to run at all, one layer above the per-replica throttling this lesson's Chapter 7 describes. The Economics of Inference turns every speedup number in this lesson into a dollar figure, on a shared baseline app, and already leans on this lesson's own α=0.7, k=4 result directly. None of the three requires re-deriving anything from this lesson — they each take its conclusions as a given and build the next layer on top.

Beyond this path, the same idle-compute argument this lesson built from scratch reappears, in different guises, across essentially every memory-bound accelerator workload — anywhere a fixed data-movement cost dwarfs the arithmetic riding on top of it, there is a version of Chapter 0's question worth asking again: what, exactly, is the idle resource, and what is the cheapest, safest thing to compute with it while the wait was happening anyway.

Common pitfalls, collected in one place

Every row in that table traces back to a specific chapter's derivation, not a vague warning — which is the whole design principle this lesson followed throughout: no claim without arithmetic behind it, and no warning without a chapter that shows exactly how the failure happens, exactly how to detect it before it ships, and exactly what number to check to confirm it's fixed.

Keep this table next to the dashboard from Chapter 7 — between the two, a team has both the real-time signals to watch and the specific failure modes those signals are watching for, which is the complete operational picture this lesson was built to hand off.

That's the lesson, cover to cover: an idle resource, a proof it can be spent safely, an exact price for spending it, six ways to spend it well, and the honest accounting for what happens once real traffic wants a share of the same budget.

Nine chapters, one number at the center of all of them, and a genuinely free lunch — rarer in engineering than the phrase usually implies, and worth the full derivation it took to earn it honestly.

Thank you for working through every one of those derivations by hand. That's what made this lunch actually free, and not just something that sounded free.

“The best way to predict the future is to invent it — but the second-best way is to guess it cheaply and check your guess for the price you were already paying.” That's not a real quote from anyone; it's simply what nine chapters of this lesson add up to. Speculative decoding isn't a trick that makes GPUs faster. It's an accounting exercise that notices a resource already being paid for and refuses to let it sit idle.