Serving Engines: vLLM, SGLang, TensorRT-LLM

Chapter 1: Continuous Batching

Chapter 0 ended with a clean number: batch 156 requests together and the same GPU serves 156 users at the same per-user speed one lonely user got alone. The obvious next move is to just... do that. Collect 156 incoming requests, run them together as one batch, repeat. This is called static batching, and it is the first thing anyone implements. It also throws most of that 156× away, for a reason that has nothing to do with the arithmetic in Chapter 0 and everything to do with a fact Chapter 0 quietly ignored: requests don't all take the same number of steps to finish.

The problem static batching can't see coming

Picture four requests grouped into one static batch, generating tokens one step at a time in lockstep — every request in the batch takes a step together, because that's what "batched" means at the hardware level: one matrix multiply, all four hidden states riding along inside it. Say their true output lengths, decided by whenever each one happens to emit an end-of-sequence token, turn out to be 100, 100, 100, and 400 tokens.

The batch can't move on to a new request until every slot is free, and a slot only frees up when its occupant finishes. So the batch runs for 400 steps, no matter what:

But only some of that work was useful. The three 100-token requests each did 100 steps of real generation, then sat in their slot — finished, done, but not evicted, because static batching only refills the whole batch at once — for the remaining 300 steps while the fourth request kept going alone:

Nobody wrote a bug. Every line of the scheduler is doing exactly what static batching is defined to do. The waste comes from a structural mismatch: real output lengths vary a lot (short answers, long answers, tool calls that loop, refusals that stop in five tokens), and a design that can only refill all slots at once, after the slowest occupant finishes, pays for that variance in idle seats every single round.

The fix: refill a seat the moment it opens

Continuous batching — also called in-flight batching, iteration-level batching, or (in the paper that introduced the idea, Orca, 2022) iteration-level scheduling — removes the "wait for everyone" rule entirely. At the end of every single decode step, the scheduler checks which slots just finished, evicts them, and immediately admits new requests from the queue into those freed seats. The batch composition can change from one step to the next. Nobody waits for a slower neighbor.

Redo the four-request example with a queue of new arrivals waiting behind it. At step 100, three slots free up simultaneously. Under continuous batching, three new requests slide in at step 101 and start their own 100 (or however many) steps immediately — those seats never sit empty. Only the very last handful of steps, once the queue itself runs dry, can go unfilled, and in a live production system with a steady arrival rate, the queue essentially never runs dry.

A worked throughput comparison

Hold the same 900 wasted slot-steps from static batching and ask: if continuous batching recovers essentially all of it, how much more total work does the same 1,600-slot-step budget produce?

That 2.3× came from a toy batch with only one long straggler. Real production traffic mixes many short chat replies with occasional long completions, tool-call loops, and reasoning chains — the length variance is typically wider than this example, not narrower, so the gap between static and continuous batching in a real deployment tends to be at least this large, and often larger.

What "the batch" actually looks like under the hood

Continuous batching needs one more piece: a way to add and remove rows from a batched matrix multiply between steps without re-launching the whole compute graph from scratch. Practically, this means the scheduler keeps a small Python-level (or C++-level) list of "currently active sequences," and on every step it: (1) drops any sequence that hit end-of-sequence or its max-token limit last step, (2) pulls new sequences off the waiting queue to refill those slots up to whatever the current max batch size is, and (3) runs one forward pass over whatever the resulting batch happens to be — which might be size 4 one step and size 6 the next.

python
class ContinuousBatcher:
    def __init__(self, max_batch):
        self.active = []            # currently running sequences
        self.queue = []              # waiting requests
        self.max_batch = max_batch

    def step(self, model):
        self.active = [s for s in self.active if not s.finished]   # evict finished
        while len(self.active) < self.max_batch and self.queue:
            self.active.append(self.queue.pop(0))                       # admit new, this step
        new_tokens = model.forward_batch(self.active)             # one matmul, mixed batch
        for seq, tok in zip(self.active, new_tokens):
            seq.append(tok)
            if tok == EOS or len(seq) >= seq.max_len:
                seq.finished = True

Compare this to the naive server from Chapter 0: that loop called model.generate() once per request, start to finish, before even looking at the queue. This one calls a single-step forward_batch() repeatedly, checking the queue between every step. The difference between those two designs, more than any other single change, is what turns a serving engine into a serving engine.

Prefill still complicates the picture

One detail this chapter has quietly glossed over: admitting a new request into a freed slot isn't free the way resuming an existing one is. A brand-new request needs its prefill pass — processing its entire prompt for the first time, which costs roughly as much compute as its whole prompt length, all at once — before it can join the token-by-token decode loop the other slots are already running. Squeeze a long prompt's prefill into the middle of an otherwise-smooth decode batch and every other request in that batch waits for it. Chapter 4 is entirely about that interaction; for now, treat this chapter's arithmetic as holding for a batch of requests already past their first token, and file away that admission itself has a cost worth revisiting.

What continuous batching does NOT fix

Keep this list honest, because the next three chapters exist precisely to fill these gaps. Continuous batching alone does not solve: how much memory each new arrival should reserve for its KV cache (Chapter 2); whether two requests that happen to share a long common prefix should redundantly recompute it (Chapter 3); or what happens when a newly admitted request's prefill is itself thousands of tokens long and threatens to stall everyone already decoding (Chapter 4). It solves exactly one problem — not wasting a freed seat — and it solves that one completely.

A second, larger worked example: what real length variance does to the waste

Four requests was small enough to trace by hand, but real chat traffic has far more length variance than one long outlier among three short ones. Try a batch of 20 requests where 15 are short replies (20 tokens each) and 5 are longer completions (800 tokens each) — a plausible mix for a support chatbot fielding both one-line answers and detailed explanations.

Widen the length variance and the waste gets worse, not better — 73.1% here versus 56.25% in the smaller four-request example. This is the general pattern: static batching's waste grows with how far the typical request falls short of the batch's slowest member, and real traffic (short acknowledgments next to long structured outputs, agentic loops that run for hundreds of steps next to one-shot lookups) tends to have more of that spread than a toy example, not less.

Two knobs a real scheduler exposes

Production continuous batchers don't admit an unlimited number of new arrivals every step — they cap concurrency with two related settings, worth naming since they show up in every engine's configuration surface under some name: a maximum number of concurrent sequences (a direct cap on batch size, tying straight back to Chapter 0's crossover-batch arithmetic) and a maximum number of total tokens processed per step (since one step's cost depends on total tokens moving through it, not just the sequence count — ten sequences each admitting a 500-token prefill chunk cost very differently than ten sequences each just decoding one token). Both caps exist so batching doesn't try to admit so much work into one step that step latency itself balloons past what any of the concurrently-decoding users can tolerate.

The failure mode continuous batching introduces on its own

Sizing the batch from arrival rate: Little's Law

Continuous batching keeps seats full, but it doesn't answer a prior question: how many seats does the scheduler actually need to configure as its maximum, given real traffic? A classic queueing-theory result, Little's Law, answers this directly: the average number of requests in the system (L) equals the average arrival rate (λ) times the average time each request spends being served (W).

Suppose a chat product sees 50 new requests arrive per second on average, and each request's decode phase runs for about 2 seconds end to end (a plausible length for a moderate chat reply at ~140 tok/s):

That number is directly checkable against Chapter 0's crossover batch of 156: 100 concurrent requests sits comfortably under the crossover, meaning this traffic level, served well, should be able to reach close to full per-user speed for everyone, with room to spare before the GPU tips into the compute-bound regime where per-user latency starts climbing. Push arrival rate to 100 requests/s at the same 2-second service time and L becomes 200 — past the crossover, meaning this single GPU can no longer serve that traffic level without either accepting degraded per-user latency or adding a second GPU. Little's Law is what turns “how much traffic can one engine handle” from a guess into an arithmetic question, once you know your own arrival rate and typical request duration.

The edge case: what if the queue is often empty?

Every advantage this chapter derived assumed a nonempty queue ready to backfill freed seats. At very low traffic — arrival rate far below service capacity — that assumption can fail: a slot frees up and there's genuinely no waiting request to put in it, because none has arrived yet. In that regime, continuous batching's utilization advantage over static batching shrinks toward zero, simply because there's nothing left to schedule more cleverly. This isn't a flaw in continuous batching — it never performs worse than static batching, since an empty queue produces the same idle slot either way — it's just a reminder that the 2.3× and 156× figures in this lesson describe the ceiling reachable under sufficient load, not a guarantee at every traffic level. Low-traffic periods are memory-bound-at-batch-1 territory again, exactly Chapter 0's starting point, no matter how sophisticated the scheduler is.

Measuring it: the metric that catches this in production

The gap between static and continuous batching shows up on a dashboard as a specific, checkable number: batch utilization, the average fraction of configured slots that are doing real work at any given moment, sampled across a serving window. A healthy continuous-batching deployment under steady load should sit close to its own achievable ceiling — not 100% (some slack for arrival variance is normal and healthy), but well above the 44–56% this chapter's worked examples measured under static batching. A production alert worth setting: if batch utilization drifts noticeably below its historical baseline with traffic otherwise unchanged, something in the admission path is silently reverting toward static-batching-like behavior — a stuck eviction check, a scheduler bug holding finished slots open, or a queue that isn't being drained as fast as it should be. The arithmetic this chapter built by hand is exactly what that dashboard number is measuring in production, just averaged over real traffic instead of one toy example.

Fairness across requests admitted at different times

Continuous batching's greedy “fill any open seat immediately” rule has an implicit ordering policy worth naming: first-come-first-served, since new arrivals are drawn from the front of the waiting queue. This is usually the right default, but it can interact badly with wildly different request lengths arriving close together. A very long request admitted at step 0 occupies its seat until it finishes, potentially thousands of steps later, while dozens of much shorter requests queue up behind it, each waiting only for whichever other seats free up in the meantime — not for the long request's seat specifically, since continuous batching never preempts a running sequence just because something shorter arrived. In the worst case, a handful of very long-running requests can occupy a large share of the configured max batch size indefinitely, leaving fewer seats available to backfill for everyone else, which is exactly the kind of scenario Chapter 4's priority and preemption machinery exists to address — continuous batching alone guarantees no idle seats, not that every seat is available to every waiting request equally.

Orca's original framing, briefly

The 2022 Orca paper that introduced iteration-level scheduling framed the problem slightly differently than this chapter's slot-based picture, and the alternate framing is worth having in your vocabulary: instead of thinking in terms of fixed “seats,” think of the scheduler making a fresh admission decision at every iteration (every single decode step across the whole system), rather than only at batch boundaries. “Iteration-level scheduling” and “continuous batching” describe the identical mechanism from two angles — one emphasizing the scheduling granularity (per-iteration, not per-batch), the other emphasizing the effect on utilization (continuous, not bursty) — and both names show up in different engines' documentation for exactly the same underlying idea this chapter derived from scratch.

A quick gut-check before moving on

Before Chapter 2 layers memory management on top of this, confirm the mental model holds: continuous batching's entire contribution is scheduling — deciding when a seat is filled — and says nothing about how much memory that seat costs to fill. A scheduler that perfectly avoids Chapter 1's slot-waste can still run the GPU out of memory if it doesn't also respect Chapter 2's budget; the two problems are genuinely separate, solved by genuinely separate mechanisms, and conflating them is the most common way to misunderstand what “continuous batching” alone actually buys a deployment.

Carry this forward

Hold onto one number from this chapter above all others: continuous batching alone, on the toy example, turned a 43.75% utilization figure into something approaching 100% in steady state — recovering most of the gap between Chapter 0's crossover ceiling and what a naive scheduler would actually deliver. The next two chapters spend that recovered headroom on memory (Chapter 2) and on not stalling anyone with a giant prompt (Chapter 4); none of that spending is possible without this chapter's fix in place first.

Chapter 2: PagedAttention

Chapter 1 fixed when a new request gets a seat. It never asked how much memory that seat should reserve. Answer that question the naive way and you rediscover a version of Chapter 0's waste, except now it's memory sitting idle instead of compute.

The naive allocation: reserve for the worst case

Recall the KV-cache byte cost from this campaign's cache lessons, re-derived here in one line: a 7B-class model with 4,096-wide hidden states and 32 layers costs 512 KiB per token, summed across every layer, in FP16. The obvious way to give a new request its own memory is to reserve, up front, enough space for the longest sequence that engine will ever allow — say a 2,048-token maximum context:

That reservation happens before a single token of output exists. Now suppose the actual average completion, in real traffic, runs about 300 tokens — a plausible mix of short chat replies and the occasional longer one:

That number should feel familiar in shape, if not in cause, to Chapter 0's 99.36% idle compute: a resource gets reserved for the theoretical maximum, and the typical case never comes close to using it.

What that costs in concurrent capacity

An A100 80GB running the 7B model has 14 GB spent on weights, leaving 66 GB for everything else. Under reserve-max allocation, at 1 GiB reserved per request:

Sixty-six is uncomfortably far from Chapter 0's 156-request compute crossover. The GPU has plenty of spare compute to serve 156 users at once — but reserve-max allocation runs out of memory long before it gets there, because it pays the 1 GiB tax on every request regardless of how many tokens that request actually ends up needing.

The fix: allocate in fixed-size blocks, like an operating system pages memory

PagedAttention, introduced by the vLLM paper (Kwon et al., 2023), borrows the exact trick operating systems use for virtual memory: never reserve one giant contiguous chunk. Instead, carve KV-cache memory into small fixed-size blocks — commonly 16 tokens each — and hand a request new blocks only as it actually generates tokens that need them.

Redo the 300-token request under this scheme. It needs enough blocks to cover 300 tokens:

The only waste left is internal fragmentation in the last, partially-filled block of each request — at most 15 tokens' worth (block_size − 1), no matter how long the sequence runs. Compare that to reserve-max, whose waste is proportional to how far short of the maximum a request falls, which for most real requests is nearly all of it.

What that buys back in concurrent capacity

This is the missing piece from Chapter 0 and Chapter 1. Continuous batching can only fill as many seats as memory allows it to reserve; PagedAttention is what lets the batch actually grow past 66 and toward — and past — the 156-request compute crossover, because it stops paying an 85% memory tax on every single seat.

The mechanism: block tables, exactly like page tables

Each request's KV cache is now scattered across whatever physical blocks happen to be free — not contiguous in memory at all. A small per-request block table maps logical block index (0, 1, 2, …, this request's own numbering) to the physical block ID actually holding that data, wherever it lives in the pool:

The attention kernel is rewritten to walk this indirection — for each query, gather the relevant keys and values from whichever physical blocks the table points to, rather than assuming they sit end-to-end in memory. That's a real engineering cost (a custom CUDA kernel, not a one-line change), which is exactly why PagedAttention was a paper and not a footnote: making attention correct and fast over non-contiguous, indirected memory took real kernel work, distinct from the memory-management idea itself.

The bonus: copy-on-write for shared prefixes

Block-level indirection buys a second win almost for free: if two sequences share an identical prefix — the classic case is parallel sampling, generating several candidate completions from the same prompt — their block tables can point to the same physical blocks for the shared portion, with a reference count tracking how many sequences currently point at each block. Only when a sequence's own generation diverges from the others (writes a token the others don't share) does it get a fresh, private copy of that one block — copy-on-write, the same trick Unix uses for fork().

Work a concrete case: 4 parallel samples from the same 1,000-token prompt, each generating 50 more tokens before stopping.

Compare that to naively duplicating the whole prompt for every branch:

Beam search gets the identical benefit for the identical reason: every beam shares its ancestor path's blocks until the beam-search step where it diverges, and only then pays for a private block.

python
class BlockAllocator:
    BLOCK_TOKENS = 16
    def __init__(self, total_blocks):
        self.free = list(range(total_blocks))
        self.refcount = {}

    def alloc(self):
        block_id = self.free.pop()
        self.refcount[block_id] = 1
        return block_id

    def share(self, block_id):        # copy-on-write: just bump the refcount, no copy yet
        self.refcount[block_id] += 1

    def write(self, block_id):        # about to diverge -- get a private copy if shared
        if self.refcount[block_id] > 1:
            self.refcount[block_id] -= 1
            new_id = self.alloc()
            # ... copy the 8 MiB of KV data from block_id to new_id ...
            return new_id
        return block_id                    # already private, write in place

Waste across the whole length distribution, not just one average

The 85.4% figure used one average length. Real traffic has a distribution, and reserve-max's waste percentage actually changes shape across it — worth tracing at several lengths against the same 2,048-token maximum:

Reserve-max's waste is worst exactly where real traffic lives — short, everyday requests — and only disappears for the rare request that happens to hit the configured maximum. Paged allocation's waste, by contrast, stays in the low single digits everywhere, because it's bounded by block size alone, not by how far a request falls short of some global ceiling.

The block-size dial: a real tradeoff, not a free parameter

Nothing says blocks must be 16 tokens. Larger blocks mean a shorter block table (less pointer-chasing for the attention kernel to walk per request) but more internal fragmentation in the worst case; smaller blocks mean tighter packing but more table entries and more indirection overhead per attention call. Compare block sizes 128 and 16 for that same 300-token request:

Sixteen is a real engineering compromise, not an arbitrary default — small enough that worst-case internal fragmentation (at most block_size − 1 tokens, 15 in this case) stays tiny relative to typical request lengths, while large enough that the block table and the attention kernel's gather step don't balloon into their own overhead for long sequences.

Fragmentation isn't the only cost this chapter's mechanism prevents

Reserve-max has a second failure mode worth naming: external fragmentation. Even when the total free memory across the pool would be enough for a new request, if that free memory is scattered in chunks smaller than the contiguous slab a new request needs, the request cannot be admitted — exactly the classic operating-systems memory-fragmentation problem. Fixed-size blocks sidestep this entirely: any free block can satisfy any request's next-block need, because every block is identically sized and a request's logical-to-physical mapping never requires contiguity in physical memory at all.

python
def gather_kv(block_table, physical_pool, logical_positions):
    """What the attention kernel does per query: walk the indirection."""
    keys, values = [], []
    for pos in logical_positions:
        logical_block, offset = divmod(pos, BLOCK_TOKENS)
        physical_block = block_table[logical_block]           # the indirection lookup
        k, v = physical_pool.read(physical_block, offset)  # gather from wherever it actually lives
        keys.append(k); values.append(v)
    return keys, values

What happens when even paged allocation runs out

Paging cuts waste dramatically, but the pool is still finite — 445 concurrent requests at this chapter's average length, not infinite. When a burst of new arrivals would push past that ceiling, the engine has to choose: reject the new arrivals outright (simple, but wastes an opportunity if the burst is brief), or preempt some already-running lower-priority request to make room, exactly the swap-or-recompute decision Chapter 4 works out in full. Paging doesn't remove the need for a preemption policy; it just raises the ceiling at which that policy has to kick in, from 66 requests to 445.

The watermark: never let the pool hit exactly zero

One more real detail: engines don't run the block pool down to literally zero free blocks before refusing new admissions. A small reserved watermark — say, holding back 2–5% of total blocks as an emergency buffer — guarantees that already-running requests always have at least one more block available for their very next token, even if a burst of admissions filled the pool right up to that line. Without a watermark, a pool that hits exactly zero free blocks has no way to let an in-progress request take even one more step, forcing an emergency preemption at the worst possible moment — mid-generation, for a request that was already almost finished. Reserving a small margin ahead of time is cheaper than discovering the pool is exactly full when a currently-running request needs its 20th block right now.

What the block-table overhead itself costs, quantified

This chapter has treated the block table as free bookkeeping. It isn't literally free — each entry is a small integer (a physical block ID), and the table itself lives somewhere in memory too. For a request holding 19 blocks (the 300-token example), at, say, 8 bytes per table entry:

Compare that to the 2 MiB of internal fragmentation the same request pays in KV memory itself. Block-table bookkeeping is utterly negligible next to the fragmentation it eliminates — which is exactly why PagedAttention's designers could freely choose a small block size (16 tokens, favoring low fragmentation) without worrying that a correspondingly larger table would eat back the savings. The tradeoff named earlier in this chapter (bigger blocks reduce table size but increase fragmentation) is real in principle, but at these byte counts, fragmentation dominates every practical decision by many orders of magnitude — block size gets chosen almost entirely on fragmentation and attention-kernel efficiency grounds, not table-size economy.

Beam search and parallel sampling, the arithmetic taken further

The 4-branch copy-on-write example used a modest branch count. Push it to a realistic beam-search width of 8, generating 40 tokens beyond a shared 500-token prompt, to see how the sharing advantage scales:

This is the same mechanism, and the same “pay the shared part once” principle, that Chapter 3 scales up to unrelated users instead of sibling beams of one request — PagedAttention's copy-on-write blocks are the substrate both chapters' savings are built on.

The name, traced to its source

“Paged” is a deliberate borrowing, not a coincidence of naming: operating-systems virtual memory solved this exact problem decades earlier, for the exact same underlying reason — a process doesn't know in advance exactly how much memory it will need, so reserving a maximum-sized contiguous block up front wastes whatever the process doesn't end up using, precisely PagedAttention's reserve-max failure mode. Fixed-size pages, an indirection table (a page table, here a block table), and on-demand allocation are the identical three ideas in both settings; the vLLM authors made that lineage explicit in the paper's own name.

Carry this forward

The number worth keeping from this chapter is the 6.7× concurrent-capacity gain — not because that exact multiplier holds everywhere, but because it demonstrates that memory management is not a secondary concern behind scheduling. A perfectly-scheduled continuous batcher (Chapter 1) sitting on top of reserve-max allocation still caps out at 66 concurrent requests on this GPU, nowhere near Chapter 0's 156-request compute crossover. Paging is what makes that crossover reachable in practice, not just in theory.

Chapter 3: Prefix Caching & RadixAttention

Chapter 2's copy-on-write example shared a prompt across four sibling branches of the same request. Now stretch that idea across unrelated requests: a customer-support bot where every single user's prompt begins with the same 1,000-token system message — instructions, tool schemas, a few worked examples — before the user's own question. A hundred different users, a hundred identical opening thousand tokens.

The redundant work, priced out

Recall from Chapter 0 that prefill — processing prompt tokens for the first time — costs about 2 FLOPs per parameter per token, the same rate as decode, but applied to every prompt token at once rather than one token per step. For this 7B model, that's still 1.4 × 10¹⁰ FLOPs per token, and at the A100's 312 TFLOPS compute-bound ceiling (the regime prefill actually runs in, since a whole prompt arrives as one dense batch of tokens):

If every one of 100 users' requests independently reprocesses the shared 1,000-token system prompt from scratch:

Cache the KV blocks for that 1,000-token prefix once, and every subsequent user with the identical opening tokens can skip straight to their own unique continuation:

And per-user, this isn't just a throughput win — it directly cuts TTFT (time to first token, the latency a user actually feels before anything starts streaming back). Every user after the first skips 44.9 ms of prefill compute they would otherwise have had to sit through before seeing a single word.

The mechanism: this is PagedAttention's block-sharing, one level up

Chapter 2 built copy-on-write, reference-counted physical blocks so sibling beams of one request could share a prompt. Prefix caching is the exact same block infrastructure applied across requests: when a new request arrives, check whether its opening tokens exactly match a prefix that's already sitting in some other (or previous) request's cached blocks. If so, point this request's block table at those existing blocks — reference count goes up, zero new computation happens for that portion — and only run prefill on whatever tokens come after the matching prefix.

python
def prefill_with_cache(prompt_tokens, block_pool):
    matched_blocks, matched_len = block_pool.longest_cached_prefix(prompt_tokens)
    for b in matched_blocks:
        block_pool.share(b)                      # reuse: bump refcount, no compute
    remaining = prompt_tokens[matched_len:]           # only the NEW suffix needs real prefill
    if remaining:
        new_blocks = model.prefill(remaining)       # actual GPU work, just for this part
        block_pool.append(new_blocks)
    return matched_blocks + new_blocks

RadixAttention: the general case, not just one fixed system prompt

The example above assumes one, unchanging system prompt shared by everybody. Real traffic is messier: multi-turn conversations where turn 3 shares a prefix with turn 2 but not with a different user's conversation; few-shot prompts that share some examples but not others; self-consistency sampling where several reasoning branches share a chunk of chain-of-thought before diverging. RadixAttention, introduced with SGLang (Zheng et al., 2023), generalizes prefix caching to any overlapping prefix among any requests, by organizing all cached KV blocks into a radix tree — a tree where each edge is labeled with a run of tokens, shared paths from the root represent shared prefixes, and branch points mark exactly where two sequences first diverge.

When a new request arrives, the scheduler walks the tree from the root, matching as many tokens as it can against existing edges — this is the "longest cached prefix" lookup, now generalized to work against every request that has ever passed through, not one hardcoded system prompt. The moment the new request's tokens diverge from every existing path, a new branch is created there, and only the tokens past that divergence point get real prefill compute.

A worked tree-sharing example

Eight self-consistency samples share a 2,000-token reasoning prefix (the same question, the same worked setup) before branching into eight different continuations of the final reasoning steps.

That "savings equals the number of things sharing the prefix" pattern is not a coincidence — it falls directly out of the arithmetic: N users each skip redoing the same fixed amount of work, so the total work drops from N× that amount to 1× it, a factor-of-N reduction, every time.

Eviction: what happens when the tree gets too big

The radix tree cannot grow forever — it competes for the exact same physical block pool that active requests' own decode needs. When memory pressure hits, RadixAttention evicts using an LRU (least-recently-used) policy at the leaf level: prune the tree's least-recently-touched leaf nodes first, working inward, since a leaf by definition has no other cached sequence still depending on it. Internal, heavily-shared nodes (like the 1,000-token system prompt every request touches) survive eviction far longer than they naturally would under plain LRU on individual blocks, precisely because their reference count keeps getting refreshed by every new request that walks through them.

Multi-turn conversations: savings that compound turn over turn

System prompts are the easy case — one fixed prefix, shared by everyone. Multi-turn chat is subtler: each new turn's prefix is the entire conversation so far, which is unique to that user, but which the same user's own next turn shares almost completely with the turn before it. Trace one conversation across four turns, each turn adding roughly 200 tokens of new exchange on top of everything before it:

This is why prefix caching matters even for workloads with zero cross-user sharing at all — a single user's own multi-turn session is, by itself, a chain of near-total prefix overlap, turn after turn.

The tension this chapter creates with Chapter 2's memory budget

Cached prefixes and active requests' own growing KV caches compete for the exact same physical block pool. Grow the radix tree too aggressively and you eat directly into Chapter 2's concurrent-capacity headroom. Revisit that chapter's 66 GiB pool, now with 20 GiB reserved for cached prefixes:

That's not necessarily a bad trade — if the 20 GiB of cached prefixes is saving enough redundant prefill work across many requests, it can easily be worth the reduced concurrent-decode headroom. But it is a genuine budget line, not a free feature, which is exactly why RadixAttention's LRU eviction at the leaf level exists: it keeps the cache from growing unboundedly and silently starving the pool that active decode depends on.

A simplified radix-tree lookup, in code

python
class RadixNode:
    def __init__(self):
        self.children = {}      # token -> child RadixNode
        self.block_ids = []     # physical KV blocks for this edge's tokens
        self.refcount = 0
        self.last_used = 0

def match_and_extend(root, tokens, now):
    node, i = root, 0
    while i < len(tokens) and tokens[i] in node.children:
        node = node.children[tokens[i]]
        node.refcount += 1; node.last_used = now   # cache hit — refresh LRU clock
        i += 1
    return node, tokens[i:]          # node = deepest match; remainder needs real prefill

Hit rate as a dashboard number, and why it plateaus

Real operators track prefix cache hit rate — the fraction of incoming prompt tokens that matched something already cached — as a first-class metric, because it feeds directly into the expected TTFT savings this chapter has been deriving. If hit rate is h (a fraction between 0 and 1) and a full cold prefill would take T:

At h = 60% and T = 178 ms (a 4,000-token prompt at the compute-bound rate), expected prefill time is 0.4 × 178 ≈ 71 ms — the exact parameter Chapter 8's worked trace plugs in. Hit rate doesn't climb linearly with cache size, though: a real prompt population has a small set of very common prefixes (shared system prompts, popular few-shot templates) and a long tail of nearly-unique ones, so the first few hundred megabytes of cache typically capture most of the achievable hit rate, and doubling cache size beyond that buys progressively smaller gains — a diminishing-returns curve, not a straight line, which is worth knowing before over-provisioning cache memory expecting linear payoff.

A subtlety: prefix matching happens at token boundaries, not text boundaries

Prefix matching compares token IDs, not raw characters — and tokenizers don't always split text the same way depending on what comes immediately after it. Two prompts that look identical as text for their first 999 characters can tokenize into different token sequences if the 1,000th character changes which sub-word boundary the tokenizer chooses near that point, especially across languages or when a shared prefix is immediately followed by very different continuations. The practical consequence: a prefix cache keyed on token IDs will very occasionally miss a match that looks obviously identical to a human reading the raw text, purely because of where the tokenizer happened to draw a boundary. This is rare enough not to be a major engineering concern, but worth knowing as a source of a mysterious “why didn't this cache hit” investigation the first time it surfaces in production logs.

Cache warmup: the cost this chapter's numbers hid

Every savings figure in this chapter compared “cached” against “uncached,” but the first request to touch any given prefix always pays the full, uncached cost — someone has to compute it before anyone can reuse it. Across a whole day's traffic hitting the same 1,000-token system prompt:

At N = 100 (this chapter's running example), average amortized cost per request for the shared portion is 0.449 ms — already negligible. At N = 2 (only two requests ever share this exact prefix), it's 22.45 ms — still a real win, just a much smaller one. This is the same “savings scale with N” pattern from earlier in this chapter, restated as an amortization curve: a prefix shared by thousands of requests over a day effectively costs nothing after the first hit; a prefix that only two requests ever share barely breaks even against the bookkeeping overhead of maintaining it in the tree at all, which is precisely why the LRU eviction policy exists — rarely-reused branches of the tree are exactly the ones it should prune first when memory is tight.

Self-consistency sampling, worked at realistic scale

Chapter 3 opened with an 8-branch self-consistency example. Push it to a scale closer to what production reasoning workloads actually run — 32 sampled reasoning paths sharing a 3,000-token problem statement and worked setup, each branch generating 150 further tokens of its own reasoning chain:

At this scale, prefix sharing isn't a minor efficiency tweak for reasoning-heavy workloads — it's the difference between a self-consistency query being computationally reasonable at all and being 32× more expensive than the actual novel reasoning work it's trying to buy. This is a direct preview of why Chapter 7 rates SGLang's tree-aware scheduling especially highly for agentic and reasoning-heavy workloads: this exact pattern — many branches, one shared root — is precisely what those workloads generate constantly.

What eviction actually removes, precisely

When memory pressure forces the radix tree to shrink, it's worth being precise about what gets reclaimed. Evicting a leaf node returns its physical blocks to the free pool — the same pool Chapter 2's allocator draws new blocks from for active decode — and decrements the reference count on every block it shared with its parent path toward the root. A shared internal node (like the 1,000-token system-prompt root this chapter opened with) is only actually reclaimed once every branch that ever touched it has been evicted, which is exactly why heavily-shared prefixes survive memory pressure far longer than lightly-shared ones: their reference count keeps getting refreshed by new requests walking through them, even as their individual sibling branches come and go around them.

Where this chapter's savings show up on the bill

Every saving this chapter derived is FLOPs and time, not dollars directly — but the translation is immediate, since GPU-time is what's actually billed. A 100× reduction in redundant prefill compute for a shared system prompt is, at a fixed GPU-hour rate, a roughly 100× reduction in the portion of the bill attributable to that redundant work specifically — not the whole request's cost, since the unique continuation still has to be paid for either way, but a real, measurable line item. This is the exact thread this campaign's dedicated cost-modeling lesson picks up and runs much further, translating every technique in this lesson into an actual dollars-per-request figure.

Carry this forward

The pattern to keep from this chapter: any time N things share a fixed-cost prefix, caching that prefix turns N× the work into roughly 1× the work, whether N is a hundred unrelated users sharing a system prompt, four sibling beams sharing a decoding path, or thirty-two reasoning branches sharing a problem statement. That single idea, applied consistently through a radix tree instead of one hardcoded special case, is what RadixAttention actually is.

One more number worth checking by hand

Chapter 4: Scheduling: Prefill vs. Decode

Chapter 1 flagged this and deferred it: prefill and decode are not the same kind of work, and dropping a big prefill into the middle of a smooth continuous-batching decode loop can stall every other request in the batch. This chapter is about exactly how badly, and the fix.

Two different regimes, revisited

Decode, one token at a time, is memory-bound — Chapter 0's whole opening argument. Prefill, processing an entire prompt's worth of tokens in one dense matrix multiply, is compute-bound — it has high arithmetic intensity because many tokens share the same weight-read, exactly like a large decode batch does. Both regimes exist inside the same engine, competing for the same GPU, often in the same continuous-batching step.

The worked stall: one big prompt, unchunked

Suppose an 8,000-token document arrives for summarization, and the naive scheduler just runs its entire prefill in one shot before returning to the decode loop that was already serving other users:

For 359 milliseconds, the GPU is entirely consumed by this one prompt's prefill. Every other request currently decoding gets zero new tokens in that window — their inter-token latency (ITL), the gap between consecutive tokens streaming back to a user, just spiked by 359 ms. For context, a comfortable reading pace is somewhere around 20–30 tokens per second, meaning users expect a new token roughly every 33–50 milliseconds; a 359 ms stall is a visible, jarring stutter, not a rounding error.

Chunked prefill: bound the damage per round

The fix is to never let one prefill occupy the GPU for longer than the ITL budget allows. Split the 8,000-token prompt into fixed-size chunks, and interleave one chunk of prefill with the ongoing decode batch's regular steps, round after round, instead of running the whole prompt at once.

Derive the right chunk size directly from the ITL budget, rather than picking a number arbitrarily. Set the budget at 50 ms (a comfortably safe margin under the 33–50 ms reading-pace window):

Round down to a clean number and this is exactly why production chunk sizes cluster around 512–2,048 tokens — it isn't an arbitrary default, it falls straight out of dividing an ITL budget by a compute-bound throughput ceiling. Redo the 8,000-token example with a 512-token chunk:

Nobody else's stream stalls for more than 23 ms at a time, instead of 359 ms once. The total prefill work is identical — still 8,000 tokens' worth of compute — it's just spread across 16 rounds instead of dumped into one.

The tradeoff chunk size doesn't erase: whose TTFT pays for it

Smaller chunks protect everyone else's ITL, but they cost the big prompt's own TTFT, because its prefill now has to wait its turn between other requests' decode rounds instead of running straight through. Model a round-robin scheduler that gives each other active request roughly 20 ms of decode time between prefill chunks (a simplifying assumption, but a realistic order of magnitude):

Read the two rows side by side. Bigger chunks (2,048) finish the big prompt's own prefill in fewer, longer rounds — better TTFT for that request, at the cost of a 91.9 ms stutter imposed on every other stream sharing the GPU during each of those rounds, blowing well past the 50 ms budget. Smaller chunks (256) keep everyone else's ITL comfortably smooth, at the cost of more than doubling the big prompt's own TTFT, because it now has to wait through 32 rounds of interleaving instead of 4. There is no chunk size that wins both; the right choice depends on whether your traffic is dominated by many concurrent interactive chats (favor small chunks, protect ITL) or by fewer, latency-tolerant big-document jobs (favor larger chunks, protect their own TTFT).

Priority and preemption

Chunked prefill handles the routine case; production schedulers also need a policy for genuine priority conflicts — an interactive chat request arriving while a long batch-summarization job is mid-decode. Preempting the batch job frees a slot immediately, but its partially-generated KV cache needs somewhere to go. Two options, and the right one depends purely on which is cheaper to redo:

Swap the preempted request's KV cache out to CPU RAM over PCIe, then swap it back in when a slot frees up again. At a typical PCIe 4 ×16 bandwidth of roughly 25 GB/s, and reusing the 512 KiB/token figure from Chapter 2 for a request that has generated, say, 300 tokens so far (150 MiB of KV):

Recompute instead: just discard the KV cache and redo the prefill for those 300 tokens once the request is readmitted, at the compute-bound prefill rate:

For this request, swapping (6 ms) beats recomputing (13.5 ms) — but that comparison flips for a request with very little generated so far (recomputing a handful of tokens is nearly free, while swapping still pays a fixed PCIe round trip) or when PCIe bandwidth is shared and congested by other traffic. Real schedulers (vLLM's included) make this choice per-request, dynamically, rather than hardcoding one strategy.

When several big prompts arrive together

The single-prompt trace understates real risk: production traffic can see several large prompts land in the same short window — several users pasting long documents within seconds of each other. If three 8,000-token prompts queue up, each needing 16 chunks at 512 tokens, and the scheduler round-robins one chunk from each per round before returning to ordinary decode:

No single round exceeds the 50 ms budget, so nobody sees one big stutter — but the cumulative drag on ITL across that window is real, and it's exactly the kind of load a capacity-planning exercise (queue depth alarms, admission control that caps how many large-prompt prefills can be in flight at once) needs to account for. Chunking bounds the worst single stall; it doesn't make concurrent big prompts free.

Deriving the swap-vs-recompute crossover precisely

Chapter 4's preemption example claimed swapping (6 ms for 150 MiB) beat recomputing (13.5 ms for 300 tokens) for that specific request, then gestured at “this flips for short requests.” Make that precise. Swapping pays a small fixed round-trip cost — queueing plus DMA setup, call it 0.5 ms — on top of the per-byte transfer time; recomputing has no comparable fixed cost, just per-token compute:

Set them equal to find the crossover token count:

Below about 21 generated tokens, recomputing from scratch is actually cheaper than paying the fixed overhead of a swap round-trip; above it, swapping wins and the gap only widens, since its per-token slope (0.021 ms) stays below recompute's (0.0449 ms) for every additional token. A scheduler that always swaps, never recomputes, is leaving a small but real optimization on the table for very freshly-started requests.

Priority classes, worked

Suppose the scheduler splits traffic into two priority tiers — interactive (weight 3) and background batch (weight 1) — and shares GPU rounds proportionally to weight rather than strictly always preempting one for the other:

At the aggregate 22,286 tok/s compute-bound ceiling (or whatever the current batch's throughput is), interactive traffic gets roughly 16,715 tok/s and background gets roughly 5,572 tok/s of the shared budget — neither starves completely, unlike strict priority preemption where background traffic could be indefinitely delayed by a continuous stream of interactive requests. Weighted fairness is a real, tunable middle ground between “always prioritize interactive” (best interactive latency, background starves under load) and “pure FIFO” (fair, but interactive users feel every background job's queue depth).

Why decode itself is never chunked

It's worth being precise about what chunking applies to. A decode step, by construction, produces exactly one new token per active sequence — there's no smaller unit of work to split it into, unlike a prefill pass which processes many prompt tokens in one shot and can be divided arbitrarily. This is why every chunking technique in this chapter is described as chunked prefill, never “chunked decode”: decode is already the smallest possible unit of GPU work per sequence per step, and the scheduling problem this chapter solves is entirely about how much prefill work to interleave alongside that already-minimal decode step, not about subdividing decode further.

A fairness lens on the chunk-scheduling policy

Round-robin chunk scheduling (one chunk per waiting prefill, in turn, between decode rounds) isn't the only policy, and it isn't automatically the fairest one under all conditions. If three prompts of very different lengths — say 1,000, 4,000, and 16,000 tokens — are all chunked at 512 tokens and served strictly round-robin, the 1,000-token prompt finishes in 2 rounds while the 16,000-token prompt needs 32, so early rounds spend disproportionate scheduling attention on requests that are almost done relative to the huge one that has barely started. A shortest-remaining-chunks-first policy would instead let the short prompt finish and free its scheduling slot faster, at the cost of the long prompt waiting comparatively longer before its own chunks resume — yet another instance of this chapter's central tension, chunk scheduling policy now trading against itself, not just chunk size.

Recomputing the ideal chunk size on faster hardware

Chapter 0's sensitivity check found an H100's compute-bound prefill ceiling at roughly 70,574 tok/s (aggregate, at its own crossover), against the A100's 22,286 tok/s. Redo this chapter's chunk-size derivation — max chunk size = ITL budget × throughput — on that faster chip, holding the same 50 ms budget:

Faster hardware doesn't just finish work sooner — it changes what “safe” chunk size even means, since the same ITL budget now buys room for a proportionally bigger chunk before it's consumed. A chunk size tuned and shipped for A100 hardware is needlessly conservative if silently redeployed on H100 without revisiting this derivation — leaving throughput on the table by chunking far more finely than the faster chip's own budget actually requires.

Connecting chunk size back to the prefix cache

Chapters 3 and 4 interact in a way worth making explicit: a high prefix cache hit rate directly shrinks how much of a prompt actually needs chunked prefill in the first place. Revisit the 8,000-token example at a 75% cache hit rate:

A well-populated prefix cache doesn't just save compute (Chapter 3's headline claim) — it directly shrinks the scheduling problem this chapter is solving, since chunking only has to manage whatever fraction of the prompt is genuinely new. This is one of several places in this lesson where two chapters' techniques compound rather than simply add: a good cache hit rate makes the chunking tradeoff strictly easier to manage, not just cheaper in isolation.

What a real ITL budget looks like across user contexts

This chapter fixed the ITL budget at 50 ms, tied to a general human reading-pace estimate. Real products tighten or loosen that budget by context. A voice assistant reading a response aloud has to keep pace with natural speech — roughly 150 words per minute, or about 2.5 words/second, translating to a budget nearer 150–200 ms per word-equivalent (looser than reading text, since spoken cadence is slower than silent reading). A code-completion tool showing an inline suggestion as someone types wants near-instantaneous feedback, well under 50 ms, since any perceptible lag breaks the illusion of the suggestion keeping up with typing. Recompute this chapter's max-chunk-size formula for the tighter, 20 ms budget a code-completion product might target:

The formula doesn't change; the input does. Knowing your own product's actual latency tolerance — not borrowing a generic 50 ms assumption — is the difference between a chunk size that's needlessly conservative (leaving throughput on the table) and one that's genuinely tuned to what users actually need.

The decode slice assumption, made explicit

This chapter's TTFT calculations assumed a fixed 20 ms “decode slice” between prefill chunks — time the scheduler gives to everyone else's ongoing decode before returning to the big prompt's next chunk. That number isn't arbitrary; it's roughly how long one continuous-batching decode step takes at a moderate batch size (Chapter 0's 7 ms step time, times a small number of interleaved rounds to give other requests a fair share before yielding back). Shrink that decode slice and the big prompt's own prefill finishes faster (fewer milliseconds lost waiting between its own chunks), at the direct cost of the very users that slice was protecting — another instance of this chapter's central tradeoff, now showing up in a parameter this chapter had, until now, held fixed rather than treated as its own dial.

The name worth remembering

If this lesson leaves you with one term from this chapter to search for later, make it chunked prefill — it's the specific name under which every major engine's documentation exposes this tradeoff as a configurable parameter, and it's the search term that will turn up the exact scheduling knob this chapter has been deriving from first principles the whole way through, whichever engine you end up running.

Carry this forward

Keep the formula, not the specific numbers: max safe chunk size equals your ITL budget times your engine's compute-bound throughput ceiling. Everything else in this chapter — the 447.6 ms versus 1,008 ms TTFT comparison, the swap-versus-recompute crossover, the priority-weighting example — is that one formula worked out under different assumptions, and every one of them will need to be re-derived, not looked up, the day your hardware, model, or latency target changes.

One more number worth checking by hand

(A 500-token prompt at the same chunk size needs only 1 round — small requests never feel this chapter's tradeoff at all; it only bites once a prompt spans multiple chunks. Keep that distinction in mind before over-engineering chunk-size tuning for a workload whose typical prompt is already shorter than one chunk. Measure your own prompt-length distribution first, then decide whether this chapter's tradeoff even applies to you at the scale you're actually running.)

(And once you've measured it, revisit this chapter's derivation with your own numbers plugged in, not the 8,000-token example used throughout — the formula is the reusable part.)

Chapter 5: TensorRT-LLM: Compiling the Graph

Everything so far — continuous batching, PagedAttention, prefix caching, chunked prefill — is a scheduling and memory-management story, and it applies whether the model runs in ordinary eager PyTorch or not. TensorRT-LLM (NVIDIA's serving engine) attacks a different layer entirely: what if, instead of running the model's ordinary PyTorch code step by step, you compiled the entire computation graph ahead of time into a single, hyper-optimized executable?

The cost eager execution pays, one kernel at a time

Ordinary PyTorch execution is eager: every operation — a matrix multiply, a bias add, a layer-norm, an activation function — is dispatched to the GPU as its own separate CUDA kernel launch, one Python-to-CUDA round trip per operation. Each launch carries real, fixed overhead on the CPU side — setting up the call, checking arguments, queuing it on the GPU — typically on the order of 5–10 microseconds, even though the GPU-side work itself might take far less time for a small operation.

Estimate how many kernel launches one forward pass through this 32-layer model actually costs. A single transformer layer, run eagerly, might issue on the order of 200 separate kernel launches: the Q, K, V projections; the attention score computation; the softmax; the attention-value multiply; the output projection; two layer-norms; the two feed-forward matrix multiplies; the activation function; and assorted reshapes, type-casts, and residual adds along the way.

Compare that against Chapter 0's actual compute time for one token: at batch size 1, real GPU compute takes microseconds, dwarfed entirely by 48 ms of overhead just launching the kernels that will run it. For small batches especially, dispatch overhead can rival or exceed the useful work being dispatched.

Graph compilation: fuse first, launch once

TensorRT-LLM compiles the model's computation graph ahead of time and applies kernel fusion: merging several consecutive operations that would each be a separate eager kernel into one combined kernel that does the same math in a single launch. A bias-add immediately followed by an activation function, for instance, becomes one fused kernel instead of two; several small matrix multiplies with compatible shapes may be fused into one larger one.

Suppose fusion collapses each layer's roughly 200 eager kernels down to about 20 fused ones — a 10× reduction, a plausible order of magnitude for a well-fused transformer layer:

This overhead reduction is on top of, and independent from, everything Chapters 0–4 already covered — it doesn't change the memory-bandwidth ceiling or the compute-bound ceiling, but it shrinks the fixed tax paid on the way to reaching either one, which matters most exactly where it hurt most: small batches and latency-sensitive single-token decode steps.

FP8: halving the bytes AND doubling the FLOPs, on Hopper

NVIDIA's Hopper generation (H100) adds native hardware support for FP8, an 8-bit floating-point format — half the bytes of FP16 for weights and activations, with tensor cores that run FP8 matrix multiplies at roughly double the throughput of BF16/FP16 on the same chip. An H100 SXM is rated at roughly 989 TFLOPS dense BF16 and roughly 1,979 TFLOPS dense FP8 — essentially exactly 2×.

Both of Chapter 0's ceilings move at once under FP8. The memory-bound floor: halving weight bytes (14 GB → 7 GB for this model) halves the time to read them, so the batch-size-1 decode ceiling roughly doubles. The compute-bound ceiling: doubled FLOPs throughput roughly doubles the aggregate throughput at the crossover batch too. Halving bytes and doubling FLOPs moves both walls together — a rare case where a single numeric-format change helps both regimes Chapter 0 identified, not just one.

When compile-time beats flexibility — and when it doesn't

Compiling an engine isn't instantaneous. Building a TensorRT-LLM engine for a given model, precision, and maximum batch/sequence-length configuration commonly takes several minutes — say, 8 minutes, for a concrete number to reason with. Compare that to vLLM's cold start, which is mostly just loading weights off disk, on the order of 30 seconds for a model this size.

If a team redeploys 50 times a day — realistic during active model or prompt-template iteration — the compile tax adds up fast:

That 6.7 GPU-hours/day is a real cost, and it only pays for itself if the resulting throughput gain (roughly 2× under FP8, from this chapter's arithmetic) saves more GPU-time in steady-state serving than the rebuilds cost. For a busy, stable production endpoint serving many GPU-hours of traffic a day, halving that serving cost dwarfs a one-time 8-minute build. For a research team redeploying dozens of times daily while iterating on the model itself, the build tax can exceed the entire benefit before the engine ever serves a production request.

Across three GPU generations and precisions, the ceilings side by side

Pull together every hardware and precision variant this lesson has touched, all measured against the same 7B model, so the pattern is visible at a glance rather than scattered across chapters:

FP8 on H100 versus FP16 on A100 is roughly a 3.3× single-user speedup and a 6.3× aggregate-throughput speedup — from one hardware generation plus one numeric-format change, with none of Chapters 1 through 4's scheduling machinery touched at all. This is why serving teams treat hardware and precision choices as a first-class lever alongside scheduling, not an afterthought: the ceiling itself moved, not just how close you can get to it.

Dynamic shapes: the other cost of compiling ahead of time

Compilation needs to know, roughly, what shapes it's optimizing for — a range of expected batch sizes and sequence lengths, often called a shape profile (a minimum, an optimal target, and a maximum). Traffic that stays inside the profiled range runs at full compiled speed. A request that falls outside it — a batch size far larger than anticipated, or a sequence length beyond the configured maximum — either falls back to a slower unoptimized path or gets rejected outright, depending on configuration. This is the same compile-time-versus-flexibility tension from earlier in this chapter, just surfacing at request time instead of deploy time: a shape profile is itself a bet about what traffic will look like, and a wrong bet costs either a rejected request or a silent performance cliff, neither of which an eager, uncompiled engine experiences, because it never committed to a shape range in the first place.

FP8 isn't free: the calibration step this chapter glossed over

Halving numeric precision from FP16 to FP8 shrinks the representable range and precision of every number in the model, and naïve conversion measurably degrades output quality on some layers more than others. Production FP8 deployment runs a calibration pass first — feeding representative data through the model to determine, per tensor, the right scaling factors that keep FP8's narrow range from clipping or under-using its available precision. Skipping this step and just casting weights to FP8 directly is a common first mistake; the throughput numbers this chapter derived assume calibration was done correctly, not that FP8 is a costless drop-in replacement for FP16.

The build-time break-even, as a formula

Generalize this chapter's 50-redeploys-a-day example into a reusable rule. Let B be build time, R be redeploys per day, H be daily GPU-hours of steady-state serving, and s be the fractional throughput speedup (0.5 for a 2× win, i.e. half the GPU-hours needed for the same volume):

Plug in this chapter's numbers — B = 8/60 hours, R = 50, s = 0.5, and ask what daily serving volume H makes the trade worthwhile:

Any production endpoint serving more than about 13.3 GPU-hours a day of steady traffic — a genuinely low bar, well under one GPU running continuously — comes out ahead compiling, even at an aggressive 50 redeploys daily. Drop the redeploy count to something more typical for a stable production service, one or two a day, and the break-even volume drops to a fraction of a GPU-hour: compiling is essentially always worth it once a shape has stabilized, which is exactly the qualitative rule stated earlier, now with the arithmetic that makes it checkable against your own traffic.

Fusion has to know about paging, or it isn't usable

One engineering wrinkle worth naming: a generic fused attention kernel, optimized purely for raw throughput on contiguous memory, is not automatically compatible with Chapter 2's block-table indirection. Compiling the fastest possible attention kernel and then discovering it assumes keys and values sit end-to-end in memory would silently break every gain PagedAttention built. Production compiled engines write their fused attention kernels to gather through the block table as part of the fused operation itself — the indirection lookup from Chapter 2's gather_kv pseudocode has to be baked into the compiled kernel, not bolted on afterward. This is a real reason building a competitive compiled engine takes serious kernel engineering, not just a generic graph compiler pointed at an arbitrary model.

Other quantization formats worth knowing by name

FP8 isn't the only reduced-precision option, and it's worth placing alongside its neighbors. INT8 weight quantization (integers instead of floating point, 1 byte per weight) gives similar memory savings to FP8 but needs a different calibration approach, since integers have no exponent to absorb outlier magnitudes. INT4 (4 bits per weight, 3.5× smaller than FP16) pushes memory savings further still, at a correspondingly larger accuracy risk, and is more commonly applied to weights alone while keeping activations and the KV cache at higher precision (a mixed-precision approach) rather than quantizing everything uniformly. Each format moves Chapter 0's memory-bound ceiling by roughly the same bytes-per-weight ratio as FP8 did in this chapter's table — the arithmetic pattern is identical, only the byte count and the accuracy tradeoff change.

Compiling across multiple GPUs: tensor parallelism adds another dimension

Everything in this chapter assumed one model fits on one GPU. Larger models (recall Chapter 0's 70B sensitivity check, which didn't fit on a single 80 GB card) need tensor parallelism — splitting each layer's weight matrices across several GPUs, so each holds a shard and they communicate partial results over NVLink during every forward pass. Compiling an engine for a tensor-parallel configuration means the ahead-of-time graph has to account for exactly how many GPUs are splitting the model and how they're connected — a 2-way split compiles a different engine than a 4-way split, even for the identical model and precision. This multiplies the shape-profile problem from earlier in this chapter: the compile-time investment now has to be repeated per parallelism configuration, not just per batch/sequence-length range, which is one more reason TensorRT-LLM's build cost pays off best on a truly stable, unchanging deployment topology rather than one still experimenting with how many GPUs to split across.

CUDA graphs: a lighter-weight middle ground

Between fully eager execution and a fully compiled TensorRT-LLM engine sits a lighter-weight technique worth naming: CUDA graphs, which capture a fixed sequence of kernel launches once and replay that whole captured sequence with a single dispatch call, without needing a full ahead-of-time compilation pipeline. It doesn't fuse operations into fewer, larger kernels the way TensorRT-LLM does — the same 200-ish kernels per layer still run — but it collapses the dispatch overhead of launching them individually into one graph-replay call, capturing much of this chapter's dispatch-overhead win with a much smaller engineering and build-time investment than full graph compilation. vLLM and SGLang both use CUDA graphs internally for exactly this reason: fixed, common shapes (a specific batch size range, ordinary decode steps) get graph-captured for a meaningful speedup, while the engine as a whole stays flexible enough to fall back to eager execution for shapes that fall outside what was captured — a genuine middle point on the compile-time-versus-flexibility spectrum this chapter has been mapping, not purely one extreme or the other.

The build-time break-even, revisited at a smaller shop's scale

Chapter 5's break-even formula (R × B < H × s) assumed a busy endpoint. Check it at a much smaller scale — an internal tool serving one team, with only 2 GPU-hours/day of real steady-state serving, and a much calmer 2 redeploys/week (roughly 0.29/day) as the model gets periodically refreshed:

0.039 < 1.0 — compiling still wins, even at this much smaller scale, because the rebuild cost itself is so small relative to almost any amount of steady serving. The break-even genuinely only favors staying eager when redeploy frequency is high (active daily iteration) and serving volume is low — exactly the combination that describes early-stage prototyping, and almost nothing else. This is worth internalizing as the chapter's real takeaway: the compile-versus-eager tradeoff isn't as close a call as it might sound in the abstract; for the overwhelming majority of production traffic levels, it tips decisively toward compiling once a shape has stabilized at all.

Where the 7.5-microsecond dispatch estimate comes from

This chapter's dispatch-overhead arithmetic leaned on a single estimate — 7.5 microseconds per kernel launch — worth grounding briefly rather than treating as an unexplained constant. That figure covers the CPU-side work of a CUDA kernel launch: the driver validating arguments, computing grid and block dimensions, and enqueueing the work onto the GPU's command stream, all before the GPU itself starts executing anything. It sits in the middle of a real range (roughly 5–10 microseconds depending on driver version, launch complexity, and whether CUDA graphs or launch-overhead-reducing APIs are in play) — which is exactly why this chapter treated it as a plausible order-of-magnitude estimate rather than a precise spec-sheet number, the same honest framing Chapter 0 used for its own FLOPs-per-parameter estimate. The conclusion this chapter draws from it — that fixed per-kernel overhead matters most at small batch sizes — holds across the entire plausible range; only the exact millisecond figures would shift with a different assumed constant.

What this chapter did NOT claim

Worth stating explicitly, since it's easy to over-read: this chapter never claimed TensorRT-LLM is “better” than vLLM or SGLang in some absolute sense — it claimed compilation trades flexibility for throughput, at a real, quantifiable build-time cost, and that trade pays off once a deployment is stable and busy enough. A workload that never stabilizes never crosses that break-even point, no matter how busy it gets, because R (redeploys) stays high indefinitely. Chapter 7 turns this from a standalone verdict into one input among several for an actual decision.

Carry this forward

The two ideas worth keeping separate in your head: kernel fusion attacks fixed per-operation dispatch overhead (helps most at small batch), and FP8 attacks the byte and FLOP cost of the operations themselves (helps at every batch size, since it moves both of Chapter 0's ceilings at once). They're often adopted together in one compiled engine, but they're solving different problems, and a deployment could in principle adopt one without the other — fusing kernels in an eager runtime via CUDA graphs, or running FP8 weights without a full ahead-of-time compile.

One more number worth checking by hand

(The same logic applies to any fixed per-request or per-operation overhead in a system, not just kernel launches — it is always the small, frequent unit of work that a fixed cost hurts most, whether that's a CUDA kernel dispatch, an HTTP request's connection setup, or a database query's parse-and-plan step. Look for fixed costs first whenever a small-scale workload seems disproportionately slow.)

Chapter 6: Disaggregated Serving

Chapter 4 patched over prefill-decode interference with chunking — both phases still share the same GPU, taking turns. There's a more radical fix: don't share the GPU at all. Run prefill and decode on physically separate fleets of GPUs, tuned independently for their own regime, and transfer the resulting KV cache between them once prefill finishes.

Why splitting the fleets is tempting

Chapter 0 and Chapter 4 already established that prefill wants compute-bound conditions (big batches of dense prompt tokens, high arithmetic intensity) and decode wants memory-bound conditions kept as clean as possible (small, latency-sensitive steps, no interruption). Putting both on one GPU means every scheduling decision is a compromise between two regimes that want opposite things from the hardware. A prefill fleet can be provisioned and scaled purely around compute-bound throughput; a decode fleet can be provisioned and scaled purely around keeping ITL tight for however many concurrent streams it holds. Neither fleet has to compromise for the other's sake, ever.

The cost this buys: transferring the KV cache

The catch is that once prefill finishes on one GPU, the resulting KV cache — potentially gigabytes of it — has to physically move to whichever GPU in the decode fleet will continue that request. This transfer has to be fast enough that it doesn't eat the latency win disaggregation was supposed to buy.

Take an 8,000-token prompt (the same size Chapter 4 chunked), at 512 KiB/token from Chapter 2:

Within a node, GPUs connect over NVLink — on an H100 system, roughly 900 GB/s of bandwidth:

Across nodes, the transfer has to go over the network — RDMA over InfiniBand, commonly 400 Gb/s per NIC on modern clusters, which is about 50 GB/s. Assume 4 NICs dedicated to KV transfer traffic, aggregated:

Over plain Ethernet without RDMA, at a more pedestrian 10–25 Gb/s (1.25–3.1 GB/s):

The pattern across all three: disaggregation is a clear net win only when the interconnect is fast enough that the KV transfer time stays a small fraction of the request's own end-to-end latency. NVLink, always yes. RDMA-class networking (InfiniBand, RoCE), usually yes for interactive workloads. Ordinary TCP/IP over Ethernet without RDMA, essentially never — the transfer cost swamps everything it was meant to save.

When it actually wins in practice

Disaggregation pays off most when prefill and decode load are decoupled — their traffic patterns don't move together. A RAG (retrieval-augmented generation) application might see huge, bursty prefill demand (long retrieved documents stuffed into context) that has nothing to do with how many decode streams are concurrently active. A single shared fleet has to be sized for the worst case of both at once; two independent fleets can each be scaled to their own actual demand curve, and the prefill fleet in particular can often use cheaper, more compute-dense hardware without needing decode's memory-bandwidth headroom.

NVIDIA's Dynamo is the orchestration layer built for exactly this: routing incoming requests to the right prefill-fleet GPU, managing the KV-transfer handoff to a decode-fleet GPU once prefill completes, and autoscaling each fleet independently based on its own queue depth and utilization — rather than requiring an operator to hand-wire this coordination themselves.

Transfer time across the traffic you'll actually see

An 8,000-token prompt was one convenient example. Real traffic spans a much wider range — short chat turns through long document-stuffed RAG prompts. Trace transfer time across that range, for both fast interconnects:

Even at the long end of this range — a 32,000-token document, near the edge of what many production systems allow — RDMA transfer stays under 80 ms, a real but bounded cost against a request whose own prefill compute at that length would already run into whole seconds. NVLink stays comfortably under 20 ms at every length shown. The interconnect choice, not the prompt length, is what decides whether disaggregation pays off; prompt length just decides by how much margin.

Independent scaling: the economic case, worked

The strongest argument for disaggregation isn't the transfer cost — it's what independent fleet scaling buys when prefill and decode demand move differently. Suppose prefill demand is bursty (peaks at 5× its average, driven by occasional large-document uploads) while decode demand is comparatively steady (peaks at 1.5× its average, ordinary chat concurrency). A single combined fleet has to be provisioned for the worst case of either spike happening, since both phases share the same GPUs:

Two independent fleets, each sized to its own demand curve:

The exact ratio depends on how much of total GPU-time each phase consumes on average, but the direction is robust: whenever the two phases' demand curves are decoupled, sizing them together forces the calmer phase to be chronically overprovisioned just to cover the bursty one's peak. Splitting them lets each fleet breathe at its own rhythm.

Different hardware per fleet

Disaggregation also unlocks a hardware choice that a combined fleet can't make: prefill wants raw compute throughput and can tolerate somewhat less memory bandwidth per FLOP, since it's compute-bound (Chapter 4); decode wants memory bandwidth above almost everything else, since it's memory-bound (Chapter 0). A combined fleet must buy one GPU SKU good enough at both. A disaggregated deployment can put a compute-dense, cheaper-per-FLOP chip on the prefill fleet and reserve the most memory-bandwidth-rich (and usually most expensive) chip for the decode fleet where that bandwidth is actually the bottleneck — extra architectural freedom a shared fleet structurally cannot exploit, on top of everything already priced out above.

Why not just always disaggregate?

As a rough rule of thumb worth stating explicitly: disaggregation tends to earn its operational overhead once a deployment is running enough replicas that prefill and decode fleets can each independently absorb their own demand spikes without either dropping below a handful of GPUs — a double-digit total GPU count is a reasonable place to start seriously evaluating it, not a hard threshold, but a useful gut check before adopting the added architectural complexity this chapter describes.

Interconnects keep getting faster — recheck the arithmetic, don't just trust last year's answer

This chapter's numbers used H100-generation NVLink (900 GB/s). Newer NVLink generations on more recent platforms push considerably higher — on the order of 1.8 TB/s aggregate on some newer multi-GPU systems. Redo the 8,000-token transfer-time arithmetic at that bandwidth:

The qualitative verdict from this chapter (NVLink: yes; RDMA: usually yes; plain Ethernet: essentially never) doesn't change with faster interconnects — it just gets more comfortably true. What can change over time is where the RDMA-versus-Ethernet line actually falls in practice, as commodity networking gradually gets faster too. The right habit isn't memorizing this chapter's specific millisecond figures as permanent facts; it's remembering the formula — KV bytes divided by achieved bandwidth, compared against the request's own end-to-end latency — and re-running it against whatever hardware you're actually deploying on.

The KV cache itself must survive the trip, correctly

Transfer time is only half the correctness story. The receiving decode-fleet GPU has to reconstruct the exact same block-table indirection (Chapter 2) the prefill GPU used, in its own local physical block pool — which almost certainly has different physical block IDs free at that moment, since the two GPUs' pools evolve independently. The transfer protocol has to carry the KV data plus enough structure to let the receiving side re-establish a valid block table pointing at wherever it actually lands locally, not assume the same physical block numbering applies on both ends. Get this wrong — copy raw bytes without re-mapping the logical-to-physical indirection — and the decode-fleet GPU silently reads wrong keys and values for some positions, a correctness bug far worse than the latency Chapter 6 quantifies, because it corrupts output instead of merely slowing it down.

A cost comparison: two fleets vs one, at fixed total GPU count

Hold total GPU count fixed at, say, 20, and compare two provisioning strategies for a workload whose average prefill-to-decode GPU-time ratio is roughly 30:70 (typical for chat-heavy traffic with occasional longer prompts). A combined fleet runs all 20 GPUs interchangeably, each doing both phases as scheduled. A disaggregated split allocates GPUs by that same ratio:

If prefill demand spikes to double its average share (a burst of long-document uploads) while decode stays steady, the disaggregated prefill fleet can absorb that spike by drawing down its own queue depth and, if autoscaling is wired up, temporarily borrowing capacity — without touching the 14 GPUs dedicated to decode, whose users see no ITL impact at all. A combined fleet facing the identical spike has no such isolation: every one of its 20 GPUs is now spending a larger fraction of its time on prefill, and Chapter 4's chunked-prefill stutter risk rises fleet-wide, for every user, decode-only or not. The isolation this chapter argues for isn't just about raw throughput — it's about containing a spike in one phase from ever touching users who are only interacting with the other.

Dynamo's role, precisely: orchestration, not a new inference kernel

Worth being precise about what NVIDIA's Dynamo actually is, since the name can suggest a fourth alternative to vLLM/SGLang/TensorRT-LLM rather than what it is: an orchestration and routing layer that sits above whichever inference engine actually runs prefill and decode on each fleet. Dynamo decides which prefill-fleet node handles an incoming request, manages the KV-transfer handshake this chapter derived the arithmetic for, and autoscales each fleet based on its own queue depth — but the actual prefill and decode compute underneath can be running vLLM, SGLang, or TensorRT-LLM workers, which is exactly why Chapter 7's engine-choice decision and this chapter's disaggregation decision are independent axes, not alternatives to choose between. A deployment can disaggregate its fleets while still running whichever inference engine won Chapter 7's weighted comparison on each side of that split.

Prefill fleets and prefix caching, together

One more interaction worth naming: Chapter 3's prefix cache lives in GPU memory, and a disaggregated prefill fleet is exactly where it's most valuable, since every prefill — cached-hit or not — now runs exclusively on that fleet rather than competing with decode for the same GPU. A well-populated prefix cache on the prefill fleet means a large fraction of incoming requests barely touch that fleet's compute at all, letting a comparatively small number of prefill-fleet GPUs serve a much larger share of traffic than raw prefill-FLOPs accounting alone would suggest — another way these techniques compound: disaggregation isolates prefill's demand curve, and prefix caching shrinks that demand curve further, and the two savings multiply rather than merely add.

The decision in one sentence

Strip away every number in this chapter and the decision rule left standing is short: disaggregate when prefill and decode demand genuinely move independently, at a scale where each fleet can flex on its own, and only when the interconnect between them is fast enough that the KV handoff stays a rounding error against the request's own latency — get any one of those three conditions wrong (correlated demand, too small a deployment, or too slow a network) and a well-tuned combined fleet with Chapter 4's chunked-prefill scheduling is simpler to operate and performs just as well.

Where this leaves the memory wall from Chapter 0

It's worth closing this chapter by noticing what disaggregation does not change: the decode fleet is still bound by exactly Chapter 0's memory-bound arithmetic, and the prefill fleet is still bound by exactly its compute-bound crossover math. Splitting the fleets doesn't invent new physics — it removes the interference between two workloads that were fighting over the same silicon, letting each one separately reach closer to the ceiling Chapter 0 already derived for it. Every chapter in this lesson, disaggregation included, is ultimately in service of that one opening chapter's numbers, not a departure from them.

Carry this forward

One test to run against your own numbers before adopting this chapter's architecture: compute KV size for your typical prompt length, divide by your actual available interconnect bandwidth, and compare that to your target end-to-end latency. If the transfer time comes out under roughly 5% of that target, disaggregation is almost certainly a net win once you're at sufficient scale to operate two fleets; above that, the interconnect itself needs upgrading before this chapter's architecture pays for itself.

One more number worth checking by hand

(Even at Ethernet speeds, a very short prompt's transfer stays tolerable in isolation — it's long-context traffic where the interconnect choice becomes make-or-break, exactly as the sensitivity table above shows. Size your interconnect for the LONGEST prompts your product actually sees, not the average, since that is the worst-case transfer time users will actually experience, and worst-case latency is what a real SLO has to be built against, not a comfortable average.)

(A single KV-transfer benchmark run against your actual network path is worth more than any of this chapter's generic bandwidth figures — real interconnects rarely hit their rated peak under production contention. Budget accordingly, and re-check after any hardware or driver upgrade.)

(Treat the 5% transfer-time-to-latency-target ratio from earlier in this chapter as a starting heuristic, not a hard rule — tighten or loosen it based on how latency-sensitive your specific product actually is.)

Chapter 7: Choosing an Engine

Six chapters of mechanism later, the honest question: given a real workload, which engine? Not which is “best” in the abstract — the three engines this lesson names optimize for genuinely different points on the tradeoffs Chapters 0–6 built, and the right choice depends on which of those tradeoffs your traffic actually hits.

An honest comparison, axis by axis

None of these rows contradicts anything derived earlier in this lesson — they're direct consequences of each engine's central design bet. vLLM bet on memory management (Ch. 2) and made it the default everywhere else builds on. SGLang bet that most real traffic has more shared structure than people assume, and built its scheduler around exploiting that structure maximally (Ch. 3). TensorRT-LLM bet that for a fixed, known, high-volume shape, paying an upfront compile cost to strip every layer of dispatch overhead (Ch. 5) is worth it.

A decision by workload shape

Interactive chat, many short turns, high concurrency. ITL matters most (Ch. 4); prefix reuse across turns and shared system prompts matters a lot (Ch. 3). SGLang's default RadixAttention scheduler is purpose-built for exactly this shape. vLLM with prefix caching enabled is a strong, more general-purpose runner-up.

Offline batch jobs — summarizing a million documents overnight, no human waiting. Aggregate throughput (Ch. 0's crossover batch) dominates; TTFT and ITL barely matter, since nobody is watching a token stream in real time. Larger prefill chunks (Ch. 4), FP8 (Ch. 5), and a stable, compiled TensorRT-LLM engine tuned for this one known shape can extract the most raw throughput per GPU-hour, since there's no interactive-latency constraint fighting the compile-time investment.

Agentic workloads — tool calls, multi-step reasoning, long and highly variable-length generations, frequently branching (self-consistency, tree search). Heavy prefix and tree sharing (Ch. 3) across branches, wide variance in output length that rewards continuous batching's steady-state utilization (Ch. 1) especially strongly, and enough architectural churn during development that TensorRT-LLM's rebuild tax (Ch. 5) is hard to justify until the agent's shape stabilizes. SGLang's tree-aware scheduling is a particularly strong fit; vLLM is a capable, flexible alternative.

Turning the comparison table into an actual number

Qualitative fit is a starting point; a real decision benefits from putting weights on what matters for your specific traffic and computing a score, rather than picking by reputation. Score each engine 0–1 on four axes for an interactive-chat workload, then weight those axes by how much this workload actually cares about each one — weights that should sum to 1:

Compute each engine's weighted score by hand — multiply each cell by its row's weight, then sum down the column:

SGLang wins this weighted comparison for interactive chat, and the arithmetic shows exactly why: it dominates on the two heaviest-weighted axes (ITL and prefix reuse) for this workload, which matters more than its comparatively smaller lead or lag elsewhere. Rerun the identical table with offline-batch weights (throughput 0.60, iteration speed 0.05, the other two roughly 0.175 each) and TensorRT-LLM's compute-throughput strength would instead dominate the sum — the ranking isn't fixed, it moves with what the weights say your traffic actually needs, which is the entire point of doing this arithmetic instead of picking by reputation.

The switching cost this matrix doesn't capture

A weighted score picks the best engine for a snapshot of your traffic today; it says nothing about the cost of moving to it from whatever you're already running. Changing engines means re-validating that outputs match (different kernels can produce tiny floating-point differences that occasionally change a generation), rebuilding deployment tooling, retraining on-call runbooks, and absorbing a migration window's worth of risk. For a workload sitting close to a tie in the weighted comparison, that switching cost alone can be reason enough to stay put — the matrix is a tool for a genuine gap, not a mandate to chase every small score difference.

The same matrix, reweighted for offline batch work

Rerun the identical scoring exercise for the offline-batch workload described earlier — no human waiting, throughput dominates, iteration speed barely matters because the job runs on a fixed, already-validated model:

The ranking flips: TensorRT-LLM edges ahead once throughput carries 70% of the weight instead of 15%, exactly because that's the one axis it dominates. Notice, too, that SGLang stays highly competitive even here — its across-the-board strength means it's rarely a bad choice, just not always the best one once a workload's weights shift hard enough toward a single axis another engine was purpose-built for.

Hybrid deployments: using more than one engine at once

Nothing forces a single choice fleet-wide. A real production system commonly runs SGLang (or vLLM) for its latency-sensitive interactive endpoint and a separately-provisioned TensorRT-LLM fleet for a nightly batch-summarization job against the same underlying model — two engines, two workloads, each matched to its own row in this chapter's matrix, rather than one engine compromising for both. The router from Chapter 8's pipeline diagram is exactly the component that would direct traffic to the right fleet based on request type, if a deployment reaches the scale where running two engines is worth the added operational surface.

Re-scoring as a workload matures

Revisit the interactive-chat score from earlier once a product has been live long enough to gather real usage data: iteration speed's weight (0.20 in the original example) naturally shrinks as the model and prompt templates stabilize, while raw throughput's weight (0.15) tends to grow as traffic volume increases and cost per request starts mattering more than it did during early testing. Recompute the weighted sum with iteration speed dropped to 0.05 and throughput raised to 0.30 (holding ITL and prefix reuse at 0.35 and 0.30):

SGLang still wins here, but the gap to TensorRT-LLM narrowed noticeably (from 0.8705 vs 0.615 to 0.8435 vs 0.6975) as throughput's weight grew — a concrete illustration of exactly the maturation pattern described earlier: as a product moves from prototype to scaled production, re-running this matrix periodically, not choosing once and never revisiting it, is what catches the moment a different engine's tradeoffs start to win.

Total cost of ownership, not just the weighted score

The scoring matrix in this chapter deliberately left out one input that real decisions can't: engineering time. A team with deep in-house TensorRT-LLM experience will pay a smaller effective “iteration speed” penalty than the generic weights above assume, because their build pipeline and debugging muscle memory for that engine are already strong; a team that has only ever operated vLLM will find SGLang's migration cheaper than TensorRT-LLM's, independent of what a generic axis score suggests, purely because less new operational knowledge has to be built from scratch. The matrix in this chapter is a starting point for a decision, adjusted with exactly this kind of team-specific context — it is not a substitute for knowing your own organization's actual capabilities.

These engines are moving targets

A decision checklist, distilled from every chapter above

Reduce the whole matrix exercise to five questions worth asking about any real workload before writing a single line of deployment configuration, each one pointing straight back at the chapter that derived why it matters:

None of these questions has a universally right answer — that's the entire point this chapter has been making. What they share is that each one is answerable with a number you can actually measure from your own traffic (prefix hit rate, ITL percentile targets, redeploy frequency, prefill-to-decode GPU-time ratio), not a guess, because every chapter before this one built the arithmetic that turns the guess into a measurement.

What this chapter's matrix cannot tell you

Every score in this chapter's tables is an estimate, not a measurement — a plausible relative ranking based on each engine's design, not a benchmark run on your actual model, hardware, and traffic. The only way to turn this chapter's exercise from an informed guess into a real decision is to instrument all three genuinely, against a representative slice of real traffic: measure actual TTFT and ITL percentiles, actual prefix hit rates, actual cold-start times on your own deployment pipeline, and actual throughput at your own typical batch sizes. This chapter's weighted-matrix method is valuable precisely because it forces you to name the axes that matter before you look at any benchmark number — so that when the real numbers come in, you already know how to weigh them, rather than being swayed by whichever engine happens to win on the one metric a vendor chose to publish.

A last worked check: does the matrix agree with the crossover arithmetic?

It's worth confirming this chapter's scoring exercise doesn't contradict Chapter 0's hard numbers — a decision framework that ignored the underlying physics would be worthless no matter how principled its weights look. Offline batch work, scored highest for TensorRT-LLM above, is precisely the workload that can run at or near Chapter 0's crossover batch continuously, with no interactive-latency constraint forcing it below that point the way a chat product's ITL budget does. TensorRT-LLM's throughput edge (Chapter 5's FP8 and fusion gains) therefore compounds directly with the ability to actually sustain that crossover batch size in this workload — the matrix's ranking and the hard arithmetic underneath it are pointing the same direction, which is the sanity check worth running on any decision framework before trusting it: does the scored outcome make sense given the physics derived earlier, or does it contradict it?

One paragraph per engine, as a final summary

vLLM is the general-purpose default: PagedAttention at its core, continuous batching from the start, fast to iterate on, broad model support. Reach for it when you don't yet know which axis of this chapter's matrix will end up mattering most, and want to start serving quickly while that becomes clear. SGLang leads specifically when shared structure across requests — prefixes, trees, branches — is a first-order property of your traffic, because RadixAttention is its scheduling core, not a bolted-on feature. TensorRT-LLM earns its build-time cost once your deployment has stopped changing shape and volume is high enough that squeezing out the last increment of throughput per GPU-hour outweighs the flexibility every other engine on this list offers by staying eager. None of the three is wrong to start with; the wrong move is picking once, never re-running this chapter's questions, and being surprised months later that the workload outgrew the original choice.

Revisit this table after reading Chapter 8

This chapter's comparisons scored each engine in isolation. Chapter 8's full lifecycle trace shows the same three engines' mechanisms working together inside one request — scheduling, prefix lookup, paged allocation, and decode, chained in sequence — which is the more accurate picture of what actually determines a workload's real-world performance: not any single engine's headline strength in isolation, but how well its particular combination of mechanisms handles your traffic's actual shape, end to end.

Carry this forward

The habit worth keeping, more than any specific score in this chapter's tables: write down the axes that matter for your workload, weight them honestly, and only then look at benchmark numbers — in that order. Reversing the order (find a benchmark, then rationalize why its winning axis was the one that mattered all along) is how teams end up running an engine that's a poor fit for their actual traffic, having convinced themselves otherwise after the fact.

One more number worth checking by hand

(Rerun this same scoring exercise on your own workload before trusting any of the three rankings this chapter computed — the method transfers directly; the specific weights and axis scores almost certainly won't, because they were chosen to illustrate the method, not measured against your own production traffic. Treat every number in this chapter's tables as a placeholder for a measurement you still need to take, on your own model, your own hardware, and your own traffic mix, before it earns a place in a real deployment decision.)

(The scoring method survives even a completely different set of axes — if your workload cares about something this chapter didn't name, add a row, weight it honestly, and rerun the same arithmetic. That extensibility is the whole reason this chapter taught the method instead of just handing over a ranking.)

(Come back to this chapter after Chapter 8's full lifecycle trace — seeing all three engines' mechanisms run together on one request makes the axis weights in this chapter's tables far more concrete.)

Chapter 8: The Full Request Lifecycle

Every chapter in this lesson studied one stage in isolation. Trace one real request through all of them, in order, with every knob this lesson introduced live at once, and the whole machine becomes one picture instead of eight.

The path, stage by stage

Watch the knobs interact

Nothing in this pipeline is independent. Raise the prefix hit rate (more requests sharing cached prefixes, Chapter 3) and stage 4 shrinks for most requests, which frees GPU cycles that stage 6 can use to admit a bigger batch, which pushes the system closer to Chapter 0's crossover throughput. Increase max batch size past that crossover and per-request decode latency (stage 6) starts to climb, exactly as Chapter 0's tail behavior predicted. Increase chunk size (stage 4) and a big prompt's own TTFT improves, but every other request's stage 6 ITL takes a bigger hit per round, exactly Chapter 4's tradeoff table. None of these knobs can be maximized independently — tightening one changes the room left for the others.

A full worked trace

One concrete request: a 4,000-token prompt, 60% of which matches an already-cached prefix (a long-running conversation), arriving at a system running batch size 140 (just under Chapter 0's 156 crossover), chunk size 512, on an A100.

Every number in that trace is a callback to a chapter you've already derived by hand. That's the entire point of this lesson: a serving engine isn't one clever trick, it's these seven stages, each solving one specific piece of Chapter 0's original waste, wired together without breaking each other.

A second trace: the cold-cache case

Contrast the worked trace above with the same 4,000-token prompt arriving with a 0% prefix hit rate — a genuinely new conversation, nothing cached:

Nothing about the decode phase changes — once admitted to the batch, this request still gets the same per-user rate everyone else does, governed purely by Chapter 0's step time, not by how the request got there. Prefix hit rate only ever affects the prefill stages (3 and 4); it has zero effect on stage 6, which is exactly why raising the hit rate frees GPU cycles rather than changing what a fully-admitted decode step costs.

Failure gallery: five ways this pipeline breaks in production

Every stage in the lifecycle above is a place a real deployment can go wrong. Five failures worth knowing by symptom, not just by mechanism, because the symptom is what you'll actually see first.

1 · Admission race — OOM despite “plenty” of free slots

Mechanism. Chapter 1's continuous batcher checks for a free seat; it doesn't always check whether the KV-block pool (Chapter 2) has room for that seat's eventual growth. A burst of admissions, each individually reasonable, can collectively outrun the block pool before any of them finish and free memory back.

Fix. Admission control that checks projected KV usage (current usage plus each candidate request's expected growth) against the pool budget before admitting, not just whether a scheduling seat is open.

2 · Prefix-cache poisoning

Mechanism. Chapter 3's cache matches on exact token sequences. If tenant or session identity isn't part of what's hashed into the cache key, two different users' near-identical prompts can collide, and one user's cached continuation gets served to another.

Fix. Scope every cache key to include tenant/session identity for anything touching private context, exactly the discipline this campaign's semantic-caching lesson covers in depth for the retrieval layer — the same failure, one layer down the stack.

3 · Chunked-prefill starvation of one huge document

Mechanism. If the scheduler always prioritizes fresh short requests' prefill chunks over continuing a long one already in progress, a steady stream of new short arrivals can perpetually push the long document's remaining chunks to the back of the queue — a queuing fairness failure, not a compute-capacity one.

Fix. Weighted or age-aware chunk scheduling (Chapter 4's priority discussion), so a request's chunks accumulate priority the longer they wait, guaranteeing eventual completion.

4 · Disaggregation network partition

Mechanism. Chapter 6's KV transfer assumes the interconnect between fleets stays up. A network partition or a congested link between the prefill and decode fleets strands finished KV caches with nowhere reliable to go.

Fix. Timeouts and fallback routing to alternate decode-fleet nodes, plus monitoring on the transfer path itself (Chapter 6's 4.4/19.5 ms budgets), not just on each fleet's own health.

5 · Silent engine-compatibility drift

Mechanism. A TensorRT-LLM engine (Chapter 5) compiled against one model checkpoint and precision configuration served against a slightly different checkpoint after an untracked model update — the engine still runs, since the shapes match, but the fused kernels' calibration no longer matches the actual weight distribution.

Fix. Version-pin engine builds to exact checkpoint hashes, and treat an engine rebuild as a mandatory step of any model update, not an optional optimization pass.

Why every stage checks the one before it, not just its own state

Look again at the seven-stage flow at the top of this chapter and notice what each stage's decision actually depends on. The scheduler (stage 2) can't decide whether to admit a request without knowing the current KV pool budget (stage 5's resource, checked in advance). The prefix lookup (stage 3) determines how much work stage 4 actually has to do. Stage 4's chunking decisions determine how much GPU time is available for stage 6's decode loop to stay smooth. None of these stages is a clean, isolated microservice with its own independent SLA — they all draw from and report back to the same shared state (the batch, the block pool, the radix tree), which is exactly why a serving engine has to be architected as one coherent system rather than a pipeline of independently-scalable stages the way a typical web backend might be. This is the deepest structural reason this lesson built every chapter's mechanism before showing the assembly: none of them are separable in a real engine, only in the explanation of one.

Reading the assembly canvas like an operator would

The tracer widget below is deliberately shaped like a real serving dashboard, not a diagram — three knobs an operator actually controls, six readouts an operator actually watches. Practice reading it the way you'd triage a real alert: if TTFT looks fine but ITL stutter is high, the fix lives in Chapter 4 (shrink the chunk size). If GPU utilization is low and per-user decode rate is already at its ceiling, the fix lives in Chapter 1 or Chapter 0 (the batch isn't full enough — raise max batch size, or check whether arrivals are keeping the queue fed). If prefill tokens needed stays stubbornly high across many requests, the fix lives in Chapter 3 (the prefix cache isn't being hit — check whether prompts are byte-identical where they should be). Every one of this lesson's chapters maps to a specific corner of this one panel; that mapping, more than any individual number, is the actual skill this lesson was built to teach.

The one sentence this whole lesson has been building toward

Every chapter's mechanism, restated as one idea: a GPU is not slow at serving LLMs, it is underused at serving one at a time, and everything from Chapter 1's continuous batching through Chapter 6's disaggregated fleets is a different, compounding answer to the exact same question Chapter 0 opened with — how do you keep this specific piece of very expensive, mostly-idle silicon doing useful work for as many people as possible, without corrupting anyone's output or making anyone wait unreasonably long for their own first token. Nothing in this lesson made the chip faster. All nine chapters made it busier.

The seven stages, mapped one last time to their chapters

Router (introduced in Chapter 6's disaggregation discussion), admission (Chapter 1), prefix lookup (Chapter 3), chunked prefill (Chapter 4), paged allocation (Chapter 2), the decode loop itself (Chapter 0's whole opening argument), and completion — every stage in this pipeline has its own chapter's worth of derived arithmetic behind it, and none of those chapters could have been skipped without leaving a gap in this trace that the numbers wouldn't have closed on their own.

One more number worth checking by hand

(Little's Law doesn't care which engine is running underneath — it only needs an arrival rate and a service time, both measurable from any production deployment's own logs, no matter which of Chapter 7's three engines happens to be serving that traffic. Pull those two numbers from your own metrics before trusting any capacity plan built on assumed traffic, and re-derive the concurrency estimate whenever either one moves by more than a small margin, since both directly determine how many GPUs you actually need.)

(This closes the loop this lesson opened in Chapter 0: a product number becomes an arrival rate, an arrival rate becomes a concurrency requirement, and a concurrency requirement becomes a GPU count — every step derived, none of it guessed.)

Sizing a whole fleet from one product number

Close the loop between this lesson's arithmetic and an actual capacity-planning decision. Suppose a chat product needs to serve 10,000 requests per day, each averaging 2 seconds of decode time, arriving roughly evenly across a 16-hour active window (a simplifying assumption — real traffic has peaks, which a real plan would pad for, but this gives the right order of magnitude).

Apply Chapter 1's Little's Law:

That's comfortably inside a single GPU's crossover batch of 156 from Chapter 0 — this traffic level needs, at steady state, a small fraction of one GPU's capacity. One A100, running one instance of any of the three engines from Chapter 7, would sit at extremely low utilization almost all the time, with huge margin for traffic bursts. Scale the same product to 1,000,000 requests/day instead:

Still under one GPU's 156-request crossover — a single well-configured GPU, running near but not past the crossover, could in principle serve this entire product's average load, though real deployments would still run multiple replicas for redundancy and to absorb bursts above the average. This is the entire lesson's arithmetic, chained end to end: a product's daily request volume determines an arrival rate, Little's Law turns that into a concurrency requirement, and Chapter 0's crossover batch says whether that concurrency fits on one GPU or needs a fleet — the exact question Chapter 6's disaggregation and this campaign's autoscaling lesson pick up from here.

What's still missing from this picture, honestly

This lesson stayed inside one deployment's serving mechanics on purpose — it's already a full syllabus. Two things it deliberately did not cover, both queued as their own lessons in this campaign: how many replicas of this whole pipeline you actually need, and when to spin them up or down as traffic changes (autoscaling, cold starts, scale-to-zero economics); and what any of this actually costs per request once you multiply GPU rental price by the throughput numbers this lesson derived (token economics, the API-vs-self-host break-even). Speculative decoding — spending the idle compute this lesson keeps rediscovering to guess several tokens ahead instead of just batching more users into it — is a complementary technique this lesson deliberately left to its own dedicated treatment, since it changes the per-token compute math in a way that interacts with, but is genuinely distinct from, everything built here.

One last worked comparison: the whole lesson's arithmetic, side by side

Stack every major numeric result this lesson derived into one table — not to memorize, but to see the shape of the whole argument at once: each row moved one piece of the same fixed GPU from wasted to used.

No single row is the “real” fix. All six compound, and Chapter 8's own worked trace is what it looks like when they run together on one request instead of being studied one at a time.

“Premature optimization is the root of all evil.” — Donald Knuth. A GPU sitting at 0.64% utilization while serving one user is not premature optimization’s target; it’s the opposite kind of mistake — the optimization that was never attempted at all, because nobody looked at the number first.