Fast Embeddings on GPUs: Ivy, Tulip, ROSE and the Host Side of Inference

Chapter 0: Search Runs on Embeddings

You type a question into a search box and an answer comes back in under a second. Somewhere in that second, a neural network turned your sentence into a list of numbers, and a much larger fleet of machines had already turned several billion documents into lists of numbers so the comparison could happen at all.

Those lists of numbers are embeddings. The model that makes them is small by modern standards, a Transformer well under the size of a chat model. And that smallness is the whole story of this lesson.

On 4 September 2026, Perplexity Engineering published Fast Embeddings on GPUs, an under the hood tour of the serving infrastructure behind their search stack. The headline finding is unusual for a performance post, because it opens by conceding most of the ground:

The concession, in their words. "Both Transformer-based models and the underlying Hopper/Blackwell architectures are mature technologies, so embedding inference on the GPU side has converged to a largely optimal implementation across various inference engines." Everybody's matrix multiplies are fast. Everybody's attention kernels are fast. If you want a win, it is not there.

And yet they report cutting the median latency of a short embedding request from 4.60 milliseconds to 1.53 milliseconds against vLLM v0.22.0 on the same hardware, running the same model, in the same precision. Same GPU work, three times faster end to end.

The gap between those two facts is what you are here to understand. It lives entirely on the host: kernel launches, Python, scheduling, memory copies, tokenization, gRPC. This lesson takes that machinery apart piece by piece.

The Shape of a Search System

Start with the setting, because it decides which numbers matter. In a typical search setup, indexed documents are mapped to a high dimensional vector space using an embedding model and stored in a vector database. Embedding a query with the same model lets you find similar documents by finding the nearest vectors. If any of that is unfamiliar, the Vector Embeddings lesson builds it from zero, and this lesson assumes it.

Now look at the same picture from the inference engine's side of the wire. The engine does not see "search". It sees requests arriving with wildly different shapes, and the post sorts them into traffic patterns:

Batch embedding

Building, expanding or re-indexing the database. Bulk documents get embedded into the vector space. Maximise throughput to minimise cost. Nobody is waiting.

↓

Scoring

Following vector search, large batches of candidate documents must be scored. Strike a balance between throughput and latency. Somebody is waiting, but for a batch.

↓

Online embedding

Querying the database. A short query must be embedded for lookups. Minimise latency. Somebody is waiting, for one sentence.

A small honesty note about the source. The post says these give rise to "two different traffic patterns" and then describes three workloads. Batch embedding and online embedding are the two poles: throughput at one end, latency at the other. Scoring is the workload in between, and the post describes it as striking a balance between the two. We keep that structure throughout, and we keep its benchmarks separate, because the engine really does behave differently in all three.

The Search Path, and Which Stage Each Pattern Stresses

Pick a traffic pattern. The stages it loads light up, the metric it optimises fills, and the published latency or throughput anchor for that pattern appears underneath. All three run on the same model, on the same engine.

What Each Pattern Actually Costs

Put real numbers on those three rows, taken straight from the post's benchmark figures. The point is not that one is bigger than another. The point is that they are measured in different units because they are asking the engine for different things.

Pattern	Request shape in the benchmarks	Metric	Published anchor
Batch embedding	Request batch size 100, four concurrent processes, sequence length 512	Embeddings per second	1499.3 emb/s on BGE-M3, against 1342.2 for the baseline
Scoring	Pre-tokenized request batch sizes 5, 25 and 50, sequence length 512	Milliseconds per request	6.17 ms p50 on BGE-M3 at batch 5, against 9.60 for the baseline
Online embedding	Pre-tokenized request batch size 1, fully sequential, sequence length 128	Milliseconds per request	1.53 ms p50 on BGE-M3, against 4.60 for the baseline

Every number on this page comes from that post or from the charts inside it. Where a value is read off an unlabelled line chart we say so. Where the two engines tie, or where the baseline wins, we say that too, because several cells do exactly that and a lesson that hid them would be teaching you the wrong lesson about where the wins come from.

When to reach for it: You run a retrieval augmented generation service and the p99 on the query path is 40 milliseconds, of which the embedding call is 12. The concrete artefact is a flame graph where the embedding span is wide but the GPU utilisation counter on that box reads under ten percent. Why this framing helps: it tells you immediately that you are in the online embedding pattern, that your bottleneck is not the matrix multiplies, and that buying a bigger GPU will change nothing. Where the throughput mindset fails: you will spend a sprint tuning batch sizes for a workload whose batch size is one and can never be anything else.

Why the Model Being Small Changes Everything

A 70 billion parameter chat model spends so long inside each matrix multiply that the host has plenty of idle time to prepare the next step. An embedding model under a billion parameters, handed a 128 token query, finishes its entire forward pass in a fraction of a millisecond.

At that scale the fixed costs stop being noise. Launching a kernel costs a few microseconds of host time whether the kernel runs for two microseconds or two milliseconds. Parse the JSON, tokenize, build the batch, launch a hundred and forty kernels one at a time, copy the result back, serialise it: those costs do not shrink when the model does.

So the engineering question for this lesson is not "how do we make the GPU faster". The GPU is fine. The question is: how much of the wall clock can we take away from everything that is not the GPU?

The thesis, stated once so you can hold the whole lesson against it. Perplexity's wins are host side wins. They show up exactly where host cost dominates: batch size one, short sequences, scoring, and many concurrent requests. They shrink toward nothing, and in a few published cells they reverse, at large batches and long sequences, where the work really is on the GPU and the two engines are doing the same fast thing.

The post names two traffic patterns as the poles of embedding serving. Which pair, and what does each one optimise?

Training and inference: one minimises loss, the other minimises latency Dense and sparse retrieval: one maximises recall, the other maximises precision Batch embedding, which maximises throughput to minimise cost, and online embedding, which minimises latency for a short query Prefill and decode: one is compute bound, the other is memory bound

Where we are going. Chapter 1 explains why an embedding engine can reuse an LLM engine's kernels without modification. Chapter 2 names the three services. Chapters 3 through 6 are the core: the cost model that justifies a dumb scheduler, the launch overhead that motivates CUDA graphs, the lazy capture that makes thousands of graphs affordable, and the LazyTensor that overlaps the host with the device. Chapters 7 and 8 cover the engine and the gateway. Chapters 9 and 10 read the kernel chart and the benchmarks honestly. Chapter 11 turns all of it into a checklist you can carry.

Chapter 1: Embeddings Are Prefill and Decode

Here is the first surprising claim in the post, and it is the one that saved Perplexity the most engineering time:

In their words. "Since we typically use small Transformer models to produce embeddings, we share the bulk of the implementation with our LLM inference code: batch embeddings are similar to compute-bound prefill, whereas online embeddings, which often run on a few tokens, are computationally similar to memory-bound decode. We thus reuse our optimized prefill and decode kernels to serve embedding models."

To see why that is true rather than a convenient analogy, you need one idea: arithmetic intensity. It is the only number that decides whether a piece of work is limited by the GPU's arithmetic units or by its memory bus, and it is easy to compute by hand.

Two Phases of an LLM, in One Paragraph Each

Prefill is what happens when you hand a chat model your whole prompt. Every token in the prompt goes through the network at once. A thousand token prompt means a thousand rows flowing through each weight matrix together. The weights get loaded from memory once and reused a thousand times. The GPU's arithmetic units are the bottleneck.

Decode is what happens after that, when the model emits one token at a time. Each step pushes a single row through the same weight matrices. The weights still have to be loaded from memory in full, and they get used once. The memory bus is the bottleneck and the arithmetic units are mostly idle.

Now map the embedding workloads onto that. Bulk indexing pushes thousands of document tokens at once: that is prefill. A user query of nine tokens pushes almost nothing: that is decode, arithmetically speaking, even though it runs only once and produces no new tokens.

The Arithmetic, By Hand

Take one dense layer, the workhorse of the model: a weight matrix of shape (d, d) applied to T token vectors. Set d = 1024 and work in BF16, so every number is 2 bytes.

Work done. Every output number is a dot product of length d, which is d multiplies and d adds. There are T rows and d outputs per row, so:

FLOPs = 2 · T · d · d

Bytes moved. The weight matrix must be read from high bandwidth memory: d · d numbers at 2 bytes each. The input rows must be read and the output rows written: 2 · T · d numbers, again at 2 bytes each.

Bytes = 2 · d · d + 4 · T · d

Now put the two numbers next to each other for a single query token, T = 1:

Quantity	T = 1 (one query token)	T = 512 (a small indexing batch)
FLOPs = 2 · T · 1024 · 1024	2,097,152	1,073,741,824
Weight bytes = 2 · 1024 · 1024	2,097,152	2,097,152
Activation bytes = 4 · T · 1024	4,096	2,097,152
Arithmetic intensity	1.00 FLOP per byte	256.0 FLOP per byte

Look at what happened. At T = 1 you move two megabytes of weights to do two megaflops of arithmetic, so you get one floating point operation per byte. The GPU can do hundreds of operations in the time it takes to fetch one byte, so it spends almost the entire layer waiting on memory. That is decode.

At T = 512 the activations have grown to exactly the size of the weights, the arithmetic has grown by a factor of 512, and the intensity is 256 operations per byte. Now the arithmetic units are the constraint. That is prefill.

The crossover, in closed form. Call the machine's balance B, the number of floating point operations it can do in the time it takes to move one byte. Setting the intensity equal to B and solving for T gives T = B · d / (d − 2B). For d = 1024 and B = 200 that is 200 · 1024 / (1024 − 400) = 204800 / 624, about 328 tokens. Below a few hundred tokens you are memory bound. Above it you are compute bound. Hold onto that number: the post independently puts GPU saturation for a sub-billion parameter model at around 512 tokens.

Roofline: Where Your Batch Sits

The classic roofline. The sloped part is the memory ceiling, the flat part is the arithmetic ceiling, and the knee is the machine balance. Drag the token count and watch a batch walk from the decode regime into the prefill regime. Widen the model and the knee moves left.

Tokens 64

Width d 1024

Balance 200

The Same Table, In Code

Nothing here needs a GPU. It is arithmetic on shapes, and running it is the fastest way to convince yourself that the two regimes are real.

python3
# Arithmetic intensity of one dense layer, in BF16, as the token count grows.
# FLOPs   = 2 * T * d * d          (one multiply-add per weight per token)
# Bytes   = 2*d*d  +  2*(2*T*d)    (weights read once, activations in and out)
d = 1024
balance = 200          # machine balance of a modern data centre GPU, FLOP per byte
print(f"{'tokens':>8} {'GFLOP':>9} {'MB moved':>9} {'FLOP/byte':>10}  regime")
for T in (1, 8, 64, 512, 4096, 32768):
    flops = 2 * T * d * d
    byts  = 2 * d * d + 4 * T * d
    ai    = flops / byts
    regime = "memory bound" if ai < balance else "compute bound"
    print(f"{T:>8} {flops/1e9:>9.3f} {byts/1e6:>9.3f} {ai:>10.2f}  {regime}")
print()
print("limit as tokens grow:", d / 2, "FLOP per byte")
print("crossover token count:", round(balance * d / (d - 2 * balance)) if d > 2 * balance else "never")

output
  tokens     GFLOP  MB moved  FLOP/byte  regime
       1     0.002     2.101       1.00  memory bound
       8     0.017     2.130       7.88  memory bound
      64     0.134     2.359      56.89  memory bound
     512     1.074     4.194     256.00  compute bound
    4096     8.590    18.874     455.11  compute bound
   32768    68.719   136.315     504.12  compute bound

limit as tokens grow: 512.0 FLOP per byte
crossover token count: 328

Two details in that output are worth pausing on. The intensity never exceeds d / 2, because in the limit the weights become free and you are left with one multiply-add per two bytes of activation traffic. And the crossover is a property of the machine and the model, not of the workload: change the GPU and the same batch changes regime.

What This Buys the Engineering Team

If bulk embedding is arithmetically prefill and online embedding is arithmetically decode, then an engine that already has good prefill kernels and good decode kernels already has good embedding kernels. Nothing about the kernel cares that no token will be generated afterwards.

The post spells out the payoff: "we can achieve massive batch inference throughput with minimal additional engineering work, while preserving low latency for online embeddings workloads." Chapter 7 goes through the small list of things that genuinely do differ, and it is a short list.

When to reach for it: Your team already runs a tuned LLM serving stack and product asks for a reranker endpoint next quarter. The concrete artefact is a decision doc choosing between standing up a second serving system and extending the one you have. Why the arithmetic framing helps: it lets you argue from FLOPs per byte that the reranker is your existing prefill path with the sampling loop deleted, so the work is plumbing rather than kernel engineering. Where the "embeddings are a different thing" instinct fails: you buy a second system, a second on call rotation, and a second set of regressions, for kernels you already had.

Why can an LLM inference engine reuse its prefill and decode kernels for embedding models without rewriting them?

Because embedding models and LLMs use the same tokenizer, so the inputs are identical Because a bulk embedding batch has the same arithmetic intensity profile as prefill and a short query has the same profile as decode, so the kernels face the same compute bound and memory bound problems Because embedding models are trained by distilling LLMs, so their weight layouts match Because both produce logits over a vocabulary, so the output stage is shared

Chapter 2: Tulips, Roses, and Some Ivy

Three services stand between an HTTP request and a GPU. They are named after plants, they are written in two different languages on purpose, and the boundary between them is the most interesting design decision in the whole post.

Here is the request that starts it, exactly as the post's architecture diagram shows it:

{ "model": "pplx-embed",
  "input": [
  "Roses are red",
  "The quick brown fox"
  ] }

Two short strings. By the time they reach a GPU they are two lists of token ids, [3630, 288, 525, 2518] and [17823, 574, 1382, 38835], and in the diagram they have been sent to two different replicas. That splitting is Ivy's doing, and chapter 8 is about why.

Ivy: the Rust HTTP Gateway

Ivy is the front door. Every Perplexity service that wants an embedding calls Ivy. Its job is all the CPU side work that has nothing to do with matrix multiplication:

JSON parsing

Turn the HTTP body into a request object. At high request rates this is not free, and it is exactly the kind of work that a systems language does better than an interpreter.

↓

Tokenization

Strings become token ids. Perplexity's in-house unigram tokenizer runs here, fully rolled out, and the post says it drastically improves latencies over off-the-shelf tokenizers.

↓

Input templating

Many embedding models want a prefix that names the task, one for a query and another for a document. Ivy applies it, so changing a template does not mean touching an inference box.

↓

Batch splitting

A large-batch request gets cut into chunks and load-balanced across replicas rather than pinned whole to one of them.

↓

Custom gRPC out

Ivy translates the request into a custom gRPC protocol for the downstream servers. Token ids on the wire, not strings.

The post gives the reason for the split plainly: "This separation allows us to configure certain parameters around tokenization and input formatting without having to touch the heavier inference instances." Tokenizer settings and prompt templates change often. Model weights and CUDA graphs do not. Putting the fast-changing pieces in a separate, cheap, stateless tier is a classic deployment argument, and it is worth more than it looks: a template change becomes a config rollout instead of a fleet restart that would throw away every captured graph.

Tulip: the Inference Server Interface

Tulip is a gRPC server implemented with Rust, tokio and tonic. Tokio is Rust's asynchronous runtime, the thing that lets one process hold thousands of in-flight requests without one thread each. Tonic is the gRPC implementation that sits on top of it.

Tulip receives gRPC inference requests, handles scheduling and batching, sends batches to the ROSE engine, and returns completed responses to clients. The post describes it as being "as lightweight of an interface over our model serving as possible". Concretely, each incoming request becomes a tokio async task. Those tasks put their sequences into a pool. A scheduling loop takes sequences out of the pool, forms a batch, hands it to ROSE, and later wakes the tasks whose results have arrived.

Why the interface is in Rust and not Python. Every millisecond Tulip spends is a millisecond the GPU is not being fed. A tokio task that is waiting on a result costs a few hundred bytes and no thread. The same design in Python fights the global interpreter lock for exactly the work that is on the critical path: waking a coroutine, copying a buffer, encoding a response. The post's closing section names free-threaded Python as the thing that would let more of this move back across the boundary.

ROSE: the Runtime-Optimized Serving Engine

ROSE implements model inference. It is primarily defined in Python, and it provides kernels, layers and definitions for a wide variety of models. It implements the forward passes, and it owns CUDA graph management specialised for embeddings.

The bridge between the Rust server and the Python engine is one function:

step(batch) → LazyTensor

The post describes it as a function "which takes a batch and returns a reference to the computation it performs on the accelerator." Read that carefully. It does not return embeddings. It returns a handle to a computation that has been launched and is still running. Chapter 6 builds that object from scratch, and it is the hinge the whole overlap story turns on.

Why Python survives in the deepest layer. It is tempting to conclude that the fix for a slow inference stack is to rewrite everything in Rust. The post does the opposite: it keeps model definitions, layers and forward passes in Python, because that is where you want to prototype a new model on a Friday. The post's own summary is "mixing highly re-usable and performant Rust primitives alongside more generic Python modeling code." The boundary is drawn where the cost is, not where the language preference is.

The Serving Path, End to End

A packet of work travels the real path from the published architecture diagram. Tap a stage to pin its responsibilities, language and what leaves it on the wire. Press play to watch a request go all the way to the GPU and the embeddings come back.

The Division of Labour, in One Table

Service	Language and runtime	Owns	Speaks	Holds state?
Ivy	Rust, HTTP gateway	JSON parsing, tokenization, input templating, batch splitting, load balancing	HTTP in, custom gRPC out	No model state. Tokenizer and templates only.
Tulip	Rust, tokio, tonic	Request pool, first come first served scheduling, batch formation, LazyTensor tracking	gRPC in, step() calls out	The in-flight request pool and the pending LazyTensors.
ROSE	Python	Kernels, layers, model definitions, forward passes, CUDA graph management	step(batch), returns a LazyTensor	Weights, captured CUDA graphs, static buffers.

When to reach for it: You are designing a tokenizer service and someone proposes putting it inside the model server "to save a hop". The concrete artefact is a proposal doc with one box instead of two. Why this split helps: tokenizer and template changes ship several times a week, and if they live in the model server every change restarts a process that then spends minutes recapturing CUDA graphs before it can serve at full speed. Where the single box fails: your cheapest, most frequently edited code becomes coupled to your most expensive, slowest-to-warm process, and you find out on the day of an incident.

Which service tokenizes the request, and what does the layer below it receive?

ROSE tokenizes, and Tulip receives embeddings Ivy tokenizes, and Tulip receives token ids over a custom gRPC protocol Tulip tokenizes, and ROSE receives raw JSON The client tokenizes, and Ivy receives token ids over HTTP

Chapter 3: Latency Is Tokens, Not Sequences

Tulip's scheduler is almost embarrassingly simple, and the post says so: "The scheduling mechanism in Tulip is very simple: requests accumulate while Tulip is dispatching work or waiting for results. From the accumulated requests, sequences are picked on a first-come, first-served basis to be run through the model."

No priority queue. No sorting by length. No bin packing. If you have read about LLM serving, where continuous batching and preemption and prefix aware routing are the whole game, this looks like a shortcut. It is not. It is a conclusion, and the argument for it is a cost model you can derive on paper.

The Two Terms of a Forward Pass

A Transformer block does two kinds of work. Count the floating point operations for a sequence of length L in a model of width d.

Dense work. The four attention projections (query, key, value, output) and the feed forward network. Every token passes through every weight matrix independently. With a feed forward width of 4d, one block is roughly 4 · 2Ld² from the projections plus 2 · 2L · d · 4d from the feed forward, which is about 12 L d² operations. It is linear in L.

Attention work. The score matrix Q · K^T is 2L²d operations, and multiplying the softmax output by V is another 2L²d. That is about 4 L² d. It is quadratic in L.

Now take the ratio, and watch almost everything cancel:

attention / dense = 4 L² d / (12 L d²) = L / (3d)

The crossover is L = 3d, and it is far away. BGE-M3 has a width of 1024, so attention only catches up with the dense layers at about 3 × 1024 = 3072 tokens. At the 512 tokens most of these benchmarks use, the ratio is 512 / 3072, so attention is about 17 percent of the dense cost. This is exactly the post's claim: "For small embedding models, at the sequence lengths we serve for, we noticed that the linear cost of dense layers is dominant over the quadratic cost of attention."

And if the cost is dominated by a term that is linear in L, then the cost of a batch is dominated by the sum of the lengths. It does not care how you cut them up. Two sequences of 256 cost the same as one of 512. That is the post's next sentence: "Thus, latency is mostly proportional to the number of tokens, not the number of sequences."

Fitting the Model to the Published Numbers

Do not take the shape on faith. The post publishes three Tulip median latencies for BGE-M3 at batch size one: 1.53 ms at 128 tokens, 2.03 ms at 512 tokens, and 9.57 ms at 4096 tokens. Three measurements, three unknowns. Fit

t(L) = c + a · L + b · L²

where c is everything that does not scale with tokens (the gRPC hop, the queue, the launch, the copy back), a is the linear dense term and b is the quadratic attention term. Solve the three by three system exactly.

LADDER 1 of 3 · RUNG 1: plain Python, no imports

python3
# Rung 1: plain Python. Fit t(L) = c + a*L + b*L*L to the three published
# Tulip p50 latencies for BGE-M3 at batch size one (1.53, 2.03, 9.57 ms).
rows = [[1.0, 128.0, 128.0**2, 1.53],
        [1.0, 512.0, 512.0**2, 2.03],
        [1.0, 4096.0, 4096.0**2, 9.57]]

def solve(m):                       # Gaussian elimination with partial pivoting
    n = len(m)
    for i in range(n):
        p = max(range(i, n), key=lambda r: abs(m[r][i]))
        m[i], m[p] = m[p], m[i]
        for r in range(i + 1, n):
            f = m[r][i] / m[i][i]
            for k in range(i, n + 1):
                m[r][k] -= f * m[i][k]
    x = [0.0] * n
    for i in reversed(range(n)):
        s = sum(m[i][k] * x[k] for k in range(i + 1, n))
        x[i] = (m[i][n] - s) / m[i][i]
    return x

c, a, b = solve([r[:] for r in rows])
print(f"fixed overhead c = {c:.4f} ms")
print(f"linear term   a = {a:.3e} ms per token")
print(f"quadratic     b = {b:.3e} ms per token squared")
print()
print(f"{'tokens':>8} {'linear ms':>10} {'quadratic ms':>13} {'quad share':>11}")
for L in (128, 512, 1024, 4096, 8192):
    lin, quad = a * L, b * L * L
    print(f"{L:>8} {lin:>10.3f} {quad:>13.3f} {quad/(lin+quad):>10.1%}")

output
fixed overhead c = 1.3766 ms
linear term   a = 1.173e-03 ms per token
quadratic     b = 2.020e-07 ms per token squared

  tokens  linear ms  quadratic ms  quad share
     128      0.150         0.003       2.2%
     512      0.600         0.053       8.1%
    1024      1.201         0.212      15.0%
    4096      4.804         3.390      41.4%
    8192      9.607        13.559      58.5%

Three numbers from a published chart just told you the whole architecture of the problem. At 128 tokens the quadratic term is two percent of the token dependent work. At 512 it is eight percent. It does not become the majority until somewhere past 4096 tokens, which sits comfortably in the region the hand model put at 3d.

And look at c. The fit says 1.38 milliseconds of the 1.53 millisecond median request does not depend on the token count at all. That single coefficient is the entire subject of chapters 4 through 8. It is the gRPC hop, the schedule, the kernel launches and the copy back. Shrinking c is what "Tulip is faster" means.

RUNG 2: NumPy, and the question the scheduler asks

python3
# Rung 2: NumPy. Same fit, then the question the scheduler actually asks:
# for a fixed token budget, does it matter how the tokens are cut into sequences?
import numpy as np

L = np.array([128.0, 512.0, 4096.0])
t = np.array([1.53, 2.03, 9.57])
A = np.stack([np.ones_like(L), L, L ** 2], axis=1)
c, a, b = np.linalg.solve(A, t)
print("c, a, b =", np.round([c, a, b], 8))

def batch_ms(lengths):
    lengths = np.asarray(lengths, dtype=float)
    return c + a * lengths.sum() + b * (lengths ** 2).sum()

print()
print(f"{'split of 2048 tokens':>28} {'model ms':>9}")
for n in (1, 2, 4, 8, 16, 32):
    lengths = np.full(n, 2048 / n)
    print(f"{str(n) + ' x ' + str(int(2048/n)):>28} {batch_ms(lengths):>9.3f}")
print()
print("token count is the price; the cut only moves the small quadratic term.")

output
c, a, b = [1.3765745e+00 1.1727800e-03 2.0000000e-07]

        split of 2048 tokens  model ms
                    1 x 2048     4.626
                    2 x 1024     4.202
                     4 x 512     3.990
                     8 x 256     3.884
                    16 x 128     3.831
                     32 x 64     3.805

token count is the price; the cut only moves the small quadratic term.

Cut 2048 tokens thirty two different ways and the predicted time moves by under a millisecond, all of it from the quadratic term getting cheaper as the sequences get shorter. The number of sequences is not a cost driver. The number of tokens is. That is the licence to schedule first come first served.

RUNG 3: torch, and where saturation shows up

python3
# Rung 3: torch, CPU. The same three coefficients as a linear solve, then the
# question the scheduler asks: what does one more token in the batch buy?
import torch

L = torch.tensor([128.0, 512.0, 4096.0])
t = torch.tensor([1.53, 2.03, 9.57])
A = torch.stack([torch.ones_like(L), L, L ** 2], dim=1)
c, a, b = torch.linalg.solve(A, t).tolist()
print(f"c={c:.4f} ms   a={a:.6e} ms/tok   b={b:.6e} ms/tok^2")

tokens = torch.tensor([64., 128., 256., 512., 1024., 2048., 4096.])
ms = c + a * tokens + b * tokens ** 2
per_tok_us = ms / tokens * 1000.0

print()
print(f"{'tokens':>7} {'batch ms':>9} {'us/token':>9} {'saved vs half the size':>23}")
prev = None
for T, m, u in zip(tokens.tolist(), ms.tolist(), per_tok_us.tolist()):
    gain = "" if prev is None else f"{prev - u:>18.2f} us"
    print(f"{int(T):>7} {m:>9.3f} {u:>9.3f} {gain:>23}")
    prev = u
print()
print("Doubling 256 -> 512 buys 2.64 us per token. Doubling 1024 -> 2048 buys 0.47.")
print("That flattening is the saturation the post puts at about 512 tokens.")

output
c=1.3766 ms   a=1.172776e-03 ms/tok   b=2.020461e-07 ms/tok^2

 tokens  batch ms  us/token  saved vs half the size
     64     1.452    22.695                        
    128     1.530    11.953                10.74 us
    256     1.690     6.602                 5.35 us
    512     2.030     3.965                 2.64 us
   1024     2.789     2.724                 1.24 us
   2048     4.626     2.259                 0.47 us
   4096     9.570     2.336                -0.08 us

Doubling 256 -> 512 buys 2.64 us per token. Doubling 1024 -> 2048 buys 0.47.
That flattening is the saturation the post puts at about 512 tokens.

The final column is the whole argument. Every doubling of the batch used to halve the cost per token. By 512 tokens the returns have collapsed to under three microseconds per doubling, and by 4096 the quadratic term has started charging you for the privilege. The post states the conclusion directly: once a batch is large enough to saturate the GPU, "which is around 512 tokens on a model under one billion parameters, packing more sequences into it does not improve efficiency."

Cost Model: Tokens Versus Sequences

The fitted model, live. The bar splits into the fixed cost, the linear dense term and the quadratic attention term. Change the total token budget and the number of sequences you cut it into, and watch which term reacts. The dashed line is the saturation point the post reports.

Tokens 512

Sequences 1

The Scheduler, in Full

Now write it. Requests accumulate while the engine is busy. When the engine frees up, take sequences off the front of the pool until you hit a token budget or a sequence limit. No sorting, no priorities. Roughly forty lines, and it is the whole mechanism.

python3
# Tulip's scheduler, in full. Requests accumulate while the engine is busy;
# when it frees up, sequences are taken first come first served up to a token
# budget. There is no sorting, no priority, no length bucketing by cleverness.
from collections import deque

class Scheduler:
    def __init__(self, token_budget=512, max_seqs=64):
        self.pool = deque()
        self.token_budget, self.max_seqs = token_budget, max_seqs
    def submit(self, req_id, lengths):
        for i, L in enumerate(lengths):
            self.pool.append((req_id, i, L))
    def next_batch(self):
        batch, tokens = [], 0
        while self.pool and len(batch) < self.max_seqs:
            rid, i, L = self.pool[0]
            if batch and tokens + L > self.token_budget:
                break                      # this one waits for the next batch
            self.pool.popleft()
            batch.append((rid, i, L)); tokens += L
        return batch, tokens

s = Scheduler()
s.submit("query", [9])                     # an online query: nine tokens
s.submit("reindex", [512] * 3)             # a bulk indexing chunk
s.submit("rerank", [180] * 4)              # a scoring request

n = 0
while True:
    batch, tokens = s.next_batch()
    if not batch:
        break
    n += 1
    who = ",".join(sorted({b[0] for b in batch}))
    print(f"batch {n}: {len(batch)} sequences, {tokens:>4} tokens, from [{who}]")
print(f"\n{n} forward passes, pool empty: {len(s.pool) == 0}")
print("the nine-token query rode out in batch 1 because it arrived first.")

output
batch 1: 1 sequences,    9 tokens, from [query]
batch 2: 1 sequences,  512 tokens, from [reindex]
batch 3: 1 sequences,  512 tokens, from [reindex]
batch 4: 1 sequences,  512 tokens, from [reindex]
batch 5: 2 sequences,  360 tokens, from [rerank]
batch 6: 2 sequences,  360 tokens, from [rerank]

6 forward passes, pool empty: True
the nine-token query rode out in batch 1 because it arrived first.

Notice what the dumb policy bought. The nine token query was first in the pool, so it went out immediately in a batch of its own rather than waiting behind fifteen hundred tokens of reindexing work. First come first served is not just simple, it is a fairness property: a small request that arrives first never queues behind a large one.

When to reach for it: You are building a reranker service and someone proposes a scheduler that sorts by sequence length to pack batches tightly. The concrete artefact is a design doc with a priority queue in it. Why the cost model helps: if latency is linear in tokens then a tightly packed batch and a loosely packed batch of the same token count cost the same, so the sorting buys you nothing while adding a head of line blocking hazard for short requests. Where the packing instinct fails: it is imported from training, where you pad to a rectangle and padding really is waste. A ragged serving batch has no padding to save.

A batch already holds 512 tokens of a sub-billion parameter embedding model. You add four more sequences of 128 tokens each. What does the cost model predict?

Almost nothing changes, because the GPU is saturated and extra sequences ride along free Cost quadruples, because attention is quadratic in the number of sequences Cost rises roughly in proportion to the 512 extra tokens, because time tracks tokens; what does not improve is efficiency per token, since the GPU was already saturated Cost falls, because larger batches always amortise better

Chapter 4: Launch Overhead: CPU versus GPU

We now know that 1.38 of the 1.53 millisecond median request has nothing to do with token count. This chapter finds the largest single piece of it.

Running a forward pass takes two processors. The post puts it plainly: "The CPU is responsible for scheduling batches and launching kernels with the appropriate parameters, while the GPU executes the relevant matrix multiplication, attention, norm or activation kernels."

What "Launching a Kernel" Actually Means

A kernel is one GPU program: a matrix multiply, a normalisation, an addition. Your Python does not run on the GPU. It runs on the host and, for each kernel, does a surprising amount of work:

1. Python dispatch

Interpret the line, resolve the operator, check dtypes and devices, walk the dispatcher to the right backend implementation.

↓

2. Shape and stride work

Compute output shapes, allocate an output tensor from the caching allocator, choose a tile configuration for the kernel.

↓

3. Driver call

Marshal the arguments into a launch descriptor and hand it to the CUDA driver, which pushes the work onto a stream.

↓

4. Return, immediately

The call does not wait. The GPU may start the kernel while the host is already on the next line. That asynchrony is the only reason any of this works at all.

Steps one to three cost a few microseconds each time, and they cost the same whether the kernel then runs for two microseconds or two milliseconds. For a chat model, where a single matrix multiply may take hundreds of microseconds, that is a rounding error. For an embedding model handling a 128 token query, it is the bill.

The picture in the post. Their diagram of a forward pass without CUDA graphs shows six host invocations across the top, in order: F.rms_norm(), torch.linear(), F.sdpa(), F.rms_norm(), torch.linear(), torch.add(). Underneath, the device row shows the kernels they produce: rms_norm, gemm, flash_attention, rms_norm, gemm, elt_add. Between several of those kernels sit hatched blocks labelled GPU IDLE. The device finished and is waiting for the host to catch up and launch the next one.

Host and Device, Kernel by Kernel

The two timelines from the post, live. Drag the token count to change how long each device kernel runs, and drag the per launch cost to change how long the host takes to issue it. The hatched blocks are GPU idle. Switch to graph mode to replace every launch with a single replay.

Tokens 32

Launch us 5.0 us

The Budget, On Paper

Do this one by hand before running it, because the arithmetic is the entire insight. Take a 24 layer encoder. Each layer issues roughly six kernels: two norms, the attention projection matrix multiply, the attention kernel itself, the feed forward matrix multiply, and a residual add. That is

6 × 24 = 144 kernels per forward pass

At 5 microseconds of host time per launch, the host needs 144 × 5 = 720 microseconds just to issue the work. Now the device side. Suppose the whole stack costs 0.72 microseconds of device time per token, a figure chosen so the arithmetic stays clean. Then:

Batch	Device time	Host launch time	Who waits
1 token	0.7 µs	720 µs	The GPU waits, and is idle 99.9% of the pass
128 tokens	92 µs	720 µs	The GPU waits, idle 87% of the pass
512 tokens	369 µs	720 µs	The GPU waits, idle 49% of the pass
1000 tokens	720 µs	720 µs	Nobody. This is the inflection point.
4096 tokens	2949 µs	720 µs	The host waits. Now you are a real GPU workload.

That number is the post's number. "Across each model, we track an inflection point, determining the minimum number of tokens at which GPU execution is more expensive than CPU-side kernel launch. Because embedding models are small, we observe that this inflection point comes at batches of thousands of tokens and tens of sequences." Our back of the envelope put it at a thousand tokens. Theirs, measured per model, sits in the low thousands. Below that line, the fastest possible GPU changes nothing.

python3
# The launch-overhead budget. A forward pass is N kernels. The host pays a fixed
# cost per launch; the device pays a cost that grows with the token count.
# Below the crossing the GPU waits on Python. Above it, Python waits on the GPU.
N_KERNELS      = 6 * 24        # six kernels per block, twenty four blocks
LAUNCH_US      = 5.0           # host cost of one eager kernel launch
GRAPH_LAUNCH_US = 8.0          # one graph.replay() for the whole pass
DEVICE_US_PER_TOKEN = 0.72     # device time for one token through the stack

host_eager_us = N_KERNELS * LAUNCH_US
print(f"kernels per forward pass : {N_KERNELS}")
print(f"host time, eager launches: {host_eager_us:.0f} us")
print(f"host time, one graph     : {GRAPH_LAUNCH_US:.0f} us")
print()
print(f"{'tokens':>7} {'device us':>10} {'eager us':>9} {'idle share':>11} {'graph us':>9} {'idle share':>11}")
for T in (1, 8, 64, 256, 512, 1024, 4096):
    dev = T * DEVICE_US_PER_TOKEN
    eager_wall = max(dev, host_eager_us)
    eager_idle = max(0.0, host_eager_us - dev) / eager_wall
    graph_wall = max(dev, GRAPH_LAUNCH_US)
    graph_idle = max(0.0, GRAPH_LAUNCH_US - dev) / graph_wall
    print(f"{T:>7} {dev:>10.1f} {eager_wall:>9.1f} {eager_idle:>10.1%} {graph_wall:>9.1f} {graph_idle:>10.1%}")
print()
print(f"inflection point, eager: {host_eager_us / DEVICE_US_PER_TOKEN:.0f} tokens")
print(f"inflection point, graph: {GRAPH_LAUNCH_US / DEVICE_US_PER_TOKEN:.0f} tokens")

output
kernels per forward pass : 144
host time, eager launches: 720 us
host time, one graph     : 8 us

 tokens  device us  eager us  idle share  graph us  idle share
      1        0.7     720.0      99.9%       8.0      91.0%
      8        5.8     720.0      99.2%       8.0      28.0%
     64       46.1     720.0      93.6%      46.1       0.0%
    256      184.3     720.0      74.4%     184.3       0.0%
    512      368.6     720.0      48.8%     368.6       0.0%
   1024      737.3     737.3       0.0%     737.3       0.0%
   4096     2949.1    2949.1       0.0%    2949.1       0.0%

inflection point, eager: 1000 tokens
inflection point, graph: 11 tokens

The two inflection points in that output are the punchline. Launching one kernel at a time, the crossover is a thousand tokens: everything below it is a host bound workload wearing a GPU costume. Replace 144 launches with one graph replay and the crossover falls to eleven tokens. Almost every real request is then GPU bound, which is the only regime where a fast GPU helps.

The Inflection Point

Host cost is a flat line: it does not care how many tokens you send. Device cost is a sloped line through the origin. Where they cross is the inflection point. Change the kernel count and the per launch cost and watch the crossing slide. The shaded region on the left is the regime where the GPU idles.

Kernels 144

Launch us 5.0 us

The Kernel That Would Not Be Captured

There is a sentence in the post that is easy to skim past and worth stopping on: "Some attention implementations rely on dynamic host-side inputs to configure kernel launches, preventing full-model prefill/dense CUDA graphs. We upstreamed changes to relevant kernels to enable them in our inference engine."

Here is what that means. Chapter 5 explains that a captured graph is a frozen recording of kernel launches with their arguments baked in. If a kernel decides its own grid size on the host, by reading a value out of a tensor and branching on it, then the recording cannot be replayed with different data, because the branch might need to go the other way. One such kernel anywhere in the model breaks the whole model graph.

So the fix was not in their code at all. It was in the attention library: move the dynamic decision from host code into the kernel, so the launch configuration becomes static. They then pushed that change upstream. Owning the stack, in this instance, meant sending a patch to somebody else's repository.

When to reach for it: You are on call for an inference service and the dashboard shows GPU utilisation at eight percent while p99 latency triples under load. The concrete artefact is a Nsight Systems trace with a dense picket fence of short kernels and visible white space between them. Why the launch budget helps: count the kernels in one forward pass, multiply by a few microseconds, and compare to the measured device time. If the product is larger, you are host bound and no amount of GPU tuning will move p99. Where utilisation graphs mislead: they average over a window, so a busy host and an idle device looks like a healthy but under fed system rather than a correctable defect.

Your embedding forward pass issues 144 kernels at about 5 microseconds of host time each, and the device work for a 256 token batch measures 184 microseconds. What is the bottleneck, and what would fix it?

The GPU, because 184 microseconds of device work is substantial; buy a faster accelerator The host, because 720 microseconds of launch cost exceeds 184 microseconds of device work; collapse the launches into a single graph replay Memory bandwidth, because BF16 weights dominate the traffic; quantise the model The network, because gRPC serialisation dominates at this size; switch to a binary protocol

Chapter 5: CUDA Graphs, Captured Lazily

The fix for 144 launches is to stop making 144 launches. A CUDA graph is a recording of a sequence of kernel launches, their arguments and their dependencies, replayable with one call to the driver. The post: "instead of launching independent kernels, a CUDA graph can be built to capture the metadata required to launch all the kernels of a forward pass with a single call to the CUDA driver. This eliminates the need to re-run expensive Python and PyTorch code for the configurations CUDA graphs can be captured for."

Read the last clause again. It eliminates the Python. On replay there is no dispatcher, no shape inference, no allocator, no argument marshalling. The driver already holds a compiled description of the whole pass and pushes it onto the stream in one go, so the kernels run back to back with no gaps.

Capture Has Two Hard Rules

Rule 1: shapes are frozen

A graph records a specific launch geometry. A batch of 4 sequences and 512 tokens gets a different graph from 8 sequences and 512 tokens. Nothing about a replay can adapt.

↓

Rule 2: addresses are frozen

The recording holds pointers, not values. Replay reads whatever now lives at the captured input address and writes to the captured output address. You copy new data into the same buffer and replay.

↻ every replay

Rule 2 is the one that surprises people. A graph is not a function you call with arguments. It is a machine wired to specific memory. The idiom is always: write your inputs into the static input buffer, replay, read the static output buffer.

The picture in the post. Their forward pass with CUDA graphs shows one host box, graph.replay(), and a single arrow down to a device row where rms_norm, gemm, flash_attention, rms_norm, gemm and elt_add run touching each other. Compare that to the diagram in chapter 4: same six kernels, six host boxes, and hatched idle blocks in between. Same work, no gaps.

Rule 1 Costs You a Combinatorial Explosion

Frozen shapes mean one graph per configuration, and for embeddings a configuration is a pair: how many sequences, and how many tokens in total. The post: "CUDA graphs must be captured for each distinct configuration, which for embeddings means a graph per sequence count and token count combination. Since this grid is expansive, we pad token counts to buckets that are multiples of 64 or 256."

Bucketing is the standard trade. Round 130 tokens up to 192 and you waste 32 percent of the batch on padding, but you reuse a graph instead of falling back to eager. Count what survives after bucketing.

python3
# How many CUDA graphs is "a graph per sequence count and token count"?
# The post pads token counts to buckets that are multiples of 64 or 256.
MAX_SEQS   = 64
MAX_TOKENS = 8192

def buckets(step_small=64, switch=1024, step_big=256, top=MAX_TOKENS):
    small = list(range(step_small, switch + 1, step_small))
    big   = list(range(switch + step_big, top + 1, step_big))
    return small + big

def pad(n, bs):
    for b in bs:
        if n <= b:
            return b
    return bs[-1]

bs = buckets()
print(f"token buckets   : {len(bs)}  ({bs[0]}, {bs[1]}, ... , {bs[-1]})")
print(f"sequence counts : 1 .. {MAX_SEQS}")
print(f"graphs to capture: {len(bs) * MAX_SEQS}")

EAGER_MS, CAPTURE_MS = 62.0, 41.0
total_s = len(bs) * MAX_SEQS * (EAGER_MS + CAPTURE_MS) / 1000.0
print(f"capture cost    : {total_s:.0f} s  ({total_s/60:.1f} minutes)")
print()
print("padding waste for a few real batches:")
for n in (100, 130, 500, 1100, 3000, 5000):
    p = pad(n, bs)
    print(f"  {n:>5} tokens -> bucket {p:>5}   {100*(p-n)/p:>5.1f}% of the batch is padding")

output
token buckets   : 44  (64, 128, ... , 8192)
sequence counts : 1 .. 64
graphs to capture: 2816
capture cost    : 290 s  (4.8 minutes)

padding waste for a few real batches:
    100 tokens -> bucket   128    21.9% of the batch is padding
    130 tokens -> bucket   192    32.3% of the batch is padding
    500 tokens -> bucket   512     2.3% of the batch is padding
   1100 tokens -> bucket  1280    14.1% of the batch is padding
   3000 tokens -> bucket  3072     2.3% of the batch is padding
   5000 tokens -> bucket  5120     2.3% of the batch is padding

2816 graphs, 4.8 minutes. Those are our numbers from our assumptions, but they land exactly where the post lands: "This still results in thousands of graphs that might take multiple minutes to capture for a typical model." Bucketing already cut an eight thousand column grid down to forty four columns, and the answer is still thousands of graphs.

The Graph Grid

Each cell is one CUDA graph: a sequence count crossed with a token bucket. Change the bucket step and the maximum sequence count and watch the grid and the capture bill move. Coarser buckets mean fewer graphs and more padding waste, and the meter shows you both at once.

Bucket 64

Max seqs 64

Why Capture Is Expensive

The post itemises the bill: "The cost of capture comes from two sources: an eager forward pass that must be executed to compile kernels and set up buffers for various kernels that need them, followed by the capture run which re-executes Python code."

Cost 1: the eager warmup pass

Kernels get compiled or selected for this shape, autotuners pick tile sizes, workspace buffers get allocated. None of this can happen during capture, because capture records launches rather than running them.

↓

Cost 2: the capture run

All the Python runs again, and this time every launch is recorded instead of executed. You pay the full dispatch cost a second time to buy the right never to pay it again.

Lazy Capture: Pay As You Serve

Here is the choice. Capture all 2816 graphs at startup and the process is useless for five minutes. Capture none and every request pays eager dispatch forever. Perplexity does neither. From the post: "We mitigate startup costs by capturing CUDA graphs lazily as the engine serves. We keep track of each configuration and ensure that it goes through an eager warmup run before triggering graph capture and replay on the second hit."

Hit 1: eager

This configuration is new. Run it eagerly, serve the request, and mark the configuration warm. The kernels are now compiled and the buffers exist.

↓

Hit 2: capture, then replay

The configuration is warm, so record the pass into a graph and replay it to produce the answer. This request is the most expensive one this configuration will ever see.

↓

Hit 3 and onward: replay

One driver call. No Python. This is the steady state, and in production it is where essentially all traffic lives.

python3
# Lazy capture as a three-state machine, one state per configuration.
# First hit: eager, and warm the kernels. Second hit: capture, then replay.
# Every hit after that: replay only.
class GraphCache:
    def __init__(self):
        self.state = {}                 # config -> "cold" | "warm" | "captured"
        self.cost = {"eager": 62.0, "capture": 41.0, "replay": 0.4}   # ms
    def run(self, config):
        st = self.state.get(config, "cold")
        if st == "cold":
            self.state[config] = "warm"
            return "eager", self.cost["eager"]
        if st == "warm":
            self.state[config] = "captured"
            return "capture+replay", self.cost["capture"] + self.cost["replay"]
        return "replay", self.cost["replay"]

traffic = [(4, 512), (4, 512), (4, 512), (1, 64), (4, 512), (1, 64), (8, 1024), (1, 64), (4, 512)]
gc, total = GraphCache(), 0.0
for cfg in traffic:
    what, ms = gc.run(cfg)
    total += ms
    print(f"seqs={cfg[0]:<2} tokens={cfg[1]:<5} -> {what:<15} {ms:>6.1f} ms")
print(f"\ntotal {total:.1f} ms over {len(traffic)} requests")
eager_only = len(traffic) * gc.cost["eager"]
print(f"eager every time would cost {eager_only:.1f} ms")
print(f"capture everything up front: {3 * (gc.cost['eager'] + gc.cost['capture']):.1f} ms before serving a single request")

output
seqs=4  tokens=512   -> eager             62.0 ms
seqs=4  tokens=512   -> capture+replay    41.4 ms
seqs=4  tokens=512   -> replay             0.4 ms
seqs=1  tokens=64    -> eager             62.0 ms
seqs=4  tokens=512   -> replay             0.4 ms
seqs=1  tokens=64    -> capture+replay    41.4 ms
seqs=8  tokens=1024  -> eager             62.0 ms
seqs=1  tokens=64    -> replay             0.4 ms
seqs=4  tokens=512   -> replay             0.4 ms

total 270.4 ms over 9 requests
eager every time would cost 558.0 ms
capture everything up front: 309.0 ms before serving a single request

Three configurations appeared in that traffic, so three requests paid eager cost and three paid capture cost, spread across the run rather than stacked in front of it. The steady state requests cost 0.4 milliseconds each.

The cost, stated honestly. The post does not pretend this is free: "Lazy graph capture has an impact on p99 latencies during startup; however, it is valuable in spreading multiple minutes of eager work across multiple hours. Quicker startup times allow us to better scale and manage embedding deployments." You trade a worse tail for the first few minutes of a replica's life against a replica that can join the fleet in seconds. If you autoscale on traffic, that trade is not close.

Warmup, Capture, Replay

Requests arrive with random configurations. Each configuration walks cold to warm to captured. The lower panel is the running latency trace, so you can watch the tall eager and capture spikes thin out as coverage grows. Compare the two startup policies side by side.

Capture and Replay, For Real

This is the second implementation ladder, and it is short because the API is short. On a CUDA machine it builds a real torch.cuda.CUDAGraph. With no GPU present it runs the identical static buffer discipline on the CPU and tells you so, which means you can rehearse the code on a laptop and read the shape of it before you ever touch a GPU.

LADDER 2 of 3 · capture and replay, with a CPU fallback

python3
# Capture and replay a whole forward pass. On CUDA this is a real
# torch.cuda.CUDAGraph. With no GPU the same static-buffer discipline runs on
# the CPU, so you can rehearse the shape of the code anywhere.
import time, torch

torch.manual_seed(0)
dev = "cuda" if torch.cuda.is_available() else "cpu"
print("device:", dev, "| real CUDA graph:", dev == "cuda")

D, T = 256, 512
W1 = torch.randn(D, D, device=dev) / D ** 0.5
W2 = torch.randn(D, D, device=dev) / D ** 0.5

# The two rules of capture: fixed shapes, fixed addresses.
static_in  = torch.zeros(T, D, device=dev)
static_out = torch.zeros(T, D, device=dev)

def forward():
    h = torch.nn.functional.rms_norm(static_in, (D,))
    h = h @ W1
    h = torch.nn.functional.gelu(h)
    h = h @ W2
    static_out.copy_(static_in + h)          # write into the captured buffer

def warmup(n=3):
    for _ in range(n):
        forward()

def timed(fn, n=20):
    t0 = time.perf_counter()
    for _ in range(n):
        fn()
    if dev == "cuda":
        torch.cuda.synchronize()
    return (time.perf_counter() - t0) / n * 1e3

warmup()                                      # the eager pass the post pays for once
if dev == "cuda":
    g = torch.cuda.CUDAGraph()
    with torch.cuda.graph(g):                 # the capture run
        forward()
    replay = g.replay
else:
    replay = forward                          # CPU fallback: same buffers, no graph

static_in.normal_()
eager_ms  = timed(forward)
replay_ms = timed(replay)
print(f"eager  forward : {eager_ms:.3f} ms")
print(f"replay forward : {replay_ms:.3f} ms")

static_in.normal_()                           # new input, same address
replay()
ref = static_out.clone()
forward()
print("replay matches eager:", torch.allclose(ref, static_out, atol=1e-5))

output
device: cpu | real CUDA graph: False
eager  forward : 0.251 ms
replay forward : 0.236 ms
replay matches eager: True

On a CPU the two timings are the same, because there is no graph and replay is just forward. That is the honest result and it is the point of the fallback: the speedup comes from removing driver and dispatch cost, and on a CPU there is no driver to remove. What the CPU run does prove is the discipline: write into static_in, run, read static_out, and the last line confirms that replaying with fresh data in the same buffer gives the same answer as running eagerly.

When to reach for it: You serve a small model at batch size one and your profiler shows dispatch overhead above half the request. The concrete artefact is a captured graph per shape bucket plus a static input buffer per bucket. Why lazy capture matters operationally: a replica that must capture thousands of graphs before it serves cannot be part of an autoscaling group, because scale up arrives minutes after the traffic that caused it. Where eager capture fails: your scale up policy and your warmup policy quietly disagree, and the fleet is always one incident behind the load.

Why does Perplexity capture CUDA graphs lazily rather than all at startup?

Because captured graphs consume GPU memory, and lazy capture keeps only the graphs that traffic actually uses Because a graph captured at startup would use stale weights Because thousands of graphs take multiple minutes to capture, and spreading that eager work across hours of serving lets a replica start quickly, at the cost of a worse p99 during startup Because CUDA graphs cannot be captured until the first real request defines the model shapes

Chapter 6: LazyTensor and Overlap

CUDA graphs made the launch cheap. This chapter makes the wait disappear, and it is the most elegant piece of engineering in the post.

Start from the fact that makes it necessary. From the post: "Through CUDA, GPU work is asynchronous. Since launching a kernel asynchronously enqueues it on a stream, host code must explicitly synchronize to read out the resulting vectors."

Three Primitives You Need First

Stream

An ordered queue of GPU work. Everything you push onto one stream runs in order, one item at a time. Push a kernel and the call returns immediately; the work happens later.

↓

Page-locked (pinned) host memory

Normal host memory can be paged out by the operating system, so the GPU cannot read it without help. Pinned memory is nailed to a physical address, which is what makes a truly asynchronous copy from device to host possible.

↓

Event

A marker you push onto a stream. It fires when everything queued ahead of it has finished. You can poll it without blocking, or block on it when you finally need the answer.

Now the object. From the post: "The LazyTensor tracks a host buffer in page-locked memory and a cudaMemcpyAsync operation via an event copying data from the device. It is kicked off after the launch of the forward pass on the same stream."

The one sentence that makes it work. "Since the copy operation must wait for all prior kernels on the stream to execute, the associated event tracks both the completion of the forward pass and the availability of the result on the CPU." One event, two meanings. Because the stream is ordered and the copy was queued behind the forward pass, an event placed after the copy cannot fire until the forward pass is done and the bytes have landed in host memory. You do not need to track the model separately from the transfer. Ordering did it for you.

What Gets Queued, in Order

Step	Who does it	Blocks?	What it leaves behind
1. Write token ids into the static input buffer	Host	No	The captured graph's input address now holds this batch
2. `graph.launch()`	Host, one driver call	No	144 kernels queued on the stream
3. `cudaMemcpyAsync` device to pinned host	Host, one driver call	No	A copy queued behind the kernels on the same stream
4. Record an event	Host, one driver call	No	A marker that fires when the copy completes
5. Return a LazyTensor holding the buffer and the event	Host	No	A handle the Rust task can await
6. `LazyTensor::synchronize()`	Host	Yes, and only here	Embeddings, already in host memory

The post's execution profile diagram makes this visual. Along the host row: graph.launch(), then LazyTensor::new(), then a gap marked with an ellipsis where other work happens, then LazyTensor::synchronize(). Along the device row: a long block labelled CUDA graph, then a block labelled cudaMemcpyAsync, then a thin coloured tick for the event. An arrow points up from that tick to the synchronize box, labelled BLOCKS ONLY HERE.

LADDER 3 of 3 · the LazyTensor itself, with a CPU fallback

python3
# A LazyTensor: a page-locked host buffer, an async copy off the device, and an
# event that says "the copy is done". Nothing blocks until you ask for a value.
import torch

dev  = "cuda" if torch.cuda.is_available() else "cpu"
real = dev == "cuda"
print("device:", dev, "| pinned buffer + CUDA event:", real)

class LazyTensor:
    """Tracks a result that is still on the accelerator."""
    def __init__(self, src):
        self.host = torch.empty_like(src, device="cpu", pin_memory=real)
        if real:
            self.host.copy_(src, non_blocking=True)   # cudaMemcpyAsync on the stream
            self.event = torch.cuda.Event()
            self.event.record()                       # fires when the copy lands
        else:
            self.host.copy_(src)
            self.event = None
        self.done = False

    def ready(self):
        return True if self.event is None else self.event.query()

    def synchronize(self):                            # the ONLY blocking call
        if self.event is not None:
            self.event.synchronize()
        self.done = True
        return self.host

D = 128
w = torch.randn(D, D, device=dev) / D ** 0.5

def step(tokens):                                     # ROSE step(): no wait
    return LazyTensor(torch.nn.functional.normalize(tokens @ w, dim=-1))

batch_a = torch.randn(4, D, device=dev)
batch_b = torch.randn(4, D, device=dev)

lt_a = step(batch_a)                # launch batch N
print("after launch A, A ready?", lt_a.ready())
lt_b = step(batch_b)                # launch batch N+1 without waiting for A
print("after launch B, host has done zero blocking waits")

out_a = lt_a.synchronize()          # block here, and only here
out_b = lt_b.synchronize()
print("A shape", tuple(out_a.shape), "row norm", round(out_a[0].norm().item(), 4))
print("B shape", tuple(out_b.shape), "row norm", round(out_b[0].norm().item(), 4))
print("both batches collected:", lt_a.done and lt_b.done)

output
device: cpu | pinned buffer + CUDA event: False
after launch A, A ready? True
after launch B, host has done zero blocking waits
A shape (4, 128) row norm 1.0
B shape (4, 128) row norm 1.0
both batches collected: True

Two things carry over from the CPU run. The structure is real: the result is a handle, the handle owns its own host buffer, and there is exactly one method that blocks. And the shape check confirms the contract, four rows of 128 numbers each with unit norm, which is what an embedding endpoint is supposed to hand back. On a GPU the same code would report ready() as false right after launch and the two launches would genuinely overlap.

Now Use It: Overlap

The post: "We leverage LazyTensors in our ROSE encoder engine to overlap GPU and CPU work. Instead of each step() call running the CUDA graph and waiting for it to finish, step() returns a LazyTensor to asynchronously track its result."

Put that beside the sentence from the CUDA graphs section and the design closes: "Since CUDA graphs minimize the CPU-side overheads, once a graph is launched, we have free time to kick off and enqueue the execution of the next batch whenever it is available."

So the loop becomes: launch batch N, get a LazyTensor, and instead of waiting, go build batch N plus one from the pool and launch that too. The post's overlapping diagram shows a device row with FWD BATCH N directly touching FWD BATCH N+1, and between them, in red, NO IDLE GAP. Underneath, the host row shows Prepare and Launch twice, the two cudaMemcpyAsync blocks each ending in an event, and a single LazyTensor::synchronize() that feeds Collect results.

python3
# What overlapping buys. Tulip's scheduler prepares batch N+1 on the host while
# batch N is still on the device. Same work, two orderings.
import asyncio, time

PREPARE_MS, DEVICE_MS, COLLECT_MS, N_BATCHES = 3.0, 8.0, 1.5, 6

async def spin(ms):                    # burn wall clock without sleeping the loop
    end = time.perf_counter() + ms / 1000.0
    while time.perf_counter() < end:
        await asyncio.sleep(0)

async def device(ms, lock):            # one GPU, one stream: strictly serial
    async with lock:
        await spin(ms)

async def blocking():
    lock = asyncio.Lock()
    t0 = time.perf_counter()
    for _ in range(N_BATCHES):
        await spin(PREPARE_MS)
        await device(DEVICE_MS, lock)  # host stands still here
        await spin(COLLECT_MS)
    return (time.perf_counter() - t0) * 1000

async def overlapped():
    lock = asyncio.Lock()
    t0 = time.perf_counter()
    async def one():
        await spin(PREPARE_MS)
        await device(DEVICE_MS, lock)
        await spin(COLLECT_MS)
    await asyncio.gather(*[one() for _ in range(N_BATCHES)])
    return (time.perf_counter() - t0) * 1000

b = asyncio.run(blocking())
o = asyncio.run(overlapped())
floor = N_BATCHES * DEVICE_MS
print(f"batches                 : {N_BATCHES}")
print(f"device floor            : {floor:.1f} ms")
print(f"blocking step()         : {b:.1f} ms   ({b/floor:.2f}x the floor)")
print(f"LazyTensor + overlap    : {o:.1f} ms   ({o/floor:.2f}x the floor)")
print(f"host work hidden        : {b - o:.1f} ms")

output
batches                 : 6
device floor            : 48.0 ms
blocking step()         : 75.1 ms   (1.56x the floor)
LazyTensor + overlap    : 52.7 ms   (1.10x the floor)
host work hidden        : 22.4 ms

The device floor is the honest lower bound: one GPU, one stream, six batches of eight milliseconds each, so forty eight milliseconds no matter what you do. Blocking on every step spends over half again as long. Returning a handle and preparing the next batch during the wait gets within about ten percent of the floor. Wall clock numbers wobble by a millisecond or two between runs; the ratio does not.

Showcase: CPU and GPU Overlap, Live

The post's overlapping diagram, running. The upper lane is the device stream: forward passes, then the copy back, then the event tick. The lower lane is the host: prepare, launch, and the one blocking synchronize. Turn overlap off and watch the idle gaps open up between forward passes. The readout tracks device idle time and effective throughput.

Prepare 3.0 ms

Forward 8 ms

When to reach for it: You maintain a Python inference loop that calls .cpu() or .item() on the result of every forward pass. The concrete artefact is a profile in which the host thread sits inside cudaStreamSynchronize for a third of the run. Why the handle pattern helps: the synchronize is not paying for compute, it is paying for ordering, and if you defer it you can spend that time building the next batch. Where the eager habit fails: a single .item() for a log line, in the middle of a loop, serialises the entire pipeline and is invisible in code review.

In the LazyTensor design, where does the host actually block, and why is one event enough to cover both the forward pass and the copy?

It blocks at graph launch; the event is recorded before the kernels so it covers everything after it It blocks only at synchronize; because the copy is queued on the same stream behind the forward pass, an event after the copy cannot fire until both are complete It blocks at every kernel launch, but each block is only a few microseconds It never blocks; the pinned buffer is coherent, so the host can read it at any time

Chapter 7: One Engine for LLMs and Embeddings

Chapter 1 argued from arithmetic that an LLM engine ought to be able to serve embedding models. This chapter is the receipt. The post: "We adapted our ROSE engine, which we originally built for LLM serving, to also handle the execution of embedding models. To minimize the effort needed to support embedding models, ROSE aggressively reuses code between LLMs and embeddings. For instance, pplx-embed serving and Qwen3.5 LLM decoding all go through the same kernels."

Not similar kernels. The same kernels. Which raises the interesting question: what is actually different?

The Dense Layers: Nothing Is Different

The post disposes of most of the model in one sentence: "For dense layers, embedding and LLM inference are identical since token vectors are processed independently."

That independence is the whole reason. A linear layer applied to a token vector never looks at any other token. Stack a million token vectors into one tall matrix, multiply, and every row comes out correct regardless of which sequence it belonged to or what task the model is doing. Norms, activations, residual adds, all the same. Roughly nine tenths of the parameters in a Transformer sit in layers that literally cannot tell the difference.

Attention: Three Real Differences

Attention is the only operator that mixes tokens, so it is the only place a difference can hide. There are three, and the post names all of them.

1. No KV cache

"When serving an embedding model, we do not instantiate a KV cache." An LLM keeps every key and value forever because token 900 will attend to token 3. An embedding pass reads the sequence once, pools it, and throws everything away.

↓

2. Ragged inputs, not padded

"In attention layers, differences are handled by adding support for ragged inputs, alongside the paged prefill and decode setups required by LLMs." One flat token axis plus a table of offsets, so sequences of different lengths sit end to end with no padding at all.

↓

3. Ragged attention kernel variants

"...and dispatch to variations of attention kernels which support the ragged format to avoid padding." The kernel reads the offset table and confines each sequence's attention to its own slice.

And the boring parts are shared too. "The supporting conversion and calibration routines are also shared with the LLMs." Weight conversion, quantisation calibration, the tooling nobody writes a blog post about. That is where the engineering time usually goes, and reusing it is most of why this integration was cheap.

Why Ragged Beats Padded

Real embedding traffic is a title, a paragraph and a whole page arriving together. Pad that batch into a rectangle and you pay for the longest row, everywhere.

The ragged layout instead concatenates every sequence into one flat token axis and carries a cumulative offsets array, conventionally called cu_seqlens. Sequence i lives in rows cu[i] through cu[i+1]. The attention kernel reads those bounds and never looks outside them, so there is no mask to apply and no padding to skip.

python3
# Padded batch versus ragged batch. Real embedding traffic is a mix of a title,
# a paragraph and a whole page, and padding charges you for the longest one.
import numpy as np

rng = np.random.default_rng(7)
lengths = np.sort(rng.integers(12, 512, size=12))[::-1]
print("sequence lengths:", lengths.tolist())

n, longest, total = len(lengths), int(lengths.max()), int(lengths.sum())
padded = n * longest
print()
print(f"ragged cells (cu_seqlens layout): {total}")
print(f"padded cells ({n} x {longest})      : {padded}")
print(f"wasted                           : {padded - total}  ({(padded-total)/padded:.1%})")

# the ragged layout itself: one flat token axis plus cumulative offsets
cu = np.concatenate([[0], np.cumsum(lengths)])
print()
print("cu_seqlens:", cu.tolist())
print("sequence 3 occupies rows", cu[3], "to", cu[4])
print("attention for sequence i is confined to rows cu[i]:cu[i+1] with no mask")
print()
print("no KV cache: an embedding pass reads every token once and keeps nothing.")
print(f"KV bytes an LLM would hold for these tokens (24 layers, 8 kv heads, dim 128, bf16):")
print(f"  {2*24*8*128*2*total/1e6:.2f} MB")
print("  embeddings keep 0.00 MB")

output
sequence lengths: [484, 460, 448, 428, 399, 354, 324, 301, 162, 154, 124, 39]

ragged cells (cu_seqlens layout): 3677
padded cells (12 x 484)      : 5808
wasted                           : 2131  (36.7%)

cu_seqlens: [0, 484, 944, 1392, 1820, 2219, 2573, 2897, 3198, 3360, 3514, 3638, 3677]
sequence 3 occupies rows 1392 to 1820
attention for sequence i is confined to rows cu[i]:cu[i+1] with no mask

no KV cache: an embedding pass reads every token once and keeps nothing.
KV bytes an LLM would hold for these tokens (24 layers, 8 kv heads, dim 128, bf16):
  361.46 MB
  embeddings keep 0.00 MB

Thirty seven percent of that padded rectangle is nothing. Every one of those cells costs a matrix multiply row, memory traffic and attention work, and produces a vector that gets discarded. The KV figure at the end is the second saving: an LLM would be holding three hundred and sixty megabytes of keys and values for those same tokens, and an embedding pass holds none of it.

Padded Rectangle Versus Ragged Strip

The same batch drawn both ways. Above, the padded rectangle: real tokens in colour, padding in grey, and every grey cell is work you pay for and throw away. Below, the ragged strip with its offsets table. Change the length spread to see how much padding costs when traffic is mixed.

Spread 70%

Sequences 9

What Sharing Actually Buys

Component	LLM serving	Embedding serving	Shared?
Dense projections and feed forward	Same weights, same kernels	Same weights, same kernels	Yes, completely
Norms, activations, residual adds	Per token, independent	Per token, independent	Yes, completely
Attention layout	Paged prefill and decode	Ragged, no padding	Kernel family shared, variant differs
KV cache	Allocated, paged, reused across steps	Not instantiated at all	No, and that is a saving
Sampling and detokenization	Present	Absent; pooling and normalisation instead	No
Weight conversion and calibration	Shared tooling	Shared tooling	Yes
CUDA graph management	Per decode shape	Per sequence count and token bucket	Mechanism shared, keying differs

The payoff the post claims from that table is a prototyping payoff: "This sharing allows us to easily serve an embedding model that was originally fine-tuned from an LLM for prototyping, evaluation, and production inference." Fine tune an LLM into an embedding model on Monday and it is servable on Tuesday, on the production path, with production kernels, because there is only one path.

When to reach for it: A research team wants to try three embedding model candidates, one of which is a fine tune of your existing chat model. The concrete artefact is a request for an evaluation harness that can serve all three at production speed. Why one engine helps: the fine tune already runs on your kernels, so the only new code is the pooling head and the ragged dispatch, and the evaluation runs on the same path production will use. Where a separate research server fails: the candidate wins in evaluation, then regresses in production because the research server padded batches and the production server does not, and nobody can reproduce the difference.

Serving an embedding model on an LLM engine, what genuinely changes in the attention layer?

The softmax is replaced by a normalisation, since there is no next token to predict Attention becomes causal instead of bidirectional The projections change shape because embeddings have a different hidden size No KV cache is instantiated, and the engine dispatches to ragged variants of the attention kernels so that mixed-length sequences need no padding

Chapter 8: Ivy: Chunk, Balance, Tokenize

Every optimisation so far lives inside one replica. This chapter is about the layer above, and it fixes a problem that no amount of single replica tuning can touch.

The post: "Ivy, our inference HTTP proxy layer, also plays an important role in performance. Because request payloads vary in production, routing individual requests to individual replicas can cause load imbalance."

Why Whole Request Routing Fails

Think about what varies. One caller sends a nine token query. Another sends a hundred documents of 512 tokens each. A third sends eight documents of 4096. The token counts differ by four orders of magnitude, and latency is proportional to tokens, so the work differs by four orders of magnitude too.

Now round robin those requests over four replicas. Replica one draws the hundred document job and is busy for a long time. Replica four draws the nine token query and finishes instantly, then idles. Round robin balances request counts, and request counts are not the thing that costs money.

The consequence for a queue. A short request that lands behind a large one waits for the whole large one. That is head of line blocking, and it lands squarely on p99 rather than p50, because it only happens to the unlucky. This is exactly the shape of the concurrency benchmark in chapter 10, where the baseline's p99 climbs much faster than its median.

The Fix: Split, Then Spread

The post: "Ivy splits large-batch requests into chunks and load-balances them between replicas, improving utilization and smoothing latency."

The architecture diagram shows it happening on a two string request: the first input goes to one Tulip replica as [3630, 288, 525, 2518], the second goes to a different Tulip replica as [17823, 574, 1382, 38835]. Two sentences, two replicas, one response.

Two design details make this work. Chunks are cut on a token budget, not a sequence count, because chapter 3 established that tokens are the price. And chunks go to the least loaded replica rather than round robin, which is what turns a split into a balance.

python3
# Ivy's job in one function: split a large-batch request into chunks and spread
# the chunks over replicas, instead of pinning one whole request to one replica.
# Latency is proportional to tokens, so chunks are cut on a TOKEN budget.
def whole_request_routing(reqs, replicas):
    load = [0] * replicas
    for i, r in enumerate(reqs):
        load[i % replicas] += sum(r)              # round robin over whole requests
    return load

def chunk_by_tokens(seqs, budget):
    chunk, used = [], 0
    for L in seqs:
        if chunk and used + L > budget:
            yield chunk, used
            chunk, used = [], 0
        chunk.append(L); used += L
    if chunk:
        yield chunk, used

def chunked_routing(reqs, replicas, budget=2048):
    load, n = [0] * replicas, 0
    for r in reqs:
        for _, tokens in chunk_by_tokens(r, budget):
            j = min(range(replicas), key=lambda x: load[x])   # least loaded first
            load[j] += tokens; n += 1
    return load, n

requests = [[512] * 100, [128] * 4, [512] * 100, [96] * 2, [4096] * 8, [64] * 3]
R = 4
a = whole_request_routing(requests, R)
b, n_chunks = chunked_routing(requests, R)

def report(name, load):
    print(f"{name:<22} {str(load):<40} makespan {max(load):>6}  spread {max(load)-min(load):>6}")

print(f"requests : {[len(r) for r in requests]} sequences, {sum(sum(r) for r in requests)} tokens")
print(f"replicas : {R}")
print()
report("whole requests", a)
report(f"chunks of <=2048 ({n_chunks})", b)
print()
print(f"perfect split would be {sum(sum(r) for r in requests)//R} tokens per replica")
print(f"tail replica shrinks by {max(a)/max(b):.2f}x")

output
requests : [100, 4, 100, 2, 8, 3] sequences, 136064 tokens
replicas : 4

whole requests         [83968, 704, 51200, 192]                 makespan  83968  spread  83776
chunks of <=2048 (61)  [34816, 33280, 34816, 33152]             makespan  34816  spread   1664

perfect split would be 34016 tokens per replica
tail replica shrinks by 2.41x

Read the spread column. Routing whole requests leaves one replica holding 83,968 tokens while another holds 192, a spread of eighty three thousand. Chunking on a token budget brings the spread down to 1,664 and puts every replica within three percent of the ideal 34,016. The makespan, which is what the slowest caller experiences, falls by a factor of 2.41.

Chunk and Balance Across Replicas

Requests arrive with wildly different token counts. Watch them land on four replicas two ways: whole requests round robin, or chunks on a token budget to the least loaded replica. The bars are per replica token load, the dashed line is the perfect split, and the readout is the makespan the slowest caller sees.

Budget 2048

Replicas 4

The Chunk Budget Is a Real Trade

Making chunks smaller improves balance and hurts everything else. Each chunk becomes its own gRPC message, its own scheduling decision and its own response to reassemble, and if the chunk falls under the saturation point from chapter 3 you also stop using the GPU efficiently.

Chunk budget	Balance	Per chunk overhead	GPU efficiency
Very small, well under 512 tokens	Excellent	Many messages, many schedules	Poor: below the saturation point
Around one to four thousand tokens	Good	Modest	Good: comfortably saturated
Very large, or no chunking	Poor: one replica takes the whole job	Minimal	Good, on one replica, while the others idle

The Tokenizer, Which Is Also on the Critical Path

One more sentence, easy to miss: "Our recent work on in-house unigram tokenization, fully rolled out in Ivy, drastically improves latencies over off-the-shelf tokenizers."

Remember where tokenization sits. For a 128 token query whose Tulip median is 1.53 milliseconds, tokenization runs before the request is even sent downstream. A tokenizer that takes a millisecond is not a detail, it is most of the budget. This is why the high concurrency benchmark in chapter 10 is the interesting one: it is the only figure that includes tokenization through Ivy and the network hop between Ivy and Tulip, which is to say it is the only figure that measures what a caller actually experiences.

When to reach for it: You run an embedding gateway and your p50 is healthy while your p99 is four times worse under load. The concrete artefact is a per replica queue depth graph where one line is always high and the others are always low. Why chunking helps: the imbalance is caused by request size variance, not request rate, so splitting large requests onto a token budget and routing to the least loaded replica attacks the actual cause. Where adding replicas fails: more replicas do not reduce variance, so the unlucky short request still waits behind the same large job and p99 barely moves while your bill does.

Why does splitting large-batch requests into chunks smooth latency, when it adds messages and scheduling work?

Because smaller batches always run faster on the GPU per token Because production payloads vary hugely in token count, so routing whole requests loads replicas unevenly; chunks on a token budget spread the real work and stop short requests from queueing behind large ones Because each chunk gets its own CUDA graph, which avoids a capture Because chunking lets the tokenizer run in parallel with the forward pass

Chapter 9: But the Kernels Still Matter

The post spent eight sections arguing that the GPU side has converged, and then titled its next section "...but the Kernels Still Matter." Both things are true, and the chart underneath is the most interesting piece of data in the whole article.

The setup: "ROSE supports a variety of attention backends. Different kernels may be suited to specific problem sizes. Over time, we integrated FlashInfer 2, FlashInfer 3 and FlashAttention 4 kernels to implement ragged attention."

A labelling note, stated once and then dropped. The chart's legend reads "FA4 faster" and "FA3 faster", while the prose names three kernels: FlashInfer 2, FlashInfer 3 and FlashAttention 4. The prose says FlashInfer 3 outperforms FlashAttention 4 on Qwen-based models at very long sequence lengths, and the chart's FA3 wins are concentrated exactly there, on pplx-embed-4b at 8192. So the chart's FA3 is the kernel the text calls FlashInfer 3, and FA4 is FlashAttention 4. From here we use the chart's own labels. The chart also notes that ties within 0.1 percent go to FA4.

Two Models, Two Attention Shapes

The chart benchmarks two models, and the difference between them is the point of the whole exercise.

	BGE-M3	pplx-embed-4b
Attention type	MHA, multi head attention	GQA, grouped query attention
Head dimension	64	128
Peak throughput in the chart	655k tok/s, at batch 24 and length 512	94k tok/s, at batch 24 and length 128
Where FA3 wins	Short and medium lengths at large batches	Long sequences, and every batch size at 8192

In multi head attention every query head has its own key and value heads. In grouped query attention several query heads share one key and value head, which cuts memory traffic and changes the kernel's blocking problem entirely. Add a head dimension of 128 instead of 64 and the tile shapes a kernel wants are different again. The post says as much: "Since performance and tuning can vary with the number and dimension of attention heads, we maintain support for multiple configurations and make a case-by-case decision when serving."

Winner Per Cell, and by How Much

The published chart, every cell. Rows are batch size, columns are sequence length, each cell shows the winning backend, its margin over the slower one, and the throughput in tokens per second. Tap any cell to pin its numbers. Switch models to see the pattern flip at 8192.

Reading the Chart Row by Row

On BGE-M3, at 64, 128 and 512 tokens, FA3 takes over as the batch grows: it wins every cell at batches 16 and 24 for those three lengths, with margins from 1.23 percent up to 2.17 percent. At 4096 and 8192 the pattern reverses completely and FA4 wins all ten cells, with the largest margins on the chart, up to 3.37 percent.

On pplx-embed-4b the story is nearly the mirror image. FA4 owns the short lengths. At 8192, FA3 wins every batch size, from 0.57 percent up to 1.16 percent. That column is the chart's version of the prose sentence about Qwen-based models at very long sequence lengths.

Look at the size of these margins. The largest number anywhere on the chart is 3.37 percent. Twelve of the twenty five cells for pplx-embed-4b are decided by less than half a percent, and one cell is a dead tie at 0.00 percent. This is what "converged to a largely optimal implementation" looks like when you zoom in: two independently developed state of the art kernels, trading cells, never separated by more than a few percent. Compare that to the three times end to end latency win from fixing the host side.

python3
# The case-by-case decision, as a lookup table. Every entry is a cell of the
# published chart: winner, margin over the slower backend, throughput in tok/s.
# The chart's FA3 is the kernel the prose calls FlashInfer 3.
SEQ    = [64, 128, 512, 4096, 8192]
BATCH  = [1, 4, 8, 16, 24]

CHART = {
 "BGE-M3": {                      # MHA, head dim 64
   1:  [("FA4",0.00,68),  ("FA4",0.51,130), ("FA4",2.09,345), ("FA4",1.71,433), ("FA4",3.37,338)],
   4:  [("FA4",0.17,212), ("FA4",1.71,350), ("FA4",0.40,550), ("FA4",1.13,473), ("FA4",2.44,367)],
   8:  [("FA4",1.09,348), ("FA4",0.14,477), ("FA3",0.97,612), ("FA4",1.38,478), ("FA4",2.96,371)],
   16: [("FA3",1.30,459), ("FA3",1.23,584), ("FA3",1.65,648), ("FA4",1.60,487), ("FA4",2.87,374)],
   24: [("FA3",2.17,521), ("FA3",1.66,607), ("FA3",2.06,655), ("FA4",0.98,491), ("FA4",3.17,376)],
 },
 "pplx-embed-4b": {               # GQA, head dim 128
   1:  [("FA4",1.79,20), ("FA4",1.96,37), ("FA4",1.47,75), ("FA4",0.12,72), ("FA3",1.01,59)],
   4:  [("FA4",2.20,59), ("FA4",1.50,77), ("FA4",0.22,82), ("FA4",0.05,73), ("FA3",1.33,60)],
   8:  [("FA4",3.21,77), ("FA4",0.29,81), ("FA4",0.22,90), ("FA3",0.43,74), ("FA3",1.16,61)],
   16: [("FA4",2.74,81), ("FA4",0.14,85), ("FA4",0.20,90), ("FA4",0.07,75), ("FA3",1.02,61)],
   24: [("FA4",2.52,85), ("FA3",0.45,94), ("FA4",0.42,92), ("FA3",0.24,74), ("FA3",0.57,60)],
 },
}

def pick(model, batch, seqlen):
    b = min(BATCH, key=lambda x: abs(x - batch))
    s = min(range(len(SEQ)), key=lambda i: abs(SEQ[i] - seqlen))
    return CHART[model][b][s]

for model in CHART:
    cells = [c for row in CHART[model].values() for c in row]
    fa3 = [c for c in cells if c[0] == "FA3"]
    tight = [c for c in cells if c[1] < 0.5]
    print(f"{model:<15} FA4 wins {len(cells)-len(fa3):>2}/25   FA3 wins {len(fa3):>2}/25   "
          f"margin under 0.5% in {len(tight):>2}/25 cells")

print()
print("FA3 wins per model, by sequence length:")
for model in CHART:
    row = []
    for i, s in enumerate(SEQ):
        col = [CHART[model][b][i][0] for b in BATCH]
        row.append(f"{s}:{col.count('FA3')}/5")
    print(f"  {model:<15} " + "  ".join(row))

print()
for q in [("pplx-embed-4b", 1, 8192), ("pplx-embed-4b", 8, 512), ("BGE-M3", 24, 512), ("BGE-M3", 1, 4096)]:
    w, m, t = pick(*q)
    print(f"serve {q[0]:<14} batch {q[1]:>2} len {q[2]:>5} -> {w}  (+{m:.2f}%, {t}k tok/s)")

output
BGE-M3          FA4 wins 18/25   FA3 wins  7/25   margin under 0.5% in  4/25 cells
pplx-embed-4b   FA4 wins 17/25   FA3 wins  8/25   margin under 0.5% in 12/25 cells

FA3 wins per model, by sequence length:
  BGE-M3          64:2/5  128:2/5  512:3/5  4096:0/5  8192:0/5
  pplx-embed-4b   64:0/5  128:1/5  512:0/5  4096:2/5  8192:5/5

serve pplx-embed-4b  batch  1 len  8192 -> FA3  (+1.01%, 59k tok/s)
serve pplx-embed-4b  batch  8 len   512 -> FA4  (+0.22%, 90k tok/s)
serve BGE-M3         batch 24 len   512 -> FA3  (+2.06%, 655k tok/s)
serve BGE-M3         batch  1 len  4096 -> FA4  (+1.71%, 433k tok/s)

The middle block of that output is the evidence for the post's sentence, read straight off the chart. For pplx-embed-4b, FA3 wins zero out of five cells at length 64 and five out of five at length 8192. For BGE-M3 the 8192 column goes the other way, zero out of five. The claim is specifically about Qwen-based models at very long lengths, and the chart says exactly that and nothing more.

What "maintain support for multiple configurations" costs. A lookup table like the one above is cheap. Keeping three attention backends alive is not: three sets of build flags, three upgrade paths, three sets of numerical behaviour to validate, and a benchmark sweep to re-run whenever any of them releases. Perplexity concluded that a few percent at the tail of their traffic was worth it. For most teams the correct read of this chart is the opposite one: pick the backend that wins most of your cells and move on.

The post says FlashAttention 4 is generally faster, with one exception. What is the exception, and roughly how large are the margins across the chart?

FlashInfer 3 wins at batch size 1, and margins reach fifty percent FlashInfer 2 wins on multi head attention, and margins are around ten percent FlashInfer 3 outperforms it on Qwen-based models at very long sequence lengths, and no margin anywhere on the chart exceeds 3.37 percent FlashInfer 3 wins whenever the head dimension is 64, and margins are around five percent

Chapter 10: Benchmarks, Read Honestly

Everything so far has been mechanism. This chapter is measurement, and the only way to read a vendor benchmark usefully is to look hardest at the cells where the vendor does not win.

The Setup, Which Matters As Much As the Numbers

Element	What the post says
Baseline	vLLM v0.22.0
Precision	BF16
Weights and inputs	Actual model weights, inputs derived from evaluation datasets
Correctness gate	All timing runs were preceded by warmup runs which verified that the divergence in cosine similarity is within 0.1 percent
Hardware	1× H200, noted on every chart
Models	BGE-M3 and pplx-embed-1-0.6b

The correctness gate deserves a moment. It is trivially easy to win an inference benchmark by computing something slightly different: a lower precision accumulation, a fused approximation, a shorter sequence after a silent truncation. Checking that the embeddings from both engines agree to within 0.1 percent cosine divergence before timing anything is the difference between a benchmark and a claim. When you build your own comparison, copy this step first.

Chart One: Low Latency Embeddings

Pre-tokenized request batch size 1, fully sequential requests, ten thousand requests, sequence lengths of 128, 512 and 4096 tokens. This is the online embedding pattern in its purest form, and it is where the host side work dominates most.

Model	Length	Tulip p50	Tulip p99	Baseline p50	Baseline p99
BGE-M3	128	1.53	1.67	4.60	7.96
BGE-M3	512	2.03	2.17	4.65	6.06
BGE-M3	4096	9.57	10.56	11.71	13.20
pplx-embed-1-0.6b	128	1.88	2.26	4.94	7.94
pplx-embed-1-0.6b	512	2.75	3.09	4.98	5.98
pplx-embed-1-0.6b	4096	15.41	16.32	16.15	17.54

Read the first column against the third and the trend is unmistakable. At 128 tokens on BGE-M3 the ratio is 4.60 over 1.53, about 3.0 times. At 512 it is 2.3 times. At 4096 it is 1.22 times. The advantage decays as the sequence grows, exactly as a host bound story predicts.

And here is the cell where the story reverses. On pplx-embed-1-0.6b at 4096 tokens, Tulip's p99 is 16.32 ms while the baseline's median is 16.15 ms. A slow Tulip request at that size is slower than a typical baseline request. The medians are 15.41 against 16.15, a 1.05 times advantage that is close to nothing. At 4096 tokens on a 0.6 billion parameter model you are firmly past the inflection point from chapter 4: the GPU is the bottleneck, both engines are running good kernels, and there is nothing left for a better harness to win.

Chart Two: Low Latency Scoring

Pre-tokenized request batch sizes 5, 25 and 50, sequence length 512, ten thousand sequential requests. This is the middle workload: several documents at once, and somebody waiting.

Model	Request batch	Tulip p50	Tulip p99	Baseline p50	Baseline p99
BGE-M3	5	6.17	6.48	9.60	11.55
BGE-M3	25	22.43	23.54	26.59	30.66
BGE-M3	50	40.45	43.77	47.29	51.04
pplx-embed-1-0.6b	5	9.51	10.21	13.93	24.24
pplx-embed-1-0.6b	25	34.34	36.29	35.04	40.41
pplx-embed-1-0.6b	50	60.96	64.51	63.68	69.06

The same decay. At batch 5 on BGE-M3 the median advantage is 1.56 times. By batch 50 it is 1.17 times. On pplx-embed-1-0.6b at batch 25 the medians are 34.34 against 35.04, a 1.02 times advantage that is inside the noise of any honest reading, and Tulip's p99 of 36.29 is above the baseline's median of 35.04 there too.

One cell is worth calling out for the opposite reason. At batch 5 on pplx-embed-1-0.6b the baseline p99 is 24.24 milliseconds against a median of 13.93, a spread of 1.7 times. Tulip's p99 of 10.21 sits 7 percent above its median of 9.51. A tighter distribution is a different kind of win from a lower median, and for anything with a latency budget it is often the more valuable one.

Chart Three: High Throughput Embeddings

Request batch size 100, four concurrent processes submitting requests, sequence lengths of 512, 1024 and 4096. This is bulk indexing, measured in embeddings per second, and it is the least flattering chart in the post.

Model	Length	Tulip emb/s	Baseline emb/s	Ratio
BGE-M3	512	1499.3	1342.2	1.12×
BGE-M3	1024	691.5	622.5	1.11×
BGE-M3	4096	125.4	116.6	1.08×
pplx-embed-1-0.6b	512	926.7	902.3	1.03×
pplx-embed-1-0.6b	1024	422.8	423.3	1.00×
pplx-embed-1-0.6b	4096	72.2	73.5	0.98×

Two of these six cells do not go Perplexity's way. At 1024 tokens on pplx-embed-1-0.6b the numbers are 422.8 against 423.3, so the baseline is ahead by 0.1 percent, which is a tie. At 4096 the baseline wins outright, 73.5 against 72.2. The post publishes both. That is the correct behaviour from a benchmark, and it is also the strongest evidence for the thesis: with a batch of 100 sequences of 4096 tokens, the GPU has four hundred thousand tokens to chew on and the host has nothing left to optimise. Both engines are measuring the same H200.

Chart Four: High Concurrency

Sequence length 512, batch size 1, but with 1, 2, 4, 8 and 16 concurrent requests. And one crucial difference from every other chart, in the post's own words: "This benchmark also includes tokenization costs through Ivy, alongside the networking overhead between Ivy and Tulip."

That makes it the only end to end figure in the article. It is also an unlabelled line chart, so the values below are read off the plot and are approximate. We mark them as such and we do not compute precise ratios from them.

Model	Concurrency	Tulip p50	Tulip p99	Baseline p50	Baseline p99
BGE-M3	1	about 3	about 3.5	about 12.5	about 15
BGE-M3	16	about 16.5	about 19	about 23	about 32
pplx-embed-1-0.6b	1	about 3.5 to 4	about 4	about 11.5	about 12.5
pplx-embed-1-0.6b	16	about 25	about 35.5	about 32	about 50

Three things are visible in that chart. At one concurrent request the gap is at its widest, which is the tokenizer and the gateway showing up in the measurement. Both baselines dip at two concurrent requests before climbing, a shape that usually means a queue or a batching timer finding a better duty cycle at low load. And at sixteen concurrent requests the baseline's p99 pulls far away from its own median, while Tulip's stays closer to its own, which is the tail behaviour you would expect from smoothing load across replicas.

Showcase: The Benchmark Explorer

All four published figures, with the exact values. Pick a chart and a model. Bars are drawn to scale, cells that tie or reverse are marked, and the readout gives the ratio for the selected group. The concurrency chart is drawn as lines and labelled as read from the chart, approximately.

Every Ratio, In One Table

python3
# Every published Tulip-versus-baseline number, and the ratio each one implies.
# Latencies in ms (lower is better); throughput in embeddings per second.
LAT = {   # (figure, model, x): (tulip p50, tulip p99, base p50, base p99)
 ("embed", "BGE-M3",           128):  (1.53, 1.67, 4.60, 7.96),
 ("embed", "BGE-M3",           512):  (2.03, 2.17, 4.65, 6.06),
 ("embed", "BGE-M3",          4096):  (9.57, 10.56, 11.71, 13.20),
 ("embed", "pplx-embed-1-0.6b",128):  (1.88, 2.26, 4.94, 7.94),
 ("embed", "pplx-embed-1-0.6b",512):  (2.75, 3.09, 4.98, 5.98),
 ("embed", "pplx-embed-1-0.6b",4096): (15.41, 16.32, 16.15, 17.54),
 ("score", "BGE-M3",             5):  (6.17, 6.48, 9.60, 11.55),
 ("score", "BGE-M3",            25):  (22.43, 23.54, 26.59, 30.66),
 ("score", "BGE-M3",            50):  (40.45, 43.77, 47.29, 51.04),
 ("score", "pplx-embed-1-0.6b",  5):  (9.51, 10.21, 13.93, 24.24),
 ("score", "pplx-embed-1-0.6b", 25):  (34.34, 36.29, 35.04, 40.41),
 ("score", "pplx-embed-1-0.6b", 50):  (60.96, 64.51, 63.68, 69.06),
}
THR = {   # (model, seqlen): (tulip emb/s, baseline emb/s)
 ("BGE-M3",            512): (1499.3, 1342.2),
 ("BGE-M3",           1024): (691.5, 622.5),
 ("BGE-M3",           4096): (125.4, 116.6),
 ("pplx-embed-1-0.6b", 512): (926.7, 902.3),
 ("pplx-embed-1-0.6b",1024): (422.8, 423.3),
 ("pplx-embed-1-0.6b",4096): (72.2, 73.5),
}

print("LATENCY  (baseline p50 / Tulip p50, and Tulip p99 against baseline p50)")
print(f"{'bench':>6} {'model':>18} {'x':>5} {'p50 ratio':>10} {'p99 vs base p50':>16}")
for (fig, model, x), (t50, t99, b50, b99) in LAT.items():
    flag = "  <-- Tulip p99 above the baseline median" if t99 > b50 else ""
    print(f"{fig:>6} {model:>18} {x:>5} {b50/t50:>9.2f}x {b50/t99:>15.2f}x{flag}")

print()
print("THROUGHPUT  (Tulip / baseline; below 1.00 means the baseline wins)")
print(f"{'model':>18} {'seqlen':>7} {'tulip':>9} {'baseline':>9} {'ratio':>7}")
wins = ties = losses = 0
for (model, s), (t, b) in THR.items():
    r = t / b
    wins += r > 1.01; ties += 0.99 <= r <= 1.01; losses += r < 0.99
    mark = "" if r >= 1.0 else "   <-- baseline ahead"
    print(f"{model:>18} {s:>7} {t:>9.1f} {b:>9.1f} {r:>6.2f}x{mark}")
print()
print(f"throughput cells: {wins} clear wins, {ties} inside one percent, {losses} losses")
print("the gains live where the host is the bottleneck: short sequences, batch one.")

output
LATENCY  (baseline p50 / Tulip p50, and Tulip p99 against baseline p50)
 bench              model     x  p50 ratio  p99 vs base p50
 embed             BGE-M3   128      3.01x            2.75x
 embed             BGE-M3   512      2.29x            2.14x
 embed             BGE-M3  4096      1.22x            1.11x
 embed  pplx-embed-1-0.6b   128      2.63x            2.19x
 embed  pplx-embed-1-0.6b   512      1.81x            1.61x
 embed  pplx-embed-1-0.6b  4096      1.05x            0.99x  <-- Tulip p99 above the baseline median
 score             BGE-M3     5      1.56x            1.48x
 score             BGE-M3    25      1.19x            1.13x
 score             BGE-M3    50      1.17x            1.08x
 score  pplx-embed-1-0.6b     5      1.46x            1.36x
 score  pplx-embed-1-0.6b    25      1.02x            0.97x  <-- Tulip p99 above the baseline median
 score  pplx-embed-1-0.6b    50      1.04x            0.99x  <-- Tulip p99 above the baseline median

THROUGHPUT  (Tulip / baseline; below 1.00 means the baseline wins)
             model  seqlen     tulip  baseline   ratio
            BGE-M3     512    1499.3    1342.2   1.12x
            BGE-M3    1024     691.5     622.5   1.11x
            BGE-M3    4096     125.4     116.6   1.08x
 pplx-embed-1-0.6b     512     926.7     902.3   1.03x
 pplx-embed-1-0.6b    1024     422.8     423.3   1.00x   <-- baseline ahead
 pplx-embed-1-0.6b    4096      72.2      73.5   0.98x   <-- baseline ahead

throughput cells: 4 clear wins, 1 inside one percent, 1 losses
the gains live where the host is the bottleneck: short sequences, batch one.

The whole lesson, in one output. Three latency cells carry a Tulip p99 above the baseline's median, and all three are the largest configuration in their chart. Of six throughput cells, four are clear wins, one is a tie and one is a loss, and the loss is the longest sequence. The claim this data supports is not "Tulip is faster than vLLM". It is "removing host side overhead is worth up to three times at small sizes and converges to nothing as the GPU becomes the bottleneck", which is a much more useful thing to know, because it tells you whether your own workload is in the regime where any of this helps.

On pplx-embed-1-0.6b at sequence length 4096, indexing throughput is 72.2 embeddings per second for Tulip and 73.5 for the baseline. What is the right conclusion?

The benchmark is flawed, since Tulip is faster in every other figure At that size the workload is GPU bound, both engines run comparable kernels, and there is no host side overhead left to remove; the harness wins live at small sizes and vanish here CUDA graphs hurt throughput at long sequences and should be disabled above 2048 tokens The baseline is using a better attention kernel, so Tulip should switch backends

Chapter 11: Owning the Stack

Eleven chapters ago the puzzle was: if the GPU side has converged, where did three times come from? Here is the assembled answer, and then a checklist you can carry into your own system.

The Whole Stack, and What Each Layer Removed

Every technique in the lesson, placed on the layer it lives in and connected to the cost it attacks. The right hand column is the request timeline: watch the segments each technique removes disappear as they switch on one at a time.

The Five Wins, in Order of Size

1. Whole-model CUDA graphs

Replaces roughly 144 kernel launches with one driver call, and deletes the Python dispatch on every replay. Moves the CPU and GPU inflection point from around a thousand tokens down to a handful.

↓

2. Lazy graph capture

Turns minutes of startup capture into work spread across hours of serving, at the cost of a worse p99 during startup. Makes replicas cheap to start, which is what makes autoscaling real.

↓

3. LazyTensor and overlap

Returns a handle rather than a value, so the host can prepare and launch the next batch while the current one runs. The device stops idling between batches.

↓

4. Ivy chunking and balancing

Splits large-batch requests on a token budget and spreads them across replicas, so short requests stop queueing behind long ones. Attacks the tail, not the median.

↓

5. Kernel selection per configuration

Three attention backends, chosen case by case. Worth up to about three percent, and only in specific cells. Last on the list on purpose.

The Checklist

One table to keep. Every row is a technique from this lesson, the cost it attacks, the signal that says you need it, and what it charges you.

Technique	Bottleneck attacked	Reach for it when	What it costs
Whole-model CUDA graphs	Per-kernel launch and Python dispatch on the host	Kernel count times per-launch cost exceeds measured device time; a trace shows gaps between short kernels	Static shapes and addresses, a graph per configuration, and any kernel with host-side dynamic launch config must be fixed first
Token-count bucketing	The combinatorial explosion of graph configurations	You need graphs but the shape grid is thousands of columns wide	Padding waste, up to about a third on an unlucky batch with fine buckets and more with coarse ones
Lazy capture	Minutes of startup before a replica can serve	You autoscale, redeploy often, or run many model variants per box	A worse p99 during the first minutes of a replica's life
LazyTensor handle plus overlap	Host idling inside a synchronize while the device works	The host thread spends a large fraction of the run blocked on stream synchronisation	Pinned host buffers per in-flight batch, and a real async story in the calling layer
First come first served scheduling	Complexity, and head of line blocking for small requests	Latency is proportional to tokens rather than sequences, which holds while sequence length stays well under three times the model width	No packing optimisation, which is fine because there was nothing to pack
Ragged attention, no KV cache	Padding waste and cache memory that embeddings never read	Mixed length batches, which is every real embedding workload	A second attention kernel variant to maintain alongside the paged one
Chunk and load balance at the gateway	Replica imbalance caused by payload size variance	p99 is several times p50 while per replica queue depths are lopsided	More messages, more scheduling, and reassembly; chunks below the saturation point waste GPU
Fast in-house tokenization at the edge	CPU time before the request is even dispatched	Tokenization is a measurable fraction of a single digit millisecond budget	A tokenizer to own and keep byte-identical with training
Multiple attention backends	The last few percent of GPU time	You have already done everything above and your traffic concentrates in cells where a specific backend wins	Three build and upgrade paths, three sets of numerics to validate, and a sweep to re-run on every release

The Diagnostic, in Order

If you inherit an embedding service and want to know which of the rows above applies, run these five checks. They take an afternoon and they will tell you whether any of this lesson is worth your time.

1. Which pattern is this?

Measure the token distribution of real traffic. Bulk indexing, scoring or online queries behave completely differently, and the same fix helps one and does nothing for another.

↓

2. Is the device even busy?

Compare measured device time for one forward pass against the wall clock of the request. If device time is a small fraction, you are host bound and the GPU is not the lever.

↓

3. Count the kernels.

Kernels per pass times a few microseconds is your launch bill. Compare it to device time. If launches win, CUDA graphs are the single largest available change.

↓

4. Where does the host block?

Find every synchronize, every .cpu() and every .item() on the request path. Each one is a place the host stopped being able to prepare the next batch.

↓

5. Is p99 a tail or a queue?

If p99 is several times p50 while per replica load is uneven, the fix is at the gateway, not in the engine. Chunk large requests and route to the least loaded replica.

What Perplexity Says They Got, and What Comes Next

The post's own summary: "The serving infrastructure composed of Ivy, Tulip and ROSE allows us to serve embeddings for Perplexity with lower latency and better throughput, resulting in more accurate search at a reduced cost compared to off-the-shelf solutions."

On the language question, which is the part most teams will argue about: "mixing highly re-usable and performant Rust primitives alongside more generic Python modeling code. Many open-source inference engines, such as vLLM, SGLang, and TokenSpeed, are integrating languages like Rust and C++ into their stack. We've invested in Rust over the past two years and have reaped great rewards in both performance and maintainability."

And the roadmap they name: tweaking the custom gRPC protocols between Ivy and Tulip to reduce network latencies, improving computational throughput in ROSE, and, as support grows across the ecosystem, using free-threaded Python to improve Python and Rust interoperability and reduce overheads. Every one of those is a host side item. Nobody is promising a faster matrix multiply.

When to reach for the whole framework: You are writing the design doc for a new embedding or reranking service and the default plan is to deploy an off-the-shelf engine behind a load balancer. The concrete artefact is a capacity model with a p99 target in it. Why this framework helps: it lets you predict, before writing code, which regime your traffic sits in, and therefore whether the default plan will hit the target or miss it by three times. Where the default plan fails: it is tuned for chat model traffic, where host overhead is genuinely negligible, and your workload is the one case where it is not.

Where This Connects

Lesson	What it adds to this one
Vector Embeddings	What the model produces and why the geometry works. The prerequisite for everything here.
Vector Databases	Where those vectors go afterwards: HNSW, IVF, product quantisation. The consumer of the throughput this lesson buys.
Embedding Ops	Drift, migrations and version compatibility. The reason a reindex job exists at all, and therefore the reason batch throughput matters.
Transformers	The forward pass whose kernels we spent this lesson launching.
RAG	The pipeline where the query path latency in chapter 0 is actually spent.
ML Inference Engineer	The wider serving discipline: batching policy, quantisation, capacity planning.

Sources

Perplexity Engineering. "Fast Embeddings on GPUs." Perplexity Blog, Research, 4 September 2026. perplexity.ai/hub/blog/fast-embeddings-on-gpus. Every quotation, figure and benchmark number in this lesson comes from that post and its charts.
Perplexity Engineering. "pplx-embed: State-of-the-Art Embedding Models for Web-Scale Retrieval." Referenced by the post for the models benchmarked here.
Perplexity Engineering. "Improving Unigram Tokenizer CPU Performance." Referenced by the post for the in-house tokenizer rolled out in Ivy.
Perplexity Engineering. "Architecting and Evaluating an AI-First Search API." Referenced by the post for the surrounding search platform.
vLLM. The baseline engine, version 0.22.0. github.com/vllm-project/vllm
FlashInfer. github.com/flashinfer-ai/flashinfer
FlashAttention. github.com/Dao-AILab/flash-attention

The closing thought. The most quotable sentence in the post is the one that concedes: embedding inference on the GPU side has converged to a largely optimal implementation across various inference engines. Everything that follows is an argument that the interesting engineering moved somewhere else. It moved into the runtime that launches the kernels, the abstraction that tracks the result, the scheduler that forms the batch and the gateway that decides where the batch goes. The model is not the system. The harness around the model is the system.