LLMRouter: One Question Per Query

Chapter 1: Routing Is a Decision Process

Six families of routers, six vocabularies. A cascade paper talks about acceptance thresholds and escalation. A contrastive paper talks about query–model matching and temperature. An agentic paper talks about trajectories, rewards, and policy gradients. They sound like different problems.

The paper's first move is to show they are one problem, written six ways. Here is the reduction, built from nothing.

Step 1: what does the router know?

Call the thing the router looks at its state. Write it s_t, with a subscript t because — as we will see in a moment — there may be more than one decision per query. The paper defines it as a triple:

Three parts, each earning its place:

Most routers ignore two of the three. That is the point of writing all three down: which portion of the state a router reads is exactly what defines its family. Read only q and you are single-turn. Read q and h_t and you are multi-turn. Read u as well and you are personalized. There is no other axis.

Step 2: what can the router do?

The candidate pool is a finite set:

and an action is drawn from that set plus one extra symbol:

Two kinds of action, and the difference is everything:

Dispatch. a_t = m sends the current state to candidate m, gets a response y_t back, and appends it to the history:

where ⊕ is just concatenation — the new response is added to what we already know. The episode continues.

Terminate. a_t = ⊥ (read "bottom", the standard symbol for a halting or undefined action) ends the episode and aggregates everything collected in h into the final answer y.

Step 3: what is the router trying to do?

The sequence of actions is the trajectory, τ = (a₁, …, a_T). A policy π is whatever rule produces those actions from states. The goal, Equation 1 of the paper:

Take it apart symbol by symbol, because every single one is doing work.

perf( y | q ) is the quality of the final answer y given the query q. It is deliberately not one metric: it "aggregates task-specific quality metrics" — exact match on a geometry answer, multiple-choice accuracy on MMLU, token-level F1 on a memory question, an execution-based pass rate on MBPP code, a persona-conditioned judge's verdict on an open-ended preference prompt. The formulation does not care which; it cares that a number comes out.

c( τ ) is the cost of the whole trajectory, not of one call. It "sums the monetary or token cost of every call in the routing trajectory." For a single-turn router that is one call. For a cascade that drafts with a small model and escalates, it is both calls. For a multi-turn router it includes the decomposition call, every sub-query call, and the aggregation call. This is the accounting rule that makes the comparison fair, and it is why multi-turn routers cannot hide their overhead.

λ ≥ 0 "controls the performance–cost trade-off." It is a scalar you choose, not a parameter you learn, and it encodes a value judgement that no amount of data can supply: how many dollars is one point of quality worth to you?

E_{q, τ ~ π} — the expectation — runs over two sources of randomness: which query arrives, and which trajectory the policy produces (a policy may sample rather than take the arg max). You are optimising average behaviour over your traffic, not worst case.

λ has units, and the units are the whole game

Here is where most readings of an objective like this go soft, so let us be brutally concrete. perf is in points (0 to 100). c is in dollars. You cannot subtract dollars from points. So λ must carry units of points per dollar to make the expression dimensionally sane.

Which means λ is answerable. Not by theory — by your finance team. Let us find the exact value where the decision flips, using the two numbers from Chapter 0.

Worked example: the break-even λ. Two policies. Largest-LLM scores 38.72 and, on our 200-in / 150-out query, costs the 671B blended tier. Smallest-LLM scores 37.94 at the 7B tier, whose blended price in the paper's sheet is $0.20 per 1M tokens.

Costs first. At $1.25 blended, 350 tokens costs (350 ÷ 1,000,000) × 1.25 = $0.0004375. At $0.20 blended, (350 ÷ 1,000,000) × 0.20 = $0.00007.

The objective for each policy is score − λ × cost. They are equal when:

Collect terms. Move the scores to one side and the λ terms to the other:

Divide: λ = 0.78 ÷ 0.0003675 = 2,122 points per dollar.

Now invert it, because the reciprocal is the number a human can actually judge: 1 ÷ 2,122 = $0.00047 per point. That is what one point of average benchmark score is costing you, per query, when you upgrade from the small tier to the large tier.

At a million queries a day, one point of average score costs 1,000,000 × $0.00047 = $471 a day, which is $171,900 a year.

Why arg max is the wrong decision rule (and when it is right)

Suppose your router has produced a score for every candidate. The obvious thing is to take the highest one. The paper is precise about when that is justified: greedy arg max "is optimal for Eq. 1 only when λ = 0."

Derive it. If λ = 0, the objective is just perf, so the best action is whichever candidate maximises predicted quality, and arg max over quality scores is exactly that. But the moment λ > 0, the quantity you want to maximise is not the score — it is score − λ × cost, which is a different ranking, because the cost term reorders it.

Worked example: how arg max fails. Three candidates. Predicted quality scores from the scorer g, and real input/output prices from Table 9. Note that we price input and output separately — gpt-oss-120b charges $0.15 in but $0.60 out, so its cost depends on the shape of the query, not just its length. For our 200-in / 150-out question:

Greedy arg max picks cogito: 74.0 is the largest number. Now set λ = 20,000 points per dollar — equivalently, you are willing to pay $0.00005 for one point — and recompute the objective for each:

gemma: 62.0 − 20,000 × 0.000035 = 62.0 − 0.70 = 61.30
gpt-oss-120b: 71.0 − 20,000 × 0.00012 = 71.0 − 2.40 = 68.60
cogito: 74.0 − 20,000 × 0.0004375 = 74.0 − 8.75 = 65.25

The winner flipped to gpt-oss-120b — neither the cheapest nor the strongest. Push λ to 40,000 and recompute: gemma 62.0 − 1.40 = 60.60; gpt-oss 71.0 − 4.80 = 66.20; cogito 74.0 − 17.50 = 56.50. Still gpt-oss, now with cogito in last place. Where does gemma finally take over? Set the two objectives equal: 62 − λ(0.000035) = 71 − λ(0.00012), so 9 = λ(0.000085), so λ = 105,882. Above that budget pressure, the 9B model wins.

Three different winners from one unchanged scorer. Nothing about the model's beliefs changed; only λ did. That is why the paper insists on separating the scoring function g from the decision rule d: g says what it believes about quality, d converts belief plus budget into an action. Confusing them is the most common structural mistake in routing systems.

Three things to notice while you drag. First, the ordering changes several times, not once — there is no single "cost-aware ranking", there is a ranking per budget. Second, the most expensive candidate is the first to fall, and it falls fast, because the penalty is linear in cost and its cost is the largest. Third, and most important: switch to a three-call trajectory and every bar drops, but the expensive ones drop three times as far. That is the arithmetic behind finding (iii). A multi-turn router does not merely have to be better — it has to be better by enough to pay for two extra calls, and c(τ) makes it pay.

What the formulation buys you, stated plainly

Two things, both practical.

It makes routers comparable. Once every router is a policy over the same state space, action space, and objective, "which is better?" becomes a well-posed question with a procedure attached: fix q, fix ℳ, fix perf, fix λ, run both, compare. Every difference you then measure is attributable to the policy, because everything else was held constant. That was structurally impossible before.

It makes routers composable. If the only difference between families is which slice of s_t the encoder reads, then personalizing a router is not a redesign — it is swapping E_q for a user-conditioned one and inheriting everything else. The paper says this explicitly: "any router is personalized by swapping in a user-conditioned E_q while inheriting the remaining components." Chapter 2 makes that surgical.

Working a full trajectory, with real prices

The abstraction only earns its keep if you can trace an actual episode through it. Here is a three-step multi-turn route, priced.

t = 1. State s₁ = (q, ∅, ∅). The query is "Compare the population growth of Japan and Nigeria since 1990 and explain the divergence." The router's decision rule emits a₁ = the 3B scaffolding model, asked to decompose. It returns two sub-queries. Tokens: roughly 120 in, 60 out. This call is not free and is charged to c(τ).

t = 2. State s₂ = (q, ∅, h₂) where h₂ = h₁ ⊕ y₁ — the decomposition is now part of the state. The router dispatches sub-query one to a 70B model: say 150 in, 400 out. At llama-3.3-70b's $0.88 in and $0.88 out that is (150 × 0.88 + 400 × 0.88) ÷ 10⁶ = (132 + 352) ÷ 10⁶ = $0.000484.

t = 3. Sub-query two to the same model, another $0.000484. Now h₄ holds the decomposition and both answers.

t = 4. a₄ = ⊥. The episode terminates and the responses are aggregated — which is itself a call to the 3B model, with a long input (both sub-answers) and a moderate output. Say 500 in, 250 out.

So c(τ) is four calls, not one. Even ignoring the two scaffolding calls' cost, the two specialist calls alone total $0.000968, against $0.0004375 for a single dispatch to cogito. The multi-turn route has already spent 2.2× the most expensive single-turn option, before we count the decomposition or the aggregation.

Single-turn routing is a bandit, not an MDP

Worth being precise about, because it changes which algorithms are appropriate.

A Markov decision process has states that transition: your action changes the world, and the next state depends on it. That is what makes credit assignment hard and what motivates value functions and temporal-difference learning.

In single-turn routing there is no transition. You observe q, take one action, receive a reward, and the episode ends. The next query is drawn independently. That is a contextual bandit — the special case of an MDP with horizon one — and it is a strictly easier problem. There is no discounting, no bootstrapping, no value function; the optimal policy is simply arg max over actions of the expected reward given the context.

Which explains something otherwise puzzling about the router zoo: most of the methods are not reinforcement learning at all. They are supervised regressors and classifiers and nearest-neighbour tables, because with horizon one, and with the reward for every action observable offline from the query–model matrix, the bandit problem degenerates further into ordinary supervised learning. You do not even have an exploration problem, because you were not restricted to observing the arm you pulled.

The paper acknowledges all three rows: it lists methods that "sample for exploration in online settings" and methods that "directly optimize trajectory-level rewards with reinforcement learning". Most of its seventeen live in row one, and that is not a limitation, it is the correct response to the structure of the problem.

The expectation over q is a claim about your traffic

One more symbol that is easy to skim past. The objective is an expectation over q, which means it optimises for the distribution of queries you actually receive. Two consequences that bite in practice.

First, a router is fitted to a traffic mix. If 60% of your traffic is code and the benchmark's generic mix is 13.8% code (516 of 3,729 test items across MBPP and HumanEval), a router trained on the benchmark has learned a prior that does not match you. Its scoring function will be systematically miscalibrated toward knowledge QA.

Second, traffic mix drifts, and the objective silently changes when it does. Nothing in the router notices. The scores it emits are unchanged; the expectation they were fitted to has moved. This is the same failure that afflicts every deployed recommender, and the same remedy applies: measure the mix, and refit when it moves.

Chapter 9 shows the strongest version of this: routers trained on benchmark queries and then asked to route multi-agent node prompts — planning instructions, verification instructions — do not merely degrade, they reorder. The expectation moved a long way.

λ is a Lagrange multiplier, and that changes how you set it

Equation 1 subtracts a cost term from a quality term, which looks like an arbitrary modelling choice. It is not. It is the standard way to convert a constrained problem into an unconstrained one, and knowing that gives you a much better handle on what λ means.

The problem you actually have is almost never "maximise quality minus lambda times cost." It is:

Maximise quality, spend no more than a budget B. That is a constrained optimisation. The standard tool is the Lagrangian: introduce a multiplier λ ≥ 0 and optimise

The constant λB does not depend on the policy, so maximising over π is the same as maximising E[perf] − λE[c] — which is exactly Equation 1.

So λ is the shadow price of the budget: the number of quality points you would gain if the budget were raised by one dollar. That interpretation is far more useful than "a trade-off weight", for three reasons.

One: it tells you which direction to move λ. If your policy comes in under budget, λ is too high — you are being more frugal than you had to be. If it comes in over, raise λ. Sweeping λ monotonically sweeps expected cost, so a simple bisection finds the λ that hits any target budget.

Two: it tells you when raising the budget is worth it. If at your operating point λ = 2,000 points per dollar, then an extra dollar of budget buys 2,000 points of expected quality across the whole query distribution. Compare that against what a dollar buys elsewhere in your system and you have an actual capital-allocation argument.

Three: it explains why the sweep in Chapter 8 is the right experiment. Sweeping λ (or β) traces the entire family of budget-constrained optima in one pass, which is exactly the frontier a deployment needs. You do not choose λ; you choose a budget, and λ is what falls out.

A units problem inside perf

One more honest wrinkle before we leave the objective. perf "aggregates task-specific quality metrics" — exact match, multiple-choice accuracy, token-level F1, execution pass rate, a judge's verdict. All are mapped to a 0–1 or 0–100 scale, which makes them addable. It does not make them comparable.

An F1 of 0.26 on a short-phrase memory answer and an accuracy of 0.70 on a four-option question are not two measurements of the same underlying quantity in different places. One has a high floor (partial token overlap is nearly free) and a low ceiling (exact phrasing is hard); the other has a floor at 0.25 from guessing and a ceiling at 1.

Add them and divide by seven, and you have a number. Chapter 7 will show precisely what that number does and does not resolve. For now, hold the caution: perf is an aggregation of incommensurable metrics, by necessity, and the aggregation is a modelling choice that nobody has a principled answer for.

The optimal policy, if you had a perfect predictor

It clarifies what every router is approximating to write down what it would do with perfect knowledge.

Suppose an oracle hands you, for query q, the true expected quality of every candidate: perf(q, m) for all eighteen. And you know every cost c(q, m) exactly, since cost is a deterministic function of token counts and prices. Then the optimal single-turn action is simply:

That is it. No learning, no search, no exploration. The entire difficulty of routing is that perf(q, m) is unobservable before you pay for it, and that the only way to learn its shape is to have paid for it many times in the past.

Which locates every router in the zoo precisely: each one is an estimator of perf(q, m), and the five components are the estimator's architecture. E_q and E_m decide what the estimate is a function of; g is the functional form; ℒ is how it is fitted; and d is the only slot that is not part of the estimator at all — it is where the known quantity, cost, enters.

Why the expectation is over q as well as τ

Two sources of randomness were listed earlier; here is why each is written down.

Randomness in τ given q exists only if d samples rather than taking an arg max. For most single-turn routers the policy is deterministic and this expectation collapses. It matters for online routers that explore, and for multi-turn routers whose decision rule is written a_t ~ d(·) — note the tilde in Table 1's multi-turn row, which is a sampling symbol, against the equals sign in the other two rows. The paper is being precise, not decorative.

Randomness in q is the one that never collapses, and it is a claim about your deployment rather than about your algorithm. A policy that is optimal for your query distribution is not optimal for someone else's. This is why the paper's four findings are all comparative statements about this benchmark's distribution, and why Chapter 9's out-of-distribution studies reorder everything.

A sanity check on the whole formulation

Before moving on, verify that the formulation reproduces the things you already believe. Three special cases:

All four fall out without special-casing. A formulation that requires an exception to express the methods it claims to unify is not unifying anything; this one does not.

Two common misreadings of Equation 1

Misreading one: "λ is a hyperparameter to tune." It is not. A hyperparameter is tuned against a validation objective; λ is the objective, or rather it defines which objective you are optimising. Tuning λ to maximise quality would drive it to zero, which is the answer to a different question. Set it from your budget, not from a sweep.

Misreading two: "the minus sign means cost is a penalty." It means cost is the other objective, converted into quality units. The distinction matters when you extend the framework: a penalty suggests you would rather it were absent, whereas an objective suggests there is a legitimate frontier along it. If cost were merely a penalty, the right λ would be as small as you can afford; because it is an objective, there are budgets at which the cheap policy is genuinely correct, not merely tolerable.

The vector-cost extension, sketched

Since Chapter 3 flagged latency as the missing axis, here is what including it would look like — it is a two-line change to the formulation and a much larger change to the experiment.

where c(τ) is now a vector — dollars, seconds, maybe carbon — and λ is a vector of exchange rates, one per cost dimension. The dot product collapses it back to a scalar, so everything downstream is unchanged: g still scores, d still decides, the matrix now has more layers.

What breaks is not the mathematics but the measurement. Latency is not a deterministic function of token counts the way price is; it depends on the provider's queue depth at the moment of the call, which is not reproducible and cannot be replayed from a stored matrix. A latency-aware benchmark would need distributions per candidate, measured over time, and would stop being a lookup table. That is the real reason this paper's cost is money.

Why terminating is an action, not a stopping condition

A small piece of notation with a real design consequence. The action space is ℳ ∪ {⊥}, so "stop" is an element of the same set as "dispatch to model 7." It could instead have been written as a separate binary decision, or as a fixed horizon. Making it an action does three things.

It makes stopping learnable by the same machinery. The scorer emits nineteen numbers instead of eighteen, and d takes an arg max over all of them. No separate stop-classifier, no separate loss.

It makes stopping costly to get wrong in a measurable way. Terminating too early yields a poor answer; terminating too late costs another call. Both show up in the same objective, so the trade-off is priced rather than heuristic.

It identifies the missing capability precisely. Chapter 7's diagnosis of the multi-turn collapse — the need for "better sufficiency estimation [and] early stopping" — is, in this notation, the observation that routers are bad at scoring ⊥. The two heuristic multi-round baselines do not score it at all; they decompose unconditionally. Router-R1, which does learn when to emit it, is 2.5× better than they are on the generic track.

So ⊥ is not bookkeeping. It is the slot where the field's most-identified open problem lives, and writing it as an action is what makes that visible.

What is being learned, one sentence per component

Before Chapter 2 opens the components up, it helps to know what each is for in terms of the objective we just built.

Read the third and fourth rows together once more. All the uncertainty is in g; all the policy is in d. That is the cleanest way to hold the whole architecture, and it is why the two are worth keeping separate in code as well as in exposition.

Chapter 2: Five Components

Chapter 1 gave us a shared objective. That alone does not let you build anything — "maximise expected quality minus lambda times cost" is a wish, not an architecture. This chapter is the architecture, and it is smaller than you expect: five slots. Fill them and you have a router. Fill them differently and you have a different router. There is no sixth slot.

Read that left to right as a pipeline. The first two turn things into representations. The third compares those representations. The fourth turns the comparison into an action. The fifth is how all of it gets fitted. Now each one, with its tensor shapes.

Component 1: the context encoder E_q

The context encoder maps the routing state s_t to whatever representation the routing decision will be based on. The paper identifies exactly two output forms, and the split is not cosmetic.

Embedding-based. The state becomes a vector. Data flow: a string goes in, a fixed-length array of floats comes out.

Inside that one arrow live four genuinely different designs, ordered by how much they look at:

Text-based. The state is kept in natural language and never becomes a vector at all. Cascades "append the draft response and a verification confidence to the query." Fine-tuned language-model routers, exemplified by Router-R1, "verbalize the whole state directly in the prompt, leaving its representation to the model's forward pass."

Notice what that costs. A text-based E_q means the router itself is a language model, so a routing decision costs a forward pass through a language model — which is what you were trying to economise on. This is not fatal (the routing model can be far smaller than the candidates; the paper's multi-turn routers use Qwen2.5-3B-Instruct) but it is a real line on the bill, and c(τ) will charge you for it.

Component 2: the model encoder E_m

A router has to represent the candidates too, or it has nothing to compare the query against. Four options, in increasing order of how much data they need:

That last column is the one production engineers should read first. It is the answer to "a new model dropped this morning, how fast can we use it?" — and it varies by three orders of magnitude across the four rows.

Component 3: the scoring function g

The scoring function "measures the compatibility between the encoded state and each candidate." Its signature is the same in every router, whatever is inside:

Eighteen numbers. That is the entire output of the whole encoding apparatus, and everything downstream sees only those eighteen numbers. The instantiations "track the encoders":

That last row deserves a pause. In a fine-tuned language-model router there is no separable scoring function — representation, comparison, and selection all happen inside one transformer, and the "score" is the probability the decoder assigns to the token sequence spelling out a model name. The five-component decomposition still applies, but three of the slots are occupied by the same object. The paper says so plainly: "the query representation, candidate representation, and selection score are combined within the language model rather than implemented as separate embedding modules."

Component 4: the decision rule d

The decision rule "converts the resulting scores into a routing action." We already met the headline result in Chapter 1: greedy arg max is the default and is optimal only at λ = 0. The alternatives are where cost-awareness actually lives:

Component 5: the learning signal ℒ

The learning signal "specifies how the components above are fit toward Eq. 1." Four forms, and the paper's framing of them is the sharpest paragraph in the whole section:

And the unifying observation: "In every case, ℒ is a surrogate for the same objective. What differs is not the goal but the form in which perf is observable — measured for every candidate by supervised routers, returned only at the end of a trajectory for agentic ones, and revealed only through comparisons when quality is user-specific."

Sit with that. The reason personalized routers use preference losses is not that their authors preferred ranking losses. It is that you cannot measure perf_u directly — there is no ground-truth score for "how much did this particular user like this answer." All you can get is "she picked A over B." The loss follows from what is observable, not from taste.

The three families, filled in

Now assemble. Here is Table 1 of the paper, which is the whole formulation on one page:

Read the columns vertically and the differences localise beautifully.

The state column grows monotonically: q, then q and h, then q and u and h. Nothing is ever removed.

The encoder column shows E_m is identical in all three rows. Candidate representation is family-independent. Only E_q changes, exactly as promised.

The action column shows the single-turn and personalized rows are the same expression — arg max over m of g — differing only in what got fed into E_q. The multi-turn row is the one that uses a sampling rule d and an indexed action a_t.

The learning-signal column shows the three forms of observability: per-candidate rewards, episode returns, pairwise comparisons.

Concept meets realization: what you actually write

The paper's Figure 4 gives the complete code for a new router. This is not pseudocode — it is the library's real interface, and it is worth reading because it shows exactly where each of the five components physically lives:

from llmrouter.models import MetaRouter, BaseTrainer

class MyRouter(MetaRouter):            # (E_q, E_m, g, d): state -> action
    def route_single(self, query):
        s      = self.encode_state(query)     # context encoder E_q
        scores = self.score(s, self.models)   # model encoder E_m + scoring g
        query["model_name"] = self.decide(scores)   # decision rule d
        return query

class MyRouterTrainer(BaseTrainer):      # learning signal L
    def loss_func(self, outputs, batch):
        return my_objective(outputs, batch)  # pointwise / pairwise / RL reward

# Train and run through the same interface as every built-in router.
router  = MyRouter(yaml_path="my_router.yaml")
trainer = MyRouterTrainer(router)
trainer.train()
answer  = router.route_single({"query": "..."})

Four of the five components live in one method. The fifth lives in one other method. Everything else — data construction, training loop, inference, evaluation, deployment — is shared infrastructure that "operates on any router unchanged."

The design consequence is stated as the library's organising principle: "Swapping a router, a candidate pool, or a training objective is therefore a configuration change rather than a reimplementation." Concretely, the candidate pool and the objective weights live in a YAML file, so an ablation that would otherwise mean forking a repository becomes a one-line edit.

The one thing the abstraction does not give you

Honesty requires the caveat. A unified interface makes routers comparable; it does not make them equally expressible. Table 1's multi-turn row has a sampling rule and an episode return, which are strictly harder to optimise than the single-turn row's arg max and pointwise fit. The paper acknowledges the practical fallout: "Multi-round and RL-based routers cannot optimize this weighted objective and are therefore run once under a single configuration."

So the sweep in Chapter 8 covers eleven routers, not seventeen. That is a limitation of the comparison, disclosed by the authors, and you should carry it forward when you read the rankings.

Walking the shapes: a kNN router end to end

Abstractions are cheap. Here is every tensor a kNN router touches, with sizes, for one query.

Two observations. First, the "model encoder" here is implicit: a candidate is represented by the set of training queries it solved, which is a row of the matrix rather than a vector anyone learned. The five-component decomposition still applies; the slot is just filled by data instead of parameters. Second, the entire router is one matrix-vector product and a sort. Inference cost is microseconds, which is what a router is supposed to be.

Contrast a text-based router. The pipeline becomes: build a prompt containing the query and eighteen model descriptions (perhaps 600 input tokens), run a forward pass through a 3B model, decode a name. That is milliseconds and a real dollar cost, and every one of those dollars is charged to c(τ). Same five slots, three orders of magnitude apart in operating cost.

The inference budget, component by component

Read the "cached" column and a design rule falls out: everything that does not depend on the query should be precomputed at startup. Eighteen model embeddings is 18 × 384 floats — 27 kilobytes. Adding a nineteenth candidate at three in the morning is one more row in that table if E_m is metadata or a learned embedding you already have; it is a full retraining run if the model side was learned jointly. Chapter 2's model-encoder table is, read this way, an operations document.

The diagnostic: which slot did you actually change?

When you read a routing paper, or debug your own, the useful first question is which of the five slots differs from the baseline. Almost every reported improvement localises to one, and the ones that localise to none are usually reporting a difference in the surrounding stack — which is exactly what this infrastructure exists to eliminate.

Two routers, built slot by slot, side by side

The best test of a decomposition is whether two methods that look nothing alike fill the same five slots without strain. Take GraphRouter, a learned graph scorer, and AutoMix, a two-model cascade with a verifier.

They share no code, no data structure, and no intuition. They fill the same five slots without a single "not applicable." That is what a good abstraction looks like, and it is why the library can claim that "data construction, training, inference, and evaluation apply unchanged."

What the configuration file has to say

The paper notes that "a short YAML file specifies the router's candidate pool and objective weights, after which the router can be invoked by name using the same commands as any built-in method." That single sentence encodes a fair amount of design. A minimally sufficient configuration has to declare:

Read that as a dependency graph and it tells you the cost of each experiment. Changing the objective weights is free. Changing the hyperparameters costs a training run. Changing the pool costs a training run plus a full N×K collection sweep. Design your experiment schedule to move the cheap knobs first.

What the abstraction cannot express

Every decomposition has a boundary, and it is worth finding this one so you know when you have left it.

Sampling the same candidate more than once. Self-consistency — ask the same model five times at temperature and take a majority vote — is a real and effective technique. The action space is ℳ ∪ {⊥}, so a multi-turn router could emit the same model repeatedly, but nothing in the formulation represents "how many samples" as a decision variable, and no router in the library does it.

Partial-credit escalation. A cascade escalates the whole query. Nothing expresses "keep the small model's first three paragraphs and ask the big one to fix the fourth."

Anything that changes a candidate. The pool is fixed and the models are black boxes. Fine-tuning a candidate, adding a system prompt per candidate, or adjusting a candidate's decoding parameters are all outside the frame — and all are things a real deployment does.

Latency, throughput, availability. c(τ) is a scalar and the paper instantiates it as money. Real dispatch decisions are also about queue depth and provider outages.

The three shapes a learning signal takes

Component five deserves its formulas, because the choice between them is forced by what you can observe, and each has a different failure mode.

Pointwise. You observe a target for every (query, candidate) pair — the whole row of the matrix. Fit a regressor:

This is Table 1's single-turn row: "fit g to per-candidate reward perf(y_m | q) − λ c_m." Its weakness is that it spends capacity getting the absolute values right when only the ordering matters for the decision. A scorer that is uniformly 10 points pessimistic makes identical decisions and has a terrible loss.

Pairwise. You observe only comparisons — this candidate beat that one, for this user. Fit a ranker:

where σ is the logistic function. This is Table 1's personalized row: "fit g to comparisons m⁺ ≻_u m⁻." It optimises exactly what the decision needs, and it is forced on you when quality is user-specific — because "how much did she like it" has no scale, only an ordering.

Trajectory reward. You observe one scalar at the end of an episode. Optimise the expected return with a policy gradient:

with R(τ) = perf(y | q) − λ c(τ). This is Table 1's multi-turn row. It is the most general and by far the highest-variance: one scalar has to assign credit across every action in the trajectory.

Every ℒ here is a surrogate, including the pointwise one

Worth being precise. Equation 1 asks for the policy that maximises expected reward. ℒ_point minimises squared error on a predicted reward. Those are not the same objective, and the gap has a name in the recommender literature: you are optimising a proxy (prediction accuracy) for a goal (decision quality).

Concretely: suppose two candidates have true rewards 0.62 and 0.61, and your scorer predicts 0.30 and 0.90. Squared error is enormous, and the decision is wrong. Now suppose it predicts 0.90 and 0.30. Squared error is identically enormous, and the decision is right. The loss cannot tell those two situations apart; the deployment can tell nothing else.

This is why the paper evaluates routers by running them rather than by their training loss, and why a low validation loss is not evidence a router will route well. If you build one, measure the thing you want.

The interface as a reproducibility contract

One last framing of what the five slots buy, because it is easy to read the abstraction as merely tidy.

A shared interface is a contract about what is held fixed. When GraphRouter and RouterDC both subclass MetaRouter and both run through the same route engine, the same evaluation module, and the same query–model matrix, the set of things that can differ between them is exactly the set of things inside route_single and loss_func. Everything else is provably identical, because it is literally the same code path.

That is a stronger guarantee than "we tried to control for the stack." It is control by construction. And it is why the paper can make the claim it makes — "measured differences reflect the routing policy rather than the surrounding stack" — as a statement about the software rather than a statement about the authors' diligence.

The slot nobody thinks about

Of the five, E_m gets the least attention and causes the most operational pain. Everyone has an opinion about the scorer; almost nobody asks how a candidate is represented until the morning a new model ships and the router cannot see it.

Go back to the model-encoder table and read only the last column. Static metadata: free. Historical profiles: one sweep. Learned embeddings: a sweep plus a retrain. Verbalized: free at inference, but the fine-tuned router has never seen the name. That column is a four-way fork in your operational tempo, decided by an architectural choice made months earlier for reasons that had nothing to do with tempo.

If your pool changes monthly, that choice is the most consequential of the five, and it is the one most papers do not discuss.

Chapter 3: The Candidate Pool

Everything so far has treated ℳ as an abstract set of K things. It is not abstract. It is eighteen specific hosted endpoints with eighteen specific price tags, and if you do not know what is in it you cannot read a single number in this paper correctly — because a router can only be as good as the disagreements available in its pool.

The eighteen

Served through two providers, Together AI and NVIDIA NIM, spanning 7B to 671B parameters. Table 9, sorted by blended average price (the mean of the input and output prices), in USD per 1M tokens:

The "Blended" column is not in the paper as a column — the paper says the blended average is the mean of input and output, and sorts by it. So we can check our understanding by reproducing the sort. Take row 3: (0.05 + 0.20) ÷ 2 = 0.125, which slots between row 2's 0.100 and row 4's 0.150. Take row 12: (0.15 + 1.50) ÷ 2 = 0.825, which slots between row 11's 0.600 and row 13's (0.50 + 1.20) ÷ 2 = 0.850. Every row checks out, which means we are reading the price sheet the same way the authors did.

The shape of a query decides the price ranking

This is the single most practically important arithmetic in the chapter, so let us do it slowly. Compare two candidates that sit close together in the blended sort but have opposite price structures:

Case A: a memory-track query. The paper's memory track retrieves the top five conversational turn-pairs and stapes them into the prompt, so the input is long and the answer is a short phrase (the prompt literally says "Reply with a short phrase only"). Call it 3,000 input tokens and 20 output tokens.

gpt-oss-20b: (3,000 ÷ 10⁶) × 0.05 = $0.00015 for input; (20 ÷ 10⁶) × 0.20 = $0.000004 for output. Total $0.000154. The input is 0.00015 ÷ 0.000154 = 97.4% of the bill.

qwen3-next-80b: (3,000 ÷ 10⁶) × 0.15 = $0.00045; (20 ÷ 10⁶) × 1.50 = $0.00003. Total $0.00048.

Ratio: 0.00048 ÷ 0.000154 = 3.1×.

Case B: a code-generation query. MBPP hands the model a task description plus tests and asks for a function body. Short in, long out. Call it 300 input tokens and 600 output tokens.

gpt-oss-20b: (300 ÷ 10⁶) × 0.05 = $0.000015; (600 ÷ 10⁶) × 0.20 = $0.00012. Total $0.000135.

qwen3-next-80b: (300 ÷ 10⁶) × 0.15 = $0.000045; (600 ÷ 10⁶) × 1.50 = $0.0009. Total $0.000945.

Ratio: 0.000945 ÷ 0.000135 = 7.0×.

Same two models. Same price sheet. The cost ratio between them more than doubles, from 3.1× to 7.0×, purely because the query changed shape. A router trained on one regime and deployed in the other will have systematically wrong beliefs about cost even if its quality predictions are perfect.

Watch what happens when you switch from "code" to "memory". gpt-oss-20b, which is cheap on both sliders, becomes dramatically the best value in the long-input regime because its $0.05 input price is the lowest in the pool by a factor of two. Meanwhile qwen3-next-80b, respectable in the blended sort at position 12, becomes an expensive choice in code and a merely-average one in memory. No single ordering of this pool is correct.

What the pool tells you about the ceiling

Here is a way to think about a candidate pool that will make Chapter 7's numbers legible. Define the oracle: the policy that, for each query, magically picks the candidate that actually got it right at the lowest cost. The oracle is not achievable — it requires knowing perf before you pay for it — but it is the ceiling any router is chasing.

The gap between the best fixed model and the oracle is the total headroom. The gap between the best fixed model and your router is what you captured. Routing research is entirely about that fraction, and it has a property worth internalising:

A detail worth noticing: the 17-model pool

The paper mentions, almost in passing, a "17-model no-cogito pool [that] drops the most expensive model (cogito-v2-1-671b)." That is not housekeeping. Dropping the single most expensive candidate changes the cost ceiling of every policy that would have selected it, and it changes what "Largest-LLM" means, since cogito-v2 and deepseek-v3.1 are tied at 671B parameters.

This is also a good moment for a caution about the rule baselines. "Smallest" and "Largest" are defined by declared parameter count, and both ends of the range have ties: mistral-7b-instruct-v0.3 and both qwen2.5-7b variants all declare 7B; deepseek-v3.1 and cogito-v2-1-671b both declare 671B. The paper does not state its tie-break. So when Chapter 0 quoted the 671B-over-7B price multiple as a band of 3.8× to 6.25× rather than a single number, the range was not hedging for its own sake — the 7B tier spans $0.20 to $0.30 blended and the 671B tier spans $1.15 to $1.25, so the ratio genuinely lies between 1.15/0.30 = 3.8 and 1.25/0.20 = 6.25 on blended prices, and further apart still on input-heavy or output-heavy traffic. Read baseline costs as a band, not a point: the same phrase "Largest-LLM" denotes two different price points depending on a tie-break the paper never publishes, and every cost-weighted comparison involving it inherits that ambiguity.

Cost is where routing research differs from every other benchmark

One last framing before we build the data engine. In ordinary ML benchmarking, the x-axis is implicit: everyone runs the same model on the same hardware and reports one number. Routing has no such luxury, because the choice being evaluated is itself the choice of how much to spend. A router that reports 45.46 without a price is not reporting a result; it is reporting half of one.

The paper builds this in at the protocol level: every response is scored "with its task metric and priced from its token counts", and every reported comparison is under an explicit weighting of the two. That is the discipline. When you see a routing result anywhere else with no cost axis, you now know what is missing.

When is the blended average the right number?

The paper sorts Table 9 by the blended average, so it is worth knowing exactly what that number assumes. Blended price is (p_in + p_out) ÷ 2, and the true cost is (n_in p_in + n_out p_out) ÷ 10⁶. Setting the blended estimate equal to the true cost:

Expand the left side: (n_inp_in + n_inp_out + n_outp_in + n_outp_out) ÷ 2. Subtract the right side and multiply through by 2:

So the blended average is exact in exactly two cases: n_in = n_out (your prompt and answer are the same length) or p_in = p_out (the provider charges the same both ways). Nine of the eighteen candidates satisfy the second condition, which is why the blended sort looks so reasonable. The other nine do not, and for those the blended figure is a fiction that happens to be convenient for sorting a table.

The break-even token ratio between two candidates

Here is a genuinely useful piece of arithmetic that falls straight out. Two candidates cost the same when:

Collect: n_in(a_in − b_in) = n_out(b_out − a_out), so the break-even ratio is

Worked example. Take A = mistral-small-3-24b ($0.10 in, $0.30 out) and B = rnj-1-instruct ($0.15 in, $0.15 out). Both have blended price $0.200 and $0.150 respectively, so the blended sort says rnj is cheaper, full stop. Is it?

Break-even ratio: (0.15 − 0.30) ÷ (0.10 − 0.15) = (−0.15) ÷ (−0.05) = 3.0.

So when your prompt is exactly three times your answer, they cost the same. Check at n_in = 3,000 and n_out = 1,000: mistral gives (3,000 × 0.10 + 1,000 × 0.30) ÷ 10⁶ = (300 + 300) ÷ 10⁶ = $0.0006. rnj gives (3,000 × 0.15 + 1,000 × 0.15) ÷ 10⁶ = (450 + 150) ÷ 10⁶ = $0.0006. Identical. Above a 3:1 ratio — every memory-track query, every agent scaffold — mistral-small-24b is the cheaper of the two, despite ranking below rnj in the paper's own sort.

Why price does not track parameter count

Three mechanisms, and knowing them stops the Table 9 anomalies from looking like typos.

Mixture-of-experts models activate a fraction of their weights. mixtral-8x7b declares 46.7B total parameters, but each token is routed to two of eight experts, so the compute per token is closer to a 13B dense model. deepseek-v3.1 declares 671B and activates a small fraction of that. Serving cost tracks active parameters, not declared ones. That single fact explains why a 671B model can be cheaper than a 140.6B one.

Serving stacks differ. Rows 7 and 8 of Table 9 are the same base model — qwen2.5-7b-instruct — served by NVIDIA at $0.20 blended and by Together at $0.30 blended. Same weights, 50% price difference. That is the serving stack, the hardware, and the vendor's margin, not the model.

Prices are commercial decisions. A provider may subsidise a flagship or price-in demand. Nothing obliges the price sheet to be a function of anything technical.

The axis this paper does not measure

Honesty about scope. The objective in Equation 1 has two terms, quality and money. Production systems have a third: latency. A 671B model on a busy endpoint may take five seconds to first token where a 9B model takes two hundred milliseconds, and for an interactive product that difference is often decisive in a way that a tenth of a cent is not.

xRouteBench does not measure latency, and the formulation does not include it. Adding it is not conceptually hard — c(τ) becomes a vector of costs and λ becomes a vector of weights — but it is not what the paper measured, and you should not read the reported frontiers as latency frontiers. If you build on this, that is the first extension worth making, and it will change the rankings again, because latency does not track price either.

Each track has a cost signature

Chapter 5 will describe the benchmark tracks properly. But since we have the price sheet in front of us, it is worth seeing now that each track lands in a different place on it. These token shapes are our estimates from the paper's prompt descriptions, not reported figures — the ratios are what matter.

Now apply the break-even ratio from earlier. mistral-small-24b beats rnj-1 above a 3:1 input-to-output ratio. Look down the third column: that is true for memory (150:1), time series (12:1), and marginally not for the others. So the correct cheapest choice between those two models flips depending on which track a query came from — and in a mixed deployment, on which query.

Costing a multi-agent query, before we get there

Chapter 9 studies routing inside multi-agent systems, where one user query becomes four to seven model calls. The pricing consequence is worth seeing now, because it is stark.

Take the Tree topology, which the paper reports as seven LLM calls per query. At 200 in / 150 out per call:

All calls to cogito-v2-1-671b: 7 × $0.0004375 = $0.0030625 per user query.
All calls to gemma-2-9b-it: 7 × $0.000035 = $0.000245.
A mixed route — say the planner and consolidator on a mid-tier model and the actors on cheap ones — lands between, and that is exactly the space a per-node router explores.

At a million user queries a day, the all-cogito Tree costs $3,062 daily, which is $1.12M a year, for one feature. That is the scale at which per-node routing stops being a research curiosity. It is also why the paper's finding that six of seven learned routers beat always-largest in this setting is the most immediately bankable result in the paper.

A design exercise: build a pool worth routing over

Since headroom is a property of ℳ, choosing the pool is the highest-leverage decision you will make. Three principles, all readable off Table 9.

Span the price range, not the size range. The pool covers 25× in input price. That range is what makes the cost term in Equation 1 able to change decisions at all. A pool where every candidate costs within 20% of every other has a cost term that never overrules the quality term, and you have built an expensive way to run arg max.

Include specialists, not only ladders. qwen3-coder-next is in the pool for a reason: the generic track contains 516 code items and a coder-specialist creates a disagreement that a pure size ladder would not. Every specialist you add is a new region of query space where the ranking inverts.

Do not include near-duplicates without a price reason. The pool has two copies of qwen2.5-7b at different prices, which is defensible as a serving comparison. Two copies at the same price would add nothing but a coin flip for the router to get wrong.

And a fourth, practical one: keep the pool small enough to sweep. Every candidate costs N calls to profile, so K = 18 means each benchmark rebuild is eighteen sweeps. Doubling the pool doubles the rebuild and roughly doubles the classifier's output dimension. The marginal candidate has to earn its column.

What dropping cogito does

The paper's aside about a "17-model no-cogito pool" is worth taking seriously as an experiment in pool design, because it changes three things at once.

It changes the cost ceiling. The most expensive blended price falls from $1.250 to $1.200 (mixtral-8x22b), so the pool's price spread narrows slightly from 25× on input to 24× ($1.20 ÷ $0.05).

It changes what Largest-LLM means. With cogito removed, the 671B tier contains only deepseek-v3.1 — and deepseek is cheaper ($1.150 blended) than mixtral-8x22b ($1.200). So the baseline's identity becomes unambiguous, which is a genuine methodological gain, and its cost changes.

It removes whatever cogito was uniquely good at. If cogito was the only candidate solving some subset of queries, the oracle drops and every router's ceiling drops with it. If it was redundant with deepseek, nothing happens except the bill.

That third possibility is the interesting one and it is exactly the kind of question the infrastructure is built to answer: rerun with the seventeen-model configuration and compare oracles. A drop tells you cogito earned its column; no drop tells you it did not. Every candidate in a production pool should have to pass that test.

Price per correct answer — the metric that combines both axes

Equation 1 combines quality and cost with a weight you have to choose. There is a weight-free combination that is often more legible to non-specialists:

Worked example, on the Chapter 0 numbers. Over 1,000 generic-mix queries at 200 in / 150 out:

Largest-LLM: spend 1,000 × $0.0004375 = $0.4375. Correct at 70.29% = 702.9 answers. Price per correct = 0.4375 ÷ 702.9 = $0.000622.

Smallest-LLM (7B tier at $0.20 blended): spend 1,000 × $0.00007 = $0.07. Correct at 57.55% = 575.5. Price per correct = 0.07 ÷ 575.5 = $0.000122.

Ratio: 0.000622 ÷ 0.000122 = 5.1×. The large model costs five times as much per correct answer, not merely per query — because the higher accuracy recovers only part of the price gap.

Now the routed version. GraphRouter scores 80.54 on this track. If its average cost per query landed at, say, $0.00015 — plausible for a policy that sends most queries to mid-tier models — then its price per correct is 0.15 ÷ 805.4 × 1,000 ÷ 1,000 = $0.000186. Better accuracy than always-largest and a third of the price per correct answer. (That cost figure is our illustration; the paper does not publish per-router average costs in Table 2, which is reported at β = 0.)

Two providers is a design decision, not an accident

Eighteen models across Together AI and NVIDIA NIM. That split does work beyond convenience.

It exposes serving as a variable. The duplicate qwen2.5-7b pair only exists because two providers serve the same weights, and it is the pool's only controlled test of "does the serving stack change quality?" — a question that matters and that a single-provider pool cannot ask.

It makes the pool robust. A single provider's outage takes out the entire action space. Two providers means the router degrades to a smaller pool rather than to nothing.

It makes the engineering honest. Two providers means two API dialects, two rate-limit regimes, two failure vocabularies, and two message-role conventions — which is why Appendix F has to specify what happens when a provider does not support a system role. A pipeline that works across two providers will probably work across five. One that works across one probably will not.

Retries multiply your cost model

One cost the price sheet does not show and every production system has. When a call fails — a timeout, a malformed response, a rate-limit rejection — you retry, and you pay again.

If a candidate has failure probability f and you retry until success, the expected number of calls is 1 ÷ (1 − f), so the effective price is

Worked example. At f = 0.02 (a 2% failure rate), p_effective = p ÷ 0.98 = 1.020p — a 2% surcharge, negligible. At f = 0.15, p ÷ 0.85 = 1.176p, a 17.6% surcharge. At f = 0.40, p ÷ 0.60 = 1.667p — you are paying two thirds more than the sticker price.

Now the important part: f differs by provider and by model. A heavily loaded endpoint for a popular model may fail far more often than a niche one. So two candidates with identical sticker prices can have materially different effective prices, and a router using sticker prices will systematically over-select the flakier one.

The paper's protocol does not report failure rates, and for a research benchmark that is reasonable — retries are collection-time engineering. For a production cost model it is not optional: measure f per candidate, and price with it.

The cheapest column, per track, at a glance

Pulling the earlier arithmetic together into the table you would actually consult. For each track's token shape, which of five representative candidates is cheapest?

Verify the third row's mistral cell: (3,000 × 0.10 + 20 × 0.30) ÷ 10⁶ = (300 + 6) ÷ 10⁶ = $0.000306. And the fourth row's oss-20B: (1,200 × 0.05 + 100 × 0.20) ÷ 10⁶ = (60 + 20) ÷ 10⁶ = $0.00008.

gemma is cheapest on the two balanced shapes and loses both input-heavy ones to oss-20B, whose $0.05 input price is half of gemma's. That single price cell — the lowest input price in the pool — makes a 20B model cheaper than a 9B one for exactly the queries where input dominates. If your product is a memory-augmented assistant, that is the most important number on the whole sheet, and it is invisible in the blended sort where oss-20B ranks third and gemma ranks first.

What a nineteenth candidate should look like

A useful exercise, because it forces you to reason about a pool as a portfolio rather than a list. Given these eighteen, what would you add?

Not another 7B generalist. There are already four (mistral-7b, two qwen2.5-7b variants, llama-3-8b). A fifth adds a column that correlates almost perfectly with existing ones, so the oracle barely moves and every learned router gets one more nearly-indistinguishable class to confuse.

Not another 671B model. There are two, they cost the most, and Table 2 says the largest tier is not where the headroom is.

Yes: something with a genuinely different failure surface. The pool has exactly one coder-specialist. It has no long-context specialist priced for long context — and the memory track showed input-dominated queries are a distinct regime. A candidate with a very low input price and modest quality would be a new corner of the frontier, not a new point on an existing edge.

Yes: something with an unusual price shape. gpt-oss-20b is valuable out of proportion to its size purely because its $0.05 input price is half the next cheapest. A candidate priced unusually along either axis creates decisions no other candidate can create.

A last sanity check on the sort

We reproduced the blended-price ordering earlier for two rows. Since the ordering is the only structure the paper imposes on Table 9, it is worth confirming the boundary cases where blended prices tie, because ties reveal what the secondary ordering is doing.

Rows 1 and 2 both blend to $0.100 (gemma-2-9b and llama-3-8b-lite). Rows 5, 6, and 7 all blend to $0.200 (mistral-7b, mistral-small-24b, qwen2.5-7b) — and note that mistral-small-24b gets there by a completely different route, $0.10 in and $0.30 out, while the other two are flat $0.20. Three candidates, one blended price, two very different cost behaviours.

Apply the break-even ratio to the pair inside that tie. For mistral-7b (0.20 / 0.20) against mistral-small-24b (0.10 / 0.30): break-even n_in/n_out = (0.30 − 0.20) ÷ (0.20 − 0.10) = 0.10 ÷ 0.10 = 1.0. So they cost the same when the prompt equals the answer in length; above that ratio — which is most real traffic — the 24B model is cheaper than the 7B one.

Check at 3,000 in / 20 out: mistral-7b gives (3,000 × 0.20 + 20 × 0.20) ÷ 10⁶ = (600 + 4) ÷ 10⁶ = $0.000604. mistral-small-24b gives (300 + 6) ÷ 10⁶ = $0.000306. The 24B model costs half as much on a memory query, despite being three times the size and identically priced in the blended sort.

The one-line summary of this chapter

If you retain nothing else from the price sheet: cost is a function of two prices and two token counts, and collapsing it to one number throws away the half of it that changes your answer.

Everything else here was that statement made arithmetic — the blended-average identity, the break-even ratio, the per-track cost signatures, the duplicate qwen pair, the 24B model that is cheaper than the 7B one on long inputs. Every one of those is invisible to a system that stores one price per candidate, and every one of them changes which candidate you should have picked.

Chapter 4: The Query–Model Matrix

Everything in this paper — the supervision, the evaluation, the benchmark, the ablations, the multi-agent study — rests on one object. It is a table. The paper calls it the query–model matrix, and once you see it, the whole architecture of the library becomes obvious, because every module is either producing it or consuming it.

Building it, three stages

The pipeline is stated in three steps, and each one is a place where hand-built research setups used to break:

The product is described in one sentence worth reading twice: "a dense query–model matrix of performance and cost that serves at once as routing supervision and as the test bed."

What is actually in the cells

Concretely, for N queries and K candidates you get two aligned N × K tables:

For the test split of xRouteBench: N = 4,767, K = 18. So each table has 4,767 × 18 = 85,806 cells, and every cell required one real API call to a real hosted model whose output then had to be parsed and graded.

Let me make the scale concrete with arithmetic you can check. 4,767 × 18 = 4,767 × 10 + 4,767 × 8 = 47,670 + 38,136 = 85,806. That is the test split only. Every router also trains on a training split, which needs its own full sweep.

What does that cost in dollars? The paper does not publish its API bill, so we cannot quote one — but we can bound the shape of it, which is more useful anyway. The mean blended price across the eighteen candidates is the sum of the blended column divided by 18. Summing the Chapter 3 table: 0.100 + 0.100 + 0.125 + 0.150 + 0.200 + 0.200 + 0.200 + 0.300 + 0.375 + 0.560 + 0.600 + 0.825 + 0.850 + 0.880 + 0.900 + 1.150 + 1.200 + 1.250 = $9.965, so the mean is 9.965 ÷ 18 = $0.5536 per 1M tokens.

At an average of 500 tokens per call, 85,806 calls consume 42.9M tokens, which costs 42.9 × 0.5536 ≈ $24. At 5,000 tokens per call — the scale of the long-context memory track — it is 429M tokens and $238. Those are our estimates, not the paper's, and the point of showing both is the sensitivity: the bill is dominated by the long-input tracks, which is the same asymmetry that made those tracks worth building.

Supervision and evaluation are the same object, used two ways

This is the part that is genuinely elegant. Look at the asymmetry:

The paper: "Evaluation reuses this path, except that a test query goes only to the candidate the router selects rather than to the whole pool." And crucially, since the matrix is already built, the evaluation does not need to call anything at all — it can look up the cell. That is what turns a benchmark run from an hour of API calls into a table join, which is what makes sweeping eleven routers across five cost weights tractable.

One nuance that keeps the accounting honest: for cascades and multi-turn routers, "every decomposition and aggregation call is priced into the trajectory cost." A cascade that drafts with the small model and escalates pays for both cells, not the better one.

Worked example: reading a row

Take one query and four candidates, with quality on a 0–1 scale and cost in dollars, for a 200-in / 150-out generic question. Prices are real; the quality values are illustrative because the paper publishes track aggregates, not per-query cells.

Check the llama row by hand: input (200 ÷ 10⁶) × 0.88 = $0.000176; output (150 ÷ 10⁶) × 0.88 = $0.000132; total $0.000308. Then 20,000 × 0.000308 = 6.16.

This single row contains the whole thesis. The two cheapest candidates got it right; the two most expensive got it wrong. Largest-LLM scores 0 here and pays the most. Smallest-LLM scores 1 and pays the least. A perfect router scores 1 for $0.000035. And the supervision signal a pointwise router learns from this row is precisely the last column: fit g to perf(y_m | q) − λ c_m, exactly as Table 1's single-turn row specifies.

Three things this makes visible. First, the correct cells are scattered, not concentrated in the expensive columns — that scatter is the headroom. Second, the oracle is much cheaper than Largest and much more accurate than Smallest, which is the shape every routing result has. Third, the learned router does not reach the oracle, and the gap it leaves is the honest measure of how hard this problem still is.

Why the configuration-file detail matters more than it sounds

The paper says the candidate pool "is declared in a single configuration file", that adding a task "requires only a prompt template and a registered metric", and that adding a candidate "requires only its endpoint and per-token price."

Translate that into the thing you would otherwise have to do. Without it, adding a nineteenth model means: finding every place the pool is enumerated; re-running the collection for all N queries; re-fitting every model-side representation (kNN profiles, Elo ratings, MF latent factors, learned model embeddings); and re-running every evaluation. With it, you edit one file and re-run one command. The paper's claim that "a new task or candidate pool enters through a configuration change rather than a re-engineered stack" is the difference between a benchmark that ages out in six months and one that survives the next model release.

The metrics that fill P

perf is not one function. LLMRouter ships built-in metrics "aligned with standard benchmark conventions":

And on top of that, the library "exposes a weighted objective that balances performance and cost, allowing routers and trainers to target performance-first, cost-sensitive, or hybrid operating points" — which is Equation 1 made into a configuration knob.

What one row of the schema holds

"Normalized into a unified schema" is doing a lot of quiet work. Here is what a single curated query record has to carry so that thirteen generic subtasks plus four exotic tracks can share one table:

The paper's phrasing is precise: each non-text asset is converted "with an optional pointer to the source image, video, or time series." Keeping the pointer is what makes the transformation reproducible rather than a one-way destruction of the original data.

Split hygiene matters more here than usual

A subtlety specific to routing. Half these routers are non-parametric — kNNRouter's entire model is the training set. If a test query leaked into the stored neighbours, kNNRouter would retrieve itself at similarity 1.0 and read off the correct answer, scoring near-perfectly for reasons that have nothing to do with routing.

So the split has to happen at stage one, before anything else, which is exactly where the pipeline puts it: queries are "sampled from source benchmarks, normalized into a unified schema, and split into training and test sets." Doing it later — splitting the matrix after collection, say — would be equivalent, but doing it at curation means every downstream module inherits the split for free and cannot get it wrong.

Chapter 9's Slack study takes the same care one level up, splitting "by session into 32 training sessions and 8 test sessions" rather than by individual record, so a user's earlier turns cannot inform the prediction of their later ones.

Label noise: what a buggy grader costs you

Every cell of P is the output of an automatic grader, and graders have failure modes. Execution-based code evaluation can fail on a correct answer with a stray import. Math answer verification can miss an equivalent form. Exact match punishes a right answer with different whitespace.

Suppose your graders are 95% reliable, so 5% of cells carry a wrong label. What does that do to a router?

Worked example. Take a query where the true best-model rate is such that a perfect router would be right 70% of the time. With 5% independent label noise on the supervision, the fitted target is a corrupted version of the truth: the router is being taught, on one query in twenty, to prefer a model that actually failed. In the limit of enough data, the learned scorer converges to the noisy conditional expectation, which is the true one shrunk toward the noise floor. And critically, the same noise is in the test labels — so a router that learned the truth perfectly would be measured at 0.95 × 0.70 + 0.05 × (some chance of accidentally matching) ≈ 0.67, and a router that learned the noise would score the same or better.

The practical upshot: grader quality is a ceiling on every router in the table simultaneously, and because it is shared, it does not change the ranking much — which is precisely why a shared pipeline is more valuable than a perfect one. Everyone is measured against the same imperfect ruler. That is what "measured differences reflect the routing policy rather than the surrounding stack" buys you.

What if you cannot afford a dense matrix?

N × K is fine at N = 4,767 and K = 18. It is not fine at N = 10,000,000 production queries and K = 40 candidates, which is 400 million calls.

The paper builds a dense matrix because a research benchmark should. In production you would sample: route most traffic normally and divert a small fraction to a random candidate, logging the outcome. That gives you a sparse matrix — and notice that one router in the zoo is designed exactly for sparse matrices, because it comes from a field where dense ratings never exist. MFRouter learns latent factors precisely so it can predict unobserved cells from observed ones.

This is worth flagging as an extension the paper does not study: every method here is evaluated with complete supervision. How each degrades as the matrix thins out is an open and very practical question, and the answer will not be uniform — kNN and Elo need only the cells they have, whereas a classifier trained to predict "which of eighteen is best" needs a full row to define its label.

Headroom, oracle, and capture rate — defined properly

The matrix lets us define the three quantities that every routing result is implicitly reporting. Let P be the performance matrix and C the cost matrix, over N queries and K candidates.

Read the difference between the first two carefully, because it is the whole subject. best-fixed takes the max outside the sum — one column, chosen once. oracle takes the max inside the sum — the best cell in each row, chosen per query. Swapping the order of a max and a sum is the entire opportunity, and the size of the swap is the headroom:

and the fraction a router captures is

Worked example, from the simulation above. Twelve queries, eight candidates. Largest gets 7 of 12, Smallest gets 6, the oracle gets 12, the learned router gets 9. So best-fixed = 7/12 = 0.583, oracle = 12/12 = 1.000, router = 9/12 = 0.750.

Headroom = 1.000 − 0.583 = 0.417. Capture = (0.750 − 0.583) ÷ 0.417 = 0.167 ÷ 0.417 = 0.40. The router picked up two fifths of what was available.

That framing is more informative than an accuracy number, because it separates two questions that a raw score conflates: how much was there to win (a fact about the pool) and how much did we win (a fact about the router). Two routers reporting the same accuracy on different pools have not achieved the same thing.

The cost-aware oracle is a different object

One subtlety. The oracle above maximises quality per row, ignoring cost. There is a second, more useful oracle: for each row, the cheapest candidate that is correct.

This is the ceiling the simulation's "Oracle" button uses, and it is the right target for a cost-aware router: perfect accuracy at the lowest achievable bill. It is also unachievable for a stronger reason than the quality oracle — not only must you know which candidates are correct, you must know it before paying, and correctness is exactly what you cannot observe in advance.

What a cell cannot capture

Every abstraction discards something. Four things the matrix loses, all of which matter in deployment:

Variance. One cell is one sample. Run the same model on the same query twice at non-zero temperature and you may get different scores. A single-sample matrix treats a 55%-reliable model and a 55%-reliable coin identically, and the router learns a mean where the variance was the interesting part.

Partial credit that the metric flattens. A code answer that fails one of five tests scores 0 under an execution pass rate. So does one that fails to parse. The router sees the same label for a near-miss and a catastrophe.

Latency. Discussed in Chapter 3, absent here.

Failure modes. A refusal, a timeout, and a confidently wrong answer are all a zero. In production those three demand very different responses, and a router trained on the flattened label cannot distinguish them.

None of these invalidate the benchmark. They tell you what a second-generation matrix would carry — several samples per cell, a graded score, a latency field, and a failure-reason field — and that is a concrete direction for anyone extending this work.

The pipeline, as code you could write

Three stages, and it is short enough to sketch. This is our rendering of the described pipeline, not the library's actual source, but the shape is what matters.

# Stage 1 — Query Curation
records = []
for task in config.tasks:                 # declared in one config file
    for ex in sample(task.source, task.n):
        records.append({
            "text":    task.template.render(ex),   # the ONE prompt every model sees
            "gold":    ex.answer,
            "metric":  task.metric,               # which grader to run later
            "asset":   ex.image_or_series,        # optional pointer, kept for re-rendering
            "split":   assign_split(ex),          # BEFORE anything else — no leakage
        })

# Stage 2 — Response Collection   (the expensive one: N x K calls)
for r in records:
    for m in config.candidates:            # endpoint + per-token price
        out = call(m.endpoint, r["text"])
        store(r.id, m.id, out.text, out.n_in, out.n_out)   # token counts are the point

# Stage 3 — Metric Scoring and Pricing
for (r, m, text, n_in, n_out) in store:
    P[r.id][m.id] = GRADERS[r["metric"]](text, r["gold"])
    C[r.id][m.id] = (n_in * m.price_in + n_out * m.price_out) / 1e6

Read the comments as the design. Splitting happens in stage one. Token counts are stored, not costs, so prices can change without a re-collection. The metric identifier travels with the record so stage three needs no knowledge of tasks. And the candidate list appears in exactly one place, which is what makes "adding a candidate requires only its endpoint and per-token price" true.

What happens when a call fails

A detail the pseudo-code glosses and every real implementation must decide. During stage two, a candidate will sometimes refuse, time out, return malformed output, or be rate-limited into oblivion. What goes in the cell?

A refusal is genuinely ambiguous: a model declining to answer is a quality outcome from the user's point of view, and is not one from the model-capability point of view. Whichever you choose, choose it once and write it down, because it silently defines what P means. The paper does not report its policy; this is a place where reading the released code would be worth the time.

Idempotency, and why the hash matters

Chapter 9 mentions the ComfyUI node that "hashes the selected datasets, candidate pool, and sample size into a metadata record and reuses an existing data directory when the record and its files are unchanged." That is worth pulling forward, because it is the single most useful engineering idea in the paper for anyone reproducing this work.

The expensive stage is stage two. If you re-run the pipeline after changing a router hyperparameter, you must not re-run 85,806 API calls. Content-addressing the inputs — task list, pool, sample size — and caching the output directory under that hash makes stage two idempotent: identical inputs, instant reuse; any input changed, automatic rebuild.

Without it, the iteration loop of routing research is hours and hundreds of dollars. With it, the loop is seconds for everything except the one change that genuinely invalidates the data. That is the difference between an experiment you run once and a research programme.

What the training split has to contain

The pipeline splits queries into train and test, and the test half is 4,767. The training half is not reported as a single number, but its composition requirements are worth deriving, because they constrain the design.

Every candidate needs coverage. A historical-profile model encoder represents a candidate by the queries it solved. A candidate with no training observations has an empty profile, and kNN will never vote for it. So the training split must include every candidate on every task family — which is why it must also be a dense sweep, not a sample.

Every task family needs enough examples to be learnable. If AIME contributes 13 test items, its training complement is presumably small too. A router cannot learn "route competition mathematics to the reasoning specialist" from a handful of examples, which is part of why the specialised behaviours the benchmark hopes to elicit are hard to elicit.

The split must preserve the mix. If training is 90% knowledge QA and test is 50%, the router's prior is wrong before it starts. Stratified splitting by subtask is the standard remedy and is the natural reading of "split into training and test sets" at the curation stage.

Together those three say the training sweep is at least as expensive as the test sweep, so the real cost of the matrix is comfortably north of the 85,806 test cells — likely double.

Eighteen columns times six graders

Stage three has a cross-product that is easy to underestimate. Six metric families, eighteen candidates, and every combination must work.

Concretely: the execution-based code grader must extract Python from eighteen models' idea of "put your code in [BEGIN]...[Done]". The math verifier must find a boxed answer in eighteen models' idea of LaTeX. The multiple-choice parser must find a letter in a parenthesis in eighteen models' idea of a final answer, some of which will helpfully add "The answer is (C), because...".

Every parser failure is scored as a wrong answer, and parser failures are not distributed evenly across candidates — a model with strong instruction-following will comply with the format more often than one without. So format compliance is silently mixed into P alongside task capability.

The matrix is also the thing you debug against

One underrated use. Once P and C exist, they answer operational questions that have nothing to do with training a router:

That last row is worth pausing on. Rows where every candidate fails are queries routing cannot help with, by definition. Counting them tells you the ceiling of your entire pool, and if the count is large the correct response is not a better router — it is a better candidate.

Reading the matrix as a portfolio

One more way to hold the object, borrowed from finance and genuinely clarifying. Think of each candidate as an asset whose "return" on query i is P[i][j] and whose "price" is C[i][j].

A fixed policy is holding one asset. Its expected return is that column's mean, and its risk is that column's variance. Routing is not diversification — you are not spreading across assets to reduce variance — it is timing: choosing which asset to hold on each individual query, using a signal available before you buy.

That framing sharpens two things. First, it explains why correlation between columns is the enemy. If two candidates succeed and fail on the same queries, their columns are perfectly correlated and there is no timing decision to make between them. Headroom is a statement about column decorrelation. Second, it explains why the oracle is unreachable in principle rather than merely in practice: the oracle uses information from after the purchase.

And it suggests a diagnostic nobody runs. Compute the correlation matrix of P's columns. Pairs near 1.0 are redundant candidates you could drop. Pairs near 0 are where all your routing value comes from. That is a five-line computation on a matrix you already have, and it will tell you more about your pool than any router's accuracy number.

Why this chapter is the hinge of the paper

Chapters 1 and 2 gave a formulation, which is a way of thinking. Chapter 5 will give a benchmark, which is a dataset. This chapter gave the thing that connects them, and it is the one that could not be skipped.

A formulation without supervision is philosophy. A benchmark without a construction pipeline is a frozen artefact that expires with its candidate pool. The query–model matrix — and specifically the automation of building one — is what turns both into a research programme. That is why the paper's own framing puts the pipeline before the benchmark: xRouteBench is described as a product of the data engine, not the other way around.

Chapter 5: xRouteBench

You now have a machine that turns a task list and a candidate pool into a query–model matrix. Point it at the right tasks and you get a benchmark. This chapter is about which tasks, and — more interestingly — about a design decision the authors made that you should agree with only after you understand its cost.

What was missing

Start with the gap, because xRouteBench is defined by it. Four prior resources, four blind spots:

So the target list writes itself. xRouteBench is built to cover, under one cost-aware protocol, "regimes in which routing decisions fundamentally differ": long-context inputs where input tokens dominate, image and video inputs that only some candidates can process, time-series inputs with multiple modality encodings, and tasks with user-specific quality preferences.

The five tracks, 4,767 instances

Add them up yourself, because the arithmetic is worth carrying: 3,729 + 314 = 4,043; + 101 = 4,144; + 127 = 4,271; + 61 = 4,332; + 100 = 4,432; + 27 = 4,459; + 308 = 4,767. The seven non-personalized sets total 4,459 — that is the number Chapter 7 will need, because Table 2 reports on those seven and the personalized track is scored separately in Table 3.

Now look at the distribution, and notice how lopsided it is. The generic mix is 3,729 ÷ 4,459 = 83.6% of the non-personalized queries. Charades-Ego is 27 ÷ 4,459 = 0.61%. That is a ratio of 138 to 1. Keep that in your pocket; in Chapter 7 it becomes the single most important fact about how the results table should be read.

Inside the generic mix

The generic track is "deliberately a mixture rather than a single task family": knowledge, commonsense, reading comprehension, mathematics, and code, all behind the same routing interface. Thirteen subtasks, and the sizes are informative:

Check the total: seven subtasks at 500 gives 3,500; four at 50 gives 200; plus 16 and 13. So 3,500 + 200 + 16 + 13 = 3,729. It reconciles.

The framing question the track is built to ask: "whether a router can recognize these distinctions within an apparently uniform text setting, rather than defaulting to one model for every natural language request." All thirteen look identical from outside — text in, text out — but "an ARC-Challenge question, an AIME problem, and a Python synthesis task reward very different capabilities and impose different output constraints."

The design decision: everything becomes text

Here is the choice that shapes the whole benchmark, stated by the paper as its "Design principle":

So: a geometry diagram is described in words by a frozen vision–language model (Gemma-3-27B-IT by default), and the description is appended to the problem. A time series is plotted, captioned by the same model, and both the caption and the raw numerical values (truncated after 200) are appended to the question. A video's available views are sampled to frames, described, and merged into one text query. A long conversation is retrieved over with a fixed Contriever encoder, and the top five turn-pairs are inserted.

Why this is right. Two reasons, both good. First, it makes the pool usable: fourteen of the eighteen candidates are text-only, and without the transformation they could not compete for a single vision query, which would collapse the vision track to a four-way choice. Second, it isolates the variable under study. As the paper puts it for the vision track: "Rendering the image once holds perception constant across the pool and turns visual mathematics into a query the whole pool can answer, so the track scores the reasoning quality of each candidate on the same input." Any difference you then measure is a difference in reasoning, not in whose image encoder is better.

The same logic runs through the memory track: "A fixed Contriever encoder embeds the question and the turn pairs, and the five most similar pairs are inserted into a brief answer prompt... Every candidate therefore receives the same retrieved evidence for a query. Differences in score reflect its use of that evidence rather than a different retrieval result."

What it costs — and the paper says so itself. Two costs, and the authors flag both.

The first is a ceiling. If Gemma-3-27B-IT's description of a geometry diagram omits an angle, no candidate can recover it. The measured "visual mathematical reasoning" of every model is capped by one frozen captioner that is not itself being evaluated. The vision track measures reasoning-over-description, which is a real and useful capability, but it is not the same thing as visual reasoning.

The second is a distribution shift, and the paper names it explicitly: "These rendered queries read as machine-written descriptions of a figure, a distribution that departs from the natural questions the routers are trained on, so the track also tests how well a router generalizes when the query type shifts." That is an admirable disclosure. It also means the vision numbers in Table 2 are partly measuring out-of-distribution robustness, not in-distribution routing skill — which is one reason they are so noisy.

The tracks that are genuinely new

Memory. LoCoMo and LongMemEval contain "questions whose supporting facts may be separated from the query by many turns or sessions. Some can be answered by recovering one explicit fact, whereas others require combining facts across sessions or resolving a later update against an earlier statement." The routing question is narrowed on purpose: retrieval is fixed, so the only thing being asked is which model best uses the evidence it was handed. And because the evidence is long, this is the track where input tokens dominate — precisely the regime Chapter 3's arithmetic showed reorders the price sheet.

TimeSeries. Built from TSRBench, covering seven skills: anomaly detection, similarity analysis, noise understanding, pattern recognition, inductive reasoning, causality analysis, and event prediction. Each series is rendered as a chart, captioned, and then both the caption and the raw values are given to every model. That dual encoding is the interesting bit: "The raw values preserve exact numerical evidence and are truncated after 200 values when necessary, while the descriptions make higher-level structure explicit."

Video. Charades-Ego records each activity simultaneously from a first-person egocentric camera and a third-person camera. The transformation "randomly withhold[s] one view to form single-view and dual-view queries." That makes video "the regime in xRouteBench where the available modality varies from query to query, and it tests whether a router still selects the right model when the visual evidence is partial." Three classification tasks are built from the annotations — predict the activity, the verb, or the object.

Personalized. This one has a completely different supervision shape, so read it carefully. Open-ended prompts come from MT-Bench and Chatbot Arena. For each dialogue, two of the eighteen candidates are sampled at random and each generates a response under the same conversation history. Then DeepSeek-V3.1, conditioned on a persona, compares the two and returns a preference or a tie. The 200 personas are sampled from PersonaHub — things like "A 71-year-old retired nurse from Italy, volunteering in hospice care" and "A 34-year-old scientist from London who is a social media influencer."

The critical implementation detail: "The persona is supplied to the judge only, not to either candidate model." Think about what that means. Both candidates answer the same question with no idea who is asking. The persona enters only in the grading. So the router's job is not "help the model adapt to the user" — it is "predict which unadapted model this particular user would happen to prefer." A subtler task, and a cleaner one, because the candidate responses stay comparable across personas.

What the design principle buys the ecosystem

"Adding a new application requires only a transformation script and a registered metric." Which is the same modularity claim as Chapter 4's, now applied to tasks instead of models. It is what makes the benchmark a living object rather than a frozen artefact: a new modality is a script, not a fork.

Are the seven track scores even on the same scale?

Here is a question the results table quietly assumes an answer to. The average in Table 2 adds a token-level F1 from LoCoMo to a multiple-choice accuracy from MMLU to an exact-match rate on geometry. All three are numbers between 0 and 100. Are they comparable?

Not really, and the evidence is in the table itself. Look down the LoCoMo column across all fourteen routers: 25.44, 26.59, 25.24, 27.64, 26.78, 24.49, 25.70, 25.89, 24.93, 25.94, 25.40, 24.60, 24.70, 24.60.

The maximum is 27.64 and the minimum is 24.49. The entire spread is 3.15 points. Divided by seven, that column contributes at most 0.45 points of variation to anyone's average — while contributing a near-constant 25.7 to everybody's total.

Compare the generic column, which runs from 12.98 to 80.56: a spread of 67.58, contributing up to 9.65 points of variation after division by seven. One track drives twenty-one times as much of the ranking as another, and neither fact is visible from the Avg column.

The prompts, and why they are published

Appendix F prints every template. That is unglamorous and it is the difference between a reproducible benchmark and a set of numbers. A few, because they show the design thinking:

Multiple choice: "Answer the following multiple-choice question by selecting the correct option (A, B, C, or D). You MUST put your final answer letter in a parenthesis." The parenthesis requirement is a parsing contract — without it, grading eighteen models with eighteen different formatting habits becomes its own research project, and format compliance would silently contaminate the accuracy scores.

MATH and AIME: "Make sure to put the answer (and only answer) inside \boxed{}." Same idea, borrowed from the source benchmark's own convention.

MBPP: "Implement the function with no irrelevant words or comments. Put your code in this format: [BEGIN] <Your Code> [Done]." Delimiters so an execution harness can extract runnable code from prose.

SQuAD: "Answer the question using the provided context. If the answer is not in the context, output noanswer." An explicit abstention token, because SQuAD contains unanswerable items.

Notice the shared shape: every template imposes an extractable output format. That is not fussiness. When you are grading 85,806 responses automatically, the fraction you can parse is a hard multiplier on every number you report, and a model that answers correctly but formats freely is scored as wrong. The templates are how you keep that multiplier near one, equally, for every candidate.

The paper is also explicit about the system/user split: "the fixed instruction is sent as a system message when the API supports that role; otherwise, the two parts are concatenated with [System Instruction] and [User Query] delimiters. Thus, all candidate models receive the same logical prompt for a given query." Eighteen endpoints across two providers do not all support the same message roles, and that sentence is how the comparison survives it.

Two more details from the personalized track

The judge's cost is excluded. "The judge's own cost is excluded from the reported cost." Correct, and worth saying out loud: the judge is measurement apparatus, not part of the deployed system. Including it would inflate every router's cost by the same constant and tell you nothing.

Ties are worth half. Answers are scored "as win, tie, or loss (1, 0.5, 0)." So a Table 3 accuracy of 50.00 is what you would get by always losing half and tying half, or by winning half and losing half — and the floor is not zero, it is whatever tie rate the judge produces. Smallest-LLM's 42.53 therefore means it is genuinely losing most head-to-heads, not merely failing to win.

The one thing the transformation cannot preserve

A closing caution on the everything-becomes-text principle, applied to the time-series track. Every series is captioned and its raw values are appended, "truncated after 200 values when necessary."

Truncation is lossy in a way that interacts badly with the task list. Anomaly detection cares about a single spike; if the spike is at index 400 of a 600-point series, it is gone. Event prediction cares about the tail. The caption is meant to carry the higher-level structure that truncation destroys — "the descriptions make higher-level structure explicit" — but a paragraph of prose is a lossy summary too, and the captioner is a 27B model looking at a rendered chart.

None of this makes the track invalid. It makes it a measurement of a specific thing: reasoning over a caption plus a truncated numeric prefix. Every candidate faces the identical handicap, so the routing comparison is fair. Just do not read 63.78 on TimeSeries as "EloRouter is good at time series."

What routing decision each track actually tests

The tracks are not five samples of one thing. Each isolates a different property of the routing problem, and reading them that way makes the results legible.

Read the "held constant" column as the experimental design. Every track fixes the thing that would otherwise confound the routing comparison. That is the same discipline as the shared pipeline in Chapter 4, applied inside each task.

The memory pipeline, step by step

Worth walking, because it is the clearest example of holding a confound constant.

And a detail that shows the care taken. For LoCoMo, whose template presents retrieved turn-pair chunks: "If no chunk is retrieved, the context field is replaced by No relevant information available." LongMemEval, whose template presents whole sessions rather than chunks, gets its own analogous constant, and the paper frames it as an empty history rather than a failed retrieval: "Here, an empty history is represented by No relevant chat history available." Two datasets, two explicit empty-context strings, each worded for the shape of its own prompt.

Notice what that buys the benchmark. Without a fixed string, an empty retrieval produces a malformed prompt that eighteen models each mishandle differently — some apologise, some hallucinate, some emit the literal template. The score difference on those queries would then measure prompt-robustness, not routing. With the constant, the empty case becomes a routable condition: every candidate sees the identical degenerate prompt, so any remaining difference is still attributable to the model. This is the same discipline as fixing the retrieval result across candidates, applied to the boundary case.

The generic mix is not evenly weighted either

The same imbalance that afflicts Table 2's average appears one level down, inside the generic track. Seven subtasks have 500 items; four have 50; HumanEval has 16 and AIME has 13.

Compute the shares. AIME is 13 ÷ 3,729 = 0.35% of the track. HumanEval is 16 ÷ 3,729 = 0.43%. Together, the two hardest reasoning benchmarks in the mix are 0.78% of it.

So a router that is perfect on competition mathematics, versus one that gets every AIME item wrong, differ by at most 0.35 points on the generic track — and by 0.35 ÷ 7 = 0.05 points on the headline average. The generic score is, arithmetically, a knowledge-QA and grade-school-maths score with a code component: MMLU, MMLU-Pro, ARC, OpenBookQA account for 2,000 of 3,729 items, and GSM8K plus MATH another 1,000.

Three tasks hide inside the video track

A last detail that is easy to miss. Charades-Ego's 27 test items are not 27 instances of one task. The paper builds "three classification tasks from the annotations, predicting the activity, the verb, or the object of the depicted action, each answered by a compact identifier drawn from the task label inventory."

So roughly nine items per task, across a further split into single-view and dual-view regimes. Which means the video column of Table 2 is aggregating three different tasks under two evidence regimes across 27 items total. That is not enough to say anything about any of the six cells, and it is worth remembering when Chapter 7 computes the standard error.

The track's value is not its numbers. It is that it establishes a regime — "the available modality varies from query to query" — that no previous routing benchmark contained, and it gives the next benchmark something to scale up.

The seven time-series skills, and why they route differently

The TimeSeries track covers "anomaly detection, similarity analysis, noise understanding, pattern recognition, inductive reasoning, causality analysis, and event prediction." Seven skills across 127 test items is about eighteen per skill — small, but the list itself is instructive because the skills make different demands on the dual encoding.

The paper acknowledges the split directly: "The raw values preserve exact numerical evidence... while the descriptions make higher-level structure explicit." And it notes a further wrinkle in how the two are combined — for perception and causality items the two-source block replaces a marker in the source question, while for the other two task types it is appended. Small, and it means the prompt structure is not uniform across the track.

One more careful design touch: the captioner receives "one instruction sampled uniformly from 20 equivalent phrasings." That is prompt-variation as a hedge against any single phrasing's idiosyncrasies leaking into every caption. It is the same instinct as prompt ensembling, applied to data construction rather than inference.

One personalized supervision record, start to finish

The personalized track's construction is unusual enough to walk through concretely.

Two properties fall out of this design and both are deliberate. First, because the persona reaches only the judge, "this construction keeps the candidate responses comparable while making the routing target sensitive to the user utility represented by the judge." The same two responses could be labelled differently for two personas, which is precisely the signal a personalized router must learn.

Second, because the router never reads the persona, it cannot cheat by pattern-matching on demographic text. It has to infer taste from votes — which, as Chapter 5 noted, is the situation every real product is in.

What xRouteBench still cannot express

The benchmark closes four gaps in prior work. Being fair means naming the ones it leaves open.

None of these is a criticism of a first unified benchmark — it closed the four gaps it set out to close, and each row above is a paper someone can now write because the infrastructure exists. That is what a foundation is supposed to leave behind.

A real caption, and a real problem solved through it

The paper prints two of the descriptions its captioner produces (Figure 8), which lets us see exactly what a routed vision query looks like. Here is the Geometry3K one, verbatim:

The question asks for x. Now — can you answer it from the description alone? Yes, and doing so is the best possible demonstration of what this track measures.

The intersecting chords theorem says that when two chords cross inside a circle, the products of their segment lengths are equal. Chord one is split into 4 and 6; chord two into x and 8. So:

Three things to notice about that. First, the caption contains every number needed — the captioning prompt's instruction to "report every visible number, symbol, angle, length, and relationship" did its job. Second, it contains the crucial relational fact ("the two chords cross at a point inside the circle") without which the theorem does not apply. Third, and critically, it does not contain the answer — the prompt's instruction to "withhold the solution" held.

That is the track working. Every one of the eighteen candidates receives that paragraph, and the ones that know the intersecting chords theorem get 3 while the ones that do not, do not. The image encoder is out of the loop entirely; what is being measured is geometric reasoning over prose.

And here is the honest flip side, visible in the same example. If the captioner had written "one chord is divided into segments of length 4 and 6, and the other into segments of length x and 8" but omitted "the two chords cross at a point inside the circle", the problem would be unsolvable for every candidate, and every one would score zero on an item that a model looking at the picture could answer. The captioner is a single point of failure for the entire column.

The second published caption, for a MathVista physics item, shows the same design under more pressure: it describes a block sliding into a spring and adds that "a caption states that the spring force does negative work, decreasing speed and kinetic energy" — text that was in the figure and had to be transcribed. Faithful transcription of embedded text is a distinct capability from describing geometry, and the whole track depends on one 27B model doing both well.

Why the video track withholds views on purpose

The Charades-Ego construction has one element that is easy to read as arbitrary and is in fact the point of the track. For each paired clip the pipeline matches the two viewpoints by identifier, selects the action segment whose egocentric and exocentric occurrences overlap most closely in time, "and we then keep the first-person view, the third-person view, or both at random."

Randomly discarding evidence sounds like sabotage. It is a controlled manipulation of how much a query provides, and it creates the one regime no other routing benchmark has: queries from the same task where the available modality differs. The paper says exactly what this tests — "whether a router still selects the right model when the visual evidence is partial."

Think about why that is a distinct routing skill. A dual-view query has more information, so more candidates can succeed and the cheap ones become viable. A single-view query is harder, so it may need the expensive ones. A router that cannot tell the two apart will either overspend on easy queries or underspend on hard ones — and the signal distinguishing them is in the query text, since a merged two-view description reads differently from a one-view description.

With 27 test items across three tasks and two regimes, the track cannot measure whether any router has this skill. It can, and does, establish that the regime is constructible and that a benchmark can hold it. That is a reasonable thing for a first version to accomplish.

Sizes and regimes, side by side

A closing summary that will be useful when Chapter 7 starts computing standard errors. Every track, its size, and what its size lets you conclude.

Read the last two columns together and the benchmark's honest shape appears: one track that can rank routers, and six that establish regimes. Both are valuable and they are not the same kind of contribution. The six exist so that the seventh is not the only thing anyone measures — and so that the next version of this benchmark knows which regimes are worth scaling up.

If you were scaling it up, the ordering is obvious from this table: Charades-Ego first, because it is the only source of the modality-varying regime and it has 27 items; then Geometry3K and MathVista, because the vision tracks show the largest between-router spread and therefore the most signal per additional item.

Chapter 6: The Router Zoo

Seventeen routers, one interface. This chapter derives the interesting ones from the five components, with arithmetic, so that when Chapter 7's table arrives you are reading mechanisms rather than acronyms.

The organising claim from Appendix B: the routers "share the candidate pool and task interface described above, but they expose different information to the routing decision." Everything below is a statement about how much of s_t a method looks at, and what it does with it.

The rule-based floor

Smallest-LLM and Largest-LLM "ignore the query and always select the candidate with the smallest or largest declared parameter count." In five-component terms: E_q is a constant, E_m is a single scalar (parameter count), g is that scalar, d is arg max or arg min, and ℒ does not exist.

They are not strawmen. They are "the two simple deployment policies against which query-aware routing should be compared" — and, as Chapter 0 showed, one of them is what most production systems actually do.

EloRouter: the query-blind learned baseline

EloRouter is the most instructive router in the zoo precisely because it barely qualifies as one. It is "query-independent, but it replaces a fixed parameter-count rule with a global ranking estimated from pairwise outcomes in the routing data. It therefore captures which model is strongest on average while deliberately discarding the variation between individual queries."

Elo itself is a rating system from chess. Each candidate holds a scalar R. The expected score of A against B is:

Worked example. Suppose model A sits at R_A = 1500 and model B at R_B = 1600. Then (R_B − R_A) ÷ 400 = 100 ÷ 400 = 0.25, and 10^0.25 = 1.7783. So E_A = 1 ÷ (1 + 1.7783) = 1 ÷ 2.7783 = 0.360. A is expected to win 36% of head-to-heads.

Now A loses one. With the standard update R ← R + K(S − E) and K = 32, A's new rating is 1500 + 32 × (0 − 0.360) = 1500 − 11.52 = 1488.48. B gains the same 11.52. Over thousands of recorded outcomes the ratings converge to a global ordering, and EloRouter simply always dispatches to the top of it.

(The paper reports that EloRouter uses "a scalar rating (Elo)" fit from "logged pairwise model outcomes"; it does not publish its K or initial rating, so the constants above are the standard chess ones, shown to make the mechanism concrete.)

The embedding-space trio: kNN, SVM, MLP

All three "make the selection query-specific from its embedding." They share E_q (an off-the-shelf sentence embedding) and differ entirely in g.

kNNRouter "retrieves similar training queries and transfers their observed best-model choices through a vote or distance-weighted vote, requiring no fitted decision function beyond the stored examples." There is no training. There is a table.

Worked example: why the two vote rules are different decision rules. Take k = 5 neighbours of an incoming query. For each, we know from the query–model matrix which candidate did best. Their cosine similarities to our query, and their winners:

Plain vote: A gets 1, B gets 3, C gets 1. B wins.

Distance-weighted vote: A gets 0.95. B gets 0.30 + 0.28 + 0.25 = 0.83. C gets 0.20. A wins.

Same neighbours, same embeddings, same stored matrix — opposite answers. And the weighted answer is the better one here, because neighbour 1 at similarity 0.95 is genuinely about the same thing while neighbours 2–4 at ~0.28 are barely related. This is the g/d separation from Chapter 2 with teeth: the "router" you built is not determined by your encoder, it is determined by a rule you might have picked without thinking.

SVMRouter and MLPRouter "fit discriminative boundaries over the same embedding space, using a kernel classifier and a multilayer perceptron, respectively." Same E_q, same data, different hypothesis class. In Table 2 they average 45.10 and 39.34 — a 5.76-point gap from nothing but the choice of classifier. That gap is the paper's whole argument for controlled comparison, in one row.

MFRouter: the recommender-system framing

MFRouter "takes a different view of the logged query–model matrix: it learns latent representations for both sides and selects the model with the strongest query–model interaction."

If that sounds like collaborative filtering, it is. Queries are users, models are items, the P matrix is the ratings matrix, and the task is predicting an unobserved entry. E_q and E_m both emit vectors in R^r, and g is their dot product.

Worked example, with r = 3. Query latent p = (0.8, −0.3, 0.5). Three model latents:

q_A = (0.6, 0.4, −0.2): g = 0.8(0.6) + (−0.3)(0.4) + 0.5(−0.2) = 0.48 − 0.12 − 0.10 = 0.26
q_B = (0.2, −0.9, 0.7): g = 0.16 + 0.27 + 0.35 = 0.78
q_C = (0.9, 0.1, 0.1): g = 0.72 − 0.03 + 0.05 = 0.74

Greedy arg max picks B. Now attach real prices — A is gemma at $0.10 blended, B is cogito at $1.25, C is gpt-oss-120b at $0.375 — and apply a cost-aware d with a penalty of 0.5 score-units per dollar-per-million:

A: 0.26 − 0.5(0.10) = 0.26 − 0.05 = 0.21
B: 0.78 − 0.5(1.25) = 0.78 − 0.625 = 0.155
C: 0.74 − 0.5(0.375) = 0.74 − 0.1875 = 0.5525

C wins, and B — the scorer's favourite — drops to last. The scoring function did not change one bit. This is Chapter 2's most useful idea made arithmetic: g holds the beliefs, d holds the budget, and the budget can overturn the beliefs.

RouterDC and GraphRouter: the two strongest single-turn scorers

RouterDC "learns query–model matching with dual contrastive objectives over query–model, query–query, and cluster-level relations." Contrastive learning pulls a query's representation toward the models that solved it and pushes it away from those that failed — the same geometry as a text-image contrastive model, with "models" in the place of "captions."

GraphRouter "passes information over a query–model interaction graph and predicts the quality of an unobserved query–model edge." This is link prediction. Queries and models are nodes; an observed (query, model, score) is a labelled edge; the router predicts labels for edges it has not seen. Message passing means a query's representation is informed by other queries that share models with it, which is exactly the transitive inference collaborative filtering is good at.

These two take the top two places on the generic track in Table 2 — 80.56 and 80.54, separated by 0.02 points, which on 3,729 queries is under one answered question. And they land in completely different places on the overall average. Chapter 7 explains why.

The cost-aware cascades: Hybrid LLM and AutoMix

These two are the only single-turn methods whose cost varies per query, and the mechanism differs sharply.

Hybrid LLM "learns from the quality gap between the smallest and largest candidates, then uses a thresholded prediction to decide whether the smaller model is sufficient." One call, always. The decision is made before spending anything, from the query alone. Cheap, but it must predict difficulty blind.

AutoMix "first obtains a draft from the smaller model and a verification signal for that draft; it retains the draft when the signal is reliable and otherwise escalates the query to the larger model." One call sometimes, two calls sometimes. The decision is made after seeing evidence, which is strictly more informed — and strictly more expensive.

Worked example: when does escalation pay? Let the small model cost c_s and the large one c_l, and let the escalation rate be r (the fraction of queries the verifier rejects). AutoMix's expected cost per query is:

because you always pay for the draft, and pay for the large model only on the escalated fraction. Put in real prices for a 200-in / 150-out query: gemma at $0.000035 and cogito at $0.0004375. At r = 0.3, E[c] = 0.000035 + 0.3(0.0004375) = 0.000035 + 0.00013125 = $0.000166 — about 38% of always-large. At r = 0.9, E[c] = 0.000035 + 0.00039375 = $0.000429, which is 98% of always-large plus the draft you threw away. The break-even is where c_s + r·c_l = c_l, i.e. r = 1 − c_s/c_l = 1 − 0.08 = 0.92.

So a cascade only makes sense if its verifier can accept most drafts. Above a 92% escalation rate in this example, you would have been better off skipping the draft entirely. The paper places both methods "with the single-turn family because they return one final model answer through a fixed cascade" — but notes pointedly, "Their extra calls are nevertheless part of the inference cost."

CausalLM Router: the router that is a language model

CausalLM Router "verbalizes the query and the available candidates, then fine-tunes a causal language model to generate the selected model name." E_q, E_m, and g are all one forward pass, as Chapter 2 described.

The appeal is obvious: no feature engineering, and adding a candidate is adding a name to a list. The cost is equally obvious and shows up in the results: it averages 38.22 in Table 2 — above Smallest-LLM's 37.94 but below Largest-LLM's 38.72, so a fine-tuned generative router with the whole query in front of it does not clear the policy that ignores the query and always picks the biggest model — and it comes dead last in Table 4's real-user evaluation at 27.97 — well under the 50% you would get by guessing on a binary preference. That is a failure mode worth naming: a generative router can produce an output that is fluent, plausible, and simply a bad choice, and nothing in the architecture penalises confident nonsense.

The multi-turn family

Three routers, all of which "treat intermediate responses as part of the routing state."

Router-R1 is a pretrained routing agent that "reasons over the query and the results gathered so far. At each step it can issue a search call to a suitable specialist, incorporate the returned evidence, or terminate and produce an aggregate answer; its policy is trained with trajectory-level reinforcement learning rather than pointwise best-model labels." And the accounting line: "The cost of its reasoning, search, and final aggregation calls is included in the routed trajectory."

kNN-MultiRound "first breaks a complex query into a small set of sub-queries, applies the kNN routing rule to each one, and combines the resulting answers." LLM-MultiRound "uses an LLM to express both the decomposition and the model choice in text." Both "can assign different candidates to different parts of a problem, but they also introduce decomposition and aggregation calls that a one-shot router does not pay for."

The personalized family

Two routers, both graph-based, both trained on comparisons.

GMTRouter "represents users, sessions, queries, candidate models, and responses as nodes in a heterogeneous interaction graph, allowing a decision for the current query to draw on earlier interactions from the same user." Five node types, so an edge can carry "this user, in this session, asked this query, got this response from this model, and preferred it."

PersonalizedRouter "similarly uses a graph-based scorer, while explicitly incorporating user features and task descriptions alongside the query and candidate representations."

Both are "trained from comparisons between candidate answers, so a high score means that a model is predicted to be preferred for this user and query, not simply that it has the highest population-average task score." Which is precisely the distinction between Table 3 and Table 2, and Chapter 9 will show that the distinction is real and that the two personalized routers rank differently against a simulated judge than against real humans.

The full inventory

Seventeen rows. Read the middle column top to bottom and you are reading Chapter 1's state triple being progressively unlocked: parameter counts only, then the query, then the query plus intermediate results, then the query plus the user.

Inside the discriminative routers, with shapes

MLPRouter. The forward pass is three lines, and the shapes tell you what it can and cannot represent:

where e is the 384-dimensional query embedding. Total parameters: 384 × 256 + 256 + 256 × 18 + 18 = 98,304 + 256 + 4,608 + 18 = 103,186. About a tenth of a megabyte. That is the whole router.

And notice the second weight matrix: W₂ has one row per candidate. This is the same fixed-head structure that a classifier has — adding a nineteenth model means adding a row, which means retraining. The model encoder is not a separate object here; it is baked into the output layer, which is precisely why the model-encoder column of Chapter 2 mattered.

SVMRouter. A kernel classifier scores by comparing the query to stored support vectors:

With an RBF kernel, K(e, e_i) = exp(−γ · ||e − e_i||²), which decays with distance. That makes an SVM a smoothed, learned relative of kNN: kNN takes the k nearest and votes uniformly; the RBF-SVM takes all support vectors and weights them by an exponentially decaying distance, with the weights α_i fitted rather than fixed at one. Given that framing, SVMRouter's 45.10 against kNNRouter's 41.30 in Table 2 is not mysterious — it is the same idea with the weights learned instead of assumed.

GraphRouter's message passing, worked small

Link prediction sounds abstract until you push numbers through it. Build the bipartite graph: query nodes on one side, model nodes on the other, an edge wherever the matrix has an observed outcome.

One round of message passing updates a model node by aggregating from its neighbours. Say model m has been observed on three training queries with representations (all in R² for legibility) and outcomes:

An outcome-weighted mean aggregation gives m the representation (0.9 + 0.8) ÷ 2, (0.1 + 0.3) ÷ 2 = (0.85, 0.20) — the centroid of what it solves, with the failure excluded. Now a new query arrives at (0.85, 0.15). Its dot product with m is 0.85 × 0.85 + 0.15 × 0.20 = 0.7225 + 0.03 = 0.7525. A second model whose solved-centroid is (0.15, 0.88) scores 0.85 × 0.15 + 0.15 × 0.88 = 0.1275 + 0.132 = 0.2595. The first model wins, and it wins because the graph told us which region of query space it succeeds in.

Real message passing learns the aggregation and runs several rounds, so a query's representation is also informed by other queries that share models with it — the transitive step that plain kNN cannot make. But the mechanism is the one above, and it explains both GraphRouter's strength on the large generic track and its fragility elsewhere: with only 27 video queries there is almost no graph to pass messages over.

RouterDC's contrastive objective, in words and shapes

Contrastive learning needs a positive and a set of negatives. For a query q, the positives are the candidates that solved it and the negatives are those that failed. The loss pulls E_q(q) toward the positives' embeddings and pushes it from the negatives':

which is a softmax cross-entropy over the eighteen candidates, with temperature τ. RouterDC is described as using dual contrastive objectives, "over query–model, query–query, and cluster-level relations" — so in addition to the query-to-model term above, there are terms pulling similar queries together and organising them into clusters. That extra structure is why it fits the large generic track so tightly, and it is a reasonable hypothesis for why it transfers so badly to Geometry3K, where it scores 16.39: a representation shaped by clusters found in 3,729 text queries has no cluster that machine-written geometry descriptions belong to.

Hybrid LLM's threshold, derived

Hybrid LLM predicts the quality gap between a small and a large model and thresholds it. Write the predicted gap as Δ(q) = predicted perf_large − predicted perf_small. Then the rule is: use the small model unless Δ(q) > θ.

What should θ be? Send to the large model only when the expected quality gain exceeds the extra cost in quality-units, which is exactly Equation 1:

Worked example. Small = gemma at $0.000035, large = cogito at $0.0004375, on our 200-in/150-out query. The cost difference is $0.0004025. At λ = 20,000 points per dollar, θ = 20,000 × 0.0004025 = 8.05 points. So escalate only when you predict the big model will be more than 8 points better on this query. At λ = 5,000, θ drops to 2.01 points and you escalate far more often. At λ = 100,000, θ is 40.25 and you almost never escalate.

One scalar, no retraining, and it sweeps the entire frontier. This is the cleanest illustration in the paper of why cost-awareness belongs in d.

What each router costs to run

Read the last column as the c(τ) column. Everything above AutoMix costs one candidate call and nothing else; everything from AutoMix down pays for calls that never appear in the answer. That single structural split predicts most of Chapter 8's cost-sweep reversals before you see a number.

Choosing k, and the failure mode at both ends

kNNRouter has exactly one important hyperparameter and it behaves the way it does in every other nearest-neighbour method.

k too small. At k = 1 the router copies the best model of the single nearest training query. If that query's label is noisy — and Chapter 4 argued that some fraction of the matrix is — the noise passes straight through. Variance is high; the decision boundary is jagged.

k too large. As k grows toward N_train, the vote is dominated by the global frequency of each candidate in the training labels, and the router converges to "always pick the model that was best most often." Which is EloRouter. kNN with k = N is a query-blind rating router, and that identity is the cleanest way to see what the query-awareness is actually buying.

So the k sweep interpolates between a memorising router and a global-preference router, and the useful setting is somewhere in the middle, found by validation. Note also that the distance-weighted variant is a softer version of the same interpolation: it never fully ignores far neighbours, but it downweights them continuously rather than by a hard cut-off.

Elo's blind spot, stated precisely

EloRouter fits one scalar per candidate from pairwise outcomes. That scalar is a sufficient statistic for exactly one model of the world: that model A beats model B with a probability that depends only on their two ratings.

Formally, Elo assumes a total order with a consistent strength gap. It cannot represent non-transitivity: A beats B on maths, B beats C on code, C beats A on long context. Fit Elo to that data and you get three ratings that are wrong about every pairing, in a way that averages out to something plausible.

And non-transitivity is exactly what heterogeneous pools have. That is the whole premise of routing. So EloRouter is fitting a model that the data provably violates — and it still finishes third of the fourteen rows in Table 2, first on three of seven tracks.

MFRouter and the cold-start problem

Matrix factorization learns a latent vector for each query and each model from observed interactions. A brand-new candidate with no observations has no latent vector, and the model has literally nothing to say about it — the standard cold-start problem.

Three standard remedies, all applicable here:

The third is the bandit answer, and it is why the paper's component list includes decision rules that "sample for exploration in online settings." An offline benchmark never needs exploration because every cell is already observed. A production router needs it every time a model ships.

Picking a router by constraint

Practical synthesis. You rarely choose a router by reading a leaderboard; you choose it by which constraints you have.

AutoMix's verifier, and the two ways it can be wrong

Cascades live or die on the verifier, and its two error types have completely different consequences. Lay them out.

The asymmetry: a false accept costs quality and a false escalate costs money. So the verifier's threshold is another instance of the λ question, and it should be set by the same arithmetic — escalate when the probability the draft is wrong, times the quality you would gain, exceeds λ times c_l.

And note where AutoMix sits relative to Hybrid LLM on the information axis. Hybrid LLM predicts difficulty from the query alone, before spending anything: cheap, blind. AutoMix predicts adequacy from an actual draft: informed, and it has already spent c_s to get there. That is a genuine trade, and which side wins depends on how predictable difficulty is in your domain from the query text — which is exactly the quantity the query–model matrix measures.

Why a generative router is fragile

CausalLM Router's poor showing has a mechanical explanation worth spelling out, because building a router as an LLM prompt is a near-universal first instinct.

The router is asked to generate a model name. That name is a token sequence, and the decoder assigns it probability by the usual next-token machinery. Three consequences:

Name frequency becomes a prior. If "llama" appears far more often than "rnj" in pretraining, the decoder is biased toward llama-shaped continuations before any routing evidence enters. Nothing corrects this.

Tokenisation is arbitrary. "gpt-oss-120b" and "gpt-oss-20b" share a long prefix and differ by one token. Two candidates that differ by 6× in size and 3× in price are one sampling step apart in output space.

There is no calibration signal. Cross-entropy on the correct name rewards confidence on the training distribution. Nothing penalises being confidently wrong off it — which is exactly Table 4's 27.97%.

Chapter 7: Results, Read Honestly

Here is Table 2, the paper's main result, under the performance-first setting (α, β) = (1.0, 0.0) — quality only, cost ignored. Seven test sets and their unweighted average. Then we take it apart.

Step one: verify the average is what you think it is

Never trust an "Avg" column you have not reproduced. Take GraphRouter's row and add the seven scores by hand:

Divide by 7: 318.24 ÷ 7 = 45.463, which rounds to the printed 45.46. Confirmed — the average is the plain unweighted mean of the seven track scores.

Check Largest-LLM the same way: 70.29 + 26.59 + 35.57 + 37.70 + 33.00 + 22.22 + 45.67 = 271.04, and 271.04 ÷ 7 = 38.72. Confirmed again.

So the headline comparison is 45.46 against 38.72. The absolute gain is 45.46 − 38.72 = 6.74 points. The relative gain is 6.74 ÷ 38.72 = 0.1741, i.e. +17.4%.

Step two: reconstruct the abstract's 14.6%

The abstract claims "learned routers achieve a 14.6% relative improvement over the strongest fixed-model baseline." We just computed 17.4%. Both cannot be describing the same arithmetic, so one of two things is true: a different router-baseline pair, or a different aggregation. Let us test the second.

Notice the problem with the unweighted mean. Charades-Ego has 27 test queries and gets one seventh of the average. The generic mix has 3,729 and also gets one seventh. In terms of queries, the video track is over-weighted by a factor of (1/7) ÷ (27/4,459) = 0.1429 ÷ 0.00606 = 23.6×.

So compute the query-weighted average instead: multiply each track score by its test-set size, sum, and divide by 4,459. For GraphRouter:

328,105.24 ÷ 4,459 = 73.58. Do the same for Largest-LLM — 262,111.41 + 8,349.26 + 3,592.57 + 2,299.70 + 3,300.00 + 599.94 + 5,800.09 = 286,052.97, then ÷ 4,459 = 64.15.

Relative gain: (73.58 − 64.15) ÷ 64.15 = 9.43 ÷ 64.15 = 0.1470, i.e. +14.7%.

Step three: notice how few questions some of these numbers rest on

This is the part of reading a results table that most people skip, and it changes everything here.

Charades-Ego has n = 27. One correct answer is worth 1 ÷ 27 = 3.70 percentage points. So every video score must be a multiple of 3.70 — and look: the entire column contains only four distinct values. 33.33 is 9 correct. 29.63 is 8. 25.93 is 7. 22.22 is 6. That is it. Fifteen routers, and the whole video comparison spans six to nine correct answers out of twenty-seven.

Geometry3K has n = 61, so one answer is 1.64 points. Verify a few: 45.90 × 61 ÷ 100 = 28.0 correct; 42.62 → 26.0; 27.87 → 17.0; 16.39 → 10.0. All integers, as they must be. So EloRouter's apparently decisive 45.90 over SVMRouter's 42.62 on Geometry3K is 28 correct versus 26 — a two-question difference.

TimeSeries has n = 127, one answer worth 0.787 points: 63.78 is 81 correct, 62.99 is 80, 45.67 is 58. EloRouter beats GraphRouter on this track by exactly one question.

Finding (i): no single router dominates

Now the finding has teeth. Look across the rows:

RouterDC is first on the generic track at 80.56 — and last of the eleven rule-based and single-turn rows on the overall average at 36.32, below both rule baselines. Only the three multi-turn routers (kNN-MultiRound 23.20, LLM-MultiRound 22.37, Router-R1 22.30) sit lower, which puts RouterDC 11th of the fourteen rows overall. Be precise about which reference class you are ranking within: "last" among the routers you would actually consider deploying is a much sharper indictment than "11th of 14", and the multi-turn block below it fails for an entirely different reason (Finding iii). Its Geometry3K score is 16.39, which is 10 correct out of 61 — the worst of any single-turn router, tied with kNN-MultiRound, and above only Router-R1's 14.75 and LLM-MultiRound's 14.29.

EloRouter, which does not look at the query at all, is first on Geometry3K (45.90), tied first on MathVista (50.00), and first on TimeSeries (63.78). A query-blind router beating query-aware ones on three of seven tracks is a genuinely uncomfortable result, and it is exactly why the paper includes it.

Hybrid LLM, CausalLM, and Smallest-LLM all tie for first on video at 33.33 — nine correct answers. Which, per the sample-size point above, means "first on video" is not a meaningful accolade.

The paper's own summary: "The winner router varies across tasks. For example, RouterDC performs best on Generic LLM Tasks and SVMRouter on LoCoMo. Though GraphRouter attains the best average on xRouteBench, it did not consistently outperform other routers in all tasks."

Under weighting, RouterDC leaps from last to near the top — because its 80.56 on the generic track finally counts for what it is worth — and EloRouter drops, because its strengths were on the three smallest tracks. Neither view is dishonest. They are answers to different questions, and if you only ever see one of them you are being told half a story.

Finding (ii): learned routing beats the strongest fixed baseline

The mechanism is worth restating because it is not "learned routers pick the big model more cleverly." The paper: "always selecting the largest model incurs the highest cost yet delivers only mediocre performance, whereas learned routers select smaller, cheaper models for many queries that the largest model answers incorrectly."

Test it against the table. Largest-LLM scores 22.22 on video — six of 27 — while Smallest-LLM scores 33.33, nine of 27. Three queries exist where the small model is right and the big one is wrong. On TimeSeries, Largest gets 45.67 × 127 ÷ 100 = 58 of 127 and Smallest gets 49.61 × 127 ÷ 100 = 63. Five more. On MathVista, n = 100 makes the arithmetic free: 33 correct against 35, two more. LongMemEval is scored by token-level F1 rather than exact match, so there is no integer count to recover, but the sign is the same — 35.57 against 36.77, a 1.20-point deficit over 101 items.

That is four of seven tracks on which the largest model is worse than the smallest, and every one of those inversions is money a router can pick up. Note carefully what the count is and is not. Four of seven does not mean the small model is better overall. Take the seven per-track differences (Largest minus Smallest) and add them up:

There it is — Chapter 0's 0.78, recovered from the track deltas. So the largest model wins the average by taking three tracks by a lot (+23.72 between them, mostly generic and Geometry3K) and losing four by less (−18.25 between them, mostly video). Counting tracks and averaging tracks are different operations and they disagree here, which is precisely why you should always report both. And under a query-weighted mean neither count applies: the 3,729-item generic track swamps the other six, so Largest-LLM's lead would widen sharply. Three readings, one row of one table.

The routing opportunity survives all three, because a router operates per query, not per average. Four losing tracks means at minimum three video queries, two MathVista queries, five TimeSeries queries and a slice of LongMemEval where the cheap model is right and the expensive one is wrong — and that is only the inversions visible at track granularity. Within the tracks Largest wins, there are more of them still, individually invisible because they are netted against the wins. Aggregate scores hide the very structure routing exists to exploit.

Finding (iii): multi-turn routing does not pay

The multi-turn block is the most dramatic thing in the table. The best multi-turn average is 23.20; the best single-turn average is 45.46. Nearly a factor of two.

The generic column is where it collapses hardest: kNN-MultiRound scores 13.99 and LLM-MultiRound 12.98, against 80.56 for RouterDC on the same queries. That is not underperformance, that is failure — and on a mix that includes four-option multiple choice, where random guessing is 25%.

What went wrong? Two named causes, and neither is "routing is bad":

Cause one: redundancy and overhead. "For many queries, one well-chosen route is sufficient, whereas additional rounds introduce redundant information and computational overhead." Decompose "What is the capital of France?" and you have made it harder, not easier.

Cause two: the scaffolding model. "Multi-turn routers also rely on a base model (Qwen2.5-3B-Instruct) to decompose queries and aggregate responses, making their performance sensitive to the capabilities of this model." Every answer, no matter which 671B specialist produced it, passes through a 3B model on the way out. That model is the bottleneck, and it is not in the pool.

Add the Chapter 1 arithmetic and the picture completes: a multi-turn trajectory pays for decomposition, several sub-query calls, and aggregation, so under any λ > 0 it starts from a deep hole and has to climb out on quality alone. It does not.

Finding (iv): personalization pays, and the encoding matters

Table 3, the personalized track: 308 preference prompts, scored by DeepSeek-V3.1 conditioned on a persona, win/tie/loss mapped to 1/0.5/0.

Both user-conditioned routers take the top two places, which "confirms the benefit of conditioning routing decisions on user context." The margin over the best user-agnostic router, EloRouter at 66.40, is 68.78 − 66.40 = 2.38 points.

And the finer point: GMTRouter's edge over PersonalizedRouter is 68.78 − 67.86 = 0.92 points, which the paper reads as showing "that the way user context is encoded and integrated remains important." Both condition on the user; they differ in how, and how matters — though 0.92 points on 308 prompts is, again, a handful of comparisons, so hold it loosely.

One thing to flag: EloRouter at 66.40 is third. A router that ignores both the query and the user is within 2.4 points of the best personalized method. Part of persona preference is simply "some models write better answers," and a global rating captures that for free. The genuinely personal signal is the remaining 2.4 points — real, but smaller than the framing suggests. Chapter 9 will show that on real humans the picture changes substantially.

How much of this table can the sample sizes actually resolve?

We have been reading differences of one and two points as if they were real. Let us check whether they can be, with the standard tool: the binomial standard error of a proportion,

Run it for each track at a representative accuracy:

Read the last two columns together. On the video track the entire observed spread across all fourteen rows — eleven points, and only four distinct values — is 1.3 standard errors. That column can distinguish approximately nothing. On the generic track the spread is ninety standard errors, and differences of two points are unambiguously real.

The same tool applies to Table 3. With n = 308 preference prompts at p = 0.65, SE = √(0.2275/308) = 0.0272, i.e. ±2.7 points. GMTRouter's 0.92-point edge over PersonalizedRouter is a third of one standard error. The paper's own claim — that the gap "shows that the way user context is encoded and integrated remains important" — is a reasonable interpretation of a directional result, but it is not something 308 comparisons can establish, and the honest reading is "both personalized routers beat all user-agnostic ones, and they are tied with each other."

The per-track winners, laid out

Seven tracks, six distinct winners, and four of the seven margins are one to three questions. That is the empirical content of "no single router dominates" — and, read carefully, it also says something the paper does not claim: on these sample sizes, most of the per-track "wins" are not wins.

The genuinely robust statements the table supports are the large ones. RouterDC and GraphRouter really are far better than everyone else on the generic mix, by twenty-something points over the rule baselines. The multi-turn family really is catastrophically worse. Learned routers really do beat both fixed policies on average. Those survive any reasonable error bar.

Diagnosing the multi-turn collapse further

Two extra observations from the table that sharpen the cause.

The collapse is not uniform across tracks. On LoCoMo, the multi-turn routers score 24.60, 24.70, 24.60 — essentially identical to every single-turn router's ~25.5. On the generic mix they score 35.64, 13.99, 12.98 against ~70. Why? Because LoCoMo's F1 metric has that compressed dynamic range: even a badly aggregated answer shares tokens with the gold phrase. The generic mix, graded by exact match and answer verification, has no partial credit — a mangled aggregation scores zero. The collapse is visible exactly where the metric is unforgiving, which points at aggregation, not at model selection, as the failure.

Router-R1 is much better than the two baselines on the generic mix — 35.64 against 13.99 and 12.98 — while being roughly tied on the other six tracks. Router-R1 is the one trained with reinforcement learning on trajectory rewards, so it has at least learned when to stop, which the two heuristic multi-round baselines never learn. That is direct evidence for the paper's own prescription that "sufficiency estimation [and] early stopping" are the missing pieces: the method that has any of it is 2.5× better than the methods that have none.

"Relative improvement over the baseline" is three different numbers

We have now computed one relative gain two ways. It is worth seeing how much the choice of baseline moves it as well, because "X% better than the baseline" is the most-quoted and least-specified sentence in applied machine learning.

Take GraphRouter's 45.46 and compare it against three defensible fixed-model baselines:

That third row deserves its own paragraph. Compute it by taking, for each track, the better of Smallest and Largest: 70.29, 26.59, 36.77, 37.70, 35.00, 33.33, 49.61. Sum = 289.29, divided by 7 = 41.33. This is a stronger baseline than either rule alone, because it is allowed to know which fixed model suits each track — a fair opponent if your deployment is track-aware.

So the same result is "+19.8%", "+17.4%", or "+10.0%" depending on a choice nobody usually states, and "+14.7%" if you also switch to query weighting. A range from 10 to 20 percent, all honest.

Which routers agree with each other

A useful diagnostic that the table supports but nobody usually runs: look at which routers rise and fall together across tracks. Compare RouterDC and GraphRouter, the two graph-and-contrastive scorers:

Identical on the big text track, and catastrophically different on the three tracks built from machine-written descriptions. The gap is not diffuse — it is concentrated exactly on the query types that Chapter 5 identified as out of distribution.

A hypothesis consistent with the mechanism from Chapter 6: RouterDC's contrastive objective organises the query space into clusters learned from the 3,729 training text queries. A geometry description generated by Gemma-3-27B belongs to no such cluster, so its representation lands somewhere arbitrary and the matching score is noise. GraphRouter's message passing is less committed to a cluster structure and degrades more gracefully.

Note also RouterDC's Geometry3K score of 16.39 — that is 10 correct of 61, and the random baseline for that track is not well-defined but Smallest-LLM manages 17. Scoring below a policy that ignores the query means the router is actively choosing badly, not merely failing to choose well. There is signal in the errors; it is just inverted.

What would it take to reach the oracle?

Chapter 4 defined the capture rate. Applying it here would require the oracle row, which the paper does not print — but we can bound the situation from what is printed.

On the generic track, the best router scores 80.56 and the best fixed policy scores 70.29. The oracle is at least as high as the best cell in each row, which is certainly above 90 for a pool with eighteen heterogeneous models on 3,729 mostly-multiple-choice questions. Take 92 as a conservative guess and the capture rate is (80.56 − 70.29) ÷ (92 − 70.29) = 10.27 ÷ 21.71 ≈ 47%.

Roughly half the available headroom, on the track where the signal is strongest and the sample is largest. That is a genuinely encouraging number and a genuinely humbling one: routing works, and it is nowhere near solved. The remaining half is what the next generation of methods is competing for.

(That 92 is our assumption, not the paper's, and the conclusion is only as good as it. The real point is the shape: state your oracle, report your capture, and the field gets a denominator it can compare across pools.)

Every inversion between the two rule baselines

Finding (ii) rests on the claim that cheap models beat expensive ones on real queries. The table lets us enumerate exactly where, which is more convincing than the average.

Four of seven tracks go to the smallest model in the pool. Three of those four are small enough that Chapter 7's standard errors swallow the margin — but LongMemEval is not, and TimeSeries at 63 versus 58 of 127 is a five-question gap on a track with a ±4.4-point standard error, which is right at the edge.

The robust statement: the largest model wins decisively on the large text track and is not reliably better anywhere else. That is enough for finding (ii), and it is a weaker and more honest claim than "small models beat large ones."

What 0.36 points means, track by track

GraphRouter leads SVMRouter by 0.36 average points. Since the average is unweighted, that gap could be produced by a change on any single track: a track's contribution to the average is its score divided by seven, so a 0.36-point average gap corresponds to 0.36 × 7 = 2.52 points on one track, everything else held equal.

Convert 2.52 points into questions on each track:

The gap between the first-placed and second-placed router on the entire benchmark can be produced by a single video question. Ninety-four generic questions would also do it, and those ninety-four would be real. But the metric cannot tell you which happened, and the top of the leaderboard is therefore not a measurement of anything stable.

A closing note on how to hold this table

We have spent this chapter finding limits in a results table, which can read as debunking. It is not. Every limit we found was found using data the authors published — the per-track scores, the test-set sizes in Table 6, the composition in Table 7. A paper that reported only the Avg column would be immune to all of this analysis and would deserve far less trust.

So hold three statements at once, and notice they are compatible:

The experiment is well designed. One pipeline, one pool, one set of metrics, seventeen routers, five scenario tracks that did not previously exist under a common protocol. Nothing about the comparison is confounded by the surrounding stack, which was the entire problem it set out to fix.

Several of its individual numbers cannot bear weight. Four of seven tracks have sample sizes at which the observed spread is a couple of standard errors. The top three routers on the average are tied.

Its four findings survive anyway. "No single router dominates" is strengthened, not weakened, by the noise — if even the ordering is unstable, certainly no router dominates. "Learned routing beats the best fixed baseline" rests on the 3,729-query track where the standard error is 0.75 points and the margin is ten. "Multi-turn does not pay" is a factor-of-two gap. "Personalization works but the encoding matters" holds for the first clause and is thin for the second.

Chapter 8: What Reverses Under Cost

Every number in Chapter 7 was measured at β = 0 — quality only, cost ignored. That is the setting nobody deploys in. This chapter turns the cost dial up and watches the leaderboard come apart, which is the paper's most immediately actionable result.

The protocol

Each router is scored by a weighted reward:

This is Equation 1 with the trade-off written as two weights instead of one λ. They are the same object: divide through by α and β/α plays the role of λ. The paper sweeps five settings, "from the quality-only (α, β) = (1.0, 0.0) to the heavily cost-weighted (0.2, 0.8)."

Only the endpoints are named. The natural reading — and the one consistent with the paper's later reference to a "β ≥ 0.4" threshold — is five settings in steps of 0.2 with α + β = 1:

Who can even be swept

Before the results, a limitation the authors disclose: "Multi-round and RL-based routers cannot optimize this weighted objective and are therefore run once under a single configuration."

Derive why from Chapter 2. Cost-sensitivity can live in the decision rule d, or it can live in the learning signal ℒ. If it lives in d — a threshold, a cost penalty applied to scores — you can move it for free, because d is a few lines evaluated at inference. If it lives in ℒ — as it does for a router whose policy was trained by reinforcement learning against a reward that already contains λ — then a new operating point means a new training run.

So the sweep covers eleven routers, and the paper's rank figure is "of eleven." That is not a flaw in the experiment; it is a real property of those methods, and a real deployment cost: an RL-trained router cannot follow your budget when it changes at the end of the quarter.

The reversals

The paper ranks each router by reward within each category as β grows, and reports that "the rankings shift dramatically along the sweep." Three specific anchors are given, and each is a different kind of failure of the β = 0 leaderboard:

The MLPRouter row is the one that should change your behaviour. If you had read Chapter 7's table and eliminated MLPRouter for scoring 27.87 on Geometry3K — 17 correct out of 61, exactly level with the query-blind Smallest-LLM baseline, and seventh of the nine single-turn routers on that track — only RouterDC (16.39) and CausalLM (24.60) do worse — you would have discarded the best available option for three of the five budget settings on that entire category.

The paper's conclusion is a prescription, not an observation: "it is practical to choose the router that matches the performance–cost requirements of the deployment at hand."

Why the reversals happen — the mechanism

Nothing mysterious is going on. Go back to Chapter 1's arithmetic and generalise it.

A router's reward under weight β is α·P − β·C, where P is its average quality and C its average cost. As a function of β (with α = 1 − β), that is a straight line:

A line with intercept P and slope −(P + C). Two routers' lines cross exactly once, at

Worked example. Router 1 has P = 80, C = 60 (on whatever normalized cost scale is in use). Router 2 has P = 68, C = 20. At β = 0, router 1 leads by 12 points. Where do they cross?

Numerator: 80 − 68 = 12. Denominator: (80 + 60) − (68 + 20) = 140 − 88 = 52. So β_cross = 12 ÷ 52 = 0.231.

Check it. At β = 0.231: router 1 gives 0.769(80) − 0.231(60) = 61.52 − 13.86 = 47.66. Router 2 gives 0.769(68) − 0.231(20) = 52.29 − 4.62 = 47.67. Equal to rounding. At β = 0.4: router 1 gives 0.6(80) − 0.4(60) = 48 − 24 = 24.0; router 2 gives 0.6(68) − 0.4(20) = 40.8 − 8 = 32.8. Router 2 now leads by 8.8.

A twelve-point quality lead evaporates by β = 0.23 and becomes an eight-point deficit by β = 0.4. With eleven routers you have up to 55 pairwise crossings scattered across the interval, which is exactly why "the rankings shift dramatically" rather than gently.

The frontier itself

Figure 6 of the paper plots each router's operating points in the performance–cost plane, one point per β. The reported shape:

"For most routers, performance and cost exhibit a clear positive correlation, with the operating points rising from the low-cost to the high-cost end. This is because increasing the inference budget unlocks more powerful and expensive models, which perform better."

That is the expected part. Here is the part that is not:

"Meanwhile, always calling the largest model incurs the highest cost yet delivers only mediocre performance and is dominated by the learned routes, since many queries that the largest model fails are solved by smaller and cheaper ones. This confirms that no single model covers all queries, which is exactly the headroom that routing exploits."

Dominated is a precise word from multi-objective optimisation, and it deserves unpacking. Point X dominates point Y when X is at least as good on every objective and strictly better on at least one. Saying Largest-LLM is dominated means there exist routed operating points that are both cheaper and more accurate. Not a trade-off. A strict improvement, for any budget-holder with any β whatsoever.

That is the strongest possible form of the paper's claim, and it is what makes routing worth the engineering: you are not buying quality with money, you are picking up money that was being burned.

A practical procedure, assembled from what we now know

What a Pareto frontier is, and why routing needs one

Two objectives, quality and cost, and no exchange rate given in advance. That is a multi-objective problem, and it has a standard vocabulary worth owning.

Plot every policy as a point: cost on the x-axis, quality on the y-axis. Policy X dominates policy Y when X is at least as good on both axes and strictly better on at least one — cheaper and no worse, or better and no dearer. A dominated policy is never the right answer for anybody, at any budget, ever.

The Pareto frontier is the set of non-dominated policies. Choosing among them requires a value judgement (that is what λ is), but choosing something off the frontier requires only a mistake.

Worked example. Four policies, with quality in points and cost in cents per query:

C is off the table for every possible decision-maker. Nobody who prefers more quality and less cost should ever pick it. And the paper's central empirical claim is that Largest-LLM is a C: "always calling the largest model incurs the highest cost yet delivers only mediocre performance and is dominated by the learned routes."

Take a moment on how strong that is. It is not "routing gives a better quality-per-dollar ratio", which is a claim you can argue with by preferring quality. It is "there exist routed policies that are simultaneously cheaper and more accurate than always using the biggest model." No preference over the two objectives makes always-largest correct.

Why fifty-five crossings

Chapter 8 said the rankings shift "dramatically" rather than gently, and the count is worth doing. Every pair of routers has a reward line, and two non-parallel lines cross exactly once. With eleven sweepable routers, the number of pairs is

Fifty-five crossing points, scattered over β ∈ [0, 0.8]. Not all of them fall inside that interval and not all of them change the top of the ranking, but the density is why the rank table looks like turbulence rather than a gentle drift. If your leaderboard is measured at one β and deployed at another, you are somewhere on the far side of an unknown number of those crossings.

Choosing β from a business constraint

The practical question is never "what is your β" — nobody has an opinion about that. It is one of two other questions, and both convert.

Case 1: you have a budget. "We can spend $X per month on inference." Divide by expected query volume to get a target cost per query, then find the point on the frontier at that cost. β is whatever value puts you there; you never have to name it.

Case 2: you have a quality floor. "Accuracy cannot drop below 70%." Find the cheapest frontier point above that line. Same procedure, other axis.

Case 3: you genuinely trade off. Then use the Chapter 1 break-even calculation. Ask: how much would we pay, per query, for one point of accuracy? Suppose the answer is "a point of accuracy is worth $200,000 a year to us and we serve 400 million queries a year." Then one point is worth 200,000 ÷ 400,000,000 = $0.0005 per query, so λ = 1 ÷ 0.0005 = 2,000 points per dollar. That number is now defensible in a budget meeting in a way that "we chose beta equals 0.4" never will be.

A caution about cascades on the frontier

One wrinkle the linear model glosses over. For most routers, cost per query is roughly fixed once the policy is fixed, so its point on the frontier is a point. For AutoMix it is not: its cost is data-dependent, since it pays for the large model only on escalated queries. Its expected cost is c_s + r · c_l, where the escalation rate r depends on the query mix.

So a cascade's position on the frontier moves when your traffic gets harder, without anyone changing a setting. On easy traffic it looks brilliantly cheap; on hard traffic it converges to always-large-plus-a-wasted-draft. That is a genuinely different risk profile from a single-dispatch router, and it does not show up in a single reported number. If you deploy a cascade, monitor r.

Constructing a frontier from scratch

Worth doing once by hand, because the procedure is three lines and most people have never run it.

Take five policies with (cost, quality) pairs, cost in cents per query:

Procedure. Sort by cost ascending. Walk the list keeping a running maximum of quality. A policy is on the frontier if its quality exceeds every cheaper policy's quality.

P1 at 62 — running max was nothing, so keep. Running max 62.
P2 at 69 > 62 — keep. Running max 69.
P3 at 71 > 69 — keep. Running max 71.
P4 at 70, not greater than 71 — dominated by P3, which is both cheaper (1.20 < 2.00) and better (71 > 70). Discard.
P5 at 74 > 71 — keep.

Frontier: {P1, P2, P3, P5}. Four survivors, one eliminated, and the elimination required no preference at all — only the two inequalities.

Now attach λ. Which frontier point wins at a given λ? Compute quality − λ × cost with λ in points per cent:

Check one cell: at λ = 6, P2 gives 69 − 6 × 0.80 = 69 − 4.8 = 64.2, and P3 gives 71 − 6 × 1.20 = 71 − 7.2 = 63.8. P2 edges it. That crossing sits between λ = 3 and λ = 6 — solve exactly: 69 − 0.8λ = 71 − 1.2λ gives 0.4λ = 2, so λ = 5.

Every frontier point wins for some interval of λ, and P4 wins for none — which is the definition of dominated, verified numerically. Run this procedure on your own routers and the "which is best" argument ends: you will have a table of intervals instead of an opinion.

What to log so you can rebuild the frontier later

The frontier is not a one-time artefact; your traffic and your prices both move. Five fields per request are enough to reconstruct it at any point:

The third row is the highest-value one and the most often omitted. If you log all eighteen scores rather than the arg max, then re-tuning your entire cost posture is an offline replay over yesterday's logs. If you log only the winner, every re-tune is a live experiment.

Operating-point stability

One last practical note. Because rankings reverse across β, a system that recomputes its operating point automatically — say, tracking a monthly budget — can flip routers mid-month, and a flip changes the quality distribution your users see.

Two mitigations, both standard control-theory hygiene. Add hysteresis: require a router's advantage to exceed a margin before switching, so you do not oscillate around a crossing point. And rate-limit the change: move β gradually rather than in a step, so quality drifts rather than jumps. Neither is in the paper, which is studying the frontier rather than operating on it; both are what you will need on the day you deploy it.

Finding the β that hits a budget, by bisection

Chapter 1 argued that λ is the shadow price of a budget and that sweeping it sweeps cost monotonically. That monotonicity gives you an algorithm: bisection on β.

Setup. Target: average cost per query no greater than $0.00020. Using the five-policy example, with costs converted to dollars (P1 = $0.0035, P2 = $0.0080, P3 = $0.0120, P5 = $0.0440 — scaled down by 100 for legibility, so treat them as relative).

Iteration 1. Try λ = 0. Winner is P5 at cost 0.0440. Over budget. So λ must rise.

Iteration 2. Try a large λ = 100 (points per unit cost). P1: 62 − 0.35 = 61.65. P2: 69 − 0.80 = 68.20. P3: 71 − 1.20 = 69.80. P5: 74 − 4.40 = 69.60. Winner P3, cost 0.0120. Still over. Rise further.

Iteration 3. λ = 600. P1: 62 − 2.10 = 59.90. P2: 69 − 4.80 = 64.20. P3: 71 − 7.20 = 63.80. Winner P2, cost 0.0080. Getting closer.

Iteration 4. λ = 2,000. P1: 62 − 7.0 = 55.0. P2: 69 − 16.0 = 53.0. Winner P1, cost 0.0035. Under budget — so the answer is between 600 and 2,000, and the true optimum is the largest budget-feasible point, which here means the crossing between P1 and P2.

Solve it exactly: 62 − 0.0035λ = 69 − 0.0080λ gives 0.0045λ = 7, so λ = 1,556. Just below that, P2 wins at cost 0.0080; just above, P1 wins at 0.0035. There is no λ that produces an average cost of exactly 0.0020 — it is between two frontier points.

That is the concave-region caveat from Chapter 1 in the flesh: to hit a budget strictly between two frontier points you need a randomised mixture. Route a fraction p of queries under P2 and (1 − p) under P1, and solve 0.0080p + 0.0035(1 − p) = 0.0020 — which has no solution here because 0.0020 is below P1's own cost. Raise the target to 0.0050 and p = (0.0050 − 0.0035) ÷ 0.0045 = 0.333: send a third of traffic to P2 and two thirds to P1.

The cost of sitting at the wrong operating point

Quantify the mistake, so it stops being abstract. From the frontier table: at λ = 6, the right choice is P2, scoring 64.2 on the weighted objective. The β = 0 leaderboard would have told you to pick P5, which scores 74 − 6 × 4.40 = 47.6.

You would be 16.6 objective points worse off, having chosen the policy that a leaderboard crowned. And in the raw axes: P5 delivers 74 quality for 4.40, P2 delivers 69 for 0.80. You paid 5.5× the money for 5 points, at a budget where you had declared points to be worth 6 per unit cost — which prices those 5 points at 30 and their cost at 3.60 units, i.e. 21.6 in objective terms. The numbers say the same thing twice.

This is the entire practical content of Chapter 8, and it is why the paper's rank figure is arguably its most useful exhibit. RouterDC going from first to tenth of eleven is not a curiosity. It is the size of the mistake available to anyone who reads only the β = 0 table.

How to read the rank figure

The paper's Figure 5 is a grid: routers down one axis, cost weights across the other, and each cell holds that router's rank under the weighted objective, "with smaller rank values indicating better performance."

Rank rather than score, and that is a deliberate choice with a trade-off. Ranks are robust — they do not require the cost normalization to be comparable across categories, and they are immune to one router's outlier score compressing the colour scale. But ranks also destroy magnitude: a router that falls from first to tenth may have fallen off a cliff or may have been beaten by nine routers within a point of each other. From the rank cells alone you cannot tell.

So read the rank grid for the pattern — does the ordering churn, or is it stable? — and read Figure 6, the performance-cost scatter, for the magnitudes. The paper reports both, and the two answer different questions.

Five operating points, as a product decision

The sweep's five settings are not abstract. Each corresponds to a recognisable product posture, and naming them makes β selectable by someone who will never look at an equation.

Notice the first row's annotation, because it is the whole chapter in one line: the setting every leaderboard reports is the one setting nobody deploys at. The paper's own main table (Table 2) is at (1.0, 0.0), and the paper immediately follows it with the sweep that shows how much that ordering moves. That sequencing is the authors telling you how to read their own headline.

What changes if you have more than one product

A practical corollary that the single-β framing hides. If you run a free tier and a paid tier, you do not have one β, you have two — and the right answer may be two different routers, not one router at two thresholds.

Why? Because the crossings are between routers, not between thresholds of one router. MLPRouter being best on Vision for every β ≥ 0.4 and near-worst at β = 0 means that a product serving both postures should be running MLPRouter on its free tier and something else on its paid tier. Both are cheap to serve; the scorers are hundred-kilobyte models. Running two is not the extravagance it would be with two frontier models.

That is not in the paper — it studies one router at a time across the sweep — but it is the direct operational reading of its own finding, and it costs almost nothing to implement.

The routers you cannot sweep, and what to do about them

Chapter 2 noted that RL-trained and multi-round routers "cannot optimize this weighted objective and are therefore run once under a single configuration." Sit with the practical consequence, because it is a deployment property and not a footnote.

A router whose cost sensitivity is baked into its training reward has one operating point. Not one you chose — one that whoever trained it chose, at whatever λ they had in mind, which for a research artefact is usually zero. Your budget changes at the end of the quarter; the router does not.

Three responses, in increasing order of effort:

That third row is a genuinely promising and unclaimed research direction. Goal-conditioned policies are standard in reinforcement learning — you train one policy over a distribution of goals and condition on the goal at inference. Doing the same with λ would give you an agentic router that can be swept, which is exactly what the field's main evaluative instrument requires and no current method supplies.

The single-number trap, one last time

Everything in this chapter reduces to one habit, so here it is stated as a habit rather than as a result.

When someone reports "our router achieves X", the reply is: at what cost, and where does it sit at my budget? A router is a point on a line whose slope is −(P + C). Reporting X reports the intercept and withholds the slope. You cannot rank lines by their intercepts unless you are certain you will only ever evaluate them at zero — and Chapter 8's table of product postures says nobody deploys at zero.

The paper models the alternative: it reports the full sweep, it reports the frontier plot, and it states plainly that "it is practical to choose the router that matches the performance–cost requirements of the deployment at hand." That sentence is the paper declining to name a winner, on purpose, and it is the most useful sentence in it.

A frontier is a promise about the future, not a summary of the past

One closing reframe. A leaderboard is retrospective: on this data, at this setting, these were the scores. A frontier is prospective: whatever budget you turn out to have, here is the best you can do at it.

That difference is why the sweep is worth its cost. Your budget is not known when you choose a router — it is set by a business that will change its mind. A frontier is robust to that; a leaderboard is not. The paper measured eleven routers at five settings, which is 55 evaluations rather than 11, and in exchange every reader can pick the row that matches a budget the authors never knew about.

Which is also the argument for reporting frontiers in your own work. It costs 5× the evaluation and it makes the result useful to people whose constraints differ from yours — which is everyone.

Chapter 9: Real Humans and Multi-Agent Systems

Everything so far has been offline: a static matrix, a fixed set of queries, a simulated judge. This chapter is the paper's answer to the question that should be nagging at you — does any of this survive contact with reality? Two experiments, and one of them delivers the most uncomfortable result in the paper.

Experiment 1: a Slack deployment with real people

The setup, exactly as reported. LLMRouter can expose any router "as an OpenAI-compatible server that integrates with OpenClaw for deployment on messaging platforms such as Slack and Discord", with "a routing memory [that] persists the interaction history h across turns." So they built a Slack app and collected preferences from actual humans.

The mechanic: "For each query, two models are sampled from a pool of ten candidates, their answers are shown in randomized positions, and the user marks one as better or declares a tie." Two design details are load-bearing. Randomized positions removes the well-known bias toward whichever answer appears first. Splitting by session rather than by record means no test user's earlier turns leak into training — the router is genuinely predicting an unseen conversation.

Then every router is trained on the human training split and scored by "how often its selection matches the human preference on held-out sessions."

The result that should reset your priors

Put Table 3 and Table 4 side by side. Same routers, two notions of "user preference" — one simulated by a persona-conditioned DeepSeek-V3.1 judge, one collected from fifteen humans in Slack.

The paper's own reading: "The simulated ranking does not fully transfer, as GMTRouter, the winner under the persona judge, drops to sixth on real users, showing that it matters to validate personalized routers against real feedback."

Two honest cautions before you over-read Table 4. First, the sample is small: 234 records, 8 test sessions, 15 users. Second, everything is measured as agreement with a pairwise preference, so the numbers are not comparable to Table 2's task accuracies. What survives both cautions is the ordering change, which is large, and the paper's conclusion from it, which is methodological rather than numerical: validate against real feedback.

Experiment 2: routing inside a multi-agent system

A multi-agent system (MAS) is "conventionally instantiated with a single base model shared by every agent" — a planner, three executors, and a summariser, all the same model. The paper asks: what if each node picks its own?

"We instead treat model choice as a per-agent decision, where a router receives the prompt of each functional node and selects the most suitable LLM for that call."

This is a stress test, and the paper says why: "Since node prompts differ substantially, covering planning, execution, and verification instructions, this setting stress-tests how well a router trained on ordinary queries generalizes." The routers are trained on the Generic LLM Tasks training split and then asked to route prompts that look nothing like benchmark questions.

Five topologies, following MultiAgentBench and GraphPlanner:

Note the call counts, because they are the cost story. A Chain query costs four model calls; a Tree or Graph query costs seven. At the 671B tier's $0.0004375 per 200-in/150-out call, a Tree query costs 7 × 0.0004375 = $0.0030625 — seven times a single dispatch, before you have improved anything. Multi-agent systems are expensive by construction, which is exactly why per-node routing has something to offer.

Table 5, and the arithmetic

Verify MFRouter's average by hand: 75.40 + 74.20 = 149.60; + 81.00 = 230.60; + 78.60 = 309.20; + 73.20 = 382.40. Divide by 5: 76.48. And Largest-LLM: 69.00 + 67.00 + 77.20 + 69.00 + 75.20 = 357.40, ÷ 5 = 71.48.

The gain: 76.48 − 71.48 = 5.00 points absolute, and 5.00 ÷ 71.48 = +7.0% relative. The paper's summary: "routing every node pays off, as six of the seven learned routers beat always selecting the largest model on average, with MFRouter attaining the best average of 76.48 against 71.48."

Count them: kNN 76.08, SVM 76.28, MLP 75.64, MF 76.48, Elo 75.32, RouterDC 75.92 — six above 71.48. GraphRouter at 69.24 is the seventh, and it is below.

The cross-table reversal

Now line this up against Chapter 7 and the ground moves again:

Ranks are over all fourteen rows of Table 2, rule-based baselines included — which is why MFRouter's 38.69 lands 8th: Largest-LLM's 38.72 sits directly above it, by three hundredths of a point.

The best router on xRouteBench is the only learned router that loses to the baseline in a multi-agent system. And the weakest of those seven routers on xRouteBench — RouterDC, lowest of the MAS seven at 36.32 — comes fourth.

Part of that is explicable and part of it is not. The explicable part: the MAS study trains on the Generic LLM Tasks split and evaluates on the Generic test split, so a router that was strong on Generic ought to transfer — and RouterDC's 80.56 on Generic (Chapter 7's best) is exactly why it does well here, while its catastrophic Geometry3K score never enters the picture. The inexplicable part: GraphRouter scored 80.54 on Generic, statistically indistinguishable from RouterDC, and yet it collapses. Whatever GraphRouter learned about benchmark queries does not survive the shift to planner, executor, and verifier prompts — the shift the paper warned about when it said node prompts "differ substantially."

One further disclosure the authors make, which matters for reading the numbers: "We adopt the topology of GraphPlanner without training its planner, since the training requires live answers from candidate models that have since been retired." So the Plan-Exec-Sum planner is untrained. Model churn is not a hypothetical hazard for this field — it already broke a reproduction inside this very paper.

The deployment surface, briefly

Two more pieces of the library exist mainly to make studies like these possible.

The OpenAI-compatible server. Any router can be exposed behind the interface every client already speaks, so "the same router evaluated offline can therefore serve live single-agent and multi-agent traffic without modification." The Slack study is that claim being cashed.

The ComfyUI canvas. "LLMRouter exposes its full routing pipeline as a graph on the ComfyUI canvas, where every node is a library component and every edge is an artifact that flows between components." Two input nodes declare the benchmarks and the pool, a data-engine node emits the query–model matrix, and each router node consumes that matrix and emits its evaluation. The router nodes are grouped in the menu by family, "so the taxonomy of §2.1 is visible in the node menu."

There is one engineering detail in there worth stealing regardless of whether you ever touch ComfyUI: "The Generate Data node hashes the selected datasets, candidate pool, and sample size into a metadata record and reuses an existing data directory when the record and its files are unchanged, which skips the costly response-collection stage on repeated runs." Content-addressed caching of the expensive stage. When your expensive stage costs 85,806 API calls, that hash is the difference between an experiment you can iterate on and one you can run once.

How much can 234 records tell you?

Apply Chapter 7's tool here too, because the Slack study is by far the smallest experiment in the paper.

Of 40 sessions, 8 are held out — a fifth. If records are distributed roughly evenly, the test set is on the order of 45 to 50 pairwise comparisons. (The paper reports the totals and the session split but not the test-record count, so this is an inference.) At p = 0.8 and n = 47:

So PersonalizedRouter's 83.05 and EloRouter's 82.20 are indistinguishable. So are the whole top four. What is not inside the error bar is the range: 83.05 down to 27.97 is more than nine standard errors, and GMTRouter's fall from first to sixth spans 68.78-to-70.70 against a field whose top has moved to 83.05 — a gap of over twelve points, or two standard errors.

So the finding survives at the level the paper states it: the ranking does not fully transfer, and the specific routers at the extremes are clearly separated. The finer orderings inside the top group are not evidence of anything.

Below chance: what it means and why it happens

Two routers score under 50% at picking which of two answers a human preferred. That deserves an explanation rather than a shrug.

Largest-LLM at 41.53. This one is almost mechanical. The policy always names the biggest model. If in the sampled pairs the bigger model is frequently not the one people preferred — because it is more verbose, more hedged, more inclined to preamble — then a policy that always backs it will be wrong more often than a coin. What Table 4 is measuring here is not a router failing; it is the folk belief "bigger is better" being tested against people and losing.

CausalLM at 27.97. Harder, and more instructive. A generative router emits a model name as text. Its errors are not centred on the truth — they are centred on whatever name the fine-tuned decoder finds most probable, which is driven by name frequency in training and by tokenisation, not by fit to the query. A systematically wrong prior produces systematically wrong picks, and nothing in a next-token objective penalises confident nonsense. Note that CausalLM also underperformed on xRouteBench (38.22 — above Smallest-LLM's 37.94, but still short of Largest-LLM's 38.72, so a query-aware generative router failed to beat a policy that never reads the query), so this is a consistent weakness rather than a one-off.

The general lesson for anyone building a router as an LLM prompt — a very common instinct — is that fluency is not calibration. The output will always be a valid model name. That tells you nothing about whether it is the right one.

The per-topology picture

Table 5 has structure worth reading down the columns, not just across the average.

The gain from routing is largest on Tree (+12.60) and smallest on Plan-Exec-Sum (+1.00). A plausible reading: Tree has the most heterogeneous node roles — a root planner, two sub-planners that refine, two actors — so per-node model choice has the most to differentiate. Plan-Exec-Sum's three executors all receive atomic sub-queries of the same kind, so there is less to choose between. Routing pays in proportion to how different the nodes are, which is the same "headroom comes from heterogeneity" principle from Chapter 3, one level up.

Also notice that Largest-LLM is not uniformly bad here: it scores 77.20 on Graph, beating five of the seven learned routers on that topology. The Graph topology has three actors answering independently and then one full-communication revision round, which is a structure that already extracts a lot from a strong model. Where the scaffold does the work, the model choice matters less.

What "without training its planner" costs the result

Worth dwelling on the disclosure: "We adopt the topology of GraphPlanner without training its planner, since the training requires live answers from candidate models that have since been retired."

Plan-Exec-Sum is therefore running an untrained planner, and it is also the topology with the smallest routing gain (+1.00). Those two facts may well be related: a planner that decomposes badly produces sub-queries whose difficulty is uninformative, and a router that cannot tell hard sub-queries from easy ones has nothing to route on. That is a hypothesis, not the paper's claim, but it is the kind of hypothesis a fully specified experiment lets you form — and it is another argument for the infrastructure, since the fix is a rebuild of the matrix against current models rather than a new research project.

What "OpenAI-compatible" actually buys

The paper states that LLMRouter "can expose any router as an OpenAI-compatible server." That is one clause, and it is the reason the Slack study exists at all.

An OpenAI-compatible server accepts the same request shape every LLM client already speaks: a list of messages, a model name, a few sampling parameters. So the router can be dropped in anywhere a model endpoint goes, with no client changes:

The consequence the paper draws: "The same router evaluated offline can therefore serve live single-agent and multi-agent traffic without modification." No re-implementation between the research artefact and the deployed one, which means the thing you measured is the thing you shipped — a property that is rarer than it should be.

It also explains the multi-agent study's mechanics. A multi-agent framework already calls a model endpoint at each node. Point every node at the router's URL and you have per-node routing, with no changes to the framework. That is why the paper could instantiate five topologies without writing five integrations.

The routing memory is h_t, in production

"A routing memory persists the interaction history h across turns." Read that against Chapter 1 and it is the deployment realisation of the state's third component.

Offline, h_t is whatever earlier models returned within one routing episode. In a Slack conversation, it is the accumulated turns of a session that may run for twelve exchanges. That is what makes the multi-turn and personalized families deployable rather than theoretical: the server holds the state that their E_q requires.

It also introduces a problem the offline benchmark never faces. Sessions have no natural end. Does h reset after an hour of silence? A day? Never? Every choice changes the state that every subsequent decision is made from, and the paper does not report a policy. If you build this, decide it deliberately, because an ever-growing h is both a monotonically increasing prompt cost and a slow drift in what your router is conditioning on.

Costing the topologies, per user query

Chapter 3 priced a Tree query. Here is the full table, at 200 in / 150 out per node call, so you can see the whole space a per-node router is choosing within.

The spread is 12.5× in every row, because it is the price ratio of the two models and the call count cancels. What the call count changes is the absolute money at stake: the same 12.5× is worth $0.00161 per query on Chain and $0.00282 on Tree.

And per node, the router has eighteen choices, so a seven-node topology has 18⁷ = 612,220,032 possible assignments. The router does not search that space — it makes seven independent local decisions — which is a simplification worth naming: a per-node greedy policy ignores the fact that a weak planner makes every downstream executor's job harder. Joint optimisation over the topology is not attempted here, and is another open door.

Three design decisions in the preference collection, and what each prevents

The Slack study is small, and its design is careful in ways that are worth stealing.

Randomised positions. "Their answers are shown in randomized positions." Human raters have a well-documented tendency to favour the first option shown. Without randomisation, that bias would attach to whichever model happened to be sampled first, and the collected preferences would encode a UI artefact as a model property.

Anonymised labels. Responses appear as "Answer A" and "Answer B." If raters could see model names, brand priors would contaminate the vote — and the whole point is to measure the answer, not the reputation.

Session-level splitting. "We split by session into 32 training sessions and 8 test sessions." Turns within a session are correlated: same user, same topic, same mood. Splitting by record would put turn 3 in training and turn 4 in test, and a router would score well by memorising the session rather than by learning the person.

Each of the three prevents a specific, plausible way of getting a good-looking number that means nothing. That is what careful experimental design looks like, and it is worth noticing that all three cost almost nothing to implement.

Fifteen users is a real limitation, and a specific one

Be precise about what 15 users does and does not undermine.

What it does not undermine: the claim that the simulated ranking fails to transfer. That claim is about a disagreement between two rankings, and a disagreement is visible with few samples as long as the effect is large — which a five-place drop is.

What it does undermine: any claim about the magnitude of personalization gains in general. Fifteen people from presumably one lab is not a sample of humanity; their preferences may be unusually homogeneous, which would make personalization look easy, or unusually diverse, which would make it look hard. There is no way to tell from 15.

What it leaves open: whether the persona judge is wrong or merely different. The judge simulates 200 diverse personas; the humans are 15 actual people. Those two populations differ in composition as well as in realism, so "the ranking did not transfer" could be a realism effect or a population effect. Distinguishing them would need the same 15 people's personas fed to the judge — an experiment the infrastructure makes possible and that nobody has run.

What a per-node router actually sees

One clarifying detail about the multi-agent setup, since it is easy to picture wrongly. The router does not see the user's query. It sees "the prompt of each functional node" — which for a planner is an instruction like "decompose the following into three atomic sub-queries", and for a verifier is "check the previous answer and improve it."

Those prompts have a very different distribution from benchmark questions. They are imperative rather than interrogative, they contain meta-instructions about format, and they embed prior nodes' outputs. A router trained on "Answer the following multiple-choice question..." has seen nothing like them.

Which reframes the +7.0% result. It is not "routing helps in multi-agent systems, as expected." It is "routers trained on an entirely different query distribution still beat always-largest by 7% when applied to node prompts" — a transfer result, and a more surprising one. It also reframes GraphRouter's failure: it is not that GraphRouter is bad at multi-agent systems, it is that GraphRouter transfers worse than the others, which is consistent with its being the most tightly fitted to the benchmark distribution.

Inside the ComfyUI graph

The visual interface is easy to dismiss as a demo. Read the node inventory and it is actually the clearest statement of the library's architecture, because every node is a module and every edge is an artefact.

Two details are worth stealing. First, the router nodes "appear in menu groups named after the three router families, so the taxonomy of §2.1 is visible in the node menu" — the formulation is not documentation, it is the information architecture. Second, "each router node reads its defaults from the same YAML configuration that the command line uses and renders every hyperparameter as a typed widget whose value, range, and options come from that file, so a canvas node and its scripted counterpart run one configuration."

That second point prevents the classic failure of visual tooling: a GUI whose behaviour drifts from the CLI's. One configuration source, two front ends, no divergence.

And on execution, a node "returns a summary of the query count, the success count, the average performance, and the routing distribution over candidates." That last field — the routing distribution — is the most diagnostic thing a router can report and is missing from most evaluations. A router that sends 95% of traffic to one candidate has effectively degenerated to a fixed policy, and its accuracy number will not tell you that. Its distribution will.

What both deployment studies have in common

The Slack study and the multi-agent study look unrelated: one is about human taste, the other about agent scaffolds. They deliver the same finding twice, from opposite directions, and that is why the paper runs both.

One study moves the evaluator and one moves the input, and both produce a reordering rather than a uniform drop. Two independent perturbations, one conclusion: router rankings are not a property of routers, they are a property of routers crossed with a distribution.

That is a stronger statement than either study alone supports, and it is the reason the paper ends where it does — with infrastructure that makes it cheap to build the matrix on your own distribution, rather than with a recommendation about which router to use.

Chapter 10: Connections and Cheat Sheet

Eleven chapters, one idea: the choice of which model answers a query is itself a learned decision, and it has a price attached. Here is where that idea sits in the wider landscape you already know, and what to carry away.

Routing is not the only kind of routing you have met

Read the rightmost column. Mixture-of-experts routing is differentiable and trained with the model; you can backpropagate into it. LLM routing is not: the candidates are black-box HTTP endpoints, there is no gradient through them, and the router's only teacher is a table of observed outcomes. That is why the whole field looks like recommender systems and bandits rather than like deep learning. If you want the layer-level version, our mixture-of-experts lesson builds it from scratch; the contrast makes both clearer.

Where each piece of this paper connects on the site

The cheat sheet

What is still open

The paper is a foundation, and foundations are defined as much by what they do not yet hold. Four gaps, all of them visible in the results we have read:

Sufficiency estimation. Named directly by the authors as the fix for the multi-turn collapse: "better sufficiency estimation, early stopping, and more effective decomposition and aggregation." Right now the multi-turn routers decompose unconditionally. Deciding whether to decompose is the unsolved problem, and it is the same problem AutoMix's verifier attacks one level down.

Sweepable agentic routers. As long as RL-trained routers must be retrained per operating point, they cannot participate in the cost frontier that is the field's main evaluative instrument. Making λ a conditioning input rather than a training constant would fix this.

Perception in the loop. xRouteBench deliberately renders every image, video, and series to text with one frozen captioner. That was the right call for isolating the routing variable, and it means the benchmark cannot yet study the routing decision that matters most in multimodal deployment: which model should look at this directly?

Model churn. The paper had to abandon training GraphPlanner's planner "since the training requires live answers from candidate models that have since been retired." Every query–model matrix has a shelf life measured in months. Infrastructure that can rebuild it from a config edit is not a convenience; it is the only way results like these stay true.

References

1. Feng, T., Yu, F., Zhang, H., Dai, Z., Yuan, L., Lei, Z., Zhang, W., Zhu, K., Yue, H., Xuan, K., Liu, G., You, J. "LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers," 2026 — arXiv:2608.06867. The paper this lesson is built on.
2. Hu, Q. J. et al. "RouterBench: A Benchmark for Multi-LLM Routing System," 2024 — arXiv:2403.12031. The precomputed single-turn benchmark xRouteBench extends past.
3. Ong, I. et al. "RouteLLM: Learning to Route LLMs with Preference Data," 2024 — arXiv:2406.18665. Matrix factorization and causal-LM routers trained on Chatbot Arena preferences.
4. Ding, D. et al. "Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing," ICLR 2024 — arXiv:2404.14618. The quality-gap threshold router.
5. Aggarwal, P. et al. "AutoMix: Automatically Mixing Language Models," NeurIPS 2024 — arXiv:2310.12963. Draft, verify, escalate.
6. Feng, T. et al. "GraphRouter: A Graph-based Router for LLM Selections," 2024 — arXiv:2410.03834. Link prediction over the query–model graph; best xRouteBench average.
7. Chen, S. et al. "RouterDC: Query-Based Router by Dual Contrastive Learning," NeurIPS 2024 — arXiv:2409.19886. Best on the generic track, last of the eleven rule-based and single-turn rows on the unweighted average.
8. Zhang, H. et al. "Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning," 2025 — arXiv:2506.09033. The agentic multi-turn router.
9. Maharana, A. et al. "Evaluating Very Long-Term Conversational Memory of LLM Agents" (LoCoMo), 2024 — arXiv:2402.17753; Wu, D. et al. "LongMemEval," 2024 — arXiv:2410.10813. The memory track, where input tokens dominate.
10. Lu, P. et al. "MathVista: Evaluating Mathematical Reasoning in Visual Contexts," ICLR 2024 — arXiv:2310.02255; Sigurdsson, G. et al. "Charades-Ego," 2018 — arXiv:1804.09626. The vision and video tracks.
11. Izacard, G. et al. "Unsupervised Dense Information Retrieval with Contrastive Learning" (Contriever), 2021 — arXiv:2112.09118. The fixed retriever that holds the memory track's evidence constant.
12. Zheng, L. et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," NeurIPS 2023 — arXiv:2306.05685. The source of the personalized track's prompts, and of the judging methodology whose limits Chapter 9 exposes.

An implementation checklist

If you take one artefact from this lesson into your own system, make it this. Every line traces to a chapter.

Objections worth having

A lesson that only agrees with its source is not teaching you to read papers. Three places where a careful reader should push back, and what the paper's defence is.

"The unweighted average over seven wildly unequal tracks is the wrong headline metric." Agreed, and Chapter 7 showed it changes the answer by 2.7 percentage points of relative gain and reorders the field substantially. The defence is that the unweighted mean is the honest way to say "across scenario types" rather than "across queries", and the paper does publish every per-track number so you can recompute. The fix is not to change the metric; it is to always print both.

"Four of the seven tracks are too small to resolve anything." Also true, and the standard errors in Chapter 7 make it quantitative. The defence is that those tracks exist to cover regimes — input-dominated cost, partial modality, modality selection — that no prior benchmark contained at all. A 27-query video track that establishes the regime exists is a better starting point than no video track. But it should not be read as a ranking, and the paper's prose is careful not to.

"The vision and time-series tracks do not measure what their names say." True: they measure reasoning over a frozen captioner's description. The paper states this explicitly and gives the design reason — holding perception constant is what makes the routing comparison valid across a mostly text-only pool. That is the right call for the question being asked, and the wrong data for a different question. Know which one you are reading.

Symbols, in one place

Reading order, if you want to go further

The reference list below is long. If you are going to read three of them, read them in this order.

First, RouteLLM. It is the shortest path to understanding the supervision problem, because it uses Chatbot Arena preference data rather than a constructed matrix, and seeing what that buys and costs makes Chapter 4's pipeline feel inevitable rather than laborious.

Second, Hybrid LLM. The cleanest possible instantiation of the five components — two candidates, one predicted gap, one threshold — and the clearest demonstration of why cost-awareness belongs in the decision rule.

Third, GraphRouter. The best-performing single-turn method on this benchmark, and the one whose mechanism most rewards understanding, because link prediction over a query–model graph is a genuinely different way to think about the problem than "classify the query."

Then come back to this paper's Appendix B, which is the most compressed useful survey of the field currently in print: seventeen methods, each in two sentences, all in one vocabulary.

Three numbers to remember

If everything else fades, these three carry the argument.

The argument, in one pass

Assemble it. No single model is best at everything, and the gap between the cheapest and most expensive fixed policies is 0.78 points for a price multiple of six or more — so there is headroom, and it is large. Capturing it means deciding per query, which is a policy, which needs an objective, which is quality minus a price times cost. Any router that implements that policy is five components: what it reads about the query, what it knows about the candidates, how it scores their compatibility, how it turns scores into an action given a budget, and how all of it is fitted.

Fitting it requires knowing how every candidate would have done on every query, which costs K calls per query and grading and pricing and a fixed prompt per task — and rebuilding that by hand for every paper is why nobody could compare anything. Automate it, and you get a benchmark spanning regimes that previously had none: input-dominated memory, partial-modality video, dual-encoded time series, user-conditioned preference.

Run seventeen routers through it under one protocol and four things become visible. No router dominates — the winner changes with the task and with the budget. Learned routing beats the best fixed policy by somewhere between 10% and 20% relative, depending on baseline and aggregation, both of which you must now state. Multi-turn routing loses to single-turn by nearly a factor of two, because it pays for decomposition and aggregation through a 3B scaffold and gets no consistent quality back. And personalization works, but the router that wins against a simulated judge drops five places against real people.

Which is why the last chapter's checklist ends where it does: build the matrix on your own traffic. Everything above is a method for answering the question. None of it answers it for you.

Four more open problems

Beyond the four already named:

Joint routing over a topology. Chapter 9 counted 18⁷ assignments for a seven-node system and noted that per-node greedy ignores every interaction between them. Whether joint optimisation is worth its supervision cost is unmeasured and now measurable.

Routing under sparse supervision. Every method here was evaluated with a dense matrix. Production never has one. Which of the seventeen degrade gracefully is an empirical question with an obvious experiment attached: mask cells at increasing rates and re-run.

Mixed policies. The Lagrangian sweep recovers only the convex hull of the frontier. Randomising between two routers can hit budgets no pure policy reaches. Nobody has tried it here.

Cost beyond money. Latency, throughput, provider availability, and safety are all absent from c(τ). Making it a vector is conceptually trivial and empirically unexplored, and it will reorder the rankings again — because none of those quantities track price.

Teaching this to someone else in ten minutes

The best test of understanding. Here is the ten-minute version, in the order that lands.

Minute 1–2: the surprise. "The biggest model in an eighteen-model pool beats the smallest by 0.78 points on a hundred-point scale, and costs roughly four to six times as much. It loses four of seven benchmark tracks outright." Let that sit. Everything else is a response to it.

Minute 3–4: the idea. "So pick per query. The thing that picks is a router. It reads the query, scores the eighteen candidates, and dispatches to one. Its objective is quality minus a price times cost, and the price is how many quality points a dollar is worth to you."

Minute 5–6: the structure. "Every router ever published is five choices: what you encode about the query, what you encode about the candidates, how you score their compatibility, how you turn scores into an action given a budget, and how you fit it. Families differ only in which part of the query state they read — just the query, the query plus what earlier models said, or the query plus who is asking."

Minute 7–8: the hard part. "Training it needs to know how every candidate would have done on every query. That is eighteen API calls per query, graded and priced. Nobody could afford to redo that per paper, so nobody's results were comparable. This paper automates it: 4,767 queries times 18 candidates, 85,806 cells, and a config file where the pool used to be hard-coded."

Minute 9–10: the findings. "Learned routers beat the best fixed policy by ten to twenty percent, depending on how you aggregate. No router wins everywhere. Multi-turn routing loses to single-turn by a factor of two because it pays for decomposition through a small model. And the personalized router that wins against a simulated judge drops five places against fifteen real humans — so validate on your own traffic."

If you can deliver those five beats without notes, you have the paper.

Where to go from here on this site

Three natural next moves, depending on what pulled at you.

If the decision-process framing was the interesting part, go to MDPs and then bandits and preference learning. Routing is a contextual bandit with offline full-information feedback, which is a corner of that space worth seeing in context.

If the matrix was the interesting part, go to recommender systems. Four of the seventeen routers are collaborative filtering wearing different clothes, and cold-start, sparsity, and implicit feedback all transfer directly.

If the evaluation methodology was the interesting part — the standard errors, the aggregation choices, the simulated-versus-real judge — go to evaluation statistics and GenAI evaluation. Most of what made Chapter 7 possible was ordinary statistical care applied to a table that published its denominators.

And if you want the layer-level cousin of everything here, mixture-of-experts routes between sub-networks inside one model with a differentiable gate. Same word, entirely different constraints, and holding both in mind makes each sharper.

Questions people ask after reading this

"Should I build a router for my product?" Only if your pool has headroom. Build the matrix on a few thousand of your own queries with three or four candidates, compute the oracle and the best fixed model, and look at the gap. If the gap is a point or two, stop — your candidates agree, and no router beats agreement. If it is ten points, you have found real money.

"How many candidates do I need?" Fewer than eighteen. The paper uses eighteen because it is a benchmark and wants coverage; a product wants the smallest pool that spans the price range and includes any specialist your traffic needs. Three or four is a reasonable start, and every added candidate costs a full N-query sweep on every rebuild.

"Which router should I start with?" kNN, because it needs no training, its behaviour is inspectable (you can look at the neighbours it voted on), and it establishes the baseline everything else must beat. Run EloRouter alongside it as a control — if kNN cannot beat a query-blind global rating, your embeddings are not carrying routing signal and a fancier scorer will not fix that.

"Where does the router itself run?" In your request path, before the model call, as a hundred-kilobyte model plus eighteen cached vectors. Its latency is a rounding error against a model call. The exception is a text-based router, which is a model call, and should be chosen deliberately rather than by default.

"Does routing conflict with prompt caching?" Yes, and this is a real production consideration the paper does not raise. Provider-side prompt caching gives you a discount for reusing a prefix with the same model. A router that sends consecutive turns of one conversation to different candidates forfeits that discount every time it switches. Your cost model should include it, and it argues for stickiness within a session — which the routing memory makes expressible, since h_t is part of the state.

"What breaks first in production?" The pool. A model is deprecated, a price changes, a provider adds a new tier, and every model-side representation your router learned is now describing something that no longer exists. Build the rebuild before you build the router.

What to build this week

A concrete, small, finishable version of everything above.

Five days, about fifteen dollars of API spend, and at the end you will know whether routing is worth anything for your product — which is more than most teams know after a quarter of arguing about model selection. Every number in this lesson was computable from an artefact of that shape; the paper's version is the same thing at 4,767 queries and eighteen candidates, with better graders.

What changed, in one paragraph

Before this paper, "which router is best" was a question you answered by reading six papers that each defined the problem differently, measured it on different pools with different metrics, and mostly did not measure cost at all. After it, the question has a procedure: fix the pool, fix the metrics, build the matrix once, run every method through the same interface, and sweep the budget. The answer that procedure returns is no router is best — which is not a disappointing answer, it is the first trustworthy one the field has had, and it comes with a method for finding the answer that is right for you.

Six things people believe that this paper corrects

Each row is a belief that is reasonable, widely held, and measurably wrong on this evidence. Notice that four of the six are wrong in the same direction — they over-trust size, complexity, or fluency — and the fifth and sixth are wrong about transfer. That is a coherent bias, and it is the bias an infrastructure paper is uniquely positioned to correct, because correcting it required running everything on one stack.

The last thing

The most quotable sentence in the paper is not one of the findings. It is the reason all four of them exist: "Every router therefore faces the same queries, candidate pool, and metrics, so measured differences reflect the routing policy rather than the surrounding stack."

That is not a result. It is a precondition for having results, and the field did not have it. Seventeen methods, one interface, one matrix, one sweep — and the first thing that falls out is that nobody wins, that the ranking reverses under cost, and that simulation does not transfer. Those are humbling answers, and they are the first ones anyone could trust.