Compound AI Systems: Globally Aligned Local Rewards

Chapter 1: What a Compound System Actually Is

Chapter 0 sketched the retriever–reranker–generator–verifier pipeline informally. Before deriving anything about how to optimize it, pin down precisely what it is — the paper's own formalism, which every later chapter builds directly on top of.

A compound system is a graph of components

Optimas represents a compound AI system as a directed acyclic graph 𝒞 = (𝒞 componentSet, ℰ), where 𝒞 = {C₁, …, C_K} is a set of K distinct components — the paper calls them task nodes — and ℰ is the set of possible directed edges connecting them. A component can be an LLM, a general machine learning model, a model selector, a retriever, anything with an input and an output. In the running example, K = 4: Retriever, Reranker, Generator, Verifier. The system's own input — the user's raw question — is treated as a special source node, C₀.

Each component C_k takes some input x_k and produces some output y_k. An edge (C_i, C_j) means C_i's output is routed as (part of) C_j's input. By default, component indices follow topological order over the edges: C_i is upstream of C_j whenever i < j — exactly the ordinary left-to-right pipeline picture.

One subtlety worth naming, because it matters for how ambitious this formalism actually is: the graph doesn't have to be fixed per instance. The edges actually used for a given input x, written ℰ(x) ⊆ ℰ, can be dynamic — a verifier's rejection might route control back through a re-retrieval step for some questions and not others. A compound system, in this formalism, is closer to a program with conditional branches than a fixed assembly line.

Configurations: the knobs you're actually allowed to turn

Every component C_k is controlled by a configuration policy v_k, formally C_k : (x_k; v_k) ↦ y_k. The configuration space 𝒕 that v_k lives in can be empty (nothing to tune — some components, like a fixed hash function, just don't have knobs), discrete (a textual prompt, or a choice of which of several candidate LLMs to call), or continuous (trained model parameters, or a numeric hyperparameter). The full joint configuration across the whole system is written v = (v₁, …, v_K).

Map this straight onto the running example, and onto one of the paper's own five real pipelines at the same time — its HotpotQA multi-hop question-answering system, which is literally a five-component version of exactly this retrieve-and-generate shape:

Notice already how little these five configuration spaces have in common. Four of them are strings a human or a search procedure edits. One is a small integer chosen from a fixed set. Elsewhere in the paper's other four systems, configurations get stranger still: Amazon's Session Analyzer and Candidate Profiler are small Qwen 2.5 1.5B models whose actual weights get updated by reinforcement learning; PubMedQA's Context Model Selector chooses, per question, which of seven different LLM APIs to call at all. Chapter 2 will show why this heterogeneity is not a minor inconvenience — it's a big part of why you can't just backpropagate through the whole thing.

Forward execution: running the system once

For a given input x and joint configuration v, the system executes its components in topological order over the active edge set ℰ(x):

where pa(C_k) denotes all of C_k's parents over ℰ(x) — everything feeding directly into it. Chain that recursively from C₀ all the way through, and you get the whole system's output, written compactly as f(x; v) := y, where y collects the output(s) of one or more components (usually the final one, though a verifier's pass/fail flag might count too).

python
def run_compound_system(x, components, config_v, edges):
    # components: dict of Ck by index, in topological order
    # config_v: dict of vk by index -- the one thing this whole lesson is about tuning
    outputs = {0: x}                          # C0 is the source node: the raw input
    for k in sorted(components):        # topological order, by construction
        parents = parents_of(k, edges, x)      # E(x) can depend on x -- dynamic planning
        x_k = gather([outputs[i] for i in parents])
        outputs[k] = components[k](x_k, config_v[k])   # yk = Ck(xk; vk)
    return outputs                             # f(x; v) -- the final answer lives in here

The optimization objective, written down

Given a dataset 𝒟 of inputs and a user-defined global reward function R : 𝒙×𝒚 → ℝ that scores the final system output — for the running example, imagine R checks the verifier-approved answer against a ground-truth answer and reports an F1 score — the whole optimization problem this session is about is one clean expression:

Find the joint configuration v that maximizes the expected global reward, averaged over the dataset. That's it. Every remaining chapter in this session is about how you actually solve this one equation for a system with four, five, or more components with wildly different configuration types — without ever needing to differentiate through the whole thing at once.

Concept → realization: what x_k and y_k actually look like

It's easy to let “C_k : (x_k; v_k) ↦ y_k” stay an abstract arrow on a whiteboard. It shouldn't. Trace one real question through HotpotQA's actual five-module pipeline, and write down, at every arrow, exactly what data type and rough size is crossing it — this is the concrete substance every later chapter's math is quietly standing on top of.

Notice what changes shape at every single arrow: a string becomes a shorter string, becomes a list of strings, becomes a shorter string again. Nothing here is a fixed-size tensor the way a hidden state in an ordinary neural network is. That's the concrete, unglamorous reason Eq. 1 can't be treated like training one more neural network — the “activations” flowing between layers are literally variable-length natural-language text, chosen by discrete decisions (which k passages, which words to keep in the hint), not continuous vectors with a well-defined gradient waiting to be computed.

Why the configuration types don't share a representation either

The same heterogeneity shows up on the v side, not just the y side. Look at what a single component's configuration space can actually contain, concretely, across the paper's five real systems:

text
# PubMedQA's Context Model Selector -- v is a DISCRETE CHOICE among 7 full LLM APIs:
{gpt-4o, gpt-4o-mini, gpt-3.5-turbo-0125, gpt-4-turbo,
 claude-3-5-haiku-20241022, claude-3-5-sonnet-20241022, claude-3-7-sonnet-20250219}

# HotpotQA's Retriever -- v is a single INTEGER from a 4-element set:
k ∈ {1, 5, 10, 25}

# Amazon's Session Analyzer -- v is a CONTINUOUS weight tensor:
Qwen 2.5 1.5B parameters, updated by PPO gradient steps

One of these v's is a categorical label picked from seven strings. One is a single machine-readable integer. One is roughly 1.5 billion floating-point numbers. There is no shared unit of “how far did v move” across those three objects — you cannot subtract a choice of Claude-3.7-Sonnet from a choice of GPT-4o and get a meaningful distance, the way you can subtract two points in weight-space. This is the concrete, load-bearing reason behind Chapter 2's Obstacle 2 (“the knobs don't share a language”) — it isn't a description problem, it's a genuine absence of shared mathematical structure between the objects being optimized.

One more distinction the taxonomy needs: per-instance vs. global configuration

There's a second axis hiding inside “discrete configuration” that matters for everything from Chapter 5 onward, so it's worth naming now. Some configurations are chosen once, for the whole system, and then held fixed for every future input. STaRK-Prime's Aggregator is exactly this: its two numeric weights, relation_weight and text_weight, are each searched over {0.1, 1.0} — a small, 2×2 = 4-combination space — using a global search across the entire training set, and whichever single combination wins gets used for every question at inference time, forever, until the next optimization pass. HotpotQA's Retriever(k) works the identical way: k is swept across {1,5,10,25} globally, and one winning value is fixed system-wide.

Other configurations are chosen fresh, per input. PubMedQA's Context Model Selector doesn't pick one LLM for the whole dataset — at inference time, it uses its trained reward model to pick, potentially, a different one of the 7 candidate LLMs for every single question, because which model is actually best can genuinely vary question to question. This is a strictly harder decision problem than the global case: a global hyperparameter only ever needs its LRF evaluated during the search phase, once, offline; a per-instance policy needs r_k evaluated live, at inference time, on every new input the system ever sees, since there's no single fixed answer to bake in ahead of time. Chapter 5's second worked toy makes this exact distinction concrete with real numbers — a per-instance model-selection decision where the wrong global default (always picking the model that looks stronger on average) actively loses to a per-question aligned choice.

Chapter 2: Why End-to-End Is Intractable

Chapter 1 wrote down one clean objective, v^★ = argmax_v 𝔼_x[R(x,f(x;v))]. The paper is blunt about what happens if you try to solve it the way you'd solve any other machine learning objective — take the gradient, descend: “directly optimizing the objective… is difficult.” Unpack exactly why into three separate, named obstacles, because each one rules out a different familiar shortcut.

Obstacle 1: most of v is not differentiable

Gradient descent needs ∂R/∂v_k for every coordinate of v. Walk through the running example's configuration types and ask, honestly, which of them that expression even means anything for.

A prompt is a string. “How would the F1 score change if I nudged this prompt in the direction of its gradient” is not a well-formed question — there is no continuous space of prompts that R is a smooth function of. A retriever's choice of k ∈ {1, 5, 10, 25} is a choice from a finite set, not a point you can perturb infinitesimally. A model selector choosing between seven candidate LLM APIs is the same story. Even where a component genuinely does have continuous, differentiable parameters — Amazon's Session Analyzer, a Qwen 2.5 1.5B model — that component usually still sits behind a sampling step (the model generates text, which then feeds a downstream component as a discrete sequence of tokens), which breaks the gradient chain right there anyway.

Net result, in the paper's own words: “as the configuration spaces are typically non-differentiable, gradient-based optimization cannot be used.” Not “is harder to use.” Cannot be used, for the overwhelming majority of a realistic compound system's actual knobs.

Obstacle 2: the knobs don't share a language

Suppose you got creative and found some optimizer for each individual configuration type separately — policy-gradient RL for the trainable weights, an evolutionary search over prompt strings, Bayesian optimization over the discrete hyperparameter. You still don't have a way to move all of v together, in one coherent update, implied by a single end-to-end loss. The paper names this precisely: “each policy v_k may control a different configuration type, so the joint policy v can span heterogeneous spaces.” A prompt edit, a PPO gradient step on model weights, and a jump between two discrete hyperparameter values are not the same kind of move, and there's no natural way to combine “how much to move” across all three inside one optimizer.step() call. Prior work — the paper's related-work section surveys this directly — mostly responds by picking ONE configuration type and optimizing only that: prompt-optimization frameworks like DSPy or TextGrad, reinforcement-learning approaches for trainable weights, model-routing methods like LLMSelector for discrete choices. Each is a real, useful tool. None of them, alone, touches a system with all three types mixed together, which the table in Chapter 1 shows every one of the paper's real pipelines actually has.

Obstacle 3: running the whole thing is expensive

Here's the one that survives even if you're willing to be inefficient. Suppose you abandon gradients entirely and just try many joint configurations by brute force, keeping whichever scores best on your data. Every single trial requires executing the entire pipeline — every component, in order — for every training example, just to get one number (the global reward R) back out. The paper states this plainly: “running the entire compound AI system during optimization to achieve global reward is costly.”

Make that concrete. This is not a number the paper computes — it's a derivation for this lesson, using the real component count, the real retriever hyperparameter set, and the real training-set size the paper reports for its HotpotQA system, to turn the word “costly” into an actual figure you can check.

HotpotQA's real pipeline has four prompt-controlled modules (Question Rewriter, Info Extractor, Hint Generator, Answer Generator) plus a Retriever with the real 4-way hyperparameter choice k ∈ {1,5,10,25}. Suppose you wanted to grid-search just 5 candidate prompts for each of the four text modules, crossed with all 4 retriever settings:

The paper's real HotpotQA training split, from Appendix C, is 1,000 questions. Evaluating every one of those 2,500 configurations properly means running the entire 5-stage pipeline on every training question:

Now compare against what the paper's own Table 3 reports Optimas actually spending on this exact system: about 2.02 thousand equivalent system runs — a real, measured number, not a derived one.

That gap — between what brute force would cost and what the paper's method actually costs — is not a minor implementation detail. It's most of the engineering problem this paper exists to solve.

python
def brute_force_runs(configs_per_module, num_modules, train_size):
    # illustrative estimate for THIS lesson -- not a number the paper reports
    total_configs = configs_per_module ** num_modules
    return total_configs * train_size

print(brute_force_runs(5, 5, 1000))   # 2,500,000 -- see derivation above
# Table 3's REAL measured Optimas cost on this exact HotpotQA system: 2,020 runs
print(2_500_000 / 2020)               # ~1,238x fewer full-system executions

Concept → realization: why sampling breaks the gradient even when weights are continuous

Obstacle 1 mentioned this in passing; it's worth making concrete, because it's the part people most often assume away. Say Amazon's Session Analyzer (a genuinely continuous, trainable Qwen 2.5 1.5B model) DID have a smooth loss surface with respect to its own weights — call that weight tensor θ. The quantity you'd actually need for end-to-end training is ∂R/∂θ, all the way from the final global reward back through every downstream component to this one. But between this component's output and the next component's input sits a sampling step: the model doesn't emit a differentiable probability distribution to the next stage, it emits one discrete, sampled token sequence — text. The next component (the Candidate Profiler) consumes that text, not the distribution it was drawn from. The moment a continuous distribution collapses into one discrete sample, the chain rule has nothing left to walk backward through; ∂(sampled tokens)/∂θ is not a well-defined quantity the way ∂(a continuous activation)/∂θ would be in an ordinary network. This is precisely why REINFORCE-style policy-gradient methods exist at all — they estimate a gradient around this exact discontinuity using the sampled reward as a proxy signal, rather than differentiating through the sample directly, and Chapter 8 shows exactly how much more data REINFORCE needs to pay for that workaround on this exact system.

Obstacle 2, seen through what prior work actually tried

The paper's own related-work section is candid about this: prior approaches to optimizing multi-component systems “typically optimize different components separately”, and it names four distinct families, each solving one slice of the heterogeneity problem and leaving the rest untouched:

A fifth family, cited in the same section, tries something structurally closer to this paper's own idea: decomposing a single global loss into per-step “process rewards” using human step annotations, hierarchical planning, or Monte Carlo Tree Search credit assignment. The paper's own stated distinction from that family is precise: those methods generally require a fixed decomposition of the loss, decided in advance, or extensive retraining whenever the system changes. Optimas's local rewards instead adapt dynamically, via the preference-pair retraining Chapter 6 covers, which is what lets them track a system that keeps moving under them during optimization — rather than being decomposed once and then going stale the moment any component changes.

The brute-force wall gets worse, not better, as ambition grows

Chapter 2's 2.5-million-run estimate used a conservative 5 candidates per text module. Real prompt search is rarely that timid. Bump it to 8 candidates per text module — still a modest number for an automated search — while leaving the retriever's real 4-way choice alone:

Compare against the same real, measured Optimas cost from Chapter 2 — which does not move just because the search considered more candidates, since it never evaluates the full cross product at once:

Brute force doesn't degrade gracefully as ambition grows — it degrades exponentially, because it's raising a base (candidates per module) to a power (number of modules). Optimas's real, measured cost is close to flat under the same change, because each iteration only ever locally optimizes one component against a cheap scalar reward, never the full cross product of every component's candidates at once.

Making “costly” visceral, in dollars

Run counts are abstract until you attach a price to them. This paragraph's numbers are an illustrative estimate for this lesson only — not a figure the paper reports — but the order of magnitude is what matters. Suppose, generously, that one full 5-stage pipeline execution (four LLM calls plus a retrieval step) costs around $0.02 in API spend, a plausible ballpark for small, fast models like GPT-4o-mini running short prompts.

Fifty thousand dollars, or a third of a million, to find one good configuration by exhaustive search — against roughly forty dollars for the real, measured cost this session has been citing all along. This is the same 1,238× and 8,111× gap from before, just converted into units a budget conversation actually uses. It's also why Chapter 3's fallback — skip the whole pipeline, tune one component against its own convenient metric — is so tempting in the first place: it's not just intellectually simpler, it's the option that doesn't require anyone to justify a five- or six-figure search budget. The trouble, as the next chapter shows with a real measured number, is that the cheap fallback can cost you something a dollar figure doesn't capture at all: a system that gets quietly, measurably worse.

Chapter 3: The Naive Fix, and the Backfire

Chapter 2 leaves exactly one path that's actually cheap: touch one component at a time, score it on something you can measure without running the rest of the pipeline, and improve that. This is what most prior optimization approaches for compound systems actually do, and the paper is direct about the consequence: “previous efforts largely focus on optimizing the policy for single types of configurations, which simplifies the optimization problem; however, this also leads to suboptimal compound systems.”

Why “suboptimal” understates it

The paper's introduction spells out the specific failure shape: “a perfectly optimized prompt can struggle to compensate for a poorly chosen model. Even when the components are individually well optimized, they may still collaborate suboptimally, as the upstream component might not have visibility into which inputs are effective for the downstream components.” Translate that into the running pipeline: a reranker's own notion of “good ranking” has no built-in awareness of what the generator, three steps downstream, will actually be able to do with a given passage. Nothing connects the two. A component optimized in isolation is optimized against whatever metric someone happened to hand it — and that metric has zero obligation to track the thing the whole system was built to do.

A real instance: DSPy on the Amazon pipeline

This isn't a hypothetical risk. DSPy — a genuinely strong, widely deployed prompt-optimization framework, using an algorithm called MIPRO that iteratively refines each module's instructions based on that module's own textual feedback — is one of the paper's own baselines, run across the same five real compound systems. Look at what it does on the Amazon behavior-driven recommendation pipeline specifically:

DSPy doesn't just fail to help. It makes the unoptimized default look good by comparison, and it lands below the accuracy of firing a bare single LLM at the task with no pipeline structure at all (20.20). Compute the size of that drop, precisely, the same way the paper does:

That's exactly the number the paper's results section reports: “their performance may be inconsistent and can even degrade the system (e.g., a 14.3% drop on the Amazon dataset).” A well-engineered, respected optimization method, applied with real care, made this specific real system meaningfully worse than doing nothing — while, on a different system in the same paper (HotpotQA), the same method genuinely helps.

Why a good optimizer can still do this

MIPRO's loop asks an LLM to look at a module's own inputs and outputs and few-shot demonstrations, judge which candidate instruction seems to produce better behavior, and rewrite accordingly. Nothing in that loop requires the judge's opinion of “better” to track Amazon's actual next-item-recommendation accuracy on this pipeline's specific downstream dynamics — how the Session Analyzer's PPO-trained representation interacts with the Candidate Profiler's output, and how that interacts with the Next Item Decider three steps later. The strength of the optimizer was never the missing ingredient. What was missing is any guarantee connecting what it's optimizing for to what the system needs.

Naming the general pattern

This is Goodhart's Law, made concrete rather than quoted as a slogan: “when a measure becomes a target, it ceases to be a good measure.” A convenient local signal — a relevance score, an LLM judge's opinion, a module's own confidence — can be a perfectly reasonable thing to look at. The moment you make it the literal quantity a search procedure maximizes, on a system it doesn't reliably track, it stops behaving like a proxy for what you want and starts behaving like something actively working against it, precisely because optimization is relentless about exploiting whatever signal you hand it.

It isn't just DSPy, and it isn't just Amazon

Look one column over in the exact same Table 2 the paper reports. TextGrad — a different prompt-optimization method, using estimated gradients from black-box LLM feedback rather than DSPy's MIPRO — does the same thing to a different system, BigCodeBench:

A second real method, a second real system, the same real direction: an optimizer that helps elsewhere makes this particular pipeline slightly worse than leaving it alone. Neither failure is a fluke specific to one algorithm's implementation. Both are the identical structural gap — a local optimization signal with no forced connection to that system's actual downstream objective — showing up wherever that gap happens to be wide enough to matter.

DSPy's full record, computed across all five real systems

Chapter 8 shows the paper's complete headline table later; it's worth pulling just DSPy's row out now and computing every delta by hand, because the pattern — not any single number — is the actual lesson:

The full record is not “DSPy is broken” and not “DSPy is great.” It's a strong win on HotpotQA, modest real wins on PubMedQA and STaRK-Prime, and real losses on Amazon and BigCodeBench — two systems out of five actively worse than doing nothing. That spread, computed from the paper's own numbers rather than summarized, is the most concrete evidence available that “this optimizer is well-engineered” and “this optimizer is safe to run on your system” are different claims. The first is true of DSPy. The second depends on facts about the specific pipeline that DSPy's own optimization loop never checks.

A third flavor of convenient-but-unvalidated: cloning by embedding similarity

The paper's baseline suite includes one more instructive case: Hierarchical Behavior Cloning (HBC), which sidesteps training a reward model entirely by scoring a candidate output on how close its text embedding is to a known-good reference output, using OpenAI's text-embedding-3-small to measure that closeness. It's a reasonable-sounding proxy — “outputs that read similarly to a good example are probably also good” — and it costs nothing to compute; no downstream Monte Carlo rollout, no pairwise preference collection, just a cosine similarity. On Amazon, it lands at 21.55% (Chapter 3's table), essentially flat against the 21.21% unoptimized baseline — not the sharp Amazon-style backfire DSPy produces, but not a real gain either. Text that sounds like a good answer and text that is a good answer are correlated, but imperfectly — and imperfectly is exactly the gap Chapter 4's alignment condition exists to close.

HBC's full record, across every system it's reported on, makes the imperfection concrete rather than abstract:

Two small real gains, three real regressions — and on HotpotQA specifically, embedding similarity doesn't just underperform, it drops the system's F1 by more than a third relative to doing nothing at all, a sharper fall than even DSPy's worst Amazon result. The mechanism is easy to name in hindsight: F1 rewards a specific, factually correct span of text; a text embedding rewards general topical and stylistic resemblance to a reference answer. Those two things agree often enough to look like a working proxy on some tasks (Amazon, PubMedQA) and diverge badly enough to actively mislead optimization on others (HotpotQA, BigCodeBench, STaRK-Prime) — and nothing about the embedding-similarity signal itself tells you in advance which kind of task you're facing. That unpredictability, not any single bad number, is the actual argument this chapter is building toward.

What MIPRO's “textual feedback” loop is actually doing, concretely

It helps to see the actual shape of the object MIPRO optimizes over, since “an LLM judges which instruction seems better” can sound more rigorous than it is. Each candidate configuration DSPy considers bundles together a module's instruction text (the prompt) with a small set of few-shot demonstrations — example input/output pairs shown in-context. The optimizer proposes new candidates, runs each on a handful of training examples, and asks an LLM to read the resulting inputs, outputs, and demonstrations, then output a natural- language critique and a preference between candidates. That critique is real, often genuinely useful-sounding text. It is also, structurally, an opinion generated with no numeric grounding in Amazon's actual next-item accuracy on this specific pipeline's downstream dynamics — nothing forces the judge's textual sense of “this instruction seems clearer” to correlate with what the Session Analyzer and Candidate Profiler, three components downstream, will actually do with the Next Item Decider's output. Chapter 4's LRF replaces exactly this step: instead of an LLM's free-text opinion, a small trained scalar head, calibrated directly against measured downstream reward.

So the question this session actually needs answered isn't “how do we optimize each component locally” — Chapter 2 already conceded that's the only tractable option. It's: what has to be true of a local reward before you're allowed to trust that optimizing it locally helps, rather than hurts, the global objective? That is a precise, answerable question, and Chapter 4 answers it.

Chapter 4: The Alignment Condition

Chapter 3 ended on a precise question: what has to be true of a component's local reward before optimizing it locally is safe for the whole system? This chapter derives the paper's answer — the local–global alignment property — from its definition of a Local Reward Function upward.

Defining a Local Reward Function

A Local Reward Function (LRF) on component C_k is a function r_k : (x_k, y_k) → ℝ that scores the component's own output y_k, given the context x_k it was given to work with. Note what this is not: it is not the reranker's own hand-picked relevance metric from Chapter 3. It's a function you're going to train — and the rest of this chapter is entirely about the one property that training has to produce.

The paper's actual implementation is worth seeing concretely, because it's simpler than it sounds: one shared LLM backbone φ encodes the concatenated text [x_k, y_k] into an embedding, and a separate, small, per-component linear head h_k projects that one embedding down to a single scalar reward:

One shared backbone (the paper uses Llama 3 8B Instruct with a LoRA adapter), K cheap linear heads on top — this scales to many components in the system without training K separate full-sized reward models from scratch. Chapter 6 covers exactly how these heads get trained; this chapter is about the mathematical target that training is aimed at.

Deriving what “aligned” has to mean

Start from what you actually need, not from the formula. You need: whenever the LRF prefers one candidate output over another, that preference should be trustworthy — acting on it shouldn't accidentally hurt the system. “Trustworthy” has to be measured against the one thing that actually matters, the global reward R, evaluated on the whole system's final output, not on C_k's output in isolation.

But R depends on everything downstream of C_k too — the same candidate y_k could lead to very different final outcomes depending on what the downstream components (with their own configuration v_−k, the “minus-k” notation meaning “everyone except k”) do with it, and downstream components can themselves be stochastic. So the fair comparison isn't “R for this one downstream run” — it's R averaged over the downstream randomness, holding the rest of the system fixed. That gives the paper's formal property (Eq. 3), the local–global alignment condition:

In words: for every input x, and any two candidate outputs y_k⁺, y_k⁻ that C_k could have produced — whenever the local reward ranks one at least as high as the other, the true expected downstream global reward of that one must also rank at least as high. That's the whole condition. Read it twice, because two things about it are easy to miss and both matter:

How the downstream expectation actually gets computed

The right-hand side isn't hand-waved in the paper — it's an explicit Monte Carlo procedure: “this involves executing the downstream components with the candidate output and the outputs from the non-downstream components fixed, capturing their stochasticity, and averaging the resulting global rewards from the final system outputs.” Walk it through concretely, for the running example's Reranker:

Repeat for a second candidate (list A), and you have both sides of a comparable pair — two numbers estimating what each candidate is actually worth, downstream, in expectation. Whichever one scores higher becomes the label the training procedure in Chapter 6 uses.

python
def estimate_downstream_reward(candidate_y_k, x, downstream_components, downstream_v, R, n_samples=8):
    # Eq. 3's right-hand side, made concrete for one candidate output
    rewards = []
    for _ in range(n_samples):           # captures downstream LLM sampling stochasticity
        y_final = run_downstream(candidate_y_k, x, downstream_components, downstream_v)
        rewards.append(R(x, y_final))
    return sum(rewards) / len(rewards)   # Monte Carlo average -- one comparable number

Why this is enough

This condition is exactly what lets you replace “run the whole downstream pipeline every time you want to compare two candidates” (Chapter 2's expensive obstacle) with “compute one cheap scalar from a trained r_k” — provided r_k was actually trained to satisfy Eq. 3. Under the paper's Theorem 4.1 (stated informally here, proved formally in the paper's Appendix B), if r_k satisfies this alignment property, then maximizing r_k over v_k and maximizing the true downstream global objective over v_k provably arrive at the same solution. That guarantee is what the rest of this session leans on. Chapter 5 makes it concrete with numbers you can check by hand; Chapter 6 shows how r_k actually gets trained toward this property in the first place.

Where the guarantee actually comes from: a derivation, step by step

Theorem 4.1 is not an assumption bolted on afterward — it's a direct consequence of what training a preference model provably does. Walk through the argument in two separate halves: first, that training r_k on downstream-derived preference labels provably produces exactly an aligned reward, not an approximately-aligned one; second, that an aligned reward is genuinely enough to guarantee optimizing it locally helps the whole system.

Half 1 — the training target IS the alignment property, not just close to it. Chapter 6 covers the mechanics of the training loss; here's why its solution is exactly what alignment needs. The preference labels r_k is trained on come from comparing two candidates' true Monte Carlo–estimated downstream rewards (this chapter's Eq. 3 walkthrough) — so the probability that a given comparison instance labels candidate A as preferred over candidate B is itself a function of the gap between their true expected downstream rewards. Model that probability the standard way (a sigmoid of the gap, scaled by some fixed sensitivity β > 0):

The paper's Lemma B.1 answers a precise statistical question about this exact setup: if preference labels are generated by a sigmoid of some true underlying score gap, what function does the log-sigmoid pairwise loss (Chapter 6's Eq. 4) actually converge to when you minimize it over enough data? The answer, proved by taking the loss's derivative with respect to the candidate scoring function and setting it to zero: the unique minimizer is that same true underlying score, up to a positive affine rescaling. Applied here, that means the reward function r_k that minimizes Chapter 6's pairwise loss satisfies

Read this literally: the trained reward is not merely correlated with the true expected downstream reward — at the loss's exact minimizer, it is the true expected downstream reward, stretched and shifted by a positive constant. And because α is strictly positive, that rescaling can never flip an order: whenever the true expected downstream reward of one candidate is at least as large as another's, r_k's ranking of them agrees. That's not an approximation of Eq. 3 — multiplying by a positive number and adding a constant is exactly the kind of transformation Eq. 3 already said was allowed (recall Chapter 4's earlier callout: alignment never required matching R's scale, only its order). The training objective and the alignment property turn out to be the same mathematical statement, viewed from two different directions.

Checking Half 1 with actual numbers. Pick two candidates with known true expected downstream rewards: 𝔼[R|y_A] = 0.3, 𝔼[R|y_B] = 0.7, gap = 0.4. Suppose the preference-label generator has sensitivity β = 2 (a modeling choice, not something to solve for here). The TRUE probability that a randomly-collected preference pair labels B as preferred:

Now check that the reward Half 1 claims you'd learn — r_k(y) = α·𝔼[R|y] with α = β = 2 — reproduces the identical probability when plugged back into the same sigmoid:

The learned reward isn't merely close to the true score — plugged back into the exact same preference model, it reproduces the exact same observable preference probability. That's what “identifiable only up to a positive affine rescaling” means in practice: r_k and the true 𝔼[R|y] are statistically indistinguishable from the preference data alone, which is precisely the sense in which training toward Eq. 4 recovers alignment exactly, not approximately, at its true minimizer.

Half 2 — alignment is genuinely enough to make local greed safe. This is the paper's Lemma B.2, and it's worth seeing the actual two-line argument, because it's shorter than it sounds. Suppose alignment (Eq. 3) holds for C_k, and suppose some update raises C_k's average local reward across the input distribution — 𝔼_x[r_k(x_k,ἂ_k)] > 𝔼_x[r_k(x_k,y_k)], where ἂ_k is the new candidate output and y_k the old one, with every other component's configuration held fixed.

1. Fix one input x. Because only C_k's configuration changed, r_k(x_k,ἂ_k) ≥ r_k(x_k,y_k) holds for (almost) every individual x, not just on average — whatever made the average go up did so by helping at the level of individual inputs.
2. Apply Eq. 3 pointwise, for this one x. Alignment says: whenever the local reward ranks ἂ_k at least as high as y_k, the true downstream expected global reward, conditioned on this x, ranks ἂ_k at least as high too.
3. Take the expectation over x (law of total expectation). An inequality that holds for every individual x still holds after averaging over all of them — averaging a collection of numbers that are each ≥ their counterpart cannot flip the direction of the overall comparison.

That's the whole proof. Nothing in it required knowing the exact numeric value of R, the internal workings of the downstream Generator or Verifier, or how large the improvement in r_k was — only that alignment held pointwise and that averages preserve pointwise inequalities. This is precisely why Chapter 7's loop can treat “r_k went up” as license to keep a change, subject to the validation gate that catches the cases where staleness (Chapter 6) has quietly broken the pointwise assumption in Step 1.

Concept → realization: the exact computation behind the slider

The widget's accuracy readout isn't decoration — it's the same pairwise-comparison logic Table 4's real 77.96% and 49.52% numbers come from, just run on 8 synthetic points instead of a held-out test set. Every pair of points gets checked once, and the sliders's “true expected downstream reward” ordering is compared against the trained reward's ordering:

python
def pairwise_ranking_accuracy(true_R, learned_r):
    # true_R, learned_r: lists of scores for the same N candidates, same order
    n = len(true_R)
    correct, total = 0, 0
    for i in range(n):
        for j in range(i+1, n):
            total += 1
            true_order = true_R[i] >= true_R[j]
            learned_order = learned_r[i] >= learned_r[j]
            if true_order == learned_order:
                correct += 1
    return correct / total   # exactly what Table 4 reports, per system, averaged

With 8 candidates there are “8 choose 2” = 28 pairs to check, every time the slider moves. At the untrained end (near 0%), the trained reward is mostly noise, and a random ordering agrees with the true one on roughly half of all pairs by chance — the same coin-flip territory Table 4's LLM Judge baseline landed in, 49.52%. At the fully-trained end (100%), Half 1's derivation from earlier this chapter applies exactly: r_k = α·R with α > 0 preserves every pairwise order, so accuracy hits 100%, the theoretical ceiling that Table 4's real 77.96% approaches but doesn't reach — a live, small- scale demonstration of exactly the gap between an idealized minimizer and what finite real-world training data actually delivers.

Chapter 5: A Worked Toy

Chapter 4's condition is precise, but precise definitions can still feel abstract until you watch them bite on actual numbers. Build the smallest possible system where alignment's absence causes real, measurable damage — and where you can check every number by hand.

The setup

Two components. A Retriever picks exactly one of two candidate passages, A or B, for a given question. A fixed Generator (same model, same prompt throughout) answers using whichever passage it's handed, scored by F1 against the ground-truth answer — this F1 score is the global reward R for this toy. This is the running four-component pipeline collapsed to its simplest possible form: one upstream choice, one downstream consequence, nothing else in the way.

Four toy questions. Passage A always looks more relevant — it shares more literal words with the question (measured as a keyword-overlap score, the naive local metric from Chapter 3). Passage B is phrased differently, using synonyms and paraphrase, and shares fewer words with the question — but B is the one that actually contains the fact the Generator needs. This is not a contrived pathology; it's a well-documented failure mode of any retrieval signal built on lexical overlap: the passage that says the answer in different words loses to the passage that merely echoes the question's vocabulary.

Look at the pattern across every row: overlap(A) > overlap(B) every single time, while F1(B) > F1(A) every single time too. Keyword overlap and downstream answer quality are not just uncorrelated here — they point in opposite directions on all four queries. This is Chapter 3's misalignment, distilled to its cleanest possible form.

Strategy 1: greedy by keyword overlap (Chapter 3's naive local metric)

For every query, hand the Generator whichever passage scores higher on overlap. Since overlap(A) beats overlap(B) every time, this strategy picks A on all four queries, no exceptions:

Strategy 2: greedy by an aligned local reward

Now suppose, instead, the Retriever's local reward r₁ was trained the way Chapter 6 describes: for each query, sample both candidates, estimate their downstream F1 via the Generator (exactly the Monte Carlo procedure from Chapter 4), and label whichever passage produced the higher F1 as preferred. Since F1(B) > F1(A) in every one of these four rows, every single training pair says “B beats A” — and a correctly trained r₁ will rank B above A on all four queries too. This is Chapter 4's alignment property holding by construction: the training labels came directly from the downstream F1 comparison itself, so r₁(x, B) ≥ r₁(x, A) ⇒ expected downstream F1 of B ≥ that of A is satisfied exactly. Greedy-by-r₁ therefore picks B every time:

The gap, and what caused it

One flip — which passage gets handed to the Generator — moves this two-component system from badly serving its users (26.25% F1) to strongly serving them (85% F1). Nothing about the Generator changed. Nothing about the ground-truth answers changed. Nothing about the questions changed. The only thing that changed between Strategy 1 and Strategy 2 is which local signal decided the Retriever's one choice.

python
queries = [
    {"overlap_A": 0.82, "overlap_B": 0.35, "F1_A": 0.20, "F1_B": 0.90},
    {"overlap_A": 0.75, "overlap_B": 0.40, "F1_A": 0.30, "F1_B": 0.85},
    {"overlap_A": 0.68, "overlap_B": 0.55, "F1_A": 0.45, "F1_B": 0.70},
    {"overlap_A": 0.90, "overlap_B": 0.20, "F1_A": 0.10, "F1_B": 0.95},
]

greedy_overlap_F1 = [q["F1_A"] if q["overlap_A"] > q["overlap_B"] else q["F1_B"] for q in queries]
greedy_aligned_F1 = [max(q["F1_A"], q["F1_B"]) for q in queries]  # aligned r1 always finds the true winner

print(sum(greedy_overlap_F1) / len(queries))   # 0.2625 -- misaligned local metric
print(sum(greedy_aligned_F1) / len(queries))    # 0.85   -- aligned local reward

A second worked toy: the model selector's leaderboard trap

The first toy used a proxy that varies per query but points the wrong direction. This second toy uses a proxy that's even more common in practice, and arguably more dangerous precisely because it sounds objective: a context-free metric — one that never even looks at the specific question in front of it.

Two components again. A Model Selector (mirroring PubMedQA's real Context Model Selector, which the paper implements as a discrete choice among 7 candidate LLMs) picks between exactly two candidate models, Model X and Model Y, for each incoming biomedical question. A fixed downstream classifier answers using whichever model is selected, scored as correct (1) or incorrect (0) against the ground-truth label — PubMedQA's real evaluation metric is exact-match accuracy, so this per-query score genuinely is binary, not a continuous F1.

Model X has the higher score on a general public leaderboard — 71% overall, averaged across thousands of unrelated benchmark questions. Model Y's leaderboard score is lower, 63%. Crucially, that leaderboard number is the same fixed constant for every query — it carries no information at all about which model is actually better on this specific biomedical question. Six real per-query outcomes:

Because leaderboard(X) > leaderboard(Y) on every row — it's a constant, so it has to be — greedy-by-leaderboard selects Model X on all six queries, with no per-query variation possible at all:

Now suppose the Model Selector's local reward was trained the way Chapter 6 describes — per query, sample both candidates, run the fixed downstream classifier on each, and label whichever one was actually correct as preferred. On Q5, neither model is correct, so there's no valid preference pair to collect from that query at all (both candidates score 0; Chapter 6's loss needs a genuine winner to learn from) — the aligned reward simply has nothing to prefer there, and the system is stuck with whichever default runs. On every other query, the aligned selector correctly identifies the true winner:

Notice what this toy adds that the first one didn't: the aligned strategy is not magic. It still gets Q5 wrong, because neither available model was capable of answering it — alignment guarantees picking the best available option, not omniscience. And notice what made greedy-by-leaderboard fail so completely here: a context-free proxy applied to a per-instance decision doesn't just point in the wrong direction sometimes, the way keyword overlap did in the first toy — it points in exactly one fixed direction on every single instance, incapable of ever being right about the cases where the weaker-on-average model happens to be the better-on-this-question model. This is also the honest reason the paper's own real PubMedQA result (Chapter 8's Table 2) shows the smallest relative improvement of the five systems, 1.8% — per-instance model selection often has less headroom to exploit than the sharper, more consistently-directional misalignments in this session's other four systems.

python
queries = [
    {"correct_if_X": 1, "correct_if_Y": 0},
    {"correct_if_X": 0, "correct_if_Y": 1},
    {"correct_if_X": 0, "correct_if_Y": 1},
    {"correct_if_X": 1, "correct_if_Y": 0},
    {"correct_if_X": 0, "correct_if_Y": 0},
    {"correct_if_X": 0, "correct_if_Y": 1},
]
# leaderboard(X)=0.71 > leaderboard(Y)=0.63 on EVERY query -- greedy always picks X
greedy_leaderboard = [q["correct_if_X"] for q in queries]
# aligned reward picks whichever was actually correct on THIS query
greedy_aligned = [max(q["correct_if_X"], q["correct_if_Y"]) for q in queries]

print(sum(greedy_leaderboard) / len(queries))   # 0.333 -- context-free proxy
print(sum(greedy_aligned) / len(queries))       # 0.833 -- per-query aligned reward

A third worked toy: the hint generator's brevity trap

One more, because the direction of the naive fix matters, not just the fact that it's naive. It would be tempting to conclude, from the first two toys, that the fix is always “prefer whatever the naive metric under-ranks.” It isn't. Alignment tracks the true objective — sometimes that means overriding the naive heuristic, sometimes it means agreeing with it. This toy shows both happening in the same five queries.

The Hint Generator (HotpotQA's real fourth module) can produce either a short hint or a long hint for the Answer Generator to work from. A convenient local metric scores hints on raw brevity — fewer tokens, higher score — on the reasonable-sounding theory that a concise hint is a clean hint. The true downstream signal is the Answer Generator's F1 once it actually tries to answer using that hint.

On Q1, Q2, and Q4, the short hint drops a disambiguating detail a multi-hop question actually needs — naming which director, not just “the director” — and F1 collapses. On Q3, both hints are adequate and the long one edges ahead slightly, since the extra context doesn't hurt. On Q5, something different happens: the long hint actually makes things worse, F1 dropping from 0.65 to 0.60, because the extra detail introduces a distracting second entity the Answer Generator latches onto incorrectly. Greedy-by-brevity always picks “short” (brevity(short) > brevity(long) on every row, by construction of the metric):

The aligned reward, trained to prefer whichever hint actually produced higher downstream F1 on each query, correctly picks “long” on Q1, Q2, Q3, and Q4 — and correctly picks “short” on Q5, because that's what the data says is actually better there, even though it agrees with the naive brevity heuristic on that one row:

python
queries = [
    {"F1_short": 0.25, "F1_long": 0.92},
    {"F1_short": 0.30, "F1_long": 0.88},
    {"F1_short": 0.70, "F1_long": 0.75},
    {"F1_short": 0.20, "F1_long": 0.85},
    {"F1_short": 0.65, "F1_long": 0.60},
]
# brevity(short) > brevity(long) on EVERY row -- greedy always picks "short"
greedy_brevity_F1 = [q["F1_short"] for q in queries]
# aligned reward: whichever hint actually produced higher F1 -- NOT always "long"
greedy_aligned_F1 = [max(q["F1_short"], q["F1_long"]) for q in queries]

print(sum(greedy_brevity_F1) / len(queries))    # 0.42 -- brevity heuristic
print(sum(greedy_aligned_F1) / len(queries))     # 0.81 -- aligned, picks short OR long per query

Chapter 6: Training a Reward to Stay Aligned

Chapter 5's “aligned r₁” was built by hand, by definition, for a toy with four rows. A real LRF has to be learned from data, and it has to keep working as the system around it keeps changing. This chapter covers both.

The training objective: a pairwise ranking loss

To make r_k satisfy Chapter 4's ranking-agreement property, the paper trains it with a pairwise log-sigmoid ranking loss — the same Bradley-Terry-style preference loss family behind RLHF reward models, applied here per component instead of once at the end of a whole system:

Read it as: take the gap between the reward assigned to the preferred output and the reward assigned to the rejected one, push that gap through a sigmoid, and push the result toward 1 (equivalently, push the loss toward 0). Crucially, this never needs the absolute downstream reward values — only which of the pair was better. That's exactly the information Chapter 5's toy training labels carried (“B beats A,” every row) — this loss is precisely the mechanism that would learn Chapter 5's Strategy 2 behavior directly from those four preference pairs.

What the loss actually penalizes, by the numbers

Two small worked cases show the loss doing its job. Suppose a well-separated, correctly ordered pair: r_k(y⁺) = 2.1, r_k(y⁻) = 0.4, gap = 1.7.

Now flip the ordering — same two scores, but assigned to the wrong candidates: r_k(y⁺) = 0.4, r_k(y⁻) = 2.1, gap = −1.7.

An order of magnitude more loss (1.868 vs 0.168) for getting the ranking backwards versus getting it right with the same margin. The loss doesn't just gently nudge toward correct orderings — it punishes reversed ones sharply, which is exactly the pressure that would eventually train r₁ out of Chapter 5's “always pick A” failure mode and into “always pick B.”

python
import math

def pairwise_loss(r_plus, r_minus):
    gap = r_plus - r_minus
    sigma = 1 / (1 + math.exp(-gap))
    return -math.log(sigma)

print(pairwise_loss(2.1, 0.4))   # ~0.168 -- correctly ordered, low loss
print(pairwise_loss(0.4, 2.1))   # ~1.868 -- backwards, much higher loss

Where the preference pairs come from

Section 4.1's collection recipe, formalizing Chapter 4's Monte Carlo walkthrough into three concrete steps:

Why one training pass isn't enough: staleness

The system's configuration keeps changing during optimization — that's the whole point of running this loop at all. An r_k trained under a past configuration v^t can go stale under an updated v^t+1, for two distinct, precisely-named reasons:

1. Upstream drift. After updating C_k itself — say the Generator gets a new prompt — the same outputs from an upstream component (the Reranker) may now lead to a different global reward than before, so r_reranker, trained against the old Generator, is measuring alignment with a Generator that no longer exists.
2. Downstream distribution shift. Downstream components (the Verifier) now receive inputs generated by the updated Generator, which may fall outside the distribution r_verifier was ever trained to score — an out-of-distribution problem, distinct from a ranking-direction problem.

Retraining every LRF from scratch after every single configuration change would defeat the entire point of using cheap local rewards. The paper's fix is a lightweight two-stage design instead:

To keep those small-batch updates stable and stop them from immediately overwriting everything learned before, the paper maintains a buffer of previously collected preference data and mixes it into every mini-batch — the same experience-replay idea from reinforcement learning, applied here to preference pairs instead of environment transitions.

It's worth seeing why “mix in the buffer” matters and isn't just conservative caution, with a small illustrative numeric picture (not a number the paper reports, but the mechanism it's guarding against). Suppose an LRF has already learned to rank 200 preference pairs correctly, 85% pairwise accuracy, from its offline Stage 1 training. A configuration change is accepted, and Stage 2 collects 10 brand-new preference pairs reflecting the new configuration, then takes a few gradient steps only on those 10. With so few examples and no anchor back to the previous 200, a handful of gradient steps can overfit hard to those 10 pairs' particular quirks — plausibly dragging accuracy on the original 200 down to something like 60–70%, even while the new 10 are fit almost perfectly. That's catastrophic forgetting in miniature: solving today's small, narrow update at the cost of yesterday's broader competence.

Mixing a sample of the buffered older pairs into the same mini-batch as the 10 fresh ones changes what the gradient step is actually optimizing: instead of “fit these 10 examples,” it becomes “fit these 10 new examples without moving off the older ones the buffer keeps represented.” The update still adapts to the new configuration — that's the whole point of Stage 2 existing — it just can't do so by quietly erasing everything Stage 1 already got right. This is the same reason experience replay exists in RL: a handful of the most recent transitions is a biased, narrow window onto the environment, and training only on it tends to overfit to whatever's most recent rather than genuinely improving the policy.

Why lightweight adaptation is even viable

This only works if a small amount of fresh data genuinely recovers most of an LRF's alignment quality — and the paper measures this directly on its real HotpotQA system. Training on just 12.5% of the available preference data yields 65.46% pairwise ranking accuracy, which the paper reports as 92.7% of the accuracy achieved with the full dataset. Back-solve the implied full-data number:

Using 25% of the data instead recovers 95.1% of full performance — check it against the same implied full-data figure:

A small mini-batch really does get you most of the way there. That's the empirical justification for Stage 2's entire design: online adaptation doesn't need to be expensive to be effective.

Concept → realization: why this loss provably converges to an aligned reward, not just approximately

Chapter 4's derivation already did the hard work here — it's worth restating in one sentence now that the actual loss form is in front of you. Because the preference label for every (y⁺, y⁻) pair comes directly from comparing their true Monte Carlo–estimated downstream rewards, minimizing this exact pairwise log-sigmoid loss is a statistical estimation problem whose unique solution — by the paper's Lemma B.1, derived step by step in Chapter 4 — is a strictly increasing function of the true expected downstream reward. The loss you're looking at right now is not a reasonable-sounding heuristic that happens to correlate with alignment. Its minimizer is alignment, given enough representative preference data. What finite, noisy, real-world data buys you instead of that idealized minimizer is exactly what Chapter 9's retriever-sweep evidence shows: alignment holding approximately, with the right configurations still identifiable as a set, rather than holding with mathematical exactness everywhere.

Concept → realization: how small “K small linear heads” actually is

It's worth putting real numbers on “cheap” here, even though this specific arithmetic is an estimate for this lesson, not a figure the paper reports directly. Llama 3 8B's hidden dimension is 4,096. Suppose the LoRA adapter (rank 32) targets the four attention projection matrices in each of the model's 32 transformer layers, each matrix roughly 4,096×4,096. A rank-32 LoRA adapter on one such matrix adds two small matrices, A (4,096×32) and B (32×4,096), instead of updating the full 4,096×4,096 = 16.8 million- parameter matrix directly:

Now add the K per-component heads on top. Each head is nothing more than one linear layer mapping the backbone's 4,096-dimensional embedding down to a single scalar reward — 4,096 weights plus 1 bias, 4,097 parameters per component. For HotpotQA's real 5-module pipeline:

This is the concrete payoff of “one shared backbone, K small heads”: adding a sixth component to the pipeline costs about 4,097 more parameters to train, not another 33.6-million-parameter adapter and certainly not another 8-billion-parameter model. The expensive part of an LRF — understanding language well enough to judge an (x_k, y_k) pair at all — is paid for exactly once, by the shared backbone. What's per-component is deliberately almost free.

How much fresh data Stage 2 actually needs — the real ablation

“Whenever a configuration change is accepted” (Table 7's real hyperparameters, Chapter 7) specifies a fresh input size of 20. That number wasn't picked arbitrarily — the paper's Appendix G.3 sweeps it directly on HotpotQA, holding everything else fixed and varying only how many new inputs feed each adaptation round:

Read the deltas by hand, relative to the peak at 20:

Performance is non-monotonic in the amount of fresh data — it rises from 10 to 20 new inputs, then falls at 30 and falls further at 40. More adaptation data is not simply better, past a point. A plausible mechanism, consistent with everything else this chapter has covered: a large fresh batch pulls each accepted-update's mini-batch further away from the buffer's older preference data, destabilizing the exact thing the buffer (this section's earlier paragraph) exists to stabilize — each adaptation step starts drifting away from a good local optimum rather than refining it. This is the concrete, measured reason Table 7 fixes fresh input size at 20 rather than “as much as you can afford”: past a certain point, more data actively hurts the exact property Stage 2 is supposed to preserve.

Chapter 7: The Optimas Loop

Chapters 4 and 6 built the two ingredients — the alignment condition, and how to train toward it. This chapter assembles them into the actual algorithm the paper runs, Algorithm 1, walked through in plain language first and then as pseudocode.

Setup

Components 𝒞 = {C₁, …, C_K}; an initial configuration v⁰; initial LRF parameters Θ⁰ (Chapter 6's Stage 1 output); a training set 𝒟 and a separate held-out validation set 𝒟_v; a total iteration budget T.

One iteration, step by step

1. Scheduler. Pick one component i_t ∼ Uniform({1,…,K}) — literally a coin flip across all K components. Not round-robin, not “whichever looks worst.” A deliberately simple design choice.
2. Local optimization on that one component, branching by its configuration type:
   • Prompt → rank candidate prompts by average local reward over a sample, keep the best (an OPRO-style search using r_it as the scoring function)
   • Trainable weights → run PPO, with r_it standing in as the critic
   • Discrete/low-dimensional (model selection, a hyperparameter like retriever k) → build a probability distribution over candidates proportional to exp{r_it(…)}, sample the update
   This produces one candidate new configuration for component i_t alone; every other component is held fixed.
3. Form the candidate policy. Take v^t, replace only slot i_t with the new candidate, call the result ḣ (v-tilde).
4. Validate. Run the whole system with ḣ on the held-out 𝒟_v, and compare the summed global reward against the current v^t's summed reward on the same set.
5. Accept / reject gate. If ḣ's validation total is strictly higher — accept: v^t+1 ← ḣ, and (because the configuration genuinely changed) run Chapter 6's Stage 2 adaptation, collecting fresh preference data and updating Θ. If not — reject: v^t+1 ← v^t, Θ unchanged, nothing spent on adaptation for a move that didn't happen.

python
def optimas_loop(components, v0, theta0, R, train_set, val_set, T):
    v_star, v = v0, v0
    theta = theta0
    for t in range(T):
        i_t = random.choice(list(components))          # uniform scheduler
        candidate_vk = local_optimize(components[i_t], v[i_t], theta[i_t])  # branch by config type
        v_tilde = v.copy(); v_tilde[i_t] = candidate_vk

        if sum(R(x, run_compound_system(x, components, v_tilde)) for x in val_set) > \
           sum(R(x, run_compound_system(x, components, v))       for x in val_set):
            v = v_tilde; v_star = v_tilde                    # accept: validation genuinely improved
            prefs = collect_preference_data(train_set, theta, k=50)
            theta = reward_model_train(theta, prefs)     # Stage 2 adaptation, only on accept
        # else: reject -- v and theta both unchanged, nothing wasted
    return v_star

Why the validation gate is the load-bearing piece

Chapter 4/5's alignment guarantee is a statement about r_k at the moment it was trained. Chapter 6 showed r_k can go stale as the system evolves. The validation gate is what converts a per-component guarantee that might occasionally be slightly wrong (due to staleness) into an actual monotonically-non-decreasing guarantee for the whole running algorithm: even if one local-optimization step proposes something that a stale LRF thought was an improvement but genuinely isn't, the gate checks it against real held-out data before it's ever accepted. The reported best configuration v^★ only ever moves when validation confirms an actual gain.

Trace one concrete rejection to see the gate doing real work. Suppose the Verifier's LRF, trained slightly before the Generator's most recent prompt update, scores a proposed new Verifier prompt as +0.05 better on average local reward — a real, measured improvement, by its own (possibly stale) lights. Step 4 then runs both the current and candidate configurations on the actual 20-example validation set. Because the Generator changed underneath it since this LRF was last trained (Chapter 6's “upstream drift”), the Verifier's notion of a good output no longer matches what the current Generator actually tends to produce — and the measured validation total for the candidate comes back lower than the current best, say by 0.03 in summed global reward. Step 5's gate rejects it outright, discards the candidate, and leaves v^t untouched. The LRF's own +0.05 belief was simply wrong, and nothing about that error propagates into v^★ — it dies at the gate, exactly where Theorem 4.1's alignment guarantee (which assumes a freshly-trained r_k) stops being trustworthy on its own.

The trust-region idea, and how it's actually implemented

In principle, each local optimization step should stay within a “trust region” of the previous configuration, so the LRF — trained on data near the old configuration — is only trusted to extrapolate a small distance:

In practice, a literal distance function d is awkward across such different configuration types — a prompt edit, a PPO gradient step, and a hyperparameter jump don't share a natural notion of “distance.” So the real implementation approximates the trust region with something simpler: a small, fixed number of update steps per iteration. Table 7's real hyperparameters: a train size of 50, a search size of 50, 3 prompt candidates per round, 3 local optimization steps, a fresh input size of 20, and a validation size of 20 — conservative by construction, rather than by an explicit geometric bound.

Concept → realization: what “3 local optimization steps” actually spends

Table 7's hyperparameters aren't just labels — each one is a concrete count of API calls and pipeline runs. For a prompt-controlled component, the paper's Appendix F spells out the two separate roles a “50” plays: train size (50) is how many examples are used while training that module's local configuration candidates, and search size (50) is how many samples are used when ranking those candidates against r_k to pick the best one — two different 50's, spent on two different jobs. Layer “3 prompt candidates” × “3 local optimization steps” on top: in the worst case, up to 3×3 = 9 distinct candidate prompts get proposed and scored across one component's full local-optimization phase for a single scheduler pick — each one scored by average r_k over the search-size sample, never by running the whole downstream pipeline. That's the concrete anatomy behind Chapter 2's abstract claim that local optimization is cheap: single-digit candidate counts, evaluated with one cheap trained scalar apiece, not the 2,500,000-run grid from that chapter's brute-force estimate.

Concept → realization: what “sample proportional to exp{r_k}” means, numerically

The third branch of step 2 — discrete or low-dimensional configurations, like PubMedQA's Model Selector — deserves one concrete pass, since “a probability distribution over candidates proportional to exp{r_k(…)}” can stay abstract otherwise. Suppose, for one question, the trained LRF scores three candidate LLMs: Claude gets r_k = 0.62, GPT-4o gets 0.58, GPT-4o-mini gets 0.51. Exponentiate each and normalize:

python
import math
scores = {"Claude": 0.62, "GPT-4o": 0.58, "GPT-4o-mini": 0.51}
weights = {k: math.exp(v) for k, v in scores.items()}
total = sum(weights.values())
probs = {k: w/total for k, w in weights.items()}
print(probs)   # {'Claude': 0.350, 'GPT-4o': 0.336, 'GPT-4o-mini': 0.314}

Notice this isn't a hard argmax — Claude has the highest local reward but only a 35% chance of actually being sampled, not certainty. That softness is deliberate: r_k is trained on finite data (Chapter 6) and can be close to a tie between genuinely competitive candidates, exactly the near-tie situation Chapter 9's retriever-k sweep will show happening for real (k=5 versus k=10, a 0.0007 gap). Sampling proportionally, rather than always taking the single highest-scoring candidate, keeps a small amount of exploration alive around close calls instead of committing permanently to whichever option happened to score a hundredth of a point higher on this particular batch of preference data.

Why “more candidates per step” isn't automatically better — the real ablation

It would be reasonable to guess that considering more candidate prompts per optimization step can only help. The paper's Appendix G.2 tests this directly on HotpotQA, sweeping the prompt-candidates hyperparameter from 3 up to 10 while holding everything else fixed:

Not monotonic in either direction — the best result comes from 7 candidates, and 5 is worse than the smaller setting of 3 right next to it. The paper's own reading of this is measured rather than triumphant: “exhaustive candidate pools are unnecessary, and results with fewer candidates remain competitive.” The practical takeaway Table 7 bakes in (3 candidates, 3 steps) isn't “the provably optimal setting” — it's “a modest, cheap setting that's already close to what a much larger search would find,” which is exactly the trust-region philosophy from the paragraph above: a search intentionally kept small enough that r_k is only ever asked to rank candidates near where it was actually trained.

Why the gate needs strictly greater, not just at least as greater

Look again at Algorithm 1's accept condition: ḣ's validation total must be strictly higher than v^t's — not “at least as high.” Concretely, on a 20-example validation set (Table 7's real validation size), suppose a proposed update's summed reward is 14.10 and the current best's is also 14.10, an exact tie. The gate rejects. This isn't pedantry: Table 2's own reported standard deviations are large relative to some of these gaps — Amazon's accuracy varies ±0.82 to ±3.78 points across methods just from run-to-run noise. A gate that accepted ties, or near-ties within noise, would let the configuration drift on pure variance, with nothing to stop it random-walking through updates that only ever look identical to the status quo. Requiring strict improvement on real held-out data is a small design choice with an outsized job: it's the difference between a guarantee that v^★ only ever moves on signal, versus one that can wander on noise.

Chapter 8: Five Pipelines, Read Honestly

Everything so far has been derived and toy-verified. This chapter holds it up against the paper's actual experiments — five real compound systems, each one, in its own way, an instance of the retriever–reranker–generator–verifier family this session opened with.

The five real systems

Two of these are the running example almost verbatim: STaRK-Prime's two scorers feeding a weighted Aggregator is a reranking stage; BigCodeBench's Unit-Test-Generator-then-Final-Code-Generator sequence is a verify-then-regenerate loop. This isn't a toy family of problems — it's how real, deployed pipelines are actually shaped.

What these five tasks actually ask a compound system to do

The metrics and component types in the table above describe the machinery. It's worth spending one paragraph on what a human would actually be asking the system to accomplish, per the paper's own Appendix C descriptions — the concrete substance behind each abbreviation:

None of these are toy benchmarks chosen because they're easy to optimize. Each one is exactly the kind of task where a single LLM call, unaided, genuinely struggles — which is precisely why Table 2's “Single LLM” row, covered later this chapter, is so much weaker than every compound system built on top of the same underlying models.

Table 2: the headline result, reproduced by hand

Average the five “Rel. Improv.” numbers by hand, exactly as the paper's own headline claim does:

That's the paper's abstract, reproduced from its own table. And this table already contains Chapter 3's inconsistency, in full context: DSPy is the worst method on Amazon (18.18, below Unoptimized's 21.21) while being the best baseline on HotpotQA (44.90) — a real number Optimas beats by exactly the reported 12.4%: (50.48−44.90)÷44.90×100% ≈ 12.43%, matching the table. Optimas is the only method in the whole comparison that improves over Unoptimized on every one of the five systems, not just on average.

Table 3: matched cost, so the win isn't just “tried more things”

Optimas is the cheapest of the three on average, while also being the most accurate. Two more real numbers the paper reports directly: REINFORCE needs “more than three times the data” Optimas needs, because it collects a single shared reward signal via costly downstream Monte Carlo rollouts across the whole system rather than a dedicated per-component signal. LLMSelector needs “2.8k times… 3x more expensive than Optimas” in full-system forward passes.

Table 4: does the alignment property actually hold, measured directly?

Pairwise ranking accuracy: how often the method's score puts the higher-global-reward candidate above the lower-global-reward one, on held-out pairs. Average both rows by hand:

The LLM Judge baseline — a GPT-4o model scoring outputs from 20 in-context examples, structurally the same kind of unvalidated signal behind TextGrad's textual feedback loop — is empirically Chapter 3's whole argument, measured directly: asking a model to informally judge which output is better, without training it against the actual downstream metric the way Chapter 6's loss does, gets you almost nothing above random guessing at this specific task.

Interpretability and efficiency, honestly reported

On HotpotQA's Answer Generator, the trained LRF learns to prefer shorter outputs — consistent with F1 penalizing verbose non-matching answers — and the resulting prompt-optimization step explicitly adds a length constraint to the module's instructions (Figure 6). This is a recoverable, inspectable signal, not a black box: perturb an output's length and watch r_k respond.

On efficiency, Chapter 6 already previewed the data-efficiency numbers (12.5% of data → 92.7% of full performance). The paper's backbone-size ablation is worth reporting exactly as measured, because it's genuinely counter-intuitive: 1B parameters → 72.58% alignment accuracy, 3B → 70.04%, 8B → 71.20%. The smallest backbone is actually the best of the three — a spread of only 2.54 points across an 8× parameter range, functionally flat. A bigger reward-model backbone is not what's buying alignment quality here.

Table 11: the “cheapest AND most accurate” claim, accounted for down to the run

Chapter 2's cost claim used a derived estimate. Table 3's numbers are real, but presented as totals. The paper's Appendix G.4 shows its actual arithmetic on the Amazon system — the one system with trainable local models, so the only one that needs a full accounting of PPO training cost alongside everything else. Every number below is real and independently checkable.

Amazon has three components in topological order, labeled A (Session Analyzer), B (Candidate Profiler), C (Next Item Decider). Optimas's total of ≈0.31k effective full-system runs decomposes into exactly three pieces:

1. LRF training (once, up front): 60 initial preference pairs collected per component, 2 full system runs needed per pair (one run per candidate in the pair) → 60 × 2 = 120 runs.
2. LRF adaptation (Stage 2, Chapter 6): across the whole optimization run, the LRFs get updated 5 times, each update collecting 10 fresh preference pairs per component, again 2 runs per pair → 5 × 10 × 2 = 100 runs.
3. Global validation (Chapter 7's gate): 20 validation inputs, 7 accepted updates actually triggered a validation pass. The naive count would be 20 × 7 = 140, but caching trims it: whichever component was just updated, only that component and everything downstream of it needs to be re-run on a validation input — the rest of the trajectory is cached from before. Averaged across A, B, and C's topological positions, that's a 2/3-of-a-full-run discount:

Sum the three pieces, exactly as Table 11 does:

DSPy and TextGrad, by contrast, only ever optimize component C's prompt, holding A and B fixed — so their cost converts differently: pre-compute a pool of 20 validation inputs by running A and B once, then run only C during each optimization step. Since A, B, and C are assumed comparably expensive, a C-only call is worth roughly 1/3 of a full system run:

Every one of these numbers was previously just a total in a table. Walked through by hand, they show precisely where Optimas's cost goes — a fixed, one-time LRF training cost, a small ongoing adaptation cost tied directly to how many updates actually get accepted, and a validation cost that's discounted by caching whatever didn't change. None of that is optional bookkeeping baked in to make the number look good after the fact; it's the same accounting Table 3's headline “0.71k average, cheapest of the three” is built from.

Why compound structure itself already beats a bare LLM, before any optimization at all

One comparison this chapter hasn't isolated yet: not “does optimizing the pipeline help,” but “does having a pipeline at all help,” compared to firing one LLM at the raw task with no structure whatsoever. Table 2 reports this “Single LLM” baseline for every system:

STaRK-Prime's Single LLM row is the sharpest data point in the whole table: exactly 0.00 MRR. Not low — zero. A single LLM call, asked to directly rank items from a biomedical knowledge base with no retrieval component feeding it candidates at all, cannot produce a meaningful ranked list; there's nothing for it to rank. This is the honest answer to a question this session has quietly assumed the answer to since Chapter 0: why build a compound system instead of just prompting one strong model harder? For at least one of the paper's five real tasks, the answer isn't “a pipeline is more accurate” — it's “a pipeline is the only way the task gets attempted at all.” Optimizing the local rewards well (this session's whole subject) is a second, separate win layered on top of a structural one.

Chapter 9: Limits & Connections

Every lesson in this series ends the way this paper's own results section does: honestly. Optimas works, and works well — but the guarantees behind it come with real, stated boundaries.

Boundary 1: alignment is trained and approximate, not exact

The paper's own Appendix G.1 sweeps HotpotQA's retriever hyperparameter k and compares the local reward against the true global reward at every value — a direct empirical test of whether alignment holds in a real, messy setting, not just in Chapter 5's clean toy:

The local reward peaks at k = 5. The true global reward peaks at k = 10 — not the same point. The top-3 configurations coincide as a set, {5, 10, 15}, but in a different order (local: 5 > 10 > 15; global: 10 > 5 > 15). Look closely at the size of the disagreement: the local reward's own top-2 gap (k=5's 0.6124 vs k=10's 0.6117) is a near-tie, just 0.0007 — while the true global reward at those same two points differs by a real 0.0047 (0.3645−0.3598), roughly 7× larger in relative terms. Alignment held well enough to be genuinely useful — the top-3 sets agree, the paper's own optimization loop lands on good configurations — but not exactly. Theorem 4.1 only claims the ranking-agreement property holds at the loss's minimizer under stated regularity conditions; it never claims numerical values match everywhere. Table 8 shows that property holding approximately, not perfectly, in a real deployed system.

Boundary 2: no guarantee of a global optimum

Theorem 4.2 proves the algorithm converges to a point where no single-component change can improve things further. The paper states its own caveat directly: “the block-coordinate (round-robin) updates adopted in Optimas do not guarantee global optimality in non-convex problems… our global convergence guarantees only hold under additional structural assumptions, such as Polyak–Łojasiewicz or Kurdyka–Łojasiewicz conditions.” Translated: two or more components changing together, in a coordinated way neither would attempt alone, might still find something better — and because every iteration touches exactly one component, the algorithm structurally cannot see that possibility.

Boundary 3: you need something to train on

The entire alignment machinery assumes two things are available: a way to cheaply sample multiple candidate outputs per component, and a computable-or-estimable global reward R for every training instance. Tasks with no automatic metric — open-ended creative writing, genuinely subjective calls with no ground truth — don't hand Chapter 6's loss the preference labels it needs. There's nothing for Eq. 4 to train against without some scorable global signal in the first place.

This isn't necessarily fatal for those tasks — it's a scoping boundary, not a dead end. Nothing in Theorem 4.1's derivation (Chapter 4) required R to be an automatic metric specifically, only that it be a well-defined scalar function you can evaluate. Reward Learning covers how R itself can be built from human preference labels when no automatic ground truth exists, exactly the way RLHF reward models are trained. In principle, Optimas's machinery could sit on top of a learned R the same way it sits on top of F1 or accuracy here. In practice, the paper's own five experiments only test tasks with clean automatic metrics — whether the whole pipeline still behaves well when R is itself a noisy, learned approximation rather than a ground-truth score is a real open question the paper doesn't claim to answer.

Boundary 4: scheduling doesn't prioritize

The uniform-random component scheduler (Chapter 7, step 1) spends iterations on every component roughly equally, regardless of how close each one already is to its own local optimum. In a system with far more than five components, that's real wasted budget. The paper's own stated future work is scaling to “even larger systems” — exactly where this becomes costly rather than cosmetic.

Boundary 5: what “convergence” is actually promising, in plain language

Theorem 4.2's convergence guarantee leans on structural assumptions with intimidating names — Polyak–Łojasiewicz (PL) or Kurdyka–Łojasiewicz (KL) conditions. Stripped of notation, a PL condition is a promise about the shape of the optimization landscape: it rules out wide, flat regions where the objective barely changes no matter which direction you move, far from any actual maximum. Without some such condition, a coordinate-ascent algorithm can get stuck making technically-non-negative but practically-negligible progress forever, never actually approaching a good solution. With it, the paper's convergence result has teeth: not just “the algorithm never gets worse” (Chapter 7's validation gate already guarantees that on its own, unconditionally) but “the algorithm's progress is bounded below by a real rate,” tied to how sharply the landscape curves near its optimum. The honest caveat, which the paper states directly, is that this stronger guarantee is conditional — it holds where the landscape happens to satisfy PL/KL, not universally, and the paper offers no proof that every real compound system's reward landscape does.

There's a second, more mundane way “convergence” is bounded in practice: everything Chapter 7's gate promises assumes the held-out validation signal is real signal, not noise. Table 2's own reported standard deviations (Amazon ranges from ±0.82 to ±3.78 accuracy points across methods, just from run-to-run variance) put a practical floor under how small a genuinely trustworthy validation improvement can be. Theorem 4.2's guarantee is about the population objective l(v); Algorithm 1 only ever sees a finite validation sample's estimate of it. When a true improvement is smaller than that sample's own noise floor, the gate can't reliably tell it apart from a false positive — a limitation of finite validation data, not of the theorem.

How Optimas compares to the alternatives

Two rows worth a sentence each, since Chapter 3 introduced both without a formal comparison. HBC's “ heterogeneous configs” is a real ✗, not an oversight in this table: it works by cloning toward outputs whose text embedding is close to a known-good reference, a mechanism that only makes sense for text-producing components — it has no natural way to touch Amazon's PPO-trained continuous weights or STaRK-Prime's numeric aggregation weights. REINFORCE is the mirror image: Chapter 8's efficiency numbers showed it needs more than 3× the data Optimas needs, because it collects one shared, noisy reward signal via costly downstream rollouts across the whole system, rather than a dedicated, cheap per-component signal — it supports continuous weights natively, but pays for that support with data efficiency, which is exactly the tradeoff Optimas's per-component LRF design was built to avoid.

OPRO earns every ✗ in its row for a simpler reason: it was never designed to touch a multi-component system in the first place. OPRO treats an LLM prompt-optimization problem as a single black-box function to search over — one target metric, one configuration, no notion of upstream or downstream components at all. It's a real, useful tool for exactly the job Chapter 2 opened with (“optimizing LLM single-step generation”), and Chapter 7's local prompt-optimization step literally reuses OPRO-style search inside Optimas, one component at a time. What OPRO was never built to answer is this session's actual question — what happens when four or five of these single-step problems are chained together and a change to one ripples into the others. Being absent from a compound-systems table isn't a failure for OPRO; it's simply outside the problem it was designed to solve.

Where this connects on the rest of the site

This session assumed you already had a retriever and a reranker to optimize — Retrieval-Augmented Generation covers how those two pieces get built in the first place. A compound system, viewed from the agent's side, is an agent harness with a formal optimization layer bolted on top; Agents & Tool Use and Session 07's agent harness cover the harness itself, whose tool and model calls are exactly this session's C_k's. This whole session assumed you already had a computable global reward R — AI Evaluation covers building one. Chapter 6's pairwise log-sigmoid loss is the same preference-loss family behind RLHF reward models, applied per-component instead of once at the end — see Reward Learning and Reward Alignment. Chapter 7's OPRO-style local optimizer for text-configured components is covered in full in Prompt Engineering.

Five numbers, if that's all you keep

Every number below was derived or reproduced by hand somewhere in this session — none of it is new. Reading them together, once, is the fastest way to check the whole argument still holds together end to end.

python
session_recap = {
    "brute_force_vs_optimas_runs": 2_500_000 / 2020,        # Ch2: ~1,238x fewer full-system executions
    "dspy_amazon_regression_pct": (18.18 - 21.21) / 21.21 * 100,  # Ch3: -14.3%, worse than doing nothing
    "toy_greedy_vs_aligned_f1_gap": 0.85 / 0.2625,       # Ch5: 3.24x, one flipped signal
    "alignment_accuracy_lrf_vs_judge": 77.96 / 49.52,   # Ch8: real, measured, both from Table 4
    "headline_avg_relative_improvement_pct": (14.3+1.8+22.1+12.4+9.0)/5,  # Ch8: 11.92%, the paper's own abstract claim
}
for k, v in session_recap.items():
    print(k, "≈", round(v, 2))
# brute_force_vs_optimas_runs ≈ 1237.62
# dspy_amazon_regression_pct ≈ -14.28
# toy_greedy_vs_aligned_f1_gap ≈ 3.24
# alignment_accuracy_lrf_vs_judge ≈ 1.57   (77.96% vs 49.52%, ~1.6x better than a coin flip)
# headline_avg_relative_improvement_pct ≈ 11.92

Read as a chain rather than five separate facts: brute force is too expensive to even attempt honestly (line 1); the cheap fallback everyone reaches for instead can quietly make things worse (line 2); a toy small enough to trust by hand shows exactly why (line 3); a trained, validated local reward measurably tracks the real objective where an untrained judge doesn't (line 4); and applied to five real, deployed-shaped systems, the whole approach delivers a real, honestly-reported, non-cherry-picked improvement (line 5). Every later number in this session traces back to one of these five.

“A system must be managed. It will not manage itself… Left to themselves in the Western world, components become selfish, competitive, independent profit centres, and thus destroy the system… The secret is cooperation between components toward the aim of the organization.” — W. Edwards Deming