Self-Play, Grounded

Chapter 1: SPICE: Two Roles, One Corpus

Start from the shape you already know from Absolute Zero: one policy, two roles, trained jointly with reinforcement learning. SPICE keeps that shape and changes exactly one thing about what each role is allowed to see.

The two roles, precisely

SPICE stands for Self-Play In Corpus Environments. A single model π_θ plays two roles by prompting alone — no separate weights, no separate checkpoints:

• Challenger (role = C): given a raw document d sampled from a corpus, generates a question q and extracts a gold answer a* directly from the document's own text.
• Reasoner (role = R): given only the question q — never the document — must answer it using nothing but what it already knows.

That last clause is the whole mechanism. The Challenger has read the document. The Reasoner has not. This is called information asymmetry, and it is the one design choice everything else in this chapter serves.

Why hiding the document is the fix, not a detail

Think about what asymmetry buys you. If the Reasoner could also read d, answering would reduce to reading comprehension — find the sentence, copy the number. Forcing the Reasoner to work from the question alone means it has to actually reason: connect the concepts the question describes to whatever the model already knows, the same way a student answers a closed-book exam question that was written from a source they never saw.

And because the gold answer a* is extracted from real document text rather than generated from nothing, it inherits the document's factual grounding. The Challenger cannot silently invent a wrong answer and call it correct — the document itself pins down what “correct” means. This is the direct fix for hallucination amplification from Chapter 0: the ground truth lives outside both roles' judgment, in text neither role wrote.

Two closed-book exams, not one

It helps to notice that the Reasoner role isn't the only place a closed-book constraint is doing work here. Compare the two roles' constraints side by side, because they're subtly different kinds of “can't see”:

Only one role is actually “closed-book” here — but that's exactly the asymmetry that makes the whole setup work as a test rather than a rubber stamp. If both roles were closed-book (Absolute Zero's situation, where neither role reads external documents), there would be no way to tell whether a proposed question is factually sound in the first place. If both roles were open-book, there would be nothing to test. Exactly one role needs the outside information, and exactly one role needs to be denied it — SPICE's whole architecture is that single asymmetric cut.

Where the documents come from

SPICE draws from a corpus D of 20,000 documents pulled from two freely available sources: Nemotron-CC-Math for mathematics, and NaturalReasoning (a subset of DCLM) for general reasoning. Each document segment runs up to 5,992 tokens, sized to fit inside the model's context window. Nothing about this corpus is small-talk or trivia — it's the kind of dense technical and scientific text a graduate-level reasoning benchmark would draw from, which is exactly the point: real difficulty comes bundled with real, verifiable content.

The split between the two sources is exactly even: 50% Nemotron-CC-Math, 50% NaturalReasoning, sampled uniformly across the whole corpus over the course of training. With T = 640 training iterations and 20,000 documents to draw from, that means each individual document gets reused roughly 2–3 times over the full run — not once-and-discarded, but nowhere near saturated to the point of memorization either. That reuse rate is worth keeping in mind for Chapter 9's discussion of whether a “near-inexhaustible” corpus is really inexhaustible: at 640 iterations it clearly isn't a bottleneck, but the number that would make it one is now concrete rather than abstract.

Two task formats, one universal verifier

Depending on what a given document actually contains, the Challenger picks between two formats: a multiple-choice question (four options, one document-grounded correct answer) or a free-form question with a typed answer — an integer, an expression, or a string, all directly extractable from the source text. Both formats share one crucial property: they can be verified by a rule-based checker (the paper uses Math-Verify for mathematical equivalence, plus exact matching for other types) without needing a specialized executor the way code tasks need a Python interpreter. That's what lets SPICE's self-play work across any subject a document corpus can cover, not just math and code.

Why free-form verification needs a real equivalence checker, not string matching

The binary correctness reward in the next section sounds simple — “does the Reasoner's answer match the gold answer” — but naive string equality would silently mark a huge fraction of genuinely correct answers wrong. A document might extract a gold answer as 1/2; a perfectly correct Reasoner might answer 0.5, or 50%, or x/(2x) in unreduced form. All four strings are mathematically identical and all four would fail a literal == comparison. This is exactly why the paper reaches for Math-Verify: a library that parses both sides into a symbolic form and checks mathematical equivalence, not textual equivalence, before deciding pass or fail. Get this step wrong and every downstream reward — the Reasoner's binary correctness signal, and through it the Challenger's variance reward from Chapter 2, since it's built out of exactly these binary outcomes — becomes noisy in a way no amount of RL tuning can fix, because the noise is in the ground truth itself, not in the policy.

Generating a valid task is itself an attempt-and-filter process

Given a document, the Challenger doesn't get exactly one shot. It takes up to N = 1,024 attempts to produce a valid (q, a*) pair — “valid” meaning correctly formatted and parsable from the Challenger's own generation. If multiple valid questions come out of the same document, one is selected at random, which keeps the training load balanced between the two roles rather than letting one document dominate. A malformed attempt doesn't just get discarded silently — it costs the Challenger a small penalty (ρ = −0.1) to discourage generating garbage in the first place.

That ceiling of 1,024 is deliberately generous, and it's worth understanding why. Most documents yield a valid question on the very first or second attempt — a well-formed passage with clear extractable facts is easy to turn into a question. But some documents are genuinely awkward: a table-of-contents fragment with no complete sentences, a passage that's mostly citations and cross-references, or a segment cut off mid-thought at the 5,992-token boundary. A generous attempt budget means training never has to hard-fail on an unlucky document — it just costs a few more (penalized) generation attempts before either a valid task is found or the document quietly contributes nothing to that iteration. Training keeps moving either way; a bad document is a minor inefficiency, never a crash.

What the Challenger's prompt actually enforces

It's worth reading the Challenger's actual instructions rather than just the reward that scores its output, because one requirement baked into the prompt itself is doing quiet, essential work. Both the MCQ and free-form question-generation prompts explicitly require that the generated question be self-contained and solvable without the source document.

Pause on why that constraint has to be there at all. The Challenger has the document in front of it while writing the question — nothing stops it, mechanically, from writing something like “according to the passage above, what value does the author give in paragraph two?” That's a perfectly well-formed question for the Challenger, whose context includes the passage. But the Reasoner never sees “the passage above,” per Chapter 1's whole information-asymmetry design — so a reference-dependent question like that would land in front of the Reasoner as a dangling pointer to nothing, unanswerable not because the reasoning is hard but because the question itself is malformed for the role that has to answer it. Requiring self-containment in the generation prompt is the mechanism that keeps every question a genuine, standalone reasoning problem rather than an accidental one-sided in-joke that only makes sense from the Challenger's vantage point.

Notice the layering here: this is a prompt-level constraint, enforced by instruction, not a reward-level one. Nothing in the Gaussian variance reward from Chapter 2 would directly punish a reference-dependent question — it only ever looks at the Reasoner's pass rate. A reference-dependent question could still get a well-calibrated 50% pass rate by accident (the Reasoner guessing on an unanswerable question is statistically indistinguishable from guessing on a genuinely hard one) and slip through the reward check clean. The self-containment requirement is a second, independent line of defense living in the prompt itself, catching a failure mode the reward alone can't see.

python
# the two roles, as prompts to the SAME model -- no separate weights
def challenger_step(model, document):
    q, a_star = model.generate(prompt=f"Given this document, write a challenging question "
                                    f"and extract its answer directly from the text:\n{document}")
    return q, a_star   # document is now DISCARDED before the next step

def reasoner_step(model, question):
    # NOTE: no `document` argument here -- this is the entire mechanism
    answer = model.generate(prompt=f"Answer step by step, box your final answer:\n{question}")
    return answer

Both roles are trained jointly, on shared weights

The objective SPICE maximizes sums both roles' expected rewards over the same policy:

Every training iteration therefore produces two kinds of trajectories from one model: Challenger trajectories scored by a reward that Chapter 2 derives in full, and Reasoner trajectories scored by a simple correctness check. Training runs for T = 640 iterations at batch size B = 128, with temperature 1.0 for both roles and a group size of G = 8 responses per question — a number that will matter a great deal in the next chapter. Implementation runs on the Oat actor-learner framework with vLLM for fast generation during both role's rollouts.

What one training iteration's batch actually looks like

The paper's own Algorithm 1 spells out one detail that's easy to gloss over when reading the objective as a single expectation: for each of the B = 128 documents sampled per iteration, the Challenger doesn't generate one attempt and stop — it generates up to N attempts (Chapter 1's 1,024 ceiling), and from those attempts, the algorithm subsamples exactly G trajectories while preserving the valid:invalid ratio that the raw attempts actually produced. If a document yielded 6 valid questions and 2 malformed attempts out of its first 8 tries, the subsampled group of G=8 keeps that same 6:2 mix rather than silently discarding every invalid attempt or over-representing whichever type happened to come first.

That design choice matters for a specific reason: it's what lets the −0.1 invalid-task penalty from Chapter 1 actually train the Challenger to stop generating malformed output, rather than becoming pure sampling noise. If the pipeline silently threw away every invalid attempt before computing a batch, the Challenger's policy gradient would never see the penalty attached to a real trajectory it actually produced — the malformed generations would vanish from the training signal instead of being punished by it. Preserving the true ratio keeps the penalty load-bearing.

What "actor-learner" actually means for this loop

It's easy to picture self-play as one model taking turns talking to itself in a single thread. In practice, SPICE's implementation splits the work the way most modern large-scale RL systems do: actors are worker processes that just generate rollouts — running the Challenger step, then the Reasoner step, over and over, as fast as vLLM can serve them — while a separate learner process collects those finished rollouts, computes rewards and advantages, and applies the actual gradient update. The two run concurrently rather than in lockstep: actors keep producing new (Challenger, Reasoner) rollout pairs while the learner is still busy updating on the previous batch.

This split matters for a reason specific to self-play: the same rollout batch that generates the Reasoner's K=8 sampled answers is also exactly what's needed to compute the Challenger's variance reward (Chapter 2 derives this dual use in full). Because both roles share one policy and one batch of rollouts, the actor-learner split doesn't need two separate data pipelines the way training two genuinely different models would — one stream of generation naturally produces both roles' training signal at once.

Chapter 2: The Variance Reward, Derived

The Reasoner's reward is the easy half: 1 if the answer matches the document-extracted gold answer, 0 otherwise. The Challenger's reward is the interesting half, and it's worth deriving in full, because it is the mechanism that turns a pile of documents into a curriculum that automatically tracks the Reasoner's improving skill.

The problem the Challenger's reward has to solve

A question that every Reasoner rollout gets right teaches nothing — it's already mastered. A question every rollout gets wrong also teaches nothing useful — there's no gradient signal distinguishing a lucky guess from real understanding, and it may simply be malformed or unanswerable. The reward needs to reward the Challenger for landing exactly at the boundary of what the current Reasoner can and cannot reliably do.

Notice that this reward doesn't act alone — it's paired with a generation prompt that already nudges the Challenger toward exactly this target before the reward ever gets computed. The paper describes the Challenger's prompt as walking the model through three explicit steps: multi-step complex information extraction from the document, deliberate difficulty enhancement of the resulting question, and self-testing the question before submitting it. The reward this chapter derives and the prompt structure work together, not separately — the prompt biases generation toward the kind of multi-step, extraction-and-reasoning question likely to land near the 50% frontier in the first place, and the reward then reinforces whichever specific attempts actually hit that target. Neither piece alone would reliably produce well-calibrated questions: a good prompt with no reward has no way to adapt as the Reasoner improves, and a good reward with no prompt structure would be searching blindly across a much larger space of possible (and mostly useless) question formulations.

From K rollouts to a variance

For each valid question, SPICE samples K = 8 responses from the Reasoner (the same group size G used for the Reasoner's own policy-gradient update) and scores each one as correct or not: l_i = 1[â_i = a*]. Treat those 8 binary outcomes as samples of a Bernoulli variable with empirical pass rate p = (number correct) / 8. The variance of that Bernoulli estimate has a closed form every statistics course derives early:

Don't take that formula on faith — it's a two-line derivation, and worth doing once so the rest of the chapter isn't standing on an unexplained fact. Variance, by its textbook definition, is Var(X) = E[X²] − E[X]². Each l_i is a binary variable that only ever takes the values 0 or 1, which means it has a convenient property no other kind of number has: l_i² = l_i exactly (0²=0, 1²=1, nothing else is possible). So E[l²] = E[l] = p, and substituting straight into the definition:

This one quantity is the entire signal. It is 0 when p is 0 or 1 (every rollout agrees — trivial or impossible) and maximized at p = 0.5, where Var = 0.25 (rollouts split evenly — maximum disagreement, maximum uncertainty about whether this question is hard or easy for the current Reasoner). You can confirm the maximum without calculus, too: p(1−p) = p − p² is a downward-opening parabola in p, and any parabola ap²+bp+c peaks at p = −b/(2a); here a=−1, b=1, giving p = 1/2 directly.

Turning variance into a reward: the Gaussian shape

SPICE doesn't reward variance directly — it rewards closeness to the ideal variance of 0.25, using a Gaussian bump centered there:

Read the pieces: the numerator penalizes distance from the 0.25 sweet spot, squared so both directions (too easy, too hard) are penalized symmetrically; the denominator (2·0.01 = 0.02) controls how sharply the reward falls off away from that peak — a narrower curve than 0.02 would demand near-perfect calibration, a wider one would tolerate sloppier difficulty targeting. At the peak (Var = 0.25 exactly) the exponent is zero and the reward is exactly 1.0, the maximum possible.

What the 0.02 width is actually buying you

It's worth sanity-checking that this isn't just an arbitrary magic number by seeing what happens if you turn the knob. A Gaussian's width parameter controls how forgiving the reward is about missing the exact 50% target. Recompute the k=7/8 (p=0.875, Var=0.1094) case from the table below at two hypothetical alternative widths, to feel the difference:

A narrower Gaussian turns a 7/8-solved question — still a meaningfully useful, not-quite-trivial question — into one the Challenger is barely rewarded for proposing at all (0.019, near the floor), which risks the Challenger avoiding anything short of a perfect 50/50 split. A wider Gaussian tolerates that same question generously (0.781), but stretched far enough, a wide-enough curve degrades toward the flat, uninformative reward Threshold already represents in the ablation below. The paper's 0.02 sits deliberately between those two failure modes — sharp enough to meaningfully separate difficulty levels, forgiving enough that a near-miss still earns a substantial reward.

The paper names that width parameter τ (a temperature, in the same sense a softmax temperature controls how sharply probability mass concentrates), writing the general form of the reward as r_C = exp(−(σ² − σ²_opt)² / 2τ), with σ² = Var(l) = p(1−p), σ²_opt = 0.25, and τ = 0.01 — which is exactly where the “2·0.01 = 0.02” denominator above actually comes from. Giving the constant a name matters here for a reason beyond bookkeeping: it's the same knob every one of the three alternative widths computed above is turning, and it's the knob you'd tune first if you were adapting this reward to a new domain with a different natural spread of difficulty.

The paper's own appendix states three properties this Gaussian shape buys over the alternatives, worth listing explicitly because Chapter 2's own three-rival ablation later in this chapter is a direct empirical test of all three claims: maximum reward occurs exactly at the 50% frontier; the reward decreases smoothly in both directions away from that peak (unlike Threshold's flat plateau); and unlike R-Zero-style's linear falloff, the Gaussian's curvature provides a stronger gradient signal specifically near the optimum, where getting the Challenger to fine-tune its difficulty targeting actually matters most.

Worked example: the full reward curve at G = 8

With exactly 8 Reasoner rollouts, only nine outcomes are possible: k correct out of 8, for k = 0 through 8. Compute the reward at every one of them — this is the exact arithmetic the training loop runs, not an approximation:

The table is exactly symmetric around k = 4, as it has to be — Var = p(1−p) is symmetric around p = 0.5 by construction, and everything downstream of it inherits that symmetry. A question the Reasoner nails 7 times out of 8 scores 0.372, roughly a third of the maximum — still meaningfully rewarded, unlike a question it aces every single time (reward 0.044, barely above the floor).

Why G = 8, and not something finer-grained?

With only nine achievable outcomes (k = 0 through 8), the reward function's true continuous shape gets sampled at just nine points along the p-axis, spaced 0.125 apart. A larger group size would sample it more finely — with G = 32, the achievable pass rates step by 1/32 ≈ 0.031 instead of 1/8 = 0.125, letting the Challenger calibrate difficulty four times more precisely, and letting the Gaussian reward actually distinguish, say, a 47% pass rate from a 50% one, which G=8 simply cannot represent as two different outcomes at all (both round to the same nearby k/8).

The cost is the same one Chapter 6 derives in full for Self-Play SWE-RL's solver group: every one of the G Reasoner rollouts used to estimate p is a full generation from the model, consuming real compute and real wall-clock time before the Challenger's single reward number is even available. SPICE's rollouts are short text-generation (a boxed answer, not an agentic trajectory), which is why G=8 was affordable enough to also double as the Reasoner's own policy-gradient group size (Chapter 3) — one design choice serving two purposes at once, at a compute cost SPICE's authors judged worth the coarser 12.5%-step discretization it implies. Chapter 6 will show the same tradeoff showing up again in Self-Play SWE-RL, at a cost per rollout high enough that the choice of G has real, unavoidable consequences for what the reward function can even represent.

Three rivals SPICE tested and beat

The paper doesn't just assert the variance reward is best — it runs a controlled ablation against three alternative Challenger reward strategies, all on the same base model, same corpus, same training budget:

• Absolute-Zero-style: reward = 1 − p (the Session 13 approach — reward the Challenger more, the more often the Reasoner fails). Simple, but it conflates “hard” with “valuable to learn from” — an impossible question (p=0) scores a perfect 1.0, same as it would under a policy that only ever proposes unsolvable garbage.
• Threshold: reward = 1 for any non-degenerate outcome (1 ≤ k ≤ 7), 0 for the two degenerate ones (k=0 or k=8). A binary pass/fail on “is this trivial or impossible,” with zero gradient distinguishing a barely-solvable question from a coin-flip one.
• R-Zero-style: reward = 1 − 2|p − 0.5|. Peaks at p=0.5 exactly like SPICE's variance reward, but falls off linearly rather than following the Gaussian's faster, smoother taper.

Putting all four formulas side by side, at the same pass rates

Numbers make the shape differences concrete faster than descriptions do. Compute all four reward strategies at the same four values of k:

Look at the k=1 and k=7 rows: the Absolute-Zero-style reward is asymmetric around 50% (0.875 at k=1 versus 0.125 at k=7) because it simply rewards low pass rate, treating “the Reasoner almost never solves it” as better than “the Reasoner almost always solves it,” regardless of the fact that both are equally far from the informative 50% frontier. Variance and R-Zero-style are both properly symmetric (0.372/0.372 and 0.250/0.250) — they correctly treat “too easy” and “too hard” as equally uninformative. Threshold, true to its name, can't tell any of these four rows apart at all.

The ranking tracks exactly how much curvature each reward function has around its peak. Threshold has none (a flat plateau of reward 1 across seven different difficulty levels) and scores worst among the three “shaped” rewards. The Gaussian variance reward has the sharpest, most informative peak and wins.

It's worth noting how mild SPICE's own invalid-task penalty (ρ = −0.1) is, relative to the reward's own [0, 1] range — a malformed question costs the Challenger roughly a tenth of what a perfectly-calibrated one earns, a gentle nudge rather than a harsh deterrent. Chapter 6 derives a structurally similar penalty for Self-Play SWE-RL's injector, and it lands in a very different place: −1.0 for a bug artifact that fails consistency validation, a full unit below zero rather than a tenth. That's not an inconsistency between the two papers — it reflects a real difference in what “invalid” costs in each domain. A malformed SPICE question is cheap to detect and cheap to discard (a quick format check); a bug artifact that fails one of Chapter 5's seven consistency checks has already consumed a full agentic rollout — tool calls, bash commands, real wall-clock time in a sandboxed repository — before the pipeline learns it was invalid. The size of each paper's invalid-penalty constant tracks the actual cost of generating something worthless in that domain, not an arbitrary choice made independently by each set of authors.

Chapter 3: Watching the Curriculum Co-Evolve

A reward that targets the frontier of the Reasoner's ability only produces a real curriculum if that frontier keeps moving as training progresses. SPICE's authors check this directly, and the result is the clearest evidence in the paper that the adversarial dynamic is actually doing something, not just decorating the loss curve.

The experiment: freeze one role, advance the other

Take checkpoints from a full SPICE training run at steps 200 through 640. Run two controlled comparisons on a held-out pool of 128 documents (128 generation attempts each):

• Fixed Reasoner, advancing Challenger: hold the Reasoner frozen at its step-200 checkpoint, and evaluate its pass rate against Challenger checkpoints from later and later in training.
• Fixed Challenger, advancing Reasoner: hold the Challenger frozen at its step-200 checkpoint, and evaluate later Reasoner checkpoints' pass rate against its (now-static) questions.

Both numbers start at the same 55% because at step 200 the two roles have barely diverged from each other yet. From there they move in opposite directions for the obvious reason: a Challenger that keeps improving generates questions the old, frozen Reasoner increasingly can't handle; a Reasoner that keeps improving increasingly handles questions the old, frozen Challenger already knows how to pose. Run both roles live, simultaneously, and this is the tug-of-war training is actually riding.

Why freeze one role at all — why not just measure the live, jointly-evolving pass rate?

This methodological choice is worth pausing on, because it's the kind of control that distinguishes a real measurement from a plausible-sounding claim. If you only ever watched the pass rate of two roles that are both improving simultaneously, you couldn't tell which role's improvement was driving the number — a rising pass rate could mean the Reasoner got dramatically better, or it could mean the Challenger quietly got worse at proposing hard questions, or some blend of both, with no way to separate the two explanations from a single curve. Freezing one role at a fixed step-200 checkpoint removes that ambiguity by construction: any change in the resulting pass rate can only be attributed to the role that's still moving, because the other one, by design, is not. This is the same logic as a controlled experiment in any other science — hold one variable fixed to isolate the effect of the other.

This chapter's ablation and Chapter 1's ablation are the same experiment, seen twice

It's worth connecting a dot across chapters explicitly, because the paper itself presents the same two underlying results in two different forms, and seeing both forms makes each one clearer. Chapter 1 introduced the “Strong Challenger” baseline — a fixed Qwen3-32B-Instruct generating questions while only the target model trains as Reasoner — as a final overall-score comparison (+7.2 vs. SPICE's +9.1 on Qwen3-4B-Base). The paper's own training-dynamics figure reports that exact same comparison a second way, as a curve over training rather than a single endpoint number: with the Challenger frozen (never trained), the Reasoner “isn't challenged enough and improves more slowly” across the whole run, not just at the final checkpoint. It's the identical underlying finding — a fixed Challenger produces a weaker Reasoner — viewed once as a bar chart and once as a training curve.

The same doubling happens for corpus grounding. Chapter 0's opening widget plotted grounded (43.9% final score) against ungrounded (40.7%) self-play as two curves over 640 iterations of training. That's not a separate experiment from this chapter's co-evolution analysis — it's the same underlying “corpus grounding ablation” the paper reports, just visualized as final training-curve endpoints rather than the frozen-vs-advancing pass-rate comparison this chapter has been deriving. Two different lenses (an ablation on what's grounded in Chapter 0, an ablation on which role advances here) end up measuring overlapping consequences of the same underlying mechanism: an adversarial dynamic that only produces mutual improvement when both grounding and co-training are actually present.

A concrete example of the questions getting harder

The paper shows this directly on one document about the scale of the solar system — the diameter of the Sun, the diameter of the Moon, the Earth-Sun distance. At step 50, the Challenger's question on this document is: “What is the diameter of the Moon?” — a direct lookup, answerable by finding one sentence. At step 480, the question on the same document becomes: given an alien moon with the real Moon's diameter, orbiting at a different distance, what star-distance would produce a perfect eclipse under the same angular-size geometry — a multi-step proportional-reasoning problem that never appears verbatim in the source text at all, even though every number in it traces back to the document.

The Reasoner's answers evolve to match. Early training: a guess — “stars are way farther than moons, so maybe 1000× farther” — landing on the wrong multiple-choice option. Late training: a full seven-step derivation setting up the ratio Moon-diameter/Moon-distance = Star-diameter/Star-distance, solving algebraically, and verifying the answer by plugging both ratios back in and confirming they match to within rounding. Same document, same underlying facts — the model's demand on itself, and its ability to meet that demand, both climbed together.

The Reasoner's actual output, quoted directly from the paper

It's worth reading the two transcripts in full rather than a paraphrase — the qualitative shift in reasoning style is exactly as telling as whether the final answer is right.

Notice step 7 in the late-training transcript especially: the model doesn't just compute an answer and stop, it plugs both ratios back in and checks they agree — a self-verification habit that appears nowhere in the early-training transcript's one-line guess. Nothing in SPICE's training signal explicitly rewards “show a verification step” — the Reasoner's reward is a flat binary correct/incorrect. This kind of behavior emerges as a side effect of optimizing that binary signal against increasingly demanding questions, not because it was directly targeted.

The update rule that keeps both signals honest: DrGRPO

Before deriving what DrGRPO changes, recall the baseline it modifies. Ordinary GRPO (Session 10) replaces a learned value-function baseline with a much cheaper one: sample a group of G responses to the same prompt, and use that group's own mean reward as the baseline every response in the group is compared against — no separate critic network needed. The per-response advantage is typically (reward − group mean) / (group standard deviation), normalized so the update's scale doesn't depend on how spread-out that particular batch of rewards happened to be.

Challenger rewards and Reasoner rewards live on genuinely different scales — the Gaussian variance reward ranges continuously from about 0.04 to 1.0, while the Reasoner's binary correctness reward is exactly 0 or 1. Feeding both into a naive shared advantage estimate would let whichever reward has more spread dominate the gradient. SPICE uses DrGRPO, computing a separate advantage for each role by centering each role's rewards around its own batch mean — no shared normalization, no cross-role interference:

Notice what's absent compared to vanilla GRPO: no division by a standard deviation. Standard GRPO normalizes advantages by within-group reward spread, which sounds harmless until you remember the Challenger's reward is designed to have low variance near its 0.25 sweet spot precisely when it's doing its job well — dividing by a small number there would blow up the gradient exactly when the signal is most meaningful, not least. Mean-centering without std-normalization avoids that trap for both roles.

The other thing standard GRPO usually has that SPICE removes: the KL penalty

There's a second difference worth naming, separate from the standard-deviation question above. Most RLHF-style policy-gradient training (including plenty of ordinary RLVR setups) adds a KL-divergence penalty term that pulls the trained policy back toward a fixed reference checkpoint — a leash that keeps the model from drifting too far from where it started, usually to protect fluency or prevent reward hacking against a narrow proxy signal. SPICE trains with that penalty's weight set to β = 0: no KL regularization at all, “to focus purely on reward optimization,” in the paper's own words.

It's worth asking why that's a safe choice here specifically, rather than a corner cut. A KL leash matters most when the reward signal is a narrow, potentially gameable proxy for something broader (human preference, say) that the reference policy already approximates reasonably well. SPICE's two reward signals aren't like that: the Reasoner's reward is a hard binary correctness check against a document-extracted gold answer (Chapter 1's fix for hallucination amplification already closes off the obvious way to game that), and the Challenger's reward is a symmetric curve with a well-defined, non-gameable peak (Chapter 2). Removing the KL leash lets both roles move as far from their pretrained starting point as the reward signal actually justifies, without a hyperparameter fighting against a training signal that's already been engineered not to need protecting from.

What actually runs this update: the real infrastructure numbers

It's easy to read “T = 640 iterations, batch 128” as an abstract training schedule without a feel for what running it actually takes. SPICE's experiments run on 8 H200 GPUs per experiment — notably modest compared to the 512-GPU runs Chapter 7 covers for Self-Play SWE-RL — using a constant learning rate of 1×10⁻⁶ (no warmup schedule, no decay), gradient checkpointing and flash attention to keep memory usage tractable, and ZeRO Stage 2 optimizer-state sharding across the 8 devices. 128 trajectories flow through each gradient update, matching the batch size B exactly — one document sampled, one valid Challenger trajectory selected, and G=8 Reasoner rollouts on it, repeated 128 times per iteration, 640 iterations total.

That 8-GPU footprint is worth keeping in your head as a reference point for Chapter 7, where the same underlying question — “how much does it cost to ground self-play in something real?” — gets answered with numbers sixty-fold larger. Text-generation rollouts (a boxed answer, no tool calls, no live sandbox) are simply far cheaper than agentic rollouts against a real, breakable codebase; the compute gap between the two papers is visible in their hardware footprints before you've read a single benchmark number.

Worked example: computing role-specific advantages by hand

Take four Challenger rollouts from Chapter 2's own reward table — say the four questions land at k = 4/8, 7/8, 8/8, and 1/8 correct, giving rewards 1.000, 0.372, 0.044, and 0.372 respectively (reading straight off the table you already derived):

The perfectly-calibrated 4/8 question gets pushed up hardest (advantage +0.553); the two 7/8-and-1/8 questions (equally far from the ideal, since Var is symmetric) get an identical small penalty; the useless 8/8 question gets pushed down the most. The policy gradient built from these four advantages nudges the Challenger's generation probability toward whatever produced that first question and away from whatever produced the third — exactly the curriculum-tracking behavior this chapter set out to explain, now visible as four concrete numbers rather than a description.

One property of mean-centering is worth checking explicitly, because it's the reason a policy gradient built this way never has a systematic bias in one direction: the four advantages sum to exactly zero.

That's not a coincidence specific to these four numbers — it's a direct algebraic consequence of subtracting the group mean from every element of the group: Σ_i(r_i − mean) = Σ_ir_i − G·mean = Σ_ir_i − Σ_ir_i = 0, for any set of rewards at all. What that guarantees for training is that within any single group, some trajectories always get pushed up and others always get pushed down — there's no group in which every response in it gets the same-signed nudge, because a group where every raw reward equals the mean is the only case where every advantage is zero simultaneously, and any deviation from that necessarily splits the group into above-mean and below-mean halves. The policy gradient is always a genuine contrast between better- and worse-than-typical outcomes within the group, never a uniform push in one direction.

That last property is worth stress-testing with a deliberately degenerate case, because it exposes exactly what this reward shape is built to guard against. Suppose all four Challenger rollouts in a group happened to land at the ideal k=4/8 (reward 1.000 each) — a batch of questions that are all equally, perfectly well-calibrated. The group mean is 1.000, and every advantage is 1.000 − 1.000 = 0.000. The policy gradient for that entire batch is exactly zero: nothing to push toward, nothing to push away from, because there's no variation within the group for DrGRPO's contrast to latch onto. That's the correct behavior, not a bug — a batch where every rollout is equally good genuinely carries no information about which specific generation choices were better or worse than which others. It's also the appendix's own stated goal for this design in different words: centering rewards around the role-specific mean, without further normalization, is meant to ensure “gradient updates reflect genuine learning signal rather than question difficulty artifacts” — and a batch with no spread in it has, by that same logic, no learning signal to extract.

Chapter 4: Results Across Four Models

A method that only works on one lucky base model is a curiosity, not a technique. SPICE's authors test on four — two different model families, two sizes each — to check whether corpus grounding is a general property of the approach or an accident of one checkpoint.

The four methods' exact configurations, side by side

Before trusting any comparison table, it's worth checking what actually differs between the rows — because a comparison where every row secretly uses different sampling counts or a different optimizer isn't really isolating the one variable it claims to. The paper's own appendix lays out every baseline's exact configuration, and it's worth reading in full once, because several of the differences turn out to matter for how to read the results correctly.

Two footnotes on that table are worth reading as carefully as the numbers themselves. First (*): Absolute Zero's invalid-response penalty isn't one flat number — it's −0.5 for a response that's incorrect but well-formatted, and a harsher −1 specifically for a formatting error, a finer-grained distinction than SPICE's single −0.1 catch-all. Second (†): R-Zero is trained for only 5 iterations in this table, not 640 — because, in the authors' own words, “R-Zero shows performance degradation after 5 iterations, so we report its best performance.” That's the same collapse pattern Chapter 0 already introduced (R-Zero's own paper reports pseudo-label accuracy dropping from 79% to 63% after 3–4 iterations) showing up a second time, independently, inside SPICE's own reproduction of the baseline — two different research teams, two different training runs, the same qualitative failure.

That footnote also means the R-Zero column throughout this chapter's tables is, in a real sense, giving R-Zero the benefit of the doubt: rather than reporting what R-Zero looks like after the full 640-iteration budget every other method gets (where its collapse would presumably make the gap even larger), SPICE's authors report R-Zero's single best checkpoint. The R-Zero numbers you're about to read are not a worst-case comparison — they're closer to R-Zero's best-case one.

The headline table

Every row below is trained with the identical SPICE recipe from Chapters 1–3: same corpus, same variance reward, same DrGRPO update. Only the starting base model changes.

SPICE beats every baseline — the frozen Strong Challenger, the ungrounded R-Zero, and the Absolute-Zero-style self-play — on all four base models, not just on average. The gains span two full model families built on genuinely different pretraining pipelines (Qwen3's dense pretraining vs. OctoThinker-Hybrid's architecture), which is the kind of cross-family consistency that rules out “this only works because of some quirk in one tokenizer or one pretraining recipe.”

Notice too that the gains span both math (average +8.9%) and general reasoning — MMLU-Pro, GPQA-Diamond, SuperGPQA, BBEH (average +9.8%). The Challenger's corpus mixes math-heavy Nemotron-CC-Math documents with general-knowledge NaturalReasoning documents, and the Reasoner improvement transfers to both categories of downstream benchmark, not just the domain the training documents happened to resemble.

Reading the per-benchmark table like a researcher, not a headline-skimmer

The overall +9.1 for Qwen3-4B-Base is an average across twelve individual benchmarks, and averages hide which specific capabilities actually moved. Naming all eleven that feed into it, before zooming into three, is worth doing once so “overall” stops being an opaque single number:

Seven math benchmarks, four general-reasoning ones, averaged together (with the two domain averages the paper separately reports — math +8.9%, general +9.8% — already telling you the gain isn't concentrated in just one domain) into the single “Overall” column this chapter's headline table reports. Three of those eleven numbers are worth pulling out individually:

The pattern across all three: SPICE's gains are largest exactly where the benchmark is hardest to game — competition-level math and pattern-resistant graduate questions — and smallest (in absolute-headroom terms) where the benchmark was already nearly solved. That's the signature you'd want to see if the mechanism really is “teaching genuine reasoning ability” rather than “teaching test-taking tricks that only work on easy benchmarks.”

The same three benchmarks, run again on a stronger base model

Qwen3-8B-Base gained the least of the four models in this chapter's headline table (+5.7, versus +9.1 for the 4B version of the same family). Reading the same three benchmarks for Qwen3-8B that Chapter 4 just broke out for Qwen3-4B makes that smaller gain concrete instead of just a smaller number on a bar chart:

Every one of these three deltas is smaller for the 8B model than the identical benchmark's delta on the 4B model — and GSM8K's gap is the most dramatic illustration why: Qwen3-8B-Base already scores 91.2 on grade-school word problems before any SPICE training at all, leaving only 8.8 points of headroom below a perfect 100 to begin with. A method cannot produce a 20-point gain on a benchmark that only has 8.8 points of room left to climb. This is the same “regression toward an already-strong baseline has less room to show a gain” effect Chapter 8's cross-paper scaling discussion returns to directly — and it's worth seeing it here, inside one single paper's own internal comparison, before trusting any claim about how gains scale with model size across different papers entirely.

How these numbers are actually scored, and by what

It's worth knowing precisely what “pass rate” means on each benchmark, because the protocol isn't uniform across the table. Evaluation is entirely zero-shot — no benchmark-specific fine-tuning or few-shot examples, testing whether the reasoning ability SPICE trained transfers without any task-specific adaptation. Most benchmarks use greedy decoding (temperature 0, fully deterministic, for reproducibility). AIME'24 and AIME'25 are the one deliberate exception: because these are small, high-variance competition-math sets, the paper instead averages accuracy over 32 sampling runs at temperature 0.6, a more statistically robust protocol on exactly the two benchmarks where a single greedy rollout would be noisiest.

There's a second, easy-to-miss detail about verification specifically: the equivalence checker used to score the final reported benchmark numbers is not the same one Chapter 1 named for training. Training uses Math-Verify, a fast, free, rule-based library, run millions of times over the course of 640 iterations — it has to be cheap, since it fires on every single Reasoner rollout during training. The benchmark numbers in this chapter's tables, evaluated exactly once per model at the end, instead use GPT-4o-based verification through the simple-evals framework, a more expensive but more permissive judge, run only as many times as there are benchmark questions. Using a cheap verifier during training and a more thorough one for final evaluation is a deliberate cost/precision tradeoff, not an inconsistency — the two verifiers serve different jobs at different scales.

And for the general-reasoning benchmarks specifically, it's worth knowing what each one is actually designed to resist: GPQA-Diamond is deliberately built from graduate-level science questions hard enough that domain experts, not just language models, get many wrong without specialized knowledge. SuperGPQA spans 285 disciplines with questions built to resist a plain web search. MMLU-Pro tightens ordinary MMLU's multiple-choice format to demand deeper understanding, not just answer-elimination. BBEH extends BIG-Bench Hard with additional complex-reasoning tasks. All four are constructed specifically to punish shallow pattern-matching, which is exactly why Chapter 4's earlier observation — that SPICE's gains land hardest on the least-gameable benchmarks — is a real signal about the trained reasoning ability, not an artifact of easy benchmarks being easy to move.

One more robustness check worth naming explicitly: on Qwen3-4B-Base, every single one of the eleven individual benchmarks in Table 1 improves under SPICE relative to the untrained base — not just the overall average. A method that boosts the average by trading off badly on a couple of benchmarks (common enough in ML that it has its own name, “regression on the tails”) would show up as a strong overall number hiding a real weakness underneath. That's not what happens here; the gain is broad-based across every benchmark measured, which is a meaningfully stronger claim than the single overall-average number by itself would support.

Where the corpus itself matters: an ablation on composition

Both source datasets contribute, but not identically. Splitting the corpus apart and training separately on each half:

Exactly as you'd expect if the mechanism is really “grounding transfers the corpus's own strengths”: the math-heavy corpus alone produces the best math score but the worst general-reasoning score, and vice versa for the general-reasoning corpus. Combining them doesn't simply average the two — it produces the best overall score, better than either specialist alone, because the Challenger can now draw the right kind of document for whichever gap the Reasoner currently has.

The paper's appendix breaks this down to individual benchmarks, and two numbers make the transfer pattern vivid rather than just directional: training the Challenger only on Nemotron-CC-Math pushes AIME'25 up by 10.2 points relative to training on NaturalReasoning alone — a domain-specific corpus buying a domain-specific skill, concretely. Running the comparison the other direction, training only on NaturalReasoning instead pushes GPQA-Diamond up by 11.9 points relative to the math-only corpus. Each corpus's specialist strength shows up as a double-digit swing on exactly the benchmark you'd predict from what the corpus actually contains — not a vague overall trend, but a specific, benchmark-by-benchmark fingerprint of what each source document actually teaches.

Task-format ablation

A second ablation checks whether both question formats (MCQ and free-form) earn their place:

Free-form questions push math scores highest on their own (open-ended answers demand real derivation, not just elimination among four options) but MCQ's reliable, unambiguous verification stabilizes general-reasoning training. Mixing both, again, beats either extreme.

The gap that widens with model size: SPICE vs. a frozen stronger Challenger

Chapter 1 introduced the “Strong Challenger” baseline — a fixed Qwen3-32B-Instruct generating questions while only the target model trains as Reasoner — and noted it underperforms full SPICE. Look at that gap across all four base models, not just the one number from Chapter 1:

A frozen 32B question-writer is a strong, fixed opponent — strong enough to help every base model improve on its own. But SPICE's co-evolving Challenger extracts additional gains beyond that fixed opponent in every single case, and the extra edge doesn't shrink as models get more capable; if anything it's larger for the OctoThinker family. A fixed 32B model, however strong at the start, eventually stops being a challenge once the trained Reasoner catches up to it — a co-evolving Challenger, by construction, never lets that happen.

Chapter 5: From Documents to Codebases

SPICE grounds its self-play in text documents. The second paper this session covers, Self-Play SWE-RL (SSR), asks a pointed follow-up question, and states it almost as a challenge in its own introduction: what can a model learn by only interacting with a Python interpreter, the way Absolute Zero does? It can master every intricacy of Python syntax and semantics. It cannot learn the much larger body of knowledge embedded in real, human-built software repositories — conventions, dependency structure, the accumulated shape of how actual systems get built — because none of that is inferable from language semantics alone. SSR's answer is to ground self-play directly in that missing substrate: real, sandboxed codebases.

The two roles, this time over code

The same self-play skeleton returns, with new names for the two roles: a bug-injection agent and a bug-solving agent, sharing one policy, both equipped with the same tool-using scaffold (Bash and a search-replace code editor) borrowed from Code World Model — the same CWM covered back in Session 12, which also happens to be the base model SSR trains on top of.

The minimal-assumptions design

This is SSR's sharpest departure from ordinary agentic RL for coding. Standard approaches (including the “baseline RL” Chapter 7 compares against) assume access to human-written issue descriptions, pre-written test suites, and known test-running commands — exactly the curated infrastructure SWE-bench was built from. SSR assumes none of that. It requires only a Docker image containing source code with dependencies already installed. The bug-injection agent has to discover how to run the test suite, build its own test parser, and understand the project's structure entirely through trial and error — the same way a new engineer explores an unfamiliar repository for the first time, minus the Stack Overflow tabs.

It's worth reading the paper's own list of what it explicitly refuses to assume, because each item is doing real work in keeping the method general-purpose rather than tuned to one benchmark's conventions. SSR does not assume access to: test parsers, existing tests, commands to run the test suite, or any prior knowledge about the programming language or test framework in use. Every one of those four is something a typical agentic-coding RL setup would ordinarily be handed as a given — here, discovering all four is folded into what the bug-injection agent has to learn to do as part of injecting a bug at all. The upside of refusing these assumptions is exactly the payoff the paper's own framing emphasizes: a method built this way applies to “diverse software projects” without per-project setup, rather than only to the specific benchmark suite it was validated against.

What a “bug” formally is: five files

A bug in SSR isn't a vague description — it's a precise, machine-checkable artifact with five parts:

That reset step is worth sitting with: even if the solver agent edits or deletes the test file itself instead of fixing the underlying bug, the true, original test file is restored before anything is scored. Reward-hacking the tests directly is closed off by construction, not by hoping the agent doesn't think of it.

There's a reason this five-file shape looks deliberately familiar if you've seen a SWE-bench problem before: once a bug artifact clears validation, the paper reformats it “similarly to SWE-bench instances, including pass-to-pass and fail-to-pass test specifications” — the same taxonomy SWE-bench itself uses to describe which tests must keep passing (pass-to-pass) and which must newly start passing once the bug is fixed (fail-to-pass). That's not a coincidence of vocabulary; it's a deliberate compatibility choice. Every synthetic bug SSR generates lands in the solver's hands shaped like the real human-curated bugs it's ultimately evaluated against in Chapter 7, which is a large part of why the training signal transfers to real SWE-bench-style evaluation at all — the format of the training task and the format of the test task are the same, even though the content (a synthetic, self-invented bug vs. a real historical GitHub issue) is completely different.

How bugs actually get injected

Two strategies, both requiring the agent to keep the project runnable afterward (a true semantic bug, not a syntax error that would trivially crash on import):

• Removal: delete a code hunk or a whole file, forcing the solver to reconstruct missing functionality from scratch — a genuinely hard repair.
• Historical reversion: mine the repo's own git log and revert a past change, producing a bug that resembles something a real developer might have actually shipped by accident.

The paper's own prompt-template appendix names a third, deliberately simpler alternative alongside these two: a direct-injection prompt that just asks the agent to introduce a bug with no scaffolding toward either removal or historical reversion — the naive version of “go write a bug.” It exists in the paper specifically as a baseline to test the other two strategies against, and Chapter 7 shows exactly how that comparison turns out: these three strategies are not equally effective, and the choice of injection method turns out to matter as much as whether self-play happens at all.

What the bug-injection agent is actually told to do, step by step

It's worth reading the shape of the real prompt rather than just the abstract description, because it reveals how much of the seven-check pipeline from earlier in this chapter is actually the agent's own responsibility to satisfy, not something checked for it after the fact. The removal-oriented injection prompt (the paper's own appendix) walks the agent through a concrete sequence: first, understand the codebase's functionality and test framework; then identify interesting test files, explicitly required to cover at least {min_passing_tests} tests spanning at least {min_changed_files} code files — the same two named thresholds from earlier in this chapter, now visible as literal parameters filled into the agent's own instructions rather than an after-the-fact filter; then build and save a runnable test command, with a hard 90-second execution budget so a single slow test suite can't stall the whole pipeline; then run it, capture the output to a log file, and read it back to confirm the selected tests actually pass. If some tests turn out to be flaky or environment-dependent rather than genuinely failing, the agent is explicitly permitted to drop them and pick a different set — as long as the minimum-test-count requirement still holds. Only after all of that does the prompt ask the agent to write the generic Python parser (test_parser.py from earlier) that turns raw test output into the pass/fail JSON mapping every later scoring step depends on.

Notice what this reveals about where the real difficulty sits: writing the actual bug — the part that sounds like the headline task — is only one step in a much longer sequence where most of the work is disciplined software-engineering process (understanding a codebase, building a reliable test harness, handling flaky tests, writing a robust parser) that has to happen correctly before the bug-writing step can even be validated. An agent that's bad at that surrounding process will fail the seven consistency checks regardless of how creative its actual bug is.

Building the actual buggy codebase the solver works in

Once a bug artifact clears every check, it still has to become a real, runnable, broken repository for the solver to open and start fixing. The pipeline is a small, precise sequence of patch applications:

bash
# 1. start from the ORIGINAL, correct codebase
git checkout original-repo/

# 2. introduce the bug
git apply bug_inject.diff

# 3. hide it from the existing test suite
git apply test_weaken.diff

# 4. (higher-order bugs only) layer in a PRIOR failed solver attempt
git apply pred_patch.diff       # from an earlier, unsuccessful repair

# 5. remove git history entirely -- prevents the solver from reading
#    commit logs / diffs to "cheat" its way to the fix instead of
#    actually understanding and repairing the break
rm -rf .git && git init && git add -A && git commit -m "buggy state"

That fifth step is easy to skip mentally but is doing real work: the bug-injection agent legitimately used git log to find historical-reversion bugs in the first place (per the strategy above), but the solver must never get to see that same history, or repairing the bug would degrade into “find the commit that did this and revert it” — a lookup, not a repair. Stripping .git before handing the environment to the solver closes that information-leakage path the same way Chapter 1's document-hiding closed SPICE's analogous leakage path.

How a prediction patch actually gets graded

Scoring a solver's attempt runs a mirror-image sequence, carefully designed so nothing about the grading process can be gamed by editing the wrong file:

The order matters: the solver's patch is applied before the test-file restoration step, which is precisely what makes editing the test files pointless — whatever the solver's patch did to test_files.txt gets overwritten by the true originals moments later, every single time, regardless of what the patch tried to do.

Seven checks stand between “an agent claimed this is a bug” and “this counts”

Before a bug artifact reaches the solver, it must pass every one of these:

1. Test files existence & coverage — every referenced test file must actually exist and must cover everything the weakening patch touches.
2. Test parser validity — the agent's own parser must reliably map raw output to pass/fail JSON.
3. Test script validity — running the script on the original (unbroken) code must produce more passing tests than a configured minimum.
4. Bug scope — the injection patch must touch at least a minimum number of files.
5. Bug validity — at least some minimum number of tests that passed before must now fail.
6. Test weakening validity — some tests that fail in the buggy state must pass again once the weakening patch hides them.
7. Inverse mutation testing — for every file touched by the bug patch, individually reverting just that file (while keeping everything else broken) must make at least one failing test pass again. Any file that fails this test is dead weight in the patch and gets rejected.

That last check is the cleverest of the seven. Ordinary mutation testing asks “can my test suite detect a random code mutation?” — SSR inverts it, using the same machinery to ask “is every single file in this bug patch actually load-bearing for the bug?” A bug patch that includes an unrelated, irrelevant file edit — noise that would confuse the solver without teaching anything — fails this check and gets rejected before it ever reaches the solving agent.

Several of those seven checks lean on named, tunable thresholds rather than a hardcoded absolute rule — min_passing_tests (how many tests must already pass on the original, unbroken codebase for the repository to even be a candidate), min_changed_files (the bug scope floor from check 4), and min_failing_tests (the bug-severity floor from check 5). Treating these as configurable parameters rather than fixed constants is what lets the same seven-check pipeline validate bugs across wildly different repositories — a tiny utility library and a sprawling enterprise codebase don't have the same natural scale of “a meaningfully large bug,” and a single hardcoded file-count threshold tuned for one would be either too strict or too permissive for the other.

Higher-order bugs: recycling failure into new training data

When the solver's repair attempt fails, that failed attempt doesn't just get discarded — it gets applied to the buggy codebase as an additional layer, creating a new, harder higher-order bug for another attempt. The paper caps this at second order (it becomes likelier to overlap with existing bugs beyond that), but even one extra layer mimics something real first-order synthetic bugs structurally can't: the layered, interdependent, multi-step edits that actual software development produces, where one imperfect fix creates the next bug down the line.

Chapter 6: Opposing Incentives, Derived

SPICE's Challenger and Reasoner are adversarial but not directly opposed — the Challenger's Gaussian reward peaks when the Reasoner is at 50%, which is also a perfectly fine outcome for the Reasoner to be in on any single question. SSR's two roles are opposed in a sharper, more literal sense: the injection reward and the solve reward point in genuinely opposite directions as a function of the same number. This chapter derives that tension from the formulas themselves.

The solver's reward: binary, simple

Every one of the solver's G = 8 attempts on a bug gets scored this way independently. Define the solve rate s as the fraction of those 8 attempts that fully succeed. If the true probability of success on a given bug is s, the solver's expected reward per attempt is:

Simple and intuitive: the solver wants s as high as possible. At s=1 (always succeeds), expected reward is +1. At s=0 (never succeeds), expected reward is −1. A rational solver policy always prefers easier bugs.

The injector's reward: piecewise, and deliberately harsher at the extremes

The injection reward is adapted directly from Absolute Zero's proposer reward (Session 13), with one added twist:

where α ∈ (0,1) is a hyperparameter, set to 0.8 in the paper's experiments. Read the three branches: an invalid bug (failed one of Chapter 5's seven consistency checks) is punished hardest, at −1.0. A valid bug that's still degenerate — either impossible (s=0) or trivial (s=1) — is punished less harshly, at −α = −0.8, because it at least passed validation even though it teaches nothing. Everything in between follows a straight line, decreasing as s rises: the injector is rewarded more for bugs the solver struggles with, exactly the opposite direction from the solver's own incentive.

Worked example: the full reward table at G = 8, α = 0.8

Just as Chapter 2 worked out SPICE's reward at every achievable value of k/8, do the same here — these are the only nine solve-rate values a real 8-sample rollout group can actually produce:

Two things jump out immediately. First, the injector's reward is monotonically decreasing across the non-degenerate range while the solver's is monotonically increasing — the literal opposing incentives named in the paper's own section heading. Second, look at the jump from k=1 (+0.775) to k=0 (−0.800): a single flipped coin — one extra failure among 8 attempts — costs the injector 1.575 points of reward, the largest single-step change anywhere in the table.

Why that cliff exists: a small sample can't tell “rare” from “impossible”

The paper's own appendix works this out formally. As the continuous formula 1−(1+α)s approaches s=0, it approaches a reward near +1 — which would reward the injector for proposing something arbitrarily close to unsolvable. But with only G=8 samples, “arbitrarily small solve rate” and “exactly zero solve rate” look statistically identical — you cannot distinguish a true 2% solve rate from a true 0% solve rate using 8 coin flips. The harsh −α penalty at the s=0 boundary is the fix: it builds in a safety margin so the injector cannot exploit that statistical blind spot by learning to target “basically impossible.”

The optimal target: not the middle, but off to one side

Because the reward at the boundary is harsher than the continuous formula alone would suggest, the injector's truly optimal strategy isn't to aim for the riskiest edge of solvability — it's to aim for a solve rate with enough margin that an unlucky sample (all 8 attempts happening to fail, even though the true rate is nonzero) doesn't trigger the −0.8 cliff. The paper's theoretical analysis (treating this as an expected-reward optimization over a binomial sampling process) finds the true optimum sits near 20% — almost exactly matching k=1 out of 8 (12.5%) in the table above, the highest-reward achievable outcome (+0.775) among all nine possible values.

The general shape behind that "20%": a Beta-family reward

The paper's own appendix generalizes this beyond one specific formula. Consider any reward of the form r(s) = s^a(1−s)^b — a broader family of curves that, like SSR's actual reward, penalizes both extremes (s near 0 and s near 1) while rewarding some interior sweet spot. This family has a clean, closed-form optimum:

which is itself a small, satisfying derivation: maximize s^a(1−s)^b by taking the derivative with respect to s, setting it to zero, and solving — the algebra collapses to exactly this ratio. Two quick sanity checks make the formula trustworthy before trusting the paper's harder, non-power-law reward:

SSR's actual reward isn't a clean power law — it's the piecewise-linear formula this chapter already derived, with the harsh boundary penalty layered on top for the small-sample reason above. But the Beta-family analysis explains why the optimum lands off-center at all, rather than at the symmetric 50% you might naively expect: the reward's asymmetric shape (rewarding low s much more than it penalizes moderately-high s, until the boundary kicks in) deliberately biases the injector toward proposing bugs on the harder side of solvable, not straight down the middle.

Why G = 8, and not something larger?

It's worth asking directly why the paper doesn't just use a bigger sample group to avoid the boundary problem altogether — a larger G would let the injector target smaller, less-discretized solve rates (with G=32, for instance, the smallest nonzero rate is 1/32 ≈ 3%, much finer-grained than G=8's 12.5% floor) and would make the true value of s statistically distinguishable from exactly zero with far less ambiguity. The tradeoff is compute: every solver attempt in the group means another full agentic rollout — tool calls, bash commands, edits — on a real sandboxed repository, which is vastly more expensive per sample than SPICE's short-answer Reasoner rollouts from Chapter 2. Group size 8 is a compute-accuracy tradeoff point, not a value with special mathematical significance the way, say, the Gaussian reward's peak at p=0.5 is — a smaller group is cheaper to run at scale, at the cost of exactly the coarse-discretization boundary problem this chapter has been deriving.

The realized reward is jagged; the expected reward is smooth

There's a subtlety worth separating out cleanly, because the worked table above and this section are actually describing two different functions of the same underlying quantity. The table computes r_inject at the nine realized outcomes a real G=8 rollout group can produce — a genuinely jagged, discontinuous function of k, with the sharp −0.8 cliff sitting right at the k=0 and k=8 endpoints. But the injector isn't actually choosing k directly; it's choosing a true underlying success probability p, and then k is a random draw from a Binomial(G, p) distribution around it. Averaging the jagged realized reward over that binomial distribution produces a genuinely smooth function of p — the paper's own appendix confirms this directly, noting that the expected reward estimated from G noisy samples smooths out the discontinuity present in the raw formula, so that “a larger range of solve rates have similar expected rewards.”

That smoothing is exactly why the earlier Beta-family analysis (r(s)=s^a(1−s)^b, itself a smooth curve) is the right tool for finding the injector's true optimal target probability p*, even though the actual training-time reward function is piecewise and discontinuous in k. The injector isn't optimizing against the jagged table directly — it's optimizing against the smooth expectation of that table under repeated sampling, and it's that smooth curve whose peak sits near 20%, matching what the Beta-family shortcut already predicted.

It's worth noting where that smoothing comes from and what it implies at the two extremes of group size. As G grows without bound, the binomial sampling distribution around any true p concentrates ever more tightly around p itself (its standard deviation shrinks like 1/√G), so the smoothed expected reward converges toward the raw, un-smoothed piecewise formula — the discontinuity at the boundary becomes sharp again in the limit. At the other extreme, small G (like the paper's actual choice of 8) keeps meaningful sampling noise around every true p, which is exactly what produces the smoothing this section describes and what makes the harsh −α boundary penalty necessary in the first place — with infinite samples per bug, the true solve rate would never be ambiguous, and the whole cliff-at-zero argument from earlier in this chapter would have nothing left to guard against.

Why a piecewise-linear reward here, and a smooth Gaussian in Chapter 2?

It's a fair question why these two papers, solving structurally the same problem, land on such differently-shaped reward curves. One real difference: SPICE's variance reward is cheap to compute exactly (it's a closed-form function of one empirical pass rate) and cheap to evaluate at many candidate shapes, which is exactly why Chapter 2's ablation could afford to test four different formulas head-to-head. SSR's reward has to additionally handle consistency-validation failure as a genuinely separate, harsher case (−1.0) that has no analogue in SPICE's world at all — SPICE's Challenger can't produce anything as catastrophically invalid as a bug patch that doesn't even apply. The extra piecewise branch isn't a stylistic choice; it's answering a question (“is this proposal even well-formed at all?”) that SPICE's simpler, single-role-output task doesn't have to ask in the same way.

Why the injector is never allowed to fully solve this game

It's worth being explicit about a limit this chapter's whole derivation is implicitly working inside of, because Chapter 9 turns it into an explicit design decision rather than an accident. This reward function has an actual optimal strategy an unboundedly capable injector could, in principle, discover and exploit — not just the “aim for ~20%” calibration this chapter derives, but stranger, adversarial exploits of the reward's own construction (Chapter 9 walks through one in full). The paper's own framing of this is direct: because the challenger role has a genuine dominant strategy available to it in principle, the authors deliberately do not want self-play to run long enough or freely enough to fully explore and exploit every implication of the game's rules — a sharp contrast with two-player zero-sum games like Go, where driving self-play all the way to convergence on the optimal strategy is exactly the goal.

That's a strange thing to say out loud about a training method: don't let it fully solve its own game. But it follows directly from everything this chapter has derived. A reward with a real, discoverable exploit benefits from a policy that stays close to sensible, grounded behavior (Chapter 9's mitigations list “don't let the challenger diverge far from its initial instruction-following behavior” as a direct countermeasure) rather than a policy that's been pushed, through unlimited self-play, all the way to whatever the reward function technically rewards most. The α parameter, the piecewise structure, the harsh boundary penalty — every mechanism in this chapter is a way to make the practically achievable optimum (calibrated difficulty targeting) sit close to the game's real behavior, precisely because the theoretically optimal one (per Chapter 9) is not something you actually want the model to find.

There's a useful flip side to this that's worth stating on its own, rather than as just a caveat: precisely because training never pushes the injector all the way to the dominant strategy, a “shallow” injector — one that hasn't learned to fully exploit the game's rules — can still be genuinely useful. It still gets to take advantage of the base model's real ability to generate and explore diverse, realistic challenges (Chapter 7's removal-plus-history ablation is exactly this diversity paying off), without ever needing to discover its own worst-case optimal strategy for that diversity to be valuable. The goal was never to solve the game perfectly — it was to get a curriculum-generator good enough to track a moving target, stopping well short of wherever “perfectly” would actually lead.

python
def r_inject(s, valid, alpha=0.8):
    if not valid:
        return -1.0                    # failed one of the 7 consistency checks
    if s == 0 or s == 1:
        return -alpha                    # valid but degenerate: impossible or trivial
    return 1 - (1 + alpha) * s          # the ideal-difficulty branch

def r_solve(all_tests_pass):
    return 1 if all_tests_pass else -1

# k successes out of G=8 solver attempts on one bug
for k in range(9):
    s = k / 8
    print(k, s, r_inject(s, valid=True), 2*s - 1)
# matches the table above exactly, row for row

Chapter 7: Does It Actually Work? (showcase)

Everything so far in Chapters 5–6 is mechanism. This chapter is the check: trained this way, on nothing but bugs it invented and repaired against itself, does the resulting model actually get better at fixing real software issues it never saw during training?

The headline comparison

SSR's base model is CWM-sft — the pre-RL checkpoint of Code World Model, a 32B open-weight code LLM (the same CWM covered in Session 12). Two evaluation sets: SWE-bench Verified (500 human-verified real GitHub issues) and the harder SWE-Bench Pro (731 enterprise-scale, long-horizon problems). The comparison that matters most is against a “baseline RL” run — trained with identical hyperparameters, on the identical set of environment images, but given the human-curated issue descriptions, tests, and evaluation scripts that SSR never sees at all. Spelled out precisely: baseline RL has access to natural-language issue descriptions, the real pass-to-pass and fail-to-pass test specifications (Chapter 5's own vocabulary for what a bug artifact ultimately looks like), and ready-made evaluation scripts — exactly the standard agentic-RL-for-coding setup, just applied to the same underlying environment images SSR trains on. Its reward is a straightforward “did the given tests pass” check. SSR's solver, by contrast, only ever sees a self-generated, reversed test-weakening diff (Chapter 5) — it has to build both halves of that specification, task and test, itself.

Two things about this result are worth sitting with. First, the delta is measured on natural-language GitHub issues — text SSR's training loop never produced or trained against, since (per Chapter 5) its solver only ever sees reversed test-weakening diffs, never English issue prose. The skill transfers from “make code satisfy a formal test specification” to “understand what a human engineer meant by this bug report,” a genuinely nontrivial generalization. Second, “consistently outperforms…over the entire training trajectory” means SSR isn't just better at the end — it's ahead of the human-data baseline at every checkpoint along the way, including early ones, despite starting with strictly less curated information to learn from.

What the two benchmarks are actually testing

SWE-bench Verified and SWE-Bench Pro aren't redundant checks on the same skill — they're deliberately different difficulty regimes. Verified is 500 GitHub issues that a human review process has confirmed are solvable and well-specified — the field's standard, human-curated sanity check. SWE-Bench Pro's 731 problems are explicitly built to be harder: enterprise-scale, long-horizon tasks that stress exactly the kind of multi-step, cross-file reasoning a single isolated bug fix doesn't require. Seeing SSR's gain hold up on both (+10.4 and +7.8 respectively) rather than only the easier one is a meaningfully stronger claim than either number would be alone — it's evidence the skill generalizes past the specific difficulty profile of one benchmark's problem set.

The exact evaluation protocol, so the number isn't a black box

It's worth being precise about how these scores are actually measured, since “+10.4 points” means something different depending on how many chances the model gets per problem. The paper evaluates with exactly one attempt per problem — no parallel test-time scaling, no sampling several candidate patches and picking the best one, no majority-vote ranking across attempts. Generation uses temperature 1.0 and top-p = 0.95, the same sampling settings used throughout training rather than a separately tuned “eval mode.” A pass@1 protocol like this is a strictly harder bar than pass@k for any k>1 — there's no second attempt to fall back on if the first patch is close but not quite right — which makes the reported gains a meaningfully more conservative measurement of what SSR training actually improved than a benchmark score computed with several attempts and the best one kept would be.

What it actually costs to run this

It's worth grounding “self-play at scale” in the literal compute involved, since agentic rollouts on real sandboxed repositories are far more expensive per sample than SPICE's short text-generation rollouts from Chapters 1–4. Training runs on 512 NVIDIA H100 (80GB) GPUs per run — split 64 for the learner and 448 dedicated to generating rollouts, reflecting how much more compute-hungry agentic tool-use rollouts are than the weight updates that consume them. Context windows run up to 131,072 tokens (long enough to hold a real agentic trajectory: tool calls, file contents, bash output, reasoning). The global batch is 16 million tokens per optimizer step, and the whole run — base model, baseline RL, SSR, and every ablation — trains for 150 global steps, roughly 2.5 billion tokens, with a 30-step learning-rate warmup toward a peak of 3×10⁻⁶.

Compare that to SPICE's much lighter footprint: 640 iterations at batch 128, generating short question-and-answer text rather than full multi-turn agentic tool-use trajectories in a live sandboxed container. The compute gap between the two papers roughly tracks the gap between “write and answer a question” and “explore a real repository, run bash commands, edit files, and verify the result” — grounding in software repositories buys real generalization (Chapter 8 compares this directly) but at a real, higher price per training sample.

The engineering that keeps 512 GPUs from stalling on each other

It's worth naming what's actually solving the coordination problem at that scale, because “512 GPUs” by itself doesn't explain how the 64 GPUs doing weight updates stay usefully busy while the 448 doing rollouts are mid-trajectory on a real, slow, tool-using agentic task. SSR builds directly on the async CWM-RL infrastructure, adopting a “large-batch, small-policy-staleness” hyperparameter regime borrowed from two pieces of prior large-scale RL infrastructure work (ScaleRL and MiniRL). Concretely, this means: pack training sequences up to the full 131,072-token context limit, accumulate gradients over 16 steps to reach the 16-million-token global batch, and — the staleness half of the tradeoff — discard any rollout that has drifted more than 8 optimizer steps out of date relative to the policy that's currently training. That discard rule is the practical answer to a real asynchronous-training problem: rollout generation and weight updates run concurrently rather than in lockstep (the same actor-learner split Chapter 1 introduced for SPICE, at far greater scale here), so some rollouts will always be scored against slightly-stale advantage estimates from a few steps back. Throwing away the ones that have drifted too far keeps the training signal close enough to on-policy to remain trustworthy, without forcing every actor to sit idle waiting for the learner to catch up exactly.

It's worth connecting this back to Chapter 6's group-size question directly: the paper names rollout latency, specifically, as the reason G=8 was chosen — described as “a typical group size of 8 for acceptable rollout latency.” Chapter 6 already derived the statistical cost of that choice (a 12.5%-step discretization floor on achievable solve rates); this is the matching engineering-side reason. Every one of the 8 solver attempts in a group is a full agentic rollout — tool calls, bash commands, file edits, against a live sandboxed repository — running concurrently across the 448 rollout GPUs. A larger group would mean waiting longer for every member of the group to finish before the learner can compute that bug's reward and move on, directly trading calibration precision against how long actors sit waiting per training step. Two different chapters, two different vocabularies (statistics in Chapter 6, systems latency here), converging on the same practical number for the same underlying reason.

How much of the gain could just be noise?

The paper is explicit about a real source of measurement uncertainty: roughly 2% paired standard error on SWE-bench Verified evaluation, meaning any single run's score can wobble by around that much just from evaluation randomness (sampling temperature, minor environment nondeterminism) even with an unchanged model. A +10.4 point gain is roughly five times that noise floor — comfortably outside what evaluation noise alone could produce, though it's the right kind of number to keep in mind before treating any individual checkpoint-to- checkpoint wiggle in a training curve as meaningful on its own, rather than looking at the overall trend across the whole trajectory the way the paper's own headline claim does.

It's worth being explicit about the arithmetic that makes “+10.4 is roughly five times the noise floor” a meaningful comparison rather than a loosely-worded reassurance. A single evaluation run's score can wobble by roughly ±2 percentage points purely from sampling randomness, with no change to the model at all — that's what a 2% paired standard error means concretely. A genuine gain has to clear that wobble by a comfortable margin before it's distinguishable from noise; 10.4 divided by 2 is roughly five standard errors of separation, which is a large enough gap that evaluation noise alone is a poor explanation for it. The paper points readers toward Eval Arena for a fuller treatment of exactly this kind of question — how much of a benchmark score is signal versus noise — which is a genuinely underexamined issue across benchmark-driven ML research more broadly, not specific to this one paper's own claims.

Ablation: does self-play actually need both halves of the loop?

The paper isolates the contribution of self-play itself by comparing three training regimes, all starting from the same base model:

• Full self-play (SSR): both roles trained jointly, online, continuously generating new bugs.
• Injection-only: the agent trains only to generate bugs (using the SSR injection reward), with no learning signal from ever attempting a repair.
• Repair-only: the agent trains only to solve bugs, drawn from a fixed, deduplicated pool collected from earlier self-play runs — no ongoing bug generation.

Full self-play wins over both ablations, measured as a resolve rate over the union of both evaluation sets — all 500 SWE-bench Verified tasks plus all 731 SWE-Bench Pro tasks pooled into one 1,231-task population, a broader and stricter population than either benchmark's own per-set number reported earlier in this chapter. Injection-only degrades because a model that never practices repairing anything doesn't get better at repairing anything — obvious in hindsight, but worth confirming rather than assuming. Repair-only underperforms for a subtler reason: its bug distribution is frozen at whatever earlier self-play produced, so it never gets access to bugs calibrated to its own current, improving skill level — the same “a fixed Challenger can't track a moving target” lesson Chapter 1 already drew for SPICE, showing up again in a completely different domain.

There's a second reason full self-play beats repair-only that's easy to miss if you think of bug-injection purely as “generating training data for the real skill.” The paper's own framing treats proposing a challenging bug as substantial learning in its own right, not just as a curriculum-generation chore: identifying which tests actually pass on the unmodified codebase, breaking functionality in a way that's meaningful rather than trivial, and weakening the test suite precisely enough to hide the bug without destroying it are all skills that overlap heavily with the skills real software engineering requires — reading an unfamiliar codebase, understanding what its test suite actually covers, reasoning about what a small change breaks. A repair-only agent never practices any of that; a full self-play agent practices it every single iteration, on top of practicing repair.

Ablation: which bug-injection strategy actually teaches something?

Chapter 5 named two injection strategies plus a naive baseline. Ranked by downstream effectiveness:

The lesson generalizes past this one paper: telling a self-play system “go generate hard tasks” is not enough on its own. How the generation is scaffolded — what actions the proposer is nudged toward — measurably changes whether the resulting curriculum teaches anything at all. A model left to its own devices under naive prompting will find the path of least resistance (a one-character edit) every time.

Worth being precise about what “removal + history” actually means mechanically, since it's easy to picture it as one blended prompt asking for both at once. It isn't — each bug-injection attempt randomly samples between two entirely separate prompts, one instructing pure removal, the other instructing the git-log-mining historical-reversion strategy from Chapter 5. The improvement over removal-only comes specifically from the historical-reversion half contributing bug patterns that look like something a real developer might have actually shipped, adding realistic diversity the removal strategy alone can't produce on its own — not from some more sophisticated single prompt that does both things simultaneously.

Ablation: does the solver's feedback even matter to the injection reward?

One more check: compare the full injection reward (which depends on the solver's actual solve rate s, per Chapter 6's formula) against a stripped-down binary variant that only checks consistency validation — {−1, +1} based purely on whether the bug artifact is well-formed, with no solve-rate feedback at all.

The result is a genuine surprise: solver feedback provides only a slight, largely negligible advantage over the consistency-only baseline. The paper's own explanation is that a single noisy solve-rate number, estimated from just 8 samples, is a weak and non-committal signal for the injector to learn from — there are too many different reasons a bug could have low solve rate (badly worded, genuinely hard, an artifact of prompt formatting) for the injector to reliably disentangle which lever it should pull. What keeps working even without solve-rate feedback is the fact that the injector's policy itself keeps evolving as training proceeds — so the bugs it proposes still track its own improving skill, just through implicit policy drift rather than an explicit reward gradient. The paper's own framing of why this still works is worth quoting directly: even without solve-rate feedback in the reward, the injector's policy is still continuously updated from both bug-generation and bug-solving, which lets it generate an evolving curriculum that naturally reflects the agent's current capability — something a static, pre-generated bug pipeline (Chapter 8 compares SSR against several) structurally cannot provide, no matter how the reward is shaped.

There's also a real asymmetry worth naming between the two roles here, which the paper states explicitly: for the solver, encountering a too-easy or too-hard bug reduces training efficiency (a wasted rollout that teaches little) but isn't actively harmful the way a degenerate solve rate is for the injector's reward. The injector's job genuinely requires calibrating difficulty precisely (Chapter 6's whole derivation); the solver's job is simpler and more forgiving — more like the Reasoner's binary correctness reward in Chapter 2 than like the Challenger's shaped one. That asymmetry between the two roles' reward requirements, not just between SPICE's and SSR's reward formulas, is part of why Chapter 2's careful Gaussian-shape ablation mattered so much for the Challenger/injector role specifically, while a much cruder binary reward suffices for the Reasoner/solver role in both papers.

Chapter 8: Three Papers, One Family

Two full sessions, three papers, one underlying research question: how do you let a model set its own curriculum without the curriculum drifting into fiction? Lay all three side by side and the family resemblance — and the real differences — both become legible at once.

What Absolute Zero's "three task types" actually meant, revisited

It's worth returning to Session 13's own mechanism with fresh eyes, now that SPICE's Challenger/Reasoner split and SSR's injection/repair split are both fully derived — the contrast sharpens what was special (and narrow) about Absolute Zero's approach. AZR's proposer constructs tasks around one program, one input, and one output — a triplet (p, i, o) — and asks the solver to recover whichever one piece is hidden:

python
# the SAME triplet, three different hidden pieces
p = lambda x: x * 2 + 1
i = 5
o = p(i)  # 11

# deduction: shown (p, i), predict o        -- "trace the code"
# abduction: shown (p, o), predict i        -- "work backward from the output"
# induction:  shown {(i,o) pairs} + a hint,  predict p   -- "infer the program"

All three are graded by the same code executor — run p(i), compare to the claimed o, done. That's precisely the source of both AZR's strength (verification is instant, free, and can't be fooled by fluent-sounding wrong code, per Chapter 0's compiler-as-referee argument) and its ceiling (every one of these three task types lives entirely inside “what can be expressed as a program's input/output behavior,” which is exactly the boundary SPICE and SSR each push past in their own directions).

Base model size: a real, honest confound to flag

Notice the base models aren't matched across the three papers — Absolute Zero's headline result uses a 7B model, SPICE tests 4B–8B models, and SSR trains a 32B model. Absolute Zero's own paper actually studied this directly, testing 3B, 7B, and 14B coder variants and reporting overall out-of-domain performance gains of +5.7, +10.2, and +13.2 points respectively — larger models benefiting more, not less, from the same self-play recipe. That pattern is worth keeping in mind alongside SPICE's own per-model results from Chapter 4, where the relationship between base-model strength and absolute gain wasn't quite so monotonic (the already-strong Qwen3-8B-Base gained the least of the four, +5.7, while the weaker OctoThinker models gained more). Different papers, different self-play mechanisms, and evidently not identical scaling behavior — another reason to treat cross-paper generalizations carefully rather than assuming one paper's scaling trend automatically transfers to the next.

That's not a flaw in any individual paper, but it does mean the cross-paper headline numbers in the table above (50.4, +9.1 to +11.9, +10.4/+7.8) are not directly comparable to each other the way SPICE's own internal ablation (Chapter 2's 40.7 vs. 44.9, same base model) is. When a comparison crosses papers with different base models, different benchmarks, and different domains entirely, the right conclusion to draw is qualitative (grounding helps, richer grounding tends to help more) rather than quantitative (SSR's method is exactly X points better than SPICE's) — a distinction worth being disciplined about whenever you're the one stitching together numbers from separate papers, not just when reading someone else's summary of them.

The apples-to-apples number worth remembering

The single cleanest comparison across all three papers isn't a cross-paper leaderboard — it's the one buried inside SPICE's own ablation from Chapter 2, where an Absolute-Zero-style reward and SPICE's own variance reward were trained on the identical base model, corpus, and infrastructure:

That's not two papers' self-reported numbers being stacked together, which is always a little suspect — it's one paper, one set of authors, controlling everything except the one variable under test. The gap you get from switching just the reward-shape formula (roughly a quarter of SPICE's total +9.1 gain over the untrained base) is a real, isolated measurement of how much the shape of the curriculum reward matters, independent of grounding.

SSR isn't the only way people generate synthetic bugs — what makes it different

Self-Play SWE-RL's own related-work section places it against a specific lineage of prior systems that also generate synthetic bug-fixing tasks at scale, and the contrast is worth drawing out explicitly, because it's easy to read “synthetic bug generation” as one undifferentiated category when it isn't.

Every system in the first five rows solves the same surface-level problem SSR solves — producing a large volume of executable, verifiable bug-fixing tasks without relying purely on scarce, expensive real GitHub issues. None of them are self-play in the sense this entire session has been building up: none of them have a bug-generating policy that is itself trained by reinforcement learning to calibrate its own output against a solver's improving skill. SWE-smith's 50,000-task corpus is enormous, but it's generated once by a fixed procedure and then used as a static training set — closer to a bigger, synthetic version of SWE-Gym's curated dataset than to an adversarial curriculum that tracks the solver as it improves. SSR's injector, by contrast, keeps generating new bugs calibrated to wherever the solver currently is, for as long as training continues — the same “a fixed corpus doesn't track a moving target the way a co-evolving generator does” lesson Chapter 1 drew for SPICE's Strong-Challenger baseline, now showing up as the dividing line between an entire family of prior systems and this session's actual subject.

It's also worth noting explicitly what SSR's tool-use scaffold (Bash plus a search-replace editor, per Chapter 5) is not reinventing: agent scaffolding for software engineering is itself an active research area, split between agentic scaffolds (an LLM driving tool-mediated interaction directly, the lineage SWE-agent's agent–computer interface popularized) and pipeline-based scaffolds (human-defined stages for fault localization, patch generation, and patch selection, trading generality for stability). SSR inherits its scaffold wholesale from CWM — the base model Session 12 already covered — rather than contributing a new one; this session's actual contribution sits entirely in how the training tasks themselves get generated, not in how the agent interacts with the sandbox once it has a task.

The same related-work section also places SSR against prior work that trains core model competence with RL rather than generating tasks with it, and the contrast there is a different one worth naming: SWE-RL applies RL to real software-evolution data (issues, PRs, code diffs) with a lightweight verifiable reward based on patch similarity, improving SWE-bench Verified solve rates while staying open-weight and mid-sized; DeepSWE trains a 32B open-weight agent from scratch with pure RL on containerized software environments using execution-based reward. Both improve model competence through RL the way SSR does — but both still train against real or realistically-curated tasks, not tasks the model itself invented and validated through self-play. CWM, SSR's own base model, sits one step earlier in this same lineage: trained as a reasoning agent whose trajectories interleave reasoning and tool use, reaching state-of-the-art results among 32B models with test-time scaling, but reached through Session 12's own training recipe rather than the self-play loop this session adds on top of it as a further RL stage.

Why "two roles, one policy" is itself a workaround, not a free design choice

It's worth asking a question this whole session has quietly sidestepped: why does every method covered so far — Absolute Zero, SPICE, SSR — use one shared policy playing two roles by prompting, rather than two genuinely separate models trained against each other the way classic multi-agent RL usually works? SPICE's own related-work section names this directly as a real, unsolved engineering difficulty, not a stylistic preference. Full-scale multi-agent RL with modern autoregressive LLMs is hard enough that prior attempts have each made a real compromise to get it working at all: one line of work substitutes RNNs for transformers to sidestep the difficulty; another restricts itself to simplified environments that don't require full autoregressive text generation; a third (closer to genuinely separate agents) shows self-play works when combined with supervised fine-tuning on proprietary models, sidestepping some of the RL-specific instability. A more recent method, SPIRAL, demonstrates that zero-sum games between separately-tracked agents can teach transferable reasoning through multi-turn interaction — but only by hand-designing careful game environments for the two agents to play in.

Seen against that backdrop, “one policy, two roles, distinguished only by which prompt it's given” is a genuinely load-bearing engineering simplification, not an incidental detail Chapter 1 mentioned in passing. It sidesteps the multi-agent RL difficulty entirely: there's only ever one set of weights to update, one optimizer, one set of infrastructure to scale — the “actor-learner” split Chapter 1 walks through would be considerably more complex if the Challenger and Reasoner (or the injector and solver) were two separately-tracked policies that also had to somehow stay synchronized with each other's progress. The cost of that simplification is exactly what Chapter 9's dominant-challenger and tunnel-vision-challenger failure modes describe: a shared policy means whatever exploit either role discovers is available to both roles simultaneously, since there's only one model to have discovered it.

SPICE's other lineage: static question-mining, and the same NaturalReasoning corpus twice over

There's a second family of prior work SPICE positions itself against, distinct from the multi-agent-RL lineage above: methods that generate synthetic training questions from a corpus without any self-play loop at all. Bootstrapping approaches like STaR and MetaMath stay bounded by whatever coverage their initial seed dataset happens to have. Self-Instruct and its reasoning-focused successor CoT-self-instruct generate synthetic prompts from a handful of few-shot examples, but the result is a fixed, static dataset generated once and then trained on — not a curriculum that adapts as the model improves. Corpus-mining methods reach broader coverage: WebInstruct harvests question-answer pairs at scale but needs hand-built rule-based filters to stay clean, and two methods worth noting by name — General-Reasoner and, notably, NaturalReasoning itself — mine millions of questions directly from web content into static, offline datasets.

That last one is worth pausing on, because it produces a small, satisfying loop back to Chapter 1: NaturalReasoning is both one of SPICE's two live training corpora and the name of a prior static-dataset-generation method SPICE's own related work explicitly contrasts itself against. The difference isn't the underlying text — it's what SPICE does with it. A static corpus-mining method reads NaturalReasoning's source material once, generates a fixed batch of questions from it, and stops; SPICE's Challenger keeps re-reading documents from the same underlying pool of source material throughout training, continuously generating new questions calibrated to wherever the Reasoner's skill currently sits (Chapter 2's variance reward), rather than a one-shot dump of questions generated before training even begins. Same raw material, two entirely different things done with it — a static dataset versus a live, adaptive curriculum — and that difference is the entire argument this session has been making since Chapter 0.

Where the two Session-14 papers explicitly cite each other

This isn't an analogy this lesson is drawing after the fact — Self-Play SWE-RL's own introduction cites SPICE directly, framing its own contribution as the natural next step: SPICE showed that grounded self-play beats ungrounded self-play for general reasoning; SSR asks whether the same principle extends from text documents to software repositories, and finds that it does. The two papers were written by different teams (SPICE from FAIR/NUS, SSR from FAIR) but share a diagnosis and a fix, applied to two different substrates.

SSR's own introduction even runs the exact thought experiment Chapter 5 opened with, nearly verbatim: what can a human learn by only interacting with a Python interpreter, the way Absolute Zero's proposer does? Every intricacy of Python — but not the much larger body of knowledge that only exists inside real, human-built codebases and can't be inferred from language semantics alone. That's the same diagnosis Chapter 0 opened this session with, now stated by the second paper's own authors as their explicit motivation for extending SPICE's corpus- grounding idea from text into code.

The tradeoff hiding underneath “richer grounding is better”

It would be a mistake to read this session as a simple ranking where richer grounding always wins outright. Richer grounding costs more to verify, as the table above's per-task verification-cost row makes concrete: a Python snippet runs in milliseconds; a rule-based text-equivalence check is nearly as cheap; a full agentic rollout against a live sandboxed repository is, by comparison, extremely expensive — which is exactly why Chapter 7 had to spend a paragraph on 512 GPUs and 2.5 billion training tokens where Chapter 1's SPICE recipe needed nothing like that scale. The right way to read this progression isn't “always ground in the richest possible substrate” — it's “match the grounding to what the target domain actually needs, and budget for what that grounding costs to verify at scale.” A team building a math tutor has no obvious use for sandboxed Docker images; a team building a coding agent has no obvious substitute for one.

Chapter 9: Limits & Connections

Grounding fixes the failure mode Chapter 0 opened with. It does not make self-play unbreakable. Both papers are unusually candid about exactly where their own methods can still go wrong, and that honesty is worth taking as seriously as the headline results.

The dominant challenger: a reward function can be defeated on paper

SSR's appendix works through this formally, and the conclusion generalizes to any self-play system where the proposer has a rich enough action space. Suppose the bug-injection agent is allowed to modify test files (it is — that's the whole point of test_weaken.diff). Then it has a genuinely dominant strategy: write a test that passes with a fixed target probability regardless of what the solver does, for instance

python
if rand(seed=hash(current_code)) < p_target: pass()
else: fail()

This test's pass rate is deterministic (seeded on the code's own hash, so it's reproducible, not literally random) but completely disconnected from whether the solver's fix is actually correct. If the injector also obfuscates the surrounding code so the solver can never identify what's really being tested, this strategy guarantees the injector its own optimal reward no matter what the solver does — and the solver learns absolutely nothing, because there was never a real bug to find in the first place. The paper's honest assessment: this is hard to rule out entirely whenever the injector has a Turing-complete action space, though a model without explicit awareness of its own training setup may not stumble onto it even if it's theoretically capable of the strategy.

The paper's own appendix walks the construction one step further, and the extra step is worth including because it patches an obvious objection: what if the consistency checks require that the original, unmodified code must still pass the test? A seeded-hash test that fails on literally every input (including the original, correct code) would get caught immediately by the “bug validity” check from Chapter 5's seven-check pipeline. The refined version handles this by special-casing the original code explicitly: if hash(current_code) == hash(original_code): pass(), checked before the pseudo-random branch. Since the original code's hash is a fixed constant known in advance, this guarantees the original passes while every other candidate solution is still gated behind the same rigged coin flip. Once this construction exists, the solver's best response is a strange one to sit with: it cannot improve its odds by trying harder, because the test's outcome has nothing to do with code quality, but it also has no reason to make things worse — so rational play from the solver's side is simply to do nothing, since no action changes an outcome that was already decided before the solver ever saw the problem.

The tunnel-vision challenger: gaming diversity, not correctness

A milder but more insidious failure doesn't require a dominant strategy at all — just a limited solver. Picture a challenger that discovers long multiplication is hard for the current solver. It can keep adjusting the operand length to sit at exactly the target difficulty, forever, without ever proposing a different kind of problem. As the solver slowly improves, the challenger just makes the numbers longer. The solver's accuracy stays pinned at the target difficulty band indefinitely — looking, from the reward curve alone, exactly like healthy ongoing learning — while never actually broadening what kinds of problems it can solve. The same failure applies directly to SSR: an injector could fixate on one bug type with increasing obfuscation, or chain a fixed number of medium-difficulty sub-bugs together, without ever covering the diversity of bugs a real solver needs to generalize across.

The mitigations the SSR authors actually propose

Three, stated directly, and notice all three are really restatements of this session's central theme:

1. Ground the challenger in large, diverse, real-world data — exactly the fix this entire session has been building toward, now stated as a defense mechanism rather than just a performance booster.
2. Don't let the challenger's policy diverge far from its initial instruction-following behavior, which limits how far it can wander toward a tunnel-vision or dominant strategy in the first place.
3. Don't expect self-play alone to improve natural-language communication skill without continued grounding in real human language — a system that only ever talks to itself, even about a well-grounded task, can drift toward its own internal shorthand rather than staying legible to humans.

Why that third mitigation isn't just caution — the argument behind it

SSR's discussion section makes this point with an analogy borrowed from multi-agent game-theory research on human-AI cooperation (citing prior work on the game Diplomacy): when two agents learn to cooperate purely through self-play, with no human data in the loop, there's no guarantee the resulting behavior resembles how humans actually cooperate. The paper's own illustration is the classic Ultimatum Game — one player proposes a split of some resource, the other can accept or reject it, and rejecting means both players get nothing. Purely rational self-play converges on “always accept any positive offer, however small,” because a tiny amount beats nothing. Real humans reliably reject unfair splits anyway, out of a sense of fairness that has nothing to do with narrow reward-maximization. Two agents trained purely against each other would converge on the rational-but-inhuman strategy, not the human one, because nothing in pure self-play ever represents what a human actually values.

The same logic applies directly to language itself: many possible communication systems are more efficient than natural human language for a narrow, specialized task — the paper notes that effective communication in games doesn't need to resemble human language, or even be compositional the way human language is, and that human languages themselves already drift into unrecognizability over long stretches of time even without any artificial pressure at all (software engineering's own explosion of project-specific three-letter acronyms is the paper's own deadpan example of this happening at human speed). A self-play loop with no ongoing anchor to real human text has every incentive to drift the same way, optimizing toward whatever internal shorthand is most efficient for the game it's actually playing, not toward staying comprehensible to the humans it's ultimately meant to serve.

SSR's own stated limitations, in its authors' words

SSR's discussion section (§5) does something worth imitating as a research habit on its own: it separates “things we know are still weak about this method” from “things we tried that didn't work,” as two distinct, honestly-labeled subsections rather than one blurred list. It's worth keeping that same separation here, because the two kinds of admission carry different weight — a stated limitation is a property of the method as published; an unsuccessful attempt is a design path the authors explored and rejected before arriving at the method this session has spent two chapters deriving.

§5.2, Limitations — three properties of the published method itself

• No hidden oracle. The solver sees the complete test specification up front. The paper explicitly flags that this could in principle enable reward-hacking by overfitting to the exact given tests — and notes, carefully, that this was “not observed in this paper,” not that it's impossible. Their own suggested next step: split the bug artifact's tests into a public half the solver sees and a private half withheld for scoring, the same public/private split SWE-bench-style benchmarks already use to keep evaluation honest.
• Unit tests are a narrow oracle. Verification is entirely test-based, which the authors themselves note covers only a slice of what real software engineering involves — code review quality, architectural fit, maintainability, and every other axis a human reviewer would actually judge a pull request on. The paper points to a higher-level, goal-based verification abstraction — citing prior work called CodeClash — as one possible way to scale verification past “did the tests pass” in the future.
• Single model configuration for both roles. This is the one easiest to miss, and it's easy to miss precisely because Chapters 5–6 never called attention to it: the bug-injection agent and the bug-solving agent are not just trained jointly — they are, at every step, the exact same weights, the same way SPICE's Challenger and Reasoner share one policy. The paper's own words: “our experiments use a single model configuration for both roles,” and they name two specific open directions this forecloses — larger Mixture-of-Experts models, to see whether more capacity changes what self-play learns, and genuinely separate policies per role, rather than one shared policy playing two parts. A separate-policy setup would be a real architectural departure from every method this session covers, not a hyperparameter tweak: it would mean the injector and solver could specialize their weights toward their own role instead of splitting one model's capacity across both.

§5.3, Unsuccessful attempts — three design paths the authors tried and abandoned

• The natural-language experiment that failed. An early attempt to have the injector synthesize natural-language issue descriptions (instead of raw test diffs) produced text that copied test patches, was logically incoherent, and collapsed to repetitive patterns — attributed to the 32B base model's limited natural-language generation ability combined with a reward signal too opaque to steer toward quality or diversity. The paper adds two more reasons this path was abandoned rather than pushed harder: automatically evaluating the quality of free-form generated text is itself an unsolved problem (there's no Math-Verify-style equivalence checker for “is this issue description good”), and high-quality real natural-language issues are scarce compared to the volume of synthetic bug-diff tasks bug injection can produce essentially without limit. One consequence worth sitting with: because SSR's solver never trains on natural language at all, its Chapter 7 gains on real GitHub issues come entirely from getting better at writing test-passing code, not from any in-domain exposure to how humans phrase bug reports.
• Repository specialization didn't help. The authors tried training on just 23 repositories drawn from SWE-bench Verified and SWE-Bench Pro (carefully filtered to commits predating any evaluation instance, to avoid leakage), hypothesizing that deep specialization in a narrow set of codebases might outperform broad exposure. It didn't — 23 repositories turned out to be too few to provide enough diversity in problem-solving patterns, reinforcing the same lesson Chapter 5's injection-strategy ablation already pointed at: variety in what the proposer generates matters as much as how much of it there is.
• Training instability at scale. Despite borrowing stabilization techniques from prior large-scale RL work (the ScaleRL and MiniRL infrastructure Chapter 7 already named), the authors report training instability — “manifesting as gibberish outputs” — that currently caps how far SSR can be scaled. Their own hedge is worth quoting rather than paraphrasing away: they attribute it to “a combination of suboptimal hyperparameter settings, intrinsic challenges of long-horizon rollouts, model pretraining dynamics, and fundamental properties of self-play learning” — four candidate causes, offered without claiming to know which one (or which combination) is actually responsible.

None of these six items — three limitations, three unsuccessful attempts — were discovered by outside critics after publication. They're in the paper's own §5.2 and §5.3, offered by the authors themselves, under headings that name exactly what kind of admission each one is rather than folding everything into one vague “limitations” paragraph. That level of self-scrutiny is itself worth learning from as a research habit, independent of the specific technical content: a paper that only shows you what worked is telling you less than one that also shows you what it tried, where that attempt hit a wall, and what the published method still doesn't solve even though it worked.

§5.4, Future work — three directions the authors point at next

Beyond the limitations and unsuccessful attempts above, the paper names three concrete open directions, worth a brief look because each one exposes a real gap in what this session has covered so far. Distribution control with seeding: the current bug-injection agent has no explicit control over where in a repository it plants a bug, which can produce duplicate bugs or a skewed distribution when sampling repeatedly from the same codebase — the authors point to seeding techniques (borrowed from a code-generation method called Magicoder) that would hand the injector a target file or code snippet to steer generation toward, trading some of the injector's current freedom for deliberately broader coverage. Synthesizing complex multi-step software tasks: everything this session covers is repository-level bug fixing — a real but bounded task shape. Real engineering work sometimes spans much larger efforts (a major version migration, standing up a new software stack from scratch) that a single bug-and-fix pair can't represent; the paper frames higher-order bugs (Chapter 5) as a first, modest step toward that harder target, not a solution to it. Efficient training for long-horizon agents: the paper is candid that outcome-based RL, of exactly the kind this whole session derives, struggles on tasks spanning months of interdependent decisions where sparse terminal rewards give almost no signal across thousands of intervening steps — a credit-assignment problem neither SPICE's short rollouts nor SSR's single-bug repairs are long-horizon enough to actually run into yet.

SPICE's own honest boundary

SPICE's authors are equally careful in their own framing: the corpus is described as “near-inexhaustible,” not infinite. Twenty thousand documents is enormous relative to one model's own knowledge, but it is still a fixed, finite dataset — the grounding fixes the collapse mode this session opened with, it does not remove every possible limit on how far self-play can be pushed before some other bottleneck appears.

Where this connects, forward and backward

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman, Caltech commencement address, 1974

Every mechanism in this session — information asymmetry, corpus grounding, sandboxed repositories, consistency validation, the harsh penalty at a reward function's degenerate boundary — is a different engineering answer to exactly that one sentence, applied to a model instead of a scientist. Self-play without an external check is a model grading its own homework. Grounding is how you build in a second opinion that the model itself cannot quietly overrule.

This session drew directly on: Liu, Jin, Kim, Yuan, Zhao, Kulikov, Li, Sukhbaatar, Lanchantin & Weston, “SPICE: Self-Play In Corpus Environments Improves Reasoning” — arXiv:2510.24684; Wei, Sun, McMilin, Gehring, Zhang, Synnaeve, Fried, Zhang & Wang, “Toward Training Superintelligent Software Agents through Self-Play SWE-RL” — arXiv:2512.18552; and (Session 13's) Zhao, Wu, Yue, Wu, Xu, Yue, Lin, Wang, Wu, Zheng & Huang, “Absolute Zero: Reinforced Self-play Reasoning with Zero Data” — arXiv:2505.03335.