Harness Learning

Ask a chat model to solve a maze on its own, in one shot, with no scratch space. It often fails: it has nowhere to try an idea, see that it is wrong, and try again. Now put the exact same model inside a small Python program that lets it write out a guess, run that guess against the maze's rules, read back whether it worked, and try once more if not. Same weights. Same training. The score on held-out mazes jumps from near zero to strong. Nothing about the model changed. The program around it changed.

That program is called a harness: the executable code that decides what the model sees, when it is called, which tools it can reach, and what happens with each answer before the next call. An agent, in this paper's sense, is not just a model; it is a model plus a harness, and the harness does real work: it is the difference between a model that fails a maze and the same model that solves it.

This creates a second, quieter problem. A person wrote the harness above, guessing that retrying would help with mazes. A different task might need a different shape: summarizing a long document might need two reading passes, not a retry loop. Someone has to write the right harness for the right task, and until this paper, that someone was always a human, hand-tuning by trial and error.

Execution gives a rich kind of feedback that training a model's weights never sees directly: a trace. Run a harness on some questions and you get, for each one, what the model was asked, what it answered, and whether it was right, often down to which step broke. That beats a single reward number by a wide margin: it says not just "this failed" but "it failed because the retry loop gave up after one try." A person reading that trace can usually see what to change. This paper asks whether a model can read the same trace and propose the same kind of fix, on tasks it has never seen.

Call the model that does the task the solver. It stays frozen throughout this paper: its weights never move. Call the model that reads the harness and the execution trace and writes a code edit the proposer. The paper's central move is training the proposer, with reinforcement learning, at one job: given a harness, a task, and feedback from running it, write an edit that makes the harness better. Once trained, the proposer can be dropped onto a brand-new task and asked to improve its harness, over and over, using only what it learns from watching that task run. No gradient touches any weight at test time; the improvement lives entirely in the code the proposer writes.

People already hand-write good harnesses, and frameworks exist to help: one popular library lets you declare a modular pipeline of model calls and search over ways to wire it together; another searches over code-based workflows directly. These are useful tools, but they share one trait: a person or a search procedure decides, one task at a time, starting over for the next. Nothing in them already knows, from experience across many tasks, roughly what fix tends to work. That is the gap this paper closes: not a better search, but a model that has practiced revising.

Why is that gap hard to close? Because the thing being learned is unusual. Most of what gets trained into a language model maps a prompt to an answer: translate this sentence, write this function, answer this question. Here, the proposer's "answer" is a tiny act of engineering judgment: read a trace, diagnose what in the surrounding program is wrong, and write code that fixes it without breaking what works. A person doing this job draws on experience with many programs that failed in similar ways. The paper bets that a model can draw on that same experience, given enough revision attempts, on enough tasks, graded honestly on whether each attempt helped.

Here is the road, in order. Each chapter builds one part of the picture above.

1. Meta-learning over code: learning to learn, with code edits standing in for weight updates.
2. One revision round: what the proposer sees, writes, and how an edit is accepted.
3. Feed, score, judge honestly: why feedback, reward, and final-grade questions never mix.
4. Reward from execution: a training signal from running a harness, with real numbers.
5. Starting from a teacher: bootstrapping the proposer before reinforcement learning begins.
6. Adapting at test time, with no gradients: the loop that runs once training is done.
7. Generalizing to unseen reasoning, then carrying it to new questions: the real scores.
8. What changes inside the harness: what the edits look like.
9. Does training on sequences help: an honest look at what did not improve.
10. The map: the paper's own limits, and where this fits.

To see why the authors call this meta-learning, it helps to look at the ordinary kind first. In a typical machine-learning setup, there is one loop: you show the model examples, it makes a mistake, you compute a gradient, and you nudge its weights a little in the direction that would have made the mistake smaller. Do that enough times and the model gets better at the task.

Meta-learning adds a second, outer loop around that one. Instead of asking "how do we get better at one task," it asks "how do we get better at getting better," across many tasks. A classic example, MAML (model-agnostic meta-learning), trains a starting point for a network's weights such that a few gradient steps on a brand-new task, one never seen during meta-training, already work well. The outer loop never sees the new task's gradient steps directly; it only sees how well the network did after the inner loop ran, and adjusts the starting point so future inner loops go better.

Harness learning keeps that same two-loop shape and swaps out what gets adapted. No network weight gets nudged inside the inner loop at all. Instead, the inner loop edits an executable harness, a Python program, and the "gradient step" is a block of code written by a model, the proposer. The outer loop cannot compute a gradient of the harness directly, because you cannot differentiate through a code edit. Instead it trains the proposer with reinforcement learning, using how well the resulting harness performs as the reward. The proposer's weights are the only thing the outer loop changes; the harness itself never has weights, only lines of code.

Why insist on this framing instead of just saying "we train a model to edit code"? Because it sets the right bar for success. A system that only gets good at editing one specific harness for one task is not interesting; a person could write that harness by hand. The meta-learning framing says the goal is a reusable revision rule: something that, dropped onto a task family it has never revised, still knows roughly what change to try. Section 5's experiments test exactly that: every unseen family is held out from both the supervised data and the reinforcement learning, so success there comes from the revision rule generalizing, not from memorizing one task's fix.

Why reinforcement learning, and not gradient-based meta-learning all the way through? Because a code edit is not differentiable. MAML takes a gradient through its inner-loop update because that update is itself a smooth operation on continuous weights. Apply, the function that turns a proposed edit into a candidate harness, is a text substitution: it either finds the named span of code or it does not, and the result either runs or it does not. There is no slope from "the proposer's weights" to "whether this edit worked," only an outcome observed by executing the result. That is exactly the setting reinforcement learning with verifiable rewards is built for: a reward computed by checking an outcome, used to train a policy that never needs to be differentiable.

This also explains a design choice worth naming: Reasoning Gym's reinforcement learning runs with no KL penalty pulling the proposer back toward its starting point, and no entropy bonus pushing it to stay exploratory (Table 7). Many reinforcement learning setups for language models add one or both, to stop the policy drifting too far, too fast, from a known-good start. Here, with supervised fine-tuning already providing a reasonable starting habit (Chapter 5) and only 16 updates of training, the paper judged extra regularization unnecessary. That says something about scale: this is a short, targeted training run, not a long, open-ended one.

One more piece completes the picture: what does "outer loop" mean with no fixed number of inner-loop steps? In MAML, the inner loop is usually a handful of gradient steps, decided in advance. Here, the inner loop is T rounds of revision, and T can be as small as one (a single edit) or as large as ten (a long test-time adaptation run). The outer-loop objective, which the next chapter turns into a concrete training loss, is to make the harness after T rounds, call it h_T, as good as possible on held-out questions, in expectation over tasks, starting harnesses, and the randomness of execution.

max_θ

The outer loop's job: find proposer weights θ that make the following quantity as large as possible.

𝔼

An average over which task is being revised, which questions are drawn, which edits the proposer happens to sample, and the fact that running a harness is itself a little random.

J(h_T; Q_eval)

The mean grader score of the harness produced after T rounds of revision, measured on a held-out question set Q_eval the proposer never sees during adaptation.

This is the paper's Equation 5, and it is the true goal. Chapter 4 shows that training does not optimize it directly, that would require knowing the whole T-round outcome before updating, but instead uses each single revision's own outcome as a stand-in. Keep Equation 5 in mind anyway: it is the yardstick every later result is really trying to move.

Zoom into a single round of the inner loop, round t. Three things exist before the proposer runs. There is a task: a description, a pile of questions, and a grader that scores an answer from 0 to 1. There is a harness from the previous round, written h_t-1. And there is an execution report: a summary of what happened the last time h_t-1 ran, including which questions it got right, which it got wrong, and excerpts of the trace for the failures.

The proposer's input, x_t, is simply those three things stitched together: the task description, the current harness's source code, and the execution report. Note what is not in there: no instruction saying "add a retry loop" or "use more tool calls." The proposer has to figure out, from the report alone, what kind of change would help.

Because one proposed edit might be unlucky, bad code, a misunderstanding, the paper does not bet everything on a single attempt. It samples G candidate responses from the proposer, independently, for the same input:

y_i

One sampled response: text that describes a code change and then states the change itself, as a block of edits.

π_θ(· | x_t)

The proposer, a language model with weights θ, treated as a distribution over possible responses given the input x_t.

G

The number of independent candidates drawn this round. In the Reasoning Gym protocol, G = 1 for test-time multistep revision (one proposal per round) and G = 8 for single-step evaluation; in multi-hop QA, G = 8 at every round.

Each raw response y_i is not yet a harness. It is a proposed edit: in the paper's structured interface, a span of existing code plus its replacement, phrased as a search-and-replace pair, together with a short explanation. A function called Apply takes the previous harness and one edit and produces a candidate harness:

h'_i

Candidate harness i: the parent harness with this one edit applied.

Apply

A mechanical function, not a model: it locates the span the edit names inside the parent's source and substitutes the replacement.

h_t-1

The harness this round started from, the one the execution report in x_t describes.

y_i

The proposer's i-th sampled edit.

Apply can fail honestly: if the named span does not match the current source, or the result does not parse as valid Python, the candidate is marked invalid rather than silently patched. An invalid candidate is not thrown away unscored, it simply cannot earn task credit, which matters a great deal once Chapter 4 defines the reward.

Now there are up to G candidate harnesses. A function called Select decides which one becomes h_t, the harness the next round will see:

h_t

The harness that is carried forward into the next round, or the one the test-time user keeps.

Select

A rule, fixed per experiment, not learned: it runs every valid candidate on the scoring questions and decides which one to keep, by a protocol stated in advance.

h_t-1

The parent harness, which Select can choose to keep instead of any candidate.

{h'_i}

The set of candidate harnesses produced this round.

Q_score

The scoring questions used to judge candidates this round. Chapter 3 explains why these must be separate from the questions in the execution report.

The two Select rules the paper actually uses are different, and both are reasonable. On Reasoning Gym, Select keeps the single candidate (G = 1) only if its development score beats the parent's; otherwise the harness does not change this round, a conservative "never go backward" rule. On multi-hop QA, Select looks at eight candidates and advances to whichever scores highest, even if every one of them, including the best, scores below the parent, because a QA harness is complex enough that a temporary dip can still be heading somewhere useful.

Imagine grading your own homework with the answer key open the whole time. Every answer looks right, because you can just copy the key. A harness that is allowed to see the same questions it will later be judged on has the same problem: it can look good not because it generalizes, but because it has, in effect, memorized the test.

The paper closes this loophole by splitting every task's questions into three disjoint roles, each used for a different job and never mixed. Call the full question pool Q. It splits into:

Q_fb questions are run through the parent harness to build the execution report that goes into the proposer's input, x_t. They tell the proposer what already went wrong. Q_score questions are run through every candidate harness this round, to compute each one's reward and decide which one Select keeps. Q_eval questions are never touched during adaptation at all; they exist purely to measure, after everything is done, how good the final harness actually is. The paper is explicit that evaluation questions are disjoint from every question used for feedback or candidate selection, during any round.

Note that Q_fb and Q_score are allowed to overlap in one specific, controlled way: in some protocols, the same development questions serve both roles, feedback and scoring, deliberately. That is a design choice, not an oversight, because the thing it protects against is candidates seeing the held-out questions, not candidates being scored on the same set they got feedback from. During training, the paper goes one step further and keeps scoring questions disjoint from the questions used to build that context's report, so even the training reward does not reward memorizing the feedback set.

The paper gives this split concrete sizes, and the sizes themselves carry a lesson. On Reasoning Gym, each task family gets 25 development instances and 75 held-out instances (Table 3): development instances cover both the Q_fb and Q_score roles, while the 75 held-out instances are Q_eval and never touched during adaptation. Three times as many questions go into the number that gets reported as go into the number the proposer ever sees. In multi-hop question answering, training builds each context from six feedback questions and sixty-four separate scoring questions, both drawn from a pool of 2,000 training questions, with a further 500 development and 300 held-out test questions kept entirely apart from that training pool (Table 21). The scoring set is more than ten times larger than the feedback set, because scoring has to be reliable enough to rank eight candidates against each other, while the feedback report only needs a handful of representative failures to describe what went wrong.

Here is a small worked example of exactly what inflation would look like, to make the abstract warning concrete. Say a harness genuinely generalizes at 52% exact match on truly new questions, the honest number. Now suppose, by accident of how a pipeline is built, 30% of the "held-out" evaluation questions had already appeared, verbatim, among the scoring questions used to select that harness during adaptation. Because the harness was explicitly selected for doing well on exactly those questions, its score on that overlapping 30% will tend to run high, not because it generalizes better, but because those specific questions already shaped which candidate got kept. Blend that inflated slice back into the reported evaluation number and the overall score drifts upward, say to somewhere in the 60s, even though nothing about the harness's real ability to handle new questions improved at all. The device above plays exactly this scenario out with a slider, so the shape of the inflation is something you can watch rather than just take on faith.

This three-way split also explains a subtlety in how rounds chain together at test time. After round t selects h_t, the next round needs a fresh report. The paper runs the selected harness on a new draw of feedback questions, Q_fb for round t+1, and builds the next report from that. So the proposer's view of the world keeps updating, new failures, new successes, on fresh material, while the evaluation questions sit untouched until the very end, when they are used once to produce the numbers in every figure this lesson is about to show.

The proposer is a language model, and the standard way to train a language model from a reward signal, without needing a reward that can be differentiated, is reinforcement learning. The general objective, for a policy π_θ that maps a context x to a response y, is to maximize the expected reward of its responses:

J(θ)

The quantity training tries to maximize: expected reward.

𝔼_x~D,...

An average over revision contexts x drawn from the training distribution D (Chapter 2's x_t, for many different tasks and parent harnesses), and over the proposer's own sampled responses.

r(x, y)

The reward for response y given context x. The rest of this chapter defines what that reward actually is here.

Here, r is not a label a human wrote. It is computed by actually executing the candidate harness that y produces. For an evaluable candidate, the paper defines the reward as:

r_i

The scalar reward assigned to candidate i.

J(h'_i; Q_score)

The candidate's mean grader score (0 to 1) on the scoring questions: how well its revised harness actually does the task.

v_i

A small auxiliary reward for the edit itself being well-formed and runnable, independent of whether it got the task right.

On Reasoning Gym, that auxiliary term has a specific recipe: v_i = 0.05·1[parses] + 0.10·1[runnable] + 0.15·ρ_run(h'_i), where ρ_run is the fraction of scoring trials that complete without crashing or timing out. Say a candidate's edit parses correctly, applies to produce a runnable harness, and 90% of its scoring trials complete, but the answers themselves are wrong half the time. Its task score is roughly 0.90 × 0.5 = 0.45 (each completed trial scored 0.5 on average), and its validity bonus is 0.05 + 0.10 + 0.15×0.90 = 0.285. Total reward: about 0.735. Compare that to an edit that fails to apply at all: it earns only whatever validity credit it accumulated before failing, with no task-score term, because there is no runnable harness to score.

Rewards alone are not enough to train with; reinforcement learning with verifiable rewards needs a way to turn a batch of rewards into a per-token update. The paper uses GRPO (group relative policy optimization). Instead of comparing a response's reward to some separately learned baseline, GRPO compares it to the other responses sampled for the same context: it normalizes rewards within the group of G candidates.

Â_i

The group-normalized advantage for candidate i: how much better or worse it did than its siblings, in standard-deviation units.

r_i

Candidate i's own reward, from the equation above.

mean_j(r_j)

The average reward across all G candidates sampled for this same context.

std_j(r_j)

The spread of rewards across the same group.

δ

A small constant that stops the division from blowing up when every candidate in a group scored identically.

Let's work one group of four candidates by hand. Say their rewards are r = [0.80, 0.50, 0.20, 0.10]. The mean is 0.40 and the standard deviation is about 0.26. Candidate 1's advantage is (0.80 − 0.40) / 0.26 ≈ 1.54: it did much better than its siblings, so every token it wrote gets pushed to be more likely next time. Candidate 4's advantage is (0.10 − 0.40) / 0.26 ≈ −1.16: its tokens get pushed to be less likely. Nothing here compares candidate 1 to some fixed external bar; it only has to beat the other three edits proposed for the exact same broken harness.

GRPO then uses Â_i inside a clipped surrogate loss, the same family of objective used across modern reinforcement learning for language models. Each token of a high-advantage response gets its probability nudged up. Each token of a low-advantage response gets nudged down. Clipping keeps any single update from moving the policy too far in one step. The clipping and the per-token importance ratios are standard machinery. What is specific to this paper is everything upstream of them: how x, y, and r get defined in the first place, which the last several equations just built from scratch.

Table 7 gives the concrete numbers for Reasoning Gym training: G = 8 candidates per context, 40 contexts per update, a learning rate of 10⁻⁶, one GRPO update per sampled batch, for 16 updates total. Twenty-five scoring instances per context, each with up to 300 seconds to run. None of this changes the shape of the equations above; it just says how many times, and how big a step, the update gets taken.

Training a 4-billion-parameter model with reinforcement learning on a brand-new skill is slow from a cold start. That skill here is editing another program's source code based on an execution trace. The model has to stumble onto a working edit by chance before it can ever get useful reward. The paper's Reasoning Gym setup avoids that cold start with an optional supervised fine-tuning (SFT) stage, using a much larger model as a one-time teacher.

The teacher is Qwen3.6-35B, roughly nine times the proposer's size. For a set of training contexts, the teacher sees the same input the 4B proposer will eventually see, the task, the seed harness, and its execution report, and proposes an edit. The paper executes the resulting harness for real and only keeps the result if it clears two bars at once: it must score at least 0.08 above the seed's development score, and at least 0.40 in absolute terms. A candidate that technically improves a terrible seed from 0.02 to 0.09 does not make the cut; a candidate has to be both an improvement and genuinely decent.

Passing the score bar is not enough either. Surviving candidates also go through automated screens: for answer echoing (did the harness just hard-code the expected answer instead of computing it), for duplicate code, and for interface violations. A separate reviewer model then checks each candidate again, for anything the automatic screens might miss, like oracle access to information the harness should not have. No held-out question text ever enters the training corpus this way.

What survives this filter is telling. Of the 432 retained examples, 48% are what the paper calls interpreter loops: a harness that hands the solver a code sandbox and lets it write and run its own computation rather than reason everything out in text. That is the teacher's favorite fix for a struggling seed harness, by a wide margin over any other structure, and it is a pattern the 4B proposer ends up leaning on too, as Chapter 9 shows.

Supervised fine-tuning itself uses a low-rank adapter, LoRA, rank 32, touching every attention and feed-forward projection in the model, trained for roughly 2.9 epochs at a learning rate of 10⁻⁴ with cosine decay. The exact duration is not guessed; it comes from four-fold cross-validation where the families are kept disjoint across folds, so the choice of how long to train is itself protected from leaking information about any one task family into its own evaluation.

The LoRA adapter deserves a moment, since it is doing more work than "a cheaper way to fine-tune." Rank 32 means each updated weight matrix is constrained to a 32-dimensional correction on top of the frozen base model, touching every attention and feed-forward projection, including the projections specific to the proposer's gated-delta-net layers. That is a narrow slice of the full parameter space, which is exactly the point: with only 432 training examples, a full fine-tune of every parameter would have far more room to memorize the specific examples than to learn a general revision habit. A low-rank adapter is a built-in form of restraint, forcing whatever the model learns to be compressible into a small correction.

How long to train that adapter is itself chosen carefully, not guessed. The paper runs four-fold cross-validation over all 432 examples, with the families kept disjoint across folds, meaning no family's examples appear in both a fold's training split and its validation split. The duration that performs best, on average, across those four folds becomes the final training length, roughly 2.9 epochs. This matters for the same reason Chapter 3's question-set split matters: choosing a hyperparameter by checking it against data it was allowed to see would quietly leak information, so the paper structures even this small decision, how long to train, to be protected from that leak.

Multi-hop question answering skips this stage entirely. Its reinforcement learning starts directly from the base, untrained Qwen3-4B proposer, no teacher demonstrations at all. That is a deliberate contrast built into the experiment design: it tests whether reinforcement learning alone, with no supervised head start, can still learn a useful revision rule when the task (rewiring a multi-hop retrieval pipeline) is different in kind from short algorithmic puzzles.

Training ends. The proposer's weights are saved and never touched again. Now hand it a brand-new task, one whose questions it has never revised a harness for, and watch what happens at test time. This is the loop Chapter 0's hero device plays out, and this chapter gives it its exact rule.

Start from a seed harness, h₀, the simplest reasonable program for the task (on Reasoning Gym, one that just calls the solver once and reads off its answer). Run it on feedback questions, build an execution report, and hand the proposer its input exactly as Chapter 2 described. The proposer writes an edit, Apply produces a candidate, and Select decides whether the candidate becomes h₁ or whether h₀ survives unchanged. Repeat for round 2, round 3, and so on, up to T rounds.

The Reasoning Gym retention rule, specifically, is the "never go backward" version from Chapter 2:

h^★_t

The retained harness after round t: the best one seen so far, by this rule's own judgment.

h'_t

This round's single candidate (G = 1 in the multistep Reasoning Gym protocol).

J(·; Q_dev)

Development score: the mean grader score on the round's development questions, the same set that supplied the feedback report.

h^★_t-1

The previously retained harness, kept if this round's candidate does not beat it.

This is a ratchet: the retained harness can only improve or stay flat on its own development questions, round over round, never get worse. That is reassuring, but it is not automatically what the paper actually reports. Figure 4 plots held-out score, a different, untouched question set, across rounds, and a held-out curve can genuinely wobble even while the development ratchet only climbs, because doing better on the questions you can see does not guarantee doing better on questions you cannot.

The multi-hop QA retention rule, recall from Chapter 2, is looser: it advances to the best of eight candidates even if every one of them scores below the parent. That looseness has a purpose at test time specifically. A two-hop retrieval pipeline is complex enough that a single round's "improvement" can be a step sideways, say adding a structural piece that only pays off once a later round also adds the piece that makes it useful. A rule that refuses anything short of immediate improvement might never take that first, temporarily unhelpful step. Reasoning Gym's simpler seed harness, by contrast, rarely needs that kind of multi-round setup, so refusing to regress is the safer default there.

It is also worth being precise about where the training data for each loop's reliability comes from. Reasoning Gym's multistep training (Chapter 4's Table 8) refreshes its revision states in two offline phases: train on a batch of stored parent harnesses, then advance each one to its best sampled revision and rescore before the next phase begins. Multi-hop QA's multistep training, by contrast, generates its revision chains online, during training itself, with 32 revision runs the evolving policy actually advances round by round. Different domains, different practical ways of assembling the chains the proposer practices on, but the same underlying question for both: does practicing on chains, rather than independent single steps, produce a better test-time adapter? Chapter 10 answers that directly, and not entirely in the chains' favor.

One more thing is easy to miss. "No parameter-space update" does not mean "no learning happens between rounds." Learning happened once, during training, and it is frozen into the proposer's weights. What changes between rounds at test time is entirely in the harness and the context the proposer reads: a new report, describing a new, slightly different harness. The proposer's weights are reused unchanged, round after round, the same way a single trained chess engine is reused unchanged across many different games, each one unfolding differently because the board is different, not because the engine learned anything new mid-game.

Reasoning Gym is a collection of synthetic task families, mazes, Sudoku, arithmetic, anagrams, logic puzzles, each with its own question generator and automatic grader. The paper trains its Reasoning Gym proposer on 21 families through supervised fine-tuning and 5 more through reinforcement learning, then tests it on 21 entirely different families, never shown during either stage. This is the sharpest test of generalization in the paper: not a harder version of a trained family, but families whose questions, failure modes, and right harness shape the proposer has never encountered.

Figure 3 reports the headline single-step number: averaged over proposals on 21 unseen families, the untrained Base proposer's revisions score 0.32. After supervised fine-tuning and reinforcement learning, the same proposer's revisions score 0.62. That is not a selection effect, not "the best of several tries," it is the mean over every sampled proposal, so the average edit genuinely got better, not just the luckiest one.

It helps to see where that 0.32-to-0.62 jump is not uniform, because a single mean number can hide a lot. Table 9 breaks the 21 unseen families apart individually, and the spread is wide. On sudoku, Base averages 0.42 and Single-step RL reaches 0.94, a family where the untrained proposer already had some traction and training sharpened it dramatically. On n_queens, Base averages just 0.01 and Single-step RL only reaches 0.26, still a large relative gain, but both numbers stay low: an eight-queens-style constraint puzzle this hard evidently resists the kind of harness fix the proposer has learned to make. On propositional_logic, every method, Base, SFT, RL, and even the 35B teacher, stays clustered between 0.05 and 0.18, a family where the paper's own revision strategies simply do not help much yet. Reading the mean alone would miss that the proposer's skill is real but uneven, strong on some families, modest on others.

The comparison with the teacher deserves a careful reading, because it is easy to overstate. Under oracle best-of-eight selection, where you are allowed to pick the single best of eight proposals after the fact, the teacher still comes out ahead: its ceiling is higher. What the trained 4B proposer wins on is the average proposal: pick any one of its eight tries at random, and on average it beats a random one of the teacher's eight tries. Supervised fine-tuning alone, before reinforcement learning, reaches a mean of 0.49, already past the untrained Base's 0.32 but still short of the teacher; reinforcement learning on top of that closes the remaining gap and crosses it. Learning from the outcomes of your own revisions, rather than only imitating a teacher's revisions, is what pushes average quality past the teacher that supplied the initial training data.

Multistep revision tells a related but distinct story. With five sequential rounds, Table 13 reports the Single-step RL proposer reaching a mean score of 0.699 on canonical questions and 0.725 on format-varied questions, both comfortably above the untrained Base's single-round 0.593 and 0.498. Figure 4 breaks this down by round: on format-varied questions, most of the five-round gain lands in round 1 for the trained proposer, while SFT alone improves more gradually, spreading its gains across rounds instead of front-loading them.

One more comparison is worth sitting with. Figure 4c compares five sequential proposals, each one building on the harness the last one left behind, against eight independent proposals, each one a fresh attempt from the same seed. The sequential run, using five proposals' worth of compute, approaches or beats the oracle best score among eight independent ones, and clearly beats their average. Building on your own best-so-far harness, using feedback from running it, gets more out of a small proposal budget than restarting from scratch each time.

It is worth asking directly why later revision rounds tend to add less than round 1 did, since that pattern shows up in every one of this chapter's curves. The seed harness here is deliberately minimal, one solver call with no scratch space. Round 1's job is usually to add the single biggest missing piece: a retry loop, a code sandbox, some way for the model to check its own work. That one addition typically accounts for most of what was wrong. Once it is in place, later rounds are left hunting for smaller, more specific fixes, say handling one particular edge case the grader penalizes, which simply have less score left to recover. Chapter 8 shows a seed harness with more moving parts already built in, and there, later rounds have correspondingly more left to find.

Reasoning Gym families are synthetic and self-contained: a maze has a clean right answer, checkable by a grader in milliseconds. Multi-hop question answering is messier and closer to how people actually use retrieval-augmented agents. A question like "What year was the university founded that employed the author of the book that inspired the 2015 film?" cannot be answered from one search. It needs two or more hops: search, read, decide what to search for next, search again, then answer.

The paper's seed QA harness already has a shape: retrieve passages for the question, summarize them, write a follow-up query from that summary, retrieve again, and answer from the summaries of both hops. This is a more complex starting point than Reasoning Gym's "call the solver once" seed, and it means there is already a sensible structure for the proposer to refine rather than invent from nothing.

Training happens on HotpotQA, with no supervised fine-tuning stage, reinforcement learning straight from the base Qwen3-4B proposer, using BM25 retrieval (a classic keyword-matching search algorithm, not a learned one) over 5.23 million Wikipedia abstracts. Then, with zero additional training, the same frozen proposer is handed MuSiQue and 2WikiMultihopQA, two benchmarks it has never revised a harness for, built from entirely different passage pools, 21,100 and 54,957 paragraphs respectively, and different question styles.

Every harness in this chapter, seed and revised alike, runs inside the same fixed budget: at most twelve solver calls and twelve retrievals per question, each solver call capped at 120 seconds, each full scoring run capped at 900 seconds. That ceiling matters for reading the results honestly. A revision that adds a third retrieval hop is not free: it spends part of the same twelve-call budget every other design choice also has to share. A harness that searches too eagerly can run out of calls before it ever answers; one of the proposer's jobs, implicitly, is finding a structure that uses this fixed budget well, not simply one that would help with unlimited calls.

The solver doing the actual reading and answering here is a frozen Qwen3-8B, twice the proposer's size, run non-thinking (it does not generate extended reasoning before answering, only the final response). That is worth noting because it means the gap between Base and the trained proposers in this chapter is entirely a gap in harness quality: the same 8B solver, reading the same passages, performs very differently depending on how the surrounding program routes evidence to it, decides when to search again, and decides what counts as a good-enough draft answer.

The most common revision keeps the first-hop summary for guiding what to search next, but changes what the final answering call reads: instead of answering from summaries of both hops, it reads the retrieved passages directly. That separates two jobs that the seed harness had bundled together, deciding what to search for next, and having enough raw evidence to actually answer, which a summary can accidentally compress away. Other revisions add a third retrieval hop outright, or add a check: if a draft answer does not literally appear anywhere in the retrieved passages, treat the draft as a new search query and try again before answering.

The numbers bear out that these changes travel. Figure 5 and Table 23 show Single-step RL's independent, single-revision proposals averaging 0.41 exact match on HotpotQA (in-domain), 0.15 on MuSiQue, and 0.31 on 2WikiMultihopQA, all above their respective seed scores of 0.39, 0.14, and 0.25. Base, the untrained proposer under the identical protocol, actually finishes below the seed on HotpotQA and MuSiQue: without training, random edits to a working retrieval pipeline are at least as likely to hurt it as help it.

Successive revision over ten rounds moves the numbers further, and here the ceiling on what's achievable gets genuinely surprising. On MuSiQue, ten rounds of sequential revision average 0.27 exact match, nearly double the 0.15 average from independent single revisions. And on both unseen benchmarks, the average final run of ten-round revision exceeds even the oracle best score picked from 80 independent proposals, a number no realistic single-shot selection rule could reach. Repeated adaptation, using feedback from your own prior edits, finds harnesses that no amount of independent first-try guessing turns up.

Numbers say revisions got better on average. They do not say what kind of program the proposer tends to write. The paper audits this directly, by classifying every generated harness into structural categories and counting.

On Reasoning Gym, without any special prompting toward a particular structure, SFT and Single-step RL overwhelmingly converge on one shape: the interpreter loop, where the solver is given a code sandbox and writes its own computation rather than reasoning everything out as free text. This matches what Chapter 5 found in the teacher's own training data, where interpreter loops were already the dominant pattern (48% of retained examples). Training does not invent a new kind of structure from nothing; it reinforces and refines a pattern the data already favored.

The paper also probes something sharper: what happens when a prompt explicitly asks for a reusable helper tool, something more sophisticated than a generic code sandbox, say a dedicated search function wired into the solver's own tool loop? It separates two bars for success. Structural composition just means a harness technically plugs a helper into the solver's tool-calling loop. Functional composition additionally requires that helper to actually contribute to the final answer, not just sit there unused.

Under this directive, structural composition happens in a real minority of cases, and functional composition is rarer still: in one unseen-family probe, only 4 of 24 harnesses meeting the structural bar also met the functional one. The paper's own inspection of the failures is specific and useful. A composed harness often pairs a genuinely correct helper, a working breadth-first search for a maze, a working backtracking solver for Sudoku, a working primality sieve, with a broken interface around it. The question parser misses a marker and never calls the helper at all. The tool message is malformed and the call fails to execute. Or the final answer-extraction step returns an incomplete result, even though the helper computed the right thing internally.

Composition rewards, an attempt to directly pay the proposer for structural composition rather than just hoping a helper directive produces it, make the same tension visible from a different angle. Under a reward that pays for successful tool use, composed tool loops decline from 25 of 40 draws at the starting checkpoint to just 1 of 40 after further training: chasing the reward through pure prompting and reward-shaping pushed the proposer away from the very structure it was meant to encourage, likely because interpreter loops were simply an easier, more reliable way to earn task credit. A combined reward, adding a smaller structural bonus on top, retains more composed loops, 16 of 40 at one checkpoint, but its round-five development score still lands below plain Single-step RL. Composition is not free: paying directly for a particular shape of harness can trade away some of the task performance that shape was supposed to protect.

In multi-hop QA, the structural audit tells a parallel story about what training preserves, rather than what it invents. Base, the untrained proposer, often collapses the seed's two-hop retrieval down to a single solver call, or removes retrieval and the solver's use of it almost entirely, a simpler harness that is easier to write but throws away the retrieval machinery doing the real work. Reinforcement learning, by contrast, consistently keeps and extends multi-hop retrieval: Figure 6e's pooled structural counts over ten rounds and three benchmarks show RL-trained harnesses overwhelmingly retaining a passages-based or summaries-based multi-hop structure, where Base skews much further toward a single call or no solver use at all.

Every chapter so far has shown something getting better. This one is different on purpose, because the paper itself reports an answer that is not a clean win, and a lesson built only from wins would be dishonest about what the paper actually found.

Chapters 4 and 6 described two training recipes. Single-step RL treats every training context, a parent harness plus a report, as independent: the proposer never sees, during training, what its own edit from the previous context led to next. Multistep RL instead builds training contexts from chains: a harness that a training-time revision run already advanced, with a report describing what that specific chain's history looks like so far. The hope is that training on these richer, chain-shaped contexts would make the proposer better specifically at the thing test-time adaptation actually does, revise a harness that already carries several rounds of its own history.

The reward update itself does not change between the two: each context, chained or not, still uses the same immediate, single-revision reward from Chapter 4. What changes is only the distribution of contexts the proposer practices on.

Here is what the numbers actually show, evaluation by evaluation. On Reasoning Gym's canonical questions, after five rounds, Multistep RL finishes at 0.701 and Single-step RL at 0.699: essentially tied, Multistep a hair ahead. On format-varied questions the ranking flips hard: Single-step RL reaches 0.725, Multistep RL only 0.649, a real gap in the other direction. In multi-hop QA's independent single-revision proposals (Table 23), Multistep RL edges ahead on HotpotQA, mean exact match 0.413 against 0.411, and on MuSiQue, 0.155 against 0.147, gaps of only a point or two either way. On 2WikiMultihopQA the pattern flips again: Single-step RL reaches 0.309, Multistep RL only 0.275, a clearer gap favoring the simpler recipe. Four evaluations, four different margins, and the winner changes each time: that is not what a reliable improvement looks like.

The paper runs one more check, aimed at a narrower question: does training on chains help merely because it lets the proposer see later-round states, or does it need the whole business of building those chains online, step by step, during training? To separate the two, it builds a fixed pool of 465 training contexts straight from already-recorded revision sequences: 240 taken from a harness's first round, 155 from later rounds, and 70 stuck at the unrevised seed. Training on this fixed, pre-recorded mixture reaches a round-five development score of 0.615, against Single-step RL's 0.660, a paired difference of about −0.045. Paired difference here just means this: for each of twelve task families, subtract one method's score from the other's, then average those twelve differences. The paper reports that average alongside one standard error, a plain measure of how much the twelve per-family differences themselves varied, so a reader can judge whether −0.045 is a real gap or just noise. Here the standard error is about 0.048, nearly as large as the difference itself, so this result does not show that merely seeing later-round states, without the online chain-building machinery, helps either.

Put plainly: training on chains of revisions, rather than single independent ones, does not reliably produce a better test-time adapter in this paper's experiments. Sometimes it helps a little, sometimes it hurts, and the direction depends on the exact evaluation. What is not in question is whether iterative improvement itself works. It does: Chapters 7 and 8 already showed proposers trained purely on independent, single-step revisions still keep improving harnesses over five and ten rounds at test time. The ability to iterate seems to come from somewhere else, perhaps simply from being a competent, well-trained reviser in general, not specifically from having practiced on revision chains during training.

Harness Learning Enables Generalizable Test-Time Adaptation was written by Alvin Zhang, Xuecheng Liu, Zixuan Wang, Fahim Tajwar, Daman Arora, Ruslan Salakhutdinov, Daniel Khashabi, Yuda Song, and Andrea Zanette, published as arXiv:2609.35738 on 28 September 2026. The work was carried out with compute support from Johns Hopkins, Carnegie Mellon's FLAME center and Babel cluster, Oak Ridge and Argonne's leadership computing facilities, and several NSF grants.

One paragraph, the paper's own claim restated. A small proposer model is trained with reinforcement learning on the outcomes of its own harness edits. It learns a revision rule that transfers to task families it never trained on. It keeps improving a harness over several rounds at test time, with no parameter update at all. On Reasoning Gym, that proposer's average revision beats a teacher nine times its size. On multi-hop QA, it carries a skill learned purely on HotpotQA over to two benchmarks it never saw revised.

Here is the core loop as pseudocode, the shape every earlier chapter built one piece of:

h = seed_harness
for t in range(T):
    report = execute(h, Q_fb)                  # Chapter 0, 6
    x = (task_description, h.source, report)    # Chapter 2
    candidates = [propose(x) for _ in range(G)]  # proposer samples G edits, Ch. 2, 4
    scored = [(c, score(c, Q_score)) for c in candidates if applies(c)]
    h_new = select_best(h, scored)               # Select: Ch. 2 rule, task-specific
    h = h_new                                    # ratchet (Reasoning Gym) or best-of-G (QA)
# proposer weights theta are NEVER updated inside this loop
final_score = score(h, Q_eval)                   # measured once, Ch. 3

This paper sits inside a fast-moving cluster of work on learned harness revision, and this site already covers several of its neighbors. Harness engineering and harness optimization cover the hand-tuned and search-based ancestors of this idea: writing and improving the program around a model before anyone trained a dedicated reviser for the job. Self-improving harnesses surveys the broader family this paper belongs to, methods that change an agent's own scaffold from feedback. RRSI studies a closely related failure mode directly: an agent that rewrites its own harness can overfit to the tasks it practices on, and proposes a regularizer against exactly that; harness learning's three-way question split (Chapter 3) and frozen-weights design are a different answer to a similar worry. Prime Agent and Meta-Harness are earlier attempts at end-to-end harness optimization that this paper's related-work section places itself against directly. Agentic harness engineering gives the wider practice of building these programs by hand, the very skill this paper tries to automate. And GRPO and DAPO is where Chapter 4's optimizer comes from in more depth, if the clipped surrogate objective and its variants are worth a closer look.

The paper is just as direct about what it has not shown. Its Reasoning Gym supervision comes from exactly one 35B teacher, so the range of harness designs the proposer ever sees during SFT is whatever that one teacher happened to favor, mostly interpreter loops; a different or stronger teacher might unlock different habits. Training uses prescribed revision prompts and a fixed, pre-written report format; the proposer never chooses, on its own, to go inspect the code or run a targeted test before writing its edit, the way a human debugging a harness would. And the solver stays frozen throughout every experiment here, so this paper does not test whether training the proposer and the solver together, letting harness improvements and task-solving improvements shape each other, would work better or worse. The authors point to exactly this as a natural next step, alongside longer, more agentic tool-use trajectories for the proposer itself.

Open questions worth sitting with: would crediting an entire revision chain, rather than each step's immediate reward, fix Chapter 10's mixed result? Does the advantage over the teacher hold up if the teacher itself is allowed to iterate across rounds, not just propose once? And at what proposer size does this skill, editing code from execution feedback, stop being learnable at all?