Self-Harness: Harnesses That Improve Themselves

Chapter 1: The Harness as a Formal Object

Before a system can edit its own harness, the harness has to be a thing — a named, bounded, versioned object, not a vague cloud of "the stuff around the model." This chapter pins that down, because every guarantee in the rest of the paper rests on the pin.

What is inside the harness

The paper's working definition: the harness is the non-parametric scaffolding that governs how a fixed language model is deployed as an agent. Concretely, it includes:

Notice what is not in the table: the model's weights, the decoding configuration, the evaluator, the benchmark environment. Those are frozen. The harness is exactly the layer you can change without touching any of them.

The run, formally

The paper's formalism is small enough to hold in one hand, and each symbol earns its place. Let M be the fixed language model — think of it as an employee whose skills you cannot change. Let h be the harness — the employee's standard operating procedure: the checklists, the tools on the bench, the rules about when to double-check. Given a task instance x, running M under h produces two things: an execution trace τ (tau — the full record of messages, tool calls, and intermediate results, like a flight recorder) and a final output y. An evaluator E — the benchmark's own verifier, an authority neither the model nor the harness can edit — maps the triple (x, τ, y) to a behavioral outcome z: pass or fail.

The lineage: how self-improvement becomes attributable

Self-Harness never edits a harness in place. It operates over a lineage — a sequence h₀, h₁, h₂, … where each transition is one bounded edit to the execution protocol. Three properties of this setup do all the scientific work:

First, attribution. The model M and evaluator E are held fixed across the entire lineage. So when the pass rate changes between h_t and h_t+1, there is exactly one place the change can have come from: the harness edit. No confound with "the model got smarter" or "the grader got easier." This is the same logic as a controlled experiment — freeze everything but one variable.

Second, boundedness. Each transition is a small edit to declared surfaces, not a rewrite of the control architecture. That keeps every step in the lineage individually reviewable: you can read the diff between h₃ and h₄ the way you read a pull request.

Third, reversibility. Because the lineage is explicit and every candidate is logged with its evaluation results, a bad direction can be abandoned by simply not promoting it. Rejected candidates never become the active harness — they remain recorded branches, evidence of what was tried.

A worked contrast: model failure versus harness failure

Take a concrete task from the paper's world: extract structured data from a file and write the result to /app/answer.txt, where a hidden verifier will read it. Two runs fail. Run one: the agent computes the wrong count because it misread a date format — a genuine reasoning error. Run two: the agent computes the correct count, prints it to the console, says "task complete," and never writes the file. The verifier finds no artifact and fails the run.

Both runs score zero. But they are different diseases. Run one might genuinely need a better model. Run two needs one sentence added to the harness: "identify the required output artifact and create an initial version early." In Chapter 8 you will see that exact sentence get discovered by the loop, for the exact model that had this exact disease. The whole bet of harness engineering — manual or automatic — is that a surprisingly large fraction of observed failures are of the second kind.

Chapter 2: Stage 1 — Weakness Mining

The loop's first stage answers a deceptively hard question: given a pile of failed runs, what actually went wrong? Not per-run — per mechanism. A harness edit is a rule that applies to every future run, so it must target a recurring behavioral pattern, not one unlucky trajectory. Weakness Mining is the machinery that turns raw failures into that kind of evidence.

From records to failure signatures

At round t, the model runs under the current harness h_t on the held-in task split. Every task instance x_i yields a record r_i = (task, trace, output, outcome) — what was asked, what happened, what was produced, and the verifier's verdict. Collect the failures: F_t is the set of records where the verifier said fail.

Now the key move. Each failed record gets a failure signature — a triple φ(r) = (c, q, m), where each slot answers one question:

Failures are then clustered by exact agreement of this signature: C_φ is the set of failed records whose triple matches φ exactly. Two runs land in the same cluster only when they agree on all three slots — same verifier rejection, same causal relationship, same underlying mechanism.

Why exact-match clustering, not semantic similarity

This is the design decision worth slowing down for. The obvious way to cluster failures would be embeddings: encode each trace, group by cosine similarity, done. The paper explicitly rejects that, and the reason is a small parable about what clustering is for.

Consider two runs that both end in a timeout. Semantically, their traces look alike — both are long, both are full of shell commands, both end with the same verifier message. But suppose run A timed out because the agent kept re-running a failing install command, while run B timed out because it launched a legitimate long computation and never checked back. These need different harness edits: run A needs retry discipline; run B needs background-execution guidance or a progress check. An embedding cluster would happily merge them — same symptom, same vocabulary — and the proposer would then write one blurry patch for two diseases.

Signature clustering keeps them apart, because their m slots differ (unbounded retry versus unmonitored long job) even though their c slots agree (timeout). The goal, in the paper's words, is not to discover latent semantic similarity among traces — it is to aggregate failures that plausibly admit the same harness-level intervention. The unit of clustering is the fix, not the vocabulary.

A tiny worked clustering

Run the definition by hand on five failed traces, exactly the way the evaluation system does:

five failed records — assign each a signature (c, q, m)r1: verifier: answer.txt missing | agent printed answer to console, never wrote file
    φ(r1) = (artifact-missing, direct-cause, skipped-artifact-write)
r2: verifier: answer.txt missing | agent wrote file, then deleted it in a cleanup step
    φ(r2) = (artifact-missing, direct-cause, destroyed-own-deliverable)
r3: verifier: timeout            | agent re-ran failing pip install 9 times
    φ(r3) = (timeout, direct-cause, unbounded-retry)
r4: verifier: timeout            | agent re-ran failing apt-get install 7 times
    φ(r4) = (timeout, direct-cause, unbounded-retry)
r5: verifier: answer.txt missing | agent printed answer to console, never wrote file
    φ(r5) = (artifact-missing, direct-cause, skipped-artifact-write)

cluster by EXACT signature agreementC_a = {r1, r5}   # skipped-artifact-write     — fix: create the artifact early, verify before concluding
C_b = {r2}       # destroyed-own-deliverable  — fix: protect verified outputs from cleanup
C_c = {r3, r4}   # unbounded-retry            — fix: retry discipline (stop after N identical failures)
# note: r1/r2/r5 share a verifier symptom (artifact-missing) but r2 is NOT merged with r1/r5 —
# its mechanism differs, and so does the harness edit it needs. three clusters, three distinct fixes.

A semantic clusterer would almost certainly have produced two groups here — "missing file failures" and "timeout failures" — and the missing-file group would have hidden two different diseases behind one symptom. The signature triple produced three groups, and each group maps to exactly one candidate intervention. That mapping is the entire point of the stage.

Ordering, and what the bundle deliberately withholds

Clusters are then ordered by support (how many failures share the signature) and estimated actionability (how plausibly a harness surface could address the mechanism), so the proposer sees the recurring, fixable patterns first. For each cluster, the evaluation system packages a structured failure pattern: cluster size, representative task instances, shared trace symptoms, verifier evidence, and the inferred agent mechanism. The collection of these is the round's evidence bundle B_t.

And here is the boundary that keeps the architecture clean: the bundle never prescribes an edit. It says "eleven runs failed because the agent retried failing commands without bound" — it does not say "add a retry cap of three." Diagnosis and treatment are separated on purpose: the evaluation system owns what is wrong, the proposer (Chapter 3) owns what to change, and the gate (Chapter 4) owns whether the change survives. Three stages, three distinct authorities, no stage grading its own work.

Chapter 3: Stage 2 — Harness Proposal

The evidence bundle says what keeps going wrong. Someone now has to decide what to change. In Self-Harness, that someone is the same fixed model, re-invoked under its current harness in a proposer role. This chapter is about the constraints wrapped around that invocation — because an unconstrained proposer is exactly the self-modifying system Chapter 0 told you to worry about.

What the proposer is allowed to see

The proposer does not get raw logs, and it does not get free rein. It gets a bounded proposal context with four ingredients:

Structured cross-case evidence instead of raw logs is not just a token-budget economy. It shapes what kind of thinking the proposer does: given eleven pre-clustered instances of "unbounded retry," it reasons about the mechanism; given eleven raw traces, it would reason about eleven anecdotes.

K parallel proposals: diversity across, minimality within

From one evidence bundle, the proposer generates K mutually distinct proposal bundles. Each bundle is a pair: the edit Δ_j itself — a function that maps the current harness h_t to a candidate harness h_t^(j) = Δ_j(h_t) — and an audit record a_j stating the targeted failure pattern, the edited harness surface, the expected behavioral effect, and the regression risks the proposer itself anticipates.

Two opposing forces govern this set, and the tension between them is the design:

Diversity is enforced across branches. The K candidates must be materially distinct — not the same idea in different wording. One branch may target a different failure mechanism; another may target the same mechanism through a different surface (a prompt rule versus a runtime policy); another may carry a different hypothesis entirely about what would help. Parallel distinct branches means the round explores several regions of harness space at once, and Chapter 4's gate can pick winners empirically instead of the proposer having to pre-commit to its single best guess.

Minimality is enforced within each branch. Every individual edit must modify only the surface needed for its chosen mechanism, preserve unrelated behavior, and never rewrite the overall control architecture. Minimality is what keeps the lineage auditable (a small diff can be reviewed and reverted) and what keeps attribution sharp (when a candidate improves the pass rate, you know which idea earned it, because the candidate contained only one idea).

The addressability filter: knowing what not to patch

Not every failure cluster deserves a proposal, and the discipline to skip some is one of the stage's quieter strengths. A cluster is a suitable target only if it is both supported by evidence and plausibly addressable by an editable surface. Some clusters fail the second test: a group of failures on genuinely hard tasks reflects task difficulty; a cluster of near-random outcomes reflects evaluation noise; a cluster where the model simply lacks the knowledge reflects a capability ceiling. None of those is a missing execution rule, and no prompt sentence or runtime policy will fix them.

The proposer is instructed to exclude such clusters rather than force them into a patch. This matters because a patch aimed at an unaddressable failure is not merely useless — it is a new rule every future run must carry, a fresh opportunity for regressions, and noise in the lineage. The best proposal for some diseases is no proposal.

What a proposal bundle looks like, concretely

one of K proposal bundles for the "unbounded-retry" cluster (illustrative shape)Δ_2: edit surface = failure_recovery_instruction
  before: "If a tool call fails, inspect the error and adapt; do not blindly retry the same action."
  after:  "If a tool call fails, inspect the error and adapt. If the SAME command has failed twice,
           do not run it again — change the approach, or check the missing dependency it implies."

audit record a_2:
  targeted pattern:  unbounded-retry (11 runs, timeout, direct-cause)
  edited surface:    failure_recovery_instruction (one declared surface, nothing else touched)
  expected effect:   repeated-failure loops end by the third attempt; time budget redirected
  regression risks:  tasks where a retry legitimately succeeds on attempt 3+ may now be abandoned

Read the last line again. The proposer is required to name the ways its own edit could backfire. That declared risk does not gate anything by itself — but it makes the branch interpretable when the regression test comes back, and it is the habit that separates an audited lineage from a pile of diffs.

Chapter 4: Stage 3 — The Regression Gate

Here is the stage that makes the whole loop trustworthy. A proposed edit, however well-argued its audit record, is just a hypothesis. Self-Harness treats it exactly the way an experimentalist treats a hypothesis: run it, measure it, and let a pre-committed rule — not the proposer's enthusiasm — decide its fate.

Two splits, two different jobs

Before the loop ever starts, the task set is partitioned once and frozen: a held-in split D_in and a held-out split D_ho. The two splits play different roles, and the difference is the intellectual core of the stage.

The held-in split is where evidence comes from: its traces, verifier outcomes, and failure clusters are what the proposer saw. So held-in performance answers: did the edit fix what it claimed to fix? The held-out split is never shown to the proposer — its traces are invisible, its failures unmined. Held-out performance therefore answers a question the proposer could not have optimized for: did the edit preserve behaviors it never knew about? It is a regression test in the classic software sense, and it is the loop's defense against the oldest failure of self-modification: fixing the case in front of you by quietly breaking the cases you cannot see.

The acceptance rule

For candidate h_t^(j), evaluate both it and the current harness h_t on both splits. Let P_in(h) and P_ho(h) be the pass counts, and define the split-wise improvements Δ_in = P_in(candidate) − P_in(current) and Δ_ho likewise. The candidate is accepted if and only if:

Work it by hand on the paper's SWE-bench split (67 held-in tasks, 33 held-out):

gate arithmetic — three candidates, current harness passes 30 held-in, 14 held-outcandidate A: held-in 30 → 32, held-out 14 → 14
  Δin = +2, Δho = 0     → +2 ≥ 0 ✓   0 ≥ 0 ✓   max(2,0) > 0 ✓   → ACCEPT
candidate B: held-in 30 → 33, held-out 14 → 13
  Δin = +3, Δho = -1    → -1 ≥ 0 ✗                            → REJECT
  # note: total passes went 44 → 46. the gate rejects it ANYWAY — a held-out regression
  # means the edit broke behavior the proposer never saw. net wins do not excuse that.
candidate C: held-in 30 → 30, held-out 14 → 16
  Δin = 0, Δho = +2     → ACCEPT  # pure held-out gains count: the edit generalized

Candidate B is the one to internalize. A rule that merely demanded "more total passes" would promote it — and would thereby teach the loop that trading unseen behavior for seen behavior is acceptable. Over many rounds, that incentive compounds into a harness overfitted to the mined failures. The conservative rule refuses the trade outright: no promotion may purchase held-in gains with held-out losses, ever. The paper is explicit that proposals trading one split against the other are rejected even if their total pass count increases.

Noise, merging, and the audit trail

Stochasticity. Agent evaluation is noisy — the same harness can pass a task on one attempt and fail it on the next. The gate's answer is repetition: candidate evaluations are repeated and the acceptance rule is applied to aggregate pass counts across repeats, so a single lucky run cannot promote an edit. (The paper's headline numbers use two repeated attempts per configuration.)

Merging. If several compatible candidates pass the gate in the same round, their edits are merged into the next harness h_t+1. If none pass, h_t+1 = h_t — the loop is perfectly content to change nothing, which is itself a safety property: no round is obligated to ship an edit.

The ledger. Validation also rejects degenerate proposals mechanically — ones that touch no editable surface, or that crash before producing a valid evaluation. And for every candidate, accepted or not, the system records the changed surfaces, split-wise outcomes, evaluation repeats, proposal summary, and the decision. Every transition in the lineage is thereby auditable after the fact: you can reconstruct not just what the harness became, but what it declined to become, and why.

Chapter 5: The Minimal Seed

Every lineage starts somewhere. The choice of h₀ — the seed harness — is a methodological decision disguised as an implementation detail, and the paper gets it right in a way worth studying: the seed is deliberately minimal.

Why minimal

Imagine seeding the loop with a rich, hand-tuned harness full of clever rules. Every subsequent measurement is now contaminated: when the final harness scores well, how much came from the loop and how much from the seed's smuggled-in human engineering? A minimal seed makes the attribution clean — essentially everything the final harness knows, the loop discovered from measured rollouts. The gains in Chapter 6 are gains the process earned.

The actual seed, line by line

The initial harness builds on the DeepAgent SDK and consists of a short benchmark-facing system prompt, the default filesystem and shell tools, and — this is the structurally important part — a set of declared editable surfaces, each a small Python builder function in one configuration file. Self-Harness may change only this file. Here is the seed, condensed to its actual content:

the seed harness — every editable surface is a declared builder functiondef build_system_prompt():
    return """You are running inside a Terminal Bench 2 Harbor task environment.
    Use the built-in filesystem and shell tools to inspect the workspace, make
    concrete edits, and verify outcomes against the actual task environment.
    Do not assume synthetic datasets, domain-specific tools, or hidden fixtures
    unless you discover them in the repo or runtime."""

def build_memory_sources():   return ["/AGENTS.md"]
def build_subagents():        return []          # none. the loop may add them
def build_skills():           return []          # none. the loop may add them

def build_bootstrap_instruction():
    return "Start by inspecting the workspace and identifying the smallest relevant edit surface."
def build_execution_instruction():
    return "Prefer concrete repo changes over generic advice, and keep edits tightly scoped to the task."
def build_verification_instruction():
    return "Before concluding, verify the result with the most targeted command, file read, or test you can run."
def build_failure_recovery_instruction():
    return "If a tool call fails, inspect the error and adapt; do not blindly retry the same action."

def build_runtime_control_policy():
    return { "enabled": False,               # the whole policy is OFF at seed
             "max_recent_tool_errors": None,    # no error cap
             "max_total_tool_messages": None,   # no loop bound
             "instruction": None }

Read the seed as a list of absences. No subagents. No skills. No runtime limits — the control policy exists as a surface but ships disabled, with every field None. The verification instruction is one generic sentence. Each absence is a question posed to the loop: will you discover that you need this? And Chapter 8 shows the answers arriving, model by model: MiniMax's loop switches the runtime policy on and sets a tool-message cap; Qwen's loop grows a dependency-verifier skill and an artifact-ensure subagent; GLM's loop rewrites the execution instruction around environment persistence. The seed's empty slots are the experiment.

The edit, as the system sees it

Because every surface is a builder function, a harness edit is literally a diff to this file. Here is the shape of an accepted MiniMax edit from Chapter 8, shown the way the lineage records it:

an accepted edit = a reviewable diff to declared surfacesdef build_bootstrap_instruction():
-   return "Start by inspecting the workspace and identifying the smallest relevant edit surface."
+   return "Start by identifying the required output artifact and creating an initial
+           version of it as early as possible; then refine it against the task environment."

def build_runtime_control_policy():
-   return { "enabled": False, "max_total_tool_messages": None, ... }
+   return { "enabled": True,  "max_total_tool_messages": 50,
+            "instruction": "You have used many tool calls; consolidate progress and
+             move toward producing the required artifact." }

Two surfaces, a handful of lines, a fully reviewable change. Compare this to "the optimizer adjusted the agent somehow" — the file-level representation is what makes the lineage of Chapter 1 more than a metaphor.

Chapter 6: Nine Combinations, Nine Wins

The machine is built. Now the measurement. The paper instantiates Self-Harness across a 3×3 grid: three base models from three different families — MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 — on three benchmarks that stress three different agent muscles.

Everything else is frozen per combination: decoding configuration, budget, tool set, environment, evaluator. Within each cell of the grid, the only thing that varies between "initial" and "final" is the harness lineage. The metric is Pass (%): the share of task attempts passing the official verifier, over two repeated attempts.

The headline table, in numbers

Four readings of one table

Reading one: universality. All nine model–benchmark combinations improve, and they improve on both splits. Not "usually," not "on average" — nine for nine, with no promoted harness degrading either split. That last clause is the gate's fingerprint: the acceptance rule made split-degrading promotions impossible by construction, and the final table confirms the construction held.

Reading two: the weakest model gains the most. Work one relative gain by hand. Qwen3.5 on Terminal-Bench held-in: initial 15.1, final 36.0. The gain is 36.0 − 15.1 = 20.9 points; relative gain is 20.9 / 15.1 = 1.384, i.e. +138% — the pass rate more than doubled. Qwen posts triple-digit relative gains on all three benchmarks (+104%, +113%, +132% overall). The intuition: a strong model already routes around a bad harness some of the time; a weaker one falls into every hole the harness leaves open, so filling the holes helps it most. Harness quality and model quality are partial substitutes.

Reading three: held-out sometimes gains more than held-in. In four of the nine combinations, the relative held-out gain exceeds the held-in gain — MiniMax and GLM on Terminal-Bench (+53% vs +16%; +33% vs +20%), MiniMax and Qwen3.5 on SWE-bench (+18% vs +13%; +116% vs +111%). Pause on how odd that should feel: the edits were mined exclusively from held-in failures, yet they help the never-seen split at least as much. That is what you would expect if the edits captured reusable execution mechanisms ("create the artifact early," "stop retrying") rather than memorized patches for specific tasks. It is the single strongest piece of evidence that Weakness Mining's mechanism-level clustering did its job.

Reading four: the ceiling case. GLM-5 on AppWorld: overall 44.4 → 85.0, an absolute jump of +40.6 points (+91%), with held-in reaching 92.2. One model–benchmark pairing had a huge amount of latent capability locked behind harness problems — pagination that stopped early, completion semantics that misfired — and unlocking them nearly doubled the score. Chapter 8 shows exactly which edits did it.

Chapter 7: Evolution Trajectories

Aggregate numbers hide the texture of the process. The paper's trajectory figures show every candidate evaluation in order — accepted edits climbing the pass rate, rejected branches going nowhere, dead ends marked and abandoned. Reading them is the closest you can get to watching the loop think.

Reading the four runs

Qwen3.5 on Terminal-Bench (18.0 → 36.7). The longest climb, and the one that most rewards a slow read. The retained edits, in the order the loop found them: an artifact-ensure subagent for late deliverables; a create-within-2-steps rule for missing files; a dependency-verifier skill for skipped imports; a use-correct-content-tags rule for schema-invalid tool content; a middleware guard triggered by tool errors; and a force-a-change loop breaker for endless exploration. Notice the surfaces: a subagent, a skill, a middleware, and prompt rules — the loop used four different kinds of editable surface, not just prompt text. The structural mechanisms (subagent creation, middleware) are the paper's evidence that Self-Harness can go beyond local failure repair into reorganizing how problem solving is structured.

MiniMax M2.5 on Terminal-Bench (42.2 → 53.9). A shorter run with a compounding finale: create output early, redirect after 50 tool calls, precheck imports — and then a combined harness merging the artifact, schema, and loop edits into one promoted configuration. Merging is the gate's plural form at work: compatible accepted candidates fold together rather than competing.

MiniMax M2.5 on SWE-bench Verified (46.0 → 52.5 overall). The repair-benchmark run converges on verification discipline: a diff detector that catches empty patches (an agent can end a run having edited nothing — the detector makes that impossible to miss), null-completion handling, local verification, and local-test enforcement — run the repository's own tests before declaring the patch done. Every retained edit here is a version of the same idea: do not let the agent claim success on evidence weaker than the verifier's.

GLM-5 on AppWorld (44.4 → 85.0). The steepest curve, driven by API-workflow mechanics: completion + pagination handling, pagination exhaustion (fetch all pages of a record list before acting — a partial list silently corrupts every downstream decision), and a progress-check reminder against turn-budget exhaustion. The lesson in miniature: what looked like a weak model on AppWorld was substantially a harness that let it act on incomplete state.

The rejected branches are load-bearing

Every run's figure shows gray crosses — candidates that were proposed, fully evaluated, and refused. This is the system working, not failing, in three distinct ways. First, rejections are how the gate expresses its selectivity: the loop explores K branches per round precisely so that most can lose. Second, rejected branches feed the next round's proposal context ("previously attempted edits"), so the proposer does not circle back to dead ends. Third, the red branch endpoints mark whole directions abandoned after evaluation — the lineage equivalent of a pruned search tree. A trajectory with no rejections would be a red flag: it would mean the gate was letting everything through.

Chapter 8: Model-Specific Medicine

The core claim of the whole paper — harness design is inherently model-specific — only becomes believable when you look at what the loop actually prescribed for each model on the same benchmark. This chapter is that comparison, plus the trace-level before-and-after stories that show the prescriptions working.

MiniMax M2.5: the wanderer

MiniMax's characteristic failure was open-endedness: long dataset exploration, structured tool output handled sloppily, and required artifacts arriving late or never. Its retained edits attack all three. The bootstrap instruction was rewritten from the seed's "identify the smallest relevant edit surface" to "identify the required output artifact and create an initial version as early as possible" — a priority inversion that front-loads the deliverable. The runtime control policy, disabled in the seed, was switched on with a cap on total tool messages, so prolonged tool use triggers redirection instead of continuing indefinitely.

The paper's trace pair on the count-dataset-tokens task shows the change in behavior: under the initial harness, the run ends after long exploration with no answer artifact at all; under the edited harness, the agent identifies the relevant metadata split, computes the required count, writes /app/answer.txt, and — the detail worth savoring — reads the file back before stopping. That final read-back is verification discipline the seed asked for generically and the evolved harness made specific and habitual.

Qwen3.5: the thrasher

Qwen's disease was destructive persistence. The paper's representative trace is worth retelling in full because it is the single best argument for harness-level intervention in the whole paper. The task requires an extractor script whose output the verifier will check. Under the initial harness: the agent creates extract.js — a good start — then hits an overwrite failure while editing it, retries the edit, fails again, retries again, and finally, in a doomed cleanup attempt, deletes /app/extract.js and stops. The verifier finds nothing on disk. The agent did not lack the ability to write the extractor — it demonstrably wrote it — it lacked a rule for what to do when editing goes wrong.

The promoted harness adds exactly that rule set: dependency prechecking, command-retry discipline, loop breaking, and — the decisive one here — a tool-error-triggered redirection toward the missing artifact. Same model, same task, after the edits: the agent recreates the extractor after the failure, fixes the parsing logic, writes the output, validates the JSON with a targeted check, and leaves the artifact in place. Zero weight updates. One failure-recovery rule, discovered from the model's own trace history.

GLM-5: the amnesiac

GLM's failures were about state that would not stay put: tools installed in one shell command silently absent in the next (each command ran in a fresh session), plus a tendency to linger in exploration while the turn budget drained. Its retained edits instruct the agent to make environment changes persist across shell sessions, to verify tool accessibility after modifying the environment, and — a genuinely strategic rule — to transition from exploration to implementation when exploration has stopped producing artifacts. In the build-task trace pair, the initial harness burns its budget on long external downloads and then rationalizes failed sanity checks; the edited harness pivots on timeout evidence, validates alternative sources early, repairs the failing render check, and only then finalizes.

The comparison, compressed

If one universal harness were possible, these three columns would converge. They do not — they barely overlap. And note the row the table cannot show: applying MiniMax's "cap at 50 tool messages" to a model that thinks in long careful chains could easily hurt it. The prescriptions are not just different; they are plausibly anti-correlated across models, which is the strongest form of the model-specificity claim.

Chapter 9: Self vs External — The Design Space

Self-Harness did not arrive alone. Within months of each other, several groups converged on "the harness is a learnable surface" and diverged on everything else. Mapping the neighbors is where the design lessons crystallize — especially the comparison with AHE, which answers the same safety question with the opposite mechanism.

The neighborhood

Self-Harness versus AHE: two answers to one safety question

The question both papers face: how do you stop self-modification from quietly breaking things? Their answers are near-perfect complements.

AHE answers with falsifiable predictions. Every edit its Evolve Agent ships must declare, in advance, which tasks it expects to fix and which it puts at risk. The next round's results grade the prediction, and edits that did not deliver are reverted. This is scientific-method machinery — edits as falsifiable contracts. But AHE's own measurements expose the weak half: its evolve model's fix predictions are genuinely informative (precision and recall roughly five times the random baseline), while its regression predictions hover near chance — regression recall around 11%, barely twice random. The agent can say what an edit will fix; it largely cannot foresee what the same edit will break. AHE names this "regression blindness" and calls closing it the clearest direction for future loops.

Self-Harness answers with a gate that requires no foresight at all. It never asks the proposer to predict regressions — it measures them, on a held-out split the proposer has never seen, before any promotion happens. Regression blindness stops mattering at promotion time when regressions are empirically caught at the gate. The cost is compute: every one of the K candidates must be fully evaluated on both splits every round, which is far more expensive than AHE's ship-then-verify cycle. The trade is prediction cheapness against measurement certainty.

The frontier argument, revisited

Meta-Harness-style external optimization works when a stronger supervisor exists. Self-Harness's closing argument is about the regime where none does. If the strongest available model is the one being deployed, its harness can only be tuned by a human, or by itself. Humans do not scale with release cadence; Self-Harness is the existence proof for the alternative — and its nine-for-nine result says the alternative is not merely possible but reliable, at least under benchmark verifiers. The honest caveat lives in that last clause, and Chapter 10 takes it up: everything rests on the verifier's authority.

One more neighbor deserves a sentence: ACE evolves what the model reads (an in-context playbook), while Self-Harness evolves the machinery the model runs inside. AHE's own comparison found that prompt-layer self-evolution missed the components carrying its gains (tools, middleware, memory) — and Self-Harness's structural discoveries (subagents, middleware) land in exactly the layers a context-only method cannot reach. The two approaches are less rivals than different floors of the same building.

Chapter 10: Limits, Connections & Cheat Sheet

The paper closes with its own boundary-drawing, and the boundaries are as instructive as the results.

Four honest limits

Bounded edits are not open-ended self-improvement. Self-Harness studies small, declared-surface edits under fixed benchmarks. It is a controlled protocol, not a system rewriting its own architecture — and the authors frame that narrowness as the point: establish the controlled result first.

Benchmark-shaped edits. Accepted edits may still reflect benchmark-specific failure patterns. "Create the artifact early" looks general; whether it helps on tasks with no artifact convention is unmeasured. The held-out split protects against overfitting to specific tasks, not against overfitting to the benchmark's overall shape — both splits come from the same distribution.

Everything rests on the verifier. The loop's ground truth is the evaluator's pass/fail and the fidelity of trace records. A noisy verifier feeds noise into every stage: miscounted failures, misattributed signatures, a gate making decisions on corrupted evidence. Weak verifiers are the single point of failure for the entire paradigm — which is why the benchmarks chosen all have executable, state-based verification.

The gate is calibrated to the stakes of a benchmark. Pass-rate non-regression on two splits is the right bar for terminal tasks. The authors say plainly that higher-stakes harness changes would require stronger acceptance gates — think formal invariants, safety-property tests, staged rollouts. The architecture generalizes; the specific rule does not automatically.

The cheat sheet

And the numbers to carry: nine of nine model–benchmark combinations improved on both splits; largest relative gain +132% (Qwen3.5 on AppWorld overall); largest absolute gain +40.6 points (GLM-5 on AppWorld, 44.4 → 85.0); held-out relative gains beat held-in in 4 of 9 combos — the mechanism-generalization signature.

Connections on this site

The immediate family: the AHE veanor is the observability-driven external loop this lesson's Chapter 9 contrasts against — read it next for the change-manifest/falsifiable-prediction design and the regression-blindness measurement. The ACE veanor covers the context-layer sibling (evolving playbooks, delta updates), and the MCE veanor takes the meta step of evolving the improvement mechanism itself. The AutoDesign veanor shows gated harness editing instantiated for a design task, with its own dev-split gate.

For the survey view: the Self-Improving Harnesses Gleam places this paper in the STOP→AlphaEvolve→DGM lineage (its Chapter 2 is a summary of exactly this paper — you now know the machinery underneath it). Harness Engineering builds the harness concept from zero, and Harness Optimization maps the optimization ladder this paper climbs.