Agentic Harness Engineering: The Harness That Debugs Itself

Chapter 1: Why Automating It Is Hard

The obvious plan writes itself: point a strong agent at the harness, hand it the logs, and say "improve this." Teams have tried. Three structural obstacles reliably wreck the naive version, and it is worth feeling each one concretely before seeing AHE's answer, because each pillar of AHE is a response to exactly one of them.

Obstacle 1: the action space is a junk drawer

"Edit the harness" is not one kind of action. A harness edit might be a sentence of English appended to a prompt, a Python class hooked into the agent's execution loop, a YAML block describing a tool's arguments, or a new entry in a memory file. These live in different files, different languages, different failure modes. In most agent frameworks they are also tangled: tool behavior is described inside the system prompt, retry logic is buried in framework code the agent cannot see, and a single behavior change requires touching three places that must stay consistent. An optimizing agent facing that tangle makes edits that break imports, contradict the prompt, or silently do nothing.

This is why most prior automation retreats to a single surface — usually the prompt, the one component that is plain text and safely editable. Prompt optimizers (GEPA, MIPROv2), skill libraries, and in-context playbooks (ACE) all evolve one component and leave the rest frozen. Chapter 6 shows the cost of that retreat: on this benchmark, the components that carry the gain are precisely the ones prompt-only methods cannot touch.

Obstacle 2: the evidence is buried in millions of tokens

One evaluation round of Terminal-Bench 2 in this paper produces on the order of 10 million tokens of raw trajectories — every message, every tool call, every stdout dump of every rollout of all 89 tasks. Somewhere in there is the signal: this task fails because the agent validates a proxy instead of the real deliverable; that one fails because a cleanup command deletes the verified artifact. No model reads 10M tokens and reasons reliably about them; an evolver fed raw logs either samples arbitrarily (and misses the pattern) or summarizes lossily (and hallucinates one).

Obstacle 3: edits are hard to attribute

Suppose the evolver makes four edits and the next round's score moves from 71% to 73%. Which edit did that? Did one edit gain three points while another lost one? Should any be reverted? Task-level pass rates are noisy — a task can flip on sampling randomness alone — and without attribution, an evolution loop degenerates into a random walk that keeps whatever coincided with an up-tick. Worse, an unconstrained self-modifier has shortcut edits available that reliably raise the score: weaken the verifier, raise the reasoning budget, swap in a bigger model. An automation scheme that cannot attribute cannot distinguish a real fix from a lucky roll or a hack.

Three obstacles, three pillars

AHE's architecture answers each obstacle with a matched observability layer — a discipline of recording each phase's artifacts in a structured form that another agent can read and act on:

Notice what the three pillars have in common: none of them adds intelligence. They are all plumbing — file layouts, report formats, JSON ledgers. That is the paper's engineering bet, and its most falsifiable design decision: spend the effort making the loop's state legible, and an ordinary agent becomes a competent harness engineer. The results chapters test that bet directly, because the same base model that fails to beat the seed under ACE's evolution scheme (68.9%, below the 69.7% seed) is the one that reaches 77.0% under AHE. Same intelligence; different observability; opposite outcome.

A useful analogy as you go: AHE gives the evolve agent the same three things a new human hire would need on day one of harness duty. A repo tour that shows where everything lives (component observability), a triage dashboard instead of raw logs (experience observability), and the team's decision log of what was tried and how it turned out (decision observability). The pillars look mundane for the same reason good onboarding looks mundane — and they are load-bearing for the same reason.

Chapter 2: Pillar 1 — Components as Files

Chapter 1's first obstacle was the junk-drawer action space. AHE's answer is a harness framework called NexAU, whose job is to make "the harness" stop being an idea smeared across a codebase and become a directory you can list.

Seven component types, seven mount points

NexAU exposes seven orthogonal component types as explicit files at fixed mount points in a single workspace. Everything the Evolve Agent may touch lives here, and nothing else does:

(An eighth file, ShortTermMEMORY.md, is session-scoped scratch space for the Code Agent — the Evolve Agent is explicitly forbidden from touching it.) Creating a file is not enough on its own: tools, middleware, skills, and sub-agents must also be registered in code_agent.yaml, whose entries resolve to Python via ordinary import paths — binding: tools.file_tools:read_file loads workspace/tools/file_tools/read_file.py. The config directory sits on sys.path; the harness is a small, inspectable Python package.

Why decoupling is the whole trick

The component types are deliberately loosely coupled: adding a middleware does not require editing the system prompt, and adding a skill touches no tool. That buys three properties that make the action space tractable:

The seed: deliberately, strategically bare

The starting harness H₀ — called NexAU₀ — is a single shell-execution tool. No middleware. No skills. No sub-agents. An empty memory file. Its entire system prompt is a short block of task-agnostic instructions (marked non-deletable, so evolution can append but never erase the baseline contract).

This looks like handicapping, and the scoreboard agrees at first: bare NexAU₀ scores 69.7%, two points under Codex. But the minimalism is a measurement decision, and the paper is explicit about why: a seed already fitted to Terminal-Bench would contaminate every subsequent edit's attribution. If the seed ships with a publish-guard middleware and scores rise, you cannot tell whether the evolution loop works or the seed's designers were smart. Starting bare forces every single component AHE adds to earn its place against measured rollouts — the final harness is, component by component, a list of things the loop itself discovered were worth building.

Concept + Realization: what an edit physically is

To keep "the Evolve Agent edits a component" from staying abstract, here is the actual shape of one middleware component from the evolved harness — the cross-step risk monitor from iteration 6, reconstructed from the paper's Figure 5. A middleware is a Python class whose hooks the agent loop calls at fixed points; this one runs after every tool call:

workspace/middleware/execution_risk_hints.py — new file, iteration 6class ExecutionRiskHintsMiddleware(Middleware):
    def after_tool(self, hook):
        command = hook.tool_input.get("command", "")
        output  = hook.tool_output.get("content", "")
        notes   = detect_risks(command, output, recent_history)   # 4 patterns, below
        return HookResult.with_modifications(
            tool_output=append_notes(output, notes))              # hint rides on the tool output

# the four risk patterns it watches for, and the hint each injects:
#  1. localhost-only reachability   -> "bind 0.0.0.0 not 127.0.0.1"
#  2. same error class repeats      -> "switch tactic, do not retry"
#  3. long steps stack after timeout-> "shorter probe or background"
#  4. shallow help/file validation  -> "exercise the real feature"

And registration — the part that makes it live — is one YAML entry:

code_agent.yaml — the registry the framework loadsmiddlewares:
  - import: middleware.execution_risk_hints:ExecutionRiskHintsMiddleware
    params: {}

Read what this component does as a piece of cognition rather than code: it watches the sequence of commands — something no single prompt instruction can do, because prompt text cannot observe execution history — and injects a corrective sentence into the tool output exactly when a risky pattern appears, so the model self-corrects on the very next turn. The hint arrives situated, at the moment of relevance, instead of as rule #47 in a prompt read 400 steps ago. Chapter 8 measures what that difference is worth; here, just register that "middleware" means behavior the prompt structurally cannot express.

Chapter 3: Pillar 2 — The Agent Debugger

Chapter 1's second obstacle: one evaluation round yields roughly 10 million tokens of raw trajectories, and the actionable signal — the recurring failure mechanism — is smeared across them. Experience observability is AHE's answer: an agent whose entire job is to turn that haystack into something another agent can actually consume, on the order of 10 thousand tokens of layered evidence. A thousand-fold distillation, with the raw material kept underneath for verification.

Trajectories as a navigable environment

The Agent Debugger does not receive trajectories in its prompt — they would not fit, and stuffing samples invites hallucinated patterns. Instead, AHE frames each trajectory as a file-based environment the debugger explores with shell tools: every message of every rollout lives in its own file, and the debugger navigates with the same generic tools any coding agent has — ls, grep, read. All rollouts of the same task are placed in one environment, so the debugger can diff a passing rollout against a failing rollout of the identical task side by side.

For each task, it must answer: what is the root cause of failure (or the pattern behind success)? The answer is written to a per-task analysis report, grounded with the task's pass/fail status. Then a benchmark-level overview is aggregated from all 89 reports into a single document — the entry point every later consumer starts from.

Layered, not summarized: the difference that matters

The design principle is progressive disclosure — the same principle behind agent skills. The corpus is a pyramid: the overview names the dominant failure patterns; per-task reports carry the root-cause analysis with evidence; cleaned traces (normalized, noise stripped) sit underneath; raw traces sit at the bottom. Each layer links down. The Evolve Agent reads the overview always, drills into specific task reports when a pattern needs detail, and touches raw traces only to verify a claim it doubts — the reports say what happened, but the traces remain the ground truth the reports can be audited against.

Why keep the lower layers at all, if the reports are good? Because the debugger is an LLM, and its reports can be wrong. A flat summary would make its errors invisible and irreversible — whatever the summary hallucinated becomes the loop's reality. The layered corpus makes every claim in every report checkable by anyone (agent or human) willing to read one level deeper. It is the same auditability instinct as the git history in Chapter 2, applied to evidence instead of edits.

Concept + Realization: what flows in, what flows out

Pin down the data shapes, because the pipeline is easy to hand-wave. Per iteration:

experience observability — the data flow, one iteration# IN:  k rollouts x 89 tasks; each rollout = messages, tool calls, outputs
#      ~10M tokens total, plus per-task verifier verdicts (pass/fail)

runs/iteration_N/
  input/
    analysis/
      overview.md            # ~1 doc: dominant failure patterns, entry point (READ FIRST)
      detail/{task_name}.md  # 89 reports: root cause + evidence + pass/fail grounding
    workspace/               # the harness that produced these runs (frozen snapshot)
    nexau_in_memory_tracer.cleaned.json   # normalized traces (fallback layer)
    agent/nexau.txt          # runtime logs — middleware init errors live HERE
  output/
    change_evaluation.json   # last round's per-edit verdicts (chapter 4)

# OUT: ~10K tokens the Evolve Agent actually reads on the default path

One operational detail earns its place in the evolver's standing orders: after creating or modifying middleware, it must read at least one agent/nexau.txt runtime log from a failed task, because middleware that crashes at init fails silently from the trajectory's point of view — the run just behaves as if the middleware were absent, static validation passes, and only the runtime log shows the stack trace. Observability includes observing your own instruments.

Chapter 4: Pillar 3 — The Change Manifest

Chapter 1's third obstacle: without attribution, evolution is a random walk that keeps whatever coincided with an up-tick. Decision observability is AHE's sharpest idea, and it is worth slowing down for, because it upgrades the loop from "agent makes plausible edits" to something structurally resembling science: every edit is a falsifiable prediction, and the next round is the experiment that tests it.

An edit is a JSON contract

When the Evolve Agent changes anything, it must append an entry to change_manifest.json. The schema is short enough to memorize and strict enough to matter:

change_manifest.json — one edit, one falsifiable claim{
  "iteration": 5,
  "changes": [
    {
      "id": "chg-1",
      "type": "new",                        // new | improvement | rollback
      "description": "Shell guard: block deletion of files already verified as deliverables",
      "files": ["tools/shell_tools/run_shell_command.py"],
      "failure_pattern": "post-validation state destruction",
      "predicted_fixes": ["path-tracing", "polyglot-rust-c",
                          "large-scale-text-editing", "configure-git-webserver"],
      "risk_tasks": ["cleanup-heavy-build"],   // where the guard might over-block
      "constraint_level": "tool_impl",          // which of the 7 component classes
      "why_this_component": "Prompt advice against destroying verified state already exists and is ignored under long-horizon pressure; enforcement must live at execution level"
    }
  ]
}

Look at which fields do the epistemic work. failure_pattern names the evidence (it must trace back to the debugger's corpus — edits are required to be evidence-driven, never speculative). predicted_fixes and risk_tasks are the prediction: concrete task names, not vibes. why_this_component forces the agent to defend its choice of component class — the field that operationalizes Chapter 2's "wrong level? pivot levels" rule. And type: rollback makes even undoing an edit a first-class, recorded decision.

The verdict: prediction meets delta

One round later, the loop has two facts per task: last round's pass rate and this round's. The attribution step intersects the manifest's predictions with those observed deltas, mechanically:

attribution — plain set arithmetic, no LLM required for the core checkflipped_up   = {t for t in tasks if failed_before(t) and passes_now(t)}
flipped_down = {t for t in tasks if passed_before(t) and fails_now(t)}

hits    = set(chg.predicted_fixes) & flipped_up      # promised and delivered
misses  = set(chg.predicted_fixes) - flipped_up      # promised, not delivered
harms   = flipped_down & set(chg.risk_tasks)         # foreseen damage
blind   = flipped_down - set(chg.risk_tasks)         # unforeseen damage (chapter 9!)

The verdicts feed a three-way decision, written into the next evidence corpus so the evolver confronts its own track record before editing again:

ROLLBACK + PIVOT deserves a pause, because it encodes a real epistemology: a failed edit does not falsify the failure pattern — the pattern came from evidence and is probably real. It falsifies the choice of component level for addressing it. The paper's own trajectory shows this playing out: the seed prompt already contained advice against destroying verified state, agents ignored it under pressure, and the fix that finally worked was the same idea moved from prose (advisory) to a tool-level guard (enforced). Chapter 10 tells that story task by task.

The enforcement half: what the evolver cannot do

A falsifiable contract only binds if you cannot cheat the experiment. AHE's controllability constraints close the loopholes an unconstrained self-modifier would exploit within the first hour of a 32-hour run:

Chapter 5: The Outer Loop

The three pillars are substrates; the loop is what runs on them. One AHE iteration has six phases, and the ordering of two of them is a design decision worth understanding, not an accident of implementation.

Algorithm 1 — the AHE outer loop (paper notation, annotated)# requires: seed harness H0, base model M, benchmark D, rollouts/task k, max iters N
H_best = H0
for t in 1..N:
    T_t  = ROLLOUT(M, H_prev, D, k)      # 1. k rollouts per task, full benchmark
    T~_t = CLEAN(T_t)                    # 2. normalize traces into canonical form
    if t >= 2:
        V_t    = ATTRIBUTE(C_prev, T_prev, T_t)   # 3. grade last round's manifest…
        H_prev = ROLLBACK(H_prev, V_t)            #    …and git-revert rejected edits
    R_t = AGENT_DEBUGGER(T~_t)           # 4. layered distillation (chapter 3)
    (H_t, C_t) = EVOLVE(H_prev, R_t, V_t)# 5. workspace edits + NEW manifest
    COMMIT(H_t, C_t, t)                  # 6. git-tag the iteration
    if pass1(T_t) > pass1(H_best): H_best = H_t
return H_best

Why attribution runs BEFORE distillation

Phase 3 (grade the old manifest) precedes phase 4 (build the new evidence corpus). The consequence: the per-edit verdicts land inside the corpus the Evolve Agent reads in phase 5. When the evolver sits down to edit, its own last-round predictions — graded — are part of the evidence in front of it. A manifest entry is thereby bound as a contract rather than a rationale: the agent cannot quietly forget that last round's clever middleware delivered none of its promised fixes, because the corpus it must read opens with that verdict. Rejected edits are already reverted by the time new edits are proposed, so the evolver always works from a harness whose every component has survived at least one grading.

Why k ≥ 2 rollouts per task

Running every task at least twice doubles evaluation cost, and the paper pays it deliberately, for a diagnostic reason. With one rollout, a task is pass or fail — and a fail tells you nothing about whether the harness can do it. With two, a third category appears, and it is the most valuable one: the partial-pass task, where one rollout succeeds and the other fails on the identical task. Partial passes hand the debugger a controlled experiment for free: same task, same harness, same model — compare the two trajectories, find the divergence step, and the difference between them IS the fragility. The evolver's standing orders make the goal explicit: find what the winning rollout did, and make that strategy the reliable default. pass@k gauges the capability ceiling, but pass@k is not the target — the loop optimizes pass@1, converting "sometimes works" into "works".

The bootstrap: one-shot explorers

One wrinkle at iteration 1: a fresh evolve agent knows nothing about NexAU's middleware API or what strong coding-agent harnesses look like. Two single-shot explorer agents run in parallel with the first iteration to seed that knowledge as skills: one explores the NexAU source code and writes a practical development guide (how to create and register components), the other surveys public coding-agent references. Both are written with a deliberate write-early-write-often discipline so partial completion still yields usable files. Crucially, the seeded skills receive no special protection: from iteration 2 onward the Evolve Agent may keep, refine, or delete them based on observed rollouts — even the bootstrap knowledge has to earn its keep.

The budget of one iteration

Ground the loop in wall-clock terms, because "ten iterations, 32 hours" hides where time goes. One iteration ≈ 3.2 hours, and the dominant cost is phase 1: 89 tasks × k≥2 rollouts with per-task timeouts up to an hour, run concurrently on the paper's dispatcher/sandbox infrastructure. Phases 2–6 — cleaning, attribution, distillation, evolution, commit — are minutes against that backdrop. The design consequence: evaluation is the scarce resource, which is exactly why every other phase is engineered to squeeze maximum evidence from each expensive rollout — k≥2 for partial-pass contrast, traces kept for audit, predictions graded against deltas the evaluation was already paying for. Compare the human alternative: a harness engineer also spends most of their cycle waiting on evaluation runs; AHE simply never sleeps between them, and never forgets what last round's run said.

Read the staircase's story. The four best-so-far jumps are not four lucky rolls — each is a nameable engineering idea landing at a nameable component level: iteration 2's contract-first workflow + tunable shell timeout (prompt + tool), iteration 5's publish-state guard protecting verified artifacts (prompt + tool), iteration 6's cross-step risk monitor (middleware — the class from Chapter 2's code listing), iteration 8's post-success hard-block with pre-turn risk salience (tool + middleware). Note the drift in that sequence: early wins are prose and simple tool tweaks; later wins are execution-level machinery. The loop discovers, in order, the same lesson Chapter 8's ablation will confirm by measurement — advisory text runs out of leverage, and enforcement mechanisms take over.

Also read the dips. Pass@1 per iteration is non-monotone — iterations 3, 7, and 9 land below their predecessors. The loop survives dips because H_best is tracked separately from H_t (a bad iteration cannot destroy the best harness found), and attribution + rollback claw back the specific edits responsible. Why dips happen at all — why the evolver cannot see regressions coming — is Chapter 9's subject, and it is the loop's honest open wound.

Chapter 6: Past Every Baseline

Ten iterations, one campaign, roughly 32 hours unattended. The paper's first research question is blunt: why agentic harness engineering, rather than human-engineered harnesses or other automated methods? Terminal-Bench 2's 89 tasks (4 easy, 55 medium, 30 hard, per-task timeout 1 hour) provide the arena; everything runs GPT-5.4 high.

Work the deltas by hand

Three comparisons carry the chapter, so compute them rather than skim them:

the arithmetic that matters# AHE vs its own seed — what did 10 iterations buy?
77.0 - 69.7 = +7.3 pp        # relative: 7.3 / 69.7 = 10.5% more tasks solved

# AHE vs Codex — machine loop vs the best human harness on the panel
77.0 - 71.9 = +5.1 pp        # and Codex STARTED 2.2 pp ahead of AHE's seed

# ACE vs the same seed — self-evolution is not automatically positive
68.9 - 69.7 = -0.8 pp        # the playbook actively hurt on this benchmark

That last line is the chapter's quiet shock, and it should update you more than AHE's win does. ACE — the strongest context-evolution method of its generation, the subject of its own lesson on this site — runs its full generation-reflection-curation machinery here and lands below doing nothing. Self-evolution is not a rising tide; it is an intervention at a specific layer, and if the benchmark's failure modes do not live at that layer, the machinery adds cost and noise.

And one consistency check worth doing whenever a paper reports per-tier and aggregate numbers — reconstruct the aggregate from the tiers, weighted by task counts:

does 77.0 actually follow from the tiers? (4 easy, 55 medium, 30 hard)(4 × 100.0  +  55 × 88.2  +  30 × 53.3) / 89
= (400 + 4851 + 1599) / 89
= 6850 / 89
= 76.97          # reported: 77.0 — checks out (rounding)

# and notice the weights: Medium contributes 4851/6850 = 71% of the score.
# whoever optimizes the aggregate is, in effect, optimizing Medium —
# remember this when chapter 8 explains the Hard-tier giveback.

The layer-mismatch diagnosis

Why do both self-evolve baselines underperform a loop running the same model on the same evidence? The paper traces both gaps to the same cause. ACE distills natural-language playbooks that the agent reads in-context — strategy prose riding the prompt. Training-Free GRPO distills semantic-advantage priors from trajectory comparisons — also, ultimately, guidance text. Neither opens the scaffolding itself to edits. But Terminal-Bench 2's recurring failures — destroyed artifacts, premature submission, proxy validation, runaway timeouts — are execution-control failures. The fix for "the agent deletes its verified deliverable during cleanup" is a guard that intercepts the delete, not a paragraph advising against it; the seed prompt already contained that paragraph, and agents ignored it under long-horizon pressure. Chapter 8 completes this argument with the ablation: AHE's gain concentrates in tools, middleware, and memory — exactly the three layers prompt-only methods cannot reach.

Also note what TF-GRPO's 72.3% says: trajectory-feedback methods do move the needle (+2.6 over seed) — reinforcing successful tool sequences is not worthless. The gap to AHE (+4.7 further) is the measured value of opening the remaining components to edits. Roughly: a third of the achievable gain lives in guidance, two-thirds live in structure.

Chapter 7: Frozen Harness, New Worlds

A harness evolved for 32 hours against one benchmark has an obvious failure mode: it memorized the benchmark. Research question 2 probes this the only honest way — freeze the evolved harness, change the world, re-measure, with no further evolution. Two transfers: a different task surface (SWE-bench-verified: 500 real GitHub issues across seven repositories) and four different base models.

Transfer 1: a different benchmark

Read the two columns together, because the token column is the real finding. Both prompt-layer baselines regress below the seed on success while spending 11–29% more tokens. The mechanism is structural: ACE's playbook and TF-GRPO's priors are text distilled from Terminal-Bench traces, and that text rides the prompt at every model call on the new benchmark — a per-call tax of now-irrelevant guidance that adds cost without reshaping the policy. AHE's knowledge lives in tools, middleware, and memory: consulted when triggered, costing nothing when not. Work the spread by hand: 679 − 461 = 218k tokens per trial, a 32% saving versus ACE — across 500 tasks, over a hundred billion prompt tokens of difference, from the choice of where knowledge is stored.

The repo-level pattern sharpens it: AHE's seed-relative gains concentrate on django (+1.8 on 231 tasks) and sphinx-doc (+2.3 on 44) — the largest, most token-expensive repositories, whose multi-step edit-and-verify loops match the structure AHE's components compress. Marginal regressions appear only on the three smallest repos (22–32 tasks each), where per-repo variance exceeds the per-repo gain.

To feel the scale of the token result, extend it across the run: 65k fewer tokens per trial than the seed, times 500 tasks, is roughly 32 million prompt tokens saved per full evaluation pass — versus ACE, 218k × 500 ≈ 109 million. And be precise about what this transfer does and does not prove. It proves the evolved components encode general coding-agent experience rather than Terminal-Bench trivia — the same guards and lessons help on GitHub-issue repair. It does not prove SWE-bench-optimal engineering: +0.4 over seed is modest, and a harness evolved on SWE-bench would presumably do better. The claim is portability without damage at lower cost — which is exactly the property a reusable harness needs, and the property both prompt-riding baselines fail.

Transfer 2: different base models

The harness was evolved with GPT-5.4 high in the driver's seat. Re-evaluate seed-versus-AHE on five alternate bases, harness frozen:

Every gain is positive — the harness is not one provider's idiom set. And the ordering is the interesting part: cross-family gains dominate within-family ones. The weakest base on the panel (deepseek-v4-flash) gains the most (+10.1); the strongest configurations of the evolution target's own family gain the least (+2.3). The paper's reading: bases further from saturation lean more heavily on the coordination patterns AHE fixed into tools, middleware, and memory, while a stronger base re-derives the same coordination from its prompt at low marginal cost. The evolved harness is, in effect, crystallized engineering experience — and experience is worth most to whoever has least of their own.

Chapter 8: Where the Value Lives

AHE evolved four component layers at once: system prompt, tools, middleware, long-term memory. Research question 3a: which of them carries the +7.3? The ablation design is clean surgery — swap exactly ONE evolved layer into the bare seed, hold the other three at seed defaults, re-run the full benchmark.

Each component owns a different failure surface

Memory (+5.6 alone): twelve boundary-case lessons — performance margins, queued-over-limit cancellation, evaluator-style closure, source-packaging layout. On Hard it scores 63.3%, above full AHE's own 53.3% and above Codex's 56.7%; on Easy the same lessons reduce to superfluous re-verification (50.0%, the worst Easy cell on the table). Tools (+3.3 alone): the shell tool has grown to 1,364 lines that auto-surface contract hints from files near each command; on Medium it lands within 0.9 pp of full AHE, but its built-in publish guard closes loops too early on Hard. Middleware (+2.2 alone): a finish-hook forcing one evaluator-isomorphic closure check before submission; it clears every Easy task and inflates turn counts on Hard. System prompt (−2.3 alone): 79 lines of universal discipline — and the only regression on the table.

Non-additivity: the sum overshoots the whole

Now the arithmetic that reframes the Hard-tier "loss" from Chapter 6. Add the three positive single-component gains:

components do not stackmemory alone      +5.6 pp
tool alone        +3.3 pp
middleware alone  +2.2 pp
                  ------
sum of parts     +11.1 pp        # if effects were independent
AHE full          +7.3 pp        # what actually happens: 3.8 pp lost to interference

# Hard tier, the extreme case:
memory alone on Hard   63.3%
AHE full on Hard       53.3%     # stacking LOSES 10 pp against one component

The mechanism is concrete, not mysterious: memory's lessons, middleware's finish-hook, and the prompt's rules all push the agent toward the same closure-style verification. On a Hard task's one-hour budget, three components independently demanding re-checks burn turns on redundant verification — time that long-horizon tasks needed for the actual work. And the loop cannot see this: the evolve agent optimizes an aggregate dominated by the 55 Medium tasks, so it converges to a Medium-heavy trade-off and gives back part of the Hard-tier memory effect. Interaction-aware evolution — optimizing the portfolio rather than the pieces — is left as future work, and this table is the evidence it will be needed.

Chapter 9: The Attribution Report Card

Chapter 4 built the manifest so every edit ships predictions. Research question 3b asks the question that design makes answerable: how good are those predictions? Across 9 evaluation rounds, compare each round's predicted-fix and predicted-regression sets against the next round's observed task flips, scoring standard precision and recall over the 89 tasks.

First, own the metric

Worked micro-example so the numbers mean something. Suppose one round's manifests predict 6 tasks will flip to pass, and next round 2 of those 6 flip, while 3 other tasks flip that were never predicted:

precision and recall, by handpredicted_fixes = {A,B,C,D,E,F}      observed_flips = {A,B,X,Y,Z}
hits = {A,B}                          # predicted AND happened

precision = hits / predicted = 2/6 = 33%   # of my promises, how many landed?
recall    = hits / observed  = 2/5 = 40%   # of what happened, how much did I foresee?

And the baseline that keeps us honest: a random predictor naming the same number of tasks by chance would score roughly the base rate of flips — the paper computes this random-prediction baseline per round. Beating it is the difference between targeting and guessing that something will change.

Reading the asymmetry

The fix panel says targeting is evidence-driven. One in three promised fixes lands, and over half of everything that improves was called in advance — five times better than chance on both axes. Edits are aimed at real, agent-anticipated targets. This is the manifest system working as designed: evidence in, falsifiable aim out.

The regression panel says foresight is nearly absent. At barely twice random, most upcoming regressions go unforeseen — roughly nine in ten of the tasks an edit breaks were never flagged at risk. The paper names it regression blindness, and offers the clean formulation: the agent can justify why an edit should help, but it cannot reliably name the tasks the same edit is about to break. This is what produces the non-monotone dips in Chapter 5's evolution curve.

Notice, finally, the meta-point: this entire chapter exists only because of pillar 3. A loop without recorded predictions could never measure its own blindness — it would just have unexplained dips. Decision observability did not make the loop prescient; it made the loop's non-prescience into a number with a confidence story, which is what lets the next paper fix it. That is what "falsifiable contract" buys even when the contract is broken.

What would fixing regression blindness look like?

The measurement points at three families of remedy, in ascending cost. Cheapest — better evidence: make passing tasks legible. If the debugger also filed brief "what this pass depends on" notes (which tool behaviors, which timing assumptions), the evolver would have dependency evidence to reason over instead of a corpus that only describes failure. Middle — impact analysis: before shipping an edit, statically ask which tasks' passing trajectories touched the edited component — a middleware that modifies shell output touches every task that shells out; those are the risk candidates. Chapter 2's file-level decomposition makes this tractable in a way a monolithic prompt never was. Costliest — test before promote: evaluate candidate edits on a held-out split and accept only non-regressing ones. That is precisely the design its contemporary Self-Harness ships — protection by measurement where foresight fails. The engineering-history rhyme is exact: humans did not fix their own regression blindness by becoming better predictors either; they built CI.

Chapter 10: Three Trajectories

Aggregates hide the actual craft. The paper's case study traces four trajectories at message level; here are the three most instructive, each mapping one failure mechanism to the component level that finally fixed it. If you internalize these three stories, you can predict the ablation table from first principles.

Trajectory 1: db-wal-recovery — a prompt rule earns its keep

The task: reconstruct a SQLite database from a corrupted write-ahead log, applying both new-row inserts and value updates encoded in the WAL, and emit /app/recovered.json. The verifier is exact — it asserts every row's fields against ground truth, including updated values on pre-existing rows. On the seed, this task passed 1 of 2 rollouts. Walk the failing rollout against the passing one — both share the same seed and the same first three steps:

The debugger had filed this failure under a pattern class worth memorizing: proxy validation instead of evaluator-isomorphic validation — closing on a surrogate check (row count, file exists, script runs) rather than the evaluator's exact assertions. And the punchline that shows why pattern-level fixes beat task-level patches: chg-1 is a 68-line append to systemprompt.md that never mentions SQLite, WAL, or this task — its predicted_fixes did not even list db-wal-recovery. The general phrasing carried it across anyway, flipping the task to 2/2 permanently. One task's symptom became a reusable rule that also happened to fix its neighbors: that is the compounding the whole architecture exists to produce.

Trajectory 2: path-tracing — when advice must become enforcement

The task: implement a path tracer rendering a scene to /app/reconstructed.ppm; the verifier compares that single file pixel-for-pixel. At iteration 4 the task scored 0/2, and the failing pattern is almost comic: the agent renders a correct image, self-checks it successfully… then issues rm -rf /app/image /app/reconstructed.ppm /app/scratch as a final tidy-up, and submits on the cleanup's exit code. The verifier finds no file. Everything about the solve was right; the harness let the agent destroy verified state at the finish line.

The seed prompt already warned against this — "do not destroy verified state" was in the advisory text, and under long-horizon pressure the agent tidied anyway. The iteration-5 fix moved the rule from prose to mechanism: a shell guard that intercepts deletion of verified deliverables and answers with a message naming the protected file. At iteration 5 the same rollout reaches the same cleanup step, the guard blocks it, the agent reads the message, acknowledges, and finishes. 0/2 → 2/2. The same round also recovered polyglot-rust-c and large-scale-text-editing (both in the manifest's predicted_fixes) — while configure-git-webserver recovered only partially, because its failure travels a state-reset path the guard still treated as overrideable; iteration 8's post-success hard-block closed that remaining gap. Fixes are iterative even when the diagnosis is right.

Trajectory 3: mcmc-sampling-stan — two components, one trap

The task: install rstan 2.32.7, fit a hierarchical beta-binomial model to 30 observations, write posterior means of two parameters to text files — and the verifier reruns the agent's own analysis.R end-to-end, asserting the first parameter lands in [2.84, 2.91] and the second in [16.1, 16.7]. The failing pattern (0/2 from iteration 1 through 5) is the proxy-then-skip sequence: compute a quick grid-integration approximation, write those numbers as the deliverable, launch the real MCMC as a background job, kill it early to "preserve the already-created deliverables", and submit on a files-exist-and-parse check. The rerun verifier then executes the unconverged sampler and gets values around 10¹⁹.

Five iterations of fixes missed it — the iteration-2 contract-first rule was advisory, and this trajectory's proxy felt like compliance. What finally flipped it at iteration 6 was two components from different levels working together: the tool-level publish-state guard (you may not finalize while the process producing your deliverable was killed unconverged) plus the step-spanning risk-monitor middleware (Chapter 2's code — pattern 2, "same error class repeats → switch tactic"). Neither alone had caught it in five tries. Some failure mechanisms span steps and surfaces, and only a combination of components covers them — which is precisely why AHE's joint, all-components evolution beats single-surface optimization, and precisely the interaction structure that (Chapter 8) also caps the stacked gains. The same coupling that catches cross-cutting failures spends double budget on redundant checks. Engineering is trade-offs all the way down.

Chapter 11: Connections, Limits & Cheat Sheet

Where AHE sits in the self-improvement landscape

AHE is one corner of a rapidly crystallizing design space: systems that improve the scaffold around a frozen model. Its nearest neighbors, and the axis each occupies:

On this site, the harness trilogy gives the wider frame: Harness Engineering (what the layer is), Harness Optimization (the ladder from prompts to optimizer code — AHE now occupies its top rungs), and Self-Improving Harnesses (STOP, AlphaEvolve, DGM — the recursive limit AHE approaches with its governance rails on). The survey veanor Code as Agent Harness summarizes this paper in one chapter; you now know the machinery behind its summary.

Limits, in the paper's own words

Three, stated without spin. Benchmark scope: evolution driven on one benchmark, transfer probed on one more; broader languages, repository-scale deployments, and human-in-the-loop workflows untested. Operating-point coupling: step budgets and timeouts were fitted to GPT-5.4 high, so cross-model gains conflate harness portability with timing fit (the non-monotone within-family numbers are the visible symptom). Governance: bounded workspace + versioned manifest + file-level rollback is real containment but not a complete guardrail stack — the paper explicitly positions AHE as a controlled research prototype, not a mature autonomous self-improvement system. Add the internal limits the results themselves exposed: non-additive component interactions cap stacked gains (Chapter 8), and regression blindness caps loop stability (Chapter 9). The future-work agenda writes itself: interaction-aware evolution, and regression foresight — or, failing foresight, Self-Harness's held-out gate.

Cheat sheet