You cannot retrain a frontier model. But the model is only half the agent — the other half is the harness: prompts, tools, skills, control flow. DarwinX runs natural selection over a population of harnesses, with the model frozen, and adds about 17 points on average across four benchmarks. The selection rule fits on an index card.
You are the engineer responsible for a coding agent. Call it Monet, because that is what the paper calls the one this story is actually about. Monet wraps a frontier model — a frozen GPT-5.5 whose weights you cannot touch, will never touch, and could not afford to touch — in a loop: read the task, think, call a tool, read the result, repeat until done.
Your benchmark is Terminal-Bench 2.1: 89 real terminal tasks — compile this kernel module, migrate this database, assemble this genome fragment, train this small model and write the metrics to a file. Each task ships its own verifier, a program that inspects the final state of the machine and returns pass or fail. No partial credit, no human judgment. Monet scores 75.5%.
The leader scores above 83. Same class of model. So the gap — nearly eight points — is not the model. It is everything wrapped around the model: the system prompt, the tools it may call, the skill documents it can consult, the control flow that decides when it verifies and when it stops. The paper's word for that wrapping is the harness, and the paper's opening claim is worth reading twice: an agent's capability is determined as much by its harness as by the underlying model.
You read the failure logs. A cluster of ML tasks keeps dying during environment setup — the agent installs packages one at a time, hits a version conflict at minute eighteen, and times out. So you add one rule to the system prompt: “Always create a fresh environment and install all dependencies in a single resolved transaction before starting work.”
Sensible. Let us score it honestly. You run each of five representative tasks five times, before and after the edit, and count passes. Five runs per task means every per-task score is a multiple of 0.2 — keep that granularity in mind; it matters for everything in this lesson.
| Task | Before (passes / 5) | After (passes / 5) | Change Δ |
|---|---|---|---|
| pandas-pipeline | 1 / 5 = 0.2 | 4 / 5 = 0.8 | +0.6 |
| bio-assembly | 0 / 5 = 0.0 | 1 / 5 = 0.2 | +0.2 |
| sqlite-migrate | 3 / 5 = 0.6 | 3 / 5 = 0.6 | 0.0 |
| kernel-module | 4 / 5 = 0.8 | 2 / 5 = 0.4 | −0.4 |
| log-parse | 5 / 5 = 1.0 | 4 / 5 = 0.8 | −0.2 |
Now do the arithmetic three different ways, because the three answers disagree, and the disagreement is the entire motivation for this paper.
Way 1: the mean. 0.52 → 0.56. The edit is a win, +4 points. Ship it. This is what a scoreboard sees, and it is what most self-improvement loops gate on.
Way 2: gains and losses separately. The edit bought +0.8 of new capability and paid for it with 0.6 of capability you already had. The kernel-module task regressed because a from-scratch reinstall of its toolchain blows the time budget; log-parse regressed because the new preamble occasionally distracts the agent into rebuilding an environment it never needed. The mean cannot see this — it happily nets a broken thing against a fixed thing and calls the difference progress.
Way 3: the solved set. Before the edit, the tasks you could solve reliably (say, at 0.8 or better) were {kernel-module, log-parse}. After, they are {pandas-pipeline, log-parse}. You did not extend your capability. You traded one capability for another — a lateral move dressed up as a gain.
Suppose you gate edits properly and only keep real wins. A subtler trap remains. Your harness is now the product of a sequence of edits, each conditioned on all the ones before. The dependency-install rule you adopted in week one shapes which failures you see in week two, which shapes what you try in week three. Robeyns et al., who built a self-improving coding agent called SICA that edits its own source in a single lineage, report exactly this: early edits bias the search and the agent plateaus.
The paper calls this path dependence. A single line of descent can only be one thing at a time. If the best harness for numerical ML tasks and the best harness for text-parsing tasks lie in different directions from where you stand, a single lineage must pick one direction and permanently forfeit the other — unless something keeps the road not taken alive. Hold that thought; it becomes the archive in Chapter 3 and it wins a benchmark outright in Chapter 7.
By 2026 the field had converged, almost embarrassingly, on one inner loop for self-improving agents. Run a batch of tasks. Reflect on the failures. Propose a small bounded edit. Check it against some held-out or regression signal. Keep or revert. Prompt optimizers do this to instructions, skill systems do it to skill documents, workflow systems do it to component graphs, and SICA and the Darwin Gödel Machine (DGM) do it to the agent's own source code.
DarwinX keeps that inner loop untouched. Its question is the one layer up: what is the right selection process to wrap around it? Concretely, three prior answers and their gaps:
| System | Search | What earns a keep | The gap |
|---|---|---|---|
| SICA | Single lineage, self-editing | Benchmark reward on the current suite | Path dependence; authors report an early-edit plateau |
| DGM | Open-ended archive, one parent mutated at a time | Score vs. the parent on a growing task subset | No merge operator: lineages that solve complementary tasks are never brought back together, and a gain carries no obligation to hold what it displaces |
| HarnessX | Staged single-lineage pipeline, variants kept isolated | Per-edit gate on the average score plus a seesaw test | Isolation contains interference but strands specialists in separate lineages, and sub-threshold regressions still accumulate |
| DarwinX | Population archive with recombination | The preserve-and-extend contract of Chapter 2 | — the subject of this lesson |
Read the DGM row again, because DarwinX defines itself against it. Archives of self-modifying agents are, in the paper's words, common ground by now — inherited from quality-diversity and open-ended search. What such systems leave unsettled is how a candidate earns its place. DarwinX's answer has three commitments: a child is admitted only under an explicit contract that bounds what a win may cost elsewhere; losers are retained because they may hold the one edit another lineage needs; and complementary specialists are merged rather than left in separate branches forever.
An obvious question before we invest ten chapters: if you have compute to burn on thousands of agent rollouts, why not spend it on fine-tuning instead? Three reasons, in ascending order of importance.
Attribution. If the weights move and the harness moves, and the score goes up, you have learned nothing about either. Freezing the model makes every reported delta a statement about the harness alone. This is a methodological control, not an ideology — the paper's own outlook section sketches co-evolving both layers, in alternating phases precisely so attribution survives.
Access. The strongest models are API-only. You cannot fine-tune them, but you can always change what you send them and what tools you hand them. Harness evolution is the self-improvement channel that works on models you do not own — and the paper demonstrates the same procedure on frozen GPT-5.5, GPT-5.6 Sol, and Opus 4.8 bases.
Durability and auditability. A harness is files: prompts, markdown skill documents, tool code. Every accepted change is a human-readable diff sitting next to the evidence that justified it. A weight delta is billions of opaque numbers. And when the base model is swapped next quarter — base models are replaced far more often than scaffolds are rewritten — the harness survives as an asset. Chapter 8 shows a harness evolved on one vendor's model transferred, unchanged, onto another vendor's.
Concreteness now, because “harness” will otherwise stay a fog. DarwinX splits the editable surface into two layers:
An edit is a change to either layer: add a skill document, rewrite a prompt rule, add a tool, change when verification runs. The model never appears in that list. Every gain in this paper flows through those files and nothing else.
A single benchmark score cannot distinguish “we learned transferable agent behavior” from “we memorized our evaluation set.” So the paper builds a ladder of four benchmarks with increasing separation between the evolution signal and the test, and reports all four:
| Rung | Benchmark | Separation from the evolution signal | Result |
|---|---|---|---|
| 1 | Terminal-Bench 2.1 | None — evolve and report on the same 89 tasks | 75.5% → 83.2% (+7.7); 84.7% frontier on a stronger base |
| 2 | TerminalWorld | Task split — evolve on 94 tasks, report on 41 disjoint ones | 61.0% → 68.3% held-out (+7.3), best on the split |
| 3 | WebArena-Infinity | Distribution and reward source — evolve on 300 synthetic intents under an LLM judge, report on 1,260 real tasks under deterministic verifiers | 43.5% → 93.0% audit-clean (+49.5) |
| 4 | SWE-bench Verified | Whole benchmark — run the Terminal-Bench harness unchanged, zero in-domain feedback | 84.2%, +3.4 over a strong reference |
Average those four matched-model gains and you get the headline: (7.7 + 7.3 + 49.5 + 3.4) / 4 = 67.9 / 4 = 16.975 ≈ 17 points, from one loop, with the model frozen everywhere. The rest of this lesson builds the machine that produced that number, mechanism by mechanism, and then walks the ladder rung by rung asking, at each step, what could have gone wrong and how the paper checked.
Natural selection needs a fitness function. Before designing the selection rule, we have to be honest about what kind of measurement an agent benchmark actually returns — because it is far noisier than most people's intuition, and every design decision in DarwinX is downstream of that noise.
Each benchmark task ships a verifier: a program that inspects the end state — files written, database rows changed, tests passing — and returns a single bit. DarwinX's fitness comes from that bit and from the trajectories the agents themselves produce. The loop never consumes the benchmark's reference solutions, and no human ever picks a winner. The paper's footnote insists the title is meant literally: no gold labels and no hand-picked winners, only survival of the fitter variant under measured fitness.
Why does this matter beyond hygiene? Because of what it makes impossible. A harness that never sees answers cannot encode answers. The only thing selection can reward is a change in how the agent works: which tools it reaches for, what it verifies before finalizing, when it keeps going. Whatever survives is procedure, not trivia — which is precisely why, in Chapter 8, a harness evolved on terminal tasks will turn out to work on GitHub issues it has never seen.
Here is the uncomfortable empirical fact. Run the same agent on the same tasks twice — same harness, same model, temperature zero — and the scores diverge. Bjarnason et al. measured swings of 2.2 to 6.0 points of pass@1 on SWE-bench Verified between identical runs. The paper notes that this is often the magnitude of a single accepted edit. Environments flake, servers hiccup, one token sampled differently at step 40 cascades into a different hour of work.
So define the measurement DarwinX actually uses. For a harness variant v and a task t, run the task k times and let
Three passes out of five gives p̂ = 0.6. This is a noisy estimate of the variant's true per-task reliability p. How noisy? Work it out. A single run is a Bernoulli trial with success probability p, and the standard error of an average of k trials is:
Read those numbers as error bars on a single task's score. At k = 1 the measurement is worth almost nothing: ±50 points. At the k = 5 the paper uses for confirmation, a task whose true rate is 0.5 will routinely measure 0.4 or 0.8. Even at k = 25 — five times the budget — you still carry ±10 points per task. Averaging over many tasks helps the benchmark-level score, but selection decisions in DarwinX are made per task (Chapter 2 compares solved sets task by task), so the per-task noise is the noise that matters.
Concrete numbers. Variant A has true solve rate 0.4 on some task; variant B has 0.6. B is genuinely better by 20 points. Compare them with one rollout each. The comparison goes wrong — A passes while B fails — with probability:
and it ties (both pass or both fail), telling you nothing, with probability 0.4×0.6 + 0.6×0.4 = 0.24 + 0.24 = 0.48. So a single-rollout duel gives the correct strict ranking only 36% of the time: 0.6 × 0.6 = 0.36. Sixteen percent of the time your selection loop confidently promotes the worse harness. Compound that over dozens of generations and the search is a random walk wearing a lab coat.
The first simulation lets you feel this. Two variants with fixed true rates duel at the k of your choosing. Watch how often the worse one wins at k = 1, and watch the flip rate collapse — but never vanish — as k grows.
Variant A's true rate is 0.45, variant B's is 0.65 — B is better by 20 points, always. Each duel samples k rollouts per variant (dots: pass or fail) and ranks them by measured avg@k. The tally tracks how often the measurement gets the ranking wrong. Raise k and watch luck lose its grip — slowly, and at a price: every unit of k is a full agent rollout you pay for.
Two things to notice while you play. First, even at k = 5 the wrong verdict still arrives a nontrivial fraction of the time — the noise is suppressed, not eliminated, and DarwinX's whole architecture assumes it never will be. Second, k = 25 is quiet but costs 25 full agent runs per variant per task. With 89 tasks and dozens of candidate variants per run, uniform high-k measurement is unaffordable. The budget has to be spent unevenly — which is the next idea.
DarwinX's answer is a graduated measurement ladder, and it is worth internalizing because it reappears in every experiment:
The paper names the principle: separate exploration from confirmation. A lucky avg@3 on a subset can get a variant into the tree — that costs little and keeps the search moving. But a lucky result cannot redirect the search, because steering rights require surviving the strict re-test. Recall during exploration, precision during confirmation. Chapter 2 turns this into a precise rule.
One more measurement decision, easy to miss and load-bearing for trusting the results. When an agent runs out of the task's declared time budget, is that the benchmark's fault or the agent's? DarwinX counts it as a real failure — timeouts at the declared budget are capability failures, and on Terminal-Bench every errored trial scores zero under the strict leaderboard rule. Only genuine infrastructure failures — the cluster died for reasons outside any agent's control — are separated, under a policy fixed in advance. Chapter 6 shows the headline barely moves even under the harshest accounting the authors could construct, which is exactly what you want to check before believing any of this.
Chapter 0 showed that a single mean hides interference. Chapter 1 showed that every measurement is noisy. Now we can state DarwinX's central rule — the condition a child harness must satisfy to earn its place — and check it by hand on real arithmetic.
A child variant c is proposed as an edit to a parent p. Score both on the same tasks at avg@k, and for each task t form the per-task change:
Then — and this is the move that fixes Chapter 0's blindness — do not sum the deltas into one number. Split them into what the child gained and what it broke:
The notation (x)+ means max(x, 0): keep the value if positive, else zero. So R adds up every task where the child got worse, and ignores the gains entirely. The fitness enabler admits a child if and only if:
In words: extend — you must be a net improvement — and preserve — the total damage you inflict on what your parent already solved must stay under an explicit tolerance δ. A win is permitted to cost something, because under noise everything wiggles. It is not permitted to cost much.
Parent harness p, six tasks, avg@5 throughout (so every score is a multiple of 0.2). Parent's scores: (0.4, 0.8, 1.0, 0.0, 0.6, 0.2). Three proposed children. Compute g and R for each — genuinely do this on paper; it is six additions per child.
| Task | t1 | t2 | t3 | t4 | t5 | t6 | g = ΣΔ | R = Σ(−Δ)+ |
|---|---|---|---|---|---|---|---|---|
| Parent p̂ | 0.4 | 0.8 | 1.0 | 0.0 | 0.6 | 0.2 | — | — |
| Child A Δ | +0.6 | −0.2 | 0 | +0.2 | 0 | −0.2 | +0.4 | 0.4 |
| Child B Δ | +0.4 | 0 | 0 | +0.2 | −0.2 | +0.2 | +0.6 | 0.2 |
| Child C Δ | +0.4 | −0.4 | −0.2 | +0.4 | 0 | −0.2 | 0.0 | 0.8 |
Check child A: gains 0.6 + 0.2 = 0.8; losses 0.2 + 0.2 = 0.4; g = 0.8 − 0.4 = +0.4. Check child C: gains 0.4 + 0.4 = 0.8; losses 0.4 + 0.2 + 0.2 = 0.8; g = 0. Now apply the contract at a tolerance of δ = 0.3:
Notice what each rejection is for. Child A is Chapter 0's dependency-install edit: a real gain financed by real breakage. Child C is the pure trade: it rearranges which tasks pass without expanding coverage at all — and a mean-gated loop would happily accept an endless sequence of such children, wandering forever. Child B is the only one that grew the frontier while holding the interior.
The three children from the table, live. Bars show per-task Δ against the six parent scores; the meters accumulate g and R left to right; the verdict chip applies g > 0 and R ≤ δ. Switch children, then switch δ — child A's fate depends on the tolerance, child B's and C's do not. That difference is the design: δ absorbs measurement noise, it does not forgive trades.
A natural instinct: never accept any regression. R = 0 or revert. Work out what that costs under Chapter 1's noise. At k = 5, a task whose true rate is unchanged still flips its measured score whenever the pass counts differ between the two five-run samples — and one flipped rollout moves p̂ by a full 0.2. A perfectly neutral child, measured across the roughly forty tasks a parent lineage solidly solves, will almost surely show a spurious −0.2 somewhere. Zero tolerance would revert nearly every honest child ever proposed and freeze the lineage solid. The tolerance δ is sized to the noise floor: small enough that systematic damage fails the test, large enough that sampling wiggle does not.
The inequality is necessary but not the whole gate. A reasoned verifier agent f reads the numbers and the evidence behind them — the child's trial trajectories ℮ and the population's shared memory Kg (Chapter 5) — and returns the verdict:
Why an agent rather than the bare threshold? Because identical numbers can mean different things. A −0.2 on a task the memory knows to be infrastructure-flaky is noise; the same −0.2 on a rock-stable task, in a trajectory showing the child's new skill actively misfiring, is damage. The verifier is where that context enters. And crucially, its promote verdict is still only provisional — the paper's two-speed design:
The paper compresses this into one sentence worth memorizing: the selector is permissive about trying an edit but strict about trusting it. A high-precision admission rule applied at the front door would freeze the lineage before complementary variants ever emerge; recall belongs in exploration, precision in confirmation. The same discipline — state your acceptance condition, then verify against it before committing — will reappear in Chapter 9 as the behavior the evolved agents themselves learned. The search and the agent converge on the same epistemics.
One bookkeeping definition completes the contract, and Chapter 3 spends it. Every node carries its accumulated ancestry of net gains:
If the root spawned a child worth g = +0.4, and that child spawned a grandchild worth g = +0.2, the grandchild's G = 0.4 + 0.2 = 0.6. Why track this instead of each variant's raw benchmark score? Because screening happens on rotating subsets — different variants are measured on different tasks, so raw scores are not comparable: a variant screened on the hard-ML subset will look worse than one screened on the sysadmin subset, regardless of merit. Chained deltas, each measured parent-against-child on the same tasks, are comparable. A child that adds real improvement on top of an already-improved parent outranks both the root and its parent even if its own screening subset was brutal.