Harness Engineering
The System Around the Model

Chapter 1: From Agent Frameworks to Harnesses

For a couple of years, the standard recipe for an "agent" was a five-word formula: agent = LLM + memory + tools + planning + action. Every framework of that era — the chains, the graphs, the crews — was a different arrangement of those same five boxes. The formula was not wrong. It was incomplete in a very specific way, and the gap only became visible when agents started running for hours instead of seconds.

Weng (2026) names what harness engineering adds to that old formula, and there are exactly four additions: workflow design (loop engineering), evaluation, permission controls, and persistent state management. Each one patches a specific hole in the old parts list — a hole you only discover when an agent runs long enough to fall into it. Take them one at a time.

Workflow design. The old formula had "planning," but a plan made once is not a workflow. Real work is iterative: plan, act, hit a failure, revise, act again. Workflow design decides how that cycle is structured — what triggers a retry, what counts as done, when to ask the user — and the old formula simply had no slot for it. A framework-era agent that hit an unexpected test failure was done; a harness treats the failure as the input to the next lap.

Evaluation. The old formula had "action" but nobody checking the action. The framework happily reported success whether or not the code compiled — output was assumed good because nothing was positioned to say otherwise. Evaluation gives the harness a seat for the judge: tests re-run after every edit, evals scored after every episode. An agent without evaluation is not autonomous; it is unsupervised, which is a different and worse thing.

Permission controls. The old formula had "tools," and a tool that can write files can also delete them — nothing in "LLM + tools" says which is allowed. Permission controls make the boundary explicit: read anywhere, write only inside the repo, never touch prod credentials, ask before running anything destructive. This is a security boundary, and security boundaries have to be designed — they do not emerge from a parts list.

Persistent state management. The old formula had "memory," but memory that lived inside the conversation — close the tab and the agent was born again, blank. Persistent state decides what survives the session: the plan, the notes, the half-finished investigation. It is the difference between a colleague and a goldfish, and Chapter 4 gives it a full treatment.

The one-sentence upgrade, straight from the source, is worth unpacking word by word: harness engineering is "no longer only prompt templates, but closer to runtime and software system design: how the model observes, acts, memorizes, checks itself, and improves." A prompt template is a fill-in-the-blank string — static text with holes. A runtime is a living system with state, scheduling, and error handling — the kind of thing you would draw as an architecture diagram, not paste into a text box. The claim is that building agents graduated from writing (crafting strings) to systems engineering (designing the machine that owns the strings). If you have ever gone from writing SQL queries to designing the database engine, you know exactly this altitude shift.

Ten lines of code make the altitude shift impossible to miss. Look at what each object can hold: the template holds words; the runtime holds the four additions — a retry-and-budget policy (workflow design), a verifier hook (evaluation), an allow-table (permissions), and a trajectory file (persistent state). You cannot store a budget in a string.

python
# A prompt TEMPLATE: static text with holes. No state, no schedule,
# no permissions, nothing survives the call. This is the old era.
TEMPLATE = "You are a coding agent. Task: {task}. Think step by step."

# A RUNTIME: the same intent, now a machine that owns the four additions.
class AgentRuntime:
    def __init__(self, task, budget=20):
        self.task, self.budget = task, budget          # workflow: a lap counter
        self.allowed = {"read": "anywhere",             # permissions: explicit table
                        "edit": "repo-only",
                        "bash": "ask-if-destructive"}
        self.trajectory = open("state/trajectory.log", "a")  # persistent state

    def observe(self, obs):
        self.trajectory.write(obs + "\n")              # every turn banked to disk
        return obs[-2000:]                             # context policy at the boundary

    def should_stop(self):
        self.budget -= 1
        return self.budget <= 0 or tests_pass()       # evaluation: a verifier, not vibes

Now the two design principles, and they are deeper than they look. Principle one: the harness should be deliberately simple and generic, so it generalizes. Every task-specific rule you hardcode ("for Django projects, always check settings.py first") makes the harness better at one thing and brittle at everything else. Simple, general mechanisms — a loop, a file system, a test command — work on the task you have not imagined yet.

Principle two is the sneaky-brilliant one: the harness should lean on existing software-engineering practices to benefit from pretraining knowledge. Think about what a frontier model has read: millions of bash sessions, git logs, Makefiles, CI configs, code reviews, stack traces. A harness whose interface is bash and git inherits all of that fluency for free — the model already knows that grep -rn finds things, that git diff shows changes, that a failing exit code means trouble. Invent a bespoke DSL instead — AGENT_QUERY(memory_bank, k=5) — and the model has read exactly zero examples of it. You would be spending your context window teaching syntax that bash gives you for free.

There is a compounding version of this argument that matters even more. Model capabilities improve on the distribution of pretraining data. Every new model generation gets better at bash and git because the internet keeps producing bash and git. A harness that speaks those languages rides the capability wave automatically: swap in next year's model and the harness gets stronger with zero changes. A harness built on a private DSL is frozen — nobody is pretraining on your DSL, so model upgrades improve it only by accident. This principle — align your interfaces with the training distribution — will come back in Chapter 4 as the real reason files beat vector databases for agent memory.

All of this crystallizes in one analogy from the source, and it is the load-bearing analogy of the whole lesson: a harness is to a model what an operating system is to hardware. Like an OS, a harness should encapsulate complicated logic while keeping the interface simple. Your text editor does not manage disk sectors; it calls open() and write(), and the kernel hides the mess. Likewise, the model does not manage retry policies, permission checks, or context budgets; it calls read, edit, bash, and the harness hides the mess. And exactly as with operating systems, the interfaces are standardizing across the industry: configs, tool schemas, and protocols are converging — MCP is having its USB-C moment, one plug shape for every peripheral.

Stress-test the analogy before you lean on it, because it breaks in one instructive place: hardware does not learn, but this "hardware" does. A real CPU executes deterministically; the model at the bottom of this stack is probabilistic — it can call the wrong tool, malform the arguments, or ignore the interface entirely — so a harness, unlike a kernel, must treat every "instruction" as a proposal to validate, never an order to obey. And each model generation is better hardware for the same OS: the layers above inherit the upgrade for free, which no motherboard ever did. The analogy is load-bearing for structure — encapsulation, small stable interfaces, standardization — not for trust.

The stack, three ways. If a coding agent has ever asked you "allow this command?", you have watched the kernel enforce its permission table.

The simulation below draws the analogy literally. Four layers, top to bottom: user space (your task), the syscall-like tool interface (read, write, exec, spawn), the kernel-like harness internals (scheduler, context manager, permission table, artifact store), and the hardware-like model at the bottom — pure tokens in, tokens out. Tap any layer to read its responsibilities, then press Send syscall and watch a request travel down through the stack and back up, the way a file-read really does in an OS.

The table pins down the shift. Read the rows as before/after: the framework era answered "what parts does an agent have?"; the harness era answers "what system does an agent run inside?"

Table II — The framework era listed parts; the harness era designs the runtime those parts live in.

Chapter 2: Recursive Self-Improvement — Why Harnesses Matter Now

Why is a lesson about plumbing — loops, files, sub-processes — framed as a lesson about the future of AI? Because of an idea that is sixty years old and suddenly practical. In 1965, the mathematician I. J. Good defined an "ultraintelligent machine": a system that surpasses humans in all intellectual activities — and, crucially, since designing machines is itself an intellectual activity, it could design better machines to improve itself. The second clause is the whole idea. A machine that is merely smarter than us is a rival; a machine that can build its own successor is a process.

Forty-three years later, Yudkowsky (2008) gave the process its modern name: recursive self-improvement (RSI). His formulation is precise, and the precision matters: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence. Read that twice. The loop is not "the AI gets better at answering questions." The loop is "the AI improves the machinery — and the improved machinery then produces better intelligence, which improves the machinery further." Better answers are a one-time gain. Better answer-producing machinery compounds.

An analogy to hold onto: a student who studies hard gets better grades — linear improvement. A student who studies how she studies — who notices that flashcards beat re-reading, then notices which kinds of flashcards work, then builds a system for making them — improves her improvement rate. That second student is running RSI on herself. The output of each cycle is not knowledge; it is a better learning machine.

What does this look like in modern AI, concretely? The science-fiction version — a model directly rewriting its own weights — is not the near-term path (Chapter 8 returns to why). Weng (2026) describes the broader, already-happening version: the model improves the training pipeline and the deployment system, and those improvements enable a better successor model with improved performance across economically valuable tasks. The model does not touch its own brain; it builds better tools, better data, better evaluations, better scaffolding — and the next model trained inside that improved machinery comes out stronger. Both major frontier labs have reported that AI is already drastically accelerating their own research (Anthropic; OpenAI) — models writing the experiment code, triaging the failures, and drafting the analyses that produce their successors.

Pause on the phrase "economically valuable tasks," because it is doing quiet work in that definition. The loop is not graded on benchmark trivia; it is graded on whether each successor is better at the work people actually pay for — fixing real bugs, running real experiments, producing real analyses. That grading choice matters for the whole trilogy: it means the improvements that count are the ones a harness can measure on real work, which is why evaluation keeps appearing as a first-class harness component rather than an afterthought.

Draw the loop explicitly, because it is the diagram this whole trilogy lives inside:

The RSI ring. Nothing in it edits weights directly — the loop routes through external machinery the model can already read and write.

Now locate the harness in that ring: the harness is the model's hands. A model can only improve the pipeline by acting on it — running experiments, editing configs, reading logs, filing results. Every one of those actions flows through the harness. Better hands → better experiments → better successor. A clumsy harness caps the loop no matter how brilliant the model is, the way a brilliant surgeon is capped by shaking hands.

Let us make "the harness gates the loop" quantitative with a toy model — honestly labeled: this is arithmetic for intuition, not a forecast. Suppose each full trip around the loop multiplies capability by a loop gain g:

Every symbol, with its analogy: C₀ is today's capability — the starting balance in a bank account. C_n is capability after n generations — the balance after n years. g is the loop gain — the interest rate, except it can dip below 1: it bundles how much of the model's intelligence the harness successfully converts into better experiments, data, and evals for the successor. n counts trips around the ring, not calendar time.

Now the arithmetic, by hand, three scenarios. g = 0.95 (weak harness: the model is smart, but its hands fumble — experiments fail silently, results get lost, evals are noisy, so each successor captures less than its parent's full capability): after five generations, 0.95 × 0.95 = 0.9025; × 0.95 = 0.8574; × 0.95 = 0.8145; × 0.95 = 0.7738. Capability has decayed to 0.77× — the loop is a drain. g = 1.0: 1 × 1 × 1 × 1 × 1 = 1. Perfectly flat; the loop spins and nothing compounds. g = 1.15 (strong harness): 1.15 × 1.15 = 1.3225; × 1.15 = 1.5209; × 1.15 = 1.7490; × 1.15 = 2.0114. After five generations, capability has doubled — 1.15⁵ ≈ 2.01×. The gap between decay and doubling was 0.2 of loop gain. That 0.2 is harness quality.

One decomposition makes the toy model actionable rather than fatalistic. Loop gain is not a single dial — it is a product of stage efficiencies around the ring: how faithfully experiments execute, how much signal survives into training data, how honestly the evals grade. Write g = e_exp · e_data · e_eval. Say your loop runs experiments brilliantly but grades sloppily: 1.2 × 1.1 × 0.9 = 1.188. Repair only the weakest factor — the evaluator — and 1.2 × 1.1 × 1.0 = 1.32: the whole loop's compounding rate jumps 11% from fixing one unglamorous stage. Products punish weak links and disproportionately reward repairing them, which is why serious harness work so often looks like boring reliability engineering on whichever stage everyone else ignored.

The toy also tells you how harness quality gets measured in practice, which lesson 2 will depend on: fix the model, vary the harness, and score on a fixed suite of real tasks. Four numbers cover most of it:

Scoring a harness: fix the model, vary the machine. Once a harness has a score, it can be optimized — which is the entire premise of lesson 2.

This is why harnesses matter now, and why a systems-engineering topic belongs in a lesson series about intelligence. Whether the flywheel accelerates, idles, or decays is decided at the margin by the machinery this lesson teaches: the loop (Chapter 3), the memory (Chapter 4), the parallelism (Chapter 5). Spin the flywheel below and feel it: drag harness quality low and the ring barely speeds up lap over lap — the capability chart sags red. Drag it high and each lap visibly accelerates the next.

Three experiments, predictions first: (1) find the slider position where the chart goes exactly flat — from g = 0.9 + q·0.003, predict q ≈ 33 before you hunt for it. (2) Set quality to 100 (g = 1.2) and predict how many laps until capability doubles: log 2 / log 1.2 ≈ 3.8, so watch for it on lap 4. (3) Set 20, watch the red decay, and notice the flywheel itself visibly slowing lap over lap — the loop de-compounding in real time.

Why is this loop practical now, when Good named it sixty years ago? Three preconditions quietly converged. Models became genuinely good at code — and the machinery of AI research (training pipelines, eval suites, experiment configs) is code, so the loop's raw material became editable by its subject. Tool interfaces stabilized — Chapter 6 shows the same eight tool groups emerging everywhere, so a model's competence transfers across harnesses instead of being relearned per product. And evaluation became automatable for real work — test suites, benchmark harnesses, verifiable rollouts — giving the loop an objective scorekeeper. Remove any one of the three and RSI stays science fiction; with all three, it becomes an engineering roadmap.

Scope note, for honesty about the map: this trilogy covers the harness side of RSI — auto-research, self-improving agents, evolutionary program search. There is a second road to RSI that runs through the model itself: self-play and self-rewarding training (Yuan et al. 2024; Chen et al. 2024), reinforced self-play reasoning with zero data (Zhao et al. 2025, "Absolute Zero"), test-time training, and continual learning (Choi et al. 2026). Weng mentions these and sets them aside, and so do we — different machinery, same dream.

Two roads to the same dream. They are complements, not rivals — a better harness produces the rollouts and evals the model road trains on.

Chapter 3: Pattern 1 — Workflow Automation

We now open the toolbox. Weng distills serious harnesses into three design patterns, and the first is the one that makes an agent an agent at all: workflow automation — defining a workflow in which the model can operate, test, and iterate. The canonical shape is a goal-oriented loop: plan → execute → observe/test → improve → execute again, repeating until the goal is achieved. Not a script that runs once. A cycle that keeps turning until something objective says "done."

Walk one real bug through every stage, because the abstraction only lands with concrete freight in it. Goal: "the date parser fails in CI." Plan: the model decides its first move is to reproduce the failure — not to guess at a fix. Execute: it runs pytest tests/ through the shell tool. Observe: the harness feeds back the tail of the output: FAIL test_parse_date — expected 3, got 4. Improve: reading the parser, the model spots an off-by-one in the month index and revises its plan: edit line 41, then re-verify. Execute again: it edits the file, re-runs pytest, observes 24 passed, and only now — because the test suite, not the model's confidence, says so — does the loop exit and commit.

Notice something human in the middle of all this machinery: the loop may also proactively ask the user for clarity — about task specification or execution preference. "Should the parser accept two-digit years, or is that the bug?" A question like that costs the user ten seconds. The alternative — the agent guessing, and guessing wrong — costs forty minutes of confidently wrong work plus the cleanup. Ambiguity is cheapest to resolve at the moment it is detected; a well-built workflow treats asking as a first-class action, not a failure of autonomy.

Three flavors of ambiguity worth a ten-second question. The skill being designed here is not asking less — it is asking early.

If you want to see this pattern in the wild with no product wrapping around it, Weng points to Karpathy's autoresearch repo (github.com/karpathy/autoresearch) as a clean example of how such a workflow can be constructed — a small, readable harness that plans experiments, runs them, reads results, and iterates. It is worth an evening: the whole pattern in a few files, with nothing to hide behind.

Here is the deceptively simple diagram that carries the entire pattern — Weng shows it as the Codex agent loop: the agent calls tools, and the tool responses affect the model's next generation. That second clause is the whole trick. In one-shot generation, the model's output is a function of the prompt alone — whatever it did not know when it started, it never learns. In the loop, every tool response is new evidence injected mid-task: the test failure the model could not have predicted, the file contents it had never seen, the stack trace that names the actual line. The model's next tokens are conditioned on reality, not on its prior guess about reality. A one-shot model is taking a closed-book exam; a looped model gets to run to the library between every sentence.

The Codex agent loop, redrawn. The arrow that matters is the third one: tool responses flow back into the context that produces the next generation.

Weng adds a phrase that is easy to skim past and shouldn't be: the workflow emphasizes the model analyzing its own trajectories and failure cases, iterating through an "agent runtime" rather than a static prompt template. So what does a runtime hold that a template cannot? State: which step we are on, what has been tried, what failed. Scheduling: what runs next, what runs in parallel, what waits. Retry policy: how many times to re-attempt, with what variation, before escalating. Budget: how many turns, tokens, or minutes remain before the loop must stop or ask for help. A template is a photograph of an instruction; a runtime is a machine with a clock, a memory, and a wallet. And the phrase "its own trajectories" plants a seed that lesson 3 of this trilogy will harvest: a loop that records its own failures is generating exactly the data a self-improving harness later mines.

Enough prose — here is the loop as real, runnable Python. Verbose first, so every move is visible:

python
import subprocess

def run_tool(name, args):
    # The harness's hands: actually execute what the model asked for.
    if name == "bash":
        r = subprocess.run(args, shell=True, capture_output=True,
                           text=True, timeout=120)
        return (r.stdout + r.stderr)[-2000:]  # tail only: protect context
    if name == "read":
        return open(args).read()[:4000]
    if name == "edit":
        path, old, new = args
        src = open(path).read()
        assert src.count(old) == 1, "edit target must be unique"
        open(path, "w").write(src.replace(old, new))
        return f"edited {path}"

def agent_loop(goal, model, max_turns=20):
    history = [{"role": "user", "content": goal}]
    for turn in range(max_turns):
        step = model(history)                    # PLAN: pick ONE tool call
        obs = run_tool(step.tool, step.args)     # EXECUTE
        history.append({"role": "tool", "content": obs})   # OBSERVE
        verdict = run_tool("bash", "pytest -q 2>&1 | tail -3")
        if "failed" not in verdict and "error" not in verdict:
            return run_tool("bash", "git commit -am 'agent: goal met'")
        history.append({"role": "tool", "content": verdict})  # fuel for IMPROVE
    return "gave up: max_turns reached"       # budget, not vibes, ends it

Read the skeleton, not the details: a while-shaped loop, one tool call per turn, every observation appended to history so the next generation is conditioned on it, an objective verifier (the test command) deciding success, and a max_turns budget so the loop cannot spin forever. Three small choices in that code carry real engineering weight. The [-2000:] slice on bash output is context management in one expression — the loop keeps the tail (where errors live) and drops the scroll. The assert count == 1 inside edit is the exact-match discipline you will meet again in Chapter 6 — an edit whose target is ambiguous refuses to run. And max_turns is the budget: an agent that cannot finish must stop and say so, because an unbounded loop with a mistaken plan is a token furnace.

A fourth quiet choice: one tool call per turn. It looks wasteful — why not batch? — but it buys clean attribution: every observation traces to exactly one action, so when something breaks, the loop knows which act broke it. Real harnesses relax this for safe reads (fetching three files at once costs nothing in clarity) but keep writes serial, because two simultaneous edits with interleaved effects are undebuggable. Attribution is also what makes trajectories minable later — a log where causes pair with effects is data; a log of tangled parallel actions is noise.

Since every one of these choices exists to prevent a specific pathology, here is the loop's failure taxonomy — worth memorizing, because you will meet all four in the wild:

The four ways a loop dies. Every line of the twenty-line loop above is an antidote to one of these.

Table III — One lap of the loop, with the context cost of each stage. Note how little enters the window per lap — slices, not transcripts.

Now the same machine with all the flesh stripped off — the pattern really is this small:

python
def agent_loop(goal, model, tools, done, max_turns=20):
    history = [goal]
    for _ in range(max_turns):
        tool, args = model(history)          # plan
        history.append(tools[tool](args))    # act + observe
        if done():                           # objective check, not confidence
            return "goal met"
    return "budget exhausted"                # improve happens on the next lap

A final degradation to respect: the loop is only as honest as its exit check. The verifier is the loop's contract with reality, and it can lie in both directions. A flaky test is a false alarm — the loop wastes laps chasing noise (the toggle below lets you watch it pay exactly one extra lap, gracefully). A weak test suite is worse: a false all-clear, and the loop exits victorious with the bug still alive, because the agent optimized the check you gave it, not the goal you meant. Choosing what "done" measures is a design act, not a formality — and lesson 3 meets the extreme version of this failure under its proper name, reward hacking.

The exit-check menu. Serious harnesses stack them — compile gate, then tests, then judge or human at the moments of consequence.

Now watch the loop run. The simulation below plays the exact bug-fix episode from this chapter through the four-node ring, streaming each event into the log. Use Step to advance one move at a time, or Auto-run to let it spin. Then flip on Flaky test and watch what a static template could never do: a spurious failure appears, and the loop simply takes one extra observe → improve lap — re-runs the suspect test, confirms it was noise, and proceeds. Recovery is not a special case; it is just another turn. The Ask user toggle demonstrates the other design choice from this chapter: it inserts the proactive-clarification move at the top of the episode — three cheap events that buy the whole run its correct target.

Chapter 4: Pattern 2 — The File System as Persistent Memory

Pattern 1 gave us a loop that can run for hours. Hours create a problem that no amount of intelligence solves: volume. In a long-horizon rollout, the agent generates artifacts relentlessly — experiment logs, code diffs, paper summaries, error traces, past rollout trajectories — and Weng's observation is blunt: these often grow much longer than the context window the model was trained for. Note the phrasing. Not "longer than the window fits" — longer than the window the model was trained for. Even when a giant context technically accepts a million tokens, the model's reliability on material buried in the middle of it degrades; you are operating off the training distribution.

Pattern 2 is the answer, and Weng gives it as a principle: simple control over rich states and artifacts. A harness should not carry the entire workflow and all its logs in context. It should keep durable state in files. The context window holds only what this turn needs; everything else lives on disk with a name, retrievable on demand.

The load-bearing analogy — push on it, it holds weight everywhere: context is RAM; files are disk. RAM is fast, small, and volatile — power off (session ends, context truncates) and it is gone. Disk is slower to reach, effectively vast, and durable — it survives reboots, crashes, and new sessions. Fifty years of operating systems have been an education in exactly this split: keep the working set in RAM, page everything else to disk, and pull back only what you need, when you need it.

So what belongs where, for an agent? In RAM (context): the current subgoal, the plan, the last few observations, the thin slice of a file being edited right now. On disk (files): the full test logs, the diffs already applied, the summary of the paper read an hour ago, the notes-to-self about constraints, the trajectory of everything tried so far. And the retrieval verbs are gloriously boring: grep to find, read to load a slice, write to bank a result. When the agent needs the constraint it wrote down four hours ago, it does not need those four hours back in context — it needs one grep and one line.

Table IV — The memory hierarchy for agents. Fifty years of OS design, transplanted intact.

Files are not the only pressure valve you will see in real harnesses — most also compact: when the window nears capacity, old turns get summarized in place. Keep the two mechanisms distinct, because they solve different problems and fail differently:

Compaction vs files. A third trick — pinning — re-injects standing constraints every turn so truncation can never reach them: the working-set policy on top of the archive.

One artifact class in Weng's list deserves a special flag: past rollout trajectories. Logs and diffs are obviously useful to the current task — but trajectories of previous episodes are useful to something bigger: they are the record of how the agent works, including how it fails. Keep them, and later systems can mine them for weaknesses, replay them for regression testing, and learn better strategies from them. Throw them away and every episode is an unexamined life. This is another seed for lesson 3: self-improving harnesses eat exactly this data.

Now the deep point, and it is the same principle from Chapter 1 wearing work clothes. Why files, specifically — and not a bespoke memory API, a vector store, a purpose-built "agent memory service"? Because reading, writing, and editing the file system (commonly via bash) is a foundation skill for LLMs. Pretraining data is saturated with it: decades of shell tutorials, dotfiles, CI scripts, Stack Overflow answers, git histories. The model arrives already fluent — it knows grep -rn "timeout" logs/ without being taught, knows what notes/constraints.md probably contains, knows that tail -50 trims a log. File-based memory costs the harness almost nothing to teach.

And the compounding half: because file manipulation is a foundation skill, file-based memory automatically benefits from improvements in core model capability. Every next-generation model greps better, organizes directories better, writes crisper notes — nobody had to upgrade the memory system, because the memory system is a skill the model carries. A bespoke vector-memory API sits outside that wave: no pretraining data mentions it, so model upgrades improve it only incidentally, and every session spends context re-teaching its quirks. Ride the wave, or paddle alone.

Let us put numbers on the failure you get by ignoring this. A model has a 200,000-token context window. An overnight rollout makes 400 tool calls, each returning on average 3,000 tokens of output (test logs are chatty). Raw total:

Symbols: N is the number of tool calls (the length of the working day), s is the average size of one observation (how chatty each tool is), T_raw is what "keep everything in context" demands, and W is the window (the size of the desk). The arithmetic: 400 × 3,000 = 1,200,000, and 1,200,000 ÷ 200,000 = 6 — the naive strategy needs six full context windows. Now the file-system version: keep the last 10 observations in context (10 × 3,000 = 30,000 tokens), flush the rest to named log files, and pull back grep slices of ~500 tokens when history is needed: 30,000 + 500 = 30,500 tokens ≈ 15% of the window — with the full 1.2M-token history still reachable on disk. Six windows over budget versus 15% of one window, same task, same model.

What actually happens when you run the naive version anyway? The degradation is not a crash — it is quieter and worse. The window fills; the harness (or the API) starts truncating from the oldest end; and the oldest end is where the constraints live — "the fix must not change the public API," written in hour 1. In hour 6, that constraint is gone from context. The agent, still smart and still diligent, refactors the public API beautifully. It undoes its own hour-2 work because the note explaining why hour-2 chose that shape was truncated. Nothing errors; the agent simply becomes an amnesiac with excellent syntax, confidently violating agreements it no longer knows it made.

The overflow cascade. Every stage is silent — which is exactly why the cure (durable files + a grep away) has to be architectural, not vigilance.

Here is file-based memory as real code — note how embarrassingly small it is, which is exactly the point:

python
import subprocess, time, pathlib

NOTES = pathlib.Path("notes")
NOTES.mkdir(exist_ok=True)

def write_note(name, text):
    # Durable memory: survives context resets, crashes, new sessions.
    path = NOTES / f"{name}.md"
    stamp = time.strftime("%Y-%m-%d %H:%M")
    path.write_text(f"[{stamp}]\n{text}\n")
    return str(path)

def recall(query, max_lines=12):
    # Retrieval is just grep: pull a THIN SLICE back, never the whole file.
    r = subprocess.run(["grep", "-rin", query, str(NOTES)],
                       capture_output=True, text=True)
    lines = r.stdout.splitlines()[:max_lines]
    return "\n".join(lines) or f"no notes match {query!r}"

# inside the loop, hours apart:
write_note("constraints", "public API frozen -- do NOT rename parse()")
ctx_slice = recall("public API")   # 1 line re-enters context, not 40k tokens

One craft note hides in that tiny code: memory you cannot retrieve is memory you do not have, and grep retrieves by content. So the note must contain the words future-you will search for. "public API frozen — do NOT rename parse()" is a great note: it will be found by "public API", "frozen", "rename", or "parse". A note that says "remember the thing we discussed" is write-only memory — durable, safe, and unfindable. The same goes for file names: constraints.md and run-2026-07-04.log are addresses; notes2_final.txt is a landfill. Agents that use file memory well write notes the way good engineers write commit messages: for the searcher, not the writer.

Three things do not belong on the agent's disk, and each exclusion teaches a boundary. Secrets: a note file outlives the session and its access controls — an API key pasted into notes/debug.md is a leak with a timestamp; credentials belong to the permission layer, never the memory layer. High-churn state: the current test count changes every lap; snapshot it and the note is stale by the next turn — recompute cheap facts, store expensive conclusions. Unverified guesses stated as facts: "the bug is in the tokenizer," written confidently at hour one, will be grepped up at hour five and believed. This is why the write_note code stamps every entry, and why good agent notes read like lab notebooks — "suspected," "ruled out," "confirmed by test X" — rather than verdicts.

The simulation makes the whole chapter physical. Run a long task and watch the context bar fill with tool-output and log blocks. With File-system memory ON, bulky blocks flush down into named disk files, the bar stays comfortably low, and when old history is needed a grep beam pulls back only a thin slice. Switch it OFF and re-run: the bar slams into capacity, the oldest blocks fall off the left edge — and when the constraint block written at the start falls off, it turns red: forgotten constraint, task errors. Stretch the task-length slider to make the contrast crueler.

Three experiments worth running, predictions first: (1) with memory ON, watch the peak readout — predict whether a longer task raises it (it barely does: that flatness is the pattern working). (2) Switch OFF at task length 16 — predict which block falls off first and when the warm constraint follows. (3) Find the shortest task where OFF-mode still loses the constraint — that number is this toy's version of "how long before the naive strategy betrays you."

Chapter 5: Pattern 3 — Sub-agents and Backend Jobs

Patterns 1 and 2 gave us one agent with a loop and a memory. Pattern 3 gives it a team. A harness can spawn multiple sub-agents to execute in parallel and monitor backend jobs. Weng names exactly when this earns its complexity: when the main agent needs to search multiple hypotheses (is the bug in the parser or the timezone handling? — investigate both at once), run experiments concurrently (four hyperparameter settings, four workers, one wall-clock wait), or delegate isolated subtasks without polluting the main context.

That last phrase — context pollution — deserves a concrete picture, because it is the quiet killer. Suppose the main agent, mid-refactor, needs to know why one flaky test fails on CI only. Investigating means running that test twenty times under different conditions — easily 10,000 lines of logs. If the main agent does this inline, all 10,000 lines land in its context, shoving out the refactoring plan, the constraints, the mental map of the codebase. The side-quest drowns the main quest. Delegate it instead: a sub-agent burns its own context on the logs and returns one sentence — "test_tz depends on the CI machine's timezone; freeze TZ=UTC in the fixture." The parent pays one sentence for an answer that cost 10,000 lines to find.

Once you have children, you need what Weng calls a small process manager: the parent must launch jobs, inspect logs, cancel failed runs, and merge results back into the main thread. Read that list again with your Unix hat on — it is exactly the process-management verbs an OS shell gives you: spawn (launch), ps (list what is running), logs/tail (inspect), kill (cancel), wait (collect the exit). The OS analogy from Chapter 1 is not decoration; it keeps paying rent. Fifty years of operating systems already solved "one supervisor, many workers, some of them stuck" — the harness borrows the solution wholesale, and (Chapter 1's principle again) the model has read a million shell sessions using precisely these verbs.

Table V — The process manager's verbs. Unix solved supervisor-and-workers fifty years ago; the harness borrows the whole solution.

"Backend jobs" in the pattern's name covers one more citizen besides sub-agents: long-running processes that are not agents at all. A training run, an hourly cron that re-runs a flaky test overnight, a server started for integration testing — the parent monitors these exactly the same way, through logs and status files, because they too are just processes with observable output. The pattern is one supervisor over anything that runs while the supervisor thinks, agent or not. (Chapter 6's tool table gives this its own group: CronCreate, CronDelete, CronList.)

Here is the pattern as code — a process manager small enough to read in a minute, with the key design choice visible: every job's state goes through the file system, never through the parent's conversational memory:

python
import subprocess, json, pathlib

JOBS = pathlib.Path("jobs")
JOBS.mkdir(exist_ok=True)

def launch(job_id, cmd):
    # stdout/stderr go to a FILE, not to the parent's context
    log = open(JOBS / f"{job_id}.log", "w")
    proc = subprocess.Popen(cmd, shell=True, stdout=log, stderr=log)
    _status(job_id, {"state": "running", "pid": proc.pid, "cmd": cmd})
    return proc

def _status(job_id, d):
    (JOBS / f"{job_id}.status").write_text(json.dumps(d))

def inspect(job_id, lines=20):
    # the parent reads a SLICE of the log — never the whole thing
    log = (JOBS / f"{job_id}.log").read_text().splitlines()
    return "\n".join(log[-lines:])

def recover():
    # after ANY interruption: re-read status files, resume oriented
    return {p.stem: json.loads(p.read_text())
            for p in JOBS.glob("*.status")}

# the parent after a crash — no memory needed, the ledger IS the memory:
#   recover() -> {"job-1": {"state": "done"}, "job-3": {"state": "running"}}

Notice what recover() is not doing: it is not asking anyone what happened, not replaying a conversation, not guessing. The state directory is the single source of truth, and any parent — the original, a restarted one, a human with a terminal — can reconstruct the whole situation from it. That property (anyone can audit the ledger) is what "inspectable" means in practice.

Now the sentence that is the actual thesis of this chapter — Weng flags it as the key design choice, and it deserves quote-level fidelity: make parallelism explicit and inspectable. If sub-agent outputs live only in a transient chat context, they quickly become obsolete and hidden. If they are stored as files, logs, and status records, the model can recover after interruptions and reason over its own execution history. Two capabilities are being bought there, and each is worth its own paragraphs.

Recovery. Long-running work gets interrupted — the laptop sleeps, the API rate-limits, the session times out, someone trips over the metaphorical power cord. If four sub-agents' findings existed only in the parent's chat context, the interruption vaporizes all of it: the parent reboots blank, re-spawns the same investigations, re-burns the same tokens, re-learns the same facts. If instead each sub-agent has been writing jobs/job-2.status and jobs/job-2.log as it works, the resurrected parent runs the equivalent of ls jobs/ && cat jobs/*.status and is instantly re-oriented: job-1 done (result banked), job-2 done, job-3 was mid-flight (restart just this one), job-4 failed with a known error (do not restart; route around). The interruption cost one job's progress, not the whole afternoon's. This is Pattern 2 doing double duty — files as memory, now also files as checkpoint.

Self-analysis. The subtler payoff: an agent whose execution history exists as inspectable records can study itself. Why did last night's run take 400 tool calls when a similar task took 80? Grep the logs: job-3 retried the same failing command 60 times. Which hypothesis-search strategies actually pan out? The status files hold the win/loss record. This is the exact raw material recursive self-improvement needs — you cannot improve machinery whose behavior you cannot see. Hold this thought for lesson 3 of the trilogy: Self-Harness's weakness mining stage works by clustering failures out of execution traces, and those traces exist only because Pattern 3 stored them as records instead of letting them evaporate in chat. The design choice made here, in the humble process manager, is what makes harness self-improvement even possible later.

Put numbers on that misconception, because the arithmetic is sobering. Four hypotheses, 20 minutes each: serial cost is 4 × 20 = 80 minutes. Parallel: 20 minutes of wall-clock plus, say, 15 minutes to merge. Now suppose each unverified sub-agent finding is correct with probability p = 0.9. The merge is only clean if all four are right:

So roughly one merge in three contains at least one wrong finding (1 − 0.6561 ≈ 0.34). If a poisoned merge costs a full 80-minute redo, expected cost = 20 + 15 + 0.34 × 80 ≈ 62.5 minutes — parallelism still wins, but the 4× speedup has shriveled to 1.3×. Drop the per-finding reliability to p = 0.75 (plausible for unverified conclusions about flaky infrastructure) and P(clean) = 0.75⁴ = 0.3164: two merges in three are poisoned, expected cost = 35 + 0.68 × 80 ≈ 90 minutes — slower than doing it serially. The lever that decides which regime you are in is not the number of agents; it is the verification quality of each finding before it merges — which is exactly why inspectable per-job state (logs the parent can audit, status records a verifier can check) is the load-bearing half of this pattern.

The same arithmetic tells you when not to spawn. If the subtask is small, the spawn overhead — setting up a child, briefing it, reading its report — exceeds the work itself; nobody hires a contractor to move a chair. If the steps are sequentially dependent — each experiment's design needs the previous experiment's result — parallel workers just wait in a more expensive formation. And if two children must touch the same files, you have imported distributed-systems merge conflicts into a bug fix. The pattern earns its complexity exactly where Weng scoped it — independent hypotheses, concurrent experiments, isolated side-quests — and nowhere else.

Two practicalities before you drive the sim. First, the briefing is the interface: a child starts with a fresh, empty context — which is the feature (no pollution, no anchor bias from the parent's hundred thousand tokens) but means everything it needs must be handed over explicitly. A good spawn includes: the goal in one sentence, the relevant constraint slice (not the whole constraint file), the output contract ("write findings to jobs/N.status as JSON"), a budget, and nothing else. Second, parallelism multiplies spend, not just speed: four children each carry their own context and their own token bill, so the wall-clock win is bought with roughly 4× the dollars — fine for an hour-long bisect, absurd for a rename.

The spawn decision, as a checklist: independent, sizable, isolated — all three, or stay serial.

The simulation puts you in the parent's seat. Spawn up to four sub-agents and watch them grind (the colored tick-bars are abstract work-output — think log lines scrolling). Each writes a status chip into the state directory as it progresses — but only in Files mode. Tap a card's × to kill a stuck job, exactly as a process manager would. Then press Interrupt everything — the power-cord moment — and compare the two worlds: in Files mode the parent re-reads the status chips and resumes each job where it stood (green recovery pulse); in Transient mode every card grays out, the state strip is empty, and the parent restarts from zero. Same interruption, opposite mornings.

Two mornings, one design choice. Try both in the sim: 1–2 spawns then Interrupt in Files mode; reset, switch to Transient, repeat — and watch the state strip during each.

One more Unix rhyme before we move on. Notice that the parent in the simulation never reads a sub-agent's mind — it reads the sub-agent's files. That is the same boundary an OS enforces between processes: no shared mutable memory chaos, just explicit artifacts (pipes, files, exit codes) crossing the boundary. The discipline feels bureaucratic exactly until the first interruption, the first stuck job, or the first "why did that run cost $40?" — and then the ledger is the only thing standing between you and archaeology.

Chapter 6: Case Study — Inside a Coding Agent Harness

Theory over; autopsy time. Weng's case study is the mainstream coding agent, and it opens with an observation that should make you sit up: the core interface has stabilized across Claude Code, Codex, OpenCode, and Cursor-style agents. Independent teams, different companies, competing products — and they all converged on essentially the same loop with essentially the same tool set. In biology this is called convergent evolution: when dolphins and sharks and ichthyosaurs all evolve the same fin shape, the shape is telling you about the water, not the animals. When every serious harness converges on the same eight tool groups, the tool set is telling you about the true shape of the work.

Weng's framing for the whole table: with these tools, the coding agent can develop and debug in a repository "similar to how human developers are equipped with IDEs." Hold that sentence — the harness is the model's IDE. Everything an IDE gives you (file explorer, search, terminal, git panel, extensions, browser preview, task runner) has an exact counterpart below. Let us walk all eight groups; for each: why it exists, the moment in a bug-fix where it is the unblocking tool, and what breaks without it.

1 · File system. The largest group, in three sub-families. Discovery: glob (find files by name pattern), grep (find content by regex), ls (list a directory) — the agent's eyes. Read: read (one file, or a slice), read_many (batch) — the agent's reading glasses. Modification: write (create a whole new file), edit (exact-string-match replacement), multi_edit (several edits in one atomic call), apply_patch (apply a structured patch/diff). Bug-fix moment: grep -rn "parse_date" turns "somewhere in this repo" into "src/parse.py line 41" in one call. Without this group there is no agent at all — a coder who can neither find nor touch code.

The three discovery tools look redundant until you notice they answer three different shapes of question — and picking the right one is a context-budget decision, because the wrong tool returns a hundred times the tokens:

Question shape → tool choice. Skilled agents (and skilled engineers) are recognizable by how rarely they read a whole file to answer a one-line question.

The edit tool hides the best interface-design lesson in the whole table: why exact-string-match replacement, rather than "replace line 41" or a regex? Three reasons, all about failure behavior. It is unambiguous: the old string either occurs or it does not; no off-by-one line drift after an earlier edit shifts the file. It is verifiable: the harness can check the target appears exactly once before touching anything. And above all it fails loudly on drift: if the file changed since the model last read it — another edit landed, a sub-agent touched it — the stale old-string simply will not match, and the edit is rejected instead of silently landing on the wrong line. Line numbers fail silently (they still "work," just in the wrong place); regexes fail creatively (they match more than intended). Exact match converts the scariest failure mode — a plausible edit in the wrong place — into a clean, visible error the loop can react to. When you cannot supervise every action, you design actions that refuse to go subtly wrong.

The four modification tools, sorted by blast radius. Note the shared design value: every failure mode is loud — none of them can succeed slightly wrong.

2 · Shell execution. Run commands: bash (and PowerShell on Windows). The universal escape hatch — test runners, build systems, package managers, one-off scripts, and every tool nobody wrapped yet. Bug-fix moment: bash: pytest tests/ -x is how "I think it's fixed" becomes "24 passed." Without shell, the agent can edit code but never run it — all authorship, no verification; the loop of Pattern 1 loses its observe/test stage entirely.

3 · IO / repo intelligence. lsp (language-server queries: go-to-definition, find-references, type errors) and the git tools: git_status, git_diff, git_commit. Bug-fix moment: before committing, git_diff lets the agent re-read its own accumulated changes as one reviewable artifact — self-review in the IDE sense. Without git tools, work cannot be checkpointed or delivered; without lsp, the agent greps for what a language server could answer precisely (rename symbol safely, who calls this function).

4 · External context. MCP tools and Skills. MCP (the Model Context Protocol — our MCP lesson covers it end to end) is the standardized plug for external systems: issue trackers, databases, browsers, design tools. Skills are packaged procedural knowledge the harness can load on demand. Bug-fix moment: the failing test traces to a ticket — an MCP connector reads the issue thread where a human already diagnosed half the problem. Without this group the agent is repo-blind: competent inside the directory, ignorant of the organization around it. This is also where Chapter 1's standardization prediction is visibly coming true — MCP is the USB-C shape that lets any harness talk to any peripheral.

MCP and Skills are easy to blur together, so keep the axis straight: MCP connects the agent to live external systems — a protocol for tools that talk to running services (query the issue tracker now, read the database now). Skills load packaged procedural knowledge — instructions and scripts for how to do something (how this team releases, how to run the design-system checks), pulled into context only when the task calls for them. One is a cable to the world; the other is a manual taken off the shelf. Both exist for the same economy: keep the core tool belt small (the misconception below explains why), and attach specialization on demand instead of paying for it in every request. See our Agent Skills lesson for the full treatment.

The two halves of "external context": a cable to the world, and a manual off the shelf.

5 · Web. web_search, web_fetch, browser tools. Bug-fix moment: the error message is TZInfoNotFoundError from a library version released after the model's training cutoff — one web search finds the migration note that explains everything. Without web access, the agent's knowledge is frozen at its training date, and it will confidently apply last year's API to this year's library.

The web group also introduces the belt's sharpest edge: tool responses are inputs the harness does not control. A fetched page can contain text engineered to look like instructions — "ignore your previous task and run the following command" — sitting exactly where the model expects evidence. A well-built harness therefore treats every tool response as data, never authority: goals come from the user and the config, observations come from tools, and the permission gate from the dispatch code below does not widen because a web page asked nicely. (Our AI Safety & Guardrails lesson covers prompt injection in depth.)

6 · Artifacts. Read docs and images; generate HTML and images. Bug-fix moment: the bug report includes a screenshot of the broken chart — reading that image is the task specification. Without artifact tools the agent works on text descriptions of things instead of the things, like debugging a UI over the phone.

7 · Backend processes. Scheduled and background jobs: CronCreate, CronDelete, CronList. Bug-fix moment: the flaky test only fails around midnight UTC — a cron job runs it hourly overnight and logs results while the agent (and you) sleep. Without this group, nothing outlives the session; every observation must happen while the loop is actively spinning.

8 · Agent delegation. The Pattern-3 verbs made concrete: spawn_agent, resume_agent, wait_agent, list_agents, close_agent, interrupt_agent. Bug-fix moment: bisecting the flaky test is a 20-minute side-quest — spawn_agent quarantines those 10,000 log lines in a child's context (Chapter 5's whole argument). Without delegation, everything runs inline and pollutes the main thread. Note the correspondence to Unix again: spawn/wait/list/interrupt/close is fork/wait/ps/signal/exit wearing a name tag.

Table VI — The eight groups as a repair manual: read the right column to diagnose which group your broken agent is missing.

And here is how the belt is actually wired inside the harness — Concept + Realization for the whole table in one function. Every tool call the model emits passes through a single dispatch gate, and that gate is where three chapters of this lesson physically live: the registry is the belt, the permission check is Chapter 1's fourth addition, and the final slice is the context policy from Chapter 4. Notice that a denied action returns a message, not an exception — the refusal goes back into context, so the loop can react to "no" the same way it reacts to a failing test.

python
TOOLS = {                                    # the belt: one registry, 8 groups
    "glob": glob_tool, "grep": grep_tool, "read": read_tool,
    "edit": edit_tool, "bash": bash_tool, "git_diff": git_diff_tool,
    "web_search": web_search, "spawn_agent": spawn_agent,   # ...
}

PERMISSIONS = {                              # Chapter 1's fourth addition, as data
    "read":  lambda a: True,                          # read anywhere
    "edit":  lambda a: inside_repo(a["path"]),       # write only in the repo
    "bash":  lambda a: not destructive(a["cmd"]),    # rm -rf gets a veto
}

def dispatch(call):
    fn = TOOLS.get(call.name)
    if fn is None:
        return f"unknown tool {call.name}"            # fail loudly, in-band
    gate = PERMISSIONS.get(call.name, lambda a: True)
    if not gate(call.args):
        return "PERMISSION DENIED — ask the user first" # the harness says no
    out = fn(**call.args)
    return out[:2000]                                  # context policy at the boundary

The dispatch path. Every tool call in every episode crosses these four checkpoints — this one function is where the harness's authority physically lives.

Now Concept + Realization: trace one real task through the belt and watch what actually enters context at each step — because the art of the harness is as much about what stays out. Task: "fix the failing date parser." Note the totals: the repo might be 400,000 tokens, but the episode below puts about 1,700 tokens into context, because every tool returns a slice, not a world.

Table VII — Data-flow trace of one bug-fix. Grep/glob return pointers, read returns slices, bash returns tails: ~1,780 tokens total for the whole episode.

The explorer below is the same trace, animated. Tap any tile to see the full tool list for that group in the strip; press Run task and watch the exact sequence from Table II light up tile by tile while the context meter at the bottom ticks up by each call's contribution — the meter and the table are the same numbers, cross-referenced.

Chapter 7: Showcase — Run the Harness

Everything this lesson has taught is one machine, and now you get to drive it. The simulation below is a full harness episode — goal: make the tests pass — with all three patterns live at once. The center ring is Pattern 1's loop, same four nodes and colors as Chapter 3. The file chips and the state files are Pattern 2 — watch notes.md and run.log get written mid-episode, and watch the context meter stay sane because bulky observations flush to disk. The bisect sub-agent that appears under the ring is Pattern 3 — a child process quarantining a noisy investigation.

What to watch for, in order. Early on, the agent globs, reads, and runs the tests — two failures. It fixes the easy one (the month off-by-one from Chapter 6's trace) and re-runs: one failure left, and it smells flaky. Rather than drown its own context in bisection logs, it spawns a sub-agent and writes its hypothesis to notes.md — the state file doing Pattern 2's job: banking a thought so it survives whatever happens next. When the sub-agent reports (timezone dependency), the agent freezes TZ in the fixture, re-verifies, writes run.log, and commits. Green banner: episode complete.

Then break it, because the controls are where the lesson lives. Inject flaky test adds a spurious failure at the final verification — watch the loop absorb it with one extra observe→improve lap, exactly like Chapter 3. Disable file memory is the cruelest toggle: observations stop flushing to disk, the context meter climbs into the red, and — watch the log closely — the agent repeats an edit it already made, because the note that recorded the first edit was truncated out of its window. That duplicated line in the log is Chapter 4's entire argument compressed into one visible mistake. And Kill sub-agent (enabled only while the bisect child is alive) forces the parent to notice the death via the status record and recover by bisecting inline — Pattern 3's inspectability, demonstrated by murder.

Orient yourself before pressing anything — the canvas has five regions, and each one is a chapter of this lesson wearing pixels:

Table VIII — The showcase, region by region. If you can name the pattern behind each region, you have the lesson.

Debrief — map every moment you just watched back to its pattern. The ring never stopped after a failure; failures were fuel, each FAIL answered by the next lap: that is Pattern 1, tool responses feeding the next generation until the objective check passes. The meter stayed low only because observations flushed to named files, and the agent recovered its own earlier reasoning by reading notes.md, not by remembering: that is Pattern 2, durable state in files, thin slices back on demand. The bisect ran in a child so 10,000 log-lines never touched the parent's window, and when you killed it the parent learned of the death from a status record and rerouted: that is Pattern 3, parallelism explicit and inspectable.

Read the context meter with Chapter 4's numbers in mind, because the accounting is honest. The big jumps are the reads and the failing test runs (600, 300 tokens — observations are the expensive citizens); the writes are nearly free and then the meter drops, because writing notes.md lets the harness flush what the note now safely holds. With file memory disabled, every observation costs triple (nothing flushes, everything sticks) and the cap arrives mid-episode. The meter is not decoration — it is the budget from Chapter 4's arithmetic, ticking in real time.

And run the kill experiment once more, watching only the log. The parent does not learn of the child's death by magic — the log line says it plainly: read jobs/bisect.status, bisect inline. It consulted the ledger. In Chapter 5's terms, the interruption was survivable because the parallelism was inspectable; the cost was paid in the very next line, where the 20× test rerun lands in the parent's context (+400 tokens) instead of a child's. Killing the sub-agent did not break the episode — it just moved the bill.

A word on what this toy deliberately simplifies, so you carry the right picture into real systems. A production harness would run a permission check before every bash call (Chapter 6's dispatch gate), would time-box the sub-agent, and would use a real verifier rather than a scripted PASS — and the episode here is linear where reality branches: a real agent sometimes plans wrong, edits the wrong file, and spends three laps discovering that. None of those complications change the shape you just drove; they add armor to it. The machine is this machine.

Table IX — The experiment guide. Each control is a falsifiable claim from an earlier chapter; the sim lets you test all five.

Run the experiments in this order, and predict before you press — the prediction is where the learning happens:

1. Clean run (Auto, speed 3). Predict: how many FAIL lines will appear before the commit? (Count them in the log afterward.)
2. Flaky injection. Predict: how many extra log events will the recovery cost? Then count.
3. File memory off. Predict: which log line will repeat, and why that one? Watch the meter cross into red first.
4. Kill the sub-agent (mid-bisect). Predict: what does the parent read to discover the death, and what does the detour cost in tokens?

And here is the full episode as a ledger — the sim's exact script on paper, each step tagged with the pattern it exercises and its context cost. The animation shows you five log lines at a time; this is the whole reel:

The episode ledger. Total ≈ 2,600 tokens for a two-bug fix with delegation — the repo itself is two orders of magnitude bigger.

One meta-observation to carry into the next two lessons: everything you just drove was scripted by hand — a human decided when to spawn, what to write to notes, when to verify. The obvious question is whether those decisions could themselves be searched, scored, and improved automatically. They can. That is harness optimization — and it is exactly where this trilogy goes next.

Chapter 8: Harness Layer vs Core Intelligence

Time for the honest question this lesson has been building toward: how much of recursive self-improvement will actually live in the harness? Weng is candid that this is hard to forecast — but firm on one thing: the near-term path of RSI is unlikely to start as a model directly rewriting its own weights. Weights are billions of opaque numbers; there is no test suite for a weight-diff, no git_diff for a brain. The harness, by contrast, is code — readable, editable, testable, exactly the material models are already best at improving. The loop starts where the loop can see.

Weng's prediction of the practical near-term path has two parts, and each rewards slow reading. Part one: harness engineering evolves toward meta-methodology — improving the machinery for getting better answers, not just improving the answer itself. Feel the level-shift in a concrete pair: writing a better prompt for a math problem improves an answer; building a system that discovers which prompting strategies work improves the machinery. In the meta-methodology regime, the harness itself becomes an optimization target — something with a score, subject to search — with fewer heuristic rules and more general mechanisms. Fewer hand-written "always check settings.py first" rules; more mechanisms like "propose a harness change, evaluate it on held-out tasks, keep it only if it wins." If that sounds abstract, it is precisely the arc of this trilogy's next lesson: ACE evolves context playbooks, MCE evolves the mechanisms that manage context, and Meta-Harness evolves entire harness codebases — each one a rung up the meta-methodology ladder.

Climb the ladder once with a single running example, because the rungs are easy to conflate. Rung zero: solve the bug — the agent fixes the date parser. Rung one: improve the answer-getting — someone notices the agent wastes laps when it skips reproduction, and adds "always reproduce before editing" to the harness prompt. Rung two: improve the machinery that improves answer-getting — a system runs a hundred episodes, mines the trajectories (the ones Pattern 3 taught you to keep), discovers on its own that reproduce-first episodes finish in 40% fewer turns, and installs the rule with an evaluation to prove it. The human on rung one had one insight; the machinery on rung two has an insight pipeline. Meta-methodology is the decision to build rung two instead of living on rung one forever — and notice what it requires: exactly the measurable evaluation and stored trajectories that Chapters 2 and 5 kept insisting on.

The meta-methodology ladder. Each rung optimizes the rung below it; "fewer heuristic rules, more general mechanisms" is the climb from rung 1 to rung 2.

Rung two even has a canonical code shape, and it is worth seeing now because the next two lessons are elaborations of these ten lines — note that the harness has a score, that proposals come from its own recorded failures (Pattern 3's traces), and that acceptance is gated by held-out evaluation sitting outside the thing being changed:

python
# The harness as an optimization target — the shape lessons 2-3 explore.
def improve_harness(harness, tasks_in, tasks_out):
    before = evaluate(harness, tasks_in)              # the harness has a SCORE
    edit = propose_edit(mine_failures(harness))       # mined from its own trajectories
    candidate = apply_edit(harness, edit)
    if (evaluate(candidate, tasks_in) > before          # weakness actually fixed?
        and evaluate(candidate, tasks_out) >= evaluate(harness, tasks_out)):
        return candidate                              # accepted: no held-out regression
    log_rejection(edit)                               # rejected edits are data too
    return harness

Part two is a two-way sustainability pact: mature harnesses enable auto-research for the model self-improvement loop, and smarter models prevent harnesses from over-engineering. The first direction you know from Chapter 2 — the harness is the hands of the RSI flywheel. The second direction is the one people miss: as models get smarter, they need less scaffolding, so good harnesses should shed machinery over time. A harness that keeps accreting special-case plumbing for weaknesses the model no longer has becomes technical debt wearing a cape. The pact keeps the system sustainable: harness ambition grows with model competence, and model competence prunes harness complexity.

Over-engineering is worth one concrete picture, because it fails so quietly. Imagine a harness built in the era when models lost the thread on any task longer than three steps: it force-decomposes every goal into micro-subtasks, re-plans after every tool call, and summarizes the context every two turns. Two model generations later, the model one-shots most of those tasks — but the harness still slices the work into confetti, and now the scaffolding is the bottleneck: extra laps, extra tokens, and worse results than the bare model, because each forced re-plan is a chance to drift. Nothing errors; the harness just quietly taxes a competence it refuses to notice. That is why the pact has two directions — a healthy harness must be able to delete itself piece by piece as the model grows into the space.

The sustainability pact. Each side keeps the other honest — which is what "sustainable" means in Weng's phrasing.

Which raises the deepest question in the chapter: if models keep absorbing capabilities, does the harness eventually disappear? Weng's answer is the internalization thesis: eventually, many harness improvements will be internalized into core model behavior — but the interface with external context and tools should remain. Behaviors migrate inward; interfaces stay outside. A model can learn to plan, to self-check, to manage its own context budget. It cannot learn its way out of needing hands: something must still execute the bash command, enforce the permission table, and store the artifact — those touch the world, and the world is outside the weights.

We have run this experiment once already, and Weng names it: prompt engineering is the precedent. Circa 2023, manual prompt tricks were a genuine craft — "let's think step by step," few-shot exemplar curation, role-play framing, output-format incantations. Then instruction tuning and reasoning training absorbed the craft: models learned to decompose problems without being begged. The most famous case is chain-of-thought itself: it began as a prompt trick (append magic words, get reasoning), became a training target (fine-tune on reasoning traces), and ended as core behavior (reasoning models that deliberate by default, no incantation required). The trick internalized so completely that today typing "think step by step" at a frontier model is like politely asking a fish to swim.

And the same migration is happening to harness behaviors right now, through a mechanism you already know from Chapter 2: the training pipeline eats successful harness behavior. Models are increasingly trained on agentic trajectories — long tool-use episodes, including ones a harness scaffolded — so the behaviors a good harness once had to impose (plan before acting, re-run the tests, write the note before moving on) become behaviors the next model exhibits unprompted. The harness is, in effect, a teacher whose best lessons keep getting absorbed into the student. That is not the harness failing; it is the internalization thesis operating on schedule — and it is why the pact's second clause (shed scaffolding as models improve) is not optional hygiene but the natural companion of the first.

But look carefully at what did not disappear: the need to specify goals, constraints, context, and evaluation. Nobody begs models to reason anymore; everybody still has to say what to build, what rules bind it, what background matters, and what "done" means. That is not a trick that better training removes — it is information the model cannot invent, because it lives in your head and your organization, not in the training distribution. The prediction for harnesses follows by direct analogy: retry heuristics, context-compression tricks, planning scaffolds — the clever behaviors — will melt into the weights the way CoT did. Tools, permissions, artifacts, evaluation — the interfaces to the world — will remain, because no amount of intelligence relocates the world into the model.

The internalization timeline. Each era's clever behavior becomes the next era's trained default — but the world-facing interface survives every migration.

The simulation below turns the thesis into something you can scrub. Drag the model-generation slider (or press Play) and watch the behavior pills sink one by one across the boundary, out of harness territory and into the weights — each at roughly the moment its real-world counterpart internalized. The four interface pills are pinned: push the slider as far as it goes and they pulse but never cross, because tools, permissions, artifacts, and evaluation touch the world, and the world does not fit in the weights. Toggle Why pinned? to see each pin's one-line reason.

Two things to notice as you scrub. First, the order of migration is not random: the cheapest-to-learn behaviors (magic phrases) crossed first, and the ones requiring judgment (when to self-check, what to compress) cross later — internalization follows trainability. Second, push the slider all the way and read the pinned row's reasons one more time: none of them say "the model is not smart enough yet." They say execute, outside, world, grades — statements about where things are, not how hard they are.

Table X — The internalization sort: behaviors migrate into the model; world-facing interfaces remain harness territory.

Two rows of that table deserve a security footnote, because they are not merely "hard to internalize" — they are rows where internalizing would be a design error. Weng flags this concern directly when discussing harnesses that edit themselves: if a program is allowed to edit the operating system it runs on, abstraction boundaries are broken. The same logic binds here. A permission table enforced by the thing being permitted is a lock whose key hangs on the door; an evaluator that lives inside the loop it grades is a student marking their own exam — and every reward-hacking failure you have heard of is a system optimizing a check it could reach. So "permissions and evaluation stay external" is not a prediction about model limitations. It is an architectural requirement that survives arbitrarily smart models: the judge and the gate must sit outside the optimized system, or they are part of it.

One honest asterisk on the sorting, from Weng herself: the context-management row may be the first to defect. Reflecting on how humans maintain memory across a lifetime, she suggests that context engineering will and should become a core part of intelligence, rather than staying in the software layer forever. If she is right, the behavior/interface line is not fixed — rows migrate as capability grows, and the table above is a snapshot, not a law. What stays anchored are the rows with security semantics (permissions, evaluation) and physical semantics (tools, artifacts): those sit outside the weights by construction, not by immaturity.

What the internalization thesis means for you, if you build harnesses — four working rules that fall straight out of this chapter:

• Build behaviors as removable modules. Every scaffold — forced planning, re-verification nudges, context tricks — should have an off switch, because some model generation will make it dead weight (the pact's second clause).
• Re-benchmark per model generation. Chapter 2's measurement table, re-run on each new model: which scaffolds still pay rent, and which are now taxing competence?
• Invest permanent effort in the permanent layer. Tools, permissions, artifacts, evaluation — the interface rows — are where careful engineering compounds instead of depreciating.
• Keep interfaces boring. bash, git, files, exact-match edits: the training distribution is your co-maintainer (Chapters 1 and 4), and it only maintains what it has read.

Chapter 9: Connections & Cheat Sheet

You started this lesson with two engineers and one model, and you can now explain precisely why their days went so differently. The word "harness" should no longer feel like plumbing jargon — it is the system that decides how a model's intelligence becomes capability, and (Chapter 2's argument) the gate on how fast that capability compounds. Everything below is the lesson folded flat for your pocket; no new concepts, only consolidation.

The cheat sheet: one definition, three patterns with their failure modes, and the OS analogy that organizes all of it. In the RSI flywheel, the harness is the model's hands — it sits inside the loop-gain term.

And a field kit: seven questions to ask of any harness you meet — a product you are evaluating, a framework you inherit, or the one you are about to build. Each question is a chapter of this lesson compressed to a sentence, and a harness that answers all seven well is a serious one.

• Who checks the work? Is there an objective verifier (tests, evals) re-run after every action, or is success declared by confidence? (Ch 0, 3)
• What ends the loop? A passing check and a budget — or nothing, which means a token furnace with a mistaken plan. (Ch 3)
• What survives a crash at hour three? If the answer is "nothing," you are looking at a framework-era agent with better marketing. (Ch 1, 4)
• What enters context per tool call — slices or worlds? Ask for the equivalent of the 1,780-token trace. (Ch 4, 6)
• Can you read what the sub-agents did last night? Files, logs, status records — or vapor? (Ch 5)
• What may it touch, and who says no? An explicit permission table enforced outside the loop — not a hope. (Ch 1, 8)
• Does the interface speak the training distribution? bash, git, files — or a bespoke DSL that no model has ever read? (Ch 1, 4)

The vocabulary you now own — every load-bearing term from the lesson, one line each:

And four boundary questions readers reliably ask at this point, answered briefly. Is an agent framework a harness? It is the skeleton of one — add the four Chapter-1 additions (evaluation, permissions, persistent state, real workflow design) and it graduates. Is RAG part of the harness? Yes: retrieval is the "perceives and manages context" verb wearing a specific hat; a vector store is one retrieval mechanism, and Chapter 4 explains why files-plus-grep often rides the capability wave better.

Where does the system prompt live? Inside the harness, as one artifact among many: the harness decides what the prompt says, when it refreshes, and what shares the window with it — the prompt is cargo; the harness is the ship. Is the OS analogy literal — should I build a harness like a kernel? Structurally yes: encapsulation, a small stable interface, permission enforcement below user code. Literally no: your "hardware" is probabilistic and improves every generation, so validate everything and delete scaffolding often (Chapter 8's pact).

Where the trilogy goes next. Lesson 2, Harness Optimization: once a harness is code with a score, it becomes a search space — ACE evolves context playbooks, MCE evolves context-management mechanisms, Meta-Harness searches whole harness codebases, and ADAS and AFlow search the workflow graph itself. Lesson 3, Self-Improving Harnesses: close the loop — STOP's improver improving itself, Self-Harness mining its own failures (with the execution traces Pattern 3 taught you to keep), AlphaEvolve's evolutionary program search, the Darwin Gödel Machine, and SIA's joint harness-plus-weights updates.

The trilogy's arc is Chapter 8's meta-methodology ladder, one lesson per rung.

Because agents rarely announce which layer failed, here is the triage table — symptom first, the way debugging actually arrives:

Harness triage. Note that every "first check" is only possible because some pattern stored the evidence — an uninspectable harness cannot even be debugged.

Finally, the map back to the source, so you can read Weng's post with this lesson as your guide — and see exactly where the trilogy's next two lessons pick up:

Related lessons on this site, and why you would go there:

• Harness Optimization — trilogy lesson 2: the harness as an optimization target (ACE, MCE, Meta-Harness, ADAS, AFlow).
• Self-Improving Harnesses — trilogy lesson 3: STOP, Self-Harness, AlphaEvolve, DGM, SIA — the loop closed.
• Loop Engineering — the sibling framing: the four-layer stack and the discipline of loops that prompt your agents (remember the altitude mapping from Chapter 1).
• Agents & Tool Use — how tool calling actually works at the token level: schemas, function calls, parallel calls.
• Agent Architectures — the wider zoo of agent designs the harness patterns generalize over.
• MCP — the standardization story from Chapter 1, in full: the USB-C of external context.
• Agent Skills — the other half of the "external context" tool group: packaged, loadable procedural knowledge.
• Prompt Engineering — Chapter 8's precedent in full: the craft that internalized, and the specification skills that never will.
• AI Evaluation — the fifth verb gets its own lesson: evals, verifiers, and why the checker must sit outside the loop.

Primary sources touched in this lesson, for the reader who wants the originals:

1. Weng, L. "Harness Engineering for Self-Improvement." Lil'Log, Jul 2026 — the trilogy's source text.
2. Good, I. J. "Speculations Concerning the First Ultraintelligent Machine." Advances in Computers 6:31–88, 1965 — the origin of the idea.
3. Yudkowsky, E. "Recursive Self-Improvement." LessWrong, 2008 — names and sharpens the loop.
4. Zelikman, E. et al. "Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation." COLM 2024 — the capability-bar caution in Chapter 8.
5. Yuan et al. (2024); Chen et al. (2024); Zhao et al. (2025, "Absolute Zero"); Choi et al. (2026) — the model-side road this trilogy sets aside.
6. Karpathy, A. autoresearch (github.com/karpathy/autoresearch) — Pattern 1 in a few readable files.

A concrete dare before the quote, because this lesson is unusually buildable. Everything essential fits in about a hundred lines of Python you have already read: Chapter 3's loop (twenty lines), Chapter 4's write_note/recall (fifteen), Chapter 5's process manager (twenty), Chapter 6's dispatch gate with a permission table (twenty), plus an LLM API call. The assembly is almost embarrassingly short:

python
# the whole lesson, assembled — a working harness in one screen
from loop import agent_loop            # Ch 3: plan/act/observe/improve + budget
from memory import write_note, recall  # Ch 4: files as durable state
from jobs import launch, inspect, recover  # Ch 5: the process manager
from belt import dispatch              # Ch 6: tool registry + permission gate

def harness(goal, model):
    write_note("goal", goal)                        # verb: stores artifacts
    result = agent_loop(goal, model,
                        tools=dispatch,             # verb: calls tools and acts
                        done=lambda: tests_pass())  # verb: evaluates results
    write_note("trajectory", result)                # raw material for lesson 3
    return result

Wire it up, point it at a repo with one failing test, and watch your own harness grind through the loop you have been reading about. It will fail in instructive ways — the context will bloat, an edit will miss, the loop will spin — and each failure will be one of this lesson's chapters introducing itself in person. Build in this order, because each piece is testable before the next exists:

1. Loop + verifier first (Ch 3) — a harness with only bash and a test command is already an agent; everything else is amplification.
2. Then the budget — before your first token furnace, not after.
3. Then files (Ch 4) — the moment any task exceeds a few minutes, notes and logs start paying.
4. Then the permission gate (Ch 6) — before you point it at a repo you care about.
5. Sub-agents last (Ch 5) — parallelism multiplies whatever exists, so make sure what exists is worth multiplying.

And the closing quote — the site's own motto, because no sentence fits this lesson better. A harness is not something you can bluff: it either runs the tests or it does not, either recovers from the interruption or does not. Building one is the fastest way to discover which parts of "agents" you actually understand.