Optimize the Harness,
Not the Artifact

Chapter 1: What Is a Harness, Exactly?

"Harness" is a word people use loosely to mean "the stuff around the model." The paper makes it precise, and the precision is what buys everything later. Here is the definition, Equation 1:

Read it left to right. π_θ is an LLM or MLLM with frozen parameters θ. x is the multimodal input — here, an academic paper PDF plus whatever source assets came with it. c is the context: the target medium and the user's constraints ("an A0 conference poster, portrait, our lab's colours"). H is the harness: everything that decides how π_θ gets called, with what, in what order, checked by what, and when to stop. y is the artifact.

The tilde matters. y is sampled, not computed. Two runs of the same harness on the same paper with the same model give two different posters, because the model is stochastic and because the loop's decisions depend on what the model said. Hold on to that: it is why Chapter 2's objective is an expectation and why Chapter 8's ten-paper ablations deserve suspicion.

The trajectory is a first-class object

Alongside y, the harness emits an execution trajectory τ, which the paper defines as the record of "the sequence of intermediate actions, states, and revisions leading to the final output."

This is not logging for the sake of logging. In ordinary supervised learning the training signal is (input, label). In meta-harness optimization the training signal is (trajectory, score) — and the trajectory is where the diagnosis lives. A score of 61 tells you the poster was mediocre. The trajectory tells you that attempts 1 through 4 all failed the same overlap check on the same lane, that the designer responded each time by shrinking the font, and that the font is now 7 pt. One of those is actionable.

Equation 1 as a type signature

If the mathematical notation feels slippery, rewrite it as a function signature and it stops being slippery immediately:

the harness, as a typedef harness(
    model:   Model,           # π_θ — frozen weights, called many times
    source:  MultimodalInput, # x — the paper PDF plus its assets
    context: DesignContext,   # c — target medium, size, constraints
) -> tuple[Artifact, Trajectory]:
    # returns the deliverable AND the record of how it was made
    ...

Three observations fall straight out of the signature.

The model is an argument, not a global. That is why the Model Track in Chapter 8 is even possible: swap the argument, keep the function, measure the difference. A harness that hardcoded its model could not be tested that way, and the paper could not have reported six models under one identical harness.

The trajectory is a return value, not a log file. Logs are written for humans and thrown away. This return value is consumed by an optimizer, which changes what belongs in it: not INFO: rendering attempt 3 but a structured record of actions, states, and revisions that a subagent can reason over and, crucially, count across runs.

The return is a distribution, not a value. Two calls with identical arguments give different artifacts. Hold on to that, because every number in Chapter 8's tables is exactly one draw.

What is actually inside a trajectory

The paper defines τ as "the sequence of intermediate actions, states, and revisions leading to the final output" and leaves the schema open. The reported numbers tell us the scale, and the scale tells us what the schema has to support.

The five components — and why five, not one

The paper decomposes H into exactly five functional components. The stated reason is worth quoting because it is an engineering justification, not a taxonomy for its own sake: the decomposition exists "to enable systematic meta-harness optimization and facilitate credit assignment."

Notice how differently these five fail, and how differently you would fix each. A prompt problem is a Context and Memory problem. A "the renderer silently drops web fonts" problem is an Execution Runtime problem. A "the agent gave up after two tries" problem is Orchestration. If you lump all five into "the agent," every failure looks the same and every fix is a guess.

The rule that makes credit assignment work

Here is the constraint that turns the five-way split from documentation into machinery. From Section 3.2 of the paper:

"Each outer-loop iteration is restricted to exactly one of the five harness components… An update may span multiple files within the selected component, but it cannot modify another component in the same iteration."

And the reason, in the paper's own words: "This restriction keeps credit assignment interpretable, as each gain or regression is attributable to a single coherent intervention rather than to several simultaneous changes."

This is a controlled experiment discipline imported into an optimizer. If iteration 17 changes the validator and the retry budget and the prompt, and the score goes up 4 points, you have learned that some combination of three things helped — which is very nearly nothing. If iteration 17 changes only the validator, the same 4 points are attributable, recordable, and reusable as evidence in iteration 18.

Worked example: attributing a gain, with and without the rule

Suppose you are optimizing by hand and you have five training papers. The current harness H_t scores them:

You make a bundled change: a new overlap checker (Evaluation and Feedback), a raised attempt budget from 6 to 12 (Orchestration), and a stricter typography spec (Tools and Specifications). New scores:

A gain of 64.0 − 58.2 = +5.8. Excellent. Now answer the only question that matters for the next iteration: which of the three changes should you build on, and which should you revert?

You cannot say. The observation is one number and the hypothesis space has three dimensions. Worse, the three could be interfering: perhaps the checker is worth +9, the typography spec is worth −3, and the budget is worth −0.2, and you are about to spend the next ten iterations elaborating a typography spec that is actively hurting you.

Under the one-component rule the same total change takes three iterations, each with a clean read:

Same final harness content on the table, completely different knowledge. The bundled version ends with "+5.8, unclear why." The unbundled version ends with "the overlap checker is worth +9, and two plausible-sounding ideas were tested and refuted." The second is what you can build a hundred more iterations on.

Play with attribution

With the rule on, every iteration produces one arrow you can trust. With it off, the optimizer moves faster and learns less — which, over 123 iterations, is the difference between accumulating design priors and accumulating changes.

Why θ stays frozen — three reasons, one of them proved by the paper

Freezing the model can look like a limitation the authors accepted. It is a choice, and it buys three specific things.

Reason one: cost. Fine-tuning a frontier MLLM on long-horizon design trajectories would need gradients through a model you probably cannot train, on data you would have to construct, at a price that dwarfs the paper's entire budget. The whole campaign — 7 days, ~123 iterations — costs less than one serious post-training run.

Reason two: attribution. If the weights moved and the harness changed, no measurement separates them. Freezing θ makes every reported delta unambiguously a harness delta. Same instinct as the one-component rule, applied one level up.

Reason three: portability — and this one is measured. A harness that never touches weights can be attached to a model that did not exist when the harness was built. That sounds like a pleasant property until you see it tested. Table 4 attaches the same DesignHarness to seven different model and code-agent configurations, and every single one improves, from +5.01 to +19.56 points. Nothing was tuned per model.

A worked walk of the five components on one real run

Abstractions stay slippery until they are annotated onto something concrete. Here is the poster run from Chapter 0, moment by moment, with the owning component named.

Read the right-hand column and you can feel what an outer-loop iteration actually is. Every entry is small, specific, testable, and confined to one component. Not one of them is "make the posters better."

The failure mode this decomposition is designed to prevent

Here is what happens without it, and it is worth spelling out because it is the default outcome of letting an agent improve itself.

An agent notices its posters overflow. It appends to the system prompt: "Be careful about text overflow." Scores nudge up. Next iteration it notices figures are sometimes low quality, and appends: "Use high-resolution figures." Then: "Do not invent numbers." Then: "Balance the columns." Twenty iterations later the system prompt is 3,000 words of accumulated superstition, no line of which has ever been individually tested, several of which contradict each other, and none of which can be removed — because nobody knows which ones are load-bearing.

A harness is code, and that is the point

One last structural observation before the maths. Because the harness is an executable system — files, prompts, tools, a runtime — the thing that edits it can be a coding agent. Not a gradient, not an evolutionary operator over a hand-designed genome: a program-editing agent that reads traces and writes a diff.

The paper places this in a lineage. "A Self-Improving Coding Agent" (Robeyns et al., 2025) and MOSS (Cai et al., 2026) "update agent source from execution evidence"; Meta-Harness (Lee et al., 2026b), HarnessX (Chen et al., 2026), Self-Harness, and Agentic Harness Engineering (Lin et al., 2026) study "searchable harness programs, composable primitives, bounded updates, and outcome attribution." AutoDesign's contribution is not the idea that harnesses can be optimized; it is a specific, evaluated instantiation for design, with a gate and a benchmark attached.

Chapter 2: The Meta Objective

We have a harness H. We want a better one. To say "better" without hand-waving we need a number attached to a harness, and that is what this chapter builds — from an expectation, through the reason it cannot be differentiated, to the strange substitute the paper uses instead of a gradient.

Scoring a harness, not an artifact

Start from the thing you can actually measure. Given one poster y made from one paper x under one context c, an evaluator returns a number. The paper calls it R_meta(y, x, c) — note that it takes all three arguments, because "is this poster good?" is unanswerable without the source it came from and the brief it was made for. A gorgeous poster about the wrong paper scores zero.

Now lift from artifacts to harnesses. A harness is not one poster; it is a machine that makes posters. Its quality is the quality of the posters it makes on average, over the tasks you care about. That is Equation 2:

Two sources of randomness sit under that expectation, and separating them is the whole art of evaluating agents:

And the objective, Equation 3, is exactly what you would write:

with the standing constraint that θ is frozen. The paper states it plainly: "the parameters θ of the underlying model π_θ in the harness remain fixed. The optimization therefore acts on the system surrounding the model rather than on the model itself."

Why there is no gradient, and what stands in for one

Walk through the three things you would need for gradient descent and watch each one fail.

You would need J to be differentiable in H. It is not, and not in a fixable way. H is source code. The derivative of "poster quality" with respect to "the wording of a validator's error message" is not a small quantity — it is a category error. Changing one line of the harness can change the entire control flow of every subsequent rollout.

You would need a notion of a small step. Programs have no neighbourhood structure. The edit distance between two files is not a meaningful distance between two harnesses: a one-character change (> to >=) can flip the accept condition for every candidate, while a three-hundred-line refactor can be behaviourally identical.

You would need J to be cheap to evaluate. It is spectacularly not. One evaluation of J means running the whole harness on every training task — each of which is a long-horizon agentic run. The paper reports one such run costing 253 tool calls, 11 editing turns, 40 minutes, and under $3. Multiply by a training set, then by a development set, then by 123 iterations. A single "function evaluation" here is minutes-to-hours and dollars, where a neural-network gradient step is milliseconds and fractions of a cent.

So what replaces the gradient? The paper's answer is Equation 5, which we will meet properly in Chapter 4, but the shape is worth seeing now:

P is a coding agent. It takes the current harness, the batch of trajectories, the batch of scores, and a persistent optimization record L, and it emits a candidate harness. It is a proposal operator, not a descent direction. The analogy is loose but useful:

Worked example: how noisy is one estimate of J?

This is the calculation nobody does and everybody should. J(H) is an expectation; what you measure is a sample mean over N_train tasks. So let us ask how precisely you can know it.

Take the paper's own dispersion as a guide. On PosterBench-mini, across the twelve systems in Table 2, scores run from 34.73 to 81.46 — but that is between systems. Within a single system across papers, per-paper scores also vary a lot, because papers differ wildly in figure count, table density, and length. Suppose a per-paper standard deviation of σ = 12 points, which is conservative for a benchmark whose system-level spread is 47 points.

The standard error of a mean over N tasks is σ/√N. With N = 10:

Now suppose you propose a harness edit and the training mean moves from 62.0 to 64.5. That is +2.5 points. Is it real? The standard error of the difference of two independent means of the same size is √2 × 3.79 = 5.37 — more than twice the observed effect.

Two caveats make this less bleak than it looks, and both are real. First, the two means are paired — the same ten papers before and after — so the between-paper variance largely cancels and the relevant standard deviation is that of the per-paper differences, which is much smaller than 12. Second, a harness edit that fixes a systematic failure moves nearly every paper in the same direction, which is exactly the regime where paired comparison is powerful. If all ten papers improve, a sign test alone gives p = 2⁻¹⁰ ≈ 0.001, no variance estimate required.

None of the above is a number the paper reports; it is standard sampling statistics applied to the paper's stated setup, and we flag it as our own analysis. The paper reports no per-iteration confidence intervals, no repeated rollouts per task, and no variance estimates for the ablation tables. Chapter 8 returns to what that costs.

Why the paired sign test is the right instrument here

We estimated the standard error of a mean and found it uncomfortably large. But the gate never compares two independent means — it compares the same tasks before and after a change. That pairing deserves to be formalised, because it turns a hopeless measurement into a usable one.

Let d_i = s'_i − s_i be the per-task change on task i. The between-task variance — the fact that paper 3 is intrinsically harder than paper 7 — cancels completely in d_i, because both terms are on the same paper. What survives is only the variance of the effect plus rollout noise.

Now the cheapest possible test. Under the null hypothesis that the edit does nothing, each d_i is positive with probability 1/2, independently. If all ten training tasks improve:

If nine of ten improve, the one-sided probability of nine or more is:

And if only six of ten improve — which can easily coexist with a healthy-looking mean gain, if one task jumped a long way:

Thirty-eight percent — no evidence at all. And yet the mean might have moved several points. The lesson is direct and portable:

Three objectives that look reasonable and are not

J(H) is an expectation — a mean over tasks. That is a choice, and other choices produce recognisably broken optimizers. Since you will have to pick one for your own system, it is worth seeing what each one optimizes toward.

Notice how the ceiling mechanism from Chapter 6 is doing quiet work in the first row. Averaging alone would let a poster with a broken render be offset by a beautiful one. Capping each record before averaging means a catastrophe cannot be averaged away — it imports a slice of worst-case thinking into a mean-based objective without inheriting min's variance. That is genuinely good metric design, and it is easy to miss because it is written as a min inside an equation.

What one evaluation of J costs, end to end

One more piece of arithmetic, because it silently explains every structural choice in Chapters 4 and 5.

One task ≈ 253 tool calls, 11 editing turns, ~40 minutes, under $3. So one evaluation of J over a ten-task set is:

Each outer-loop iteration needs the candidate measured on train and on dev — two such evaluations. At roughly 123 iterations:

The paper reports the campaign taking 7 days. So the runs were roughly ten-way parallel — exactly what you would expect, since tasks within a batch are independent by construction. This is not a footnote: the task-independence of the rollout stage is what makes the method feasible at all. If evaluating J required tasks in sequence, the same campaign would take over two months.

What "human design priors" means in this objective

One phrase in the abstract deserves unpacking here rather than later: the framework "aligns with human design priors." Concretely that alignment enters through R_meta, and it enters before optimization starts.

The paper's procedure: an evaluator coding agent is given "reference artifacts annotated by humans along seven quality dimensions" — Faithfulness, Coverage, Density, Visual Evidence, Layout, Readability, Aesthetics. From those examples the agent implements R_meta, "combining rule-based checks for directly measurable properties with VLM-based judgments for perceptual properties such as aesthetics." And then, critically: "Once constructed, R_meta remains fixed during autonomous optimization."

So human preference is compiled once into a program, and the program is then frozen while the harness is optimized against it. That freeze is the anti-reward-hacking measure at the objective level, in the same way the dev gate is the anti-overfitting measure at the update level.

Human-in-the-loop: a second argument to P

The paper also allows directional guidance in natural language. When a human supplies g_t, the proposal becomes:

The stated motivation is not "humans are better"; it is a search-dynamics problem: "the coding agent acting as P may converge prematurely to a locally satisfactory harness configuration, at which point outer-loop optimization stagnates. Guidance can inject task-specific heuristics or redirect the search toward alternative improvements."

Figure 1(a) of the paper shows exactly this shape on one representative paper: autonomous optimization improves the initial harness, then plateaus, and human guidance redirects the search and yields a further gain. If you have ever watched a local search flatten out and needed to kick it, that curve will be familiar. The paper is careful about what the human does and does not do: "the human provides observations or high-level directions rather than directly editing the harness or evaluator implementation."

Chapter 3: The Inner Loop

The inner loop is what makes one poster. It runs inside a fixed harness, it does not change the harness, and it is the thing whose trajectory the outer loop will later read like a doctor reads a chart. Get it exact and Chapter 4 becomes easy.

Two abstract modules, deliberately underspecified

AutoDesign starts from what the paper calls "a minimal inner-loop scaffold consisting of two abstract modules": a designer M_design and a critic M_critic. That is the entire initial harness. Equation 4:

with the initialisation that y₀ and f₀ are empty, "so that the first step produces an initial draft from (x, c) alone."

Read the arguments carefully, because they encode a real design decision. The designer sees the previous artifact and the previous feedback, plus the source and context. The critic sees only the current artifact, plus source and context — not the previous feedback, not its own history. The critic is memoryless with respect to the loop; it is a function from a candidate to a diagnosis. That keeps it from anchoring on complaints it made two attempts ago about a region that no longer exists.

Unrolling the recursion

Substituting Equation 4 into itself makes the dependency chain visible. Writing it out for four steps:

This is a Markov chain over artifacts: y_k depends on the past only through (y_k−1, f_k−1). It has a well-known failure mode — oscillation. If fixing the overlap requires shrinking a lane, and shrinking the lane causes a density violation, and fixing the density violation requires growing the lane, the loop can ping-pong until the budget runs out. Hold that thought: the finalized harness's answer to it is the blocking/non-blocking split we are about to meet.

The critic is two things, not one

Here is where the optimized harness diverges sharply from the textbook self-refine picture. In the final DesignHarness, "rule-based validation and visual critique jointly instantiate M_critic." Two feedback sources with completely different characters:

That last row is a piece of engineering worth pausing on. The paper's control flow: "If the candidate passes all of them, the inner loop terminates and the candidate proceeds directly to finalization. Otherwise, the validator returns localized diagnostics… When a candidate fails the blocking checks, it is also rendered into a medium-specific preview and inspected by a critic VLM."

Blocking versus non-blocking checks

The validator emits two kinds of signal, and the distinction is the loop's termination logic.

Blocking checks are, in the paper's phrasing, "deterministic blocking checks… these checks determine whether the candidate satisfies the requirements for terminating refinement." Unsafe or missing assets. Broken provenance links. Severe overflow or overlap. Typographic and layout constraint violations. All-or-nothing: pass everything, or keep going.

Non-blocking checks come back alongside the blocking diagnostics and cover "properties such as content coverage, information density, and numerical consistency with the source." These inform the repair without preventing exit.

Why split them this way? Because the two categories differ in whether "more is always better." Overlap is binary-ish: text either collides or it does not, and you can always satisfy it. Information density is a trade-off against readability — a poster can be too dense or too sparse, and any threshold you pick is a judgement call. Making a judgement call blocking would produce exactly the oscillation described above. Making a hard geometric fact non-blocking would let clipped posters ship.

The attempt budget and what happens when it runs out

"The final implementation permits at most K = 12 refinement attempts."

Twelve is not a magic number; it is a number the outer loop chose, by editing the Orchestration component. Somewhere in those 123 iterations, a proposal changed the budget and the gate admitted it. That is what it looks like when a hyperparameter stops being a hyperparameter and becomes an optimization variable.

Two exits from the loop:

Two details in Exit B carry real weight. "The retained attempt history" means every attempt is kept, not just the last one — so the fallback can promote attempt 7 over attempt 12 if 7 was better. A loop that only kept the current candidate would be forced to ship whatever the last edit produced, which after eleven failed repairs is often the worst artifact of the run. And "retaining essential safety and integrity constraints" means the fallback relaxes quality requirements, not safety ones: it will ship a slightly ugly poster, never one with a broken provenance link.

Worked example: the paper's own five-attempt trace

Section 5.4 traces one real poster run and names the events. These are the paper's numbers, on its own internal 0–1 scale for that figure:

Total movement: 0.78 − 0.36 = +0.42 over nine attempts, inside a budget of twelve. Average gain per reported step, over the four reported deltas: (0.06 + 0.20 + 0.00 + 0.16) / 4 = 0.42 / 4 = +0.105.

Now read the shape rather than the total, because the shape is the lesson. The first repair — unclipping the analysis lane — buys almost nothing (+0.06). It removes a defect without improving the design. The big jump comes at A5, when the header is refitted and the composition rebalances. Fixing a violation and improving a poster are different activities, and the trace shows the loop doing the first three times before the second pays off.

The A5 → A6 step is the other instructive one: a full attempt, a real edit ("the scaling of evidence"), and zero score movement. Under an attempt budget of 12, an attempt that buys nothing is 8.3% of the budget spent. This is the cost the outer loop is trying to drive down — not "make posters better" so much as "stop wasting attempts."

And the paper names the intended behaviour explicitly: "edits stay localized to the failing region while valid layout and source-derived content are retained across revisions." That is the whole reason the artifact is editable HTML rather than a regenerated image, which is Chapter 7's subject.

Walk the loop

Switch to "no blocking checks" and watch the exit condition become the VLM's opinion — the loop keeps running past a perfectly deliverable candidate because a perceptual critic can always suggest one more thing. Switch to "no attempt history" and watch the fallback lose the ability to promote an earlier, better candidate.

One attempt, traced through every representation

Concept is cheap; realization is where understanding lives. So here is a single refinement step with the actual objects named — what exists, what type it has, who reads it, and what it costs.

Two things in that table are load-bearing and easy to miss.

Step 6 is where the discipline is. The validator does not reason about layout from HTML source. It reasons about a render. This distinction is the difference between "the CSS says max-height: 400px, so it should fit" and "the rendered box is 434 pixels tall and clips its last line." Only the second is a fact. Anyone who has debugged a webpage by reading its stylesheet knows exactly which of those two is worth having.

Step 1 happening once is a decision, not an optimization. It could plausibly re-run per attempt. It does not, and the consequence is semantic rather than economic: with one ingestion, the twelve attempts differ only in design, never in what the paper says. Re-ingesting would give you twelve slightly different readings of the same source, and a poster whose claims drift between revisions — a faithfulness failure produced by a caching decision.

The oscillation the blocking/non-blocking split prevents

Chapter 3 claimed that gating on a trade-off makes the loop thrash. Let us actually watch it thrash, because the mechanism is instructive.

Suppose Density were a blocking check requiring at least 70% information occupancy, and overlap were blocking too. Play the loop forward:

Twelve attempts, roughly 276 tool calls, 40 minutes, and the loop never terminates cleanly — not because the model is weak, but because the exit condition is unsatisfiable. Two blocking constraints pull in opposite directions and the loop has no representation for "trade these off," only for "satisfy both."

Now the paper's actual arrangement. Overlap is blocking, because it is objectively satisfiable: some layout exists with no collisions. Density comes back as a non-blocking observation. So A2 exits cleanly at 64% occupancy, having shipped a slightly sparse but entirely legible poster, and the density observation flows into the trajectory where the outer loop can see it — and if 64% occupancy shows up across many runs, the outer loop can address it in the artifact specification, where a trade-off belongs.

Reading the reported trace as a diagnosis

Return to the paper's five reported checkpoints and ask what an outer-loop optimizer, handed ten traces of this shape, would conclude.

None of these conclusions is available from the final score of 0.78. All of them are available from the trace. That is the entire argument for treating τ as a first-class return value rather than as logging, and it is why Chapter 4's optimizer reads trajectories before it reads scores.

The multimodal advantage, stated precisely

One sentence in the paper's ablation discussion explains a whole column of results, and it belongs here because it is about the inner loop's information flow:

"MLLMs have an additional repair signal unavailable to text-only LLMs: at each attempt, the rendered preview from the preceding attempt is supplied as visual context for the next repair. This lets the model inspect the artifact it is editing and localize layout, clipping, or visual-evidence failures that are not fully captured by textual diagnostics alone."

Unpack the data flow. Without vision, the designer's picture of its own output is: the HTML it wrote, plus a textual diagnostic like "element #analysis overflows its container by 34px." With vision, it additionally receives the actual pixels. The difference is not cosmetic. "Overflows by 34px" does not tell you whether the fix is a smaller font, a shorter sentence, a taller lane, or moving a figure — four repairs with very different consequences for the rest of the composition. A picture disambiguates in one glance what a diagnostic string cannot express at all.

Chapter 4: The Outer Loop

The inner loop makes a poster. The outer loop makes a better poster-maker. It runs four stages per iteration, and the paper names them: rollout, evaluation, update proposal, acceptance. This chapter walks the first three; the fourth is important enough to get its own chapter.

Stage 1 — Rollout

At outer iteration t, the current harness H_t runs on the whole training task set:

Each element is a source paper x_i plus a context c_i that "specifies the target medium and the corresponding design requirements." Each execution produces two things: an artifact yⁱ_t and a trajectory τⁱ_t. The collection is written τ_t = { τⁱ_t }.

Both indices matter. The superscript i is the task; the subscript t is the outer iteration. τ_t is therefore an entire batch of complete agentic runs — not one trace, a table of them. That is the object the optimizer reads.

Cost check, since this is the expensive stage. If a single run is roughly the paper's reported long-horizon figure — 253 tool calls, 11 editing turns, 40 minutes — then one rollout batch over ten tasks is on the order of 2,530 tool calls and, if serialized, nearly seven hours. Run it for both train and dev, for both H_t and every candidate H′_t+1, across 123 iterations, and the seven-day figure in the contributions list stops sounding leisurely and starts sounding tight. Parallelism across tasks is not an optimization here; it is a requirement.

Stage 2 — Evaluation

Every artifact is scored by the frozen optimization-time evaluator:

The paper draws a hard line here that is easy to skim past and impossible to overstate: "This optimization-time evaluator is distinct from the frozen PosterBench protocol used for final system comparison." Two evaluators, two jobs. R_meta is the training signal. PosterBench is the exam. Chapter 6 is entirely about why they must be different objects.

Stage 3 — Update proposal

Now Equation 5 in full:

Four inputs. The current harness. The batch of trajectories. The batch of scores. And L, "an optimization record… serving as persistent context across outer-loop iterations." The prime on H′ is doing real work: it marks a proposal, not an update. Whether it becomes H_t+1 is the gate's decision.

P is instantiated as a coding agent that "sequentially assumes the roles of a planner and a code editor." The two roles are worth separating because they need different things.

The word "recurrent" is the whole filter

The planner synthesizes "structured evidence of recurrent failures." Not failures — recurrent failures. This single adjective is what keeps the optimizer from chasing noise, and it is worth making concrete.

Suppose your rollout batch of five papers produces these observations:

Row three is the interesting one. Recurrence is necessary but not sufficient: a finding also has to land in a component. "Aesthetics is low everywhere" is a symptom without an address. The planner's job is to turn it into one — for instance, "in 5 of 5 runs the palette is chosen ad hoc per section," which is a Tools and Specifications problem with an obvious bounded fix.

One iteration, end to end, with numbers

Abstract stages become concrete the moment you walk one. Here is outer-loop iteration t = 17, invented in its particulars but sized by the paper's reported figures.

Rollout. H₁₇ runs on the ten training papers. Ten complete agentic runs, roughly 2,530 tool calls in total, about 40 minutes each in parallel. Out come ten posters and ten trajectories.

Evaluation. R_meta scores each artifact:

Update proposal — the planner. The coding agent cannot read ten full traces in one context, so it fans out. Ten subagents, one per trajectory, each returning a compressed finding. Suppose they return:

The planner synthesizes: row one is the only finding that is both recurrent and addressable. It writes a plan naming the failure mode, the component (Tools and Specifications), and the intended change: make figure-plus-caption an atomic element in the artifact specification, so a layout engine cannot separate them and the validator can check the pairing.

Update proposal — the code editor. The same agent switches role and implements the change: edits to the specification files, plus whatever validator hook the spec implies. Multiple files, one component. Out comes H′₁₈.

Measurement. The candidate runs on train and on dev. Ten more runs each, ~5,060 more tool calls.

Per-task differences: +5, +5, +1, +7, +2, +5, +8, +2, +2, +5. All ten positive — the sign-test signature of a systematic fix, exactly as Chapter 2 predicted, with a one-sided probability under the null of 1/1024.

Acceptance. Suppose J_dev moves 63.0 → 64.5. Then 70.7 > 66.5 and 64.5 ≥ 63.0: both conditions hold, and H′₁₈ is promoted to H₁₈. The record gains a checkpoint, the plan, the diff, the scores, and the decision Accept. Iteration 18 begins with a harness that binds captions to figures — and with a written note that a two-of-ten budget-exhaustion signal is pending, awaiting more evidence.

The cost of an iteration, tabulated

Two observations. First, the incumbent's numbers are never recomputed — lines 7–12 of Algorithm 1 carry forward whichever tuple survives, so a rejection costs exactly the same as an acceptance and no more. Second, the proposal stage is nearly free in rollout terms and expensive in reading terms, which is precisely why the 224 subagents exist: they parallelise the only stage that does not parallelise across tasks.

What the record buys you that a score history does not

The paper says L "supports comparison, reproducibility, and rollback across iterations." Take each one seriously.

Comparison. Because each iteration touched exactly one component, the record is effectively an experiment log: fifty-four accepted interventions, each with a component label and a measured delta. That is enough to answer "which component yielded the most?" — a question the paper does not report but which its own data structure makes answerable.

Reproducibility. A repository checkpoint per iteration means any intermediate harness can be rebuilt and re-run. Without that, "the harness improved over seven days" is an unfalsifiable claim about a system that no longer exists in any of its intermediate forms.

Rollback. This is the one that matters in practice. Suppose iteration 40 is accepted — it passed both gates — and only at iteration 55 does a human notice that posters have all started looking the same. Without checkpoints, unwinding means guessing which change did it and hand-reverting. With checkpoints, you bisect: check out iteration 40, 47, 51, and find the culprit in a handful of steps. The gate is a filter with a false-accept rate; checkpoints are what make its errors recoverable.

The optimization record L: what persists and what does not

For each completed iteration t, the paper says L stores: the harness H_t; the trajectories and scores; the selected harness component; the update plan and the corresponding code changes; and the acceptance decision — "with a repository checkpoint preserving the harness implementation at that iteration."

And one thing it pointedly does not store: "Trajectories and scores from the development set are not included in the record."

That omission is the entire integrity of Chapter 5. If dev trajectories entered L, then L is fed to P next iteration, and P would be optimizing against dev by proxy — slowly, indirectly, and undetectably. The barrier has to be enforced at the record, not just at the prompt, because L is persistent: a single leak contaminates every future iteration.

No tree search

One sentence rules out a whole family of algorithms you might otherwise assume: "the outer loop maintains a single active harness at each iteration and does not perform tree search over harness variants."

So this is hill climbing with a strict acceptance test, not beam search, not a population, not MCTS over programs. One incumbent, one challenger, one comparison, repeat.

Given that a single rollout batch costs hours and dollars, refusing to branch is not timidity, it is arithmetic. And the paper is honest about the consequence: it is exactly the premature-convergence risk that motivates the human-guidance channel from Chapter 2. Figure 1(a)'s plateau-then-jump is what a single-incumbent hill climber looks like when it runs out of moves.

Algorithm 1, line by line

The paper's Algorithm 1 fits in sixteen lines. Here it is with the reasoning attached to each one.

Chapter 5: The Acceptance Gate

If you take one thing from this paper into your own work, take this chapter. Everything else is a well-executed instantiation of ideas that are in the air. The gate is the discipline that stops the whole thing from being a very expensive way to overfit ten papers.

The condition

Equation 6:

Two conditions joined by AND. Both must hold or the candidate is discarded and H_t is retained. Look at the comparison operators, because they are different on purpose:

Where the operators come from — and what happens if you flip them

Play the two obvious alternatives forward and the asymmetry justifies itself.

If dev were also strict (>). Consider a genuine fix: the validator gains a check for a figure-caption provenance failure that occurred in four of the ten training papers. In the dev papers, all figures happen to already have clean captions. Dev score: unchanged. Under a strict dev condition you reject a correct, well-evidenced, harmless improvement. Repeat that policy for 123 iterations and you accept almost nothing — you have built a system that only admits changes that happen to help two disjoint samples simultaneously, which for narrow fixes is mostly luck.

If train were also non-strict (≥). Now a change that moves nothing on train and nothing on dev is admitted. Harmless? No. It is admitted into H, into the checkpoint, into L, and into every future iteration's context. Complexity accumulates with no evidence attached to it. Ten such acceptances and the harness has ten unexplained pieces, each of which the next optimizer will treat as intentional. Strictness on train is a complexity budget.

Worked example: three proposals, decided by hand

Let the training set be five papers and the development set three. The incumbent H_t scores:

Compute the two baselines, by hand:

Now three candidates arrive, one per iteration.

Candidate A — a new deterministic overlap check in Evaluation and Feedback.

Both conditions hold. H_t+1 = A. Train +6.0, dev +2.0 — the smaller dev gain is normal and expected; the fix was designed against training evidence.

Candidate B — a hand-tuned layout template lifted from the highest-scoring training paper.

This is the case the gate exists for. Train jumps +7.4 — the largest gain of the three, and by far the most tempting. Dev falls −6.33. A template shaped around five specific papers is a memorised solution wearing an improvement's clothes. Without the dev condition this is accepted, becomes the incumbent, and every subsequent proposal is built on top of a harness that has quietly specialised to five documents.

Candidate C — a refactor of the ingestion prompt with no behavioural intent.

Dev actually improved by +1.0. Doesn't matter. Train tied, and a tie on train means no evidence. Note also that the per-paper scores moved — 68 vs 69, 59 vs 60 — while the mean did not. The gate is deliberately blind to that churn: unattributable movement is not improvement.

The leak that would silently break everything

The paper closes the loophole in one sentence: "Results on the development set are used exclusively by the acceptance gate and are never exposed to P when constructing an update proposal."

Think about why this needs saying. P is a coding agent with a long persistent context. If it could see dev scores, it would — entirely reasonably, and without any instruction to cheat — start proposing edits that help dev, because that is what optimizers do with the signals they are given. Nothing about that would look like misbehaviour in the trace. It would look like a very effective optimizer. And it would be, right up until you evaluated on anything else.

The paper cites Nguyen et al. (2026), "Recursive self-evolving agents via held-out selection," as the precedent for using an independent split to gate persistent updates, and positions AutoDesign alongside it: RHI (Lee et al., 2026a) "keeps its evaluation prompt on the evaluator side of the update loop, yet the resulting pairwise history remains a task-local learning signal," whereas AutoDesign "uses an independent development acceptance gate for harness updates."

The gate read as a hypothesis test

The paper presents Equation 6 as a rule, not as statistics. But it is a test, and reading it as one tells you exactly which errors it makes and how often.

Every gate makes two kinds of mistake, and they trade against each other:

The asymmetry is stark, and it justifies the whole design. A false accept is permanent and compounding; a false reject costs one iteration out of about 123. So the correct operating point is heavily conservative, and that is exactly what an AND of two conditions gives you: the candidate has to clear both, and clearing both by luck is much less likely than clearing one.

Put rough numbers on it. Suppose a useless change has a 50% chance of appearing to improve train by luck, and independently a 50% chance of not appearing to hurt dev. Then:

The false-accept rate halves. And the cost is that a genuine but narrow fix — one that helps train and leaves dev flat — still passes, because the dev condition is ≥ and not >. That is the asymmetry earning its keep: the strict condition sits where evidence exists, the permissive one where it may not.

Gate variants, and what each one costs

You will be tempted to modify Equation 6. Here are the modifications people reach for and what each does.

How much information does the dev set actually give up?

Chapter 5 claimed dev is a consumable resource. Quantify it, because the quantity is small and surprising.

Each iteration extracts one bit from dev: pass or fail on the second condition. Over T iterations the loop has at most T bits of information about the dev papers. With T = 123:

This is the multiple-comparisons problem in its cleanest form. Nothing was ever trained on dev. Yet the final harness is precisely the one that happened to satisfy 123 consecutive dev conditions, so its dev score is optimistically biased, in exactly the way a hyperparameter chosen by a 123-point sweep on a validation set gives an optimistic validation number.

The two things that reach the proposer, and the one that does not

It is worth being very concrete about the information barrier, because in an implementation it is easy to breach by accident.

Look at the last row. The cleanest way to guarantee dev trajectories never leak is to never produce them. That is a real engineering lesson: an information barrier enforced by absence cannot be breached by a refactor, a logging change, or an over-helpful context builder. A barrier enforced by an instruction can be, and eventually will be.

Note also what the accept/reject decision itself carries. It is written to L, and it is a function of dev. So one bit per iteration does cross the barrier — by design, because the proposer needs to know whether its last idea survived. That single bit is the entirety of the leak, it is deliberate, and it is why the count of 123 bits above is the right accounting.

Watch the gate work, and watch it fail

Under the paper's policy the two curves stay coupled and both rise, slowly, with a lot of rejections. Under a train-only gate the train curve rises faster and further — and the dev curve rolls over and heads down. That divergence is not a bug in the optimizer. It is the optimizer working perfectly on the wrong objective.

What the gate cannot do

Three honest limits, so you deploy this idea with the right expectations.

It cannot detect evaluator bias. Both J_train and J_dev are computed by the same R_meta. If the evaluator systematically rewards something undesirable, the gate happily certifies edits that exploit it on both splits. The paper says as much and routes the fix through a human.

It cannot create statistical power it does not have. With small task sets, both J values are noisy sample means. A candidate can pass the gate by luck. Over many iterations, luck-passing accumulates in one direction, because passing is what gets kept.

It cannot stop dev from being consumed. As above — selection on dev is still selection. The only real defence is a third split that the loop has never influenced, which is why the frozen benchmark is a structural necessity and not just good reporting hygiene.

Chapter 6: Two Evaluators, On Purpose

A system that optimizes itself against a score needs two scores. One to learn from, one to be judged by. If they are the same object, the final number is a report of how well the system optimized its own exam, and it means nothing. This chapter builds both, and works the arithmetic of the second one by hand until the headline number's provenance is fully visible.

The separation, stated three times by the paper

The paper repeats this because it is load-bearing. From Section 3.2: "This optimization-time evaluator is distinct from the frozen PosterBench protocol used for final system comparison." From Section 5.1: "PosterBench is a frozen external evaluator, separate from the optimization-time evaluator R_meta… PosterBench evaluates completed systems and is neither optimized nor modified by the outer loop." From Appendix A.4: the two "use the same quality vocabulary… but serve a different role."

PosterBench, concretely

The corpus. A 100-paper Main Track spanning five disciplines — AI/ML, biomedicine and health, climate and earth environment, economics and policy, and physics and astronomy — plus PosterBench-mini, "a shared 10-paper subset" used for the controlled ablations. Every system receives "the same source paper and associated source assets, and its output is rendered to a common poster format before scoring."

The five disciplines are not decoration. A harness tuned on machine-learning papers has learned that evidence looks like a bar chart and a table of ablations. An economics paper's evidence is a regression table; a climate paper's is a map; a physics paper's may be a single instrument schematic. A benchmark that spans them is testing whether the harness learned design or learned the shape of NeurIPS submissions.

The seven dimensions, with the paper's weights and score modes:

Sum: 10 + 10 + 15 + 10 + 20 + 25 + 10. Add them: 10+10 = 20; +15 = 35; +10 = 45; +20 = 65; +25 = 90; +10 = 100. The weights sum to 100, so the rubric score is naturally on a 0–100 scale.

Equation 7: the weighted rubric

For paper p_i and candidate artifact A_i, the evaluator returns seven scores q_i ∈ [0,10]⁷ and forms:

The division by 10 rescales each dimension from 0–10 to 0–1 so the weights carry the units. A perfect poster scores q = (10,10,10,10,10,10,10) and gets ∑α_j × 1 = 100.

Equation 8: ceilings, and why the metric is not a weighted average

This is where the score stops being linear, and it is the most misread part of the protocol.

Four ceiling families, bounding, in the paper's words: "severe layout damage, insufficient presentation viability, confirmed visible failures, and protected render-integrity violations, respectively; inactive ceilings are 100." And the magnitude: "A standard P0 gate caps a score at 40, and more severe gate types may set a lower cap."

Order of operations is everything: cap first, then average. The paper warns explicitly that "the metric columns are dimension means, while Overall is the mean of capped poster scores and cannot generally be recovered by reweighting those displayed means."

Worked example: reconstructing the headline number — and failing informatively

Table 1 of the paper reports, for AutoDesign (DesignHarness + Claude Code + Claude 4.8) on the 100-paper Main Track: Overall 78.32, with dimension means

Apply Equation 7 to those means, one term at a time:

The reconstruction is off by 2.13, exactly as the paper says it must be. And the residual is not an error — it is a measurement. It is how much AutoDesign lost, on average, to record-level ceilings. Because min can only lower a score, the residual for any system must be ≥ 0. That gives us a free correctness check on our reading of Equation 7: compute it for every system in Table 1 and see whether all nine residuals come out non-negative.

All nine non-negative. That is not luck — it is what Equation 8 forces, and it is decent evidence that we are reading the protocol correctly.

Worked example: estimating how often a gate fires

We can push one step further, with assumptions stated up front. Suppose the only active ceiling is the standard P0 cap of 40, and suppose a gated record had roughly the system's mean rubric score before capping. Then each gated record loses (rubric − 40) points, and if a fraction f of records are gated:

For AutoDesign: f ≈ 2.13 / (80.45 − 40) = 2.13 / 40.45 = 0.053, about 5 posters in 100.

For Claude Code: f ≈ 6.85 / (76.86 − 40) = 6.85 / 36.86 = 0.186, about 19 posters in 100.

For Claude Design: f ≈ 4.01 / (74.88 − 40) = 4.01 / 34.88 = 0.115, about 12 in 100.

Read as an order of magnitude, that is a real engineering story: bare Claude Code produces a critically broken poster roughly one time in five, and attaching DesignHarness cuts that to roughly one in twenty. Which is precisely what a harness full of blocking validators, render checks, and fallbacks is for.

How R_meta gets built, and why that procedure is unusual

Most systems that need a learned reward train one. AutoDesign does something different, and the difference is worth dwelling on because it is cheap and reusable.

The procedure, from Section 3.2: humans annotate reference artifacts along the seven quality dimensions. An evaluator coding agent is given those annotated examples and implements the evaluator — writing code that "combin[es] rule-based checks for directly measurable properties with VLM-based judgments for perceptual properties such as aesthetics." Then it is frozen.

Reward hacking, dimension by dimension

Any fixed evaluator can be gamed given enough optimization pressure, and 123 iterations of a competent coding agent is real pressure. So ask, for each dimension, what the cheapest exploit would be — and what in the protocol blocks it. This is our analysis, not the paper's, but every defence named is something the paper actually specifies.

Read the third column as a whole and a design principle emerges: every dimension is checked by a dimension that pulls the other way. Faithfulness against Coverage. Coverage against Density and Readability. Density against Readability. Layout against the homogeneity check. The rubric is not seven independent measurements; it is a set of opposed pairs, and that opposition is what makes it hard to game with a single trick.

Programmatic versus VLM, and why the split falls where it does

Three dimensions are programmatic-plus-VLM, two are pure VLM, two are pure programmatic. That is not arbitrary.

Add the weights: 35 points fully deterministic, 20 points fully model-judged, 45 points a hybrid with a deterministic floor. So at most 20 of 100 points depend entirely on a model's opinion, and even those are conditioned on a rendered image and a source brief, with the system identity withheld.

That ratio is the protocol's real defence against evaluator drift. A benchmark that was 100% LLM-judge would be reproducible only as long as the judge model was, and would move under you at every model update. A benchmark that was 100% programmatic could not measure whether a poster communicates. Sixty-five percent anchored in deterministic checks, with a model consulted where no algorithm exists, is a defensible place to land — and it is worth copying.

Two more protocol details that change how you read the tables

The blinded style-homogeneity check. "For batches of at least 20 readable posters, a blinded style-homogeneity check may only reduce the professional-aesthetics score; it is not applied to PosterBench-mini, whose 10-poster scale is below that threshold."

So a system that makes 100 posters that all look like the same template can lose aesthetics points on the Main Track for exactly that — and cannot lose them on mini, where the check does not run. This is a real anti-degeneracy measure: it penalises the obvious way to game a design benchmark, which is to find one good template and apply it to everything. It also means Main Track and mini scores are not measured under identical rules, which matters in Chapter 8.

What the VLM judge sees. "Each VLM judgment receives the rendered image, a compact paper brief, and selected grounding signals, but no system identity or generation prompt." The judge does not know which system made the poster. That removes the most obvious source of judge bias, and it is the automatic-evaluation counterpart of the system-blind human study in Chapter 9.

Compute a score yourself

Drag the gate slider on AutoDesign and watch how few gated records it takes to erase a lead built out of dimension scores. At 20% gated, a system with an 80.45 rubric averages 80.45 − 0.20×40.45 = 80.45 − 8.09 = 72.36 — below bare Codex. Hard failures are not a rounding error in this protocol; they are the dominant term.

Chapter 7: What the Harness Became

Seven days, 224 subagents, at least 123 iterations, 54 accepted updates. What came out the other end? The paper answers by inspecting the final implementation rather than replaying the history, and identifies four stages. This chapter walks them with the data flow attached — what representation the artifact is in at each point, who consumes it, and why each engineering decision was made that way.

Stage 1 — Paper ingestion

The job: turn (x, c) into "a structured, provenance-aware context for subsequent generation and revision."

What it extracts, per the paper: document metadata and section outline; key passages supporting the main claims; figures and tables "together with their source locations."

What it produces: two objects, and the split matters.

And the property the whole thing is built around: "Every extracted element retains a reference to its location in x, so that source-derived statements and visual materials used in the artifact can be traced back to the source and checked during revision."

This is provenance as a data structure, not as a promise. A claim on the poster is not a string; it is a string with a pointer. That pointer is what makes a deterministic faithfulness check possible: the validator can ask "does every number on this poster resolve to a source location?" and get a yes/no without any model in the loop. Compare the alternative — asking a VLM "is this poster faithful?" — which is slower, stochastic, and unfalsifiable.

Stage 2 — Generation and revision

"The designer module is implemented as a coding agent that generates or revises the artifact from the ingested source context using the tools and skills available in the harness." At step k it conditions on y_k−1, f_k−1, and the ingested context — instantiating M_design from Equation 4.

The representation decision is the one to dwell on: "the artifact remains as editable HTML files throughout refinement… allowing revisions to be implemented as localized code edits without requiring regeneration of the entire output." For critique it "can be rendered or exported as a medium-specific preview, such as PNG, PPTX, or MP4."

Compare the three representations a design agent could carry, and the consequences follow mechanically:

The paper ties this to prior work on structured representations for editability (Qu et al., 2025; Liu et al., 2026b) and to the system prompt's own requirement: "Keep final text native and editable."

Stage 3 — Validation and critique

Chapter 3 covered the control flow; here is the data flow. The candidate y_k fans out into three consumers, each seeing a different projection of the same artifact:

Note that the two feedback sources are merged before reaching the designer, not delivered as two separate messages. That is a real design choice: it puts the conflict resolution in the harness rather than in the model. If the validator says "the analysis lane overflows by 34px" and the critic says "the analysis lane feels cramped, give it room," a merged signal can say "widen the analysis lane" once. Two unmerged signals invite the designer to make two edits, one of which undoes the other.

The paper places this pattern in its lineage: "This feedback-to-revision pattern is related to recursive self-refinement and agent-as-a-judge approaches (Madaan et al., 2023; Zhuge et al., 2025)."

Stage 4 — Finalization

The last mile, and it is more than a save button. Finalization "applies the remaining post-processing, such as final rendering adjustments, mathematical typesetting, and inlining of referenced assets, to produce a self-contained output."

Three operations, three reasons:

And recall the fallback path from Chapter 3: if the twelve-attempt budget runs out, the harness "uses the retained attempt history and applies a sequence of fallback mechanisms to identify a deliverable candidate while retaining essential safety and integrity constraints." Both exits — clean pass and fallback — converge on the same finalization stage. There is one delivery path, which means there is one place where "self-contained" is enforced.

Provenance, made mechanical

"Every extracted element retains a reference to its location in x" is one sentence, and it is the load-bearing sentence of the whole ingestion stage. Watch what it buys by comparing two systems that both claim to be faithful.

System A extracts strings. The brief says: "the method improves accuracy by 4.2 points." To check faithfulness you must ask a model whether that claim is supported by a twelve-page PDF. The check is slow, stochastic, and its failure mode is confident agreement.

System B extracts strings with pointers. The brief says: "the method improves accuracy by 4.2 points" — sourced from Table 3, row 4, column 5, page 7. Now three checks become deterministic:

None of these needs a model. All three are blocking-checkable. This is what the paper means by "source-derived statements and visual materials used in the artifact can be traced back to the source and checked during revision" — provenance is not a promise made in a prompt, it is a data structure that makes a class of hallucination mechanically detectable.

What ingestion must produce, field by field

The paper names the outputs at the level of "a content brief and a medium-specific artifact plan." Reading the downstream requirements backwards tells you what has to be in them, because every later check needs a field to check against.

Read the third column and you are reading the PosterBench dimension list in disguise. That is not a coincidence: the evaluator defines what failure means, and the outer loop grows exactly the ingestion fields needed to make those failures preventable.

Why finalization inlines assets — a concrete failure

"Inlining of referenced assets, to produce a self-contained output" reads like housekeeping. It is not.

Consider what the artifact looks like mid-loop: an HTML file plus a directory of extracted figure images, referenced by relative path. That is exactly right during refinement — the designer edits the HTML, the renderer resolves the paths, everything works. Now ship it.

The ordering matters too. Inlining during refinement would bloat every intermediate file with base64 payloads that the designer has to scroll past on every edit, and mathematical typesetting during refinement would keep changing element geometry underneath the layout checks. Both operations belong exactly where the optimized harness put them: after the layout is final, before delivery.

Mapping the stages back onto the five components

The appendix gives the mapping explicitly, and it is worth tabulating because it tells you where the optimizer was allowed to work.

The appendix also flags a distinction that is easy to lose: the image-native evaluator inside the design harness "is distinct from the outer-loop evaluator R_meta and from the frozen PosterBench protocol used for final comparison." Three evaluators now, at three levels — the critic inside one poster's loop, the training signal across tasks, and the exam across systems. Keeping them straight is a real part of understanding this system.

The four stages as answers to four questions

Architecture diagrams are decoration unless each box answers a question you could otherwise have gotten wrong. Here are the four questions DesignHarness ended up answering, and what a system that skips each one produces.

Every one of those four failures is something the paper's five-component decomposition can locate, and something the outer loop could plausibly have discovered from trajectories. Which is the claim being made: this architecture was not designed, it was found — one gated component edit at a time, starting from a designer and a critic.

What the paper actually shows, and what it only asserts

Be precise here, because this is the chapter most vulnerable to over-reading. The paper characterizes DesignHarness "by examining the final implementation obtained through meta-harness optimization." It reports the four stages, and it reports summary statistics for the campaign that produced them. It does not report a chronological trace of which iteration added which stage.

The appendix is explicit about this, and unusually careful: the architecture figure "is neither a second taxonomy nor a record of individual outer-loop iterations," and its diagonal path "highlights additions in this architectural view; it does not denote individual meta-harness iterations."

The other three media, and what it would take to make them count

DesignHarness already emits slides, webpages, and conference videos. The paper calls these pilots and refuses to score them. Its reason is a checklist, and the checklist is the most useful transfer advice in the paper:

The last row is the deepest point. It is tempting to think the seven dimensions are "quality" and would carry over. They are not; they are quality for a poster, weighted for a poster's viewing conditions. Move the medium and the weights have to be re-argued from the communication setting up. The paper's own future-work section says exactly this, and its restraint in not reporting slide or video numbers is the same restraint that makes its poster numbers worth reading.

What it can already make besides posters

"The resulting implementation supports multiple output media, including academic posters, presentation slides, videos, and web pages." The paper's pilot artifacts show paper-to-slide, paper-to-webpage, and paper-to-conference-video outputs.

But it is scrupulous about the status of those: "PosterBench formally evaluates academic posters only; the slide, webpage, and video artifacts therefore remain pilots." Which is the correct call, and the reason is stated in the same paragraph — each medium "needs source–output data, an evaluator, a rendering and validation gate, and an objective tailored to its communication setting." Without an evaluator there is no R_meta, without R_meta there is no outer loop, and without a benchmark there is no honest number. A demo is not a result.

Chapter 8: Reading the Tables Honestly

The numbers are good. Some of them are better than the paper claims and some are weaker, and the difference is always in what was held fixed and how many papers were involved. This chapter reads every table the way you would read one from a competitor.

The Main Track: 100 papers, every system

The comparison the abstract leads with is the matched one, and it is a fair fight: AutoDesign and Claude Design both run on Claude Code with Claude 4.8. Only the design harness differs.

The second comparison is the one worth staring at, because it is the harness's true contribution over doing nothing:

And here is the first genuinely uncomfortable number in the paper, which the paper reports without flinching: bare Codex on GPT 5.5 scores 73.37 — higher than Claude Design (70.87), higher than OpenDesign (69.45), higher than bare Claude Code (70.01). A coding agent with no design harness at all beats two purpose-built design systems, one of them commercial.

The three human-crafted workflows

PosterGen (56.71), Any2Poster (49.09), and Paper2Poster (44.61) are hand-designed pipelines, all running on Claude 4.8. They lose to every agentic configuration except the weakest models. But look at where they lose, using the dimension columns:

Any2Poster has the best Layout score of any system in the table — 9.59, better than AutoDesign's 8.55. A hand-built pipeline with a fixed template produces geometrically flawless posters. It scores 49.09 anyway because Coverage is 5.44 and Aesthetics is 3.10: a perfectly laid-out poster that omits half the paper.

Paper2Poster is the mirror image: Density 8.36 (second-highest in the table) with Coverage 2.35 and Aesthetics 1.69. Dense and empty at the same time — which, given Density's operational definition ("information occupancy, OCR text coverage, blank interiors, and pasted paper-body screenshots"), is exactly what pasting screenshots of the paper body would produce.

The ablation that carries the paper: attaching the harness

Table 4 holds the model and the coding agent fixed and toggles only the presence of DesignHarness, on PosterBench-mini.

Verify the abstract's averages by hand. Before:

After:

Both match the abstract exactly. The improvement is 67.39 − 54.99 = 12.40 points, and every one of the seven configurations improves — no exceptions, no regressions.

The pattern inside the gains — a derivation the paper does not do

Look at the two columns together. The two strongest baselines gain the least (+5.59, +5.01); the weakest baseline gains the most (+19.56). Let us quantify that. Compute the correlation between baseline score and gain across the seven rows.

Baseline mean is 54.99 (above). Gain mean: (5.59 + 5.01 + 17.82 + 12.92 + 14.01 + 11.87 + 19.56) = 86.78, and 86.78 / 7 = 12.40 — which of course equals the difference of the means, a useful arithmetic check.

Running the product-moment correlation over the seven paired deviations gives

Strongly negative. And the consequence shows up in the spread: the standard deviation of scores across the seven configurations falls from 14.28 before the harness to 10.06 after — a 30% compression.

What that means in plain terms: the harness substitutes for model capability. A frontier model already knows most of what the harness encodes about provenance, layout discipline, and when to stop; a weaker model does not, and the harness supplies it. DeepSeek V4 Pro with DesignHarness (54.29) outscores bare Doubao Seed 2.1 (54.01) and gets within 10 points of bare Claude Code (69.55) — a jump that would otherwise require a much stronger model.

Cost versus performance

Four points on the paper's Pareto frontier, on PosterBench-mini, using a normalized designer-only API cost proxy:

The paper's summary: "Doubao reaches 88% of the GPT-5.5 score at 27% of its cost." Check it: 71.83 / 81.46 = 0.8818 → 88%. And 2.75 / 10.02 = 0.2745 → 27%. Both exact.

Now a step the paper does not take. The marginal column is non-monotone: the middle step costs $1.79 per point while the final step costs $0.35 per point. That means Claude 4.8, although Pareto-optimal (nothing cheaper scores higher), sits below the chord connecting Doubao and GPT-5.5. Compute the chord's height at Claude 4.8's price:

So if you had a budget of $7.63 per poster and could split it across posters, you would be better off running Doubao on some and GPT-5.5 on others than running Claude 4.8 on all of them. That is a portfolio argument, not a per-poster one — but it is exactly the kind of decision a cost table exists to inform, and it is invisible unless you check convexity.

The instability nobody should skip: mini versus Main

PosterBench-mini is "a shared 10-paper subset" of the 100-paper Main Track. Same frozen evaluator. Now put the same two systems side by side on both:

Since mini is a subset of Main, we can back out the scores on the other 90 papers. For the Codex configuration:

And for the Claude Code configuration:

On the ten mini papers, Codex leads by 6.90. On the other ninety, Claude Code leads by 1.16. The ten-paper subset is not a small-scale replica of the benchmark; it is a sample that happens to favour one configuration by seven points.

Two caveats on our own derivation, in fairness. First, the blinded style-homogeneity check runs on batches of at least 20 readable posters and therefore applies to the Main Track but explicitly not to mini — so the two are not scored under literally identical rules, and part of the mini → Main shift for the Codex configuration could be that check reducing its professional-aesthetics score. Second, the 90-paper figures are exact only if mini is a strict subset of the Main Track scored identically, which the paper's "shared 10-paper subset" wording supports but does not spell out arithmetically.

The Coding Harness Track, and a result to hold at arm's length

Table 3(b) fixes the design harness to AutoDesign and the model to GLM 5.2, and varies only the coding agent:

Read that top row carefully. With a mid-tier open model (GLM 5.2, whose bare score is 50.32), swapping the coding harness from Claude Code to Kimi Code moves the score from 64.33 to 82.31 — a 17.98-point swing, larger than the entire design-harness effect on strong models, and higher than any number in the Main Track table.

Take that seriously and take it sceptically at the same time. Seriously: it says the coding harness — the agent scaffold that runs the code edits — interacts strongly with the model, and that "which coding agent" is a first-class variable rather than an implementation detail. Sceptically: it is N = 10, single-run, on a subset we have just shown can move a ranking by seven points. The paper reports no repeats, no confidence intervals, and no per-paper breakdown for this track.

The Model Track

Table 3(c) fixes AutoDesign and Claude Code and varies the model: Claude 4.8 (74.56), Seed 2.1 Pro (71.83), Kimi K2.7 (70.12), GLM 5.2 (64.33), LongCat 2.0 (55.13), DeepSeek V4 Pro (54.29). A 20.27-point spread from model choice alone, under an identical harness. So the harness does not erase model quality — it compresses the spread (the −0.84 correlation above) without eliminating it.

Explore the ablations

Everything the tables do not tell you

Chapter 9: Asking Humans

An automatic benchmark that a system was optimized near is a suspicious witness, even when the optimization used a different evaluator. So the paper runs a system-blind human study and, more usefully, asks how well the benchmark and the humans agree. This chapter builds the statistics from zero.

The design

Eleven volunteer reviewers. Four systems — AutoDesign, Claude Code, OpenDesign, Claude Design — on the 100 Main Track papers. Each task shows two anonymous posters made from the same paper, with the paper's title, abstract, and PDF as shared context. No system, model, or harness identity is disclosed. The reviewer picks the left poster, the right poster, "approximately equal," or skip, and may flag a critical failure. Presentation order is balanced left–right.

The full roster size is derived in the paper:

where C(4,2) = 4! / (2! · 2!) = 24 / 4 = 6 is the number of unordered system pairs: {AD,CC}, {AD,OD}, {AD,CD}, {CC,OD}, {CC,CD}, {OD,CD}. Every pair on every paper. The paper explains why this matters for the estimator: it provides "equal paper coverage and a connected graph for Bradley–Terry estimation" — connectivity is required, because if some system were never compared against the rest, its strength would be unidentifiable.

Bradley–Terry, derived

Give each system a latent strength β_i ∈ R. The Bradley–Terry model (1952) says the probability that i is preferred to j is that system's exponentiated strength as a share of the pair's total:

Divide numerator and denominator by exp(β_i):

So Bradley–Terry is a logistic function of the strength difference — which means three things immediately. Only differences are identifiable (adding a constant to every β changes nothing, so you fix a reference). Equal strengths give exactly 0.5. And the model is inverted by the logit:

Ties are handled by giving each side half a win; skips are dropped.

Worked example: recovering the strengths from the reported rates

The paper gives AutoDesign's tie-adjusted empirical preference against each baseline. Convert each into a strength gap with the logit.

Set β_AutoDesign = 0 as the reference. Then β_CC = −0.460, β_OD = −0.536, β_CD = −0.735. Now the model makes predictions about pairs the table never showed us. For Claude Code versus Claude Design:

So Bradley–Terry predicts Claude Code beats Claude Design about 57% of the time. Similarly σ(0.536 − 0.460) = σ(0.076) = 0.519 for Claude Code over OpenDesign, and σ(0.735 − 0.536) = σ(0.199) = 0.550 for OpenDesign over Claude Design. That is what a model buys you over a table of raw rates: predictions for comparisons, and a single ranking that is forced to be transitive.

Worked example: verifying the headline 64.0%

The paper reports AutoDesign's "probability of beating a uniformly sampled alternative" as 64.0%. With three opponents, that is the average of the three win probabilities implied by the fitted strengths. Compute each from σ:

Exactly the reported figure. And note what the round trip proves: we went from the empirical rates to strengths back to probabilities and landed on the paper's fitted estimate. That only happens when the pairwise data are close to Bradley–Terry-consistent — that is, when there are no large intransitivities of the form "A beats B, B beats C, C beats A." The consistency is itself a small quality signal about the judgments.

How much of the study actually got done

Reported: 936 submitted responses — 933 ranking judgments and 3 skips. The roster is 600 tasks per reviewer, and there are 11 reviewers.

The paper is upfront about this: the "0/600" counter "denotes this complete roster, rather than a requirement to submit 600 judgments," and "submitted non-skip decisions are retained and uncompleted assignments are not imputed." Good practice — no imputation, no silent filling.

But the density is worth computing. If 933 judgments spread over 600 distinct tasks, that is 1.56 judgments per task on average, and volunteers rarely spread evenly. Many of the 600 paper–pair cells got one judgment or none. That thinness is the main reason the bootstrap interval is 22.6 points wide, and it is why the crossed resampling — over papers and reviewers — is the right choice: both are sources of variation with few draws.

The paper also reports a nominal Krippendorff coefficient of 0.101 as "an agreement diagnostic over paper–system-pair items, not the ranking estimator." Read plainly: reviewers agree with each other only slightly more than chance on which of two posters is better. That is expected for aesthetic and communicative judgment and is not an indictment of the study — but it is the reason you need many judgments to see a signal, and 933 across 600 cells is not many.

Does the benchmark predict the humans?

This is the most useful part of the section, because it validates the automatic protocol rather than the system.

Poster-level correlation. Comparing each poster's PosterBench Score with its tie-adjusted human preference gives r = 0.34, with a paper-cluster bootstrap 95% interval of [0.22, 0.44]. Square it: r² = 0.1156, so the benchmark accounts for roughly 12% of the variance in poster-level human preference. Positive, significant, and modest.

The paper does not oversell it: "This is a useful property of the protocol rather than a requirement that it duplicate human preference: PosterBench also evaluates Faithfulness, Coverage, Density, Visual Evidence, Layout, Readability, and Aesthetics, whereas each blind judgment asks for an immediate pairwise choice." A rubric that checks numeric grounding against the source is measuring something a reviewer glancing at two posters simply cannot see.

Agreement as a function of margin. This is the sharper result. Using 919 of the 933 non-skip judgments (14 are excluded because the two posters had identical PosterBench Scores and so there is no benchmark-preferred direction), the probability that the benchmark-preferred poster matches the human choice rises with the score gap:

The paper draws the same conclusion: "the benchmark offers more than a global system ranking: a large score gap identifies comparisons in which human preference is substantially more consistent."

Feel the model

Where the strengths come from — fitting Bradley–Terry

We inverted the reported rates into strengths, which is fine when you have one rate per pair. With many noisy comparisons you fit instead, and the fit is worth seeing because it explains what "ties contribute one half-win" is actually doing.

Let w_ij be the number of times system i was preferred over j. Each comparison is a Bernoulli trial with success probability σ(β_i − β_j), so the log-likelihood of the whole dataset is the sum over ordered pairs:

Maximise that over β and you have the estimate. Three properties follow, and each one shows up in the paper's reporting.

Only differences are identifiable. Add a constant c to every β_i and every difference β_i − β_j is unchanged, so ℓ is unchanged. The likelihood has a flat direction. That is why the paper does not report raw strengths at all — it reports "the probability of beating a uniformly sampled alternative," a quantity that depends only on differences and is therefore well defined.

Connectivity is required. If the comparison graph split into two components — some systems never compared against the others — the relative strength across the split would be unconstrained by the data and the likelihood would be flat in a second direction. The paper's full roster of all six unordered pairs on every paper guarantees a connected graph by construction, which is exactly why it says the roster "provid[es] equal paper coverage and a connected graph for Bradley–Terry estimation."

Ties as half-wins is the maximum-entropy choice. Bradley–Terry in its basic form has no tie outcome; a judgment is a win or a loss. Faced with "approximately equal," you can drop it (throwing away real information — a tie is genuine evidence that two systems are close), model it explicitly (the Davidson extension, which needs another parameter and more data than 933 judgments can support), or split it. Splitting adds 0.5 to w_ij and 0.5 to w_ji, which pulls the estimated difference toward zero — precisely the effect a tie should have. It is the cheapest correct-in-spirit option, and with a thin dataset the cheapest option that does not add parameters is usually right.

Why the bootstrap is crossed, and why it has to be

The paper reports "2,000 crossed bootstrap resamples of papers and reviewers." That word crossed is doing real statistical work.

A naive bootstrap would resample the 933 individual judgments. That treats every judgment as an independent draw — and they are emphatically not. Two judgments on the same paper share whatever makes that paper easy or hard to poster. Two judgments from the same reviewer share whatever that person happens to like. Ignoring both correlations shrinks your interval, and produces confident nonsense.

What r = 0.34 does and does not license

A poster-level correlation of 0.34 between PosterBench Score and human preference is easy to over- or under-read. Both mistakes are common enough to be worth heading off.

The under-read: "0.34 is weak, so the benchmark is bad." The two quantities are not measuring the same thing on purpose. PosterBench audits numeric grounding against the source, OCR-level readability, render integrity, and coverage against a brief. A reviewer glancing at two posters for thirty seconds sees none of that. A benchmark that correlated 0.95 with a snap aesthetic judgment would be a worse benchmark for a scientific artifact, not a better one — it would have stopped measuring faithfulness.

The over-read: "positive and significant, so score gaps mean preference gaps." Square it: r² = 0.34² = 0.1156. The benchmark explains about 12% of the variance in poster-level human preference. Eighty-eight percent is elsewhere — taste, the specific paper, the reviewer, the moment. Any argument of the form "our system scores 3 points higher, therefore people prefer it" is unsupported, and the paper's own margin analysis says so numerically: at 0–3-point gaps, agreement is 51.9%.

Now do the same reading at the other end. At gaps of 20 or more, agreement is 74.4% — 24.4 points above chance. Convert that into a Bradley–Terry strength gap to feel its size:

Compare that with the largest system-level gap the study found, AutoDesign over Claude Design, at Δβ = 0.735. A 20-point PosterBench margin corresponds to a larger perceived difference than the gap between the best and worst systems in the field. Which is a fair summary of the protocol's resolution: it cannot see small differences, and when it sees a large one, that difference is very real.

What the human study does and does not establish

Chapter 10: What Generalises

Strip away the posters. What is left is a recipe for improving any long-horizon agent whose outputs can be scored, and a set of conditions under which the recipe applies. This chapter names both, reads the paper's own future-work section, and connects everything to the rest of this site.

The transferable pattern, in six lines

When this applies to your system — and when it does not

The paper's own future directions, read straight

Other media. DesignHarness already produces slides, webpages, and conference videos as pilots. The paper refuses to call them results, and gives the requirement: each medium "needs source–output data, an evaluator, a rendering and validation gate, and an objective tailored to its communication setting." It also flags what might transfer: "shared context construction, preference memories, and repair histories could nevertheless provide a substrate for reusing experience across media, provided that the transfer is evaluated against medium-specific objectives."

Better component selection. Right now the planner picks which of the five components to edit. The paper wants that to be principled: "A selector should choose next bounded update from failure attribution, uncertainty, expected improvement, and component interactions." That is a bandit problem in all but name — five arms, noisy rewards, and interactions between arms.

Evaluator evolution, with guardrails. The sharpest sentence in the future-work section: "Any adaptive evaluator must remain versioned and anchored by frozen reference tasks, adversarial probes, and periodic human audits so optimization doesn't reward-hack a moving target." If the evaluator can change and the optimizer can influence it, you have closed a loop that has no fixed point except whatever is easiest to score well on.

Co-evolution with training. "Harness optimization can complement model post-training: long-horizon trajectories and repair outcomes provide execution-time supervision, whereas the model supplies the reasoning and coding capabilities. Joint training should preserve this division while evaluating both layers against shared held-out objectives." Note that the held-out discipline is carried forward into the joint setting — the paper is consistent about that.

Where this sits among its neighbours

How this fails in the wild — five ways, and what to watch

You will build a version of this. Here are the ways it goes wrong, each with the symptom you will actually observe first and the instrument that catches it.

What to instrument on day one

If you only build six dashboards, build these. Each corresponds to a question this paper had to answer.

Notice how many of these the paper itself does not report. Acceptance rate can be inferred (54 of at least 123); the rest cannot. That is not a criticism so much as an observation about where this field's reporting norms currently sit — and an easy way for your write-up to be better than the state of the art.

Five sentences that survive if you forget everything else

Cheat sheet

Where to go from here on this site

Build a small version this week

References

1. Luo, Y., Jiang, H., Zou, J., Huang, X., Yan, W., Li, H., Yue, Z., Li, J., Chen, X., Zhao, X., Liu, J., Cui, J., Shen, Z., Li, X. "AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design," 2026 — arXiv:2608.13560. The paper this lesson is built on. Code: github.com/Yaxin9Luo/AutoDesign.
2. Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., Finn, C. "Meta-Harness: End-to-End Optimization of Model Harnesses," 2026 — arXiv:2603.28052. The harness-as-optimization-target framing AutoDesign builds on.
3. Nguyen, M., Nguyen, Q., Vuong, P. "Recursive Self-Evolving Agents via Held-Out Selection," 2026 — arXiv:2606.28374. The precedent for gating persistent updates on an independent split — the direct ancestor of Chapter 5.
4. Ren, Z. et al. "Self-Improvements in Modern Agentic Systems: A Survey," 2026 — arXiv:2607.13104. Where the model-versus-scaffold distinction is laid out.
5. Robeyns, M., Szummer, M., Aitchison, L. "A Self-Improving Coding Agent," 2025 — arXiv:2504.15228; Cai, Q. et al. "MOSS: Self-Evolution through Source-Level Rewriting," 2026 — arXiv:2605.22794. Agents that edit their own source from execution evidence.
6. Chen, T. et al. "HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry," 2026 — arXiv:2606.14249; Lin, J. et al. "Agentic Harness Engineering," 2026 — arXiv:2604.25850; Lee, H. et al. "Recursive Harness Self-Improvement," 2026 — arXiv:2607.15524; Karten, S. et al. "Continual Harness," 2026 — arXiv:2605.09998; Liu, Z. et al. "Adaptive Auto-Harness," 2026 — arXiv:2606.01770. The immediate neighbourhood.
7. Madaan, A. et al. "Self-Refine: Iterative Refinement with Self-Feedback," NeurIPS 2023. The artifact-level inner loop AutoDesign starts from.
8. Shinn, N. et al. "Reflexion: Language Agents with Verbal Reinforcement Learning," NeurIPS 2023; Wang, G. et al. "Voyager," 2023 — arXiv:2305.16291; Zhao, A. et al. "ExpeL: LLM Agents Are Experiential Learners," AAAI 2024 — arXiv:2308.10144. Experience that persists across attempts but does not edit the harness.
9. Yuksekgonul, M. et al. "TextGrad: Automatic Differentiation via Text," 2024 — arXiv:2406.07496; Khattab, O. et al. "DSPy," ICLR 2024; Agrawal, L. et al. "GEPA: Reflective Prompt Evolution," ICLR 2026. Component- and pipeline-level optimization.
10. Zelikman, E. et al. "STOP: Self-Taught Optimizer," COLM 2024 — arXiv:2310.02304; Hu, S., Lu, C., Clune, J. "Automated Design of Agentic Systems," ICLR 2025; Zhang, J. et al. "AFlow," ICLR 2025; Zhuge, M. et al. "GPTSwarm," ICML 2024. Search over workflow structure.
11. Zhang, J. et al. "Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents," ICLR 2026 — arXiv:2505.22954; Wang, W. et al. "Huxley-Gödel Machine," 2025 — arXiv:2510.21614; Schmidhuber, J. "Gödel Machines," 2006 — arXiv:cs/0309048. The self-rewriting lineage, from proof-based ideal to empirical evolution.
12. Zhuge, M. et al. "Agent-as-a-Judge: Evaluate Agents with Agents," ICML 2025. Process-level evidence alongside final-outcome assessment.
13. Pang, W. et al. "Paper2Poster," NeurIPS 2025 D&B; Zhang, Z. et al. "PosterGen," 2025 — arXiv:2508.17188; Vinaykumar, A. et al. "Any2Poster," 2026 — arXiv:2606.02915; Sun, T. et al. "P2P," ICLR 2026; Choi, J. et al. "PosterForest," ACL 2026; Wang, H. et al. "SciPostLayout," CVPRW 2024. The paper-to-poster baselines in Table 1.
14. Liu, Z. et al. "DesignAsCode," 2026 — arXiv:2602.17690; Qu, Y. et al. "IGD: Instructional Graphic Design," ICCV 2025; Si, C. et al. "Design2Code," NAACL 2025. Why the artifact stays as editable code.
15. Bradley, R. A., Terry, M. E. "Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons," Biometrika 39(3/4):324–345, 1952. The 74-year-old model behind Chapter 9.