The Wiki Never Rolls Back

Chapter 1: The Key Insight

In 2026, Andrej Karpathy sketched an idea he called the "LLM Wiki": rather than letting an agent's hard-won experience evaporate at the end of every run, compile it into something persistent — it survives across runs — and compounding — each new run makes it richer, not just different. This paper takes that sketch and asks a concrete engineering question: can an agent's experience be compiled into exactly that kind of knowledge, and used to drive skill evolution?

"Persistent" and "compounding" sound like two words for the same thing, but they're pulling different weight. Persistent just means it doesn't vanish — a log file on disk is persistent. Compounding is the stronger claim: each new addition makes the WHOLE thing more useful, not merely bigger. A wiki that only ever appended noise would be persistent without being compounding. WikiSkill has to earn the second property, not just the first, and that's what most of this lesson's remaining chapters are actually about.

Chapter 0 already showed you the emotional shape of the answer — a note that survives a rollback beats an agent with no memory at all. This chapter turns that feeling into a design decision you can actually build.

Notice, too, what the paper is explicitly NOT claiming by citing Karpathy's gist. It isn't claiming the idea of a persistent knowledge layer is novel in the abstract — wikis, changelogs, and institutional memory all predate this paper by decades. What's novel is the specific engineering question: how do you wire a persistent knowledge layer into an automated skill-evolution loop, with real gating, real rollback, and real agents reading and writing it under a defined protocol? That's an implementation question, and this lesson's remaining eleven chapters are the answer.

The Tempting Wrong Idea

Here's a tempting simplification: just keep making the SKILL.md file better. Every iteration, read what went wrong, edit the file, save it, move on. One evolving artifact, one job. It's simpler to build than three separate layers with three separate rules.

This isn't a strawman — it's genuinely how a first attempt at "automated skill improvement" would most naturally get built, and something close to it is exactly what a naive read of "agents should improve their own instructions" suggests. The failure mode only shows up once you ask a specific, slightly unglamorous question: what happens on the FIRST rejection?

It breaks the moment you ask what happens on rejection. A skill file that's also the record of the attempt that produced it has nowhere to go when that attempt is rolled back — the edit reverts, and whatever insight motivated the edit reverts with it, because insight and procedure were living in the same file. The next iteration starts from a position that looks identical to the position before the rejected attempt, except now it's missing the one piece of information that would stop it from proposing the exact same rejected idea again.

You could patch around this, of course — keep a separate changelog next to the skill file, append every rejected diff to it by hand. But now you've reinvented a second file with its own rule (append-only, never rolled back) sitting next to a first file with a different rule (freely reversible). Congratulations: you've reinvented two of WikiSkill's three layers by trying to avoid building them on purpose.

Trace It Through Two Iterations, Both Ways

Suppose iteration 3 proposes adding a rule to the active skill, and validation rejects it. Watch what each design does next.

Same rejection, same iteration count, same starting skill file — and only one of the two designs actually learns anything from the rejection. That's not a difference in the AGENT's intelligence between the two scenarios; it's entirely a difference in what the workspace itself made possible for the agent to notice.

Run that same trace-through for five more iterations instead of two, and the gap only widens. Every iteration the single-file design forgets its rejections is another iteration it can re-propose them; every iteration the three-layer design remembers is one more line in an audit trail the Proposer can consult before it ever wastes a validation pass on a repeat.

Three Jobs, Three Different Rules

WikiSkill's actual move is to stop asking one artifact to do three jobs, and instead give each job its own file, governed by its own rule:

• What happened — the raw execution trace. This should never change once written; it's the ground truth of what the agent actually did.
• What we've learned — the wiki. This should only ever grow; every iteration's insight adds to it, and nothing already there gets deleted just because a later proposal failed.
• What to do about it — the active skill. This should be freely reversible; if a change makes things worse, throw it away and go back to what worked, with zero cost to the knowledge that produced the attempt.

Read those three bullets again as three answers to "what should survive a rollback." The raw trace was never at risk — it's immutable, so the question doesn't even apply to it. The wiki is designed to survive by construction. Only the active skill is meant to be disposable, and that's exactly the property Chapter 8's gate depends on.

Watch how differently "compiled" behaves from "scattered" as iterations accumulate. On the left, each of the three prior methods keeps some form of history, but it's tied to that one run's internal bookkeeping — nothing connects iteration 2's insight to iteration 3's proposal except whatever got folded into the next optimization pass. On the right, one artifact just keeps collecting.

Notice what the left side is actually showing: each method's dot at iteration N doesn't connect to a dot at iteration N−1. There's no line, because there's no shared, standing artifact for a line to point through — whatever happened at iteration N−1 already got folded into that iteration's own proposal-and-gate cycle and is, for practical purposes, gone. The right side never does that. Every ring stays. The blob only gets bigger.

Try clicking through all five iterations on the right panel specifically, in order, reading each ring's label as you go. You end up with a small, ordered history of exactly what the wiki learned and when — something you cannot reconstruct from the left panel no matter how carefully you click through it, because the left panel's earlier dots were never designed to still mean anything once a later iteration passed them by.

That's the whole test, really: can you reconstruct the history of a system's learning by inspecting its current state alone? For the wiki, yes — every ring is still there, still labeled, still clickable. For a bounded frontier or an epoch-scoped reflection log, the honest answer is "only partially, and only for whatever happened recently enough to still be resident."

Naming the Three Layers

Here are their names, previewed — Chapter 4 is the full interactive tour of what's actually inside each one, using the paper's own real worked example rather than the illustrative sim above:

Three rows, three rules, and every mechanism in Chapters 5 through 8 is ultimately just an implementation detail of one of these three rows.

That last sentence is worth internalizing before Chapter 2, because it's the lens every remaining chapter uses. The Inference Agent, the Wiki Maintainer, the Skill Proposer, and the gating mechanism each get read and write permissions to exactly some of these three layers — never all of them.

The specific pattern of who can touch what turns out to be almost the entire design of the paper. Here's the shape of it, previewed before you've even met the four components by name — Chapters 5 through 8 fill in every cell:

Notice the one row that's conspicuously absent: nothing in this table lets the Inference Agent read wiki/. That's not an oversight — it's a deliberate restriction the paper tests directly, and Chapter 11's ablation shows exactly what happens to skill quality when you remove it.

Hold that specific absence in mind through Chapter 5, where you'll watch the loop run and notice the same gap live — a step that reads skills but never reads the wiki, by design, not by oversight.

What "Bounded" Actually Costs You

It's worth being concrete about why "scattered" isn't just a less tidy version of "compiled" — it's a structurally different guarantee. Take EvoSkill, the closest of the three prior methods to keeping something wiki-like: it maintains a frontier of candidate programs, but that frontier is bounded. New entries compete with old ones for a fixed number of slots.

Picture ten iterations of evolution. In WikiSkill, all ten iterations' worth of patterns are still sitting in wiki/patterns/ at iteration ten, however small or old any individual one is. In a bounded-frontier method, an insight from iteration two that never got reinforced by a later, higher-scoring candidate can simply fall off the end of the frontier by iteration ten — not because it was wrong, but because something else scored better and the slot count is fixed.

That's the practical shape of "scattered." It isn't that EvoSkill throws information away carelessly; it's that boundedness was never designed to answer the question "what did we learn," only "what's our current best guess." Those are different questions, and the wiki is WikiSkill's answer to the first one specifically.

One clarification before Chapter 2 makes this comparison exhaustive: the wiki being unbounded doesn't mean it's unmanaged. Chapter 6's Wiki Maintainer still decides what's worth writing down, and index descriptions still have to earn their place by being specific rather than vague. "Unbounded" describes the storage guarantee — nothing gets deleted for capacity reasons — not a license to write everything down indiscriminately.

Hold onto that distinction — between an unbounded STORAGE guarantee and an unmanaged, indiscriminate DUMP — because it resolves what would otherwise look like a contradiction two chapters from now, when Chapter 3 introduces a wiki that starts at W₀ = ∅ and Chapter 6 shows exactly how selective the Wiki Maintainer actually is about what earns a place in it.

That's the whole chapter, compressed: three jobs, three files, three rules — and every future chapter is just filling in the details of exactly one row from the table above.

One quiz, then Chapter 2 hands you the vocabulary needed to judge every claim this lesson makes from here forward.

Answer it from the flow diagrams above, not from memory of a definition — the whole point of tracing it through twice was to make the answer something you can point at, not something you have to recall.

Chapter 2: Skills & Rivals

Every one of the four methods in this lesson — Trace2Skill, EvoSkill, SkillOpt, and WikiSkill — shares the same outer loop: roll out an agent, analyze what happened, propose an edit, gate the edit on a validation set. If you only glance at the loop shape, all four look nearly identical. The entire paper is about one design choice living inside that shared loop: what happens to the analysis after the edit is applied or rejected.

Before this lesson can judge that design choice, you need two pieces of vocabulary: what a "skill" literally contains as a file, and what each of the three prior methods keeps and discards. Both are quick, and Chapter 6 onward leans on them constantly.

Treat this chapter as a glossary you're allowed to skim quickly and return to — nothing here requires the depth of a full chapter's worth of derivation, and everything here gets used, by name, repeatedly for the rest of the lesson.

Why bother with three rivals at all, instead of jumping straight to WikiSkill's own architecture? Because "compiled beats scattered" is only an interesting claim if the alternatives are genuinely competitive on everything else — same rollout discipline, same gating rule, same benchmarks. If Trace2Skill, EvoSkill, and SkillOpt were sloppier systems overall, a win against them wouldn't tell you the persistent wiki specifically was the reason. They aren't sloppy; Chapter 9's results treat them as the real, hard-to-beat baselines they are.

All three rivals are recent, 2026-vintage systems in their own right — this isn't a comparison against a strawman from years ago. Ni et al.'s Trace2Skill, Alzubi et al.'s EvoSkill, and Yang et al.'s SkillOpt were each published as serious contributions to exactly this problem, and WikiSkill's authors cite and reproduce them faithfully rather than reimplementing a weaker version to beat.

What a Skill Actually Is

Strip away the framing and a skill is unglamorous: a directory, with a name, sitting on disk. Nothing about it touches model weights — it's pure filesystem, which is exactly what makes it auditable, diffable, and, critically for Chapter 8, something you can delete and go back to what worked. Every skill directory in this paper contains exactly two files.

"Directory" is the right word, not "file" — the paper's own definition allows a skill to bundle scripts and other resources alongside its two markdown files, though the two examples in this chapter's widget keep to just the required pair for clarity.

SKILL.md carries the actual procedure — a name and a concise description in its frontmatter (so an agent can decide relevance without opening the file), followed by the full instructions and the conditions under which they apply. PURPOSE.md does a different job entirely: it maps the skill back to the wiki patterns that motivated its creation or its most recent edit, so a reader — human or agent — can always answer "why does this skill exist" without archaeology.

The "conditions under which they apply" phrase is doing real work, too — the paper's own framing splits every skill into applicability conditions on top of the instructions themselves. That's why both examples above carry a "When to Apply" AND a "When NOT to Apply" section: a procedure without a stated boundary is a procedure an agent might reach for on tasks it was never validated against.

That frontmatter split — a short description up top, full instructions below — isn't decorative. It's what the paper calls progressive disclosure: an agent deciding whether a skill is relevant only ever has to read the one-line description first. It only pays the cost of loading the full procedure once it's already decided the skill is worth applying. Multiply this by however many skills are active at once, and the difference between "read every skill's full text every turn" and "skim descriptions, then open only what's relevant" is the difference between a system prompt that fits and one that doesn't.

Notice this is the SAME discipline Chapter 6's index.md applies to wiki patterns, and Chapter 7's proposer applies to reading traces on demand instead of upfront. "Show a short summary first, load the full content only if it's actually relevant" isn't a one-off trick used once in this paper — it's the load-bearing pattern that makes the whole system tractable inside a finite context window, reused identically at three different layers.

Compare Example A to Example B above. Both are syntactically valid SKILL.md files — correct frontmatter, correct sections, nothing malformed. Only one of them would survive validation gating, and it isn't a coincidence which one. Example B's instructions are true in the sense that nothing in them is factually wrong, and useless in the sense that they don't tell an agent what to physically DO differently the next time it's about to repeat itself. That gap — between "true" and "actionable" — is exactly the distinction the real case study in Chapter 4 turns on, where a skill this abstract really was proposed, and really was rejected.

Notice that this table isn't judging Example B on tone, length, or effort — it's a checklist of exactly three things a Wiki Maintainer's pattern page and a well-formed skill both need: a specific trigger, a concrete example, and a named source. Chapter 6 hands you the identical checklist again, applied to index.md entries instead of skills, because the same "specific enough to act on" bar governs both layers.

If you're tempted to think Example B could pass with a slightly longer write-up, try it: add three more sentences of the same flavor — "be thorough," "check your assumptions," "don't rush." Length isn't the missing ingredient. None of those additions name a trigger, a concrete failing/successful pair, or a specific wiki pattern, so none of them would move Example B out of the "rejected" column no matter how many you add.

Why "Just a Directory" Is the Whole Point

It's tempting to read "a skill is a directory with two markdown files" as an implementation detail — a storage format the paper happened to pick. It's worth pausing on it, because that choice quietly unlocks three things that a weights-based representation of the same knowledge could never give you.

First, it's diffable. A patch operation like the one Chapter 6 formalizes — append, replace, insert_after — is a well-defined, human-readable edit on text. There's no equivalent "patch operation" you could apply cleanly to a set of model weights that means "add this one procedural rule and nothing else."

Second, it's portable across models. A skill directory doesn't reference any particular model's parameters, tokenizer, or architecture — it's plain text that gets injected into a system prompt. That's the entire mechanism Chapter 10's cross-model transfer results depend on: a skill evolved by one model family can be handed, unmodified, to a completely different model family, because nothing about the artifact is model-specific to begin with.

Third, it's disposable without loss. Chapter 1 already made this point about the wiki/skills split in the abstract; here it is in its most literal form. Deleting a skill directory and starting over costs exactly the disk space of that directory — nothing about the model's weights, or the wiki's accumulated patterns, is at risk.

Stack those three properties up and a pattern emerges: every one of them is a property text has and weights don't. That's not a coincidence — it's the entire reason this paper's authors, and the broader "agent skills" literature they're building on, chose a filesystem representation over a learned one in the first place.

Three Prior Methods, One at a Time

The table below compresses each method into four cells. Before you read it, it's worth meeting each one on its own terms — because the differences aren't cosmetic, they're structurally different answers to the same design question.

Read them in the order they're presented — roughly increasing sophistication of the analysis step, before this chapter's table flattens all three into a single row for direct comparison.

Trace2Skill centers on parallel analysis and hierarchical merging. Every training rollout gets its own dedicated success analyst or error analyst, run in parallel — extracting effective strategies from tasks that passed, and root-causing the ones that failed. Those per-trajectory analyses then get consolidated through a recursive, tree-shaped reduction: pairs of analyses merge into one, those merge into fewer, and so on, until a single patch set survives at the root. That patch gets applied to the skill document and gated on validation, same as everyone else. What Trace2Skill never keeps is the tree itself — once the merge finishes and the patch lands, the intermediate analyses that fed it are gone.

EvoSkill reframes evolution as a search problem. Rather than analyzing every trajectory, it schedules training tasks round-robin by category, rolls the agent out, and feeds only the failures to its proposer — passing tasks aren't inspected at all. The proposer's context is a flat history of what's been tried before and whether it worked. Candidate skill files get materialized, scored on validation, and slotted into a bounded frontier of the best performers found so far. "Bounded" is the operative word — Chapter 1 already showed you what that costs.

SkillOpt runs the most elaborate pipeline of the three: a six-stage cycle its authors call ReflACT — Rollout, Reflect, Aggregate, Select, Update, Evaluate. Every epoch, the agent rolls out on training tasks, and the system reflects on the full execution trace, successes and failures both, generating candidate patches. Those patches get hierarchically aggregated and selected down to one update, applied to a single monolithic skill document. It's the closest of the three to WikiSkill's own thoroughness — and still, everything it carries forward lives inside that one pipeline's own epoch-to-epoch state, not as a file a completely different process could open and read.

Line them up by how much of each iteration's information they actually inspect, and a spectrum emerges: EvoSkill looks at failures only; Trace2Skill and SkillOpt both look at successes and failures; WikiSkill's Wiki Maintainer looks at successes and failures too, stratified deliberately (Chapter 6) rather than examining everything unfiltered. More inspection isn't automatically better — SkillOpt inspects the most raw material of any method here and still doesn't win Chapter 9's benchmarks. What matters is what happens to the inspection's OUTPUT afterward, which is this chapter's entire point.

Trace an identical failure — say, an agent that keeps mis-parsing a spreadsheet's header row — through all four methods, and the difference in what survives becomes concrete rather than abstract:

• Trace2Skill would fold this into whatever merge-tree pass happens to be running, producing a patch. Ask again five iterations later "why was this fixed," and the merge tree that explains it no longer exists to answer.
• EvoSkill would log the failure into its flat proposal history, but only if a proposal was actually generated from it — and only until the frontier gets crowded enough that this entry falls off.
• SkillOpt would fold it into that epoch's ReflACT cycle — useful in the moment, but tied to the pipeline's own internal epoch bookkeeping rather than a file anyone else can browse independently.
• WikiSkill would get a dedicated pattern page under wiki/patterns/, indexed by a one-line description in index.md, readable by name, forever, by anything that opens the file.

Three Prior Answers to "What Persists?"

Now the comparison this whole lesson keeps pointing back to. All four rows below share the roll out → analyze → propose → gate loop; the differences are entirely in the three middle rows.

This last row is a teaser, not a derivation — Chapter 11 works out exactly where each of these formulas comes from.

Read that middle row again: three different systems, three different answers to "what does the proposer see," and none of them describe a separate, standing, independently-queryable knowledge base. EvoSkill's "flat feedback history" is the closest cousin, and it's still bounded to whatever fits in one frontier data structure, not organized into named, indexed pattern pages a proposer can browse by relevance.

One deliberate omission from this comparison, worth naming before it looks like an oversight: general-purpose automatic prompt optimizers like GEPA never appear as a baseline here. The paper is explicit about why — prior work already shows that specialized skill-evolution pipelines consistently outperform general prompt optimization on this kind of task, so a fair fight is among the specialists. WikiSkill is being measured against the best of its own category, not against a strictly weaker class of tool included to make the numbers look better.

It's worth being precise about what "specialized" buys here. A general prompt optimizer treats the ENTIRE system prompt as one thing to tune. A skill-evolution framework, WikiSkill included, treats procedural knowledge as a first-class, separately-addressable unit — you can create one skill, patch another, and leave a third completely untouched, all in the same iteration. That granularity is a structural advantage independent of any single paper's specific algorithm.

You now have the two pieces of vocabulary this chapter promised: what a skill directory physically contains, and what each of the three rivals actually persists between iterations. Chapter 3 is where the rest of this lesson's notation gets grounded — concretely, in the same countable, real-scale numbers Chapter 0 already used, before a single unfamiliar symbol appears on the page.

Keep the four-column table above bookmarked, mentally or otherwise — Chapters 6, 7, and 8 each open by restating exactly one of its four "WikiSkill" cells in full, and having the contrast already loaded makes each of those chapters land faster.

One quiz, and Chapter 3 starts counting — literally, with a benchmark's real train/val/test split, before a single new symbol shows up.

If the anatomy widget's Example B still looks passable to you, reread the checklist table one more time before answering — the quiz below is testing the exact same specific-versus-vague distinction, just applied to a different prior method's memory instead of a skill file.

Chapter 3: The Setup, Formally

Before a single Greek letter shows up, here are real numbers you could count on your fingers. ALFWorld — the household-task benchmark behind Chapter 0's clock-and-desk loop — is evaluated on 39 training tasks, 18 validation tasks, and 134 test tasks, using an "admissible actions" tool that just tells the agent which moves are legal from wherever it's standing. SpreadsheetBench, the largest of the five benchmarks in this paper, runs 80 training / 40 validation / 280 test tasks with a bash shell for code execution. None of that is a simplification for this lesson — it's the paper's real Table 6.

Every formula in this chapter is describing something at exactly that scale. Nothing here needs a bigger mental model than "a countable pile of tasks, split three ways."

Four short equations, one small table, one hand-worked example — that's the entire chapter. If Chapters 0 and 1 gave you the feeling and the architecture, this one gives you just enough vocabulary that Chapter 4's file explorer stops looking like a diagram and starts looking like an actual computation you could trace by hand.

Read this chapter once, straight through, before touching the widget below — the notation earns its keep only once you've seen where every symbol actually points.

A Dataset, Split Three Ways

Start with the data itself. Each benchmark is a set of task instances, each one paired with a correct answer:

x_i is one task instance (a math problem, a spreadsheet transformation, a household goal); y_i is its ground-truth answer. D splits into three disjoint pieces — D_train (rollouts the Inference Agent practices on), D_val (what gates every skill-update decision), and D_test (touched only once, at the very end, to report a final number). No task instance appears in more than one split.

"Disjoint" is doing real work in that last sentence. If even one task leaked between D_train and D_val, a skill could get credit on the validation split for having effectively memorized that one task during training rollouts — and the whole point of a held-out validation split, catching skills that don't generalize, would quietly stop working for exactly the tasks that leaked.

N, the total task count, differs benchmark to benchmark — 177 for LiveMath, 400 for SpreadsheetBench — but the SHAPE of D never does. Every one of the five benchmarks in this paper is exactly this one object: a set of (task, answer) pairs, split three disjoint ways. Nothing about ALFWorld's household simulator or SpreadsheetBench's code execution changes that shape; the differences live entirely inside what x_i and τ_i look like, not in D itself.

One Task, One Trajectory, One Score

Now zoom into a single task instance. The agent doesn't just output an answer — it interacts with tools, step by step, and only the final step actually commits to an answer. That interaction has a name:

Different benchmarks give τ_i wildly different shapes without changing the notation at all. LiveMath's trajectories are essentially one turn — read the question, emit an answer. ALFWorld's can run for dozens of turns as the agent navigates a simulated house. The tuple (o₁, a₁, …, o_T, a_T) doesn't care how big T is; it's the same object whether T is 1 or 40.

π is the agent, conditioned on task x_i and whichever skill set S is currently active (Chapter 5 is where S starts actually changing). τ_i is the resulting trajectory — a random variable, because the same agent given the same task and the same skills can still act slightly differently run to run. Concretely, a trajectory is a sequence of observation/action pairs, τ = (o₁, a₁, o₂, a₂, …, o_T, a_T), where the final action a_T is the one that actually emits a predicted answer ý_i.

Step through a small, illustrative trajectory below — deliberately simplified, and not copied from any real execution log — to see exactly what "observations in, actions out, one scalar score at the end" looks like as concrete text rather than as a tuple of Greek letters.

Reset the widget and step through it a second time, counting out loud: o₁, a₁, o₂, a₂, o₃, a₃. Six items, three pairs, T = 3. That's the entire content of the subscript notation in the math block above — a counting exercise, not an abstraction.

T itself varies task to task and benchmark to benchmark — it isn't a fixed constant anywhere in this paper's formalism, just whatever length a given trajectory happens to run.

The Back button matters as much as Next here. Step forward to the final action, then step back to o₂ and forward again — nothing about a₂'s search result changes on replay, because the trajectory you're stepping through is a fixed, already-completed record. That's a small, hands-on preview of what "immutable" (Chapter 1's raw layer) is going to mean starting in Chapter 4: a trajectory, once it happened, doesn't get to happen differently on a second look.

Once a_T emits ý_i, a domain-specific scoring function checks it against the ground truth:

For most of this paper's benchmarks that score is binary — exactly right or exactly wrong — but the formula leaves room for partial credit where a benchmark defines it. This one scalar, per task, is the entire signal that everything downstream (Chapters 6 through 8) is built to interpret and act on.

Reread the widget's final score card once more with this formula in hand: "✓ f(ý, y) = 1 — ý = 'Hugh Everett III, 1957' matches ground truth" is not decoration, it is the formula, filled in with this one task's actual values, spelled out in full so you can see every symbol land somewhere concrete.

Notice, too, what f never sees: it never sees the trajectory τ_i itself, only the final answer ý_i compared against the ground truth y_i. A trajectory that reasoned brilliantly through a wrong intermediate step and still landed on the right final answer scores exactly the same 1 as one that reasoned cleanly the whole way through. That's precisely why the Wiki Maintainer's job (Chapter 6) can't just read the score — f(ý,y) tells you WHETHER a task passed, and nothing at all about WHY.

Roll an agent out across every task in a split, and you get a whole set of trajectories, T_split = { τ_i ∼ π(x_i; S) }_{(xi, yi) ∈ Dsplit}. Average the per-task scores across that set and you get one number for the whole split:

This is the single number that gates every decision starting in Chapter 8 — not a per-task pass/fail, but the average across an entire validation split.

Notice the subscript alone tells you which split you're looking at: R(T_train) is a number nobody in this paper's pipeline actually gates on — it's diagnostic, if you computed it, but not load-bearing. R(T_val,k) is the number Chapter 8's entire gating mechanism revolves around. R(T_test) is the number that ends up in Chapter 9's results tables, computed exactly once, after evolution is finished.

Worked by Hand: SealQA's Ten Validation Tasks

SealQA's validation split has exactly 10 tasks — small enough to score by hand. Suppose a candidate skill set gets these ten binary outcomes on D_val: pass, pass, fail, pass, pass, fail, fail, pass, pass, pass.

Seven passes, three failures, ten tasks: sum the scores, divide by the count. That's the entire computation — no weighting, no discounting, just an arithmetic mean over the split.

Now flip a single task — task 3 goes from fail to pass. The new sum is 8, and R(T_val) jumps from 0.70 to 0.80: a full 10-percentage-point swing from one task's outcome. Compare that to SpreadsheetBench's 40-task validation split, where flipping one task moves R(T_val) by only 2.5 points. This is exactly why Appendix C's paired bootstrap significance testing exists, and exactly why the paper reports every number as an average across three independent evolution runs rather than a single pass — a validation split this small can make an individual gating decision noisy in a way a bigger split wouldn't.

Do the same arithmetic once more, this time on OfficeQA's 24-task validation split, just to feel the scale difference in your hands rather than take it on faith: one flipped task there moves R(T_val) by 1/24 ≈ 4.2 percentage points — smaller than SealQA's 10 points, still nowhere near SpreadsheetBench's 2.5.

The State This Whole Lesson Tracks

Everything from Chapter 4 onward is really tracking one evolving pair of variables:

S_k is the active procedural skill set after iteration k — the Skills layer from Chapter 1. W_k is the persistent knowledge base after iteration k — the Wiki layer. Both start empty, at iteration 0, before any evolution has happened at all.

"Starting empty" is worth sitting with for a moment, because it means WikiSkill assumes NOTHING about the task domain going in — no seed skills, no bootstrapped wiki, no warm start of any kind. Every pattern in wiki/patterns/ and every file in skills/ that you'll see from Chapter 4 onward was discovered entirely from scratch, by the system itself, watching its own agent fail and succeed.

One more piece of notation, since it'll recur constantly starting in Chapter 5: the subscript k always means "after iteration k finishes," never "during." R_best, introduced properly in Chapter 8, follows the same convention — it's a running high-water mark, updated only at the moment an iteration's gating decision resolves, never partway through one.

Five entries, and that's the entire notation budget for the rest of this lesson — nothing new gets introduced without being motivated first, per the house rule this lesson has followed since Chapter 0.

The Five Benchmarks, By the Numbers

Notation done, arithmetic done — one more grounding pass before Chapter 4 puts all of it to work on a real case study.

Here is the paper's actual Table 6 — every benchmark this lesson's numbers (Chapters 9 through 11) will draw from, with real scale, real tools, and nothing rounded off for this lesson's convenience.

Look at the spread. SpreadsheetBench's 80/40/280 split is nearly five times ALFWorld's 39/18/134 — the paper isn't testing one narrow task shape at one fixed scale, it's testing whether the same four-component loop holds up across benchmarks that don't even agree on how big "enough data" is.

Notice, too, that most splits stay roughly in a 1:0.5:3.5 train:val:test ratio even though the absolute sizes vary five-fold — SealQA is the one clear outlier, with a proportionally larger test split (16:10:85). That rough consistency isn't a coincidence — it's inherited directly from the prior baseline papers this study is fairly comparing against, per the paper's own note that "all task splits... are strictly matched with prior work."

The tools column tells its own story. LiveMath gets nothing at all — the model reasons directly from the problem statement to a multiple-choice answer, no tool calls involved, which is exactly why it's the one benchmark in this paper where Chapter 3's trajectory tuple degenerates to a single (o₁, a₁) pair. Every other benchmark is genuinely multi-step: SealQA needs a live web search tool because the questions are scholarly and time-sensitive; SpreadsheetBench needs a real bash shell because the task is executing code, not describing it; OfficeQA's glob/grep/read trio is there because the task is finding one needle in a haystack of Treasury bulletins; ALFWorld's admissible-actions tool exists because the environment itself defines what's even a legal move at each step.

One more discipline worth naming before it becomes relevant: D_test is touched exactly once, at the very end, after all evolution is finished. Every gating decision in Chapters 5 through 8 runs against D_val only. If the system peeked at test-set performance mid-evolution and let that influence which skills survive, the final reported number would be measuring how well the system learned to pass the test — not how well it generalizes. That's not a paper-specific quirk; it's the same train/validation/test discipline any properly-run machine learning evaluation depends on, applied here to skill evolution instead of parameter updates.

Add every column in Table 6 and the whole study spans 1,125 task instances across five benchmarks and, as Chapter 9 will show, five different models — every one of them run through the exact four-step loop this lesson builds up starting next chapter. That's the scale claim underneath "consistent across benchmarks and model families": not a handful of cherry-picked wins, a genuinely wide sweep.

Five models times five benchmarks times three independent evolution runs each (Appendix C's variance-reduction discipline, already mentioned above) means every single number in Chapter 9's results table represents dozens of full evolution runs compressed into one cell. Keep that in mind when a comparison table later in this lesson looks like "just a number" — it never is.

Chapter 4 is next, and it's the reward for sitting through this chapter's notation: the exact same D, τ, f, R, and (S,W) you just met, now populated with the paper's real ALFWorld case study instead of toy numbers.

No new symbols appear in Chapter 4 at all — only real file names, real validation scores, and the five symbols from this chapter's cheat-sheet table, doing real work for the first time.

Chapter 4: The Workspace

Everything from here forward is either reading from, or writing to, one of exactly three directories. This chapter is the map — and it uses the paper's own worked example (Figure 3, ALFWorld, the Qwen-3.6-27B agent from Chapter 0) as the tour's real content, not a hypothetical one.

Chapter 0 opened with an agent stuck picking a clock up, examining it, and putting it back down. Here is what the workspace behind that exact case study actually looked like, file by file, iteration by iteration — simplified for presentation the same way the paper's own Figure 3 is, but not invented for this lesson.

Every file name, every validation score, every accept/reject outcome below is scripted directly from the paper's own case study text and Figure 3's simplified excerpts — nothing here is a made-up illustration standing in for the real mechanism. If you cross-reference this widget against the paper's own Figure 3, the names and numbers will match.

Keep this legend visible as you scrub the widget below — the three border colors on the badges above are the exact same three colors bordering the three panels, so you never have to guess which panel is which layer.

The labels under each arrow above name which paper component performs that step — the Wiki Maintainer consolidating raw traces into patterns, the Skill Proposer turning the updated wiki into one candidate change. A third, dotted arrow closes the loop conceptually at the bottom; showing it actually animate, end to end, is Chapter 5's entire job.

If you missed it the first time through, the stepper's five stops are deliberately small — jump straight to iteration 1 from iteration 0 and watch both panels change in the same click. There's no faster way to see the asymmetry than triggering it directly, rather than reading about it.

One more thing worth trying on your own, after you've played through all five iterations once: drag the stepper backward, past iteration 2, back to iteration 0. Watch what does and doesn't shrink. The Wiki and Skills panels shrink honestly — they show exactly what existed at that point in time. The Raw panel does not. Once a trace has appeared in this widget, it stays visible in the Raw panel no matter how far back you scrub — a small, deliberately literal dramatization of "immutable" that goes further than the other two panels' otherwise-accurate time travel. (The Reset button, unlike the stepper, does clear it — that's a genuine restart, not a rewind.)

Reading a Metadata Strip

Click into skills/break-repetition-loop/SKILL.md at iteration 4 specifically and look at the drawer's metadata line: active since: iter 1 · refined: iter 4. That one short line is answering two different questions at once, and it's worth separating them explicitly.

"Active since: iter 1" answers a gating question — when did this skill last pass Chapter 8's strict inequality? "Refined: iter 4" answers a completely different question — when was this file's TEXT last patched, whether or not that patch happened to coincide with a fresh gating pass? A skill can be refined without a new file appearing (it's the same directory, edited in place), and the metadata strip is precisely how you'd know that without re-deriving it from the file's contents.

Compare that to skills/goal-directed-action/SKILL.md's metadata: created: iter 0 · status: rejected — rolled back at iter 1. No "active since," because it never had one. This is the same discipline Chapter 8's ledger enforces in wiki/skill-impact.md — every file's status line tells you not just what's true now, but exactly which iteration made it true.

Count the Files Yourself

Step the widget all the way to iteration 4 and count what's actually sitting in each panel — the live count under the panels tracks this as you scrub, but it's worth doing once by eye too. raw/traces/ holds five execution logs, one per training rollout that fed an iteration. wiki/ holds three permanent files (index.md, logs.md, skill-impact.md) plus two pattern pages under patterns/. skills/ holds exactly two files — SKILL.md and PURPOSE.md — for the one skill, break-repetition-loop, that survived to iteration 4.

That last number is worth sitting with. Five training rollouts, two wiki patterns, one surviving skill built from two files. The funnel only gets narrower moving right — raw is the widest layer because it's cheap to write and never has to justify its own existence; skills is the narrowest because everything in it had to survive an actual validation gate.

What Actually Flows Between the Layers

In prose, once, before you forget the shape you just clicked through: raw traces get sampled down to a manageable subset, which the Wiki Maintainer consolidates into pattern edits, an index update, and a log entry — all inside wiki/. The wiki's index, its skill-impact ledger, and a training-outcome summary then go to the Skill Proposer, which produces exactly one atomic proposal, applied to skills/ as a candidate. Gating (fully dramatized in Chapter 5) either lets that candidate persist or makes it ghost out — but either way, wiki/skill-impact.md permanently records which one happened. Every arrow in that sentence is a real read or a real write; nothing in this architecture is implicit.

It's worth naming what this chapter deliberately does NOT show you yet: how the Wiki Maintainer decides what counts as a pattern worth writing down (Chapter 6), how the Skill Proposer decides between creating a new skill versus patching an existing one (Chapter 7), and the exact arithmetic behind the gate itself (Chapter 8). This chapter's job was narrower and, in a sense, more important than any one of those — it's the map that makes the next three chapters' components locatable at all. Read Chapter 6 without this chapter's tour, and "the Wiki Maintainer writes to wiki/patterns/" is just a sentence. Read it after, and you already know exactly what that directory looked like a moment before and a moment after.

Five Iterations, One Row Each

Here's the whole case study compressed into a single reference table — useful for a quick recap without re-scrubbing the widget, and worth bookmarking mentally, since Chapter 5 walks through these same five rows again, live.

Every quiz in this lesson is answerable straight from the chapter it closes — this one included. Reread the warm callout above once more before answering if you want to; the answer really is sitting in that one sentence.

Chapter 5 is next, and it changes exactly one thing about everything you just clicked through: instead of you dragging a stepper across a finished five-iteration history, the loop runs live, one of its four internal steps at a time, and you watch each iteration's outcome get decided in front of you instead of inspecting it after the fact.

Nothing you learned here gets discarded — it gets reused. The same three-panel color coding, the same five iterations, the same file names carry straight through.

Keep this chapter's file names close by while you read the next one — take-examine-move-loop.md, goal-directed-action, break-repetition-loop, multi-operation-loop.md — because Chapter 5's dashboard reuses every one of them without re-introducing what they are.

Chapter 5: The Loop, Running

Chapter 4 showed you the filesystem after five iterations had already happened. This chapter is where those five iterations actually happen, one step at a time, in front of you — the same case study, but now you're driving the loop instead of touring its aftermath.

Same file names, same rejection at val=0.72, same acceptance at val=0.78 — nothing about the underlying story changes between the two chapters. What changes is the camera angle: static filesystem snapshot, then a live control panel.

The loop has exactly four steps, and they run in the same order, every iteration, forever (or until the early-stop rule fires, which you'll get to see near the end of this chapter). Press Step to run one stage at a time, or Run to play out a whole iteration's four stages back to back.

Before you touch anything: predict the shape of the R_best sparkline across the first five iterations. You already know two of the five real outcomes from Chapter 4 — iteration 0 rejected, iteration 1 accepted — and Chapter 4's file explorer told you iterations 2 and 3 produced no skill proposal at all. Sketch the line in your head (flat, jump, flat, flat, jump again) before you press Step for the first time, then check yourself against what actually renders.

Nothing about this widget requires you to have read Chapters 6 through 8 first. Every rule it enforces — the trace-read minimum, the strict gating inequality, the unconditional ledger append — is stated here in full, exactly as the paper states it. Those later chapters go deeper into WHY each rule is shaped the way it is; this one is where you first feel each rule actually bind.

Watch the two counters on the dashboard that never share a color with the accept/reject verdict: Wiki patterns and Ledger entries are always teal, never green or red, because they don't participate in the gating decision at all. Run a rejected iteration and a accepted iteration back to back — the R_best bar and the Active skills count only move on acceptance, but the two teal counters climb (or hold, never fall) either way. That's the asymmetry from Chapter 4, now live instead of historical.

Step through the Skill Proposer stage carefully at least once. The finish() action stays disabled with a live counter until the trace-read minimum clears — this is not a UI flourish, it's the paper's literal Appendix E.3 rule: "You MUST read at least 4 execution traces before proposing a skill change." An agent that tried to propose after reading one trace would, in the real system, simply not be able to call finish() yet.

Try the history strip once you've played through a few iterations. Each dot is read-only — clicking one shows you that iteration's Step 4 outcome exactly as it happened, without re-simulating anything or disturbing where you actually are in the live loop. That distinction — reviewing the past versus rewinding the present — only matters because R_best, the active skill count, and the wiki counters are all live, mutable state, and jumping around in history has to not corrupt any of it.

Green dots mean accepted, red mean rejected, and a dim grey dot means the Skill Proposer looked and concluded no_action was the right call — a real, named outcome in the paper's own proposal schema, not a UI placeholder for "nothing happened."

Run the loop all the way to iteration 5 and beyond, past where the real case study's data ends, and watch the "(illustrative continuation)" tag appear on the stage panel's header. This lesson never lets a synthetic number pass itself off as one of the paper's real reported figures — anywhere a number isn't directly from the source, it says so, right where you're looking at it.

The Same Loop, as Pseudocode

Now that you've driven it by hand, here's what you just drove, written down. This isn't the paper's line-numbered LaTeX — it's a clean equivalent, so the formalism lands after the intuition, not before it.

Require: D_train, D_val, metric R, max iterations K
S_0 ← empty skill set,  W_0 ← empty wiki
R_best ← R(baseline rollout of S_0 on D_val)

for k = 1 .. K:
    if R_best == 1.0:  break                      # early stop

    T_train,k ← roll out agent on D_train using S_(k-1)      # Step 1
    T_sample,k ← stratified sample of T_train,k              # Step 2 input
    W'_k ← WikiMaintainer(W_(k-1), T_sample,k)                # Step 2

    P_k ← SkillProposer(W'_k, S_(k-1), T_train,k)             # Step 3
    S'_k ← Apply(S_(k-1), P_k)

    T_val,k ← roll out agent using S'_k on D_val              # Step 4
    if R(T_val,k) > R_best:
        S_k ← S'_k;  R_best ← R(T_val,k);  accepted ← true
    else:
        S_k ← S_(k-1);  accepted ← false                 # rollback SKILLS only

    W_k ← Update(W'_k, P_k, R(T_val,k), accepted)             # wiki NEVER rolls back

return S_K, W_K

Match it against what you drove: Steps 1–4 in the sim are literally the four indented blocks under the for loop above, in order. The early-stop check sits BEFORE the roll-out, not after — which is exactly why, in the sim, hitting R_best = 1.0 replaces the next Step button with a banner instead of quietly finishing the iteration first. And the very last line, W_k ← Update(...), runs unconditionally, outside the if/else — the one line in this entire algorithm that both branches of the accept/reject decision pass through on their way out.

One rule this pseudocode states plainly that's easy to lose in the visuals: K, the maximum iteration count, is a hyperparameter the researchers set, not something the algorithm discovers. Early stopping can end a run before K iterations if R_best hits 1.0; nothing in Algorithm 1 lets a run continue PAST K, no matter how promising the trend looks on iteration K−1's sparkline.

Hover any of the four boxes in the loop diagram near the top of the widget for its exact equation number — the same discipline Chapter 4's arrow tooltips used, just applied to all four stages this time instead of two data-flow edges.

You've now driven every piece of the architecture Chapters 0 through 3 only described in prose and Chapter 4 only let you inspect after the fact. Chapters 6 through 8 zoom back in on each of the four steps you just ran — the Wiki Maintainer's root-cause analysis, the Skill Proposer's investigation discipline, and the exact arithmetic behind the gate — now that you already know, from having run it yourself, what each one is FOR.

Chapter 6: The Wiki Maintainer

Chapters 4 and 5 showed you the Wiki Layer from the outside — files that never vanish, a counter that only ever climbs. This chapter opens the one component whose entire job is writing into that layer: the Wiki Maintainer.

Two Write-Ups Of The Same Failure

Picture a SpreadsheetBench task that just failed. Here are two write-ups of the exact same trace.

Only one of these is useful to the next iteration. The shallow write-up names a symptom — something went wrong, somewhere, involving a tool. The root-cause write-up names a mechanism specific enough that a future skill could actually prevent it: re-fetch state before every write, never trust an offset computed before a prior write already happened. Guess which one the Wiki Maintainer is built to produce.

The Wiki Maintainer is the agent M_WM in the paper’s own notation (§3.2.2). At iteration k it receives the full wiki context W_k-1 alongside a sampled subset of this iteration’s rollout traces, and it produces the intermediate wiki state:

Eq. 2 — the entire wiki context in, a handful of this iteration’s traces in, an updated wiki context out. Notice what is not an input: the active skill set. The Maintainer never reads skills/ directly — its job is understanding what happened, not what to do about it.

Why Only A Sample

A training split can run to 80 tasks (SpreadsheetBench) or more. Feeding every single trace from every single task into one context window, every iteration, would blow past any model’s usable context long before iteration 5. So the system samples first (Appendix C).

The budget is deliberately asymmetric: up to 5 failing traces (to perform root-cause analysis of errors) and up to 3 passing traces (to identify effective strategies and prevent regressions in working behaviors) — Appendix C, verbatim. Each individual execution log is capped at 15,000 characters before it ever reaches the prompt. Fail-heavy, not fail-only: three working examples ride along specifically so the Maintainer doesn’t only ever see what’s broken.

Patch, Don’t Rewrite

Once the Maintainer has decided what a trace means, it doesn’t rewrite the relevant pattern page from scratch. Updates are applied using incremental, patch-based editing — appending, replacing, or inserting text spans (§3.2.2). The Maintainer’s own system prompt (Appendix E.2) spells out exactly three operations:

target must be an EXACT substring of the existing content — the Maintainer can’t patch a line it can’t quote verbatim. Every edit is meant to be minimal: change only what needs to change.

Try the three operations yourself against a real (synthetic, paper-consistent) pattern page.

Watch what stays untouched in every case: the file’s heading, its evidence list, everything except the exact substring the operation targets. That’s the whole point of patch-based editing — a pattern page accumulates evidence across five, six, ten iterations without ever losing its earlier history to a careless full rewrite.

index.md Is The Gatekeeper

A pattern page can be perfectly written and still never get read. The Skill Proposer (next chapter) doesn’t open every file in wiki/patterns/ on every iteration — it starts from wiki/index.md, a one-line-per-pattern catalog, and decides from THAT one line whether a page is worth opening at all. The Maintainer’s own system prompt is explicit about the bar (§E.2): each entry must state PROBLEM + ROOT CAUSE + FIX in one or two sentences, specific enough that relevance can be judged without opening the page.

What The Maintainer Actually Returns

Mechanically, the Maintainer doesn’t hand back prose — it hands back a JSON object with a fixed shape (§E.2), which the outer-loop harness then applies to the filesystem on its behalf:

{
  "create_patterns": [ {"name": "...", "content": "..."} ],
  "update_patterns": [ {"name": "...", "edits": [ {"op": "...", ...} ]} ],
  "update_index":    "<full updated index.md content>",
  "append_log":      "brief summary of this iteration's findings"
}

update_index and append_log are REQUIRED on every single call — even an iteration with zero new patterns still appends a one-line log entry and re-confirms the complete index. Nothing about the wiki’s bookkeeping is optional.

There’s no hard cap on how many patterns get created or updated per iteration — the Maintainer decides based on the traces and the current wiki state. What IS fixed is the discipline: root-cause, not symptom; patch, not rewrite; one clear sentence in the index, or the work downstream never happens.

What The Maintainer Is Actually Told To Look For

The Maintainer’s own system prompt (§E.2) doesn’t just say “find root causes” and leave it there — it gives a concrete four-step checklist for “Deep Trace Analysis” that has to run before a single pattern page gets touched:

Step 2 is easy to skip in a rush and is exactly why the sampling budget from earlier in this chapter always sets aside up to 3 passing traces alongside the failures — there is no “compare successful vs. failed” without at least a few of each in the same context window. Two more rules from the same prompt shape every pattern page you’ve seen so far in this lesson: capture BOTH success and failure patterns (not just what went wrong), and keep pages to 10–30 lines — “not essays.” A pattern page is a diagnostic note for a future agent to skim, not a postmortem report for a human to read at leisure. The prompt also forbids duplicate patterns outright: “do NOT create duplicate patterns — update existing ones with new evidence,” which is the other half of why patch operations exist at all — without them, every new piece of evidence for an already-known failure mode would either get dropped or spawn a near-duplicate page, fragmenting the exact knowledge the wiki exists to consolidate.

How Much Does The Wiki Actually Grow?

The paper measured this directly (Table 4). Across models, Qwen-family agents write noticeably LONGER skills than Gemma or Gemini do, and the pattern-accumulation rate varies just as much:

Notice that MORE proposed almost never means MORE accepted — every model proposes roughly 2–5× as many skill creations and edits as end up passing Eq. 4’s gate. The wiki, though, keeps every one of those proposals’ outcomes in skill-impact.md regardless — the acceptance ratio you’re looking at IS the gate from Chapter 8, just counted up across a whole run instead of one iteration.

Chapter 7: The Skill Proposer

The Wiki Maintainer understands what happened. It never writes a single line into skills/. That job belongs to a separate agent entirely: the Skill Proposer, M_P in the paper’s notation.

Give the Proposer its literal toolset up front, because everything interesting about this chapter is a consequence of just two tools (§E.3):

That’s it. No batch-edit tool, no “propose five candidates and let validation sort it out” tool. The Proposer operates as a multi-turn ReAct agent (Yao et al., 2023) — it reasons, calls read_file, reasons about what it read, calls read_file again, and only once it has investigated enough does it call finish. Producing a proposal:

Eq. 3 — the updated wiki, the active skill set going in, and this iteration’s training rollouts. Note what it is NOT handed pre-selected: which pattern pages or which traces matter. That’s the Proposer’s own job to figure out, live, via read_file.

Given, Not Fetched

To avoid context-window exhaustion on long histories, the Proposer is NOT handed a fixed set of pre-sampled traces the way the Maintainer is. Instead, at the start of every iteration it’s initially given three things: the wiki index I(W′_k), the historical skill-impact tracker (skill-impact.md), and a concise pass/fail summary of every training task’s outcome. Everything else — specific pattern pages, specific raw traces — it has to go get.

Step through a scripted (but paper-accurate) run below. Watch the “traces read” counter, and watch finish() stay locked.

The gate is literal, not a metaphor for “be thorough.” The Proposer’s own system prompt states a hard rule (§E.3, Rule 4): “You MUST read at least 4 execution traces before proposing a skill change.” Everything the widget above enforces — the disabled button, the live counter — is that one sentence, made interactive.

Two Shapes, One Rule: Exactly One Skill

Once unlocked, finish() takes exactly one of two JSON shapes — and in either case, the proposal targets a SINGLE skill. Not a batch of five hedged candidates. One atomic bet.

create ships a full two-file bundle — skill_md with YAML frontmatter plus When-to-Apply plus instructions, and purpose_md mapping the skill back to whichever wiki patterns motivated it. patch ships only a short edit list against an existing skill, using the same three patch operations Chapter 6 already put in your hands — append, replace, insert_after. If a change would touch most of the file, the Proposer’s own instructions say to use create instead of forcing a patch that barely resembles a patch.

The Five Rules, In Full

The Proposer’s system prompt closes with a short numbered list. You’ve already met three of them in this chapter; here are all five together, because each one closes off a specific failure mode a less-constrained agent could fall into.

Rule 5 is worth sitting with, because it’s the one instruction that most directly explains why Figure 3’s case study has exactly ONE named skill across five iterations, not five. break-repetition-loop is created once, at Iteration 1, and PATCHED (not replaced, not superseded by a new skill) at Iteration 4 when new evidence arrives. The Proposer had a genuine choice at Iteration 4 — create a brand-new skill, or patch the existing one — and Rule 5 is the reason it chose the latter.

Where The Proposal Actually Lands

It’s worth seeing where skill_md’s content physically ends up, because the Proposer never talks to the Inference Agent directly — the connection is a single template placeholder. Every one of the five benchmark system prompts (§E.1) is built around the exact same slot:

You are a knowledgeable question-answering assistant with access to
web_search and read_file tools.
{skill_section}
## Task
You will receive a factual question. To answer it: ...

(SealQA’s Inference Agent prompt, §E.1 — the other four benchmarks follow the identical pattern: task framing, then {skill_section}, then task-specific rules.) {skill_section} is where the full content of every skill in S_k-1 gets injected, verbatim, before a single rollout begins. Trace the whole loop through this one placeholder: the Proposer writes skill_md → Gating accepts it into S_k → next iteration’s Inference Agent sees it appear, in full, exactly where {skill_section} sits in its own system prompt. Nothing about retrieval, nothing about relevance-matching at inference time — every active skill is simply THERE, every single rollout, the moment it’s accepted.

Applying A Proposal: Same Rule, Higher Stakes

Once finish() is called, a separate step — not the Proposer itself — turns the proposal into an actual candidate skill set: S′_k = Apply(S_k-1, P_k). For a create proposal, Apply just writes two new files, skills/<name>/SKILL.md and skills/<name>/PURPOSE.md, into the workspace. For a patch proposal, Apply walks the SAME three patch operations Chapter 6 put in your hands — append, replace, insert_after — against the target skill’s existing SKILL.md.

That reuse matters more here than it did for a wiki pattern page. If a replace operation’s target string doesn’t match the skill file’s CURRENT content exactly — because, say, an earlier iteration already edited the exact line the Proposer thinks it’s targeting, and the Proposer’s view of the file is stale — Apply has nothing valid to act on. This is precisely why Rule 1 (read the wiki first) and the ≥4-trace minimum exist together: the more thoroughly the Proposer has actually read the CURRENT state of things before proposing an edit, the less likely its patch targets text that has already moved.

Chapter 8: Gating & Rollback

You’ve already watched this rule fire a dozen times, once per iteration, across the last two chapters’ sims. Here it is as one line:

Eq. 4. Everything else in this chapter is what that one strict “greater than” sign is actually protecting.

The Console

Drag the candidate’s validation score below. Release it and watch what happens on both sides of the screen at once — the skill side, and the wiki side.

Notice the ledger grows on every single release, accept or reject. That’s not decoration — it’s Eq. 4’s other half, the part that never shows up in the formula itself: the wiki is never rolled back, regardless of the acceptance decision. Accumulated patterns and logs persist across all iterations, full stop.

What Gets Written, Every Time

Following each validation evaluation, the outer-loop harness — not the Maintainer, not the Proposer, a separate programmatic step — appends one entry to wiki/skill-impact.md (§3.2.4):

This is the exact schema Chapter 7’s Proposer opens FIRST, every iteration, before reading a single pattern page or trace — it’s how “don’t repeat what already failed” becomes an enforceable fact rather than a hope.

Reading The Rejection Backward

Prior to the evolution loop, R_best is initialized to the baseline validation score R(T_val,0) — the empty skill set S₀, evaluated once before anything evolves. If validation ever reaches the maximum (R_best = 1.0), the loop terminates early — there’s nowhere left to climb. Try the button above; that’s the exact banner condition.

Does Gating Just Accept Everything Early And Stop?

A reasonable worry about a strict, greedy gate: maybe it accepts a couple of easy wins in the first iteration or two and then plateaus, with nothing interesting happening later. The paper measured exactly this (Appendix Table 5), grouping every ACCEPTED update by when it happened — early (iterations 0–1), middle (2–4), or late (5–7).

By benchmark, the split runs 39–58% early / 26–48% mid / 10–28% late, with SealQA the standout for continued refinement — 33% of its accepted updates land in the MIDDLE stage and 28% in the LATE stage, more than any other benchmark. Across every model, the early stage never accounts for a MAJORITY of accepted updates on its own — the loop keeps finding genuinely new, gate-clearing improvements well past iteration 1, which is exactly what you’d expect if the wiki really is compounding useful evidence over time rather than front-loading everything it has to offer.

How Much To Trust One Score

Validation splits here are small — SealQA’s is only 10 tasks (Table 6, Chapter 3). A single R(T_val,k) computed from 10 tasks can swing noisily from one run to the next, which is exactly why every reported number in this lesson is already an AVERAGE across three independent full evolution runs, and why the paper backs its bold-face “best” markings with paired bootstrap significance testing (1,000 iterations, p < 0.05, Appendix C) rather than simply picking whichever number is numerically largest. When a top-ranked method isn’t statistically distinguishable from one or more competitors, the paper bolds ALL of them — that’s the “multiple bold = statistical tie” footnote you’ll see under Chapter 9’s full Table 1 reproduction. Eq. 4’s gate itself doesn’t get this luxury mid-run — it has to make an accept/reject call on ONE validation pass, right now, with whatever noise that one pass carries. The three-run averaging and bootstrap testing you’re about to rely on in Chapters 9 and 10 are how the paper reports trustworthy FINAL numbers; they are not available to the gate while the loop is actually running.

R_best Only Ever Climbs

One more invariant worth stating plainly, because it’s easy to lose in the middle of the loop: R_best itself follows the exact same one-way-ratchet shape as the wiki’s pattern count, just measured in a different unit. It starts at R(T_val,0), and Eq. 4 only ever updates it UPWARD, on acceptance — a rejected candidate leaves R_best exactly where it was. So across a whole run, R_best traces out a monotonically non-decreasing staircase: flat through every rejected iteration, stepping up on every accepted one, until it either hits 1.0 (early stop) or the run’s iteration budget K is exhausted. That shape is precisely what the sparkline-style dashboards in Chapters 4 and 5 were built to make visible — a candidate score can drop below the line at any point (that’s a rejection), but the line itself never does.

Mechanically, a rollback is a real filesystem operation, not a metaphor: the candidate’s SKILL.md/PURPOSE.md (for a create) or the just-applied edit (for a patch) is discarded, and skills/ reverts to exactly the file contents it had going into this iteration, S_k-1. Nothing partial survives on the skills side — either the whole candidate persists, or none of it does. Compare that with the SAME iteration’s effect on wiki/skill-impact.md: one new row, unconditionally, describing exactly what was tried and that it failed. One layer resets to a clean prior state; the other layer never resets at all — the same asymmetry from Chapter 4, now stated as a literal disk-level operation instead of a diagram.

Chapter 9: Results — Table 1, Explored

WikiSkill wins on average across every one of five models. That headline is true and it is also, by itself, boring — the interesting finding is buried inside a 30-cell table nobody reads cell by cell. This chapter puts the table under your hands instead of your eyes.

Five Models, Five Methods, Five Benchmarks

Click a model row to expand it. Toggle methods off to isolate just the gap you care about. Every number below is the paper’s own Table 1, verified cell by cell against the source text.

WikiSkill’s per-model, per-method improvement over the strongest COMPETING skill-evolution method is +3.3 (Qwen-4B), +5.1 (Qwen-9B), +10.0 (Qwen-27B), +5.8 (Gemma-31B), +12.0 (Gemini-3.5-Flash) points. Those five numbers, read left to right, are already the next section’s finding — they climb with model size.

Gains Grow With Scale

Restrict to just the Qwen family — same architecture lineage, three sizes — and the trend is unmistakable.

The same trend shows up sharpest on one benchmark: SpreadsheetBench. WikiSkill improves the three Qwen sizes by +6.5 (4B), +9.3 (9B), and +40.9 (27B) points respectively — the gap between “skill evolution helps a little” and “skill evolution helps enormously” is itself a function of how much capability the underlying model already has to execute what the skill describes.

Existing methods, meanwhile, are inconsistent in a way WikiSkill mostly isn’t: EvoSkill improves Qwen-9B on LiveMath from 28.2% to 58.1%, then DEGRADES Gemma-31B on the very same benchmark, 33.9% down to 29.8%. SkillOpt degrades Gemini-3.5-Flash on SealQA, 29.4% down to 28.2%. Stronger AND more reliable, not just stronger on average.

Full Reference Table

All 25 model–benchmark cells, bold marking the best score (multiple bold = statistical tie under the paper’s paired bootstrap test, p < 0.05).

Gemini-3.5-Flash’s ALFWorld row is identical across every method (85.9%) because Gemini reaches 100% validation performance before any skill ever evolves — the loop early-stops per Eq. 4 before it has a chance to change anything on this one benchmark.

Five Benchmarks, Not One Trend

Averages hide how unevenly the benefit lands. Hold the model fixed at Qwen-3.6-27B and look at what WikiSkill adds, benchmark by benchmark: +11.6 points on OfficeQA, +14.1 on SealQA, +24.8 on ALFWorld, +28.0 on LiveMath, and +40.9 on SpreadsheetBench. Same model, same method, a 3.5× spread between the smallest and largest gain depending purely on WHICH task it’s applied to.

Work the SpreadsheetBench number by hand from the table above: Qwen-3.6-27B scores 40.8% with no skill, 81.7% with WikiSkill. 81.7 − 40.8 = 40.9 — the exact figure the paper reports. Do the same for OfficeQA: 53.7 − 42.1 = 11.6. Every “+X points” claim in this lesson is a subtraction you can check yourself directly against the reference table, not a number to take on faith.

LiveMath is the most RELIABLE benefactor across the whole study — gains of 20.6 to 39.6 points across all five models, never negative once. ALFWorld runs 14.0 to 29.3 points across the four models WikiSkill actually evolves skills for (Gemini-3.5-Flash excluded — recall, it early-stops on ALFWorld before ever needing to). Cross-reference this against Chapter 6’s Table 4 breakdown by benchmark: SpreadsheetBench also produces the LONGEST skills on average (142.5 lines) and the MOST wiki patterns (9.8) of any benchmark, while LiveMath produces the shortest skills (84.6 lines) and fewest patterns (4.4). SpreadsheetBench isn’t just where WikiSkill helps most — it’s also where the wiki has the most to say, and LiveMath is the opposite on both counts. That’s not a coincidence you have to take on faith either — a benchmark with more distinct, codifiable failure modes to catalog is a benchmark where a compounding wiki has more material to work with, and (per this chapter’s numbers) more room to help.

Chapter 10: Transfer & Scale — Table 2, Explored

The single most surprising number in this paper: a skill evolved by Qwen-3.5-4B — the SMALLEST model tested — pushes Gemma-4-31B’s LiveMath score from 33.9% to 73.1%. The skill came from the weakest model in the study. Sit with that before reading the mechanism.

The Transfer Matrix

Pick a benchmark. Every row is an inference model; every column is where the skill it’s running actually came from. The highlighted cell in each row is that model’s OWN self-evolved skill — the diagonal.

Within the Qwen family, stronger models derive greater value from the SAME procedural knowledge: Qwen-3.6-27B’s own SpreadsheetBench skill improves Qwen-3.5-4B, Qwen-3.5-9B, and Qwen-3.6-27B itself by +18.4, +26.2, and +40.9 points respectively over their no-skill baselines — one skill, three different-sized recipients, a bigger benefit the bigger the recipient.

Negative Transfer, Anatomized

Transfer is not free. Same source benchmark, same target model, two different sources — one wrecks the target, one helps it enormously.

A Qwen-3.5-4B-sourced skill drops Gemini-3.5-Flash from a 50.5% no-skill baseline down to 18.1%. A Qwen-3.6-27B-sourced skill, on the exact same target model and benchmark, lifts it to 63.4%. The paper’s error analysis names two specific causes (§4.2.2):

1. Granularity mismatch. Qwen-3.5-4B’s skill encodes low-level workarounds — single-line Python commands, string-conversion rules — that help the smaller model avoid execution failures but CONSTRAIN a stronger model like Gemini-3.5-Flash from writing comprehensive end-to-end scripts.
2. Budget exhaustion. Fragmented diagnostic procedures introduce redundant tool calls that can burn through Gemini-3.5-Flash’s interaction budget before the task even completes.

Discovery vs. Execution

Sometimes a model writes down a genuinely good procedure and then can’t reliably follow its own advice.

Qwen-3.5-4B’s own OfficeQA skill HURTS its author — 30.2% down to 28.5% — while the SAME skill, transferred with no changes, HELPS Qwen-3.6-27B climb from 42.1% to 52.9%. Trajectory analysis suggests the smaller model gets distracted by long document contexts and fails to follow its own detailed multi-step search instructions, reverting to default reading behavior; the stronger model reliably executes the same structured navigation the weaker model merely discovered.

Read A Full Matrix Row By Hand

Switch the explorer above to SpreadsheetBench and find Qwen-3.5-9B’s row. Five numbers, one model, one benchmark: no-skill baseline 24.3%; running a Qwen-3.5-4B–sourced skill, 25.0% (barely moves it — a skill sourced from an even smaller model doesn’t have much to offer here); running its OWN self-evolved skill, 33.6%; running a Qwen-3.6-27B–sourced skill, 50.5%; running a Gemini-3.5-Flash–sourced skill, 48.8%. Read those five left to right and a clean ordering falls out: weaker-than-self source ≤ self < stronger-than-self sources. That ordering is exactly what “stronger models derive greater value from the same procedural knowledge” predicts — it isn’t only true when comparing across DIFFERENT recipients, it shows up inside a single recipient’s own row, sorted by where the skill came from.

ALFWorld makes the opposite point — near-universal benefit, not the SpreadsheetBench story of hit-or-miss. Look at Qwen-3.5-4B’s own row: no-skill 24.4%, its own skill 53.7%, a Qwen-3.6-27B–sourced skill 57.0%. Every single source helps this recipient substantially on this benchmark — there’s no negative-transfer story to tell here at all. Not every benchmark–model pair carries the SpreadsheetBench cautionary tale; some pairs just reliably work, which is worth remembering before generalizing “transfer is risky” into a universal rule instead of a benchmark-and-source-specific finding.

Put the two findings side by side and a practical takeaway falls out that the paper states directly: stronger source models do not necessarily produce better skills, and procedural knowledge developed by one model’s experience can transfer across model scales and families — but WHETHER it transfers well depends on the specific pairing of source and target, not on a simple “bigger source is always better” or “same-family transfer is always safe” rule. If you were choosing which model to use to evolve a skill for deployment on a DIFFERENT target model, this chapter’s explorer is the actual tool you’d want — pick the target benchmark, scan the row, and see for yourself which source scored highest for that specific target, rather than assuming the largest or most expensive source model automatically wins.

Chapter 11: Ablations, Case Study, Cost & Connections

What if you just… didn’t build the wiki? This chapter answers that literally — the paper ran exactly that experiment.

Strip It Out, Piece By Piece

Using Gemini-3.5-Flash, the paper independently varies wiki access for two components: the Inference Agent (during training rollouts) and the Skill Proposer (during skill development). When the Skill Proposer has no wiki access, the Wiki Maintainer is removed too — there’s no persistent knowledge left to accumulate, so nothing is left to maintain.

The headline swing: with the Inference Agent’s wiki access OFF, giving the Skill Proposer wiki access moves average performance from 48.7% to 63.7% — +15.0 points, the single largest lever measured anywhere in this paper, with especially large jumps on LiveMath (51.3% → 72.6%) and SpreadsheetBench (49.9% → 76.6%). Without persistent knowledge accumulated across iterations, the Skill Proposer struggles to resolve intricate, recurring failure modes.

The second finding runs the other way. With the Skill Proposer’s wiki access already ON, ALSO giving the Inference Agent wiki access during training rollouts REDUCES average performance — 63.7% down to 60.9%, with a substantial drop on LiveMath specifically (72.6% → 64.8%). The paper’s hypothesis: when the Inference Agent can read both skills and the wiki during rollouts, some task-solving knowledge gets pulled directly from the wiki rather than from the skills, which makes the resulting trajectories less informative for skill development downstream.

The Real Case Study, End To End

You’ve driven this exact example twice already — Chapter 4’s file explorer, Chapter 5’s running loop. Here it is one more time, straight from the paper’s own Figure 3 (Qwen-3.6-27B, ALFWorld), with nothing simplified.

Informed by the Iteration 0 rejection, the Proposer synthesizes break-repetition-loop at Iteration 1 — concrete, action-level, accepted. As new loop variants keep showing up across rollouts, the Wiki Maintainer accumulates fresh evidence, and by Iteration 4 the Proposer refines the SAME skill with a new rule, guided by pattern evidence the earlier iteration never had. Nothing here is hypothetical — it’s one real run, traced start to finish.

Why This Scales: O(1) vs. O(N)

WikiSkill’s Skill Proposer runs a fixed number of ReAct turns per iteration, independent of how many training tasks exist. Every prior method’s optimizer call count grows with the training set. Drag N_train below (illustrative — batch size B held fixed here so the asymptotic shapes stay visible; the paper’s own experiments run full-batch, B = N_train, which is a different, and also valid, way to read the same formulas).

In WikiSkill’s own experiments, batch size is set to the full training size (B = N_train) across every dataset, so N_train/B collapses to exactly 1 and the call count becomes 1 + T_ReAct — a small constant (roughly 10–20 ReAct turns), completely independent of how many training tasks exist. Trace2Skill can’t escape O(N_train) even with full-batch: it runs one dedicated analysis call per individual trajectory, no matter how batching is configured.

Work SpreadsheetBench’s own numbers by hand: N_train = 80 (Table 6, Chapter 3), B = N_train = 80, so N_train/B = 1. Take T_ReAct = 15, the middle of the paper’s reported 10–20 range. WikiSkill’s call count is (1 + 15) × 1 = 16 optimizer calls, total, for the whole iteration. Trace2Skill, same N_train = 80, same B = 80, branching factor c = 4: 80 + (80/80)·(1/(4−1)) + 1 = 80 + 0.33 + 1 ≈ 81 calls — roughly 5× more, purely because it runs one dedicated analysis call per individual training trajectory no matter how the batch is sliced. Scale N_train up to OfficeQA’s 172-task TEST split (not the 50-task train split — illustrating the shape at a larger N) and the gap only widens: WikiSkill stays fixed near 16, Trace2Skill climbs past 170.

Limitations, Straight From The Paper

What This Paper Deliberately Is Not

Two boundaries the paper draws around its own contribution, both worth stating explicitly because they explain choices you’ve seen throughout this lesson.

Not a skill-retrieval paper. As soon as a skill library grows past a handful of entries, SOMETHING has to decide which skills are relevant to a given task — that’s skill retrieval, and it’s an active, separate research area (§6). WikiSkill sidesteps it entirely: every experiment in this lesson full-injects the ENTIRE active skill set into the Inference Agent’s system prompt, every single rollout, via the {skill_section} placeholder from Chapter 7. That’s a deliberate scoping choice, not an oversight — it isolates skill QUALITY as the thing being measured, with no retrieval-failure noise mixed in. It’s also the first line item in the Limitations table above: this paper genuinely doesn’t tell you what happens once a library has hundreds of skills competing for a limited context window.

Not a general harness optimizer. A separate line of work optimizes the broader agent harness — prompts, context management, tools, memory, workflows, all at once. The paper is explicit that this is complementary, not competing: WikiSkill “focuses specifically on evolving reusable procedural skills while holding the broader agent harness fixed” (§6). Every Inference Agent system prompt you saw quoted in Chapter 7 stays fixed across the whole evolution run — only the {skill_section} slot inside it ever changes.

Cheat Sheet

Connections. The Skill Proposer’s multi-turn tool-use loop is a direct application of ReAct (Yao et al., 2023), cited by name in the paper. For the filesystem-native skill format itself — the SKILL.md / frontmatter / applicability-conditions convention Chapter 2 opened with — see Agent Skills. And for the general-purpose prompt optimizer this paper deliberately positions itself against — “we focus our comparison on dedicated skill-evolution frameworks rather than general automatic prompt optimizers,” the paper’s own words — see GEPA.