Code World Models

Chapter 1: What “World Model” Means Here

Chapter 0 used “world model” loosely. Make it precise before going any further, because the exact shape of this definition is what every later chapter builds on top of.

The general recipe: state, action, next state

In control theory and reinforcement learning, a world model (also called a transition model or dynamics model) is a learned function that takes the current state of a system and an action taken in it, and predicts the resulting next state:

This is a genuinely general recipe — it is the same object whether the “world” is a physical robot arm, a video game, or, as this session covers, a running computer program. What changes from domain to domain is only what counts as a state and what counts as an action. CWM’s central move is picking those two things for code, and then training on trajectories that make the mapping explicit rather than implicit.

CWM's two state/action pairs

The paper’s own words, from the abstract: CWM is “mid-train[ed] on a large amount of observation-action trajectories from Python interpreter and agentic Docker environments.” That sentence names two separate world-modeling loops, and it is worth keeping them distinct, because Chapters 3 and 4 build each one from a completely different data pipeline.

Both loops share the same shape: context (source code, or a repository) sets the scene, then a stream of (action, observation) pairs unfolds, one action producing one new observation, over and over. This is a different training signal from ordinary next-token prediction on a file, even though both are, mechanically, sequences of tokens being predicted left to right — the difference is what those tokens are labels for. In a normal training file the token stream is “what a human wrote next.” In an observation-action trajectory, alternating spans of the token stream are labeled “what a deterministic system did next, given this exact action” — a categorically different, and much more strongly grounded, supervision signal.

Why “mid-training” and not pre-training or post-training

CWM sits this world-modeling data at a specific point in the training pipeline: after an ordinary large-scale pre-training run, but before the reasoning-focused reinforcement learning post-training phase (Chapter 5 walks the full pipeline). The paper’s stated reasoning is direct: exposing the model to grounded observation-action data at scale, before RL begins, “should help improve coding performance by grounding our model’s predictions in the underlying dynamical systems and provide a superior starting point for RL.” In other words, teach the model what code does first, cheaply, at huge scale, via supervised-style trajectory prediction — then spend the comparatively much more expensive RL phase teaching it what to do with that grounding, rather than spending RL compute re-deriving execution semantics from scratch through trial and error.

Why "observation" and not "state" — a distinction worth keeping precise

Notice the paper is careful to say observation, not state, and that precision matters. Later, when the RL post-training phase runs (Chapter 5 covers it), the paper describes its training environments as partially observable Markov decision processes, or POMDPs: an agent — here, CWM itself — produces actions based on the sequence of action-observation pairs it has seen so far and an initial prompt, without ever having direct access to the environment's full internal state.

For the Python-tracing loop this distinction is almost invisible, because a traced function's local variables are essentially its entire relevant state — the observation is close to complete. For the agentic Docker loop it matters enormously: an agent editing a real repository never sees the full state of that repository at once. It sees whatever a bash command happened to print, or whatever a file-view tool happened to show. Two different actions against the identical underlying repository state can produce wildly different observations, depending only on which command the agent chose to run. Learning to act well under partial observability — knowing which action will reveal the state information you actually need next — is a meaningfully harder problem than learning to act when the full state is hand-delivered to you, and it is the reason ForagerAgent's toolset (Chapter 4) deliberately includes a dedicated "view or navigate inside a file" action: half of solving a software task under partial observability is knowing what to go look at.

What the loop looks like as actual code, not just a diagram

It's easy to let "observation-action trajectory" stay an abstraction. It shouldn't — the paper's own RL infrastructure (built to run this same loop at massive scale during post-training, Chapter 5 covers the full pipeline) implements the state/action recipe from the very top of this chapter as a two-method interface, and seeing that interface makes the abstraction concrete. Every environment CWM interacts with, whether during mid-training data collection or later RL, exposes exactly two operations:

python
# Illustrative reconstruction of the environment interface described in
# the paper's RL systems section -- the literal code-level shape of
# s' = f(s, a) from the top of this chapter.
class Environment:
    def start(self, sample):
        # begins a new episode from a dataset sample; returns an
        # initial hidden state plus the observation (prompt) the
        # agent actually gets to see
        return initial_state, initial_observation

    def step(self, state, action):
        # action = a sequence of tokens (a Python line, or a tool call)
        # returns the new hidden state plus everything the agent needs
        # next: the latest action, the new observation, and a reward
        return new_state, new_observation, reward

Notice how directly this maps onto both loops from earlier in this chapter. For the Python-interpreter loop, step's action is one line of code and its returned observation is the resulting local-variable JSON dict — exactly Chapter 3's trace format. For the agentic Docker loop, step's action is a bash command or file edit and its returned observation is whatever the repository's sandbox actually printed back. One interface, two very different environments plugged into it — which is itself evidence that "world model" here isn't a loose metaphor. It's a specific, implementable software contract that both of CWM's data streams satisfy.

One structural detail is worth calling out because it foreshadows something Chapter 5 covers in more depth: the environment, not just the agent, gets a say in the interaction — it can force a context switch, erasing past history or restarting a trajectory from scratch, enabling trajectories that span more than one contiguous conversational context. That capability matters for the same reason partial observability does: a real software-engineering task rarely fits neatly inside one uninterrupted stream of turns, and an environment abstraction that can't represent a restart or a context reset would be modeling a simpler world than the one real software engineering actually happens in.

Running this loop at the scale a 32B model needs

It's one thing to define start and step on paper; it's another to run millions of them fast enough to matter, especially once this same loop is being used not just to collect mid-training data (Chapters 2–4) but to drive RL post-training itself (Chapter 5). The paper's RL infrastructure splits GPUs into two roles that run continuously and asynchronously from each other: workers keep performing rollouts — calling start and then step repeatedly against real environments — while separate trainers consume the resulting trajectories and update the policy. Workers stream completed trajectory batches to trainers as soon as they're ready, and trainers push updated weights back to workers periodically, rather than the whole system pausing in lockstep between a generation phase and a training phase. One detail is worth sitting with because it's a genuinely subtle correctness question: when a worker receives a new set of weights mid-rollout, it doesn't restart the in-progress trajectory from scratch — it continues generating the rest of that trajectory using its existing KV-cache, now paired with the newly updated policy. That's a deliberate throughput-versus-purity tradeoff, keeping GPUs continuously busy rather than idling them waiting for perfectly synchronized weights, at the cost of a trajectory occasionally being generated by a policy that changed partway through it.

The paper's own summary of the goal

The paper's conclusion states the vision in one sentence, worth quoting exactly because it is the thesis this entire session unpacks: “coding and agentic use cases of LLMs will benefit from having a world model, a learned transition function between states conditioned on actions.” Everything from here through Chapter 8 is either building that transition function's training data (Chapters 2–4), building the model that learns it (Chapter 5), demonstrating it works (Chapter 6), or measuring what it buys you (Chapters 7–8).

Chapter 2: Building Ground Truth at Scale

Chapter 1 makes the goal sound simple: run code, record what happens, train on it. There is a large practical problem hiding in that plan. You cannot just download a few million GitHub repositories and start running their test suites. Most of them will not even build — missing dependencies, undocumented setup steps, environment assumptions baked in by whoever last touched the project years ago. Before CWM can trace a single line of anyone else's code, it needs somewhere safe, repeatable, and working to run that code in.

Executable repository images

The paper's solution is what it calls executable repository images: Docker containers, one per source repository, each preconfigured so the repository's code and tests can actually run without any further setup. Docker buys isolation (nothing a traced program does can escape the container) and repeatability (the same image produces the same environment every time it is spun up) — both essential properties once you are about to run untrusted, historical, sometimes-broken code at scale, over and over, to harvest training data from it.

Building 35,000-plus of these by hand would not scale. The paper runs two different automated pipelines in parallel to get there.

Pipeline one: RepoAgent

RepoAgent is an LLM-backed agent tasked directly with the setup job: configure a target repository's development environment, locate its test files, and get a meaningful fraction of those tests passing. To help it, the pipeline feeds RepoAgent human-readable documentation extracted from the target repository — READMEs, setup guides, contributing docs. This measurably improves RepoAgent's success rate. It also introduces a real weakness the paper names explicitly: human-targeted documentation can be wrong, or stale, because nothing forces it to stay accurate. A README that says “run pip install -r requirements.txt” can be years out of date and nobody notices, because the only consequence of it being wrong is a slightly annoyed new contributor, not a build failure the whole team sees.

Pipeline two: Activ, repurposing CI

That weakness motivates pipeline two directly. Machine-targeted instructions — specifically, a repository's own GitHub Actions CI configuration — cannot silently rot the same way, because GitHub itself will visibly flag a failing build to every maintainer. If the CI config is wrong, someone finds out fast, and it gets fixed. That reliability is exactly what a data pipeline needs, so the paper builds Activ (“Act in virtual”): it repurposes each repository's existing GitHub Actions workflows as an automated, always-current build recipe, running them locally via the open-source act library rather than on GitHub's own infrastructure. Because most CI workflows were never designed to be run by a third party, and are not limited to plain build-and-test jobs, Activ modifies the target repository's source to trigger an early exit right after the first successful build, and injects a pytest fixture (since CI jobs run in separate, transient containers that would otherwise vanish) that captures that container's build state at the moment tests execute. The captured state is then committed and pushed as the resulting image.

The number this produces

Running both pipelines in parallel, the paper reports over 35,000 unique executable repository images. This is the substrate everything in the next two chapters is built from: Chapter 3's Python tracing needs working repositories to trace inside, and Chapter 4's ForagerAgent needs working repositories for an agent to actually act inside. Neither is possible without this chapter's pipeline running first.

What "executable" buys you that a static snapshot doesn't

It's worth being explicit about why a static, unrun copy of a repository (which is trivial to obtain — just clone it) would not have been enough. A static snapshot can tell a model what code looks like. It cannot tell a model what a specific test does when it runs against a specific commit, what error message a specific mutation produces, or what a specific agent action changes about the repository's state. Every one of those requires actually executing something inside a working environment — and that is precisely what an executable image, and only an executable image, provides.

The engineering trick inside Activ, one level deeper

It's worth pausing on exactly how Activ solves its trickiest constraint, because the trick generalizes well beyond this one pipeline. GitHub Actions runs every workflow job in its own separate, disposable container, and those containers are torn down the moment the job finishes — there is no built-in mechanism to reach in afterward and ask “what did the filesystem look like right before the tests ran?” Two changes make that possible. First, the pipeline edits the target repository's source so that, the instant one build succeeds, the workflow exits early rather than continuing on to whatever else it would normally do — there's no reason to let a CI job keep running once the one thing you actually wanted (a working, populated environment) has already happened. Second, it injects a fixture into the repository's own pytest configuration, engineered to fire automatically the moment test execution begins — and it is that fixture, running inside the transient container, that actually captures and exports the build state, since nothing outside that container can reach in and grab it after the fact. The resulting image is then committed and pushed before the container disappears. Every piece of this design exists to solve one specific fact about how CI infrastructure works: the environment you want only exists for a few seconds, inside a box that is designed to vanish.

RepoAgent versus Activ, side by side

Neither pipeline alone would have reached 35,000-plus images: RepoAgent covers repositories without usable CI that a documentation-guided agent can still bring up; Activ covers repositories with CI, more reliably, at lower per-repository engineering cost. Running both in parallel is itself an engineering decision worth noticing — it trades a bit of pipeline complexity for meaningfully broader coverage than either method alone would achieve.

Collapsing a build matrix down to one

A real repository's CI configuration often tests against a whole matrix of environments — several Python versions crossed with several operating systems, for instance — because the maintainers want confidence their package works everywhere a user might install it. Activ needs exactly one of those combinations to succeed, not all of them, so the pipeline deliberately collapses that matrix down to a single entry, selecting the most broadly compatible Python version and Ubuntu variant rather than attempting (and paying for) every combination the original CI config specifies. It also modifies each workflow to continue on non-critical failures rather than aborting the whole job the moment any one step errors, and layers in several independent early-exit triggers — capture succeeded, progress has stalled, or a timeout was hit — so a single slow or partially broken repository can't stall the entire data-collection pipeline behind it.

The same design also generalizes past Python specifically: for pytest-based repositories, the injected fixture is what signals success, but non-Python repositories (the paper names JavaScript's Jest test framework as an example) use a modified workflow step instead, probing for whichever test framework the repository actually uses before triggering capture. The mechanism — detect that the environment is genuinely usable, then capture and freeze it before the container disappears — is the same one described earlier in this chapter; only the specific signal used to detect "it's ready" changes per language ecosystem.

One number that connects this chapter to the next

Chapter 3 will report over 120 million traced Python functions. It's worth pausing here, before moving on, to connect that huge number back to this chapter's much smaller one — 35,000-plus executable images — because the ratio between them says something real about how the two pipelines actually feed each other:

That average is doing a lot of work in one number. It says the payoff from Chapter 2's engineering — building one more working, isolated, repeatable Docker environment — is not a single trace. It's thousands of them, because once a repository is executable at all, every function inside it, called with every fuzzed or generated input the pipeline can throw at it, becomes another training example. Chapter 2's pipeline looks, on its own, like the least glamorous part of this session's story — no clever trace format, no architecture diagram, just Docker plumbing. It's also the multiplier that makes everything downstream possible at the scale it happens at.

Chapter 3: Python Tracing

With 35,000-plus working repositories in hand, Chapter 3 covers the first of CWM's two world-modeling data streams: Python execution tracing, which the paper also calls neural code interpretation data. The goal is a dataset where every training example pairs a line of Python with the exact effect that line has on the program's memory.

The trace format

Every trace step is built from the same rule, stated precisely in the paper: the observation contains the local variables and stack-frame metadata immediately prior to executing a line of code; the action is that specific line; the next observation captures the resulting local variable state, plus event metadata such as return statements. Global variables and external side effects are deliberately left out — the trace tracks local state only. A set of custom tokens marks the structure: <|trace_context_start|> opens a trace, <|frame_sep|> separates steps, an event token (<|call_sep|>, <|line_sep|>, <|return_sep|>, or <|exception_sep|>) names what kind of step it is, <|action_sep|> separates the state from the code line, and <|arg_sep|> introduces a return or exception argument. The local-variable state itself is written as a plain JSON dictionary. Chapter 6 walks one of these traces line by line in full.

One compression trick matters enough to call out now: variables whose value did not change since the previous step are written as an ellipsis, "..", rather than being restated. Without this, a trace of a function with ten local variables running for a thousand steps would restate all ten variables' full values a thousand times, even when only one of them changes on any given step — a huge, mostly redundant token cost. The ellipsis convention keeps trace length proportional to how much actually changes, not to how many variables happen to be in scope.

Four sources, four different ways to get ground truth

The paper builds this dataset from four distinct sources, each solving a different piece of the “where do I get traceable code from” problem.

1. Function-level tracing

Starting from Python functions collected from online sources, the pipeline generates input–output pairs using a combination of fuzzing and prompting Llama3-70B-Instruct, then traces execution by capturing the Python interpreter's stack frames at each event (executed lines, return statements, exceptions). This is the largest single source: over 120 million traced Python functions.

2. CodeContests solutions tracing

For competitive-programming style code, the pipeline prompts Llama-3.1-70B-Instruct to generate Python solutions to problems from the CodeContests training set, deliberately filtering to keep a balance of both correct and incorrect submissions (a model needs to see broken code trace too, not only working code). This produces 262,000 generated solutions. Each is traced using inputs from the problem's own unit tests, and long or oversized traces (more than 10,000 line events, or more than 1 MB on disk) are filtered out, leaving 33,000 effective code snippets and 70,000 traces.

Two small worked numbers are worth sitting with here, because they show how aggressively this stage prunes:

Roughly seven in eight generated solutions get thrown away — either because they were too long or too large to trace cheaply, or (implicitly, given the balance requirement above) because the pipeline is deliberately curating for a mix rather than keeping everything indiscriminately. What survives is smaller, but each surviving snippet is traced from more than one angle.

3. Repository-level tracing

This source traces the actual unit tests of the executable repository images from Chapter 2 — over 21,000 traceable images. For a subset of repositories, the pipeline additionally walks the repository's own git log to pull in older commits, since a working build recipe for the current commit is not guaranteed to work for a commit from months earlier (dependency versions drift). The pipeline attempts tracing for up to 40 historical commits per repository, but deliberately caps successful traces at 4 per repository, specifically to avoid over-representing any single, easy-to-trace repository in the final dataset. The result: around 70,000 execution-traced commits.

Raw traces from real pytest runs are messy — deeply nested function calls, huge call stacks. Two post-processing steps clean this up. First, traces are “episodified”: function-level episodes are extracted from the raw trace with a configurable stack depth, and a stochastic “step-in” probability decides whether a called function's own trace gets folded into its caller's episode or kept as a separate one — simulating variable execution depth so the dataset isn't uniformly shallow or uniformly deep. Second, the source code context needed to make sense of each trace is gathered and compressed from the target repository, then formatted the same way as every other trace source.

4. Natural language tracing

The fourth source rewrites the strict JSON-style traces into step-by-step natural-language descriptions, generated by prompting Qwen3-32B-FP8 (with its thinking mode off) to narrate the function-level and CodeContests traces in prose instead. The paper's stated motivation: natural language is closer in domain to how LLMs reason elsewhere, which should ease knowledge transfer, and it opens the door to things the strict JSON format cannot easily express — semantic commentary (the paper's own example: “this operation preserves the structure property of the max heap”), or compressing a trace by skipping the repetitive parts of a loop's logic rather than restating every iteration mechanically. Cases where Qwen's final narrated output disagrees with the ground-truth trace are filtered out. The result: 75 million trajectories from standalone functions, plus 110,000 from CodeContests.

What one step of this format looks like, reconstructed

python
# Illustrative reconstruction of one trace STEP -- not literal training
# tokens, just the structure Section 2.2 describes in prose.
trace_step = {
    "event": "line",               # call | line | return | exception
    "state_before": {                # locals immediately BEFORE this line runs
        "a": "..",                  # unchanged since the last step -- compressed
        "b": "..",
        "y": "1",                   # changed -- restated in full
    },
    "action": "for i in range(b):",  # the exact source line about to execute
}
# globals and external side effects are deliberately excluded

Adding it all up: how big is this corpus, really?

It's worth pausing to combine the four sources into one picture, because no single number in Section 2.2 communicates the total scale on its own — it's spread across four very differently-sized pipelines.

That combined number dwarfs the ForagerAgent stream from Chapter 4 (3 million trajectories) by roughly sixty-five to one — a useful thing to hold onto heading into Chapter 4, and a reminder that the two world-modeling streams are not remotely balanced in raw volume. Function-level tracing is cheap to generate (fuzz a function, run it, record the trace) and so it dominates by sheer count; ForagerAgent trajectories are expensive to generate (an LLM agent has to actually work through a multi-turn software engineering task inside a live Docker container) and so there are far fewer of them, even though, as Chapter 8's ablation shows, both streams turn out to matter for very different downstream skills.

Two formats, one underlying signal, is also worth noticing as a design pattern in its own right: the same underlying ground truth (an interpreter's actual execution) gets represented twice — once as a strict, compact, machine-checkable JSON trace, and once as a looser, more verbose natural-language narration of the same events. Training on both means the model doesn't just learn one rigid notation for “what happened here” — it learns to express the same grounded knowledge in whichever register a given task actually calls for, structured or conversational.

Chapter 4: ForagerAgent

Python tracing (Chapter 3) teaches CWM what happens when code that already exists gets run. It says nothing about what happens when an agent acts inside a repository — edits a file, runs a shell command, reads test output, decides what to do next. That is CWM's second world-modeling data stream, and it comes from an agent the paper calls ForagerAgent.

What ForagerAgent does

ForagerAgent collects multi-step trajectories by prompting an LLM — Llama3-70B-Instruct or Qwen3-235B-A22B with its thinking mode off — with a software-engineering task inside a specific repository, drawn from the same 35,000-plus executable repository images Chapter 2 built (with SWE-bench repositories and their forks explicitly filtered out, to keep the eventual SWE-bench evaluation in Chapter 8 uncontaminated). The action set is the standard SWE-Agent toolset: create a file, edit a file, run a bash command, or view and navigate inside a file. A trajectory ends when the agent believes the task is solved, or when a hard limit on tokens, turns, or API cost is hit.

Two task families

Mutate-fix tasks (synthetic). Start from a repository whose tests all pass. Identify functions that can be verified by the repository's own test suite. Then apply one of five kinds of mutation, chosen to break the function in a way its tests will catch:

Every candidate mutation goes through two checks before it's used: it must survive being parsed against the function's abstract syntax tree (some mutations simply don't apply to a given function), and applying it must actually cause the associated unit tests to fail. Only then does ForagerAgent get pointed at the mutated function and instructed to inspect it, run its tests, and fix the failure.

Issue-fix tasks (real). These use real GitHub history directly: check out the commit immediately before a real bug-fixing pull request, hand the agent the actual GitHub issue description for context, and task it with resolving the failing tests. The pipeline verifies both that tests genuinely fail before the fix and that the historical fix was necessary and sufficient to resolve the issue — so the task is a faithful replay of a real bug, not a fabricated one.

Post-processing: dedup, and training on failure too

To avoid overfitting to repetitive interaction patterns pulled from the same source repository, trajectories are represented by the concatenation of their actions, encoded with MinHash, and near-duplicates are dropped so that any two kept trajectories from the same repository have pairwise Jaccard similarity below 0.5.

More importantly: ForagerAgent trajectories are not filtered by whether the agent actually succeeded at fixing the bug or issue. The paper is explicit about why — the goal is a comprehensive world model of agentic interaction with code environments, not a demonstration set of expert behavior to imitate. Following that same logic, the model is trained to predict both the agent's own turns and the environment's responses, though loss on 50% of the environment observations is stochastically masked, since they tend to be low-diversity (a passing-test message looks like every other passing-test message).

The numbers, and a worked cross-check

Overall, ForagerAgent produces 3 million trajectories from 10,200 executable images across 3,150 underlying repositories, split 55% issue-fix to 45% mutate-fix. The five mutation-type percentages in the table above are a nice arithmetic check: they should sum to the 45% mutate-fix share, since mutate-fix trajectories can only be one of those five types.

That consistency lets you convert the percentages into absolute trajectory counts directly:

Three hundred sixty thousand trajectories, from a single mutation type — swapping + for -, or < for ≤, and watching an agent chase down the resulting failure — is on its own a larger, more targeted “operator confusion” training set than most dedicated benchmarks of any kind.

What a trajectory actually costs to generate

Every ForagerAgent trajectory is bounded by a hard limit — on tokens, on the number of turns, or on API cost — whichever it hits first, at which point the trajectory ends whether or not the task was resolved. This is a real, practical constraint of the pipeline, worth naming honestly: generating 3 million multi-turn agentic trajectories, each one an LLM actually reasoning, calling tools, and reading back real environment output, is not free. Capping each trajectory's budget is what makes 3 million of them tractable at all — the cost per trajectory has to stay small enough, multiplied by 3 million, to be worth spending compute on mid-training data rather than on the RL phase that comes later.

How near-duplicate detection actually decides "too similar"

The dedup step deserves a concrete illustration, since “MinHash” and “Jaccard similarity below 0.5” can otherwise stay abstract. Jaccard similarity between two sets is just the size of their overlap divided by the size of their union — a number between 0 (nothing shared) and 1 (identical). Represent each trajectory by the set of distinct actions it took; two trajectories that both, say, open the same file, run the same test command, and make a similar edit will share most of their action-set, and score close to 1. MinHash is simply a fast way to estimate that similarity without actually comparing every pair of trajectories directly, which would be far too slow at 3-million-trajectory scale — it hashes each trajectory's action set down to a small fixed-size signature such that similar sets produce similar signatures, letting near-duplicates be found by comparing cheap signatures instead of full sets.

Keeping the evaluation honest: decontamination

One filtering step matters enough to restate on its own: every repository used to seed ForagerAgent tasks — and every fork of it — is checked against the set of repositories used in SWE-bench, and excluded if it matches. Without this, a repository that ForagerAgent trained extensively on could also appear, verbatim or near-verbatim, in the SWE-bench Verified evaluation Chapter 8 reports pass@1 scores on, and any resulting score gain would be partly memorization rather than genuine skill transfer. Decontamination is a quiet, unglamorous step, and it's precisely the kind of step that's easy to skip under time pressure and expensive to discover was skipped only after a benchmark number turns out to be inflated.

Chapter 5: The 32B Architecture, Honestly

Chapters 2 through 4 built the data. This chapter is about the model that data trains — a 32-billion-parameter, dense, decoder-only transformer — and the reasons behind each design choice, stated as plainly as the paper states them, including the reasons that are pragmatic rather than glamorous.

Dense, not sparse — and the honest reason why

CWM is a dense model: every parameter is used on every forward pass, unlike a mixture-of-experts model that only activates a subset of its parameters per token. The paper's stated reason is not “dense models perform better” — it's “ease-of-use in downstream open source research.” A dense checkpoint is simpler for other researchers to load, fine-tune, and inspect than a sparse one, and since the entire point of releasing CWM is to seed further research on code world modeling, that practical property outweighed whatever efficiency a sparse architecture might have bought.

Attention: Grouped-Query Attention, and a number that checks itself

CWM uses Grouped-Query Attention (GQA): 48 query heads, but only 8 key-value heads, each 128-dimensional. GQA lets multiple query heads share the same key/value heads, which shrinks the KV cache without shrinking the model's representational width in the query direction — if this sounds familiar, it's the exact mechanism Session 08 of this course flagged as a direct engineering response to KV-cache pressure. Here it is in a real, shipped 32B model.

Two numbers worth checking against each other, the way a careful reader should always cross-check an architecture table:

Neither number is stated directly in the paper's architecture table — both are exactly the kind of sanity-check arithmetic that catches a misread table before it propagates into a wrong mental model.

Interleaved local and global attention, 3:1

Rather than every layer attending over the full context, CWM interleaves local attention blocks (a sliding window of 8,192 tokens) with global attention blocks (up to 131,072 tokens) in a 3-to-1 ratio — three local layers for every one global layer. Local attention is far cheaper per token; global attention is what actually lets information from far earlier in a long agentic trajectory or a long trace reach the current position. Mixing them is a standard long-context cost-control trick, applied here at a scale that matters: CWM's execution traces and agentic trajectories can run to tens of thousands of tokens.

Two more derived numbers, worked out from the stated ratio and window sizes (not stated directly by the paper, but a direct consequence of them):

The rest of the architecture, and why it's chosen this way

Two-stage pretraining

Training happens in two stages that share a learning-rate schedule and optimizer state but differ in datamix and maximum document length.

Two honest engineering admissions in that mid-training row are worth reading closely. First, the much larger 33M-token batch size at long context: the paper explains that smaller batches produced “lackluster performance” on long-context data, because fewer documents per batch increases gradient-estimate variance — a bigger batch was needed just to keep training stable, not to go faster. Second, the 30% rehearsal fraction of the original pre-training mix, deliberately upweighted toward higher-quality math and long-context data: the paper states this “proved essential in retaining performance on standard evaluations,” a direct admission that mid-training purely on CWM-specific data, without rehearsal, would have degraded the model's general capability.

Hyperparameters, and the scaling laws behind them

Training uses AdamW (β₁=0.9, β₂=0.95, weight decay 0.1, gradient clipping at norm 1.0), 2,000 steps of linear warmup, then cosine decay from a peak learning rate of 8×10^-4, decaying by 100× over a schedule sized for 13T total tokens (the last 5T of which is the mid-training tail).

None of those numbers were guessed. The paper fits scaling laws first: define compute budget C = M·D, where M is FLOPs per token (a linear term plus an attention term that accounts for self-attention's real cost, especially at long context) and D is total training tokens. Rather than the compute-optimal ratio from Chinchilla-style scaling, CWM deliberately fixes D/M = 40 — roughly 8× more data relative to model size than compute-optimal — because a model meant to be served cheaply after training benefits more from being over-trained on data than from being training-compute-optimal. A quasi-random search across eight compute scales, from 2×10¹⁸ to 2×10²⁰ FLOP, fits:

Bigger compute budgets get a lower peak learning rate and a larger batch size — the same qualitative trend other scaling-law papers have found, fit fresh here because, as the paper notes, its different pretraining data mix makes borrowing someone else's fitted constants unreliable.

python
# the two scaling-law fits from Section 4.3, as callable functions
def lr_for_compute(C):
    return 19.29 * C ** (-0.177)

def batch_size_for_compute(C):
    return 30.17 * C ** 0.231

# GQA cache-savings sanity check from earlier in this chapter
def gqa_savings(n_query_heads=48, n_kv_heads=8):
    return n_query_heads / n_kv_heads   # 6.0 -- matches 48/8 above

Post-training, part one: SFT

Mid-training (the two-stage pipeline above) ends with a model that has been grounded in code semantics, but has not yet been shaped to follow instructions or reason step by step. That's the job of the next two phases: supervised fine-tuning (SFT), then reinforcement learning. SFT runs for 100 billion tokens across 50,000 steps, with a 2-million-token batch size and 32,768-token sequences — and the two numbers check each other directly:

The learning-rate schedule makes a deliberately un-fancy choice: 1,000 steps of warmup, then a constant 1×10^-5, not a cosine decay. The paper's own reasoning is refreshingly pragmatic — preliminary experiments found a constant rate reached similar evaluation scores to cosine annealing, while also leaving the model at a high, still-warm learning rate exactly when RL is about to begin, rather than having decayed it toward zero by the end of SFT the way cosine scheduling would. A subtler admission sits alongside this: SFT performance measurably dropped at longer sequence lengths, which the paper attributes to how its dataloader sequence-packs examples per dataset — small datasets at large context sizes get seen in fewer distinct training steps than larger ones do, quietly starving them of gradient updates relative to their size.

The SFT datamix itself closes an interesting loop back to earlier chapters: it reserves about 30% for rehearsal of the mid-training mix (which, recall, is itself 30% rehearsal of the original pre-training mix — a nested rehearsal, each stage protecting what the previous one taught), specifically to avoid overfitting to the SFT distribution ahead of RL. It also folds in agentic SWE RL trajectories generated by earlier iterations of CWM itself, filtered by rejection sampling — the model's own better attempts from a previous training round becoming training data for the next one. And it includes external reasoning trajectories from the OpenMathReasoning and OpenCodeReasoning datasets, both derived from DeepSeek-R1, because the paper found their benefit “carries through” all the way to the final, post-RL model.

One more mechanical detail worth knowing, because Chapter 8's benchmark numbers depend on it: SFT introduces <|reasoning_thinking_start|> and <|reasoning_thinking_end|> tokens that wrap any reasoning text. Loss on the opening token is masked, so the model never actually learns to generate it on its own — which means the SFT model defaults to non-reasoning behavior, and reasoning mode has to be explicitly turned on by injecting that start token at the beginning of its response. This is exactly the “small budget” versus “large budget” toggle Chapter 7's CruxEval table draws on.

Post-training, part two: the RL algorithm, honestly

CWM's RL phase uses a variant of Group Relative Policy Optimization (GRPO) — a policy gradient method that estimates the value of an action by sampling a group of completions for the same prompt and comparing rewards within that group, rather than training a separate value model the way classic PPO does. The paper is unusually candid about exactly where and why it deviates from the original GRPO recipe, and each deviation is a real, specific engineering fix for a real, specific failure mode:

That last row has genuinely specific thresholds worth seeing once, because they show what "explicitly reject" means in practice rather than leaving it vague: a token is flagged as gibberish if its vocabulary id exceeds 100,000 (out of the 128,256-token vocabulary from Chapter 5's architecture table — large ids correspond to rare, late-merged byte-pair-encoding tokens) and its log-probability is below −log(128,256)−2, meaning the model itself assigned it unusually low confidence. Both conditions have to hold together, which is what keeps this from misfiring on merely uncommon-but-legitimate tokens.

Where the SWE-bench number actually gets trained: Agentic SWE RL

Chapter 4 built ForagerAgent as mid-training data — a way to expose the model to agentic interaction cheaply, at scale, without filtering for success. RL's Agentic SWE environment is the direct next step in that same lineage, now training the model to actually solve tasks well, end to end, with a real reward signal. Its toolset is deliberately minimal: a stateful bash session as the core tool, with edit, create, and submit implemented as thin, "de-sugared" wrappers around bash commands — a design explicitly inspired by how Claude Sonnet 3.5 approaches agentic coding. Trajectories can run up to 128 turns inside a 131,072-token context window, ending when the agent submits a patch via git diff along with a written summary of how it resolved the issue.

The reward is a hybrid, and the exact shape of it is a real design decision worth understanding: if every hidden test passes, reward is 1. If not, the reward comes from comparing the submitted patch's similarity to the true oracle fix — but instead of using that similarity score continuously (as the SWE-RL paper this idea is adapted from does), CWM thresholds it: similarity above 0.5 earns a reward of 0, and similarity at or below 0.5 earns −1. The paper's stated reason for discretizing what could have been a smooth signal is training stability — it avoids rewarding low-similarity patches at all, while still giving the model some gradient signal on hard issues where it can't yet produce a fully test-passing fix, by rewarding partial credit for getting structurally close to the real answer.

Chapter 6: Tracing a Function By Hand

Time to see the format from Chapter 3 in full, on a real example straight from the paper's own prompting guide. Here is the function:

python
def f(a, b):
    y = a
    for i in range(b):
        y += y * i
    return y

# the model is asked: assert f(1, 3) == ??

Work the arithmetic by hand first

Before looking at what CWM predicts, compute the real answer, the ordinary way — this is the ground truth every prediction below will be checked against.

f(1, 3) returns 6. Simple enough to check by hand in under a minute — which is exactly the point of using this as the worked example. The question isn't whether the answer is hard to get; it's whether a model can produce the entire step-by-step path to that answer, not just the final number, in the exact format Chapter 3 described.

What CWM is trained to emit for this exact call

Using the trace format from Chapter 3 — source context first, then a stream of frames, each an event type, a JSON state, and an action — here is the full fourteen-frame trace CWM predicts for f(1, 3), reconstructed directly from the paper's own worked example:

trace format
<|frame_sep|><|call_sep|>{}<|action_sep|> def main(): # << START_OF_TRACE
<|frame_sep|><|line_sep|>{}<|action_sep|> return f(1,3)
<|frame_sep|><|call_sep|>{"a": "1", "b": "3"}<|action_sep|> def f(a, b):
<|frame_sep|><|line_sep|>{"a": "..", "b": ".."}<|action_sep|> y = a
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "1"}<|action_sep|> for i in range(b):
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "..", "i": "0"}<|action_sep|> y += y * i
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "..", "i": ".."}<|action_sep|> for i in range(b):
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "..", "i": "1"}<|action_sep|> y += y * i
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "2", "i": ".."}<|action_sep|> for i in range(b):
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "..", "i": "2"}<|action_sep|> y += y * i
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "6", "i": ".."}<|action_sep|> for i in range(b):
<|frame_sep|><|line_sep|>{"a": "..", "b": "..", "y": "..", "i": ".."}<|action_sep|> return y
<|frame_sep|><|return_sep|><|action_sep|> return y <|arg_sep|>"6"
<|frame_sep|><|return_sep|><|action_sep|> return f(1,3) <|arg_sep|>"6"

Notice how tightly this tracks the hand arithmetic above. y only ever appears with a real value on the exact frames where it changed — frame 5 (after y = a sets it to 1), frame 9 (after the first loop body sets it to 2), and frame 11 (after the second loop body sets it to 6, surfacing on the next for i in range(b): check — the one that discovers range(3) is exhausted and exits the loop). Frame 12, the return y line itself, shows y back as "..", because nothing changes its value on that particular step; it was already set to 6 as of frame 11. Every other frame shows it as "..", exactly the compression rule from Chapter 3. And the final two frames (13 and 14) are both return events, not line events — one for f returning to whatever called it, one for main's own call expression finishing — each carrying the same argument, "6", because that value is genuinely propagating up through two separate stack frames.

What did that 14-frame trace actually cost, in tokens?

Chapter 7 reports the paper's own measured averages for this exact format: roughly 18.8 tokens per predicted state and 10.0 tokens per predicted action, averaged across the function-level validation set. Apply those averages to the trace above as a rough estimate — 12 of the 14 frames carry a state dictionary (every frame except the two return frames, which carry only an action and an argument):

That estimate lands comfortably below the paper's own reported average of 497 tokens for full trace prediction — which makes sense, since f(a, b) is a short, simple function with a small loop, while the CruxEval and function-level validation sets this session's numbers are drawn from include plenty of longer, more complex functions with correspondingly longer traces. The relationship between a function's complexity and its trace length is not incidental, either: more lines executed, more distinct variables touched, more loop iterations run, all directly lengthen the observation-action sequence CWM has to predict — trace length is a real, measurable proxy for how much work a given piece of execution actually does.

Why four separate event types, not one

It's worth asking why the format bothers distinguishing call, line, return, and exception as separate event tokens rather than just streaming a flat sequence of states. Each one marks a genuinely different kind of transition that a program's control flow can make, and each carries different information. A call event is the only place new local variables get bound for the first time (frame 3 above: a and b appear out of nowhere, because that's the moment the function starts executing with real arguments). A line event is the ordinary step-forward case this example is mostly built from. A return event carries an argument — the value being handed back — and marks the current stack frame closing, which is why frames 12 and 13 above have no local-variable state at all: there's no "current line" left to report locals for, just a value propagating upward. This session's worked example never triggers one: an exception event marks control flow leaving a frame abnormally, carrying the raised exception instead of a return value — the same slot in the format, repurposed for the case where a function doesn't return at all, it fails. Separating these isn't a cosmetic choice; each event type is a different shape of transition, and collapsing them into one undifferentiated stream would throw away exactly the structural information that makes a trace useful for reasoning about control flow, not just data flow.

What an exception event would look like, illustrated

The worked example above never crashes, so it's worth seeing an exception frame at least once, even in a small illustrative form built for this lesson (not the paper's own text). Take a one-line variant: call f(0, 3) instead of f(1, 3). Nothing about the trace format changes — only where it ends:

trace format, illustrative
<|frame_sep|><|call_sep|>{"a": "0", "b": "3"}<|action_sep|> def f(a, b):
<|frame_sep|><|line_sep|>{"a": "..", "b": ".."}<|action_sep|> y = a
<!-- y is now 0; every iteration of "y += y * i" leaves it at 0 -->
<!-- suppose the function instead divided by y somewhere in the loop -->
<|frame_sep|><|exception_sep|><|action_sep|> y = 1 / y <|arg_sep|>"ZeroDivisionError: division by zero"

Structurally this is identical to a return frame — no state dictionary, just an action and an argument — and that symmetry is the point: both are ways a stack frame can end, and the format treats "ended with a value" and "ended with a failure" as siblings, not as fundamentally different kinds of event. A model trained to predict both is learning, in the same breath, what normal termination looks like and what abnormal termination looks like — which is exactly the pair of skills Chapter 7's program-termination stress test goes on to probe directly.

What happens when a function calls another function

This session's worked example never calls a second function from inside f, but real code constantly does — and Chapter 3's repository-level tracing source specifically handles this with its "episodify" step: a configurable stack depth plus a stochastic step-in probability decide whether a called function's own execution gets folded into its caller's trace (as a nested sequence of call / line / return frames, one level deeper) or treated as a separate episode entirely. Nothing about the format from Chapter 3 has to change to support this — a call event is already how frame 3 of this chapter's own trace begins, marking exactly the same kind of transition whether it's the very first frame of a trace or the fifth function called from three levels deep inside one.

Beyond tracing what's given: predicting unwritten code

The paper shows one more early prototype worth naming here, because it inverts the direction of Chapter 6's example. Instead of tracing a function that's already fully written, prompt CWM with only a natural-language description or a formal assertion — no function body at all — and ask it to predict a plausible execution trace anyway. The model then reconstructs a function body directly from the actions in its own predicted trace: if it predicted a trace where a variable gets incremented inside a loop condition on range(b), the corresponding code has to be whatever produces exactly that sequence of actions. The paper is explicit that this is early and prototype-stage, not a finished capability, but the direction is a natural extension of everything Chapter 6 demonstrated: if a model can reliably predict what code does, that same skill can run in reverse, as a constraint on what code it should write.

Chapter 7: What Trace Prediction Unlocks

Chapter 6 showed the mechanism working on one function. This chapter asks the harder question: what is this actually good for, beyond being a neat party trick on a five-line example?

The vision: a "neural debugger"

The paper is explicit that this is a first step, not a finished tool, but it states the ambition plainly: CWM's trace prediction could be extended into what it calls a “neural debugger” whose capabilities go beyond a traditional debugger — jumping to future lines of code in constant time (instead of a real debugger's forced step-by-step crawl through everything in between), predicting what inputs would be needed to reach an arbitrary target program state (the inverse of normal execution), and learning abstract representations of program state useful for verification, debugging, or generation, all without needing a live execution environment attached. The paper also shows an early prototype of using trace prediction to help write code that isn't fully specified yet — predicting a plausible execution trace for a function constrained only by natural language or formal assertions, then reconstructing the function body from the predicted actions. Read this section for what it is: an honest “here's the direction, we haven't built the whole thing yet,” not a shipped feature.

How you actually ask CWM to trace

Chapter 6's 14-frame trace didn't happen by magic — eliciting it requires a specific instruction, given to CWM as part of the prompt, spelling out the exact grammar Chapter 3 introduced. The prompting guide states the required structure as three explicit rules, worth reading in something close to their original form because they show how mechanical and unambiguous the format has to be for a model to reliably produce it:

trace prediction instructions, paraphrased from the prompting guide
1. The trace prediction starts with <|trace_context_start|> and ends
   with a final <|frame_sep|> token.
2. For each execution step: begin with <|frame_sep|>, followed by the
   event token (<|call_sep|>, <|line_sep|>, <|return_sep|>, or
   <|exception_sep|>). After call/line events, put the local-variable
   state as a JSON dict, then <|action_sep|>, then the source line.
   After return/exception events, put <|action_sep|> directly, then the
   source line, then <|arg_sep|>, then the return or exception value.
3. Provide the final assertion, with the correct output obtained after the
   <|return_sep|> event, inside [ANSWER] and [/ANSWER] tags.

Every element of Chapter 6's worked trace traces back to one of these three rules: rule 2 is exactly why a call frame's JSON dict comes before its source line, why a return frame skips the JSON dict entirely, and why the final return value appears after <|arg_sep|> rather than being folded into the state dictionary. None of the structure in Chapter 6 is incidental formatting — it's the literal grammar the model is instructed, and trained, to follow.

The measured payoff: CruxEval-Output, four ways

Where the paper does have hard numbers is a controlled comparison on CruxEval-Output — given a function and its inputs, predict exactly what it returns. Four different ways of answering the same question, each with a real measured score:

Table 8 of the paper reports both checkpoints on this same comparison: CWM-SFT is the supervised checkpoint straight out of mid-training, before the RL phase (Chapter 5); CWM (final) is the model after RL. Don't assume RL always adds a fixed few points across the board — on the first two rows it actually costs a little accuracy (67.8% → 66.6%, 59.1% → 58.1%), while on the last two rows it does the opposite: full trace prediction ticks up slightly (87.3% → 87.7%), and natural-language reasoning jumps by roughly eleven points (83.3% → 94.3%). Whatever RL is doing here, it is not a uniform bonus applied equally to every mode.

Read the second row carefully — it's the counterintuitive one. Trying to shortcut the format by predicting only the final return value (skipping every intermediate line) actually performs worse than not tracing at all — roughly 8–9 points worse than classic few-shot with no trace whatsoever. The full, line-by-line trace, by contrast, beats classic few-shot by roughly 21 points and beats the shortened single-step version by roughly 30 points (comparing the final, RL-trained CWM's own scores across rows). Whatever value CWM gets from execution-trace training, it comes specifically from working through the intermediate states, not from having merely been exposed to the format.

That derived ratio is the honest engineering tradeoff hiding inside this table: full trace prediction is the cheap, still-highly-accurate mode; natural-language reasoning is the expensive mode that buys a further few points at more than double the token cost. Neither is free, and neither is universally the right choice — it depends on whether that last handful of accuracy points is worth roughly doubling the inference cost.

Trace fidelity, measured directly

Beyond just the final answer, the paper checks how well-formed and how accurate the intermediate trace itself is, on held-out CruxEval and function-level validation data:

This is the scale-level confirmation of what Chapter 6 showed on one example: it isn't that CWM happens to get one hand-picked function right. Format compliance sits above 99% and per-step correctness sits in the high-90s across thousands of held-out traces.

A harder stress test: does a program halt?

Whether a program terminates is, in the general case, provably undecidable — no algorithm can determine termination for every possible program. The paper doesn't claim CWM solves the general problem; it builds a preliminary benchmark, HaltEval-prelim, of 115 terminating and 115 non-terminating Python programs, translated from C benchmarks in the SVCOMP and TPDB termination-verification competitions and manually checked for correctness after translation. Scoring is deliberately conservative: a claimed non-termination is only rewarded if the program actually times out (5 seconds) when run, and a non-termination guess against a program that is actually terminating is never rewarded — even if that program happens to run long enough to hit the timeout anyway.

Direct-answer CWM barely beats the constant classifier's coin flip. With reasoning enabled, it jumps to 0.94 — a 57-point swing from turning reasoning on, and comparable to Qwen3-32B's own reasoning-mode score, on a task the paper itself calls harder than expected to see strong results on at all. The honest caveat, stated directly: these are small, self-contained programs, and real-world termination is highly imbalanced — vastly more terminating loops than non-terminating ones in practice — unlike this benchmark's clean 50/50 split, so success here is not a guarantee of success on real, messy codebases.

A third stress test: predicting a program's Big-O complexity

One more grounded-reasoning task, evaluated on BigO(Bench): given existing code, predict its time and space complexity class (complexity prediction); or write a solution to a coding problem while satisfying a specified complexity requirement (complexity generation). Both tasks demand something closer to Chapter 6's simulation skill than to memorized pattern-matching — you cannot look up “this snippet is O(n log n)” from having seen similar-looking code before; you have to actually reason about how the number of operations scales as the input grows, which is a claim about execution behavior across an entire family of inputs, not about any single run.

Read across the whole table and a pattern repeats: CWM leads clearly on time complexity, on both the prediction and the generation variants, and it is explicitly the best code-only generator regardless of complexity constraints (76.1% pass@1 with no complexity requirement at all — the highest of any model compared). On space complexity specifically, it trails Qwen3-32B on every metric in the table. The paper does not paper over this asymmetry — it states plainly that CWM performs worse on space complexity than time complexity, an honest, specific admission rather than an average score that would hide which half of the task the model is actually weaker at.

Chapter 8: SWE-bench, Math, and the Data That Earned It

Chapter 0 opened with a question: does grounding a model in execution actually change what it can do, or is this an interesting idea that doesn't move real benchmarks? This chapter closes that loop with the paper's own controlled experiment, then with its headline results.

The ablation that proves the thesis

Section 7.1 runs the cleanest experiment in the whole paper. Take 8-billion-parameter models, pretrain all of them identically for 6T tokens, then vary only the final 1T tokens of mid-training datamix — adding GitHub PR data, then Python tracing, then ForagerAgent data, one at a time, on top of each other:

This table is the paper's central empirical claim, made visible in one place: Python tracing and ForagerAgent move different metrics, and neither alone gets you both. Tracing data drives a massive, specific jump in CruxEval-O (predicting what code returns) while leaving SWE-bench essentially flat. ForagerAgent data drives SWE-bench (agentic, multi-turn repository tasks) while barely touching CruxEval-O. Each data source teaches a different skill, because each data source is a world model of a different environment — one of a Python interpreter, one of an agent acting in a repository.

The final model, at full scale

The 8B ablation isolates why the data matters. The paper's headline numbers are from the final, 32-billion-parameter, fully post-trained (SFT + RL) CWM:

How the 65.8% is actually earned: best@k

The headline SWE-bench Verified number uses test-time scaling, not a single greedy attempt. For each problem, CWM generates k candidate patches in parallel, plus 40 novel unit tests meant to verify patch correctness and reproduce the original bug (following the Agentless approach), keeping only the top-5 majority tests that actually reproduce the error (following SWE-RL). Candidate patches are then filtered by how many existing tests they pass, executed against the filtered novel tests, and the highest-pass-rate patch is submitted — ties broken first by majority vote, then by fewest tokens. This whole procedure is called best@k, and at k=16 it reaches 65.8%. A far simpler alternative, plain majority voting with no test generation or execution at all, reaches 58.4% — a real gap, but proof that even the cheap version of test-time scaling helps substantially over a single greedy attempt (53.9%).

That gap is worth sitting with: it says a correct patch exists among the candidates far more often than best@16 manages to pick it out. Improving the selection step, not just generating more candidates, is real remaining headroom.

What it actually took to train and RL a 32B model like this

Every number in this chapter so far has been about what the model can do. It's worth spending one short section on what producing it actually cost in raw compute, because none of the earlier chapters' elegance — the trace format, the ablation table, best@k selection — happens without a training system that can actually push that much data through a 32-billion-parameter model.

Two thousand and forty-eight H100 GPUs, running for the trillions of tokens Chapter 5 walked through, is the actual machine behind this session's story. A few of the efficiency techniques that make that tractable are worth naming, because they're honest engineering, not glamorous research: linear layers train in fp8 low precision during pre-training and mid-training, roughly doubling the nominal FLOPs available on Hopper-generation GPUs compared to bfloat16 — but the paper reports that fp8 precision specifically hurt RL performance, so RL training reverts to bfloat16 for those same linear layers. That is a real, measured tradeoff the team hit and reported honestly rather than a design decision made purely on paper. Training also leans on FlashAttention-3 for faster, lower-memory attention, and on PyTorch's activation checkpointing (an integer-linear-program solver decides, given a memory budget, which activations to keep versus recompute) to fit a model this size and this long-context at all.

What RL training actually changes about the agent's behavior

Chapter 8's ablation table (earlier in this chapter) shows that ForagerAgent-flavored mid-training moves SWE-bench Verified pass@1. The paper also tracks two specific behavioral changes during the Agentic SWE RL phase from Chapter 5, on the same SWE RL environment, measured across training steps rather than just at the end:

File-localization recall here means: of all the files the true, human-authored fix actually touched, what fraction did the agent also edit during its own attempt? A low recall means the agent might be editing the wrong file entirely, no matter how good its code is once it gets there. Both numbers move in the direction you'd hope RL training would push them — the agent learns, purely from a sparse pass/fail reward signal (Chapter 5's threshold-based reward, recall, offers no direct instruction to "test more" or "look in the right file") to test its own work more often and to find the right place to make a fix more reliably. Neither behavior was hand-coded into the reward function; both emerged as instrumentally useful strategies for maximizing that reward, which is itself a small, concrete illustration of the paper's larger claim that a model already grounded in environment dynamics (from mid-training) has an easier time learning what to do with that grounding during RL.

Robustness, honestly: change the harness, the score moves

One more experiment deserves attention precisely because it's the kind most papers wouldn't bother running: does CWM's SWE-bench Verified score hold up if you swap out the agent harness — the actual scaffolding of prompts and tool-calling glue — for a completely different, third-party one? The paper tests this directly, with Mini-SWE-Agent and OpenHands standing in for its own harness:

Same model, same evaluation set, and nearly an 18-point swing depending purely on the scaffolding wrapped around it. The paper's own framing is honest rather than defensive: resolve rates genuinely degrade under different agents, tool implementations, or restricted tool choices, but performance “remains reasonable across the board” — it doesn't collapse to near-zero the way window attention collapsed in Session 08 of this course, but it is far from harness-invariant either. This is exactly the practical argument for why Session 07's agent-harness material and Chapter 4's ForagerAgent toolset design matter as much as the model weights themselves: a better model wrapped in a worse harness can score below a comparable one wrapped well.

Where CWM does not lead: multi-lingual and terminal benchmarks

Two more results round out an honest picture. On Aider Polyglot, which tests coding ability across six programming languages with a second-attempt self-correction step, CWM reaches 35.1% using the “whole file” edit format — comparable to other whole-file models like Gemini 2.0 Pro (35.6%), but far below top “diff” format models such as o3-pro (84.9%) or DeepSeek R1 (71.4%). The paper states plainly that CWM “was not optimized for this format and does not reach competitive performance with it” — a direct, unhedged admission of a specific weakness rather than a benchmark it simply omits.

On Terminal-Bench, where an agent solves complex tasks by operating directly inside a tmux terminal session, CWM reaches 26.25% accuracy with the Terminus-1 agent at its default 50-turn budget — placing it on the official leaderboard below GPT-5-driven agents and o4-mini's best (Goose-harnessed) configuration, but above Gemini 2.5 Pro, another o4-mini configuration running the same Terminus-1 harness, Grok 3 Beta, Gemini 2.5 Flash, and Qwen3-32B. Even the ranking of the same underlying model (o4-mini) shifts by harness here — one more concrete instance of this chapter's harness-sensitivity point. Neither result is cherry-picked to make CWM look uniformly dominant; both are reported because they're what the evaluation actually showed.

Honest comparison, not a highlight reel

On math specifically, the paper does not claim a win. Against same-size baselines — Qwen3-32B, Magistral-small-1.2-24B, gpt-oss-20B — the paper states plainly that CWM “performs slightly worse across the board, with [a] notable gap compared to gpt-oss-20B (high) on AIME.” CWM's strength, by the paper's own framing, is specifically agentic and execution-grounded coding, not blanket superiority on every reasoning benchmark it's compared against.

Chapter 9: Limits & Connections

Every session in this course ends by taking the paper's own honesty about its limits seriously, and by connecting what was built here to the rest of the map.

What CWM explicitly is not

• Not a production model. Released under a noncommercial research license specifically so the community can explore code world modeling; the paper states plainly that CWM's performance in production and real-world scenarios has not been evaluated by Meta.
• Not a chatbot. RLHF was deliberately skipped — the paper states it does not intend to develop a general-purpose chatbot. CWM has seen some instruction-following data during SFT, but has not been optimized for general chat use, and the paper explicitly warns against using it as one.
• English only. Not evaluated or optimized for other languages.
• Python-only execution grounding. Every trace in Chapters 3 and 6 is Python. Extending trace-style world modeling to other programming languages, or to symbolic (rather than concrete) execution, is explicitly left as future work.
• Planning is not built yet. The deeper vision — actually planning with the learned world model, using either formal inference or informal reasoning — is, in the paper's own words, an area where “the capabilities are there, we just need to find out how to make the most of them,” comparing today's code world models to LLMs before chain-of-thought reasoning existed. This release is explicitly a starting point for that research, not the destination.

Transparency, honestly: where external models entered the pipeline

Worth naming plainly, because it's easy to lose track of across ten chapters of pipeline description: CWM's training data leaned on other, external LLMs in four specific places, and the paper is explicit about all four rather than letting them stay implicit. ForagerAgent (Chapter 4) used Llama3-70B-Instruct and Qwen3-235B-A22B (thinking off) as the agent doing the actual acting. Trace-to-natural-language conversion (Chapter 3) used Qwen3-32B-FP8 (thinking off) to narrate the strict JSON traces into prose. Function tracing (Chapter 3) used Llama3-70B-Instruct to generate input-output pairs and CodeContests solutions. SFT (Chapter 5) incorporated DeepSeek-R1 reasoning trajectories, via the OpenMathReasoning and OpenCodeReasoning datasets, with the paper noting explicit mitigations applied to those datasets — algorithmic bias filtering and cybersecurity protections. No other external LLM tokens went into training beyond these four named uses.

The paper also ran an automated Preparedness Report assessment, checking CWM's capabilities against Meta's own Frontier AI Framework in two catastrophic-risk domains — cyber and chemical/biological — benchmarked against other capable open-weight models (Qwen3-Coder-480B-A35B, Llama 4 Maverick, gpt-oss-120B). The stated conclusion: CWM's release is unlikely to meaningfully raise risk in either domain beyond what the existing open-source ecosystem already presents, placing it in the “moderate” risk band the framework defines — alongside a further honest note that CWM's undesirable-propensity rates sit comparable to most open-source models, though some (gpt-oss-120B specifically named) score substantially lower. None of this changes anything about the technical content of Chapters 0–8, but it's part of what “releasing a research model honestly” actually looks like in practice: naming exactly where external systems entered the pipeline, and publishing a risk assessment rather than only a capability one.

Bridge: two very different world models

This site's general World Models lesson covers Genie, DeepMind's world model built from unlabeled video — trained on roughly 200,000 hours of 2D platformer gameplay footage, with zero action labels, discovering its own latent action space purely from watching pixels change. Setting Genie next to CWM makes the contrast in this session sharper by comparison:

Same core recipe — predict the next state given an action — but code hands a world-model builder something video never does: a deterministic, infinitely repeatable, free source of exact ground truth. That is a large part of why execution-trace world modeling for code is such fertile, comparatively low-hanging research ground right now.

Bridge: the agent-harness lessons

ForagerAgent's action set in Chapter 4 — create a file, edit a file, run a bash command, view or navigate a file — is exactly the harness-design problem Session 07 of this course, Agent Harness: Workflow Memory and SDK Design, covers in depth: how an agent's available actions get defined, sandboxed, and remembered across a long-running task. The general Harness Engineering treatment of tool-use loops lives at Agents & Tool Use. ForagerAgent is a concrete instance of that same loop, purpose-built to generate world-modeling training data rather than to solve tasks for a user.

What was actually released, and why that structure matters

Consistent with the “this is a testbed, not a finished product” framing running through this chapter, CWM's release is deliberately layered rather than shipping only the final model. The paper states checkpoints are released after each major stage — after mid-training, after SFT, and after RL — alongside the inference code, rather than only the single most-capable final checkpoint. That structure is itself a statement about what the release is for: a researcher who wants to study what mid-training alone contributes, without the RL phase's own effects layered on top, can start directly from the mid-training checkpoint rather than trying to reverse-engineer its contribution from the finished model. The paper's own framing in its opening pages makes the intent explicit — first and foremost, the release exists “to enable novel research on improving code generation with world modeling,” and a single end-to-end checkpoint would make that kind of stage-by-stage research meaningfully harder.

What this session actually established

“Program testing can be used to show the presence of bugs, but never to show their absence.” — Edsger W. Dijkstra, 1972 ACM Turing Award Lecture, “The Humble Programmer”

A model that can genuinely simulate a program's execution — rather than only run it once and observe the result — is a small step toward the kind of reasoning Dijkstra was gesturing at: not testing by running, but reasoning by simulating. CWM's own results say that step is real, current, and still early.