A training environment is a small world with a score. If the score reads what the agent says, a lazy agent wins. If it reads the world the agent changed, only real work pays.
Learn how to build the worlds that agents train and are tested in: the contract, the tasks, the checkers, the simulated users, and how to run thousands of them without fooling yourself.
Pick an agent and a grader, then press Run the episode. All three agents say the same five words. Watch which grader notices. Then we build, piece by piece, the whole environment around that one idea.
You need a little Python and a rough idea that a reward is a number that says how well the agent did. We build the rest from zero.
Toy task: “Cancel order W-1042 and leave W-2210 open.” The words-judge is simplified to one rule, it approves any confident “Done”. A real LLM judge is better than that, and Chapter 2 measures what it still misses. The checker compares the database with the goal state, the way τ-bench scores its episodes.
Chapter 0
See why the world an agent trains in decides what it learns, and meet the six parts every such world has
A team trains a support agent. The agent reads a customer message, calls tools on an order database, and writes a reply. On Monday its training score is 0.2. By Friday the score is 0.9. The team ships it.
The next week, customers complain. The agent says “Done, your order is cancelled” and the order is still open. Nobody changed the model between Friday and Monday. The score was real. It simply measured the wrong thing.
The score came from a grader that read the agent’s last message. The agent learned the cheapest way to earn that score: say “Done” with confidence. The hero at the top of this page is that exact failure, made small. Press it again if you skipped it.
An agent is a language model in a loop. It reads something, picks an action such as a tool call, reads the result, and repeats until it stops. Think of a clerk who works through a stack of forms one at a time.
An environment is everything outside the model that the agent acts on. It holds the world (a database, a file system, a web page), the tools that change the world, the task, and the score. Think of the clerk’s office, desk and supervisor, all in one.
Reinforcement learning (RL) is learning by trying. The agent attempts a task, receives a number called the reward, and its weights shift toward attempts that earned more. It is how you train a dog with treats, except the treats are numbers and the dog runs a million times a day.
In ordinary training, the dataset teaches the model. In agentic RL, the environment plays three roles at once. It is the dataset, because it decides which tasks the agent sees. It is the teacher, because it replies to every action. It is the examiner, because it decides what counts as good.
That makes the environment a research asset on its own. SWE-Gym (Pan et al., arXiv 2412.21139) packaged 2,438 real Python task instances, each with a runnable codebase, unit tests and a written task. Training agents in it gave up to 19% absolute gains in resolve rate on SWE-Bench Verified and Lite. The paper’s contribution was the world, not a new algorithm.
The same holds for evaluation. OSWorld (Xie et al., arXiv 2404.07972) holds 369 computer tasks. Each one ships with a setup config that builds the starting state and a script that checks the result. Humans complete over 72.36% of the tasks. The best model in that paper reaches 12.24%. The benchmark is the environment.
So a builder of environments carries a heavy duty. An agent can never be better than the world rewards it for being. If the world pays for a shortcut, the agent will find the shortcut, because finding shortcuts is exactly what optimisation does.
Every environment in this lesson has the same six parts. Click through them below. The next chapter writes each one as real JSON and as Python you could run.
The warm box is the agent. The blue box is the environment. Pick a part to light it up and read what it does.
The six names follow the Gym style of environment. Gymnasium, a widely used Python library of RL environments, documents it: reset returns the first observation, step returns the next observation, a reward and flags that say the episode ended. The 12-step count is an illustration.
Notice that the agent only ever touches two arrows. It sends an action and receives an observation. Everything else, the state, the reward, the stopping rule, lives inside the blue box where the agent cannot reach. That wall is the design. Keep the wall solid and the score means something. Let the agent see or edit what is behind it, and you are back in the opening story.
We build one environment piece by piece. Each chapter adds a part and then breaks it, because a part you have not seen fail is a part you do not understand yet.
Chapter 1
Write one tool-calling episode as plain records, and learn the five calls that every training world must answer
A customer writes: “Please cancel my kettle order, W-1042.” An agent must read that line, look the order up, cancel it, and tell the customer. That is one task. It takes three turns.
Now ask a plumbing question. How does a training program, which knows nothing about kettles, run this task a million times? It needs a fixed shape. Every world must accept the same kind of input and give back the same kind of output. That shape is the contract. Think of a vending machine: the buttons and the coin slot are the contract, and what sits inside can change.
The records are JSON, a plain text format for structured data: curly braces hold named fields, square brackets hold lists. Programs read it easily, and so do you.
reset builds a fresh world and returns the first observation. An observation is what the agent is allowed to see now. Here it is only the customer’s message.
json{"user": "Please cancel my kettle order, W-1042."}
An action is what the agent sends back. For a tool-calling agent it is a tool name plus arguments. Here the agent looks the order up first.
json{"tool": "get_order", "args": {"order_id": "W-1042"}}
step takes that action, runs the tool against the world, and returns four things: the next observation, a reward (a number for this turn), and two flags that say whether the episode has ended.
json{"obs": {"order_id": "W-1042", "status": "open", "items": ["blue kettle"]}, "reward": 0.0, "terminated": false, "truncated": false}
The reward is 0.0 here. Nothing is finished yet. Most agent tasks give a sparse reward: zero on every turn, then one number at the end. The next two chapters explain who computes that number and how.
Gymnasium also asks every environment to declare what it accepts and returns. The action_space lists the valid actions. The observation_space lists the valid observations. For a tool-calling agent these are not boxes of numbers. They are schemas: a list of tool names, and for each tool the argument names and types. A schema lets the environment reject {"tool": "cancel_ordr"} before it touches the world. It also lets you generate fake actions to test the environment without any model at all.
An episode is one run from reset to the end. Gymnasium, the standard RL environment library, returns (observation, reward, terminated, truncated, info) from step. Its docs define the two flags carefully.
terminated means the task reached an end that belongs to the task: the customer says thanks, or the goal is met. truncated means a limit from outside the task cut the run short: a step budget, a clock. The old single done flag was split into these two in Gymnasium v0.26.
The split matters for learning. A terminated episode has a true last state, with nothing after it. A truncated one has a future that was never seen. Training code that treats both as “nothing comes next” will undervalue states near the limit.
Try it. Step through two agents with the same four-step budget. One finishes. One wanders until the budget stops it.
Choose an agent, then press Step. The left panel shows the exact records. The right picture shows the order rows and the step budget.
Illustrative episode on a toy retail database. The record shapes follow the Gymnasium step return. The reward is 1.0 only when the final database matches the goal state (W-1042 cancelled, W-2210 open).
Here is a small environment class. Read it top to bottom. Each line maps to a record above.
pythonimport random class RetailEnv: max_steps = 4 def reset(self, *, seed=None, options=None): self.rng = random.Random(seed) # same seed, same world self.db = make_db(self.rng) # the world: orders, users self.goal = make_goal(self.db, self.rng) # the state we want at the end self.t = 0 return {"user": self.goal.message}, {"task_id": self.goal.id} def step(self, action): self.t += 1 obs = self.run_tool(action) # may change self.db terminated = action["tool"] == "reply" # the task's own ending truncated = self.t >= self.max_steps and not terminated # an outside limit reward = self.score() if (terminated or truncated) else 0.0 return obs, reward, terminated, truncated, {"t": self.t}
Three design decisions hide in these lines. First, reset takes a seed, so a failed episode can be replayed exactly. Second, score reads self.db, the world, never the agent’s text. Third, a truncated episode is still scored, so a half-finished job can earn something. Chapter 6 uses that.
Symptom: a batch of episodes ends with reward 0 and no tool result. Measurement: count how many actions failed to parse, and put the count in info. If 30% of actions are malformed, you are training on typos.
Fix: return an error as an observation, for example {"error": "unknown tool: cancel_ordr"}, and let the agent try again. The error costs a step. That teaches the agent to read tool schemas carefully, and it keeps the episode alive long enough to learn from.
Chapter 2
Learn the two ways to score an episode, and choose the one that cannot be talked into a mistake
A hotel manager wants to know if room 12 was cleaned. She can read the housekeeper’s note: “Room 12 done, spotless.” Or she can open the door and look. The note is cheap. The door is honest.
Scoring an agent is the same choice. The agent’s transcript is the note. The world state is the room: the rows in a database, the files on a disk, the response of an API. A state check opens the door. A text judge (a person or a model that reads the transcript and decides) reads the note.
Many agent tasks leave a footprint. A refund adds a row. A code fix changes a file, and tests either pass or fail. A calendar task moves an event. If the footprint exists, you can score from it.
The sources use this idea again and again. τ-bench (Yao et al., arXiv 2406.12045) compares the database state at the end of a conversation with an annotated goal state. OSWorld ships, for each of its 369 tasks, a custom execution-based script that inspects the machine afterwards. AppWorld (Trivedi et al., arXiv 2407.18901) scores with state-based unit tests.
AppWorld adds a point worth remembering. Its tests allow many valid ways to finish a task, and they also look for unexpected changes, which it calls collateral damage. An agent that cancels the right order and also cancels another one has not succeeded.
Compare three things: the world before, the world after, and the goal. Reward 1.0 needs two conditions. Every goal field must match, and nothing outside the goal may have changed.
pythondef state_match(before, after, goal): # before, after: {"W-1042": "open", "W-2210": "open"} goal: {"W-1042": "cancelled"} changed = {k for k in after if after[k] != before.get(k)} right = all(after.get(k) == v for k, v in goal.items()) spill = changed - set(goal) # collateral damage return 1.0 if right and not spill else 0.0
Worked example. Before: W-1042 open, W-2210 open. Goal: W-1042 cancelled. If the agent cancels only W-1042, then changed is {W-1042}, right is true, spill is empty, and the reward is 1.0. If it cancels both orders, changed is {W-1042, W-2210}, spill is {W-2210}, and the reward is 0.0 even though the goal field matches.
Some tasks have no footprint. “Write a polite reply to an angry customer.” “Summarise this report.” The output is the text itself, so you need someone to read it. A judge with a written rubric (a checklist of what good looks like) is the usual answer. Chapter 4 covers rubrics and how they break.
The rule of thumb is simple. Use a state check wherever the task changes something. Use a judge only for what the state cannot show. Many good environments use both: a state check for the facts and a judge for the tone.
Every episode answers the same task: cancel W-1042 and leave W-2210 open. The green or red mark says whether the grader’s verdict matches what really happened. Switch graders and watch the marks.
Toy rules, not a real model. The words judge approves any message that sounds finished. The log judge also checks that a cancel call appears, but not which order it hit. The checker runs state_match. Real LLM judges sit between the first two rows, and Chapter 4 shows how to measure where.
Sometimes a judge is the only option. Then measure it. Take 50 episodes that have a state check too. Score them with both. If the judge agrees with the checker on 41 of 50, agreement is 82%. Read the 9 disagreements one by one. Do they share a cause, such as a long confident message, or a tool call the judge never saw? Each shared cause is a fix to the judge’s prompt or inputs. An unmeasured judge is a guess with a number attached.
Symptom: good agents score zero. Measurement: read 20 zero-reward episodes. If five of them did the job in an unexpected order or wording, the check is too narrow. Fix: check the outcome, not the path. Compare the final status of the order, not the exact sequence of tool calls.
The opposite error is a check that is too loose, for example one that only checks the goal field. The collateral rule above closes that gap.
Chapter 3
Build a task generator with a difficulty knob, and place each task where an agent can still learn from it
You write 40 retail tasks by hand. It takes a week. After a day of RL, the agent solves 38 of them. Did it learn to handle customers? Or did it memorise 40 stories? You cannot tell, and you have no new tasks to find out.
The fix is to stop writing tasks and start writing the thing that writes tasks. A task generator is a program that takes a seed (a number that fixes the random choices) and a difficulty, and returns a task with its goal state. Think of a worksheet printer for a maths teacher: one template, endless fresh pages.
A template is a task with blanks: “Cancel the {item} order.” The generator fills the blanks from the world it built: a real item name, a real order id. A seed makes this repeatable. Seed 7 always gives the same world and the same task, so a failure can be replayed.
Real projects do this at scale. Reasoning Gym (Stojanovski et al., arXiv 2505.24760) offers over 100 data generators and verifiers across areas that include algebra, arithmetic, graph theory and logic. Its complexity is adjustable, so one generator can keep making fresh tasks at a chosen difficulty.
τ2-bench (Barres et al., arXiv 2506.07982) builds its tasks from atomic components with controlled complexity. An atomic goal is one small change to the world: cancel one order, refund one order, change one address. A task is a list of atomic goals. More goals, harder task.
Why does stacking goals make a task harder? Suppose the agent gets each atomic goal right with probability 0.9, and the goals are independent. All of them must be right, so you multiply.
One goal: 0.9. Three goals: 0.9 × 0.9 × 0.9 = 0.729. Five goals: 0.590. Eight goals: 0.430. The numbers are illustrative, but the shape is real. Difficulty grows smoothly with the count, and you pick the count.
Other knobs work the same way. Add distractor orders that look like the target. Add a constraint such as “the one placed in March”. Add a tool the agent must discover. Each knob is a parameter of the generator, and each has a measured effect on the pass rate.
A task the agent always solves teaches nothing: every attempt earns the same reward. A task it never solves also teaches nothing, because every attempt earns zero. The learning happens where the agent sometimes wins. Chapter 8 turns this into a number. For now, remember the word curriculum: an ordering of tasks that keeps the agent near that middle as it improves.
The panel shows a task that your seed and goal count built. The curve shows the expected pass rate for an agent with the per-goal skill you set. Move the goal count along the curve and watch the task grow.
The task text is generated live in your browser by a small seeded generator, so the same seed and goal count always give the same task. The pass rate is skillgoals, which assumes independent goals: illustrative. The 20% to 80% learning band is a rule of thumb for this lesson, not a number from a paper.
pythondef make_task(seed, n_goals, n_distractors=2): rng = random.Random(seed) orders = [make_order(rng) for _ in range(n_goals + n_distractors)] goals = [] for order in rng.sample(orders, n_goals): # distinct orders, no overlap kind = rng.choice(["cancel", "refund", "change_address"]) goals.append(atomic(kind, order, rng)) task = Task(orders=orders, goals=goals, text=phrase(goals)) task.reference = solve(task) # a known-good action list return task # Gate: keep a task only if its own reference solution earns reward 1.0 env = RetailEnv(); env.load(task) assert replay(env, task.reference) == 1.0
The last two lines are the most important. Every generated task ships with a reference solution, and the environment must accept it. This gate is the first guard against a bad task.
Symptom: one template has a pass rate of exactly 0% across every model you try. Measurement: replay the reference solution. If it fails, the task is unsolvable, usually because the generator picked a goal that conflicts with the world, such as refunding an order that is already cancelled. Fix: add the replay gate and drop every task that fails it.
A second symptom is quieter. The pass rate rises fast and never falls with difficulty. Measurement: count distinct template ids in a batch. If three templates make 90% of tasks, the agent is learning three tricks. Fix: sample templates evenly and track coverage.
Chapter 4
Pick the right checker for each task family, and harden it until the cheapest way to score is to do the work
A coding agent gets a bug report: “total([]) returns None, it should return 0.” The reward is 1.0 when the tests pass. The agent opens the test file, deletes the failing test, and runs the suite. Everything passes. Reward 1.0.
That is not a clever trick. It is the system working as written. The verifier is the program that turns an episode into a reward, and this one pays for “tests pass” instead of “the bug is fixed”. Those are not the same thing, and the gap between them is where agents live.
State match. Compare the world with a goal, as in Chapter 2. It fits tool-use tasks where the work leaves a footprint. τ-bench and AppWorld both work this way.
Unit tests. Run code the agent wrote or changed, and count passing tests. It fits software tasks. SWE-Gym gives every one of its 2,438 instances a codebase with an executable runtime and unit tests. The runtime is what makes the verdict trustworthy: the tests actually run.
Rubric graders. A rubric is a checklist of yes or no questions about the output. A judge model answers each question and the answers are added up. It fits text with no footprint. Here is a small one for a support reply.
json{"criteria": [ {"id": "asked_for_id", "check": "The agent asked for the order id before acting.", "weight": 1}, {"id": "stated_policy", "check": "The reply states the 30-day refund rule correctly.", "weight": 2}, {"id": "no_invention", "check": "The reply promises nothing the tools did not do.", "weight": 2} ]}
Each check is a plain yes or no. Weighted answers add up: 1 + 2 + 2 = 5 points, so a reply that misses only the policy line scores 3 of 5, which is 0.6. Small, specific checks are far more stable than one question such as “Is this reply good?”.
Exact answer match. The simplest family fits tasks with one right answer, such as arithmetic or logic puzzles. Reasoning Gym ships its generators together with verifiers. A verifier of this kind parses the final answer, tidies the format so that “1,000” and “1000” agree, and compares. Its failure is a false zero: a right answer in an unexpected format. Measure it by sampling zero-reward episodes. Fix it by normalising, or by accepting a short list of equivalent forms.
RL is an optimiser. It finds the lowest-effort path to the highest reward. So for each verifier, ask one question: what is the cheapest thing that scores 1.0 without the real work?
For visible unit tests the answers are easy to list. The agent can delete the failing test. It can special-case the expected value (if xs == []: return 0 only for the test input). It can fix the bug and quietly break something else that has no test. For a rubric, a long and confident reply often scores well even when it promises things that never happened.
The fix for each is a hardening step. Run tests from a clean checkout so edits to test files are thrown away. Keep hidden test cases the agent never sees. Add regression tests that passed before and must still pass. Try each switch below and watch the exploits stop paying.
The task is to fix total([]). Only the first behaviour is honest. Each bar is the reward the verifier pays. Turn on the switches and see which exploits stop paying.
Illustrative rules, one per switch. Clean checkout discards edits to test files. Hidden cases catch hard-coded outputs. Regression tests catch breakage elsewhere. A long confident reply with no code change fails the tests whatever you switch.
pythondef test_reward(repo, patch, hidden, regression): box = fresh_checkout(repo) # tests the agent never touched box.apply(patch.only_files("src/")) # edits to tests/ are dropped fixed = box.run(hidden).all_passed # cases the agent never saw intact = box.run(regression).all_passed # behaviour that must not break return 1.0 if fixed and intact else 0.0
You will not think of every exploit. So measure. Symptom: the training reward climbs, but a held-out check by humans does not. Measurement: sort episodes by reward, and read the top ones first, not the bottom. The best-scoring episodes are where exploits hide. Fix: every exploit you find becomes a new test or a new rule, and you re-score old episodes with it.
Also keep the reward simple. A 0 or 1 that you trust beats a clever blend of six sub-scores that the agent can trade against each other. Add partial credit only for a reason, and Chapter 6 gives the reason.
Chapter 5
Add a simulated customer to the world, and measure whether the agent succeeds every time, not just once
So far the task arrived whole: “Cancel W-1042.” Real customers do not talk like that. Yara writes, “I want to send the kettle back.” She does not know the order id. She remembers her email. She will refuse store credit because she wants the money on her card. The agent must find all this out by talking.
To train on tasks like this, the environment needs a person in it. People are slow and expensive, so we use a user simulator: a language model that plays the customer. It is the clerk’s customer, played by an actor who has read the script.
τ-bench (Yao et al., arXiv 2406.12045) tests agents on tool-agent-user interaction. Each conversation runs between a simulated user played by language models and an agent that has domain API tools and written policy guidelines. At the end, the database state is compared with an annotated goal state. So the pieces from earlier chapters all appear: a state check, tools, a task, and now a talking user.
The user needs a hidden scenario: private facts and a private goal that the agent cannot read. The simulator is told the scenario and told to reveal it only when it makes sense, as a person would.
json{"user_instruction": "You are Yara. You want to return the blue kettle. You do not remember the order id, only your email, yara@example.com. You are in a hurry. If offered store credit, refuse: you want the refund on your original card.", "goal_state": {"W-1042": {"status": "returned", "refund_to": "card"}}}
The loop now has two speakers. The agent can call a tool, which the environment answers, or reply, which the user answers.
pythondef run_episode(agent, user, env, scenario): msg = user.start(scenario) # the user speaks first while True: action = agent.act(msg) # a tool call or a reply if action.tool != "reply": msg = env.call(action) # tool result returns to the agent else: msg = user.respond(action.text) # the simulated user answers if user.is_done(msg): break return env.check_state(scenario.goal_state) # the score reads the database, not the chat
Look at the last line. The score still reads the world. The user is there to make the task realistic, not to grade. That separation keeps a charming or confused simulator from changing the reward.
τ2-bench (Barres et al., arXiv 2506.07982) goes further. In many real tasks, such as phone support, the user also acts: she restarts the router while the agent guides her. The paper calls this a dual-control environment, with a Telecom domain where both agent and user use tools in a shared world. Its user simulator is tightly coupled with the environment, so its behaviour is constrained by its tools and observable states. The authors report that performance drops significantly when agents move from a no-user setting to dual control.
A customer does not care that your agent could have solved her problem. She cares that it does, every time. So τ-bench adds a metric called pass^k (“pass hat k”). Here is the idea in this lesson’s own words: run each task k times, and count the task only if all k runs succeed.
The paper reports that state-of-the-art function-calling agents such as gpt-4o succeed on fewer than 50% of tasks, and are quite inconsistent: pass^8 is under 25% in retail. A system that wins half the time looks fine once and falls apart when repeated.
Worked example with three tasks. The agent succeeds on task A every time (p = 1.0), on task B 80% of the time, on task C half the time. With one try, it averages (1.0 + 0.8 + 0.5) / 3 = 0.767. With four tries in a row, we need all four right: (1.0 + 0.84 + 0.54) / 3 = (1 + 0.4096 + 0.0625) / 3 = 0.491. With eight: (1 + 0.168 + 0.004) / 3 = 0.391. The same agent, three very different grades.
Warm is pass@k: at least one of k tries works. Purple is pass^k: all k tries work. Choose how the agent’s skill is spread, then drag k.
Illustrative skill profiles, not measured. Flat: every task has a per-try success of 0.70. Mixed: 40% of tasks at 1.00, 30% at 0.70, 30% at 0.20 (a mean of 0.67). Each try is treated as independent, and pass^k is the share of tasks where all k tries succeed, as defined in this chapter.
Symptom: the agent’s score changes a lot when you swap the user model. Measurement: run the same agent against two simulators on the same 50 scenarios and compare. A gap of ten points means the simulator is part of what you are measuring. Also read transcripts for leaks, such as the user stating the whole goal in the first line.
Fix: tie the simulator to the scenario, give it only what a real user would know, restrict what it can claim, and keep the reward on the database. Re-check with a second simulator each time you change it.
Chapter 6
Make a twelve-step job teachable, by paying for progress, bounding the cost and saving the world between steps
Yara closes her account. The job has twelve steps: find her three open orders, cancel them, refund two, remove her saved card, export her data, and confirm by email. An agent does eleven steps perfectly and slips on the last one.
With a 0-or-1 reward, that agent earns 0. So does an agent that did nothing. Training sees no difference between them. This is the central trouble with a long horizon, the number of steps before the job ends. Success needs a long chain of correct steps, so success is rare, so most rewards are zero, so there is little to learn from.
Many RL methods train from a group of attempts at the same task. They compare each attempt with the group’s average. An attempt that did better than average gets pushed up. If all eight attempts score 0, every attempt equals the average, and nothing is pushed anywhere. The gradient, the direction the weights should move, is zero.
So a long-horizon environment must create differences between attempts. There are three tools, and each has a cost.
A milestone is an intermediate state that a checker can confirm: “all three orders are cancelled”, “the card is removed”. Partial credit pays a fraction of the reward for each milestone reached. Eleven good steps now earn something.
The risk is farming. If milestones pay, an agent may collect the easy ones and skip the hard last step. One common design caps the partial part: reward = 0.5 × (milestones reached / total) + 0.5 × (final goal state matches). Full reward needs the real finish.
A budget is a limit on cost: steps, tokens or dollars per episode. A timeout is a limit on wall-clock time, per tool call or per episode. Without them, one stuck agent can hold a training worker for hours. Both end the episode as truncated, the flag from Chapter 1.
Set the budget with care. If the reference solution takes 12 steps and the budget is 10, no agent can ever succeed. A common starting rule is one and a half to two times the reference length. That rule is a convention, not a law, so check it by measuring how often good attempts run out of steps.
A tool timeout should come back as an observation, such as {"error": "tool timed out after 30 s"}, so the agent can react. The 30 s is a value you choose.
A snapshot is a saved copy of the whole world at one step: the database, the step counter and the random-number state. You can restore it later, and the world is exactly as it was.
pythonsnap = env.snapshot() # {"db": ..., "t": 8, "rng": ...} for _ in range(8): env.restore(snap) # same world, same step 8 rollout(agent, env) # eight different endings
Snapshots save compute. If an agent slips at step 9 of 12, a retry from the snapshot at step 8 costs 4 steps, not 12, so 8 steps (67%) are saved. They also make fair comparisons: eight continuations from the same starting state differ only in what the agent chose.
Each row is one attempt at the twelve-step job. A bar runs until the agent slips or the budget ends. Switch the reward style and watch how many different rewards the eight rows give.
Illustrative simulation with seeded random slips, so the eight attempts are the same on every visit. A milestone sits every second step (6 in all). An attempt ends at its first slip or when the budget ends, and a step that is not reached earns nothing.
Milestone farming. Symptom: training reward rises, but the share of attempts that reach the final goal does not. Measurement: track both numbers on held-out tasks. Fix: cap the partial credit, as above.
Budget too tight. Symptom: success is exactly zero even for strong models. Measurement: replay the reference solution under the budget. Fix: raise the budget until the reference passes with room to spare.
Snapshot drift. Symptom: a restored world behaves differently from the original run. Measurement: restore, replay the same actions, and compare the end state byte by byte. Fix: put every source of randomness inside the snapshot. Chapter 7 returns to this.
Chapter 7
Run many sealed, repeatable environments side by side, and keep the expensive GPU from waiting on a slow tool
Your training run wants 4,096 attempts in flight at the same time. Each attempt needs its own world: its own database, its own files, maybe its own desktop. If two attempts share one database, one agent’s cancelled order shows up in another agent’s world, and the scores are junk.
So each world lives in a container: a sealed box on a machine, with its own files and programs, cut off from its neighbours. Think of a lunchbox. Many fit in one bag, and the sandwich in one box never touches the fruit in another. A container starts from an image, a frozen copy of the world built once and started many times.
A trainer cannot learn a new interface for every world. AgentGym (Xi et al., arXiv 2406.04151) describes its platform with this goal: diverse environments exposed through a uniform format, in real time and concurrently. The practical shape is a small server in each container with two doors.
httpPOST /reset {"seed": 7, "task_id": "cancel-w1042"} -> {"obs": {"user": "Please cancel my kettle order, W-1042."}, "info": {...}} POST /step {"action": {"tool": "get_order", "args": {"order_id": "W-1042"}}} -> {"obs": {...}, "reward": 0.0, "terminated": false, "truncated": false}
This is the Chapter 1 contract, carried over a network. The trainer holds a list of such servers. It can talk to a retail world and a code world in the same loop, because both answer the same two calls. Our lesson on sandbox fleets shows how this works for whole desktops, and agent environments covers what such worlds contain.
OSWorld shows why the image matters. Each of its 369 tasks carries a setup configuration that builds the starting state, plus an execution-based script that checks the result. The image plus the config is what makes the task start the same way each time.
An environment is deterministic if the same seed and the same actions always give the same results. Without that, you cannot replay a failure, compare two agents fairly, or trust a score. Almost every hidden source of randomness hides in plain sight.
The clock is the usual culprit. Suppose a task says “cancel the orders placed this month” and the code calls datetime.now(). At midnight on the last day of the month, the goal changes under a running agent. The fix is to give the environment its own fake clock, set by the seed.
Other sources are the same in spirit: random-number calls without a seed, unordered sets, generated ids, network calls to live services, and threads that race. Test for all of them at once with a replay check, run nightly on every task.
pythondef replay_matches(env, task_id, seed, actions): hashes = [] for _ in range(2): env.reset(seed=seed, options={"task_id": task_id}) for a in actions: env.step(a) hashes.append(env.state_hash()) # one fingerprint of the whole world return hashes[0] == hashes[1]
The model runs on a GPU, the costly part. The tools run in containers, the cheap part. A single attempt alternates between them: the model thinks for g seconds, a tool runs for e seconds, and the GPU sits idle for those e seconds.
Worked example, with illustrative numbers. Let g = 2 s and e = 6 s. One attempt keeps the GPU busy for 2 of every 8 seconds, which is 25%. Run four attempts at once and the GPU always finds one ready: 4 × 2 / 8 = 100%. More containers fixed the idle time without touching the model.
A 12-step episode needs 12 × 2 = 24 s of GPU. At full use that is 3600 / 24 = 150 episodes an hour per GPU. A cold start adds to each episode: if a fresh container takes 20 s to boot, the episode takes 12 × 8 + 20 = 116 s instead of 96 s. A warm pool, containers booted ahead of time, removes that cost from the loop.
Bars show how busy one GPU is as you add parallel environments. The marked bar is your setting. Slow tools and cold starts make you need more containers.
Illustrative model: the model takes g = 2 s per step, an episode has 12 steps, a cold start takes 20 s, and one GPU serves rollouts without queueing. GPU use is min(1, parallel × 12 × g / episode time). The chosen number of environments doubles with each notch.
Symptom: the dashboard shows GPUs at 30% while hundreds of containers are alive. Measurement: log, for each step, the time spent in the model and in the tool. If tool time dominates, add environments or move slow tools off the critical path. Fix: more parallel containers, a warm pool, and a timeout on every tool call so one hung container cannot stall a batch.
Another symptom: scores jump between two runs of the same code. Measurement: the nightly replay check above. Fix: find the clock, the generated id or the race, and move it under the seed.
Chapter 8
Score your own tasks with three measurements, and keep the ones that can still teach the agent something
You have 24 tasks and one week of training time. Some are broken. Some are trivial. Some are too hard today but fine next month. How do you know which are which, before you spend the week?
You run the current agent on every task, several times each, and read the results. Three measurements do most of the work.
Run each task k times, say k = 8, and count the wins. The pass rate p of a task is wins divided by k. Plot p for every task and you get the spread: how many tasks sit near 0, near 1 and in between.
A warning on noise. Eight tries is a small sample. If a task’s true pass rate is 0.10, the chance of zero wins in eight tries is 0.98 = 0.43. So “0 of 8” does not prove the task is impossible. This is why Measurement 2 exists.
More tries shrink the noise. With 32 tries and a true pass rate of 0.10, the chance of zero wins falls to 0.932 = 0.034. So 32 tries on a few suspect tasks tell you far more than 8 tries on all of them. Spend the extra runs where the answer is unclear.
The solvable rate is the share of tasks that something can solve: the generator’s reference solution, a stronger model, or a person. Run the reference through the environment. If it earns 1.0, the task is solvable. If not, the task or its checker is broken. Chapter 3’s gate does this at generation time, and you repeat it as the world changes.
Now every task with p = 0 falls into one of two boxes. Reference passes: the task is too hard right now, so keep it for later. Reference fails: it is broken, so fix or drop it. A third group, tasks with p = 1, are retired: nothing is left to learn.
Recall Chapter 6. A group of attempts teaches only if the rewards differ. For a task with pass rate p and a group of 8, the chance that the group mixes wins and losses is 1 − p8 − (1 − p)8. Read the first term as “all eight win” and the second as “all eight lose”. Take them away and what is left is the chance of a mixed group.
Worked values. At p = 0.5: 1 − 0.0039 − 0.0039 = 0.992. At p = 0.1: 1 − 0 − 0.430 = 0.570, because 0.18 is only 0.00000001. At p = 0.02: 1 − 0 − 0.851 = 0.149. At p = 0 or 1: exactly 0. Tasks near the middle almost always give a usable group. Tasks near the edges rarely do.
Each dot is one task, placed at its pass rate. The teal band is what you keep. Drag the two ends and watch how much signal the kept tasks carry.
Illustrative: 24 invented pass rates, the same on every visit, with 4 tasks at 0%, 3 at 100% and 17 between. The signal for a task is 1 − p8 − (1 − p)8, the chance that a group of 8 attempts mixes wins and losses, and the read-out averages it over the tasks you keep.
pythondef audit(tasks, agent, k=8): rows = [] for t in tasks: wins = sum(run(agent, t, seed=s) == 1.0 for s in range(k)) p = wins / k rows.append({"id": t.id, "p": p, "signal": 1 - p**k - (1 - p)**k, "ref_ok": run_reference(t) == 1.0}) return rows # keep: ref_ok and 0 < p < 1. fix: not ref_ok. park: ref_ok and p == 0. retire: p == 1.
Symptom: a task looks well balanced at p = 0.5, yet training on it never helps. Measurement: check whether the wins come from luck, such as a coin-flip answer with two options, rather than skill. Re-run with more seeds and look at why attempts win. Fix: change the task so luck cannot carry it.
Also remember that the agent improves. A band that works today goes stale in a week. Re-run the audit on a schedule and move tasks between the boxes. Reasoning Gym’s adjustable complexity and τ2-bench’s controlled complexity exist for this reason: you turn the knob instead of rewriting the task.
Chapter 9
Find the shortcuts, leaks and flaky checks in your own environment before the optimiser finds them for you
A coding agent is asked to fix a bug. It lists the files in its container and spots check.py, the grader, sitting next to the code. It opens it. The grader ends with return score. The agent rewrites that line to return 1.0. Reward 1.0, two steps, no bug fixed.
Nobody taught the agent to cheat. RL rewards whatever scores. If a two-step route to 1.0 exists and a nine-step honest route exists, the two-step route wins, and training makes it more likely each round. This is reward hacking: scoring through a gap in the environment instead of through the intended work.
Shortcuts. A tool is more powerful than the task needs. A refund_any tool with no policy check lets the agent skip the business rules. Fix: put policy checks in the tool code, not in the prompt.
Leaks. The answer is somewhere the agent can read. It might be a goal file in the container, a debug field in an API response ("expected_refund": 34.50), or the benchmark’s own solutions on the open web. Fix: keep goals and graders outside the container, strip debug fields, and cut the network or allow only a short list of sites.
Writable graders. The checker lives in the agent’s world and the agent can edit it. Fix: run verification in a separate process, on a fresh copy, after the episode ends.
Flaky checks. The same actions get different verdicts on different runs. This is subtle because it looks like noise, not cheating. Suppose a check wrongly passes 25% of wrong answers. A policy that submits again and again will see a pass in about four tries (1 / 0.25), and RL will teach it to do exactly that.
An agent trained to maximise reward drifts toward whichever route to 1.0 is cheapest in steps and in risk. So think of your environment as a map of routes. Each hole you leave open adds a route. Each hole you patch removes one. The device below shows five routes with their step costs. Patch the holes and watch the cheapest route change.
Each bar is a way to reach reward 1.0, with its cost in steps. Red routes are hacks. Press a switch to patch that hole.
Illustrative costs in steps: honest work 9, read the goal file 3, rewrite the grader 2, abuse the unchecked tool 4, look the solution up online 4, retry a flaky check about 4 (1 / 0.25, the false-pass rate assumed here). A patched route is removed. The point is the ordering, not the exact counts.
You cannot list every hole by thought alone. Use three measurements.
Read the top. Sort episodes by reward, and read the best ones (Chapter 4). A short, high-reward episode on a hard task is a red flag.
Plant canaries. A canary is a fake secret, such as a unique string in a file the agent should never open. If it shows up in any action or observation, the agent got in.
pythonCANARY = "zk-7f3a91" # planted in /task/answers.json, which no tool should expose def leaked(trajectory): return any(CANARY in str(s.action) or CANARY in str(s.obs) for s in trajectory)
Replay for flakes. Take 20 finished trajectories and replay each 5 times in fresh containers. If 4 of the 20 produce mixed verdicts, the flake rate is 4 / 20 = 20% of trajectories. Any number above zero means the check, or the world, depends on something unseeded (Chapter 7).
Worked example of a canary hunt. You run 1,000 episodes and the canary string shows up in 37 of them. That is 3.7%. You read those 37 and find that all of them called get_order, and the tool’s reply held a debug field copied from the goal table. One line of code strips the field. You re-score the earlier runs and find the 37 episodes had inflated the training reward. The canary found the leak. The re-score measured the damage.
Symptom: training reward climbs while a held-out human check does not. Measurement: the three audits above. Fix: patch the hole, then re-score the old episodes with the patched environment, so you know how much of the earlier gain was real. Keep a small suite of known exploits as tests that must always score 0.
Chapter 10
Keep the numbers, the checklist and the core code for the next time you build a world for an agent
An environment is a contract: reset builds a seeded world and returns an observation, and step takes an action and returns an observation, a reward and the terminated and truncated flags. Where the task changes the world, score the state, not the words, and add a collateral rule. Generate tasks from seeded templates with a difficulty knob, and gate each one with a reference solution. Pick the verifier for the task family (state match, unit tests, rubric) and harden it. Add a simulated user when the task is a conversation, and measure reliability with pass^k. For long jobs, pay for milestones with a cap, bound cost with budgets and timeouts, and save snapshots. Run worlds in containers behind one uniform door, make them deterministic, and keep the GPU busy. Audit pass rates, solvable rate and learning signal. Then hunt for the cheap routes: leaks, writable graders, unchecked tools and flaky checks.
| What | Number | Why it matters |
|---|---|---|
| Gymnasium step return | 5 values: observation, reward, terminated, truncated, info | The standard contract; done was split in v0.26 |
| τ-bench, retail | top agents under 50% of tasks; pass^8 under 25% | One win is not reliability |
| τ2-bench | dual-control Telecom domain | Users who act too make tasks harder |
| AppWorld | 9 apps, 457 APIs, about 100 users, 750 tasks | A rich world with state-based tests and collateral checks |
| AppWorld, GPT-4o | about 49% normal, about 30% challenge | Difficulty tiers inside one benchmark |
| OSWorld | 369 tasks; humans over 72.36%; best model 12.24% | Each task has a setup config and a check script |
| SWE-Gym | 2,438 instances; up to 19% absolute gain; 32.0% and 26.0% on two SWE-Bench sets | A training environment is a result in its own right |
| Reasoning Gym | over 100 generators and verifiers, many areas | Procedural tasks with adjustable difficulty |
| Five goals at 0.9 each (illustrative) | 0.95 = 0.590 | Difficulty compounds with task length |
| Mixed group chance (illustrative) | 1 − p8 − (1 − p)8; 0.992 at p = 0.5 | Tasks near the middle carry the learning signal |
| GPU fill (illustrative) | g = 2 s, e = 6 s: 4 environments fill 1 GPU | Parallel worlds hide tool latency |
reset(seed) give the same world twice? Run the replay check.pythonclass RetailEnv: max_steps = 12 def reset(self, *, seed=None, options=None): self.rng = random.Random(seed) # Ch 7: every random call goes through this self.task = make_task(seed, options["n_goals"]) # Ch 3: seeded template self.db, self.t = self.task.world(), 0 self.start = copy(self.db) # for the collateral rule return {"user": self.task.text}, {} def step(self, action): self.t += 1 obs = run_tool_with_timeout(self.db, action, seconds=30) # Ch 6: policy lives in the tool code (Ch 9) terminated = action["tool"] == "reply" truncated = self.t >= self.max_steps and not terminated reward = self.score() if (terminated or truncated) else 0.0 return obs, reward, terminated, truncated, {"t": self.t} def score(self): # Ch 2 and 4: state, not words done = state_match(self.start, self.db, self.task.goal) miles = self.task.milestones_hit(self.db) / self.task.n_milestones return 0.5 * miles + 0.5 * done # Ch 6: capped partial credit def snapshot(self): return {"db": copy(self.db), "t": self.t, "rng": self.rng.getstate()} def restore(self, s): self.db, self.t = copy(s["db"]), s["t"]; self.rng.setstate(s["rng"])
Wrap this class in the two-door server from Chapter 7, run it in a container from a frozen image, and audit it with Chapter 8’s function. That is a training environment.
Environments are the bridge between a model and a skill. The model supplies guesses. The environment supplies the truth about those guesses, at scale. Everything in the post-training stack that learns from verifiable outcomes, such as RL on code, tools and computer use, relies on a world that can say “yes” or “no” and mean it.
reset, step, terminated and truncated.Ready to test yourself? Open Teach and explain one chapter on a whiteboard, or open Sparks for single-idea cards. Then go back to the grader at the top and try all three agents again, now knowing what each grader can see.