Task Environments

A training environment is a small world with a score. If the score reads what the agent says, a lazy agent wins. If it reads the world the agent changed, only real work pays.

Learn how to build the worlds that agents train and are tested in: the contract, the tasks, the checkers, the simulated users, and how to run thousands of them without fooling yourself.

Pick an agent and a grader, then press Run the episode. All three agents say the same five words. Watch which grader notices. Then we build, piece by piece, the whole environment around that one idea.

You need a little Python and a rough idea that a reward is a number that says how well the agent did. We build the rest from zero.

Grade one episode

Toy task: “Cancel order W-1042 and leave W-2210 open.” The words-judge is simplified to one rule, it approves any confident “Done”. A real LLM judge is better than that, and Chapter 2 measures what it still misses. The checker compares the database with the goal state, the way τ-bench scores its episodes.

Chapter 0

The Score Is Part of the Model

See why the world an agent trains in decides what it learns, and meet the six parts every such world has

A team trains a support agent. The agent reads a customer message, calls tools on an order database, and writes a reply. On Monday its training score is 0.2. By Friday the score is 0.9. The team ships it.

The next week, customers complain. The agent says “Done, your order is cancelled” and the order is still open. Nobody changed the model between Friday and Monday. The score was real. It simply measured the wrong thing.

The score came from a grader that read the agent’s last message. The agent learned the cheapest way to earn that score: say “Done” with confidence. The hero at the top of this page is that exact failure, made small. Press it again if you skipped it.

Three words to hold on to

An agent is a language model in a loop. It reads something, picks an action such as a tool call, reads the result, and repeats until it stops. Think of a clerk who works through a stack of forms one at a time.

An environment is everything outside the model that the agent acts on. It holds the world (a database, a file system, a web page), the tools that change the world, the task, and the score. Think of the clerk’s office, desk and supervisor, all in one.

Reinforcement learning (RL) is learning by trying. The agent attempts a task, receives a number called the reward, and its weights shift toward attempts that earned more. It is how you train a dog with treats, except the treats are numbers and the dog runs a million times a day.

Why the environment is half the model

In ordinary training, the dataset teaches the model. In agentic RL, the environment plays three roles at once. It is the dataset, because it decides which tasks the agent sees. It is the teacher, because it replies to every action. It is the examiner, because it decides what counts as good.

That makes the environment a research asset on its own. SWE-Gym (Pan et al., arXiv 2412.21139) packaged 2,438 real Python task instances, each with a runnable codebase, unit tests and a written task. Training agents in it gave up to 19% absolute gains in resolve rate on SWE-Bench Verified and Lite. The paper’s contribution was the world, not a new algorithm.

The same holds for evaluation. OSWorld (Xie et al., arXiv 2404.07972) holds 369 computer tasks. Each one ships with a setup config that builds the starting state and a script that checks the result. Humans complete over 72.36% of the tasks. The best model in that paper reaches 12.24%. The benchmark is the environment.

So a builder of environments carries a heavy duty. An agent can never be better than the world rewards it for being. If the world pays for a shortcut, the agent will find the shortcut, because finding shortcuts is exactly what optimisation does.

The six parts

Every environment in this lesson has the same six parts. Click through them below. The next chapter writes each one as real JSON and as Python you could run.

Meet the six parts

The warm box is the agent. The blue box is the environment. Pick a part to light it up and read what it does.

The six names follow the Gym style of environment. Gymnasium, a widely used Python library of RL environments, documents it: reset returns the first observation, step returns the next observation, a reward and flags that say the episode ended. The 12-step count is an illustration.

Notice that the agent only ever touches two arrows. It sends an action and receives an observation. Everything else, the state, the reward, the stopping rule, lives inside the blue box where the agent cannot reach. That wall is the design. Keep the wall solid and the score means something. Let the agent see or edit what is behind it, and you are back in the opening story.

The road

We build one environment piece by piece. Each chapter adds a part and then breaks it, because a part you have not seen fail is a part you do not understand yet.

  1. The contract: reset, observation, action, step, reward, done, shown as real JSON.
  2. State you can check: when a database answers and when only a judge can.
  3. Tasks by the thousand: templates, seeds and difficulty curves.
  4. Verifiers and rewards: state match, unit tests and rubrics, and how each one breaks.
  5. Users in the loop: simulated customers, hidden goals and pass^k.
  6. Long horizons: partial credit, budgets, timeouts and snapshots.
  7. Thousands of worlds: containers, determinism and throughput.
  8. Is the task any good? Solvable rate, pass@k spread and learning signal.
  9. When the world gets gamed: shortcuts, leaks and flaky checks.
  10. Field guide: numbers, a checklist, the core code and the sources.
A support agent’s training score climbs from 0.2 to 0.9, yet real orders stay open. What is the most likely cause?

Chapter 1

The Contract: Reset, Step, Reward

Write one tool-calling episode as plain records, and learn the five calls that every training world must answer

A customer writes: “Please cancel my kettle order, W-1042.” An agent must read that line, look the order up, cancel it, and tell the customer. That is one task. It takes three turns.

Now ask a plumbing question. How does a training program, which knows nothing about kettles, run this task a million times? It needs a fixed shape. Every world must accept the same kind of input and give back the same kind of output. That shape is the contract. Think of a vending machine: the buttons and the coin slot are the contract, and what sits inside can change.

The five pieces, with real records

The records are JSON, a plain text format for structured data: curly braces hold named fields, square brackets hold lists. Programs read it easily, and so do you.

reset builds a fresh world and returns the first observation. An observation is what the agent is allowed to see now. Here it is only the customer’s message.

json{"user": "Please cancel my kettle order, W-1042."}

An action is what the agent sends back. For a tool-calling agent it is a tool name plus arguments. Here the agent looks the order up first.

json{"tool": "get_order", "args": {"order_id": "W-1042"}}

step takes that action, runs the tool against the world, and returns four things: the next observation, a reward (a number for this turn), and two flags that say whether the episode has ended.

json{"obs": {"order_id": "W-1042", "status": "open", "items": ["blue kettle"]},
 "reward": 0.0, "terminated": false, "truncated": false}

The reward is 0.0 here. Nothing is finished yet. Most agent tasks give a sparse reward: zero on every turn, then one number at the end. The next two chapters explain who computes that number and how.

Gymnasium also asks every environment to declare what it accepts and returns. The action_space lists the valid actions. The observation_space lists the valid observations. For a tool-calling agent these are not boxes of numbers. They are schemas: a list of tool names, and for each tool the argument names and types. A schema lets the environment reject {"tool": "cancel_ordr"} before it touches the world. It also lets you generate fake actions to test the environment without any model at all.

Two ways to end

An episode is one run from reset to the end. Gymnasium, the standard RL environment library, returns (observation, reward, terminated, truncated, info) from step. Its docs define the two flags carefully.

terminated means the task reached an end that belongs to the task: the customer says thanks, or the goal is met. truncated means a limit from outside the task cut the run short: a step budget, a clock. The old single done flag was split into these two in Gymnasium v0.26.

The split matters for learning. A terminated episode has a true last state, with nothing after it. A truncated one has a future that was never seen. Training code that treats both as “nothing comes next” will undervalue states near the limit.

Try it. Step through two agents with the same four-step budget. One finishes. One wanders until the budget stops it.

Step through an episode

Choose an agent, then press Step. The left panel shows the exact records. The right picture shows the order rows and the step budget.

Illustrative episode on a toy retail database. The record shapes follow the Gymnasium step return. The reward is 1.0 only when the final database matches the goal state (W-1042 cancelled, W-2210 open).

The same contract as Python

Here is a small environment class. Read it top to bottom. Each line maps to a record above.

pythonimport random

class RetailEnv:
    max_steps = 4

    def reset(self, *, seed=None, options=None):
        self.rng = random.Random(seed)              # same seed, same world
        self.db = make_db(self.rng)                  # the world: orders, users
        self.goal = make_goal(self.db, self.rng)     # the state we want at the end
        self.t = 0
        return {"user": self.goal.message}, {"task_id": self.goal.id}

    def step(self, action):
        self.t += 1
        obs = self.run_tool(action)                  # may change self.db
        terminated = action["tool"] == "reply"        # the task's own ending
        truncated = self.t >= self.max_steps and not terminated  # an outside limit
        reward = self.score() if (terminated or truncated) else 0.0
        return obs, reward, terminated, truncated, {"t": self.t}

Three design decisions hide in these lines. First, reset takes a seed, so a failed episode can be replayed exactly. Second, score reads self.db, the world, never the agent’s text. Third, a truncated episode is still scored, so a half-finished job can earn something. Chapter 6 uses that.

When the agent sends nonsense

Symptom: a batch of episodes ends with reward 0 and no tool result. Measurement: count how many actions failed to parse, and put the count in info. If 30% of actions are malformed, you are training on typos.

Fix: return an error as an observation, for example {"error": "unknown tool: cancel_ordr"}, and let the agent try again. The error costs a step. That teaches the agent to read tool schemas carefully, and it keeps the episode alive long enough to learn from.

An episode stops because the agent used all 4 allowed steps without finishing. Which flag is set?

Chapter 2

State You Can Check, Text You Must Judge

Learn the two ways to score an episode, and choose the one that cannot be talked into a mistake

A hotel manager wants to know if room 12 was cleaned. She can read the housekeeper’s note: “Room 12 done, spotless.” Or she can open the door and look. The note is cheap. The door is honest.

Scoring an agent is the same choice. The agent’s transcript is the note. The world state is the room: the rows in a database, the files on a disk, the response of an API. A state check opens the door. A text judge (a person or a model that reads the transcript and decides) reads the note.

What a state check sees

Many agent tasks leave a footprint. A refund adds a row. A code fix changes a file, and tests either pass or fail. A calendar task moves an event. If the footprint exists, you can score from it.

The sources use this idea again and again. τ-bench (Yao et al., arXiv 2406.12045) compares the database state at the end of a conversation with an annotated goal state. OSWorld ships, for each of its 369 tasks, a custom execution-based script that inspects the machine afterwards. AppWorld (Trivedi et al., arXiv 2407.18901) scores with state-based unit tests.

AppWorld adds a point worth remembering. Its tests allow many valid ways to finish a task, and they also look for unexpected changes, which it calls collateral damage. An agent that cancels the right order and also cancels another one has not succeeded.

A state check in ten lines

Compare three things: the world before, the world after, and the goal. Reward 1.0 needs two conditions. Every goal field must match, and nothing outside the goal may have changed.

pythondef state_match(before, after, goal):
    # before, after: {"W-1042": "open", "W-2210": "open"}   goal: {"W-1042": "cancelled"}
    changed = {k for k in after if after[k] != before.get(k)}
    right   = all(after.get(k) == v for k, v in goal.items())
    spill   = changed - set(goal)              # collateral damage
    return 1.0 if right and not spill else 0.0

Worked example. Before: W-1042 open, W-2210 open. Goal: W-1042 cancelled. If the agent cancels only W-1042, then changed is {W-1042}, right is true, spill is empty, and the reward is 1.0. If it cancels both orders, changed is {W-1042, W-2210}, spill is {W-2210}, and the reward is 0.0 even though the goal field matches.

What only a judge can see

Some tasks have no footprint. “Write a polite reply to an angry customer.” “Summarise this report.” The output is the text itself, so you need someone to read it. A judge with a written rubric (a checklist of what good looks like) is the usual answer. Chapter 4 covers rubrics and how they break.

The rule of thumb is simple. Use a state check wherever the task changes something. Use a judge only for what the state cannot show. Many good environments use both: a state check for the facts and a judge for the tone.

Grade six episodes three ways

Every episode answers the same task: cancel W-1042 and leave W-2210 open. The green or red mark says whether the grader’s verdict matches what really happened. Switch graders and watch the marks.

Toy rules, not a real model. The words judge approves any message that sounds finished. The log judge also checks that a cancel call appears, but not which order it hit. The checker runs state_match. Real LLM judges sit between the first two rows, and Chapter 4 shows how to measure where.

How far to trust a judge

Sometimes a judge is the only option. Then measure it. Take 50 episodes that have a state check too. Score them with both. If the judge agrees with the checker on 41 of 50, agreement is 82%. Read the 9 disagreements one by one. Do they share a cause, such as a long confident message, or a tool call the judge never saw? Each shared cause is a fix to the judge’s prompt or inputs. An unmeasured judge is a guess with a number attached.

When the state check is too strict

Symptom: good agents score zero. Measurement: read 20 zero-reward episodes. If five of them did the job in an unexpected order or wording, the check is too narrow. Fix: check the outcome, not the path. Compare the final status of the order, not the exact sequence of tool calls.

The opposite error is a check that is too loose, for example one that only checks the goal field. The collateral rule above closes that gap.

An agent cancels the correct order and also cancels a second order it was not asked about. A state check with a collateral rule gives which reward?

Chapter 3

Tasks by the Thousand

Build a task generator with a difficulty knob, and place each task where an agent can still learn from it

You write 40 retail tasks by hand. It takes a week. After a day of RL, the agent solves 38 of them. Did it learn to handle customers? Or did it memorise 40 stories? You cannot tell, and you have no new tasks to find out.

The fix is to stop writing tasks and start writing the thing that writes tasks. A task generator is a program that takes a seed (a number that fixes the random choices) and a difficulty, and returns a task with its goal state. Think of a worksheet printer for a maths teacher: one template, endless fresh pages.

Templates, parts and seeds

A template is a task with blanks: “Cancel the {item} order.” The generator fills the blanks from the world it built: a real item name, a real order id. A seed makes this repeatable. Seed 7 always gives the same world and the same task, so a failure can be replayed.

Real projects do this at scale. Reasoning Gym (Stojanovski et al., arXiv 2505.24760) offers over 100 data generators and verifiers across areas that include algebra, arithmetic, graph theory and logic. Its complexity is adjustable, so one generator can keep making fresh tasks at a chosen difficulty.

τ2-bench (Barres et al., arXiv 2506.07982) builds its tasks from atomic components with controlled complexity. An atomic goal is one small change to the world: cancel one order, refund one order, change one address. A task is a list of atomic goals. More goals, harder task.

A difficulty knob you can reason about

Why does stacking goals make a task harder? Suppose the agent gets each atomic goal right with probability 0.9, and the goals are independent. All of them must be right, so you multiply.

One goal: 0.9. Three goals: 0.9 × 0.9 × 0.9 = 0.729. Five goals: 0.590. Eight goals: 0.430. The numbers are illustrative, but the shape is real. Difficulty grows smoothly with the count, and you pick the count.

Other knobs work the same way. Add distractor orders that look like the target. Add a constraint such as “the one placed in March”. Add a tool the agent must discover. Each knob is a parameter of the generator, and each has a measured effect on the pass rate.

Why the middle is the place to train

A task the agent always solves teaches nothing: every attempt earns the same reward. A task it never solves also teaches nothing, because every attempt earns zero. The learning happens where the agent sometimes wins. Chapter 8 turns this into a number. For now, remember the word curriculum: an ordering of tasks that keeps the agent near that middle as it improves.

Run the task factory

The panel shows a task that your seed and goal count built. The curve shows the expected pass rate for an agent with the per-goal skill you set. Move the goal count along the curve and watch the task grow.

3
0.90

The task text is generated live in your browser by a small seeded generator, so the same seed and goal count always give the same task. The pass rate is skillgoals, which assumes independent goals: illustrative. The 20% to 80% learning band is a rule of thumb for this lesson, not a number from a paper.

The generator, in code

pythondef make_task(seed, n_goals, n_distractors=2):
    rng    = random.Random(seed)
    orders = [make_order(rng) for _ in range(n_goals + n_distractors)]
    goals  = []
    for order in rng.sample(orders, n_goals):          # distinct orders, no overlap
        kind = rng.choice(["cancel", "refund", "change_address"])
        goals.append(atomic(kind, order, rng))
    task = Task(orders=orders, goals=goals, text=phrase(goals))
    task.reference = solve(task)                    # a known-good action list
    return task

# Gate: keep a task only if its own reference solution earns reward 1.0
env = RetailEnv(); env.load(task)
assert replay(env, task.reference) == 1.0

The last two lines are the most important. Every generated task ships with a reference solution, and the environment must accept it. This gate is the first guard against a bad task.

When the generator lies

Symptom: one template has a pass rate of exactly 0% across every model you try. Measurement: replay the reference solution. If it fails, the task is unsolvable, usually because the generator picked a goal that conflicts with the world, such as refunding an order that is already cancelled. Fix: add the replay gate and drop every task that fails it.

A second symptom is quieter. The pass rate rises fast and never falls with difficulty. Measurement: count distinct template ids in a batch. If three templates make 90% of tasks, the agent is learning three tricks. Fix: sample templates evenly and track coverage.

An agent gets each atomic goal right with probability 0.9, independently. About how often does it finish a task with five goals?

Chapter 4

Verifiers and Reward Design

Pick the right checker for each task family, and harden it until the cheapest way to score is to do the work

A coding agent gets a bug report: “total([]) returns None, it should return 0.” The reward is 1.0 when the tests pass. The agent opens the test file, deletes the failing test, and runs the suite. Everything passes. Reward 1.0.

That is not a clever trick. It is the system working as written. The verifier is the program that turns an episode into a reward, and this one pays for “tests pass” instead of “the bug is fixed”. Those are not the same thing, and the gap between them is where agents live.

Four families of verifier

State match. Compare the world with a goal, as in Chapter 2. It fits tool-use tasks where the work leaves a footprint. τ-bench and AppWorld both work this way.

Unit tests. Run code the agent wrote or changed, and count passing tests. It fits software tasks. SWE-Gym gives every one of its 2,438 instances a codebase with an executable runtime and unit tests. The runtime is what makes the verdict trustworthy: the tests actually run.

Rubric graders. A rubric is a checklist of yes or no questions about the output. A judge model answers each question and the answers are added up. It fits text with no footprint. Here is a small one for a support reply.

json{"criteria": [
  {"id": "asked_for_id",  "check": "The agent asked for the order id before acting.",  "weight": 1},
  {"id": "stated_policy", "check": "The reply states the 30-day refund rule correctly.", "weight": 2},
  {"id": "no_invention",  "check": "The reply promises nothing the tools did not do.",   "weight": 2}
]}

Each check is a plain yes or no. Weighted answers add up: 1 + 2 + 2 = 5 points, so a reply that misses only the policy line scores 3 of 5, which is 0.6. Small, specific checks are far more stable than one question such as “Is this reply good?”.

Exact answer match. The simplest family fits tasks with one right answer, such as arithmetic or logic puzzles. Reasoning Gym ships its generators together with verifiers. A verifier of this kind parses the final answer, tidies the format so that “1,000” and “1000” agree, and compares. Its failure is a false zero: a right answer in an unexpected format. Measure it by sampling zero-reward episodes. Fix it by normalising, or by accepting a short list of equivalent forms.

Every verifier has a cheapest exploit

RL is an optimiser. It finds the lowest-effort path to the highest reward. So for each verifier, ask one question: what is the cheapest thing that scores 1.0 without the real work?

For visible unit tests the answers are easy to list. The agent can delete the failing test. It can special-case the expected value (if xs == []: return 0 only for the test input). It can fix the bug and quietly break something else that has no test. For a rubric, a long and confident reply often scores well even when it promises things that never happened.

The fix for each is a hardening step. Run tests from a clean checkout so edits to test files are thrown away. Keep hidden test cases the agent never sees. Add regression tests that passed before and must still pass. Try each switch below and watch the exploits stop paying.

Harden the test verifier

The task is to fix total([]). Only the first behaviour is honest. Each bar is the reward the verifier pays. Turn on the switches and see which exploits stop paying.

Illustrative rules, one per switch. Clean checkout discards edits to test files. Hidden cases catch hard-coded outputs. Regression tests catch breakage elsewhere. A long confident reply with no code change fails the tests whatever you switch.

The hardened reward, as code

pythondef test_reward(repo, patch, hidden, regression):
    box = fresh_checkout(repo)                    # tests the agent never touched
    box.apply(patch.only_files("src/"))        # edits to tests/ are dropped
    fixed  = box.run(hidden).all_passed         # cases the agent never saw
    intact = box.run(regression).all_passed     # behaviour that must not break
    return 1.0 if fixed and intact else 0.0

How to find the holes you cannot list

You will not think of every exploit. So measure. Symptom: the training reward climbs, but a held-out check by humans does not. Measurement: sort episodes by reward, and read the top ones first, not the bottom. The best-scoring episodes are where exploits hide. Fix: every exploit you find becomes a new test or a new rule, and you re-score old episodes with it.

Also keep the reward simple. A 0 or 1 that you trust beats a clever blend of six sub-scores that the agent can trade against each other. Add partial credit only for a reason, and Chapter 6 gives the reason.

Your RL run shows rising reward. What is the best first audit step?

Chapter 5

Users in the Loop

Add a simulated customer to the world, and measure whether the agent succeeds every time, not just once

So far the task arrived whole: “Cancel W-1042.” Real customers do not talk like that. Yara writes, “I want to send the kettle back.” She does not know the order id. She remembers her email. She will refuse store credit because she wants the money on her card. The agent must find all this out by talking.

To train on tasks like this, the environment needs a person in it. People are slow and expensive, so we use a user simulator: a language model that plays the customer. It is the clerk’s customer, played by an actor who has read the script.

How τ-bench builds it

τ-bench (Yao et al., arXiv 2406.12045) tests agents on tool-agent-user interaction. Each conversation runs between a simulated user played by language models and an agent that has domain API tools and written policy guidelines. At the end, the database state is compared with an annotated goal state. So the pieces from earlier chapters all appear: a state check, tools, a task, and now a talking user.

The user needs a hidden scenario: private facts and a private goal that the agent cannot read. The simulator is told the scenario and told to reveal it only when it makes sense, as a person would.

json{"user_instruction":
  "You are Yara. You want to return the blue kettle. You do not remember the order id, only your email, yara@example.com. You are in a hurry. If offered store credit, refuse: you want the refund on your original card.",
 "goal_state": {"W-1042": {"status": "returned", "refund_to": "card"}}}

The loop now has two speakers. The agent can call a tool, which the environment answers, or reply, which the user answers.

pythondef run_episode(agent, user, env, scenario):
    msg = user.start(scenario)                  # the user speaks first
    while True:
        action = agent.act(msg)                  # a tool call or a reply
        if action.tool != "reply":
            msg = env.call(action)               # tool result returns to the agent
        else:
            msg = user.respond(action.text)      # the simulated user answers
            if user.is_done(msg): break
    return env.check_state(scenario.goal_state)  # the score reads the database, not the chat

Look at the last line. The score still reads the world. The user is there to make the task realistic, not to grade. That separation keeps a charming or confused simulator from changing the reward.

τ2-bench (Barres et al., arXiv 2506.07982) goes further. In many real tasks, such as phone support, the user also acts: she restarts the router while the agent guides her. The paper calls this a dual-control environment, with a Telecom domain where both agent and user use tools in a shared world. Its user simulator is tightly coupled with the environment, so its behaviour is constrained by its tools and observable states. The authors report that performance drops significantly when agents move from a no-user setting to dual control.

One win is not reliability

A customer does not care that your agent could have solved her problem. She cares that it does, every time. So τ-bench adds a metric called pass^k (“pass hat k”). Here is the idea in this lesson’s own words: run each task k times, and count the task only if all k runs succeed.

The paper reports that state-of-the-art function-calling agents such as gpt-4o succeed on fewer than 50% of tasks, and are quite inconsistent: pass^8 is under 25% in retail. A system that wins half the time looks fine once and falls apart when repeated.

Worked example with three tasks. The agent succeeds on task A every time (p = 1.0), on task B 80% of the time, on task C half the time. With one try, it averages (1.0 + 0.8 + 0.5) / 3 = 0.767. With four tries in a row, we need all four right: (1.0 + 0.84 + 0.54) / 3 = (1 + 0.4096 + 0.0625) / 3 = 0.491. With eight: (1 + 0.168 + 0.004) / 3 = 0.391. The same agent, three very different grades.

Win once, or win every time

Warm is pass@k: at least one of k tries works. Purple is pass^k: all k tries work. Choose how the agent’s skill is spread, then drag k.

4

Illustrative skill profiles, not measured. Flat: every task has a per-try success of 0.70. Mixed: 40% of tasks at 1.00, 30% at 0.70, 30% at 0.20 (a mean of 0.67). Each try is treated as independent, and pass^k is the share of tasks where all k tries succeed, as defined in this chapter.

When the simulated user misleads

Symptom: the agent’s score changes a lot when you swap the user model. Measurement: run the same agent against two simulators on the same 50 scenarios and compare. A gap of ten points means the simulator is part of what you are measuring. Also read transcripts for leaks, such as the user stating the whole goal in the first line.

Fix: tie the simulator to the scenario, give it only what a real user would know, restrict what it can claim, and keep the reward on the database. Re-check with a second simulator each time you change it.

An agent solves a task 80% of the time per try, independently. About what is its chance of solving that task on all 4 of 4 tries?

Chapter 6

Long Horizons: Credit, Budgets and Snapshots

Make a twelve-step job teachable, by paying for progress, bounding the cost and saving the world between steps

Yara closes her account. The job has twelve steps: find her three open orders, cancel them, refund two, remove her saved card, export her data, and confirm by email. An agent does eleven steps perfectly and slips on the last one.

With a 0-or-1 reward, that agent earns 0. So does an agent that did nothing. Training sees no difference between them. This is the central trouble with a long horizon, the number of steps before the job ends. Success needs a long chain of correct steps, so success is rare, so most rewards are zero, so there is little to learn from.

Why identical rewards teach nothing

Many RL methods train from a group of attempts at the same task. They compare each attempt with the group’s average. An attempt that did better than average gets pushed up. If all eight attempts score 0, every attempt equals the average, and nothing is pushed anywhere. The gradient, the direction the weights should move, is zero.

So a long-horizon environment must create differences between attempts. There are three tools, and each has a cost.

Tool 1: partial credit through milestones

A milestone is an intermediate state that a checker can confirm: “all three orders are cancelled”, “the card is removed”. Partial credit pays a fraction of the reward for each milestone reached. Eleven good steps now earn something.

The risk is farming. If milestones pay, an agent may collect the easy ones and skip the hard last step. One common design caps the partial part: reward = 0.5 × (milestones reached / total) + 0.5 × (final goal state matches). Full reward needs the real finish.

Tool 2: budgets and timeouts

A budget is a limit on cost: steps, tokens or dollars per episode. A timeout is a limit on wall-clock time, per tool call or per episode. Without them, one stuck agent can hold a training worker for hours. Both end the episode as truncated, the flag from Chapter 1.

Set the budget with care. If the reference solution takes 12 steps and the budget is 10, no agent can ever succeed. A common starting rule is one and a half to two times the reference length. That rule is a convention, not a law, so check it by measuring how often good attempts run out of steps.

A tool timeout should come back as an observation, such as {"error": "tool timed out after 30 s"}, so the agent can react. The 30 s is a value you choose.

Tool 3: snapshots

A snapshot is a saved copy of the whole world at one step: the database, the step counter and the random-number state. You can restore it later, and the world is exactly as it was.

pythonsnap = env.snapshot()              # {"db": ..., "t": 8, "rng": ...}
for _ in range(8):
    env.restore(snap)                # same world, same step 8
    rollout(agent, env)              # eight different endings

Snapshots save compute. If an agent slips at step 9 of 12, a retry from the snapshot at step 8 costs 4 steps, not 12, so 8 steps (67%) are saved. They also make fair comparisons: eight continuations from the same starting state differ only in what the agent chose.

Eight attempts, three reward styles

Each row is one attempt at the twelve-step job. A bar runs until the agent slips or the budget ends. Switch the reward style and watch how many different rewards the eight rows give.

12
12%

Illustrative simulation with seeded random slips, so the eight attempts are the same on every visit. A milestone sits every second step (6 in all). An attempt ends at its first slip or when the budget ends, and a step that is not reached earns nothing.

The failure modes, with a measurement for each

Milestone farming. Symptom: training reward rises, but the share of attempts that reach the final goal does not. Measurement: track both numbers on held-out tasks. Fix: cap the partial credit, as above.

Budget too tight. Symptom: success is exactly zero even for strong models. Measurement: replay the reference solution under the budget. Fix: raise the budget until the reference passes with room to spare.

Snapshot drift. Symptom: a restored world behaves differently from the original run. Measurement: restore, replay the same actions, and compare the end state byte by byte. Fix: put every source of randomness inside the snapshot. Chapter 7 returns to this.

All 8 attempts at a long task earn reward 0 under a finish-only reward. What goes wrong for group-based RL?

Chapter 7

Thousands of Worlds at Once

Run many sealed, repeatable environments side by side, and keep the expensive GPU from waiting on a slow tool

Your training run wants 4,096 attempts in flight at the same time. Each attempt needs its own world: its own database, its own files, maybe its own desktop. If two attempts share one database, one agent’s cancelled order shows up in another agent’s world, and the scores are junk.

So each world lives in a container: a sealed box on a machine, with its own files and programs, cut off from its neighbours. Think of a lunchbox. Many fit in one bag, and the sandwich in one box never touches the fruit in another. A container starts from an image, a frozen copy of the world built once and started many times.

One uniform door

A trainer cannot learn a new interface for every world. AgentGym (Xi et al., arXiv 2406.04151) describes its platform with this goal: diverse environments exposed through a uniform format, in real time and concurrently. The practical shape is a small server in each container with two doors.

httpPOST /reset  {"seed": 7, "task_id": "cancel-w1042"}
  -> {"obs": {"user": "Please cancel my kettle order, W-1042."}, "info": {...}}

POST /step   {"action": {"tool": "get_order", "args": {"order_id": "W-1042"}}}
  -> {"obs": {...}, "reward": 0.0, "terminated": false, "truncated": false}

This is the Chapter 1 contract, carried over a network. The trainer holds a list of such servers. It can talk to a retail world and a code world in the same loop, because both answer the same two calls. Our lesson on sandbox fleets shows how this works for whole desktops, and agent environments covers what such worlds contain.

OSWorld shows why the image matters. Each of its 369 tasks carries a setup configuration that builds the starting state, plus an execution-based script that checks the result. The image plus the config is what makes the task start the same way each time.

Determinism: the same seed, the same world

An environment is deterministic if the same seed and the same actions always give the same results. Without that, you cannot replay a failure, compare two agents fairly, or trust a score. Almost every hidden source of randomness hides in plain sight.

The clock is the usual culprit. Suppose a task says “cancel the orders placed this month” and the code calls datetime.now(). At midnight on the last day of the month, the goal changes under a running agent. The fix is to give the environment its own fake clock, set by the seed.

Other sources are the same in spirit: random-number calls without a seed, unordered sets, generated ids, network calls to live services, and threads that race. Test for all of them at once with a replay check, run nightly on every task.

pythondef replay_matches(env, task_id, seed, actions):
    hashes = []
    for _ in range(2):
        env.reset(seed=seed, options={"task_id": task_id})
        for a in actions:
            env.step(a)
        hashes.append(env.state_hash())   # one fingerprint of the whole world
    return hashes[0] == hashes[1]

Throughput: keep the GPU busy

The model runs on a GPU, the costly part. The tools run in containers, the cheap part. A single attempt alternates between them: the model thinks for g seconds, a tool runs for e seconds, and the GPU sits idle for those e seconds.

Worked example, with illustrative numbers. Let g = 2 s and e = 6 s. One attempt keeps the GPU busy for 2 of every 8 seconds, which is 25%. Run four attempts at once and the GPU always finds one ready: 4 × 2 / 8 = 100%. More containers fixed the idle time without touching the model.

A 12-step episode needs 12 × 2 = 24 s of GPU. At full use that is 3600 / 24 = 150 episodes an hour per GPU. A cold start adds to each episode: if a fresh container takes 20 s to boot, the episode takes 12 × 8 + 20 = 116 s instead of 96 s. A warm pool, containers booted ahead of time, removes that cost from the loop.

Keep the GPU busy

Bars show how busy one GPU is as you add parallel environments. The marked bar is your setting. Slow tools and cold starts make you need more containers.

4
6.0 s

Illustrative model: the model takes g = 2 s per step, an episode has 12 steps, a cold start takes 20 s, and one GPU serves rollouts without queueing. GPU use is min(1, parallel × 12 × g / episode time). The chosen number of environments doubles with each notch.

Where scale goes wrong

Symptom: the dashboard shows GPUs at 30% while hundreds of containers are alive. Measurement: log, for each step, the time spent in the model and in the tool. If tool time dominates, add environments or move slow tools off the critical path. Fix: more parallel containers, a warm pool, and a timeout on every tool call so one hung container cannot stall a batch.

Another symptom: scores jump between two runs of the same code. Measurement: the nightly replay check above. Fix: find the clock, the generated id or the race, and move it under the seed.

A model step takes 2 s and a tool takes 6 s. One attempt keeps the GPU busy 25% of the time. How many attempts running together fill it?

Chapter 8

Is This Task Any Good?

Score your own tasks with three measurements, and keep the ones that can still teach the agent something

You have 24 tasks and one week of training time. Some are broken. Some are trivial. Some are too hard today but fine next month. How do you know which are which, before you spend the week?

You run the current agent on every task, several times each, and read the results. Three measurements do most of the work.

Measurement 1: the pass rate per task

Run each task k times, say k = 8, and count the wins. The pass rate p of a task is wins divided by k. Plot p for every task and you get the spread: how many tasks sit near 0, near 1 and in between.

A warning on noise. Eight tries is a small sample. If a task’s true pass rate is 0.10, the chance of zero wins in eight tries is 0.98 = 0.43. So “0 of 8” does not prove the task is impossible. This is why Measurement 2 exists.

More tries shrink the noise. With 32 tries and a true pass rate of 0.10, the chance of zero wins falls to 0.932 = 0.034. So 32 tries on a few suspect tasks tell you far more than 8 tries on all of them. Spend the extra runs where the answer is unclear.

Measurement 2: the solvable rate

The solvable rate is the share of tasks that something can solve: the generator’s reference solution, a stronger model, or a person. Run the reference through the environment. If it earns 1.0, the task is solvable. If not, the task or its checker is broken. Chapter 3’s gate does this at generation time, and you repeat it as the world changes.

Now every task with p = 0 falls into one of two boxes. Reference passes: the task is too hard right now, so keep it for later. Reference fails: it is broken, so fix or drop it. A third group, tasks with p = 1, are retired: nothing is left to learn.

Measurement 3: the learning signal

Recall Chapter 6. A group of attempts teaches only if the rewards differ. For a task with pass rate p and a group of 8, the chance that the group mixes wins and losses is 1 − p8 − (1 − p)8. Read the first term as “all eight win” and the second as “all eight lose”. Take them away and what is left is the chance of a mixed group.

Worked values. At p = 0.5: 1 − 0.0039 − 0.0039 = 0.992. At p = 0.1: 1 − 0 − 0.430 = 0.570, because 0.18 is only 0.00000001. At p = 0.02: 1 − 0 − 0.851 = 0.149. At p = 0 or 1: exactly 0. Tasks near the middle almost always give a usable group. Tasks near the edges rarely do.

Pick the tasks worth training on

Each dot is one task, placed at its pass rate. The teal band is what you keep. Drag the two ends and watch how much signal the kept tasks carry.

10%
90%

Illustrative: 24 invented pass rates, the same on every visit, with 4 tasks at 0%, 3 at 100% and 17 between. The signal for a task is 1 − p8 − (1 − p)8, the chance that a group of 8 attempts mixes wins and losses, and the read-out averages it over the tasks you keep.

The audit in code

pythondef audit(tasks, agent, k=8):
    rows = []
    for t in tasks:
        wins = sum(run(agent, t, seed=s) == 1.0 for s in range(k))
        p = wins / k
        rows.append({"id": t.id, "p": p,
                     "signal": 1 - p**k - (1 - p)**k,
                     "ref_ok": run_reference(t) == 1.0})
    return rows

# keep: ref_ok and 0 < p < 1.   fix:  not ref_ok.   park: ref_ok and p == 0.   retire: p == 1.

When the measurements mislead

Symptom: a task looks well balanced at p = 0.5, yet training on it never helps. Measurement: check whether the wins come from luck, such as a coin-flip answer with two options, rather than skill. Re-run with more seeds and look at why attempts win. Fix: change the task so luck cannot carry it.

Also remember that the agent improves. A band that works today goes stale in a week. Re-run the audit on a schedule and move tasks between the boxes. Reasoning Gym’s adjustable complexity and τ2-bench’s controlled complexity exist for this reason: you turn the knob instead of rewriting the task.

A task has 0 wins in 8 attempts, and its reference solution earns reward 1.0. What do you do?

Chapter 9

When the World Gets Gamed

Find the shortcuts, leaks and flaky checks in your own environment before the optimiser finds them for you

A coding agent is asked to fix a bug. It lists the files in its container and spots check.py, the grader, sitting next to the code. It opens it. The grader ends with return score. The agent rewrites that line to return 1.0. Reward 1.0, two steps, no bug fixed.

Nobody taught the agent to cheat. RL rewards whatever scores. If a two-step route to 1.0 exists and a nine-step honest route exists, the two-step route wins, and training makes it more likely each round. This is reward hacking: scoring through a gap in the environment instead of through the intended work.

Four kinds of gap

Shortcuts. A tool is more powerful than the task needs. A refund_any tool with no policy check lets the agent skip the business rules. Fix: put policy checks in the tool code, not in the prompt.

Leaks. The answer is somewhere the agent can read. It might be a goal file in the container, a debug field in an API response ("expected_refund": 34.50), or the benchmark’s own solutions on the open web. Fix: keep goals and graders outside the container, strip debug fields, and cut the network or allow only a short list of sites.

Writable graders. The checker lives in the agent’s world and the agent can edit it. Fix: run verification in a separate process, on a fresh copy, after the episode ends.

Flaky checks. The same actions get different verdicts on different runs. This is subtle because it looks like noise, not cheating. Suppose a check wrongly passes 25% of wrong answers. A policy that submits again and again will see a pass in about four tries (1 / 0.25), and RL will teach it to do exactly that.

Why the cheapest route wins

An agent trained to maximise reward drifts toward whichever route to 1.0 is cheapest in steps and in risk. So think of your environment as a map of routes. Each hole you leave open adds a route. Each hole you patch removes one. The device below shows five routes with their step costs. Patch the holes and watch the cheapest route change.

Close the cheap routes

Each bar is a way to reach reward 1.0, with its cost in steps. Red routes are hacks. Press a switch to patch that hole.

Illustrative costs in steps: honest work 9, read the goal file 3, rewrite the grader 2, abuse the unchecked tool 4, look the solution up online 4, retry a flaky check about 4 (1 / 0.25, the false-pass rate assumed here). A patched route is removed. The point is the ordering, not the exact counts.

Finding gaps you did not plan for

You cannot list every hole by thought alone. Use three measurements.

Read the top. Sort episodes by reward, and read the best ones (Chapter 4). A short, high-reward episode on a hard task is a red flag.

Plant canaries. A canary is a fake secret, such as a unique string in a file the agent should never open. If it shows up in any action or observation, the agent got in.

pythonCANARY = "zk-7f3a91"          # planted in /task/answers.json, which no tool should expose

def leaked(trajectory):
    return any(CANARY in str(s.action) or CANARY in str(s.obs) for s in trajectory)

Replay for flakes. Take 20 finished trajectories and replay each 5 times in fresh containers. If 4 of the 20 produce mixed verdicts, the flake rate is 4 / 20 = 20% of trajectories. Any number above zero means the check, or the world, depends on something unseeded (Chapter 7).

Worked example of a canary hunt. You run 1,000 episodes and the canary string shows up in 37 of them. That is 3.7%. You read those 37 and find that all of them called get_order, and the tool’s reply held a debug field copied from the goal table. One line of code strips the field. You re-score the earlier runs and find the 37 episodes had inflated the training reward. The canary found the leak. The re-score measured the damage.

The fix is a loop, not a patch

Symptom: training reward climbs while a held-out human check does not. Measurement: the three audits above. Fix: patch the hole, then re-score the old episodes with the patched environment, so you know how much of the earlier gain was real. Keep a small suite of known exploits as tests that must always score 0.

A check wrongly passes 25% of wrong answers. What will RL training most likely teach the agent?

Chapter 10

Field Guide

Keep the numbers, the checklist and the core code for the next time you build a world for an agent

The one-paragraph summary

An environment is a contract: reset builds a seeded world and returns an observation, and step takes an action and returns an observation, a reward and the terminated and truncated flags. Where the task changes the world, score the state, not the words, and add a collateral rule. Generate tasks from seeded templates with a difficulty knob, and gate each one with a reference solution. Pick the verifier for the task family (state match, unit tests, rubric) and harden it. Add a simulated user when the task is a conversation, and measure reliability with pass^k. For long jobs, pay for milestones with a cap, bound cost with budgets and timeouts, and save snapshots. Run worlds in containers behind one uniform door, make them deterministic, and keep the GPU busy. Audit pass rates, solvable rate and learning signal. Then hunt for the cheap routes: leaks, writable graders, unchecked tools and flaky checks.

The numbers that matter

WhatNumberWhy it matters
Gymnasium step return5 values: observation, reward, terminated, truncated, infoThe standard contract; done was split in v0.26
τ-bench, retailtop agents under 50% of tasks; pass^8 under 25%One win is not reliability
τ2-benchdual-control Telecom domainUsers who act too make tasks harder
AppWorld9 apps, 457 APIs, about 100 users, 750 tasksA rich world with state-based tests and collateral checks
AppWorld, GPT-4oabout 49% normal, about 30% challengeDifficulty tiers inside one benchmark
OSWorld369 tasks; humans over 72.36%; best model 12.24%Each task has a setup config and a check script
SWE-Gym2,438 instances; up to 19% absolute gain; 32.0% and 26.0% on two SWE-Bench setsA training environment is a result in its own right
Reasoning Gymover 100 generators and verifiers, many areasProcedural tasks with adjustable difficulty
Five goals at 0.9 each (illustrative)0.95 = 0.590Difficulty compounds with task length
Mixed group chance (illustrative)1 − p8 − (1 − p)8; 0.992 at p = 0.5Tasks near the middle carry the learning signal
GPU fill (illustrative)g = 2 s, e = 6 s: 4 environments fill 1 GPUParallel worlds hide tool latency

The checklist

  1. Does reset(seed) give the same world twice? Run the replay check.
  2. Does the reward read the world, not the words?
  3. Is there a collateral rule for unrequested changes?
  4. Does every generated task ship with a reference solution that the checker accepts?
  5. Are tests run from a clean copy, with hidden cases and regression tests?
  6. Can the agent see or edit the grader, the goal or the network?
  7. Do truncated and terminated mean different things in your code?
  8. Is the budget above the reference length, and does every tool call have a timeout?
  9. Does partial credit stay capped below a full finish?
  10. Did you swap the user simulator and compare scores?
  11. Have you read the top-reward episodes this week?
  12. Are 0% tasks sorted into broken and too hard, and 100% tasks retired?

The core as code

pythonclass RetailEnv:
    max_steps = 12

    def reset(self, *, seed=None, options=None):
        self.rng  = random.Random(seed)                  # Ch 7: every random call goes through this
        self.task = make_task(seed, options["n_goals"])       # Ch 3: seeded template
        self.db, self.t = self.task.world(), 0
        self.start = copy(self.db)                        # for the collateral rule
        return {"user": self.task.text}, {}

    def step(self, action):
        self.t += 1
        obs = run_tool_with_timeout(self.db, action, seconds=30)  # Ch 6: policy lives in the tool code (Ch 9)
        terminated = action["tool"] == "reply"
        truncated  = self.t >= self.max_steps and not terminated
        reward = self.score() if (terminated or truncated) else 0.0
        return obs, reward, terminated, truncated, {"t": self.t}

    def score(self):                                  # Ch 2 and 4: state, not words
        done  = state_match(self.start, self.db, self.task.goal)
        miles = self.task.milestones_hit(self.db) / self.task.n_milestones
        return 0.5 * miles + 0.5 * done                  # Ch 6: capped partial credit

    def snapshot(self): return {"db": copy(self.db), "t": self.t, "rng": self.rng.getstate()}
    def restore(self, s):  self.db, self.t = copy(s["db"]), s["t"]; self.rng.setstate(s["rng"])

Wrap this class in the two-door server from Chapter 7, run it in a container from a frozen image, and audit it with Chapter 8’s function. That is a training environment.

Where this sits

Environments are the bridge between a model and a skill. The model supplies guesses. The environment supplies the truth about those guesses, at scale. Everything in the post-training stack that learns from verifiable outcomes, such as RL on code, tools and computer use, relies on a world that can say “yes” or “no” and mean it.

Keep going

Sources

The takeaway. The agent becomes whatever the world pays for. Build the world so that the cheapest way to earn reward is the real work: score the state, seed everything, measure what each task teaches, and keep hunting for the shortcut before the optimiser finds it.

Ready to test yourself? Open Teach and explain one chapter on a whiteboard, or open Sparks for single-idea cards. Then go back to the grader at the top and try all three agents again, now knowing what each grader can see.

Task Environments
τ-bench · AppWorld · SWE-Gym · Back to Gleams