Agent Environments

You have built an AI program that can work a calendar app. You give it one job, ‘Put a dentist appointment on Friday at 3 pm’, and run it twenty times to see how reliable it is. Nineteen passes out of twenty: 95%. You are pleased. Then you watch a replay of run 3. The program opens the calendar, sees a dentist appointment already sitting on Friday at 3 pm, and stops. It did nothing, and it passed.

So where did that appointment come from? From run 2. Run 2 really did the job. Then nobody took the appointment away, and every run after it started with the job already done.

This lesson is about that gap: the gap between what a program really did and the number a test reports about it. To see the gap clearly, we need a few plain words first. None of them is hard, and each one names something you can already picture.

What an agent is

Picture a new assistant who can see your screen and use your mouse and keyboard, and nothing else. You tell them a job in words. They look at the screen, click, type, look again, and keep going until they think the job is done.

A program built around an AI model that looks at a screen and works the mouse and keyboard to finish a job you describe in words is an AI agent. The model behind it is the same kind you have chatted with. The difference is in its replies. A chat assistant answers with sentences; an agent answers with clicks and keystrokes.

One job, given in words, such as ‘Put a dentist appointment on Friday at 3 pm’, is a task. Think of it as one question on an exam paper. One attempt at one task, from the moment it starts to the verdict, is a run: one sitting of the exam. The opening scene had one task and twenty runs.

What a fair test needs

Think of an exam at a school. Every student sits the same paper. Each one sits at a clean desk, with the paper face down. One marking scheme decides every grade. Take away any of the three, and the grades stop meaning anything.

A test for an agent needs the same three things: the same task, the same starting point, and a fixed way to judge the result. Each of the three has a name.

The exam room comes first. The computer, with its apps and files, set up so the agent can be tested on it, is the test world. Engineers call this test world an environment. In this lesson the two words mean the same thing, and we use both.

Then the clean desk. Everything in the world at the moment a run begins, the files, the calendar entries, the open windows, is the starting state. Two runs are only comparable if they begin from the same starting state.

Then the examiner. A small program that looks at the world after the run and says pass or fail is the checker. It holds the marking scheme. The share of runs the checker passed is the agent’s score, the exam mark: 19 passes in 20 runs is a score of 95%.

Where the 95% came from

Now walk the first three runs slowly. At the start of run 1, the calendar holds three appointments: Standup on Monday at 9, Lunch with Sam on Wednesday at 12, and Gym on Friday at 10. The agent reads the task and puts the dentist on Thursday by mistake. The checker looks at Friday at 3 pm and finds nothing. Run 1 fails.

Run 2 begins with run 1’s wrong Thursday appointment still in the calendar. This time the agent adds the dentist on Friday at 3 pm. The checker looks, finds it, and says pass. So far, one fail and one pass.

Run 3 begins with both leftovers in place: the wrong Thursday appointment and the right Friday one. The agent opens the calendar, sees a dentist on Friday at 3 pm, and decides the job is already done. It stops. The checker looks at Friday at 3 pm, finds the appointment that run 2 made, and says pass.

Putting the world back to the same starting state before every run is a reset. Think of wiping the whiteboard before the next class. This test world never reset.

Something from one run that is still there when the next run begins is a state leak. It is the last student’s answers left on the desk. Every run from 3 to 20 sat down at a desk with the answer already written on it.

A leak does two things

A leak does two separate things, and both corrupt the score.

First, it changes what the checker finds. The checker reads the calendar at the end of the run. A leftover appointment satisfies it, whoever made it and whenever.

Second, it changes what the agent sees. In run 3 the agent saw the job done and stopped. That behaviour looks reasonable: a sensible assistant would not book a second dentist appointment. This is why the second effect is easy to miss. The replay looks fine, and the score is still wrong.

Step through it yourself. The device shows each run’s calendar twice: as it was when the run started, and as it was when the run ended. Watch what moves from one run to the next.

The reset fixes the first lie

Now switch the device to Reset the world and step through again. Every run starts from the same three appointments. Run 3 gives up and fails, run 4 succeeds, and run 5 picks Thursday again. After five runs the score is 2 of 5, and that is what this toy agent really does.

Over all twenty runs in the device at the top of the page, a reset brings the score down from 95% to 8 of 20, 40%. That is the truth for this toy agent. In a clean world it does the job on 8 runs, puts the dentist on the wrong day on 5, gives up on 5, and on 2 runs it does the job but also deletes the Gym appointment. All of these numbers are illustrative: we made the agent up to show the idea.

The checker can lie on its own

A reset is not the end of the story. Look at run 6 with a reset. The agent added the dentist on Friday at 3 pm, and it also deleted the Gym appointment. A checker that only looks for the dentist finds the dentist and says pass.

A checker that says pass when the task was not done gives a false pass. One that says fail when the task was done gives a false fail. Think of a smoke alarm. Ringing for burnt toast is a false pass; staying quiet in a fire is a false fail. Both make the alarm less useful, in different ways.

Deleting Gym is a change the task did not ask for. A change like that is a side effect: the mechanic who fixes your brakes and scratches the door on the way out. Nobody would call run 6 a success. Yet the checker that looks for the event cannot see the damage, because it never looks anywhere else.

A different checker can. Before the run, it notes every appointment. After the run, it lists everything that changed. It passes only if the one change is the right appointment and nothing else moved. That is the second setting of the device at the top, Reads what changed. On run 6 it sees ‘+ Dentist Fri 3 pm’ and ‘− Gym Fri 10 am’, and it fails the run.

Two levers, both ways

Now the four settings at the top of the page make sense. Two levers, the reset and the checker, give four scores for one agent that never changes.

The leak can push the score up: 95% with the checker that looks for the event. It can also push the score down. With the checker that reads what changed, every run from 3 onwards changed nothing, because the agent stopped. A run that changes nothing fails that checker. So the score falls to 5%: the agent is punished for runs it did well. A lie in either direction is still a lie.

Here is the whole ledger, counted run by run. Without a reset, a run’s starting state is the previous run’s final state. So once any run leaves the right answer behind, the checker sees the right answer in every run that follows.

worked exampleThe toy agent in a clean world (illustrative)
  20 runs: 8 do the task, 5 pick the wrong day, 5 give up, 2 do it but delete Gym
  the truth = 8 of 20 = 40%

Reset before every run
  checker reads what changed: passes only the 8          =  8 of 20 = 40%   0 wrong verdicts
  checker looks for the event: passes the 8 and the 2    = 10 of 20 = 50%   2 false passes (runs 6, 13)

Keep the last run's world
  run 1 adds Dentist on Thursday; run 2 adds Dentist on Friday at 3 pm
  from run 3 on, every run starts with Friday's dentist, so the agent stops
  checker looks for the event: run 2 + runs 3 to 20      = 1 + 18 = 19 of 20 = 95%   11 false passes
  checker reads what changed: only run 2 changed it      =  1 of 20 =  5%   7 false fails

Same agent, four scores: 5%, 40%, 50%, 95%. Only one of them is the truth.

Read the false passes in the third block carefully. Eleven of the nineteen passes were not earned. Nine of those eleven are runs where this agent, in a clean world, would have picked the wrong day or given up. The other two are runs 6 and 13, where it would have deleted Gym.

Why this matters beyond a toy

A fixed set of tasks, each with its own setup and checker, that everyone runs so their scores can be compared, is a benchmark. It is a standard exam that every school sits, so one school’s marks mean the same as another’s.

Benchmark scores for computer-using agents are reported to two decimal places, such as 12.24%. Teams compare them to decide which agent is better. Two decimal places only mean something if every run started clean and every checker read the right thing.

There is a second reason, and it is bigger. Agents are also trained inside these same worlds. An agent tries a task, the checker’s verdict is what it learns from, and it is changed so that it passes more often. Chapter 1 names this kind of training. A world that leaks, or a checker that lies, teaches an agent the wrong lesson. Our toy agent, trained on the leaky world, would learn that stopping early is a fine way to pass.

The rest of the lesson takes these two levers apart, one real test world at a time. Here is the road, in order:

1. The four parts of a test world: the starting state, what the agent sees, its hands, and the checker, with one run stepped through them.
2. Reading a task file: three real task descriptions, read line by line from setup to score.
3. Worlds made of websites: frozen copies of websites, restarts between agents, and checks on the page.
4. A phone inside a computer: a pinned clock, random task details, and a checker for every row.
5. Run it again: why one score wobbles, and how many runs a fair comparison needs.
6. Checkers that tell the truth: false passes, false fails, and checkers an agent cannot fool.
7. When the world breaks: telling a broken world from a weak agent.
8. Field Guide: the checklist, the numbers and where each came from.

Sit where the agent sits. A picture of the whole screen arrives: a calendar, drawn as a grid of 1920 by 1080 tiny coloured dots called pixels, with one line of text: ‘Put a dentist appointment on Friday at 3 pm.’ You answer with one move: click at the point 1688 across and 652 down. Two seconds later a new picture arrives. The cell you clicked is open, waiting for a title. You type ‘Dentist’. Another picture. You press Enter. Another picture. You say you are done. Then, somewhere you cannot see, a program reads the calendar and writes down a 1.

That short scene holds every part of a test world. This chapter takes them apart, gives each one its name, and then shows how two real test worlds put the same parts together in different ways. (The click point is made up. The screen size and the two-second wait are the real defaults of a desktop test world we meet below.)

The loop, in plain words

Strip the scene down and it is a loop. The agent sees something, acts, sees the result, and acts again, until it says it is done. Then a verdict arrives from outside the loop.

A stage play is a good picture of it. There is the set as it stands when the curtain rises. There is the window the actor looks through. There are the actor’s hands. And there is the critic, who reviews the stage after the curtain falls. A test world has the same four parts.

The set is the starting state from Chapter 0, put back by the reset before every run. What the agent is shown at each moment, usually a picture of the screen plus the task’s words, is the observation. It is the window.

One thing the agent does, a click, some typing or a key press, is an action. The part of the test world that carries the action out on the real computer is the action executor: the agent’s hands. The agent only decides what to do. The executor moves the mouse and presses the keys.

The critic is the checker from Chapter 0. It reads the world after the run and gives the verdict. It sits outside the loop on purpose: the agent never talks to it.

One round of the loop, observation in and action out, after which the world has changed a little, is a step, like one move in a board game. The opening scene had four steps: click, type, Enter, done. A whole run, from reset to verdict, is called an episode in code. The prose of this lesson keeps saying run; the code and the expert views say episode.

What the agent cannot see

The calendar app keeps its appointments in a file on the computer’s disk. The agent never sees that file. It sees only pictures of the screen.

Everything inside the computer that could matter, including what the agent cannot see, is the state. Think of a card game: the state is the whole table, including the cards still in the deck. The screenshot shows only part of the state, the part that happens to be drawn on screen right now.

The agent sees a screenshot, never the whole state. A world like that is partially observable: you play with your own hand visible and the deck hidden.

This is why the checker reads the state directly and never looks at the last picture. A picture can look right while the saved data is wrong. A spreadsheet can look finished on screen while the file on disk is still blank; Chapter 2 meets exactly that case. The microMDP and microPOMDP lessons build these ideas formally. This page needs only the plain version.

Rewards, and training

The number the checker hands back, 1 for success, 0 for failure, sometimes a fraction, is the reward. It is the mark on the paper. A task with two parts might give 0.5 when only one part is done.

Test worlds are not only for testing. Training by trial and reward, where the agent tries, gets a reward, and is changed to earn more next time, is reinforcement learning. A dog learning a trick for treats is the classic picture. The dog tries things, the treat says which try was right, and the right tries become habits.

The same four parts serve both jobs. A test reads the rewards and adds them up. Training learns from them. That is why the leak in Chapter 0 is dangerous twice over: in a test it gives a wrong number, and in training it teaches a wrong habit.

One real desktop test world writes all of this down as the formal model that the microPOMDP lesson teaches: states the agent cannot fully see, observations, actions, and a reward between 0 and 1 paid at the final step.

The interface: three functions

Many teams build agents, and many teams build test worlds. They need to plug into each other without rewriting anything. So programmers agree on a shape. The set of functions a test world promises to offer, with what each one takes and returns, is its interface. Think of the sockets on a wall: any plug of the right shape fits any socket.

The usual interface has three functions. reset() sets up the starting state and returns the first observation. step(action) carries out one action and returns the next observation. evaluate() runs the checker and returns the reward. Start the game, make a move, count the points.

Here is a whole test world for our calendar, built from scratch in about twenty lines of Python. It is a toy: the world is a dictionary, not a real app. But it has all four parts, and it never leaks.

pythonSEED = {"Mon": {(9, "Standup")}, "Tue": set(), "Wed": {(12, "Lunch with Sam")},
        "Thu": set(), "Fri": {(10, "Gym")}}            # the starting state

class CalendarEnv:
    def reset(self):
        self.world = {d: set(e) for d, e in SEED.items()}   # a fresh copy, every run
        self.start = {d: set(e) for d, e in self.world.items()}
        return self.observe()
    def observe(self):                                  # what the agent is shown
        return sorted((d, h, t) for d, evts in self.world.items() for (h, t) in evts)
    def step(self, action):                             # the action executor
        kind, day, hour, title = action
        if kind == "add":
            self.world[day].add((hour, title))
        done = kind in ("add", "noop")
        return self.observe(), 0.0, done, {}            # reward 0 until the end
    def evaluate(self):                                 # the checker: read what changed
        added   = {(d, h, t) for d in self.world for (h, t) in self.world[d] - self.start[d]}
        removed = {(d, h, t) for d in self.start for (h, t) in self.start[d] - self.world[d]}
        return 1.0 if added == {("Fri", 15, "Dentist")} and not removed else 0.0

Walk it from the top. SEED is the starting state: one set of appointments for each day. reset() copies the seed, so the agent’s edits land on the copy and the seed itself never changes. That one line is the whole difference between Chapter 0’s honest world and its leaky one.

reset() also keeps a second copy, start, taken at the moment the run begins. The checker needs it later, to see what changed. observe() is the observation: a sorted list of every appointment. It stands in for the screenshot.

step() is the action executor. It unpacks the action, adds the appointment if the action says add, and reports whether the run is over. It always returns a reward of 0.0, because the verdict comes later.

evaluate() is the checker that reads what changed. It builds the set of appointments added since start and the set removed since start. It passes only if exactly the right appointment was added and nothing was removed. That is the hero’s Reads what changed, in four lines.

Run it twice, the way a test would, and print what comes back:

pythonenv = CalendarEnv()
print(env.reset())                                       # run 1 starts clean
obs, reward, done, info = env.step(("add", "Fri", 15, "Dentist"))
print(reward, done, env.evaluate())
print(env.reset())                                       # run 2 starts clean again
obs, reward, done, info = env.step(("add", "Thu", 15, "Dentist"))
print(reward, done, env.evaluate())
# output
[('Fri', 10, 'Gym'), ('Mon', 9, 'Standup'), ('Wed', 12, 'Lunch with Sam')]
0.0 True 1.0
[('Fri', 10, 'Gym'), ('Mon', 9, 'Standup'), ('Wed', 12, 'Lunch with Sam')]
0.0 True 0.0

Both runs start from the same three appointments, because reset() hands each run a fresh copy. In both runs step() says reward 0.0 and done True. Only evaluate() tells them apart: the first run scores 1.0, and the second, which put the dentist on Thursday, scores 0.0. Nothing from the first run is visible in the second.

Two real shapes

Real test worlds keep the three functions, but they differ in one important detail: where the reward comes out. Two of them show the two choices.

The first is a test set of 369 tasks on a real Linux desktop, called OSWorld. Its agent works a whole computer: a web browser, an office suite, a terminal. The second is a toolkit called BrowserGym that puts many web test worlds behind one interface. Its agent works a web browser.

In the desktop world, the agent’s moves are lines of Python for a library called pyautogui: a Python library that moves the mouse and presses keys on a real screen, like a robot hand on the mouse. The opening click is pyautogui.click(1688, 652). In the web world, the agent writes short calls such as click('a51'), where a51 names one button on the page.

OSWorld’s step() returns four things: the observation, the reward, a done flag and an info dictionary. The reward is always 0. The source says so in a comment, and nothing below that line changes it. Here is the function, cut to its spine:

python · OSWorld desktop_env.pydef step(self, action, pause=2):
    reward = 0  # todo: Define reward calculation for each example
    done = False  # todo: Define episode termination condition for each example
    info = {}
    ...
    elif action == 'DONE' or (type(action) == dict and action.get('action_type') == 'DONE'):
        done = True
        info = {"done": True}
    ...
    time.sleep(pause)
    observation = self._get_obs()
    return observation, reward, done, info

The reward is set to 0 at the top and never changed: success is decided later, by evaluate().

So in the desktop world, success comes from one evaluate() call, made once, after the run has ended.

BrowserGym’s step() returns five things. Each task in it has its own checker, a function called validate(), and the toolkit calls it inside every single step:

python · BrowserGym env.pyreward, done, user_message, task_info = self._task_validate()
...
# new step API wants a 5-tuple (gymnasium)
terminated = done or (
    self.terminate_on_infeasible and self.infeasible_message_received
)  # task or agent can terminate the episode
truncated: bool = False
return obs, reward, terminated, truncated, info

The task’s validate() runs inside every step, so the reward can arrive on any step and end the run.

Look at the two flags in that return. A flag that says the run is over is done. Some test worlds split it in two. Terminated means the task itself ended the run. Truncated means something outside cut it off, such as a cap on steps. Think of the final whistle of a football match, against the referee stopping play because time ran out.

A limit on how many steps one run may take is a step cap: the time limit on the exam. When a run hits the cap, it ends whether the agent is finished or not.

Step one run through each shape and watch where the reward appears. The calendar task is the same in both; only the plumbing differs.

Why two designs

Why would anyone build a world whose step() always says 0? Build the interface from what a fair test needs, and the choice falls out on its own.

1. A fair test needs a known start. So the first function must build the starting state and show the agent its first view: reset() returns an observation.
2. The agent acts one move at a time and must see the result of each move. So step(action) returns an observation.
3. The verdict must come from the world, not from the agent’s own claim. So it is a separate reading of the state: evaluate() returns a reward.
4. Where should that reading happen? If the checker needs the world to be finished first, a file saved or a browser closed so it writes its settings, it can only run once, after the last step. That is OSWorld’s choice. If the task can be judged as it goes, a chat answer or a page that shows the result, the check can run after every step and stop the run early. That is BrowserGym’s choice, and its cost is the same check repeated on every step.
5. So a training loop must know which shape it has. Adding up step() rewards works for the second shape. For the first shape it gives 0, every time.

Each design buys something. A checker that runs once at the end can do slow, heavy work first: save the open file, restart the browser, copy files out of the machine. Chapter 2 shows all three in real task files. A checker that runs after every step can end the run the moment the task is done, and it can answer a chat message in the middle of a run. But it pays its cost on every step.

Both worlds also spend time waiting, on purpose, so that the screen settles before the next picture. OSWorld waits 2 seconds after every action before it takes the next screenshot, and it stops a run at 15 steps. BrowserGym slows every browser action by 1,000 milliseconds by default, and it allows 30 steps on its web benchmarks. Here is the arithmetic for one run of the calendar task, and for the waiting alone:

worked exampleOne run of the desktop test world (the calendar task; the actions are illustrative)
  reset(task)              → screenshot 1920 × 1080 = 2,073,600 pixels, and the task's words
  step 1  click(1688, 652) → reward 0, done False        then a 2 s pause
  step 2  write('Dentist') → reward 0, done False        then a 2 s pause
  step 3  press('enter')   → reward 0, done False        then a 2 s pause
  step 4  DONE             → reward 0, done True         then a 2 s pause
  evaluate()               → 1.0
  sum of step rewards      = 0 + 0 + 0 + 0 = 0           what a careless training loop records
  the run's real reward    = evaluate() = 1.0            what it should record

Pauses alone, at the desktop world's defaults (15-step cap, 2 s after each action)
  per run at the cap       = 15 × 2 s = 30 s
  across all 369 tasks     = 369 × 30 s = 11,070 s = 3.1 hours of waiting, before any thinking

The web toolkit's defaults (1,000 ms slow-down per action, 30-step cap on its web benchmarks)
  per run at the cap       = 30 × 1 s = 30 s

Three hours of pure waiting for one pass over the desktop benchmark, before the agent has thought about anything. Chapter 7 shows how real teams cut that time down.

The score from Chapter 0 now has an exact form. It is an average of the checker’s rewards, one per run, each one read from that run’s final state. Hover or tap any symbol to see what it means.

score

The share of runs the checker passed, written as a fraction or a percentage. Also called the success rate.

1 ÷ N

N is the number of runs: tasks times reruns of each.

evaluate(s_T⁽ⁱ⁾)

The checker’s reward for run i, read from its final state s_T: 1, 0, or a fraction for tasks with several parts.

8 runs at 1 + 12 runs at 0 = 8;  8 ÷ 20 = 0.40, the hero’s honest score

What an observation is made of

Back to the pixels of the opening scene. How many pixels across and down a screen has is its resolution. OSWorld’s screen is 1920 by 1080 by default: 2,073,600 pixels in every screenshot, about two million coloured dots in each observation.

Next to the picture, OSWorld can hand over a text view. A text list of everything on the screen, each button and box with its name, its text and its place, that the computer keeps for screen readers used by blind people, is the accessibility tree. Think of the floor plan of a building, written out room by room.

The text view is huge. Language models read text in small pieces, often a short word or part of a word. These pieces are tokens, the beads on a string of text. The raw accessibility tree of one desktop screen is usually over 1 million tokens long. So agents get a filtered table instead, with five columns: tag, name, text, position and size.

How many tokens a screenshot costs depends on the model that reads it, so this lesson gives no number for it. What it can give is the size of each screen. The device below puts four real screens side by side, at one scale.

BrowserGym’s observation is a dictionary. Its keys include goal, url, screenshot, axtree_object (the accessibility tree) and last_action_error. BrowserGym also stamps every element on the page with an id of its own, a bid, so an action can say click('a51') and mean exactly one button.

What an action is made of

OSWorld’s agent writes one line of pyautogui per step, such as pyautogui.click(1688, 652) or pyautogui.write('Dentist'). It also has three special words: WAIT, FAIL and DONE. WAIT lets the screen catch up. DONE ends the run. FAIL means: this cannot be done.

Some tasks are built for exactly that answer. A task that cannot be done on purpose, to see whether the agent says so instead of pretending, is an infeasible task. It is an exam question with no right answer, set to catch bluffing. 30 of OSWorld’s 369 tasks are infeasible. Chapter 6 shows the strange score this gives an agent that answers FAIL to everything.

BrowserGym’s agent writes short calls such as click('a51') or fill('a52', 'Dentist'). BrowserGym turns each call into commands for the browser. If a call fails, say because the id does not exist on the page, the error message comes back in the next observation as last_action_error. The agent can read it and try something else.

One interface, many worlds

One shared interface is why BrowserGym can run six web benchmarks with the same agent code, and why another benchmark’s own harness is built on BrowserGym’s core. Here is the whole agent loop, as BrowserGym’s paper prints it:

python · BrowserGym paper, Figure 3import gymnasium as gym
# environment creation
env = gym.make("browsergym/miniwob.choose-list")
# interaction loop
obs, info = env.reset()
while True:
  action = ... # agent reasoning is performed here
  obs, reward, terminated, truncated, info = env.step(action)
  if terminated or truncated: break

Every web world behind the toolkit runs this same loop; only the name inside gym.make() changes. The EnvHarness lesson goes one step further: it reshapes existing test worlds, where a run starts and what the agent sees, only through their reset and step functions.

A teammate writes a new task for the desktop test world: ‘Install Spotify.’ A person could check it by looking. But this check must run thousands of times, at night, with nobody watching. What exactly must the teammate write down so that a machine can set the stage, let the agent loose, and decide pass or fail on its own?

What the file must say

A machine needs four answers, written down before the run starts. Where does the world start? How is it set up? What words does the agent get? And how is the result checked? Leave out any one, and somebody has to sit and watch.

A small text file that holds everything about one task is a task file, the recipe card for one run. It is written in JSON: names and values inside curly brackets, which people and programs can both read. A line such as "snapshot": "os" is one name and its value.

Here is a real one, for exactly that Spotify task, from the desktop test world of Chapter 1. Read it once without worrying about the names. Then find the four answers: the line that says where to start (snapshot), the list that says how to set up (config), the words the agent gets (instruction), and the block that says how to check (evaluator).

json · OSWorld os/94d95f96….json{
  "id": "94d95f96-9699-4208-98ba-3c3119edf9c2",
  "snapshot": "os",
  "instruction": "I want to install Spotify on my current system. Could you please help me?",
  "source": "https://help.ubuntu.com/lts/ubuntu-help/addremove-install.html.en",
  "trajectory": "trajectories/",
  "config": [
    {
      "type": "execute",
      "parameters": {
        "command": [
          "python",
          "-c",
          "import pyautogui; import time; pyautogui.click({SCREEN_WIDTH_HALF}, {SCREEN_HEIGHT_HALF}); time.sleep(0.5);"
        ]
      }
    }
  ],
  "related_apps": [
    "os"
  ],
  "evaluator": {
    "func": "check_include_exclude",
    "result": {
      "type": "vm_command_line",
      "command": "which spotify"
    },
    "expected": {
      "type": "rule",
      "rules": {
        "include": [
          "spotify"
        ],
        "exclude": [
          "not found"
        ]
      }
    }
  },
  "proxy": false,
  "fixed_ip": false,
  "possibility_of_env_change": "low"
}

The other lines are bookkeeping. id names the task. source records where the idea came from, an Ubuntu help page. related_apps groups tasks by app. The last three lines arrived in 2025, and we come back to them at the end of the chapter.

Where to start: a computer inside a computer

The desktop test world does not run the agent on a real laptop. A whole computer simulated inside another computer, with its own screen, disk and operating system, is a virtual machine. Think of a ship in a bottle: a complete ship, sealed inside something else.

A virtual machine has one great gift for testing: its whole state can be saved. A saved copy of a virtual machine’s whole state, like a save point in a video game, is a snapshot. Going back to it is reverting. Whatever the agent did, a revert puts every file and setting back as it was.

The line "snapshot": "os" names the save point this task starts from. Before every run, OSWorld reverts the machine to its snapshot and then runs the task’s setup. The current code adds one shortcut: it skips the revert when nothing has touched the machine since the last one, to save time.

Why not one snapshot per task?

If snapshots are so good, why not save one per task, already set up, and revert to it? Derive the answer in four short steps.

1. A task needs an exact starting state. The simplest way to get one is one snapshot per task.
2. A snapshot holds a whole machine. OSWorld’s authors write that one per example would need “gigabytes of space” each. The benchmark has 369 tasks but only 302 different starting states. Even at an illustrative 2 GB each, that is 604 GB.
3. Most starting states differ from each other only in a file or an open app. So keep a few shared snapshots, and describe each task’s difference as a short list of small jobs.
4. The price: those jobs can fail or race each other (Chapter 7). The prize: a task is a few hundred bytes of JSON instead of gigabytes.

A list of small jobs the test world does before the agent starts, such as downloading a file or opening an app, is the setup steps: setting the stage before the curtain. The file calls them config. The real step types include download, open, launch, execute, activate_window, sleep and chrome_inject_js.

The Spotify task has one setup step. It runs a line of Python that clicks the centre of the screen and waits half a second, so the desktop has focus before the agent starts. The words {SCREEN_WIDTH_HALF} are filled in with half the screen width when the task runs.

How to check: a getter, a metric, an expected value

Now the evaluator block. A person checks a result by looking. A machine checks it in two moves: fetch one fact, then compare it with the right answer.

A small function that fetches one piece of the final state, a file, a setting, a command’s printout, is a getter: a lab technician who brings back one sample. A function that compares what the getter fetched with the right answer and returns a score is a metric: the examiner comparing an answer with the answer key.

The right answer, prepared in advance by the task’s author, is the expected value. When the right answer is a whole file, that file is a gold file.

In the file, func names the metric. result describes the getter: its type picks a getter function named get_ plus the type. expected holds the right answer. For Spotify, the getter vm_command_line runs which spotify inside the machine. The command ‘which’ prints where a program is installed, if it can find it.

The metric check_include_exclude then passes only if that printout contains ‘spotify’ and does not contain ‘not found’. Here are the real metric functions behind this task and the next one, with two debug print lines removed:

python · OSWorld metrics/general.pydef check_include_exclude(result: str, rules: Dict[str, List[str]]) -> float:
    if result is None:
        return 0.
    include = rules.get("include", [])
    exclude = rules.get("exclude", [])
    if all(r in result for r in include) and all(r not in result for r in exclude):
        return 1.
    else:
        return 0.

def exact_match(result, rules) -> float:
    expect = rules["expected"]
    if result == expect:
        return 1.
    else:
        return 0.

Read the first one aloud. If the getter returned nothing, score 0. Otherwise, take the words that must appear and the words that must not. If every must-appear word is in the printout and no must-not word is, score 1. The second one is simpler still: score 1 only if the value is exactly the expected one.

The Chrome task: a side door and a restart

The second file asks the agent to turn on Chrome’s ‘Do Not Track’ setting. A setting lives inside the browser, where no command line can see it. So the setup opens a way in.

A numbered door a program opens so other programs can talk to it is a port, a service hatch. Chrome’s remote debugging port lets another program read its settings. The setup starts Chrome with --remote-debugging-port=1337, then uses a small tool, socat, to forward port 9222 to that door, so the checker can reach it.

json · OSWorld chrome/030eeff7….json{
  "id": "030eeff7-b492-4218-b312-701ec99ee0cc",
  "snapshot": "chrome",
  "instruction": "Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?",
  "source": "https://www.surreycc.gov.uk/website/cookies/do-not-track",
  "config": [
    {
      "type": "launch",
      "parameters": {
        "command": [
          "google-chrome",
          "--remote-debugging-port=1337"
        ]
      }
    },
    {
      "type": "launch",
      "parameters": {
        "command": [
          "socat",
          "tcp-listen:9222,fork",
          "tcp:localhost:1337"
        ]
      }
    }
  ],
  "trajectory": "trajectories/",
  "related_apps": [
    "chrome"
  ],
  "evaluator": {
    "postconfig": [
      {
        "type": "chrome_inject_js",
        "parameters": {
          "js": "if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }",
          "wait_for_load": false
        }
      },
      {
        "type": "sleep",
        "parameters": {
          "seconds": 1.0
        }
      },
      {
        "type": "launch",
        "parameters": {
          "command": [
            "pkill",
            "chrome"
          ]
        }
      },
      {
        "type": "launch",
        "parameters": {
          "command": [
            "google-chrome",
            "--remote-debugging-port=1337"
          ]
        }
      },
      {
        "type": "sleep",
        "parameters": {
          "seconds": 3
        }
      }
    ],
    "func": "exact_match",
    "result": {
      "type": "enable_do_not_track"
    },
    "expected": {
      "type": "rule",
      "rules": {
        "expected": "true"
      }
    }
  },
  "proxy": false,
  "fixed_ip": false,
  "possibility_of_env_change": "low"
}

After the agent stops, the checker runs its own steps before it reads anything. Steps the checker runs after the agent stops and before it reads anything, such as saving the open file, are the postconfig: collecting the papers before marking them.

Here the postconfig takes the focus off whatever box was active, waits 1 second, closes Chrome, starts it again with the same port, and waits 3 seconds. Then the getter enable_do_not_track reads the setting, and exact_match compares it with the text ‘true’.

The file does not say why it restarts Chrome. A likely reason is that a fresh start shows only the settings Chrome has kept, not ones still being typed. Checkers need this kind of scaffolding often. The paper lists remote-debugging ports for Chrome and for the VLC media player, an extension for the VS Code editor, and decrypting the Thunderbird email app’s account data.

The spreadsheet task: save before you read

The third file is a spreadsheet task. Its setup downloads a spreadsheet from a public file store into the machine’s home folder, /home/user/, and opens it. The instruction ends with a warning: “Finish the work and don’t touch irrelevant regions, even if they are blank.”

json · OSWorld libreoffice_calc/01b269ae….json{
  "id": "01b269ae-2111-4a07-81fd-3fcd711993b0",
  "snapshot": "libreoffice_calc",
  "instruction": "Fill all the blank cells in B1:E30 with the value in the cell above it. Finish the work and don't touch irrelevant regions, even if they are blank.",
  "source": "https://www.youtube.com/shorts/VrUzPTIwQ04",
  "config": [
    {
      "type": "download",
      "parameters": {
        "files": [
          {
            "url": "https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/libreoffice_calc/01b269ae-2111-4a07-81fd-3fcd711993b0/Student_Level_Fill_Blank.xlsx",
            "path": "/home/user/Student_Level_Fill_Blank.xlsx"
          }
        ]
      }
    },
    {
      "type": "open",
      "parameters": {
        "path": "/home/user/Student_Level_Fill_Blank.xlsx"
      }
    }
  ],
  "trajectory": "trajectories/01b269ae-2111-4a07-81fd-3fcd711993b0",
  "related_apps": [
    "libreoffice calc"
  ],
  "evaluator": {
    "postconfig": [
      {
        "type": "activate_window",
        "parameters": {
          "window_name": "Student_Level_Fill_Blank.xlsx - LibreOffice Calc",
          "strict": true
        }
      },
      {
        "type": "sleep",
        "parameters": {
          "seconds": 0.5
        }
      },
      {
        "type": "execute",
        "parameters": {
          "command": [
            "python",
            "-c",
            "import pyautogui; pyautogui.hotkey(\"ctrl\", \"s\");"
          ]
        }
      },
      {
        "type": "sleep",
        "parameters": {
          "seconds": 0.5
        }
      }
    ],
    "func": "compare_table",
    "expected": {
      "type": "cloud_file",
      "path": "https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/libreoffice_calc/01b269ae-2111-4a07-81fd-3fcd711993b0/Student_Level_Fill_Blank_gold.xlsx",
      "dest": "Student_Level_Fill_Blank_gold.xlsx"
    },
    "result": {
      "type": "vm_file",
      "path": "/home/user/Student_Level_Fill_Blank.xlsx",
      "dest": "Student_Level_Fill_Blank.xlsx"
    },
    "options": {
      "rules": [
        {
          "type": "sheet_data",
          "sheet_idx0": 0,
          "sheet_idx1": "EI0"
        }
      ]
    }
  },
  "proxy": false,
  "fixed_ip": false,
  "possibility_of_env_change": "low"
}

The checker’s postconfig brings the spreadsheet window to the front, waits half a second, presses Ctrl+S, and waits half a second again. Then compare_table compares the agent’s saved file (vm_file) with a gold file (cloud_file), by the cell values of their first sheets.

Why press Ctrl+S? Because the getter copies the file on disk, and an agent can finish every cell on screen and never save. Without the postconfig, that run would fail although the work is done. The state on screen and the state on disk are two different things, and only one of them is what the checker reads. The postconfig exists to close that gap.

Now read all three files the way the machine does: line by line, each line acting at one moment of the run.

Several checks in one task

Some tasks need more than one check. Then func is a list of metrics, and a field called conj says how to combine their results. Its two values work like the words and and or.

With "and", the default, every check must hold. The checker runs them in order. If any check returns 0, it stops at once and the task scores 0. If none does, the score is the average of the results, so partial credit survives.

With "or", any check may hold. If any check returns 1, the checker stops at once and the task scores 1. If none does, the score is the largest result.

Two more rules sit in front of every metric. If the task is one of the 30 infeasible ones from Chapter 1, it scores 1 only when the agent’s last action was FAIL. For every other task, a last action of FAIL scores 0 before any check runs. And a getter that finds no file makes the task score 0. Try the rules:

The whole rule is three lines of OSWorld’s evaluate(), inside the loop over the checks, and one line after it:

python · OSWorld desktop_env.py, evaluate()if self.metric_conj == 'and' and float(metric) == 0.0:
    return 0
elif self.metric_conj == 'or' and float(metric) == 1.0:
    return 1
...
return sum(results) / len(results) if self.metric_conj == 'and' else max(results)

Written as two rules, one for each value of conj:

r₁ … r_k

The results of the task’s k checks, each 0, 1 or a fraction.

and

Every check must hold: one 0 sinks the task; otherwise partial credit is the average.

or

Any check may hold: one 1 is enough; otherwise the best partial result counts.

‘and’ over 1, 0.5 and 1 = (1 + 0.5 + 1) ÷ 3 = 0.83

What all this costs to write

OSWorld has 369 tasks. 268 use one app, 101 span several apps, 84 were adapted from other benchmarks, and 30 are infeasible. Per app, they run from 15 tasks for the email app to 47 each for the spreadsheet and slides apps, plus the 101 that span several apps. Between them they need 134 different checking functions.

The authors timed the work. Each task’s setup took about 1 person-hour to build, and each checker about 2. The whole benchmark took about 1,800 person-hours, 400 of them spent on four rounds of double checks.

worked exampleWhy shared snapshots (the per-snapshot size is illustrative)
  starting states in the benchmark            = 302
  one saved copy each, at 2 GB (illustrative) = 302 × 2 GB = 604 GB
  shared instead                              = a few snapshots + a short list of setup steps per task

Combining checks (OSWorld's rule)
  'and', results 1 and 0.5  → no zero, so the mean: (1 + 0.5) ÷ 2 = 0.75
  'and', results 1 and 0    → a zero: stop at once, score 0
  'or',  results 0 and 0.5  → no one: the largest, 0.5
  'or',  results 1 and 0    → a one: stop at once, score 1
  last action FAIL, a normal task                  → 0, before any check runs
  last action FAIL, one of the 30 infeasible tasks → 1

What the checks cost to write
  369 tasks × (1 h setup + 2 h checker) = 369 × 3 h = 1,107 person-hours
  share of the team's ~1,800 hours      = 1,107 ÷ 1,800 = 61.5%

Where the bar sat at release
  humans 72.36%, best agent 12.24%  →  72.36 ÷ 12.24 = 5.9 times

So setup and checking alone took about three fifths of the whole effort. The rest went to writing the tasks, the double checks and the tools. People finished 72.36% of these tasks; the best agent when the benchmark came out, a language model reading the accessibility tree, finished 12.24%. People were 5.9 times better.

The checkers kept costing after release. After repairing the whole benchmark in 2025, the same team wrote: “providing reliable rewards consumes more human resources than we imagined.” Chapter 7 tells that story.

Three new fields

The 2025 files carry three new fields at the bottom. proxy says whether to send this task’s web traffic through a go-between server. fixed_ip says whether to give the machine a fixed internet address. possibility_of_env_change flags tasks whose outside world may change under them, such as a website that gets redesigned. A Windows benchmark, Windows Agent Arena, adapted about two thirds of its 154 tasks from these Linux files and kept the same file structure.

Every Monday you run your agent ten times on the same forum task: create a forum named sci_fi with the description ‘A wild place for sci-fi enthusiasts’. Week 1: 4 of 10 pass. Week 5: 0 of 10. Nobody touched the agent. You replay a week-5 run and watch it do everything right. Then you open the checker’s log: it looked for the description box and found nothing. Over the weekend, the website had been redesigned.

The weekly numbers are illustrative. The task is real: it is task 580 of the most used web test world, which this chapter takes apart. The story shows the two problems every web test world must solve. The web does not hold still, and it cannot be put back.

Why not test on the live web

The real website on the internet is the live site. Testing an agent on it looks like the most honest test possible. It is the opposite.

A live site changes under you. Pages are redesigned. Prices, posts and stock change every hour. Pop-ups and robot checks appear. Some sites block traffic that comes from data centres, which is where test machines usually live. The repair team behind the desktop benchmark lists every one of these as a reason that tasks broke.

Worse, a live site cannot be reset. If your agent posts a message, orders a product or deletes a file, that change is real. It stays there for the next agent, and for everyone else on the internet. Chapter 0’s leak becomes a leak you cannot even clean up.

Freeze a copy

So copy the site and run the copy yourself. The real painting stays in the museum; a perfect copy goes on the classroom wall, where you can touch it. A copy of a website that runs on your own machine, with its own fixed data, is a replica.

The most used web test world, WebArena, runs four replica sites on your own machine. There is a shop, a forum like Reddit, a copy of the GitLab code-hosting site, and the shop’s admin panel. Next to them sit a map, a calculator, a scratchpad and an offline copy of Wikipedia.

Their data is built into the replica in advance: products, posts and users that never change unless an agent changes them. Nobody redesigns a replica over the weekend. That fixes the first problem. The second, putting the site back after an agent has changed it, needs one more idea.

Restart between agents

Each replica site ships in a sealed package. A frozen recipe that holds a program and everything it needs is an image. A running copy made from it is a container. Think of a cake mould and the cakes it makes: you can eat a cake, but the mould stays the same.

That gives a reset that is both cheap and total. Throw the container away, start a new one from the image, and the original data is back, every product and every post. WebArena’s reset is exactly this: stop and delete the site’s container and start a new one from the original image. Its authors say it takes “from a few seconds to one minute” per site.

Notice where that reset lives. Every one of WebArena’s 812 task files says "require_reset": false. The reset is not written in the task file at all. It happens outside it, between agents, done by whoever runs the benchmark.

BrowserGym, the toolkit from Chapter 1, wraps that restart in one function, prepare_backend(). Its authors give the reason in plain words: it restores the sites “so that any alteration performed by a previous agent (e.g., changing the user’s bio description) is wiped off for the next agent to start in a clean, fresh server state.”

Nothing in WebArena changes at random from run to run. Each task is one fixed instance, with the same data every time. Compare three ways to run the forum task for ten weeks: on a frozen copy restarted before every run, on a frozen copy never restarted, and on the live site.

Tasks from templates

Writing 812 tasks one by one would take forever, so WebArena writes fewer and fills them in. A task with blanks, such as ‘What is the top-{n} best-selling product in {year}’, is a template, a form letter. Filling the blanks gives one task.

WebArena has 241 templates and 812 tasks: 3.37 tasks per template on average. 187 tasks use the shop, 182 the admin panel, 180 the code site, 109 the map and 106 the forum, and 48 cross from one site to another. Here is task 580, the forum task from the opening, as it sits in the benchmark’s task list:

json · WebArena test.raw.json, task 580{
 "sites": ["reddit"],
 "task_id": 580,
 "require_login": true,
 "storage_state": "./.auth/reddit_state.json",
 "start_url": "__REDDIT__",
 "geolocation": null,
 "intent_template": "Create a new forum named {{name}}, with a description of {{description}}, and include {{sidebar_list}} in the sidebar?",
 "instantiation_dict": {
  "name": "sci_fi",
  "description": "A wild place for sci-fi enthusiasts",
  "sidebar_list": ["New", "Classic", "Movies", "Post my novel", "Random"]
 },
 "intent": "Create a new forum named sci_fi, with a description of A wild place for sci-fi enthusiasts, and include ['New', 'Classic', 'Movies', 'Post my novel', 'Random'] in the sidebar?",
 "require_reset": false,
 "eval": {
  "eval_types": ["program_html"],
  "reference_answers": null,
  "reference_url": "",
  "program_html": [
   {
    "url": "__REDDIT__/f/sci_fi/edit",
    "locator": "document.querySelector(\"#forum_description\").value",
    "required_contents": {"must_include": ["A wild place for sci-fi enthusiasts"]}
   },
   {
    "url": "__REDDIT__/f/sci_fi/edit",
    "locator": "document.querySelector(\"#forum_sidebar\").value",
    "required_contents": {"must_include": ["New", "Classic", "Movies", "Post my novel", "Random"]}
   }
  ]
 },
 "intent_template_id": 7
}

Short lists are joined onto one line for width; every key and value is unchanged. __REDDIT__ is filled in with the address of your own forum replica when the benchmark runs.

Read it from the top. sites says which replica the task uses. require_login and storage_state say the agent starts already logged in, from a saved login, so it never has to type a password. start_url is the page the browser opens first.

intent_template is the form letter, with its blanks in double curly brackets. instantiation_dict holds what goes in each blank. intent is the result: the exact words the agent gets. require_reset is false, as in every task, because the restart happens outside the file.

Then eval, the checker. Its type is program_html, and it holds two checks. Both open the forum’s settings page. The first reads the description box; the second reads the sidebar box. The next section explains how.

Three ways to check a web task

A web task can be checked in three ways. Compare the agent’s answer text: a string match. Compare the address the agent ended on: a URL match. Or open a page and read a value from it: page content. Think of checking a student’s spoken answer, the room they ended up in, or the work left on their desk.

To read a value, the checker must find it on the page. The page as the browser holds it, every heading, box and button nested inside another, is the DOM (document object model). It is like the nested folders of a filing cabinet. A short piece of JavaScript, the language web pages run, that finds one thing on a page is a locator: a librarian’s call number.

Task 580’s first check opens /f/sci_fi/edit and runs document.querySelector("#forum_description").value. In plain words: find the box whose id is forum_description, and read what is typed in it. The check then requires that text to include ‘A wild place for sci-fi enthusiasts’. The second check does the same for the sidebar box and its five items.

String checks come in three kinds. exact_match wants the answer word for word. must_include wants some words inside it. fuzzy_match asks another AI model whether two answers mean the same thing. Another AI model asked to grade the answer is a model judge, a second examiner who reads for meaning.

Across all 812 tasks, 325 (40.0%) are checked by a string match alone, 282 (34.7%) by page content alone, and the other 205 combine page content, address and answer checks. When a task has several checks, its score is their product. So every check must pass, because a single 0 makes the product 0.

Each way of checking reads one thing, and is blind to everything else. Try all three on one forum task, against five ways a run could end.

The page-content check is the best of the three, and it still lets one wrong ending through. ‘Not a wild place for sci-fi enthusiasts’ includes every required word. Chapter 6 builds checkers that close gaps like this one.

How the agent sees and acts on the web

WebArena hands the agent the page as a screenshot, as the DOM, or as the accessibility tree from Chapter 1. Every element carries a number, so an observation line reads [1582] Add to Cart, and the action that presses that button is click [1582].

The agent has 12 actions: noop, click, hover, type, press, scroll, tab_focus, new_tab, tab_close, go_back, go_forward and goto. People finished 78.24% of a sample of these tasks, taking 110 seconds each on average. The best agent at release finished 14.41%: about one task in seven, where people managed about four in five.

Web tasks are quick for people next to desktop tasks. On a sample of WebArena tasks, people took a median of 35.38 seconds. On the desktop benchmark of Chapter 2 they took a median of 111.94 seconds, about three times as long.

One interface for many web worlds

BrowserGym runs WebArena, its picture-heavy sibling, and four other web benchmarks, all behind the one reset and step interface of Chapter 1. Each task inside it is a small class with two functions. setup(page) prepares the page and returns the goal. validate(page, chat_messages) checks the page after every action.

A library that drives a real web browser from code is Playwright, a puppeteer for a browser. BrowserGym turns each of the agent’s short calls into Playwright commands. Some of its benchmarks draw each task’s details from a seed, a number that makes random choices repeatable (Chapter 4 explains seeds). WebArena does not: its tasks are fixed.

A second design: copies of famous sites, reset in a click

Restarting a container is total, but it takes seconds to a minute and needs your own machine. A newer benchmark takes another route. REAL is a set of 11 replicas of well-known sites, among them a shop like Amazon, a calendar like Google Calendar and an email app like Gmail. They are hosted on the public web, and the benchmark has 112 tasks.

The replicas hold still by design: fixed data, fixed dates and time zones, and the ability to replay a run. And each site keeps its state in one place: a small notebook each website keeps inside your browser, where it can save data between visits. That notebook is the browser’s local storage, like a notebook a shop keeps behind its counter.

So resetting is one page visit. Going to the site’s /clear page empties its notebook back to the default. And checking compares the notebook at the start of the run with the notebook at the end. The difference is exactly what the agent changed, nothing more. REAL has four special pages:

Its checks are written as queries into that difference. A short query that picks one value out of a JSON record, written in a little language called jmespath, is a street address that names exactly one house. Here is a real task from the Amazon-like shop:

json · REAL omnizon-10.json{
  "id": "omnizon-10",
  "goal": "Click on \"buy now\" on any product, increase its quantity to the maximum allowed, update the delivery date to the last available, and place the order.",
  "website": {
    "id": "omnizon",
    "name": "Omnizon",
    "similarTo": "Amazon",
    "previewImage": "/websitePreviews/omnizon_preview.jpg",
    "url": "https://evals-omnizon.vercel.app"
  },
  "difficulty": "hard",
  "challengeType": "action",
  "possible": true,
  "evals": [
    {
      "description": "quantity of 30 (max)",
      "type": "jmespath",
      "expected_value": 30,
      "query": "orderDetailsDiff.added.\"0\".order.cart[0].quantity"
    },
    {
      "description": "latest delivery date",
      "type": "jmespath",
      "expected_value": "Monday, Jul 21",
      "query": "orderDetailsDiff.added.\"0\".order.selected_delivery"
    }
  ],
  "points": 1,
  "config": {},
  "version": "v1"
}

Read the first query aloud: in the difference between the notebook before and after, take the first order that was added, its cart’s first item, and its quantity. It must equal 30, the most the shop allows. The second query reads the delivery date of that same order. Every assertion must match exactly for a pass.

Questions with written answers cannot be checked by a query, so REAL sends them to a model judge with a rubric, and passes only grades above 0.8. Chapter 6 looks closely at that line.

Pictures

Some web tasks need eyes. VisualWebArena adds 910 tasks on a new classifieds site with 65,955 listings, plus the shop and the forum. Its checkers can ask an image model a yes-or-no question about a picture on the page, or compare two pictures for likeness. The task ‘Add something like what the man is wearing to my wishlist’ is checked by going to the wishlist and asking two questions about the item there: ‘Is this a polo shirt?’ and ‘Is this shirt green?’

Here are the counts behind this chapter, with the toy forum weeks next to them:

worked exampleWebArena, counted from its task files
  templates 241, tasks 812             → 812 ÷ 241 = 3.37 tasks per template
  per site: 187 + 182 + 180 + 109 + 106 + 48 cross-site = 812
  checks: answer text only 325 (40.0%), page content only 282 (34.7%),
          the other 205 mix page, address and answer   → 325 + 282 + 205 = 812
  humans 78.24%, best agent at release 14.41%          → 78.24 ÷ 14.41 = 5.4 times

The forum week, without a restart (illustrative)
  week 1, run 2 creates sci_fi; runs 3 to 10 find it already there → 9 of 10 pass
  weeks 2 to 10: the forum is there from the start                → 10 of 10 pass
  the agent's real rate all along                                 → 4 of 10

REAL, counted from its task files
  sites 11; tasks 14 + 11 + 11 + 10 + 10 + 10 + 10 + 10 + 9 + 9 + 8 = 112
  best agent at release 41.07% = 0.4107 × 112 = 46 tasks exactly

Your agent’s next job is on a phone: ‘Delete the following expenses from the expense app: Bus pass, Train ticket, Coffee.’ You want to run it a thousand times tonight, each time on a fresh phone with the same expenses in it. You cannot buy a thousand phones, wipe them by hand, and check each one’s screen by eye. So you build the phone too.

The three expense names are illustrative. The task’s shape is real: it is one of the phone test world’s expense tasks, and by the end of the chapter you will have read its checker line by line.

A phone made of software

Chapter 2 ran a desktop inside a computer. A phone works the same way. A phone simulated inside a computer is an emulator: a flight simulator, but for a phone. It runs the real phone software, draws the real screen, and answers real taps.

The phone test world in this chapter is AndroidWorld. It runs a simulated Pixel 6 phone on Android 13, with a screen of 1080 by 2400 pixels. It has 116 tasks across 20 real apps.

The apps are fixed versions from an open app store, and none of them needs a login. That choice removes two quiet sources of change. Nothing updates itself overnight, and no account can expire or stop to ask for a password.

A way in

To set up a task and to check it, the test world must reach inside the phone without touching its screen. A tap on the screen would be the agent’s job, and it would be slow and fragile besides.

A command channel from the computer into the phone, like a cable you can type into, is ADB, the Android Debug Bridge. Think of a service tunnel into a building: the staff use it, the visitors never see it. Through it the test world reads the phone’s files, its apps’ stored records and its settings.

Putting a phone back

Before every task, the task’s own setup function does three things. It sets the phone’s clock. It restores each app the task uses to a saved state. And it fixes the task’s random choices, which the next section explains.

Tasks that use an app’s stored records also empty them first, and the code says why in a comment. Here is the start of every task that works on an app’s records:

python · AndroidWorld sqlite_validators.pydef initialize_task(self, env: interface.AsyncEnv) -> None:
    """Initializes the task environment."""
    self._clear_db(env)  # In case the previous run crashed.
    super().initialize_task(env)
    self._clear_db(env)
    if NOISE_ROW_OBJECTS in self.params:
      self.add_rows(self.params[NOISE_ROW_OBJECTS], env)

The records are cleared twice: once in case the previous run crashed, and once after the parent class has done its own setup. Then the task adds its own rows.

That first line is Chapter 0’s lesson written into code. A run that crashed halfway never reached its clean-up step, so whatever it left must be swept away before the next run begins.

The clock deserves a paragraph of its own. Every task starts with the phone’s clock at 15:34 on 15 October 2023, in universal time (UTC, the world’s reference clock). That day was a Sunday.

Why pin it? Tasks say things like ‘tomorrow’ and ‘last week’. If the clock ran free, ‘tomorrow’ would mean a different date on every night you ran the test. With the clock pinned, ‘tomorrow’ always means Monday 16 October 2023. The same task means the same thing on every run.

A task is a small class

Each AndroidWorld task is a small Python class with a fixed set of parts. A template holds the task’s words with blanks. generate_random_params() chooses what goes in the blanks. initialize_task() sets up the phone. is_successful() is the checker; it returns 0, 1, or a fraction. tear_down() cleans up afterwards. The goal the agent reads is the template with its blanks filled.

Here is a whole task, as printed in AndroidWorld’s paper: send a text message.

python · AndroidWorld paper, App. D.2class SendSms(TaskEval):
  """Task sending and validating a text message has been sent.

  It checks the SMS telephony database, which is located at:
  /data/data/com.android.providers.telephony/databases/mmssms.db."""

  template = (
      "Send a text message using Simple SMS Messenger to "
      "{number} with message: {message}"
  )

  def initialize_task(self, env: interface.AsyncEnv) -> None:
    """Sets up the initial state of the task."""
    super().initialize_task(env)
    clear_sms_database(env.base_env)

  def is_successful(self, env: interface.AsyncEnv) -> float:
    """Checks if the SMS was sent successfully."""
    super().is_successful(env)
    messages = get_messages(env.base_env)
    return check_message_exists(
        phone_number=self.params["number"],
        body=self.params["message"],
    )

  def teardown(self, env: interface.AsyncEnv) -> None:
    """Clears the SMS database."""
    super().teardown(env)
    clear_sms_database(env.base_env)

  @classmethod
  def generate_random_params(cls) -> dict[str, Any]:
    number = generate_random_number()
    message = generate_random_message()
    return {
        "number": number,
        "message": message,
    }

Walk it from the top. The docstring names the file where the phone keeps its text messages. The template has two blanks, a number and a message. initialize_task() empties the message database. is_successful() reads the messages and asks whether one exists to the right number with the right text.

teardown() empties the database again. generate_random_params() draws a phone number and a message. Notice that setup and tear-down both empty the database, so no message from an earlier run can ever satisfy the checker.

Random, but repeatable

The details a task’s template draws at random, such as which three expenses to delete, are its parameters: the numbers filled into a practice problem. Drawing them at random means one task can be run in endless variants. An agent cannot pass by memorising one answer.

But a test must also be repeatable: run the same test twice, get the same task twice. Random and repeatable sound like opposites. A seed reconciles them.

The starting number for a random number generator is its seed. The same seed always gives the same ‘random’ draws. Think of shuffling a deck in exactly the same way every time: the order looks random, and you can deal it again.

AndroidWorld seeds Python’s random numbers from each task’s seed. Its authors ran their main experiments with seed 30. A run setting chooses how many different parameter draws to try per task. Draw a few tasks yourself.

Check the data, not the screen

The checker reads the apps’ own records, not the screen. A program’s organised store of records is a database. Records of one kind sit in a table, and each record is a row. Picture a spreadsheet with one line per expense: the spreadsheet is the table, each line a row.

Most phone apps keep SQLite, a small database stored in one file, like a notebook in the app’s drawer. Through ADB, the checker can open that file and read every row.

AndroidWorld’s authors chose this on purpose. In their words: “Using the underlying system state is much more durable than matching superficial UI changes.” A screen can be redrawn, scrolled or covered by a pop-up. The rows underneath are the truth.

So their checkers ask plain questions of the stored state. Does this calendar event exist? Does this text message exist? Does this file exist? Only when no record holds the answer do they read the screen, for example to see what a running timer displays. Tasks with two parts score a fraction: half for each part done.

The rows you must not touch

Now the expense task from the opening. Its harder version puts the 3 named expenses in the table, and 50 more besides. Extra rows put in the table on purpose, which the agent must leave alone, are noise rows: decoy files in a drawer.

The goal is built from the three drawn rows: ‘Delete the following expenses from …: a, b, c.’ So the table starts with 53 rows. How should the checker decide? Derive it one condition at a time, each one added because of an ending the last one missed.

1. The task names 3 rows. The obvious check asks: are those 3 gone? Call this the targets check.
2. An agent that deletes all 53 rows passes the targets check. To catch it, require every other row from the start to still be there: the keep check.
3. An agent that deletes the 3 rows and changes one amount in another row passes both, if ‘still there’ only means ‘a row with that id exists’. So compare whole rows, every field: a row that is still there must be identical.
4. An agent that deletes the 3 rows and adds a new one passes all of that. So require that nothing in the final table was absent at the start: the nothing new check.
5. Together: final rows = starting rows minus exactly the named ones. In counts: 53 − 3 = 50 rows, each identical to its starting row.

That is exactly the real checker. It lists every row when the task starts, lists them again at the end, and runs this function:

python · AndroidWorld sqlite_validators.pydef validate_rows_removal_integrity(before, after, ids, id_name) -> bool:
  """Validates that specified rows have been removed correctly from the rows list and that the remaining rows are unaltered."""
  for row_id in ids:
    if not any(row for row in before if getattr(row, id_name) == row_id):
      raise ValueError(f"row ID {row_id} not present in before.")

  # Validate the removal and intactness of other rows
  for row in before:
    # If the row ID is in the list of removed row IDs
    if getattr(row, id_name) in ids:
      if row in after:
        return False
    elif row not in after:
      # Make sure we didn't remove other rows.
      return False

  # Check that no new unexpected rows have been added
  for row in after:
    if row not in before:
      return False

  return True

The real signature carries type hints; they are dropped here for width. row in after compares whole rows, every field, so a changed amount counts as a missing row.

Read it as three loops. The first makes sure the named rows existed at the start; if one did not, the task itself is broken, and it says so. The second walks every starting row: a named row must be gone, and any other row must still be there, identical. The third walks every final row: none may be new.

Try five endings of one run against the obvious check and the full one.

Four of the five endings are wrong, and the obvious check passes three of them. Every one of those three is a run that did damage the task never asked for.

worked exampleThe phone world
  screen 1080 × 2400 = 2,592,000 pixels = 1.25 × the desktop's 2,073,600
  every task starts at 15:34 UTC on Sunday 15 October 2023 → 'tomorrow' is always Monday 16 October
  116 tasks across 20 apps; a seed draws each run's details, and the same seed draws them again

Deleting 3 named expenses from 53 rows (3 named + 50 noise)
  right ending:          53 − 3 = 50 rows, each identical to before     → pass
  'clear all':            0 rows: the 3 are gone, 50 others are missing → fail
  one extra deleted:     49 rows: a noise row is missing               → fail
  one amount changed:    50 rows, but one differs from before          → fail
  only 2 deleted:        51 rows: a named row is still there           → fail
  the targets check alone passes 4 of these 5 endings: 3 of them falsely

Scores
  best agent 30.6%, people 80.0%           → 80.0 ÷ 30.6 = 2.6 times
  same agent, older phone image: 28.4%      → 30.6 − 28.4 = 2.2 points lower

How a phone agent acts

A phone agent answers with a small JSON object, one per step. Its action_type is one of 13 words: click, double_tap, scroll, swipe, input_text, navigate_home, navigate_back, keyboard_enter, open_app, status, wait, long_press and answer.

Its other fields say where and what: index (an element’s number on the screen), x and y (a point on the screen), text, direction, app_name and goal_status. So a tap is {"action_type": "click", "index": 12}, and typing is {"action_type": "input_text", "text": "Coffee", "index": 7}; the numbers here are illustrative. The agent says it is finished with a status action.

Budget and results

Each task allows about twice the steps a person needed. The code sets the budget from a task’s complexity: up to 10 steps for complexity 1, and 11 to 20 for complexity 2.

The best agent at release finished 30.6% of the tasks; people finished 80.0%, about 2.6 times as many. The same agent on an older phone image, a Pixel 5 on Android 12, scored 28.4%. The phone itself is part of the test world too, and swapping it moved the score by 2.2 points.

Not every task changes anything. Some are questions, such as how much you spent on coffee last week. Their answers are worked out from the starting state by counting, adding or reading one value, so the checker knows the right answer before the agent starts.

You run your phone agent on all 116 tasks and it scores 27.6%. A colleague runs the very same agent the next morning, with a different random seed, and gets 33.2%. A third run gives 26.3%. Nobody changed a line of code. Which number is the agent’s score?

These three numbers are real. AndroidWorld’s authors reran one agent with three seeds and reported exactly these scores. Every test world in this lesson so far was reset perfectly, and still the score moves. This chapter finds out why, and how many runs it takes before a score can be trusted.

Where the wobble comes from

Four things can change between two runs of the same agent on the same tasks.

1. The task’s random details. A new seed draws different expenses, different names, different numbers (Chapter 4).
2. The model’s own choices. Given the same screenshot twice, a language model may not pick the same next action twice.
3. The world’s timing. A page that loads a little slower can change what the agent sees on its next step (Chapter 7).
4. The clock. A task that says ‘tomorrow’ changes meaning unless the clock is pinned, as AndroidWorld pins it.

A process that gives the same result every time from the same start is deterministic, like a vending machine that always drops the same snack for the same button. A test world aims to be deterministic. The agent inside it usually is not.

Pin what you can

The first defence is to pin every source you can. Fix the seed. Pin the clock. Restore the same snapshot before every run. And set the model’s temperature to 0.

A setting that controls how much chance a model uses when it picks its next word is its temperature. At 0 it always picks the most likely word. Think of how adventurous a cook is with a recipe: at 0 the cook follows it to the letter, every time.

AndroidWorld’s main experiments used seed 30 and temperature 0, in the authors’ words “for reproducibility”. BrowserGym’s authors write that temperature 0 “can reduce most stochasticity”, that is, most of the chance. The desktop benchmark’s own first agents ran at temperature 1.0, so reruns of them would wobble more.

Even pinned, it moves

Pinning helps, but it does not finish the job. AndroidWorld’s authors ran tasks 20 times each to find out how much a single result can be trusted. They found three things.

With one fixed seed, the agent “appears completely incapable of solving some tasks due to ‘bad luck’ with the seed”. With a new seed on each try, it solved those same tasks fairly often. And, “surprisingly, performance varies even with a fixed seed”. A fixed seed can hide what an agent can do, and the model adds wobble of its own.

See the wobble for yourself. The device runs a toy agent that truly passes 29% of tasks, on 116 tasks, again and again. Nothing about the agent changes between runs.

Why a score wobbles: coin flips

The device is not doing anything mysterious. Each task is like a weighted coin that lands ‘pass’ with the agent’s true rate, call it p. A score is the share of heads in N flips. Flip 116 coins at p = 0.29 again, and you get a slightly different share. The derivation of how much is short.

1. One task run is a coin flip: pass with chance p, fail with chance 1 − p.
2. A score over N tasks is the average of N such flips. If one flip is 1 or 0, its spread, which statisticians call its variance, is p(1 − p). That is largest at p = 0.5 and zero at p = 0 or 1.
3. Averaging N independent flips divides the variance by N. So the variance of the score is p(1 − p) ÷ N, and its typical distance from p is the square root of that.
4. Scores from many reruns pile up in a bell shape around p, and 95% of them land within 1.96 of those typical distances.

How far one score would typically move if you ran the whole test again is its standard error. Think of a coin flipped 100 times: you expect 50 heads, but 46 or 54 would not surprise you. For a score made of N pass-or-fail runs, the standard error is the square root of p(1 − p) ÷ N.

SE

The standard error: how far one score would typically move on a rerun.

p

The true pass rate, as a fraction: 0.29 for 29%.

N

The number of runs in the score.

√(0.29 × 0.71 ÷ 116) = 0.0421, so ±4.21 points

The worked number

At 29% on 116 tasks, the standard error is 4.21 points. Now hold the three real reruns up against it. Their mean is 29.03%. They sit 1.43, 2.73 and 4.17 points from that mean: every one within one standard error. The 6.9-point spread between the lowest and the highest is about what chance alone predicts.

About two standard errors either side of a score is where 95 of every 100 reruns would land. That range is a 95% band, the margin of error you see on an election poll. Here it runs from 20.7% to 37.3%: a band almost 17 points wide, for one run of one agent.

Every benchmark’s wobble

The same arithmetic applies to every headline score in this lesson. The table gives each one with its number of tasks, its standard error, and the half-width of its 95% band, which is 1.96 standard errors. A score of 30.6% ± 8.39 means the band runs from 22.2% to 39.0%.

Two patterns stand out. The smaller the benchmark, the bigger the wobble: the web benchmark’s 812 tasks give the tightest band, and the two benchmarks with about 115 tasks the widest. And a score near 50% wobbles most, because p(1 − p) is largest there; the repaired desktop score of 60.76% wobbles more than the original 12.24% on the very same 369 tasks.

How many runs you need

Now turn the formula around. Instead of asking how wide the band is, ask how many runs make it narrow enough. To pin a 40% score within ±5 points, 95 times in 100, solve 1.96 × √(p(1 − p) ÷ n) = E for n.

n

The number of runs needed.

1.96

How many standard errors cover 95 of every 100 reruns.

E

The half-width you want, as a fraction: 0.05 for ±5 points.

3.8416 × 0.24 ÷ 0.0025 = 368.8, so 369 runs

That happens to equal the desktop benchmark’s 369 tasks. It is a coincidence, but a useful one to remember: one pass over that benchmark pins a score near 40% to about ±5 points, and no better.

Telling two agents apart takes more, because both scores wobble, and so does the gap between them. You also want to see the gap when it is real, most of the time. The chance that a test notices a real difference of a given size is its power, the strength of a magnifying glass. The usual target is 80 in 100.

n

The number of runs needed for each of the two agents.

0.84

Adds the power: seeing a real gap 80 times in 100.

δ

The gap between two agents you want to see, as a fraction.

2

Both scores wobble, so the gap’s wobble is twice one score’s variance.

2 × 7.849 × 0.24 ÷ 0.0025 = 1,507 runs per agent

So to tell two agents 5 points apart near 40%, each needs about 1,507 runs: about four times the whole desktop benchmark, for each agent. The Statistics of Evaluation lesson derives these formulas from scratch, and Error Bars for Evals applies them to model evaluations. Try other targets:

Where the extra runs come from

A benchmark with seeds can rerun each task with new details. BrowserGym’s authors ran 5 seeds of each of 125 small web tasks, 625 runs, and 10 seeds of each of 33 work tasks, 330 runs. They ran the fixed benchmarks, WebArena with 812 tasks and VisualWebArena with 910, once each.

For a fixed benchmark, the only way to get more runs is to rerun the same tasks. That measures the model’s own wobble, but not the variety that new task details would bring. It is better than one run, and it is not the same as more tasks.

With few tries the honest band is wide. Suppose one agent passes a task 3 times in 20 tries and another passes it 8 times (illustrative counts). Using the method AndroidWorld’s authors used for their 20-try study, the 95% band for 3 of 20 runs from 5.2% to 36.0%, and for 8 of 20 from 21.9% to 61.3%. The two bands overlap from 21.9% to 36.0%, so 20 tries cannot tell those two agents apart on that task.

A design question: things that happen during a run

Suppose you want your calendar world to deliver a new invitation in the middle of a run, to test whether the agent copes with a surprise. When should it arrive? If it arrives at a fixed number of seconds, a slow run and a fast run see it at different steps. Two runs of the ‘same’ task then differ by an accident of timing.

So the arrival must come from the seed too: after step 3, say, for this seed, and after step 5 for the next. That is an illustrative design. None of the benchmarks in this lesson is documented as delivering surprises mid-run, but anyone who builds one meets this question first.

worked exampleOne run of 116 tasks at a true rate of 29%
  SE  = √(0.29 × 0.71 ÷ 116) = √(0.2059 ÷ 116) = √0.001775 = 0.0421 → ±4.21 points
  95% band = 29.0 ± 1.96 × 4.213 = 29.0 ± 8.26 → 20.7% to 37.3%

The three real reruns
  mean = (27.6 + 26.3 + 33.2) ÷ 3 = 29.03%
  distances from the mean: −1.43, −2.73, +4.17 points: each within one SE
  spread = 33.2 − 26.3 = 6.9 points

How many runs to pin 40% within ±5 points, 95 times in 100
  n = 1.96² × 0.4 × 0.6 ÷ 0.05² = 3.8416 × 0.24 ÷ 0.0025 = 368.8 → 369
  at 30%: 322.7 → 323        at 50%: 384.2 → 384

How many runs each, to tell two agents 5 points apart at 40% (95% confidence, 80% power)
  n = 2 × (1.96 + 0.8416)² × 0.24 ÷ 0.0025 = 2 × 7.849 × 96 = 1,507 per agent
  that is 1,507 ÷ 369 = 4.1 times the desktop benchmark, for each agent

Two single runs: 31% against 27% on 116 tasks
  SE of the gap = √(4.213² + 4.213²) = 5.96 points
  the gap is 4 ÷ 5.96 = 0.67 of one SE: well inside the noise

Here is the same arithmetic in a few lines of Python, with its output:

pythonimport math
p, N = 0.29, 116
se = math.sqrt(p * (1 - p) / N)
print(round(100 * se, 2), round(100 * (p - 1.96 * se), 1), round(100 * (p + 1.96 * se), 1))
print(round(1.96 ** 2 * 0.4 * 0.6 / 0.05 ** 2))
print(round(2 * (1.96 + 0.8416) ** 2 * 0.4 * 0.6 / 0.05 ** 2))
# output
4.21 20.7 37.3
369
1507

A colleague sends you a new agent with a note: ‘It scores 8% on the desktop benchmark, and it never clicks anything.’ You read the code. For every task, it answers at once: FAIL, this cannot be done. How does an agent that does nothing score 8%?

The answer

Remember Chapter 1: 30 of the desktop benchmark’s 369 tasks cannot be done on purpose. They are the infeasible tasks, set to catch an agent that bluffs. And remember Chapter 2’s rule: on an infeasible task, a last action of FAIL scores 1.

So an agent that answers FAIL to everything collects all 30 of those tasks and nothing else. That is 30 ÷ 369 = 8.13%. The best agent when the benchmark came out scored 12.24%, so the do-nothing agent reaches two thirds of it without a single click.

The web benchmark has a cousin of the same trap. 36 of its 812 tasks expect the answer ‘N/A’, because what they ask for does not exist. Those answers are graded by a model judge, which may or may not accept an answer that is only ‘N/A’. So an agent that answers ‘N/A’ to everything collects at most 36 ÷ 812 = 4.4%. Guess the first bar before it grows:

Two kinds of error, named

Chapter 0 gave the plain names. The formal names are these: a false pass is a false positive, and a false fail is a false negative. A checker is a test with two error rates, like a smoke alarm. It sometimes rings for toast, and it sometimes stays quiet in a fire.

The do-nothing agent is not a bug in the checker’s code. The checker is doing exactly what it was told, for an answer nobody expected an agent to give everywhere. That is the uncomfortable part: a checker can be correct line by line and still hand out free points.

The desktop benchmark’s own authors saw a milder version in real agents. In one setting, agents scored 16.67% on the impossible tasks against 13.34% on the possible ones. In the authors’ words, part of the reason is that agents “tend to easily output FAIL and refuse to continue trying”. Some of those impossible-task passes were agents giving up, not agents spotting the trap.

What a noisy checker does to a score

Every real checker makes some mistakes of both kinds. What do they do to a score? Derive it in five lines.

1. Split the runs into those the agent really did, a share p, and those it did not, a share 1 − p.
2. Of the runs it did, the checker passes all but its false fails. Call FN the share of good runs it fails. It passes p × (1 − FN).
3. Of the runs it did not, the checker passes its false passes. Call FP the share of failed runs it passes. It passes (1 − p) × FP.
4. The reported score is the sum of the two.
5. Its distance from the truth is (1 − p) × FP minus p × FN. It is positive, an inflated score, when false passes dominate. That happens for weak agents, which have many failed runs to be falsely passed. It is negative for strong agents.

p

The true share of runs where the agent did the task.

FN

False fail rate: the share of good runs the checker fails.

FP

False pass rate: the share of failed runs the checker passes.

reported

The score the checker prints.

0.40 × 0.95 + 0.60 × 0.10 = 0.44: a true 40% reads as 44%

Work three agents through it with an illustrative checker that passes 10% of failed runs and fails 5% of good ones. A true 10% agent reads as 18.5%: it gains 8.5 points. A true 40% agent reads as 44%. A true 90% agent reads as 86.5%: it loses 3.5 points. The gap between the weakest and the strongest shrinks from 80 points to 68.

That is the pattern. False passes lift weak agents a lot, and false fails cap strong agents. A noisy checker pulls every score toward the middle and squeezes the gaps between agents, which are exactly what a benchmark exists to measure.

What each kind of checker reads

Every checker in this lesson reads one thing. Each is blind to something different. Here are the families met so far, side by side:

Model judges

A model judge reads meaning, so it forgives different words. That fixes some false fails: ‘the cheapest is North Air’ and ‘North Air has the lowest price’ can both pass (an illustrative airline). But a judge brings errors of its own.

WebArena’s looser string checks are graded by a language model. REAL’s judge is told “Don’t be too strict”, grades each answer from 0 to 1 against a rubric, and passes it only above 0.8. A short written description of what a right answer must contain is a rubric: the examiner’s notes on what earns marks.

The line sits exactly at ‘greater than’. A grade of 0.80 fails and a grade of 0.81 passes, though no reader could tell those two answers apart. The JEV as a Judge lesson studies how such judges go wrong.

Now find the blind spots yourself. Each of the four cases below pairs a checker as written with a sturdier one, and two ways a run could end.

Every sturdier version does the same thing in a different place. It moves the check closer to what the task really means: a program that runs, not a file that exists; the exact words asked for, not some of them; the value a formula computes, not how it is spelled.

Why it matters even more in training

In a test, a blind spot adds a few points to a score. In training it does far more damage, because the checker’s verdict is the reward. An agent that is changed to earn more reward will find the blind spot, if the blind spot is easier than the task.

An agent that learns to satisfy the checker instead of doing the task is reward hacking. Think of a student who learns the examiner’s habits instead of the subject. The do-nothing agent from the opening is a reward hack written by hand. A trained agent can find one on its own, and nobody tells it that it cheated.

Designing a checker that cannot be fooled

Put the lessons of the last four chapters together and a checker needs seven habits. The Field Guide repeats them as a checklist.

1. Read the state the task changes, not the agent’s words about it.
2. Read the whole change: what should change, and what must not (Chapter 4’s whole-table check).
3. Say what must be absent too. The picture benchmark added a must_exclude check, and the Spotify checker excludes ‘not found’.
4. Accept every right answer. The desktop benchmark’s repair added extra gold answers and gave full marks if any matched, after ‘functionally equivalent’ spreadsheet formulas were being marked wrong.
5. Run known agents through it before any real one: a do-nothing agent, an always-FAIL agent, a reference agent that does the task by script, and an agent that does the task plus a side effect. A checker is not finished until it fails the first, second and fourth, and passes the third.
6. Read a sample of passes by hand, and count the false passes you find.
7. Keep the checker’s code and expected values out of the agent’s world, so the agent cannot read the answer key.

worked exampleThe do-nothing agent (real counts, derived from the benchmarks' own rules)
  desktop: 30 impossible tasks pass on FAIL → 30 ÷ 369 = 8.13%
           best agent at release 12.24%     → 8.13 ÷ 12.24 = 0.66: two thirds of it, with zero clicks
  web:     36 tasks expect 'N/A'            → at most 36 ÷ 812 = 4.4%
           best agent at release 14.41%     → 4.4 ÷ 14.41 = 0.31 of it

A noisy checker (illustrative: 10% of failed runs passed, 5% of good runs failed)
  true 40%:  0.40 × 0.95 + 0.60 × 0.10 = 0.380 + 0.060 = 0.440 → reads 44.0%  (+4.0)
  true 10%:  0.10 × 0.95 + 0.90 × 0.10 = 0.095 + 0.090 = 0.185 → reads 18.5%  (+8.5)
  true 90%:  0.90 × 0.95 + 0.10 × 0.10 = 0.855 + 0.010 = 0.865 → reads 86.5%  (−3.5)
  gap between the 10% and 90% agents: true 80 points, reported 86.5 − 18.5 = 68 points

The model judge's line (REAL's code: pass if grade > 0.8)
  grade 0.80 → fail        grade 0.81 → pass

Last week your agent passed a word-processor task 16 times out of 20. This week, on cheaper machines, it passes 11. Nobody touched the agent. You replay one of the failures: the document opens, and the cursor sits on line 1, though the task’s setup was supposed to put it on line 9 before the agent ever started.

The counts are illustrative; they come from the device below. The setup step is real: the desktop benchmark’s repair team described exactly this one. This chapter is about failures that belong to the test world, and how to tell them from failures that belong to the agent.

Two kinds of failure

A run that passes or fails for reasons that have nothing to do with the agent is flaky, like a light switch that works most mornings. Flaky runs make a score wobble in a way no rerun formula from Chapter 5 accounts for, because the cause is not chance in the agent. It is a fault in the world.

A failure caused by the test world itself is an infrastructure fault. One caused by the agent’s own choices is an agent fault. Think of a broken clock in the exam room against a wrong answer on the paper. A score mixes both unless you separate them, and only one of them says anything about the agent.

The race

Here is the real setup step from the opening: pyautogui.press("down", presses=8, interval=0.01). In plain words: press the down arrow eight times, 10 milliseconds apart. Starting from line 1, eight presses land the cursor on line 9.

The step quietly assumes something. It assumes the word processor has finished loading, with the document open and the cursor on line 1, before the first press. Nothing in the setup makes sure of that. On a fast machine the app is usually ready in time. When the app was slower to load, the presses arrived while it was still opening, and they went nowhere.

Two things that must happen in order, with nothing making sure they do, form a race condition. Think of two people reaching for the same door from both sides: usually one gets there first, and sometimes they collide. Here the two racers are the app loading and the keys being pressed.

The device below runs twenty starts of this task. The agent in it is perfect: whenever the run starts right, it does the task. Every failure you see is the world’s.

Measure before you blame

How do you tell the two kinds of failure apart without watching every replay? Check the starting state before the agent’s first action. The setup promised a particular start; check that it delivered.

The simplest check compares the first observation, before the agent moves, with what the setup should produce: here, the cursor on line 9. A stronger check reads the state itself and computes its fingerprint: a short code calculated from the files, rows and settings the task depends on. Two identical states always give the same fingerprint, and almost any difference changes it. Compare it with the fingerprint the setup should have made.

A run whose start is wrong is an infrastructure fault. Set it aside and rerun it; never score it against the agent. There are other signs of an infrastructure fault worth logging on every run:

• An error that comes back from the action executor. BrowserGym returns it to the agent as last_action_error; log it for yourself too.
• A checker that crashed, or that found no file. The desktop benchmark scores a missing file 0, so the agent takes the blame for whatever went wrong.
• A setup step that raised an error before the agent started.

The fix: wait for events, not for time

The tempting fix is a longer wait. The right fix is a different kind of wait. A step that waits until its job is truly done is blocking. One that starts the job and moves on at once is fire-and-forget. Think of waiting for the kettle to boil, against switching it on and walking away.

The desktop benchmark’s step that opens a file was fire-and-forget: it asked the app to open the document and moved straight on to the key presses. The repair team made it blocking, so the key presses wait until the file is open. They also re-tuned the pauses between actions for each size of rented machine they used.

The device shows why a longer fixed wait is the wrong fix. A fixed 2 seconds works on both machines here, but it spends 40 seconds of waiting on every 20 runs, needed or not. And a still slower machine would break it again. Waiting until the file is open is never too short and never longer than needed. It is both correct and the fastest.

Small faults compound

One flaky step can look harmless. A run is a chain of steps, and a chain is only as clean as all its links. If each step goes through cleanly with chance r, a run of T steps goes through cleanly with chance r multiplied by itself T times:

r

The chance that one step goes through with no infrastructure fault.

T

The number of steps in the run: 15 at the desktop benchmark’s default cap.

P(clean run)

The chance that the whole run is free of infrastructure faults.

0.99¹⁵ = 0.860: one hiccup per hundred steps spoils 14% of runs

At 15 steps, 0.90¹⁵ = 0.206, 0.95¹⁵ = 0.463, and 0.99¹⁵ = 0.860. Even one hiccup in a hundred steps, an illustrative rate, spoils 14% of 15-step runs. A world that is right 90% of the time per step finishes only one run in five cleanly. Infrastructure must be far more reliable per step than the agent, or its faults swamp the score.

A benchmark repaired in public

The desktop benchmark’s own team did this work at scale. They released a repaired version, OSWorld-Verified, after collecting, checking and fixing more than 300 problems reported by its users. It took about two months, for a team of about ten people.

The causes they list read like this lesson’s table of contents. Websites changed, and robot checks and blocked addresses appeared (Chapter 3). Setup steps raced the apps they set up (this chapter). Some instructions were unclear. And some right answers were ones the checker did not expect, such as different but equivalent spreadsheet formulas (Chapter 6).

Their rule was to change checkers, not tasks, so that old and new scores stay comparable. The 369 tasks stayed, with repaired checkers and instructions. Files moved from a cloud drive to a public dataset store. Some websites were swapped for others, and booking windows in some tasks were widened from 4 months to 8 or more, so the dates in a task would not run out.

The machine image shrank from 50 GB to 25 GB, because the office apps took a minute to open the first time on a fresh rented machine. Leaderboard entries must now be run by the benchmark’s maintainers, not reported by their authors.

On the repaired benchmark, the best agent reported by July 2025 reached 60.76%, against 72.36% for people: 84% of the human rate. And one model’s score ranged from 9.1% to 23.0% depending only on how many steps it was allowed. The step cap is part of the world too. The team’s own summary: “providing reliable rewards consumes more human resources than we imagined.”

Many worlds at once

One full run of a benchmark is slow. The Windows benchmark’s authors estimate 10 to 20 steps per task, at 10 to 30 seconds per step, so a run done one task after another takes “one or more days”. And one ordinary computer can hold only a few desktop test worlds at once, because each one is a whole virtual machine.

Renting computers by the hour in someone else’s data centre is the cloud: renting a hall instead of building one. Running about 50 test worlds in parallel on rented machines brought a full Windows run down to as little as 20 minutes. Each machine was an 8-core computer listed at $0.384 an hour.

The desktop benchmark followed the same path: from 8 to 16 worlds on one server, to up to 50 on rented machines, and from 10 hours or more to minutes. Budget a run yourself:

worked exampleThe flaky setup (illustrative load times for 20 runs, from 0.58 s to 1.31 s on the fast machine)
  fast machine, fixed 1 s wait:      4 of 20 starts wrong (runs 5, 9, 13, 18) → 16 of 20 pass
  slower machine (loads × 1.25):     9 of 20 starts wrong                    → 11 of 20 pass
  fixed 2 s wait:                    0 wrong on both, waiting 20 × 2 = 40 s in all
  wait until the file is open:       0 wrong, waiting 16.73 s (fast) or 20.91 s (slower) in all

Small faults compound over a 15-step run
  0.90^15 = 0.206      0.95^15 = 0.463      0.99^15 = 0.860
  one hiccup per 100 steps (illustrative) → 1 − 0.860 = 14% of runs spoiled

A full run, serial against parallel (20 s per step is illustrative)
  369 tasks × 15 steps × 20 s = 110,700 s = 30.75 hours on one machine
  on 50 machines at once: 110,700 ÷ 50 = 2,214 s = 36.9 minutes
  reported: 10 hours or more down to 20 minutes = 30 times faster
  50 machines × $0.384 an hour × 20 ÷ 60 hours = $6.40 at the listed price

The repaired benchmark
  best reported agent 60.76%, people 72.36% → 60.76 ÷ 72.36 = 0.84 of the human rate

A test world for a computer-using agent has four parts: a starting state that a reset puts back, an observation, an action executor, and a checker that reads the final state. The desktop, web and phone worlds in this lesson differ mainly in how they reset (revert a virtual machine and run its setup steps; restart a container; restore its apps and pin its clock; empty the browser’s notebook) and in how and when they check (once at the end, or after every step; a file, a page value, the agent’s own answer, a database’s rows, or a model acting as judge). When a score surprises you, look there first: at a leak in the reset, at a blind spot in the checker, at a setup step that raced the app it was opening, and at a wobble you never measured.

The checklist

Run it in order the first time you build a test world of your own, and reread items 8 and 12 before you trust any score it hands back.

1. Every run starts from the same state: reset before every run, and compare a fingerprint of the starting state with the expected one before the agent’s first action.
2. The world holds still: a frozen replica instead of a live site, fixed app versions, a pinned clock. A task that must touch the live world is flagged as one that may change.
3. Random task details come from a seed, and the seed is recorded with the score.
4. You know where the reward comes out: from step() after every action, or only from evaluate() at the end.
5. The checker reads the state the task changes: a file, a setting, database rows, the browser’s local storage. Not the screen, and not the agent’s own words about it.
6. The checker reads the whole change: what should change, and that nothing else did.
7. The checker says what must be absent, and accepts every right answer, not one memorized string.
8. Before any real agent runs, the checker has failed a do-nothing agent, an always-FAIL agent and a perfect-plus-damage agent, and passed a scripted perfect agent.
9. Someone has read a sample of the runs the checker passed, by hand, and counted the false passes.
10. Setup steps wait for events, such as a file finishing its open, never for a fixed number of seconds.
11. Infrastructure faults, a wrong start or a checker that crashes, are logged and rerun, never scored against the agent.
12. Every score is reported with its number of runs and its standard error, and no gap smaller than that wobble is called a win.
13. The step cap, the machine and the model’s temperature are reported with the score: each one changes it.

The numbers, and where each one comes from

The test worlds, compared

The core as code

One sketch carries the whole lesson. Its names are ours, not any library’s, but its shape is every benchmark in this chapter: a seeded, fingerprinted reset, a loop that ignores most step rewards, one checker call at the end, and a standard error reported with the score.

python · a sketch, not a real libraryimport hashlib, json, math, random

def fingerprint(state: dict) -> str:
    """A short hash of the state a task depends on: files, rows, settings."""
    return hashlib.sha256(json.dumps(state, sort_keys=True).encode()).hexdigest()[:12]

def run_benchmark(env, agent, tasks, seeds, max_steps=15):
    scores, infra, notes = [], [], []
    for task in tasks:
        for seed in seeds:
            random.seed(seed)                                  # the task's random details
            obs = env.reset(task, seed=seed)                   # 1. a clean, seeded start
            if fingerprint(env.read_state(task)) != task.expected_start(seed):
                infra.append((task.id, seed, "wrong start"))  # the world's fault: rerun it, never score it
                continue
            for t in range(max_steps):                         # 2. the loop
                action = agent.act(obs)
                obs, _, done, info = env.step(action)          # step rewards ignored: some worlds always say 0
                if info.get("action_error"):
                    notes.append((task.id, seed, t, info["action_error"]))  # read these: agent or world?
                if done:
                    break
            try:
                scores.append(env.evaluate())                  # 3. one verdict, from the final state
            except Exception as e:
                infra.append((task.id, seed, f"checker crashed: {e}"))
    n = len(scores)
    p = sum(scores) / n
    se = math.sqrt(p * (1 - p) / n)                            # the score's wobble
    return {"score": p, "se": se, "runs": n, "infra": infra, "notes": notes,
            "seeds": list(seeds), "max_steps": max_steps}

Walk it in two sentences. The three numbered steps are the contract from Chapter 1: reset, loop, evaluate. Everything else is this lesson’s defences: the seed, the start fingerprint, the infrastructure log, and the standard error reported with the score.

Sources and their setups

Where this lesson sits

This lesson opens Act 5 of the Computer Use & Multimodal Agents path, the worlds agents are tested and trained in. Just before it comes The Model Is Not The Computer: Prime Agent, the harness that drives an agent. Next comes the OSWorld paper lesson (coming soon), which walks the desktop benchmark from Chapter 2 end to end, and then Tools, Exec-Approvals and Sandboxing, on fencing in what an agent’s actions can touch. Later in this act: Sandbox Fleets (coming soon), on running a thousand isolated desktops that never leak, then τ²-Bench, where the user can change the shared state too, and EnvHarness, which reshapes test worlds through their reset and step functions.

Around it: microMDP and microPOMDP build the formal model behind Chapter 1; Agent Evaluation builds graders and trial counts for tool-using agents; The Statistics of Evaluation and Error Bars for Evals go deeper on Chapter 5; JEV as a Judge studies model judges from Chapter 6. Coming soon in this campaign: Computer-Use Agents (the loop from the agent’s side), Trajectory Data (where training runs come from), and Agentic RL (training an agent on these worlds’ rewards).

Go back to the calendar at the top of the page: keep the last world, then switch the checker. Every part now has a name. The calendar is the state of Chapter 1; Before each run is the reset machinery of Chapters 2 to 4; The checker is Chapters 4 and 6; and the twenty squares are Chapter 5’s wobble in miniature. Then open Teach and draw the four parts of a test world from memory, with the two places a score can lie.