OSWorld

You ask a friend to install Spotify on your laptop while you make tea. When you come back, how do you know it worked? You do not watch a video of every click. You open the list of apps and look for Spotify. If it is there, the job is done, however your friend did it.

Now replace the friend with a program. That is what this lesson is about: how to test a program that uses a computer for you, and what such a test found when people and programs took it side by side.

A new kind of helper

Start with the program. Think of autocomplete that has read a whole library. You type the start of a sentence and it guesses the rest, one word at a time, and it has seen so much text that its guesses are often good. Chatbots like ChatGPT are built on one of these. It is a program trained on a huge amount of text to continue it: give it a question and it writes an answer, one word at a time. It is called a language model.

Now give that reader eyes. A multimodal model is a language model that also takes pictures as input, so it can be shown a screenshot. It reads the photos as well as the words.

Picture a helper sitting at your desk with your hand on the mouse. Build a program around such a model that works a computer for you: it looks at the screen, picks one action, does it, and looks again. It is called an agent. An action is small and exact: move the mouse to a point and click, type some letters, press Enter.

Why build one? An agent like this could do the boring parts of a job: change the setting, fill in the spreadsheet, file the email. But only if it finishes reliably. Nobody hands their computer to a helper that gets it right one time in ten. So we need a number for “reliably”, and a fair way to measure it.

A test everyone shares

Think of a standard driving test. Every learner drives the same kind of route and the examiner fills in the same sheet, so two drivers' results mean the same thing. For agents, a fixed set of test jobs plus a fixed way of scoring them is a benchmark. Two teams that run the same benchmark can compare their numbers.

One job in the benchmark is a task: an instruction in plain English plus the computer it starts on. “Install Spotify”, given a computer without Spotify, is a task. One question on the test.

Grading by the recording

How should the examiner score a task? Many earlier agent tests used the obvious answer. An expert does the job once while a program saves every click and every key press. That saved list is a recording: an answer key that holds only one answer. The agent is then graded on how many of its steps match the recording.

This grading has two problems. The first is that most jobs have more than one right way. You can install Spotify from the app store, clicking through it. Or you can open a terminal, a window where you type commands as text instead of clicking, and type one command. Both leave Spotify installed. A grader that wants the recorded route fails the second helper anyway. The paper makes exactly this point about earlier benchmarks built from recordings: their scores punish valid alternative solutions.

The second problem is the other way round: matching steps is not the same as finishing. A helper can copy the first four steps of the recording and then stop. It has matched more than half of the recording, and nothing is installed.

Try both problems yourself. The device grades three helpers on the same job, two ways at once.

Look at what went wrong, in general terms. Say a job has k valid routes and the recording holds one of them. A grader that demands the recorded route fails every helper on the other k − 1 routes: correct work, scored as failure. A grader that gives credit for each matched step pays a helper that walks half the route and stops: no result, scored as progress.

A check on the finished computer is immune to both mistakes, because it never looks at the route. It only asks whether the thing you wanted actually happened.

Grading by the result

So grade the result instead, the way you checked your laptop for Spotify. For each task, write a short program that inspects the computer after the agent stops and returns a score. Think of a building inspector who walks through the finished house and never reads the builder's diary. That program is a checker. Grading by running a check on the real result is called execution-based evaluation.

Here is the real checker of the Spotify task, in words; Chapter 4 shows it as code. After the helper stops, the checker runs one command, which spotify. It prints where the Spotify program lives, or nothing at all if Spotify is not installed. If the printout contains the word “spotify” and does not contain “not found”, the score is 1. Otherwise it is 0.

The cost is real. Somebody must write a checker for every single task, by hand, and make sure it cannot be fooled. Chapter 4 counts the hours.

A real computer, not a toy

Other earlier tests did run checks on the result, but in small, simplified worlds: plain web pages, or one app at a time. Those worlds cannot hold a job that crosses apps, such as “take the numbers from this email and put them in a spreadsheet” (our example). Real office work crosses apps all the time.

So the paper uses a real computer. Two words first. The software that runs all other software and draws the desktop is the operating system: the building's foundation and plumbing. Windows, macOS and Linux are operating systems. Ubuntu is a free, popular version of Linux.

Picture a ship in a bottle that really sails. The paper runs Ubuntu inside a virtual machine: a whole computer imitated by software, running as one program on a real computer. Chapter 2 opens the bottle. Inside it, eight everyday apps are installed: Chrome, VLC, Thunderbird, VS Code, LibreOffice Calc, Writer and Impress, and GIMP.

The benchmark is called OSWorld. It has 369 tasks on Ubuntu, each with its own setup and its own hand-written checker.

What the test found

Run an agent on all 369 tasks and count the share that the checker scored as done. That share is the success rate, a pass rate like a driving school's. Some checkers give part credit, so “done” can be a fraction of a task; Chapter 1 shows how.

People finished 72.36% of the tasks: 267 of 369. The best model of the day finished 12.24%: about 45 tasks' worth. When two percentages sit on the same ruler, the plain difference between them is measured in percentage points. Here it is 72.36 minus 12.24, which is 60.12 percentage points. Put another way, people finished almost six times as many tasks.

The paper sums up why in one line. The models struggled most with finding the exact spot to click, and with knowing how each app works. Its own words are “GUI grounding and operational knowledge”. The GUI is the windows, buttons and menus you click, the graphical user interface: the dashboard of a car. Grounding and operational knowledge get proper definitions in Chapters 3 and 9.

The hero at the top of the page showed one of each. The AI seeing pixels knew to pick Bing and clicked one row low: a grounding miss. The AI reading text searched Privacy, Appearance and Autofill and never found the setting: a gap in knowing the app.

worked exampleGrading one helper two ways (the install task; helpers and steps illustrative)
  the recording, 7 steps:  open Ubuntu Software · click search · type "spotify" · open the result
                           click Install · type the password and Enter · DONE
  a terminal helper:       open a terminal · type "sudo snap install spotify" and Enter
                           type the password and Enter · DONE
  steps that equal the recording at the same position:  0 of 7    → recording grade: fail
  the checker:  which spotify  →  "/snap/bin/spotify"
                contains "spotify": yes · contains "not found": no   → 1

The gap at release
  people       72.36% of 369 = 267 tasks
  best model   12.24% of 369 = 45.17 task credits, about 45 tasks (some tasks give part credit)
  gap          72.36 − 12.24 = 60.12 percentage points
  ratio        72.36 ÷ 12.24 = 5.91: almost six times as many

A grade that disagrees with the computer

What happens. A team grades its helper against recordings. On the install task one helper scores 0, though Spotify is on the machine. Another helper on the same task gets 57% of the steps right, though nothing was installed.

What you measure. Open the finished computers. Run which spotify on both. The first prints a path; the second prints nothing. The grade and the computer disagree, in both directions.

What you change. Grade the final state. A short check that runs on the finished computer cannot be fooled by the route the helper took.

Here is the road, in order. Each chapter explains one part of the instrument at the top of the page.

1. The rules of one task: states, views, actions, the single score at the end, and the laziest way to score.
2. A real computer in a box: snapshots, setup, the server inside the machine, and saving before the check.
3. What the agent sees, and how it acts: a picture or a text list in, one line of code out.
4. A checker for every task: real task files, read line by line, and how to fool a weak checker.
5. 369 tasks and the people who tried them: the wall of tasks, and how people did.
6. Four ways to show the screen: what each model was shown, and what one look costs.
7. Reading the scores honestly: every result, with error bars.
8. Small changes, big drops: smaller screenshots, longer memory, a moved window.
9. Right plan, wrong pixel: the click that lands one row low, and when knowing a shortcut wins.
10. Fixing the test: 300 repairs and the scores that came after.
11. Map: the numbers, the checklist, the run loop, and where this sits on the path.

Replay the person from the top of the page, slowly. Six steps. After the fifth, the screen shows “Bing” in a box. But the checker does not read that box. It reads a settings file on the disk, a file the person never saw and the agent never sees.

That gap, between what is on the screen and what is in the computer, is the first rule of every task in the benchmark. The agent sees the screen. The checker reads the computer. They are not the same thing, and the score arrives only once, when the agent stops. This chapter writes those rules down, one piece at a time, and then uses them to find the laziest way to score.

The screen is not the computer

Think of a card game where the deck lies face down. The whole deck is everything about the game at one moment. In the same way, everything about the computer at one moment is its state: every file, every setting, every open window, every program running in the background, what is in memory. The state is the whole deck, face down.

The agent never gets the deck. It gets a hand. At each step it is shown the instruction plus a view of the screen: a screenshot, a text list of what is on the screen, or both. That part of the state is its observation: the cards in its hand. The agent never sees the whole state, so the problem is called partially observable.

This matters because the checker is free to read any part of the state, including the parts the agent cannot see. In the Bing task the checker reads Chrome's settings file. In a spreadsheet task it reads the saved file on disk. The agent has to change the hidden cards by playing the visible ones.

Moves

A move in this game is one thing the agent sends: a click at a point, some typing, a key press. That is an action. One round of look, decide, act is a step, one move in the card game.

The agents write each action as one line of Python. They use a library that moves the mouse and presses keys from code, a remote control you operate by writing. The library is called pyautogui. Here are five actions from the paper's own table, each in plain words:

• pyautogui.click(x, y) clicks at a point on the screen.
• pyautogui.write('text') types some text.
• pyautogui.press('enter') presses one key.
• pyautogui.hotkey('ctrl', 'c') presses keys together, here Ctrl and C.
• pyautogui.scroll(-200) scrolls the page.

After each action the computer answers: a window opens, a menu closes, a file changes. The dealer turns over the next card. The new state that an action leads to is the transition. Some transitions show on the screen. Others happen only inside the computer, where the agent cannot look.

Step through the winning run yourself and watch both sides. The left side is all the agent ever gets; the right side is part of what the checker can read.

Only one press changed the settings file: the fifth, when the agent picked Bing. Every other press changed only the screen. And the reward row stayed at zero until the very end. Those are the next two rules.

Three special words, and a limit

Besides real actions, the agent may send one of three special words, the way a card player says “stick”, “fold” or “pass”. DONE means “I have finished”. FAIL means “this cannot be done”. WAIT means “give the computer a moment”.

The game also has a shot clock. In the paper's runs an agent gets at most 15 steps per attempt, the step limit, and at most 30 minutes per task. The attempt ends at DONE, at FAIL, or at the step limit, whichever comes first.

One attempt at one task, from the starting state until the agent stops or runs out of steps, is an episode: one hand of cards. Its record, every observation and every action in order, is a trajectory: the written log of that hand. When a run fails, the trajectory is what you read to find out why. Chapter 9 reads some.

One score, at the end

Now the score. The number the checker returns after the episode is the reward: 1 for done, a fraction for partly done, 0 otherwise. Think of the points for a hand, counted only when the hand is over.

The reward arrives once. During the episode, OSWorld's code hands back a reward of 0 after every single step. The real score comes from the checker, after the agent stops. So the agent is never told how it is doing. It must judge its own progress from the screen.

Some checkers give partial credit: they run several checks and average them, so a task can score 0.5 (Chapter 4 shows how). That is why a benchmark score can be a fraction of a task.

Here is that rule in OSWorld's code, trimmed to the lines that matter.

pythondef step(self, action, pause=2):
    self.action_history.append(action)
    reward = 0                                  # no reward during the episode
    done, info = False, {}
    if action == "FAIL": done, info = True, {"fail": True}
    if action == "DONE": done, info = True, {"done": True}
    if action == "WAIT": time.sleep(pause)
    # otherwise: run the action's code inside the virtual machine (Chapter 2)
    time.sleep(pause)
    return self._get_obs(), reward, done, info

def evaluate(self):
    last = self.action_history[-1] if self.action_history else None
    if self.evaluator["func"] == "infeasible":
        return 1 if last == "FAIL" else 0       # impossible task: only FAIL is right
    if last == "FAIL":
        return 0                                # gave up on a doable task
    # otherwise: run this task's checker (Chapter 4)

The real file also accepts each action as a small dictionary; the logic is the same.

Read it top to bottom. step records the action, sets reward to 0, and checks for the three special words: FAIL or DONE end the episode, WAIT sleeps. Then it pauses so the screen can settle and returns the new observation, the reward of 0, and whether the episode is over. evaluate runs once, after the end. It looks at the last action and applies the rule you are about to meet.

The trick questions

A test that only asks doable questions cannot tell a careful helper from a reckless one. So the authors added trick questions with no right option. At release, 30 of the 369 tasks were impossible on purpose: 8.1%. One asks for a setting the app simply does not have. The only right answer is FAIL. Such a task is infeasible. Ten of them sit in GIMP, five in the bare system and five in VS Code.

The rule is strict both ways. On an impossible task, a final FAIL scores 1 and anything else scores 0. On a doable task, a final FAIL scores 0, even if the agent happened to get close.

The formal name

Think of a board game where only today's board matters. How you got here does not change what your next move does. That is the Markov property: the next state depends only on the present state and the action, not on how you got there. A real computer is only roughly Markov, because clocks tick and background programs run on their own. The paper treats it as Markov, and so will we.

Put the pieces together and you have the paper's formal model. All five parts together make a partially observable Markov decision process, a POMDP. Here it is as one line.

S

Every possible state of the machine: files, settings, windows, memory.

O

What the agent is shown at each step: the instruction, plus a screenshot, a text list of the screen, or both.

A

Every action it may send: a line of pyautogui code, or DONE, FAIL or WAIT.

T

The transition: how the machine responds to an action.

R

The reward: the checker's score, given once at the end.

⟨ ⟩

The angle brackets just group the five parts into one object: one task's game.

In the Bing task, S is every possible state of that Ubuntu machine. O is the instruction plus the screen. A is the set of pyautogui lines plus DONE, FAIL and WAIT. T is Ubuntu and Chrome reacting to each click. R is the checker that reads Chrome's settings file. Two of the five are functions, and their shapes say a lot.

S × A

A state together with an action.

→ S

Gives the next state.

T

The transition: given where the machine is and what the agent did, where the machine goes.

R

The reward: given the final state and the last action, a score.

[0, 1]

A number from 0 to 1: 1 is done, a fraction is partly done.

Notice what R takes: a state and an action, not a history. The score depends only on how the machine ends up and on the last thing the agent said. Spelled out case by case, the paper's reward rule reads like this.

r

The reward of one episode, given once, after it ends.

the goal

What the task's checker tests: here, “Bing” in Chrome's settings file.

FAIL

The special word for “this cannot be done”. Right on the 30 impossible tasks, wrong everywhere else.

The score of a whole benchmark

One task gives one reward. The benchmark's number is simply their average: add the rewards of all tasks and divide by the number of tasks.

N

The number of tasks: 369.

r_i

The reward of task i, from 0 to 1.

Σ_i

Add up over all tasks.

0.1224 × 369 = 45.17 task credits (GPT-4 reading the text list)

Run it backwards for the best model of 2024. Its 12.24% times 369 tasks is 45.17. That is not a whole number, so some of its tasks gave part credit. “About 45 tasks” is the honest way to say it.

The laziest agent

Now put the rules to work. Every test has a floor: the score of a helper that does the same trivial thing every time, like answering “C” to every multiple-choice question. For this benchmark, the trivial thing is to give up. What does an agent score if it answers FAIL to every one of the 369 tasks? Guess before you look.

The arithmetic is short. The 30 impossible tasks each score 1 for a final FAIL. The other 339 each score 0. So the success rate is 30 divided by 369, which is 8.13%. This is the benchmark's floor.

That floor sits above 21 of the 26 runs in the paper's main results table, including every run that saw only screenshots. The paper does not report this floor; we computed it from its own rules. It does not mean those agents were worse than giving up on every task: they did not give up, and they also earned their points in different places. It does mean that a score below about 8% needs to be read with care.

One real number shows that the agents rarely gave up even when they should have. GPT-4V with numbered marks (both explained in Chapter 6) scored 16.67% on the impossible tasks: it correctly said FAIL on just 5 of the 30.

worked exampleThe success rate as a sum of rewards
  N = 369 tasks, one episode each, reward r between 0 and 1
  GPT-4 reading the text list: 12.24%  →  sum of r = 0.1224 × 369 = 45.17
  45.17 is not a whole number: some tasks gave part credit

The floor: answer FAIL to every task
  impossible tasks at release   30 of 369, each FAIL scores 1   →  30
  doable tasks                 339 of 369, each FAIL scores 0   →   0
  success rate = 30 ÷ 369 = 0.0813 = 8.13%
  main-table runs below 8.13%:  21 of 26                          (our arithmetic)
     screenshot only 6 of 6 · text list 6 of 8 · pixels + text 4 of 6 · numbered marks 5 of 6
  GPT-4V with numbered marks on the 30 impossible tasks: 16.67% = 5 of 30

Here is one episode in OSWorld's shape, from scratch. It is a sketch, not the exact API, but every line maps onto a rule of this chapter.

pythondef run_episode(env, agent, task, max_steps=15):
    obs = env.reset(task)                      # revert, set up, first look (Chapter 2)
    for t in range(max_steps):                 # at most 15 steps
        action = agent.act(task["instruction"], obs)
        obs, reward, done, info = env.step(action)   # reward is always 0 here
        if done:                               # DONE or FAIL ends the episode early
            break
    return env.evaluate()                      # the one real score, from 0 to 1

An agent that learned to give up

What happens. A team trains a new agent and it scores 8% on the benchmark, ahead of a 5% rival. They celebrate.

What you measure. Split the score: doable tasks on one side, impossible tasks on the other. Every point the new agent earned came from the 30 impossible tasks. It had learned to give up.

What you change. Report the two halves separately, the way the paper's own analysis does (Chapter 8), and print the floor beside every score. Then run every checker on the untouched starting computer: an agent that says DONE before doing anything must score 0 on every doable task.

The person at the top of the page has finished. Chrome now searches with Bing. On another run, Spotify got installed; on another, a spreadsheet changed. The next task needs a computer where none of that ever happened. You have 369 tasks. How do you put the computer back, every time, in seconds?

Here is the plan in plain words. Put the computer back exactly as it was. Arrange today's files on the desk. Let the agent work. Then save everything before the inspector comes in. This chapter builds each of those four moves, and shows what goes wrong when one is skipped.

A computer inside a computer

Picture a workshop full of ships in bottles, each one a real ship that really sails. The workshop is the host: the real computer that runs the virtual machines and the agent. Each bottle is a virtual machine, the whole computer imitated by software that Chapter 0 introduced. To the programs inside, it looks like a real computer with its own screen, disk and keyboard.

In OSWorld each virtual machine runs its own copy of Ubuntu 22.04 with the eight apps installed. Whatever the agent does inside stays inside. It can delete files, install programs or crash an app, and the host is untouched. That matters when you hand a computer to a program that is still learning: a bad click cannot reach your real files.

Why not a lighter box?

There is a lighter way to fence programs in. Every operating system has a core that controls the hardware: memory, disk, screen, network. That core is its kernel. A container is a box that shares the host's kernel. Think of a separate room in the same house, against a separate house.

The shared kernel is the catch. A container can only run the host's kind of operating system, because it borrows the host's core. A virtual machine brings its own kernel, so the same tool can run an Ubuntu, a Windows or a macOS image, on ordinary hosts or on ARM hosts, the chip family in most phones and in newer Macs.

The paper lists three more reasons. Virtual machines can save and restore their whole state cheaply, which gives a fast reset. Many of them can run side by side on one host. And they can run headless: with no screen attached, like an orchestra playing in a sealed studio. Later the project also learned to run its virtual machines inside containers; the Map at the end has that story.

Why Ubuntu

Most people use Windows. In 2023, Windows ran on 69.5% of desktop computers, macOS on 20.4%, ChromeOS on 3.2% and Linux on 3.1%. So why build the benchmark on Linux?

Because Windows and macOS are closed and licensed, and macOS may not legally run on hardware that Apple did not make. A benchmark that thousands of people will copy and run must be free to copy. So the main benchmark is Ubuntu. The paper adds 43 Windows tasks for a side study, and Chapter 8 shows how scores on the two systems line up.

Save points

Think of a save point in a video game: the game records everything, and you can jump back to it after any disaster. A saved copy of a virtual machine's whole state is a snapshot. Going back to it is a revert. One revert takes the machine back to the saved moment in a single move.

The obvious design is one snapshot per task. The paper explains why it did not do that. A snapshot stores the machine's whole hardware state, most of which has nothing to do with the task, so each task would need gigabytes of space. With 369 tasks that adds up fast.

So OSWorld shares a handful of snapshots among all tasks and builds each task's own starting point on top with a short script. There are 13 named snapshots for the 369 tasks. The exact computer a task starts on is its initial state: the board before the first move. The scripts produce 302 distinct initial states.

This is a trade-off, and it is worth stating plainly. To be fair, every attempt at a task must start from the identical initial state. There are two ways to get there. Store the whole machine for every task: fast to restore, gigabytes each. Or store a few shared machines and rebuild the differences with a script: small, a little slower, and the script must be exact. OSWorld takes the second road.

Setting the stage

A stagehand sets the props before the curtain rises. The short instructions run before the agent starts are the task's setup: download a file, open it, run a command. OSWorld runs them in three stages:

1. Start the machine and revert it to the task's snapshot.
2. Prepare files. Download them to the host, then copy them into the machine over the network.
3. Run commands. Open the file, launch an app, move the cursor, resize a window.

Every setup step has a type. Here is how often each type appears across all 369 task files.

A task usually has several setup steps, so these counts add up to more than 369. Downloading a file and launching an app are by far the most common. Read the numbers as counts of steps, not shares of tasks: one task can download several files, so 278 out of 369 does not mean that 75% of tasks download a file.

Orders through a door

Now the agent works. Its actions are written on the host, but they must happen inside the machine. Picture a stagehand behind the set taking notes passed under a door. Inside the machine runs a small program that waits for orders from the host and carries them out: a server. It listens at a numbered network door, a port, number 5000.

The orders travel as JSON: a text format of named fields inside curly braces, like a filled-in form. Here is one action on its way into the machine.

httpPOST http://<vm_ip>:5000/execute
{
  "command": ["python", "-c",
              "import pyautogui; import time; import platform; pyautogui.FAILSAFE = False; … pyautogui.click(1300, 328)"],
  "shell": false
}

Read it line by line. The first line sends the order to the machine's address, at port 5000, to a door named /execute. The command field is a list: run python, and with -c, run the one line of Python that follows.

That line starts with a fixed prefix that the host adds to every action. It loads pyautogui and two helper modules. Then it switches off pyautogui's fail-safe: normally pyautogui stops at once if the mouse reaches a corner of the screen, a safety catch for people testing scripts. The harness switches it off so a click near a corner cannot abort a run. The agent's own action comes last: here, the click on Bing.

The last field, "shell": false, says to run the command directly. A shell is the program inside a terminal that reads a typed command and runs it, a clerk who reads your request aloud to the right office. Here there is no clerk; the server starts Python itself, and gives it two minutes before it gives up.

Then the way back. After the pause the host asks the server for a screenshot, at /screenshot. The server takes a PNG picture of the screen and pastes the mouse cursor into it, so the agent can see where the pointer is. The host can also ask for the text list of the screen, at /accessibility, a long nested text that Chapter 3 opens up. Together with the instruction, that is what the agent receives each step.

python{
  "screenshot": b"\x89PNG…",                     # 1920 × 1080, with the cursor pasted in
  "accessibility_tree": "<desktop-frame>…</desktop-frame>",   # the text list (Chapter 3), or None
  "terminal": None,                              # optional terminal output
  "instruction": "Can you make Bing the main search engine when I look stuff up on the internet?"
}

The observation the agent receives after each step: a picture, a text list or both, an optional terminal printout, and the instruction.

Lock up before the inspection

The agent says DONE. Is it time to check? Not yet. Think of an inspector who asks the builder to lock up before the walk-through. The steps the checker runs after the agent stops and before it looks are the postconfig: bring a window to the front, press Ctrl+S, wait.

Here is why they exist. The checker reads persistent state: files on the disk, settings files, the output of commands. Anything the agent changed only inside an app's memory is invisible to it. A spreadsheet edit lives in Calc's memory until someone saves it. So the postconfig must turn memory into disk before the checker reads.

Here is the real ending of a spreadsheet task, from its task file.

json"postconfig": [
  {"type": "activate_window", "parameters": {"window_name": "Multiply_Time_Number.xlsx - LibreOffice Calc", "strict": true}},
  {"type": "sleep",   "parameters": {"seconds": 0.5}},
  {"type": "execute", "parameters": {"command": ["python", "-c", "import pyautogui; pyautogui.hotkey(\"ctrl\", \"s\");"]}},
  {"type": "sleep",   "parameters": {"seconds": 0.5}}
]

One line per item. Bring the window named after the file to the front. Wait half a second. Run one line of Python inside the machine that presses Ctrl+S. Wait again. Only then does the checker copy the file out and read it.

The Chrome task from the top of the page ends differently. Its postconfig closes Chrome, starts it again, and waits 3 seconds before the checker reads the settings file. The task file does not say why. A safe reading is the same reason: closing Chrome makes sure the new setting has reached the disk. Run the spreadsheet task yourself, with and without the save.

Across all the task files, the most common postconfig steps are waits (295 of them), commands (204) and bringing a window forward (137). The checker almost never trusts that the agent left things tidy.

Time

Every action needs a moment to land: a menu animates open, a dialog appears, a file saves. So OSWorld pauses after every action before it takes the next screenshot: 2 seconds by default, and 3 seconds in the run command its instructions give. With the 15-step limit that is up to 45 seconds of pauses per task, inside the 30-minute cap.

For 369 tasks run one after another, the waiting adds up. A later version of the project runs up to 50 machines at once and gets through the whole set in under an hour; Chapter 10 has the details.

worked exampleWhy not one snapshot per task
  369 tasks, 302 distinct starting states
  one private snapshot per task: "gigabytes" each, says the paper
    even at 1 GB each: 369 × 1 GB = 369 GB                       (illustrative lower bound)
  what OSWorld does: 13 snapshot names shared by all 369 tasks
    libreoffice_calc 74 · chrome 72 · libreoffice_impress 51 · gimp 32
    libreoffice_writer 31 · os 31 · vscode 27 · thunderbird 20 · multiapps 12
    base_setup 11 · vlc 4 · vs_code 3 · multi_apps 1
    74 + 72 + 51 + 32 + 31 + 31 + 27 + 20 + 12 + 11 + 4 + 3 + 1 = 369
  then a few setup steps rebuild each task's own files and windows

The spreadsheet task, timed
  setup        revert "libreoffice_calc" · download Multiply_Time_Number.xlsx · open it
  agent        up to 15 steps, each followed by a 3 s pause → at most 15 × 3 = 45 s of pauses
  postconfig   bring Calc forward · wait 0.5 s · Ctrl+S · wait 0.5 s = 1.0 s of waiting
  checker      copy the file out · read cell E3 · is it within 0.001 of 191.6667?

Two of those snapshot names, vscode and vs_code, and multiapps and multi_apps, look like the same thing spelled two ways. They are listed as the task files spell them.

A right answer that scored 0

What you see. The agent's last screenshot shows the right total in cell E3. The checker scores 0.

What you measure. Copy out the file the checker read and open it. E3 still holds what it held before the agent started. The edit lived only in Calc's memory; the file on disk never changed.

What you change. Save before you check. This task's postconfig brings Calc forward, waits half a second, presses Ctrl+S and waits again, so the checker reads what the agent did. When you write your own tasks, ask of every checker: where does it read from, and what puts the answer there?

Sit in the agent's chair. You cannot move your eyes, hover, or watch the mouse. Every few seconds someone slides a photograph of a screen across the table, and you must answer with one instruction, written in numbers: click 1,300 pixels from the left and 328 down. Then the next photograph arrives.

That is the agent's whole job, in plain words. It is handed a still photo or a typed inventory of the screen, and it must answer with an exact address to click. This chapter opens both inputs and the one output: how a picture becomes something a model can read, what the inventory looks like, and how a line of code becomes a real click.

A screen is graph paper

Look closely at any screen and it is a grid of tiny squares of colour, like squares on graph paper. Each square is a pixel. A screen's resolution says how many: 1920 × 1080 means 1,920 pixels across and 1,080 down. A point on the screen is named by its coordinates (x, y): x counts pixels from the left edge, and y counts pixels down from the top.

The benchmark's screens are 1920 × 1080, chosen because it was the most common screen size in 2023. So the Bing row of the toy screen, centred at (1300, 328), is 1,300 squares from the left and 328 squares down. Every screenshot the agent receives also has the mouse cursor pasted in, so it can see where its pointer is.

How a model reads a picture

A model does not read letters or pixels one by one. It reads in small pieces called tokens. Text is cut into pieces, often parts of words: “screenshot” might be two tokens. A picture is first shrunk, then cut into square tiles, and each tile becomes a fixed number of tokens.

A model can only hold so much at once, like a reader who can keep a few pages in mind. The most tokens it can read at once is its context window. Everything the agent is shown, the instruction, the screens, the history, must fit inside it, and every token costs time and money.

The paper does not say how many tokens a screenshot cost its 2024 models, or how they resized pictures. So we use one documented rule we can compute with: OpenAI's vision documentation for its current models. It goes like this. Fit the picture inside 2048 × 2048. Shrink it so its short side is 768 pixels. Count the 512-pixel tiles that cover it. Charge 85 tokens, plus 170 for each tile.

tiles

How many 512-pixel squares cover the shrunk picture.

170

The tokens charged for each tile.

85

A fixed charge for the picture as a whole.

Walk it for one 1920 × 1080 screenshot. It already fits inside 2048 × 2048. Shrinking the short side from 1,080 to 768 scales everything by 768 ÷ 1080 = 0.711, giving 1365 × 768. Across, 1365 ÷ 512 = 2.67, so 3 tiles; down, 768 ÷ 512 = 1.5, so 2 tiles; 6 tiles in all. That is 85 + 170 × 6 = 1,105 tokens.

85 + 170 × 6 = 1,105 tokens (one 1920 × 1080 screenshot, our arithmetic)

Notice one consequence. A model that follows this rule never sees the full 1920 × 1080 detail. It sees 1365 × 768, about 0.71 of the size in each direction. A 28-pixel menu row arrives about 20 pixels tall.

The text list

There is a second way to show a screen, and it was built for people who cannot see it. Blind users rely on screen readers, programs that read the screen aloud. To make that possible, the operating system keeps a list of every button, menu and text box on screen, with its name, its kind and its position. Think of a building's floor plan beside a photo of the building. That list is the accessibility tree. We call it the text list.

On Ubuntu, OSWorld reads it through the desktop's accessibility service and writes it out as XML: a text format that nests labelled items inside each other, like an outline with indents. The outermost item is <desktop-frame>, the whole desktop.

Raw, it is enormous: usually more than a million tokens for one screen, because it lists every hidden, empty and decorative item too. So OSWorld filters it. It keeps an item only if it is showing, visible and usable; if it is a useful kind, such as a button, link, menu, list item, text box, table cell or heading; if it has a name or some text; and if it sits on the screen with a real size.

Each item that survives becomes one row with seven columns: tag (the kind of item), name, text, class, description, the position of its top-left corner, and its size. Here is the start of the list for the toy settings screen, as the model receives it.

text listtag          name                                   text    class  description  position (top-left x&y)  size (w&h)
push-button  Customize and control Google Chrome                                 (1872, 72)               (36, 36)
link         Search engine                                                       (70, 390)                (310, 44)
combo-box    Search engine used in the address bar  Google                       (1240, 226)              (120, 28)
list-item    Bing                                                                (1240, 314)              (120, 28)

The header is OSWorld's own; the rows are illustrative. In the real list the columns are separated by tab characters; here they are lined up with spaces so you can read them.

Read the header column by column. tag is the kind of item: a push-button, a link, a combo-box (a drop-down box), a list-item. name is its label, what a screen reader would say. text holds any text inside it: the combo-box currently shows “Google”. class and description are extra hints, often empty. Then the position of the top-left corner, and the size.

Now the Bing row. It is a list item named Bing. Its top-left corner is at (1240, 314), and it is 120 pixels wide and 28 tall. That is everything a model needs to click it exactly, without looking at a single pixel.

How long is the list after filtering? That depends on the screen. A busy spreadsheet lists every cell. To describe a spread of lengths, use a percentile: the value that a given share of the measurements fall below. 90% of the people in a room are shorter than its 90th percentile height. The paper measured the filtered lists: 90% of single screens fit in about 6,000 tokens, and the 90th percentile is 6,343.6 tokens.

Compare that with the picture. 6,343.6 ÷ 1,105 = 5.7: a long text list costs about as much as five or six screenshots. And OSWorld cuts every list off after its first 10,000 tokens, so on a very busy screen part of the list is simply never shown.

See the two views side by side. Pick any item, on the picture or in the list, and watch where its click comes from.

An observation can also carry a third view: the text printed by a terminal, if one is open. That is the third view the environment offers. The paper's baseline agents use the first two, the picture and the text list, and Chapter 6 compares them.

Acting in code

Now the output side. The agent answers with Python, using pyautogui, the remote control you operate by writing. Here are the paper's own examples of actions, each with one line of explanation.

pythonpyautogui.moveTo(1300, 328)        # move the mouse to a point
pyautogui.click(1300, 328)         # click at a point
pyautogui.write('spotify')         # type text
pyautogui.press('enter')           # press one key
pyautogui.hotkey('ctrl', 'c')      # press keys together
pyautogui.scroll(200)              # scroll one way …
pyautogui.scroll(-200)             # … and the other
pyautogui.dragTo(900, 500)         # drag to a point
pyautogui.keyDown('shift')         # hold a key down …
pyautogui.keyUp('shift')           # … and let it go

The functions are the paper's Table 2; the coordinates and the typed word are ours.

The full set of actions an agent may send is its action space: the buttons on its remote control. Here the action space is any valid pyautogui code. An agent may write several lines at once, even a loop that clicks down a column.

Why code and not a short menu of buttons? Because earlier web benchmarks used fixed lists of actions, and those lists could not express everyday moves such as a right-click or a Ctrl+click. Code can say anything a mouse and keyboard can do.

The paper also defines a second, simpler action space: thirteen named actions, each with its arguments, plus WAIT, FAIL and DONE. It is called computer_13. The baseline agents use the pyautogui form.

computer_13MOVE_TO(x, y)              CLICK(button, x, y, num_clicks)   MOUSE_DOWN(button)   MOUSE_UP(button)
RIGHT_CLICK(x, y)          DOUBLE_CLICK(x, y)                DRAG_TO(x, y)        SCROLL(dx, dy)
TYPING(text)               PRESS(key)                        KEY_DOWN(key)        KEY_UP(key)
HOTKEY(keys)               plus WAIT, FAIL, DONE

Grounding: from a name to a point

Your friend says she lives in “the blue house by the bakery”. Knowing that is one thing. Finding her front door is another. The agent's plan says “pick Bing”; the action needs (1300, 328). Turning a description of a target into the exact point to click is grounding.

With the text list, the agent can compute it. A row gives the top-left corner (x, y) and the size (w, h). The middle of the item is half the width to the right and half the height down. OSWorld rounds the halves down to whole pixels.

x, y

The top-left corner of the item, from its row in the text list.

w, h

Its width and height in pixels.

⌊ ⌋

Round down to a whole pixel.

x_c, y_c

The point to click.

(1240 + 120 // 2, 314 + 28 // 2) = (1300, 328) (the Bing row)

OSWorld itself uses exactly this rule when it draws numbered marks on a screenshot (Chapter 6). And real models write it out. In one of the paper's logged runs, a model's own comment reads “Adding half of the width and height to get the center”, and its next line computes 1793 + 60, 920 + 17. Chapter 9 reads the rest of that run, and it does not end well.

Pixels only

With only a picture, no positions are given at all. The model must estimate coordinates by looking, the way you would guess a point on a photograph without a ruler.

There is a second step when the picture was shrunk. Shrinking a picture by a ratio before the model sees it is downsampling: reading a page from across the room. The model then answers in the coordinates of the small picture, and the harness must scale that answer back to the screen. A point a fraction f of the way across the small picture sits the same fraction f across the screen, so you divide by the ratio.

r

The shrink ratio: 0.4 means the picture is 0.4 times the screen's size.

x_image

Where the model points, in the small picture's pixels.

x_screen

Where the click lands on the real screen.

At ratio 0.4 the model sees 768 × 432. Every pixel of error in the small picture becomes 1 ÷ 0.4 = 2.5 pixels on the screen. Hold on to that number: Chapter 9 shows what a few pixels do to a 28-pixel row.

worked exampleOne screenshot, in tokens (OpenAI's documented rule; our arithmetic)
  screen              1920 × 1080
  fit in 2048 × 2048: it already fits
  short side to 768:  scale 768 ÷ 1080 = 0.711 → 1365 × 768
  512-pixel tiles:    1365 ÷ 512 = 2.67 → 3 across;  768 ÷ 512 = 1.5 → 2 down;  3 × 2 = 6 tiles
  tokens              85 + 170 × 6 = 85 + 1,020 = 1,105

One screen as a text list
  raw XML tree        usually more than 1,000,000 tokens
  filtered table      90% of single observations fit in about 6,000; 90th percentile 6,343.6
  raw ÷ filtered      at least 1,000,000 ÷ 6,343.6 = 158 times smaller            (our arithmetic)
  against a picture   6,343.6 ÷ 1,105 = 5.7 screenshots' worth

From a row to a click
  row:  list-item  Bing  (1240, 314)  (120, 28)                                  (illustrative row)
  x = 1240 + 120 // 2 = 1240 + 60 = 1300
  y =  314 +  28 // 2 =  314 + 14 =  328
  action: pyautogui.click(1300, 328)

From a shrunk picture back to the screen
  ratio 0.4: the model sees 768 × 432
  it answers (520, 131) → (520 ÷ 0.4, 131 ÷ 0.4) = (1300, 327.5) on the screen   (illustrative answer)
  one pixel off in the small picture = 1 ÷ 0.4 = 2.5 pixels off on the screen

The button that was never shown

What you see. On a big spreadsheet, the agent that reads the text list keeps writing that it cannot find the Save button. The button is plainly on the screen.

What you measure. Count the tokens of the filtered list for that screen. Every cell of the sheet is a row, so the list runs past 10,000 tokens, and OSWorld cuts it there. In this example (illustrative), the toolbar's rows come after the cut. The agent was never shown the button.

What you change. Filter harder for this kind of screen, or give the agent the picture as well. The paper also warns that some programs expose little or nothing to the text list at all, which is one reason it calls the picture-only agent the long-run goal.

Open one of the 369 task files. It is a few dozen lines of text. Somewhere in those lines are the instruction, the stage directions for setting up the machine, and the inspector's entire checklist. We read one line by line. Then you try to cheat the inspector.

In plain words: every task carries its own inspector's checklist. Fetch this one thing from the finished computer, compare it with that, give 1 or 0. The whole benchmark is only as good as those checklists, so this chapter reads them closely.

The task file

Think of a recipe card for one dish: the ingredients, the steps, and how to tell when it is done. A short JSON file that holds everything about one task is its task file. Each of the 369 tasks has one.

People wrote them by hand. The authors drew on software documentation, tutorials, how-to sites, question-and-answer forums, course material and blogs, so the tasks look like things real users ask for. Nine computer-science students, all authors of the paper, wrote the tasks over three months.

That took about 1,800 person-hours, where one person-hour is one person working for one hour: 650 hours for tasks in one app, 750 for tasks that use several apps, and 400 for double-checking. Then about 400 more went into four rounds of checks. Each task's setup took about 1 person-hour to build, and its checker about 2. Every task was cross-checked by two other authors.

Every file has the same top-level fields: id, snapshot, instruction, source, config, trajectory, related_apps and evaluator. Today's files also carry proxy, fixed_ip and possibility_of_env_change, added later (Chapter 10). Here is the spreadsheet task from Chapter 2, exactly as it sits in the project's repository.

{
  "id": "357ef137-7eeb-4c80-a3bb-0951f26a8aff",
  "snapshot": "libreoffice_calc",
  "instruction": "I have calculated the total work hours from the everday hours. And I have an hourly rate. Now I want to multiply the total hours with the hourly rate to get a total earned amount. However, I can't get a correct answer by directly multiply the two cells. Here the \"total hours\" is of time and \"hourly rate\" is just a number. How can I get the correct product of them? Help me fill in the cell the correct answer. Don't touch irrelevant blank regions.",
  "source": "https://www.reddit.com/r/excel/comments/17zny8u/calculating_total_amount_earned_from_total_hours/",
  "config": [
    {"type": "download", "parameters": {"files": [{"url": "https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/libreoffice_calc/357ef137-7eeb-4c80-a3bb-0951f26a8aff/Multiply_Time_Number.xlsx", "path": "/home/user/Multiply_Time_Number.xlsx"}]}},
    {"type": "open", "parameters": {"path": "/home/user/Multiply_Time_Number.xlsx"}}
  ],
  "trajectory": "trajectories/357ef137-7eeb-4c80-a3bb-0951f26a8aff",
  "related_apps": ["libreoffice calc"],
  "evaluator": {
    "postconfig": [
      {"type": "activate_window", "parameters": {"window_name": "Multiply_Time_Number.xlsx - LibreOffice Calc", "strict": true}},
      {"type": "sleep", "parameters": {"seconds": 0.5}},
      {"type": "execute", "parameters": {"command": ["python", "-c", "import pyautogui; pyautogui.hotkey(\"ctrl\", \"s\");"]}},
      {"type": "sleep", "parameters": {"seconds": 0.5}}
    ],
    "func": "compare_table",
    "result": {"type": "vm_file", "path": "/home/user/Multiply_Time_Number.xlsx", "dest": "Multiply_Time_Number.xlsx"},
    "options": {"rules": [{"type": "check_cell", "sheet_idx": 0, "coordinate": "E3",
                 "props": {"value": {"method": "approx:0.001", "ref": 191.6667}}}]}
  },
  "proxy": false,
  "fixed_ip": false,
  "possibility_of_env_change": "low"
}

Here is the same file, one field at a time.

1. id: the task's unique name, a long random string so no two tasks collide.
2. snapshot: "libreoffice_calc", the saved machine to start from (Chapter 2).
3. instruction: the request, 85 words, kept exactly as a real user wrote it, typos and all (“everday”, “directly multiply”). It asks for the total pay: hours worked times an hourly rate, in the right cell.
4. source: the forum thread the task came from.
5. config: the two setup steps. Download the spreadsheet from a public file store into /home/user, then open it.
6. trajectory: a folder for recorded runs.
7. related_apps: LibreOffice Calc.
8. evaluator.postconfig: the four steps of Chapter 2. Bring Calc forward, wait, press Ctrl+S, wait.
9. evaluator.func: "compare_table", the function that judges the result.
10. evaluator.result: vm_file, which copies the saved file out of the machine.
11. evaluator.options.rules: one rule. On the first sheet, cell E3 must be within 0.001 of 191.6667.

Getters and metrics

An inspector often has an assistant. The assistant fetches the paperwork; the inspector reads it and decides. A checker has the same two parts. The part that fetches one piece of the final state, a file, a command's output, a browser setting, is a getter. The part that compares it with what the task expects and returns the score is the metric.

In the file above, vm_file is the getter and compare_table is the metric. Across all 369 task files, the getters fetch a file from the machine 258 times, a command's output 40 times, the browser's current address 14 times, and a web page's contents 12 times: 44 kinds in all. On the expected side, the task compares against a reference file 213 times and against a rule written in the task file 172 times.

The paper counts 134 evaluation functions, far more than the 5 to 7 of earlier runnable benchmarks (Chapter 5 has the table). Today's task files name 118 distinct metrics. The most used are a table comparison (61 times), a slide-deck comparison (49), the impossible-task rule (27), the include and exclude check (23), a JSON check (22), an exact match (18) and a document comparison (16). The two counts measure different things: the paper's 134 counts its whole function library, while 118 counts names in today's files. Do not mix them.

Reaching inside each app

Each app keeps its state in its own way, so each needed its own way in. Office files are opened with Python libraries for spreadsheets, documents and slides. Thunderbird's account data is read from its profile files, and decrypted with a reverse-engineering tool. VLC is asked through its own web interface. VS Code needed a custom extension. GIMP's results are read as image files and settings files.

Chrome is driven through its remote-debugging port, a door that lets another program control the browser. A small tool called socat forwards that door to where the checker can reach it. That is exactly why the Bing task's setup starts Chrome with --remote-debugging-port=1337 and then launches socat.

Four real checkers, in plain words

• The Bing task. The getter reads the search engine's name from Chrome's settings file. The metric match_in_list scores 1 if the name is “Microsoft Bing” or “Bing”.
• The Spotify task. The getter runs which spotify. The metric check_include_exclude needs “spotify” in the printout and forbids “not found”.
• A Thunderbird task. The checker reads the text list itself, looking for the typed account address in the right form field.
• A Chrome task from the paper's Table 1. The checker reads the browser's cookies and scores 0 if any cookie still belongs to a site that should have been cleared.

Here are the two simplest metrics, from OSWorld's code, trimmed (the originals also print their inputs).

pythondef check_include_exclude(result, rules):
    if result is None:
        return 0.
    include = rules.get("include", [])
    exclude = rules.get("exclude", [])
    if all(r in result for r in include) and all(r not in result for r in exclude):
        return 1.
    return 0.

def match_in_list(result, rules):
    return 1. if result in rules["expected"] else 0.

Read the first one. If the getter fetched nothing, the score is 0. Otherwise it takes the include list and the exclude list from the rule. It scores 1 only if every include string appears somewhere in the result and no exclude string does. Note the word in: it asks whether a piece of text appears anywhere inside the result, not whether a line equals it. The second metric is one line: 1 if the fetched value is one of the expected values, else 0.

And here is the Spotify task's checker, as it sits in its task file.

json"evaluator": {
  "func": "check_include_exclude",
  "result":   {"type": "vm_command_line", "command": "which spotify"},
  "expected": {"type": "rule", "rules": {"include": ["spotify"], "exclude": ["not found"]}}
}

func names the metric. result names the getter and its argument: run this command in the machine and keep what it prints. expected is the rule to compare against.

Several checks at once

Some tasks need more than one check. 54 task files list several metrics, joined by “and” (25 files) or “or” (29 files). “And” stops at the first 0; otherwise it averages the scores. “Or” stops at the first 1; otherwise it takes the best. This is where partial credit comes from: three checks that score 1, 0.5 and 1 average to 0.83.

The impossible tasks have a metric of their own, named infeasible. It follows Chapter 1's rule: 1 for a final FAIL, 0 for anything else.

Two ways a checker goes wrong

A checker is a function of the final state, so it can be wrong in two directions. Picture an inspector who passes a house with no roof, or fails one because the paint is the wrong shade. A checker that scores 1 for a job that was not done gives a false pass. One that scores 0 for a job that was done gives a false fail. Draw the two-by-two: job done or not, against score 1 or 0. The two off-diagonal cells are the two errors.

Each error follows from the form of a rule. Start with text. A piece of text inside a longer text is a substring: a word hidden inside a longer word. A rule that asks for a substring accepts any text that contains it, including text that says the opposite.

Meet the task that shows it. Its instruction, verbatim from the task file: “Hey, my LibreOffice Writer seems to have frozen and I can't get it to close normally. Can you help me force quit the application from the command line? I'm on Ubuntu and I don't want to restart my computer or lose any other work I have open.”

A program that is running right now is a process: a cook still at the stove, not the recipe. Force-quitting a frozen app means ending its process. The checker runs a short shell command with two parts. It checks whether a LibreOffice process is still running, and whether the terminal's saved history of typed commands contains the word kill. It prints one line for each answer.

The include list is “no libreoffice is running” and “killed from terminal”. The exclude list is “libreoffice is still running” and “not killed from terminal”. Look closely at the last one. “not killed from terminal” contains “killed from terminal”. Without the exclude list, the checker would pass a helper that never touched the terminal. Try it.

Numbers have the mirror problem. How far a number may sit from the expected one and still count is the tolerance: the width of the bullseye. A rule with a tolerance rejects any value farther than that from the reference. So a correct answer that someone rounded fails whenever the rounding moved it further than the tolerance allows.

Here is the tolerance rule from OSWorld's metric helpers, trimmed.

pythonif rule["method"].startswith("approx"):                 # e.g. "approx:0.001"
    threshold = float(rule["method"].split(":")[1])
    return abs(float(value) - rule["ref"]) <= threshold

Now the spreadsheet task makes sense. It needs one fact about spreadsheets: a time is stored as a fraction of a day. Twelve hours is stored as 0.5, because it is half a day. So multiplying a time cell by an hourly rate gives a tiny, wrong number, unless you first multiply by 24 to turn days into hours. That is exactly the user's complaint in the instruction.

We do not know the real file's cells, so take one pair that gives the reference value (illustrative): 7 hours 40 minutes at 25 an hour. Multiplying directly gives 7.99, which is 183.68 away from 191.6667: score 0. Multiplying by 24 first gives 191.666…, which is 0.00003 away: score 1. And a helper that works it out by hand and types the rounded 191.67 is 0.0033 away, more than the 0.001 allowed: score 0, a false fail.

worked exampleCombining several checks (54 task files do this)
  "and":  scores 1, 0.5, 1  → no zero, so the average: (1 + 0.5 + 1) ÷ 3 = 0.83
          scores 1, 0, 1    → stops at the 0: the task scores 0
  "or":   scores 0, 0.5, 0  → no one, so the best: 0.5
          scores 0, 1, …    → stops at the 1: the task scores 1

The tolerance on cell E3 (cell values illustrative; rule and reference real)
  a time cell holds a fraction of a day: 7:40 = 7.6667 h ÷ 24 = 0.3194
  multiply directly:      0.3194 × 25      = 7.99      |7.99 − 191.6667|      = 183.68  → 0
  multiply by 24 first:   0.3194 × 24 × 25 = 191.6666… |191.6666… − 191.6667| = 0.00003 → 1
  type the rounded value:                    191.67    |191.67 − 191.6667|    = 0.0033 > 0.001 → 0

The substring trap (checker strings real; final state illustrative)
  closed with the window's X → prints "no libreoffice is running" and "not killed from terminal"
  include "no libreoffice is running": found
  include "killed from terminal":      found, inside "not killed from terminal"
  exclude "not killed from terminal":  found → score 0
  with no exclude list → score 1: a false pass

A pass for a job never done

What you see. An agent closes the frozen Writer window with its X button and says DONE. A draft checker, with only the include list, scores it 1.

What you measure. Print the checker's output: “no libreoffice is running” and “not killed from terminal”. Search it for “killed from terminal”. It is there, right after the word “not”.

What you change. Add an exclude list that vetoes the near miss, as the real task file does. Then test every checker on at least one wrong final state, not only on a right one.

Picture all 369 tasks as tiles on a wall. One says: change the heading's alignment. One says: fill in a spreadsheet with the cities where three conferences met, year by year. One says: remove the subtitles from this video. Now picture a few computer-science students, who have never seen these tasks or these apps, working along the wall with a stopwatch.

That is this chapter, in plain words. Lay all the jobs out as tiles, sort them by app, and ask how many a newcomer finished and how long each took. The answers set the ruler that every model in the paper is measured against.

The wall, by app

Sort the tiles by the app each task lives in. The bare operating system has 24 tasks: things like installing a program or changing a system setting. LibreOffice Calc has 47, Impress 47 and Writer 23. VLC has 17, Thunderbird 15 and Chrome 46. VS Code has 23 and GIMP 26. And 101 tasks use several apps at once.

Five categories

Think of the sections of an exam. The paper groups the apps into five categories: OS, Office, Daily, Professional and Workflow. OS is the bare system, 24 tasks. Office is the three LibreOffice apps, 47 + 47 + 23 = 117 tasks. Daily is Chrome, VLC and Thunderbird, 46 + 17 + 15 = 78. Professional is VS Code and GIMP, 23 + 26 = 49. Workflow is the 101 tasks that cross apps.

A recipe that needs both the oven and the blender is harder to follow than one that needs only the oven. A task that needs two or more apps is a workflow task. 268 tasks stay in one app, 72.6% of the wall, and 101 cross between apps, 27.4%. Earlier runnable benchmarks could not hold that second kind at all.

Where the tasks came from

Most tasks were written fresh. 84 of them, 22.8%, were adapted from five earlier benchmarks, so that results can be compared across test sets. Thirty are impossible on purpose, 8.1%, as Chapter 1 showed. And the 369 tasks start from 302 distinct initial states, so most tasks start from a computer arranged for them alone.

The instructions are short. On average one is 33.36 tokens long, about a sentence or two. Workflow instructions are the longest, at 51.24 tokens, because they must say what to move from where to where. GIMP's are the shortest, at 16.23: “tone down the brightness of my photo” is a typical GIMP request.

Against earlier benchmarks

How does this compare with what came before? The table below has three columns of facts for each benchmark. Tasks is how many test jobs it has; some are built from templates, a few patterns filled in many ways, and the number in brackets counts the patterns. Where the tasks run says whether there is a runnable environment at all, and what kind. Checker functions counts the distinct functions used to score results.

Read it top to bottom. Some earlier sets are much larger, but several have no runnable environment at all, only recordings, so they cannot run a checker on a result. The runnable ones had 5 to 7 checker functions; MiniWoB++ has 125, one per tiny task. OSWorld is the only benchmark in the paper's table with a runnable environment you can extend and tasks that cross apps.

The people

Who were the people? Computer-science college students with basic software skills, who had never seen these tasks or these apps before. They worked without help, on the same machines and the same checkers as the models.

They finished 72.36% of the tasks: 267 of 369. And they were slow. Sort all their times from fastest to slowest and take the one in the middle: that middle time is the median, the person in the middle of a queue sorted by height. The median task took 111.94 seconds. A similar study of 100 tasks from a web benchmark had a median of 35.38 seconds, so these tasks took 3.16 times longer. Many took 900 seconds or more: fifteen minutes on a single job. On that web sample, people finished 88%.

People are steady

Split people's score by category and it barely moves: 75.00% on OS tasks, 71.79% on Office, 70.51% on Daily, 73.47% on Professional and 73.27% on Workflow. The paper's own words: “fluctuating around 70% without exceeding a 5% variance”.

By single app the range is wider, because some apps have few tasks. Thunderbird was hardest for people, at 46.67%: 7 of 15. Impress was easiest, at 80.85%: 38 of 47.

The paper also sorted tasks by how long a person needed. Easy tasks, under 60 seconds, people finished 84.91% of the time. Medium tasks, 60 to 180 seconds, 81.08%. Hard tasks, over 180 seconds, only 49.57%. Hold those three numbers for Chapter 8, where an agent's numbers sit beside them.

Adding categories up correctly

Here is a trap that catches people who report results by group. A class average should count every pupil once, not every table once. An average where each group counts in proportion to its size is a weighted average. The overall success rate counts tasks, not categories, so it is weighted by how many tasks each category holds.

Derive it from the definition. Total credit is the sum over categories of each category's rate times its number of tasks. Divide by all 369 tasks. So each category's rate is weighted by its share of tasks, n_c ÷ N.

c

One category: OS, Office, Daily, Professional or Workflow.

n_c

The number of tasks in it: 24, 117, 78, 49, 101.

N

All tasks: 369.

rate_c

The success rate within that category.

Big categories move the total most. Office, with 117 tasks, counts almost five times as much as OS, with 24. A plain average of the five category rates would treat them as equals, and for a model whose scores swing between categories that gives a very different number. Work it for people, then for the best model of 2024.

worked exampleCategories
  OS 24 · Office 47 + 47 + 23 = 117 · Daily 46 + 17 + 15 = 78
  Professional 23 + 26 = 49 · Workflow 101
  24 + 117 + 78 + 49 + 101 = 369

People, category by category
  OS 75.00% of 24 = 18     Office 71.79% of 117 = 84     Daily 70.51% of 78 = 55
  Professional 73.47% of 49 = 36     Workflow 73.27% of 101 = 74
  18 + 84 + 55 + 36 + 74 = 267        267 ÷ 369 = 72.36%

GPT-4 reading the text list, weighted by size
  20.83% × 24 ÷ 369 = 1.355    3.58% × 117 ÷ 369 = 1.135    25.64% × 78 ÷ 369 = 5.420
  26.53% × 49 ÷ 369 = 3.523    2.97% × 101 ÷ 369 = 0.813
  1.355 + 1.135 + 5.420 + 3.523 + 0.813 = 12.246 → 12.25%   (the paper prints 12.24%: rounding)
  the plain average of the five: (20.83 + 3.58 + 25.64 + 26.53 + 2.97) ÷ 5 = 15.91%   (wrong; our arithmetic)

Time
  median 111.94 s here; 35.38 s on a 100-task web sample;  111.94 ÷ 35.38 = 3.16 times longer

For people the two averages nearly agree, because people were steady across categories. For the model they differ by 3.7 points, because it did well in the small categories (OS and Professional) and badly in the large ones (Office and Workflow).

A slide that averaged the wrong thing

What you see. A summary slide says GPT-4, reading the text list, “averages 15.9% across the five categories”. The paper says 12.24%.

What you measure. Weight each category by its number of tasks. OS (24 tasks, 20.83%) and Professional (49, 26.53%) are small and high; Office (117, 3.58%) and Workflow (101, 2.97%) are large and low. Weighted, it is 12.25%.

What you change. Average over tasks, not over groups. If you report group scores, give each group's size beside it.

The authors had one simple agent design and a dozen models to put in it. Every model got the same briefing card, the same instruction and the same 15 steps. The one thing they varied was how the screen was shown: as text, as a picture, as both, or as a picture with every clickable item numbered. Four choices, four very different requests.

Same helper, same job, same briefing card; the only thing that changes is how the screen is described to it. This chapter builds the request each model received, piece by piece, and counts what one look at the screen costs.

A baseline

The first runner on a new track sets the time to beat. A simple, standard agent that later work tries to beat is a baseline. The paper's baselines are deliberately plain: no planning tricks, no extra tools, just a model in a loop. That is the point. They measure what the models can do on their own, and every later agent on this benchmark is compared with them. The hero's 12% and 5% are two of them.

The briefing card

Picture a briefing card handed to the helper before every move. The text sent to the model with each request is the prompt. Its fixed first part, which explains the job and the rules, is the system prompt. Here are its key lines, verbatim; “…” marks lines we skipped.

system promptYou are an agent which follow my instruction and perform desktop computer tasks as instructed.
…
You are required to use `pyautogui` to perform the action grounded to the observation, but DONOT use
the `pyautogui.locateCenterOnScreen` function to locate the element you want to operate with …
You need to to specify the coordinates of by yourself based on your observation of current observation,
but you should be careful to ensure that the coordinates are correct.
…
When you think you have to wait for some time, return ```WAIT```;
When you think the task can not be done, return ```FAIL```, don't easily say ```FAIL```, try your best to do the task;
When you think the task is done, return ```DONE```.

My computer's password is 'password', feel free to use it when you need sudo rights.
First give the current screenshot and previous things we did a short reflection, then RETURN ME THE CODE
OR SPECIAL CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE.

The odd grammar is the original's. The same text sits in the paper's appendix and in the project's code.

Read it rule by rule. The model must answer in pyautogui code. It must not use pyautogui's built-in image search, a function that finds a button by matching a picture of it; it has to work out the coordinates itself, and “be careful to ensure that the coordinates are correct”. That one sentence is the whole grounding problem of Chapter 3, handed to the model.

It may answer WAIT, FAIL or DONE. But notice “don't easily say FAIL”: the prompt itself pushes against giving up. That is one reason Chapter 1's agents rarely said FAIL, even on the 30 impossible tasks. The computer's password is given, so the agent can install software. And the answer must be a short reflection on the screen and on the steps so far, then only code.

Memory and settings

Each request carries the current observation plus the last 3 observations and actions, in chat form, as if the model were reading back its own recent messages. Older steps fall away.

The model also needs settings for how it writes. Think of how adventurous a cook is with the spice rack. How much chance the model uses when it picks each next word is its temperature: 0 always picks the likeliest word. The baselines use temperature 1.0. Top-p 0.9 means the model picks only among the likeliest words that together hold 90% of the probability. Each answer may be at most 1,500 tokens long. And as before: at most 15 steps and 30 minutes per task.

The authors first tried another idea, which did not work. Showing a new cook three finished plates is few-shot prompting: putting worked examples inside the prompt. They gave the models example observations and actions. With screenshots only it scored 2.79%, so they dropped it.

pythonSETTINGS = {"remember_steps": 3, "temperature": 1.0, "top_p": 0.9, "max_answer_tokens": 1500,
            "max_steps": 15, "time_limit_minutes": 30, "screen": (1920, 1080)}

The fixed settings of every baseline run, as the paper gives them; the names of the keys are ours.

The four views

Now the one thing that varies.

1. Text list only. The filtered table of Chapter 3. The model still has to write coordinates, which it can compute from the positions in the rows.
2. Pixels only. The raw 1920 × 1080 screenshot. The model must estimate coordinates by looking.
3. Pixels and text list, both at once.
4. Numbered marks. Picture numbered seats on a theatre plan. OSWorld draws a numbered box on the screenshot around every item of the filtered text list, using the list's own positions and sizes, not a separate image model. This is Set-of-Mark prompting; we call it numbered marks.

With marks, the model can answer with a mark's number instead of a coordinate, written as tag_12. Before running the model's code, OSWorld writes a line such as tag_12=(1300, 328) above it, using the same centre rule as Chapter 3. The marks come with a shorter text list: each mark's number, kind, name and text, with no positions.

pythontag_vars = ""
for i, (x, y, w, h) in enumerate(boxes):              # one box per filtered text-list item
    tag_vars += "tag_" + str(i + 1) + "=" + "({}, {})".format(int(x + w // 2), int(y + h // 2)) + "\n"
code = tag_vars + model_code                           # e.g. "...tag_12=(1300, 328)\npyautogui.click(tag_12)"

How a mark becomes a click, trimmed from OSWorld's agent code. The model only names the mark; OSWorld does the grounding arithmetic.

Each view puts a different load into every request. Try them, and change how many past steps the model remembers.

What one request costs

Build the cost of one request from parts: tokens per screen (Chapter 3), times screens per request. With 3 remembered steps, a request holds 4 screens: the current one and three remembered ones. Here is the arithmetic for each view.

worked exampleOne request, screen tokens only, remembering 3 past steps (4 screens)
  pixels:          4 × 1,105                  =  4,420 tokens
  text list:       4 × 6,343.6                = 25,374.4 tokens at the 90th percentile
  pixels + text:   4 × (1,105 + 6,343.6) = 4 × 7,448.6 = 29,794.4 tokens
  numbered marks:  4 × 1,105 = 4,420 tokens of pictures, plus four short mark lists (size not reported)
  text list cap:   10,000 per screen → at most 4 × 10,000 = 40,000 tokens of text

From a mark to a click
  mark 12 is the Bing row: box at (1240, 314), size 120 × 28                (illustrative box)
  OSWorld writes   tag_12=(1240 + 120 // 2, 314 + 28 // 2) = (1300, 328)
  the model writes pyautogui.click(tag_12)

Be careful with what these numbers mean. The paper reports the text list's 90th percentile, not its median, so most screens cost less than 6,343.6 tokens; a long list is the expensive case. And these counts cover only the screens. The briefing card, the instruction and the model's own past answers come on top.

Who is who

Now the names. GPT-4 reads text only. GPT-4V is GPT-4 with vision: it also reads pictures. GPT-4o is a later model that does both. Some models read only text, so they could run only with the text list; models that read pictures ran in the views that include a screenshot.

That gives 8 text-list runs, 6 pixel runs, 6 pixels-plus-text runs and 6 marks runs: the 26 dots of Chapter 1, and the 26 rows of Chapter 7's table.

Marks that got in the way

What you see. Numbered marks usually help a vision model point at things. Here, GPT-4V with marks scored 11.77%, below its 12.17% with pixels and the text list.

What you measure. Look at the screens the marks were drawn on. A spreadsheet has hundreds of cells, and every cell gets a box and a number. The boxes crowd each other and cover the text they label. And some jobs, such as dragging across a range of cells, need a precise point that no box number can name. Switch on the busy screen above and pick numbered marks to see it.

What you change. Those are the paper's own two explanations: too many elements on high-resolution screens, and actions that need exact coordinates. Mark fewer items, or let the agent fall back to coordinates. Then check the size of the effect: 0.40 points is far inside the error bars of Chapter 7.

Two results come out the same week. One agent scores 12.24%. Another scores 12.17%. A headline crowns the first by name. Before you agree, ask one question: if each agent had been given a different 369 tasks of the same kind, how far would its own number have moved?

Every score in this chapter is one throw of the dice over 369 tasks. Before you compare two of them, you need a sense of how far each could have landed with a different set of tasks drawn the same way.

Twenty-six runs, one table

The paper ran a dozen models through its baseline agent, each one shown the screen in up to four of the ways Chapter 6 built: the text list, pixels, both together, or numbered marks. Not every model can see a picture, so the text-list column holds eight runs and each of the other three holds six, for 26 runs in all. Every row below is one model in one view; every column is one of Chapter 5’s five categories, plus the overall score across all 369 tasks. People’s row sits at the bottom, for comparison.

Read a few cells before moving on. GPT-4, reading only the text list, scored 20.83% on OS tasks and 2.97% on workflow tasks: the same model, under the same rules, swings 17.86 points just from which tasks it faces. The table holds a lot. The next two sections pull out what the paper itself says it means.

Four things the paper reads in it

First, every model stays far from people. With pixels only, the strongest vision models of the day scored between 5.26% and 5.80%. With the text list, the strongest language models scored between 2.37% and 12.24%. People scored 72.36%. No view and no model gets close.

Second, models swing hard between categories while people barely move at all. Chapter 5 showed people staying between 70.51% and 75.00% across all five groups. GPT-4o, shown pixels and the text list together, scored 41.67% on OS tasks and only 6.16% on Office tasks: a task you can often finish by typing one command came out far easier than a task of clicking inside a document. Workflow tasks, which need more than one app, mostly scored under 5% across the whole table; GPT-4V and GPT-4o were the exceptions, reaching 6.04% and 7.56%.

Third, the extra channel does not help everyone the same way. For GPT-4V, adding the text list to pixels, then swapping in numbered marks, both lifted the score well above pixels alone: 5.26% with pixels, 12.17% with pixels and text, 11.77% with marks. Claude-3-Opus moved the same direction: 2.42%, then 4.41%, then 6.72%. Gemini-ProV moved the other way: 5.80% with pixels alone, then down to 3.48% with pixels and text, then down again to 1.06% with marks. The same extra information helped two models and hurt a third.

Fourth, one model that is reported elsewhere as a close match for GPT-4V falls well short of it here. Across the three views both saw, Claude-3-Opus scored between 2.84 and 7.76 points below GPT-4V: 2.84 points on pixels alone, 7.76 points with pixels and text, 5.05 points with marks. Chapter 9 reads one concrete reason why.

Why a measured rate wobbles

Before comparing any two of these numbers, you need to know how much a single score alone can tell you. Picture each of the 369 tasks as a coin. It lands “done” with some true chance p and “not done” otherwise. Running the benchmark once is flipping 369 such coins once each and counting the heads.

Flip the same kind of coins again, on a different 369 tasks drawn the same way, and the count of heads would not come out identical. It would land near the first count, but it would wobble. Measured as a rate, that typical wobble has a name: the standard error, how much the rate would shift if you drew a different set of tasks the same way. Its formula falls straight out of counting coin flips.

p

The measured success rate, as a fraction: 0.1224 for GPT-4 reading the text list.

n

The number of tasks: 369.

SE = √(0.1224 × 0.8776 ÷ 369) = 0.01706, about 1.7 percentage points

That is the size of the wobble around 12.24% that comes purely from the luck of which 369 tasks got asked.

The simple range, and where it breaks

The everyday way to turn a wobble into a range is to step a fixed number of standard errors out each way. About 95% of the time, a normal wobble stays within 1.96 standard errors of the true value, so stepping that far in both directions gives a range that should catch the true rate 95 times out of 100. This is the Wald interval.

z

How many standard errors cover 95% of a bell curve: 1.96.

GPT-4, text list: 0.1224 ± 1.96 × 0.01706 = 0.1224 ± 0.0334 → 8.90% to 15.58%

Try the same recipe on a smaller number and it breaks. CogAgent, shown numbered marks, scored 0.99%. Its Wald range runs from −0.02% to 2.00%. A success rate cannot be negative. The method that works fine in the middle of the scale hands you an impossible number near either end of it.

Wilson’s range

A better range asks a different question: for which true rates would a measurement of 12.24% not be surprising? Solving that question, instead of just stepping out a fixed distance from the measured rate, gives the Wilson interval: a range nudged slightly toward 50% that never leaves 0% to 100%, however small or large the measured rate is.

centre

The middle of the range, nudged slightly toward 50%.

half-width

How far the range reaches on each side of the centre.

The device below draws every overall score in Table 5 with its own Wilson range. Switch between the four views, tap any two bars to compare them, and switch on the naive toggle to watch the Wald range slip below zero near the bottom of the scale.

worked exampleGPT-4 reading the text list: p = 0.1224, n = 369, z = 1.96
  standard error   √(0.1224 × 0.8776 ÷ 369) = √0.00029111 = 0.01706
  Wald             0.1224 ± 1.96 × 0.01706 = 0.1224 ± 0.0334  →  8.90% to 15.58%
  Wilson
    z² ÷ n            3.8416 ÷ 369 = 0.010411
    denominator       1 + 0.010411 = 1.010411
    centre            (0.1224 + 3.8416 ÷ 738) ÷ 1.010411 = (0.1224 + 0.005205) ÷ 1.010411 = 0.12629
    inside the root   0.00029111 + 3.8416 ÷ (4 × 369²) = 0.00029111 + 0.00000705 = 0.00029816
    half-width        1.96 × √0.00029816 ÷ 1.010411 = 1.96 × 0.017267 ÷ 1.010411 = 0.03350
    interval          0.12629 ± 0.03350  →  9.28% to 15.98%

Where Wald breaks: CogAgent with numbered marks, p = 0.0099
  Wald     0.0099 ± 1.96 × √(0.0099 × 0.9901 ÷ 369) = 0.0099 ± 0.0101  →  −0.02% to 2.00%  (below zero)
  Wilson   0.37% to 2.62%

Three comparisons (Wilson, 95%)
  GPT-4, text list  9.28 to 15.98   vs  GPT-4V, pixels + text  9.22 to 15.90  →  overlap: no winner (0.07 points apart)
  GPT-4, text list  9.28 to 15.98   vs  GPT-4V, pixels only    3.41 to  8.03  →  no overlap: a real gap
  people           67.58 to 76.67   vs  the best model         9.28 to 15.98  →  no contest

Three comparisons

Put three pairs through the device above and the arithmetic settles three arguments at once. GPT-4 reading the text list, 9.28% to 15.98%, against GPT-4V shown pixels and the text list, 9.22% to 15.90%: the two ranges sit almost exactly on top of each other. Neither model wins this comparison; the benchmark cannot tell them apart. GPT-4 reading the text list against GPT-4V shown pixels alone, 3.41% to 8.03%: the ranges do not touch at all, so that gap is real. And people, 67.58% to 76.67%, against the best model of 2024, 9.28% to 15.98%: there is no contest, in any view.

What the bars leave out

These ranges are not the whole story. Each number in Table 5 is one run, at temperature 1.0: a second run of the very same model would land somewhere else too, and these bars say nothing about that kind of wobble. The 369 tasks are a set the authors built by hand, not a random sample of all the work people do on a computer, so the ranges describe only the sampling that happens inside this particular set. And the overlap test used here is a cautious one: two ranges can touch a little while a sharper test, one that compares the two models task by task on the same 369 tasks, can still find a real difference underneath.

For more on reading a score with the right caution, see evaluation statistics, error bars for evals, and the trials table in agent evaluation.

Here is the whole method as a dozen lines of Python, the same lines the device above runs live.

pythonimport math
def wilson(p, n, z=1.96):
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return centre - half, centre + half

print(wilson(0.1224, 369))   # (0.0928, 0.1598)  GPT-4, text list
print(wilson(0.0099, 369))   # (0.0037, 0.0262)  CogAgent, marks

A headline by 0.07 points

What you see. A blog post ranks every row of the table and crowns the top one the winner, by 0.07 percentage points.

What you measure. Put a 95% range on each score. GPT-4 with the text list spans 9.28% to 15.98%; GPT-4V with pixels and text spans 9.22% to 15.90%. The ranges are nearly identical.

What you change. Say “no difference on this evidence.” To separate two agents this close, run more tasks, rerun each agent several times, or compare them task by task on the same set.

Take 28 tasks the agent handles reasonably well: it solves about half of them. Now move the window a little to the right. Then make the same window smaller. Then open a few other windows around it. A person sitting at that computer would barely notice any of this. Watch the agent instead.

Move the window

The authors ran this exact test. They found 28 tasks on which GPT-4V with numbered marks (Chapter 6) scored 50.79%, about 14.2 of the 28 tasks’ worth of credit. Then they reran the same 28 tasks three more times, changing nothing about the instruction or the goal, only the window the agent had to work in. Moved a little, the score fell to 36.5%. Crowded by a few other open windows, it fell to 25.39%. Shrunk to a smaller size, it fell to 15.04%, the steepest drop of the three.

How big is a drop?

A fall from 50.79% to 15.04% is 35.75 points. But points alone hide how much of the agent’s ability that is. Divide the loss by what the agent had to begin with, and you get the share of its ability that the change took away: the relative drop.

s_old

The score before the change: 50.79%.

s_new

The score after it.

worked exampleThe window study: 28 tasks, GPT-4V with numbered marks
  as written    50.79%  = about 14.2 of 28 tasks’ credit
  moved         36.5%   drop 1 − 36.5 ÷ 50.79  = 1 − 0.719 = 28.1%
  cluttered     25.39%  drop 1 − 25.39 ÷ 50.79 = 1 − 0.500 = 50.0%
  resized       15.04%  drop 1 − 15.04 ÷ 50.79 = 1 − 0.296 = 70.4%

Resizing the window took away 70.4% of what the agent could do, moving it took away 28.1%, and crowding it with other windows took away half. The paper’s own summary of this whole set of studies, among its five takeaways: agents are not robust to changes of layout or to added noise on the screen. One plausible reason, ours rather than the paper’s: a smaller window makes every clickable target smaller too, and Chapter 9 shows exactly how fast a small target turns a tiny aiming error into a miss.

How coarse these numbers are

Keep a sense of scale on all of this. The window study ran on 28 tasks, so on that study one single task is worth 100 ÷ 28, or 3.6 percentage points. The resolution and history studies you are about to read ran on a 10% subset of the benchmark, about 37 tasks, where one task is worth 100 ÷ 37, about 2.7 points. A difference of a point or two in either study is a difference of one task, not a trend.

Sharper or smaller pictures

Chapter 3 introduced downsampling: shrinking a picture by a ratio before the model ever sees it. The authors tried four ratios of the full 1920 × 1080 screenshot: 0.2, 0.4, 0.6 and 0.8, which come out to 384 × 216, 768 × 432, 1152 × 648 and 1536 × 864.

Two different findings came out of it, for two different ways of showing the screen. With pixels only, higher resolution meant higher success, in a fairly steady climb. With numbered marks, the picture is smaller than its number: 768 × 432, the 0.4 ratio, did better than the full-size screenshot, and the smallest ratio, 0.2, was clearly worse than the rest. The paper prints the shape of these two curves, not the exact success numbers behind them, so the device below draws the curves as illustrative and keeps only the two ends and the one named ratio as real anchors.

worked exampleScreenshot sizes (OpenAI’s documented rule; our arithmetic)
  0.2    384 × 216    1 tile    →  85 + 170 × 1 =   255
  0.4    768 × 432    2 tiles   →  85 + 170 × 2 =   425
  0.6   1152 × 648    6 tiles   →  85 + 170 × 6 = 1,105
  0.8   1536 × 864    shrunk to 1365 × 768, 6 tiles  →                 1,105
  1.0   1920 × 1080   shrunk to 1365 × 768, 6 tiles  →                 1,105
  a 28-pixel menu row at 0.4 is 28 × 0.4 = 11.2 pixels tall

pythondef shrink(w=1920, h=1080, r=0.4):
    return round(w * r), round(h * r)                # 0.4 → (768, 432)

def tokens(w, h):                                     # OpenAI's documented image rule
    s = min(1, 2048 / max(w, h)); w, h = w * s, h * s
    if min(w, h) > 768:
        s = 768 / min(w, h); w, h = w * s, h * s
    tiles = math.ceil(int(w) / 512) * math.ceil(int(h) / 512)
    return 85 + 170 * tiles                           # 1920×1080 → 1,105; 768×432 → 425

Longer memory

The authors also varied how many past steps the agent could remember: 1, 2, 3, or all of them. With numbered marks, more history helped. With screenshots only, it did not. The paper reads this as the vision models being weak at using pictures from several steps back, rather than just the one in front of them. And memory is not free: Chapter 6 priced one remembered screenshot at about 1,105 tokens, so a longer memory is also a bigger bill.

The text list’s length

About 90% of single observations of the text list fit inside roughly 6,000 tokens, Chapter 6’s number again. Yet relying on only the current screen, with no memory of what just happened, “leads to agents making repeated errors”, in the paper’s own words. Memory helps the agent notice it is repeating itself, and it costs tokens either way.

Hard for people, hard for agents?

The authors also split the benchmark several different ways and checked GPT-4V with numbered marks against how hard each subset had been for people. Sorted by how long a person needed, easy tasks (under 60 seconds) got 16.78%, medium tasks (60 to 180 seconds) got 13.12%, and hard tasks (over 180 seconds) got only 4.59%, while people on those same three groups scored 84.91%, 81.08% and 49.57%. Sorted by app count, single-app tasks got 13.74% against multi-app tasks’ 6.57%. Sorted by whether the task was possible at all, feasible tasks got 13.34% against the impossible tasks’ 16.67%, the floor from Chapter 1 showing through again.

Does the table add up?

Before trusting a split table, check it the way you checked the category weights in Chapter 5: multiply each row’s share of tasks by its rate and add the pieces back together. Single and multi-app should add back to the view’s overall score, and so should feasible and infeasible. Try both.

worked exampleDoes Table 6 add up? (share of tasks × rate)
  single-app 72.63% × 13.74 + multi-app 27.37% × 6.57  = 9.98 + 1.80  = 11.78   matches the overall 11.77
  feasible   91.87% × 13.34 + impossible 8.13% × 16.67 = 12.26 + 1.36 = 13.61   does not match 11.77

The single-app and multi-app split checks out almost exactly. The feasible and infeasible split does not: it reconstructs to 13.61, not 11.77. One of the two printed numbers in that second pair is almost certainly a small typo somewhere upstream. We report both splits exactly as the paper printed them and build nothing further on the feasible-infeasible numbers alone.

Another operating system

The paper also built a 43-task analytic set for Windows, to ask whether a model’s weaknesses are specific to Ubuntu. GPT-4V with screenshots only scored 4.88% on Ubuntu and 2.55% on Windows, and the two sets of per-task scores had a correlation coefficient of 0.7: a number from −1 to 1 that says how closely two lists rise and fall together. A 0.7 says the tasks that are hard on one system tend to be hard on the other too, so most of what this chapter and the next teach about failure is not an Ubuntu-only story.

A lab window that never leaves the lab

What you see. An agent that passes a task on the lab’s own maximised window fails the same task on an ordinary laptop, where the same app sits in a smaller window beside two others.

What you measure. Rerun the same tasks with only the window changed, exactly as the paper did: 50.79% falls to 15.04% when the window is resized, a 70.4% drop, and to 25.39% when other windows crowd it.

What you change. Test agents on varied window layouts, not one tidy maximised screen, and report the relative drop beside every score. The paper itself names robustness to window changes as a direction for future work.

Read the agent’s own note to itself, word for word, from the paper’s appendix: “Coordinates for the Brightness-Contrast option in the Colors dropdown. Assuming each menu item is approximately 28 pixels apart.” The plan is sensible. The next screenshot shows a dialog called Shadows-Highlights. It clicked the wrong row.

The agent knew what to do and aimed at the right thing. Its hand landed a row too low, and then it kept trying.

The count

The authors did not guess at this. They read the full trajectories of 550 failed GPT-4V runs, across every way of showing it the screen, and checked each one for clicks that missed their mark. More than 75% of them, more than 412 runs, contained at least one click that landed in the wrong place. In the authors’ own words, the agent “fails to click the correct coordinates despite planning detailed and accurate steps in their code comments, indicating strong planning but weak execution capabilities”.

Why a small error is enough

Picture aiming a dart at the centre of a small target. You rarely hit the exact centre. Most throws land close to it, fewer land far away, and the shape of “how far away, how often” traces a bell curve. How far a typical click strays from where it was aimed is its spread, written σ (sigma). A click lands on target if it strays less than half the target’s width to either side, and less than half its height up or down.

The share of a bell curve that falls within some distance of its centre is a well-known function called erf. One axis at a time, the chance of staying inside half the target’s width (or height) is erf of that half-width divided by σ√2. Because the two axes (left-right and up-down) are independent, the two chances multiply.

w, h

The target’s width and height in pixels: 120 and 28 for a menu row like Bing’s.

σ

The spread: how far a typical click strays, in pixels.

erf

The share of a bell curve that falls within a given distance of its centre.

A plan is a sequence of clicks that all have to land. If each click is about as risky as the last, the whole plan succeeds with that one chance raised to the power of how many clicks it needs.

k

The number of clicks that must all land: 6 for the Bing path of the hero.

Aim at Bing’s row from the hero, 120 pixels wide and 28 tall, with clicks that stray a typical 12 pixels in any direction (illustrative numbers, chosen to show the geometry). Sideways the row is almost impossible to miss. Up and down, with only 28 pixels of room, the odds get much tighter. Try it below: shrink the row, widen the spread, and watch exactly how many of 200 simulated clicks still land on Bing.

worked exampleAiming at the Bing row: 120 × 28 pixels, clicks spread 12 pixels (illustrative)
  sideways:  erf(120 ÷ (2 × 12 × 1.4142)) = erf(60 ÷ 16.97) = erf(3.536) = 0.99999
  up/down:   erf(28 ÷ (2 × 12 × 1.4142))  = erf(14 ÷ 16.97) = erf(0.825) = 0.7567
  one click lands:  0.99999 × 0.7567 = 0.757  →  75.7%
  six such clicks:  0.757⁶ = 0.188            →  18.8%

The paper’s count
  failed GPT-4V runs read: 550;  with clicks that missed: more than 75% = more than 412

The brightness hunt, from the agent’s own code
  Colors menu:           (344 + 28, 64 + 12)          = (372, 76)
  guess, 6th item down:  (344 + 28, 64 + 12 + 28 × 6) = (372, 244)  →  Shadows-Highlights opened
  guess, 5th item down:  (344 + 28, 64 + 12 + 28 × 5) = (372, 216)  →  Exposure opened
  then the 6th again, then the 2nd: Color Balance, Color Temperature, until the steps ran out

Misses snowball

A missed click rarely ends the story by itself. The paper names two ways it compounds. Miss, nudge the aim, miss again, and you have burned several steps of the 15-step budget on one action: repetitive clicks. Or the stray click lands on the wrong thing entirely, opening a pop-up or the wrong application, and now the screen itself has changed under the agent in a way its plan never accounted for: environmental noise. The paper also notes agents that never learn to close a pop-up, or that get drawn toward an advertisement instead of the task.

The brightness hunt, line by line

Here is one full failure, straight from the paper’s appendix, walked through comment by comment. The task: tone down the brightness of a photo in GIMP. The agent opens the Colors menu using the same “add half the width and half the height” centring rule from Chapter 3: a button at (344, 64) sized so that its centre sits at (372, 76).

python# Action 2
# Coordinates for the `Colors' menu
# Adding half of the width and height to get the center
colors_menu_x, colors_menu_y = 344 + 28, 64 + 12
pyautogui.click(colors_menu_x, colors_menu_y)

# Action 3
# Coordinates for the `Brightness-Contrast...' option in the `Colors' dropdown
# Assuming each menu item is approximately 28 pixels apart
brightness_contrast_option_x, brightness_contrast_option_y = 344 + 28, 64 + 12 + 28 * 6
pyautogui.click(brightness_contrast_option_x, brightness_contrast_option_y)

# Action 4
# Coordinates for the `Cancel' button in the `Shadows-Highlights' dialog
# ...
brightness_contrast_option_x, brightness_contrast_option_y = 344 + 28, 64 + 12 + 28 * 5
pyautogui.click(brightness_contrast_option_x, brightness_contrast_option_y)
# ... More Wrong Duplicate Actions; Can not find the right coordination.
# Failed

Read it the way the device above would score it. The agent guesses that Brightness-Contrast sits six menu rows down, at 28 pixels apiece: (372, 244). That opens a dialog called Shadows-Highlights instead, so the agent backs up by one row to the 5th item down, (372, 216), and gets Exposure. It tries the 6th row again, then the 2nd. Its own log records what happens next, verbatim: “More Wrong Duplicate Actions; Can not find the right coordination”, until the step budget runs out. The paper’s own diagnosis of this run is blunt: the agent “does not know which menu in the toolbar is for brightness adjustment and instead randomly tries until exhausting the maximum number of steps”. That not-knowing has a name: operational knowledge, knowing how a particular app actually works, which menu holds which tool, what a double-click does, which command does the job in one step instead of six guesses.

Knowledge can also skip the clicking

Not every success in the paper comes from aiming better. One of its clean wins skips the aiming entirely. The instruction, verbatim: “I downloaded an episode of Friends to practice listening, but I don’t know how to remove the subtitles. Please help me remove the subtitles from the video and export it as “subtitles.srt” and store it in the same directory as the video.” The agent opened a terminal and typed two commands of a video tool called ffmpeg, no mouse involved at all.

bashffmpeg -i video.mp4 -map 0:s:0 subtitles.srt           # copy the first subtitle track into subtitles.srt
ffmpeg -i video.mp4 -c copy -sn no_subtitles_video.mp4   # copy the video and sound, leave subtitles out

Six steps, no fiddly clicking, no row to miscount. Knowing the one right command beat any amount of careful aiming.

Easy for whom?

The paper notes, and shows examples of, a pattern that cuts both ways: tasks a person finds simple can be hard for an agent, and tasks a person finds tedious can be easy for one. Agents finished “force close a process” with one typed command, pkill soffice.bin, and “monitor the system CPU for 30 seconds” with sar 1 30. The same agents failed “bold the font on this slide and add a note” and “erase all the highlighted marks in this document”, both ordinary mouse work for a person. The category numbers from Chapter 7 agree: GPT-4o, shown pixels and the text list, scored 41.67% on OS tasks, which often reduce to one command, and only 6.16% on Office tasks, which are mostly clicking inside a document.

A model that plans like a person but points badly

Claude-3-Opus, Chapter 7 showed, scored 2.84 to 7.76 points below GPT-4V across every view both models saw. The paper’s own examples of what went wrong read like a catalogue of small, specific misreadings: it treated double-clicking a file as merely selecting it rather than opening it, read column B in a spreadsheet as column C, and typed into a replace box in a code editor without ever pressing the button that actually replaces all the matches.

The device above runs on the same arithmetic as this short function. It is the formula, not a trick.

pythonimport math
def p_hit(w, h, sigma):
    r2 = math.sqrt(2)
    return math.erf(w / (2 * sigma * r2)) * math.erf(h / (2 * sigma * r2))

print(p_hit(120, 28, 12))         # 0.757
print(p_hit(120, 28, 12) ** 6)    # 0.188

A plan that was right until the last inch

What you see. The run ends at its step limit. The trajectory shows the same menu opened again and again, with a different dialog cancelled each time.

What you measure. For every click, compare the item the agent’s own comment names with the item actually sitting under the click, using the positions in the text list. The comment always names Brightness-Contrast; the item under the click changes every time, and the agent’s coordinates move in whole 28-pixel steps of its own guessing.

What you change. Take the position straight from the text list instead of estimating it from the picture, or pick a route that needs no aiming at all: a keyboard shortcut or a command. Then fix the agent itself. Better grounding is the first research direction the paper names, and it is exactly where the next lessons on this path pick up.

A year after the benchmark came out, messages kept arriving. This task’s website has moved its button. This instruction can be read two ways. This checker fails a correct document because one small detail differs. Each message was a crack in the test, and every crack bent somebody’s score.

Before you compare an old score with a new one, check that the test, the time allowed and the person keeping score are all the same.

What the authors said was missing

The paper closes by naming its own gaps. Models that ground their clicks robustly even when a window moves. Memory that can actually use a screenshot from several steps back, not just read it and forget it. Agents that explore and reflect instead of only reacting to the newest screen. Safety: an agent that can use a computer could, for instance, get past a CAPTCHA puzzle meant to keep robots out, and the paper measures no harmful side effects at all, only task success. And more kinds of tasks, collected in easier ways than nine students working for three months.

The repair

On 28 July 2025, the authors announced OSWorld-Verified: a repaired version of the benchmark built from more than 300 pieces of user feedback, each collected, checked and fixed, over about two months of work by a team of roughly ten people. The kinds of problems ranged widely. Websites had changed their layout since 2024. Anti-robot defences and CAPTCHAs now blocked steps that used to work. Tasks tied to real dates, like booking something, had quietly become impossible. Some instructions turned out ambiguous. Some checkers had been too strict all along; one document checker now returns a score from 0 to 1 instead of all-or-nothing. Some tasks had timing problems, such as a file-open step that could block forever. Along the way the virtual machine images shrank from 50 GB to 25 GB, the task files moved from Google Drive to Hugging Face, and three new fields appeared in every task file: proxy, fixed_ip, possibility_of_env_change. The team’s own summary of the whole effort is worth quoting exactly: “providing reliable rewards consumes more human resources than we imagined.”

The task set changed

Fixing the test also changed what it tests. The repaired set drops 8 tasks that need a working Google Drive connection, leaving 361 of the original 369. Today’s task files hold 27 tasks marked impossible, down from 30 at release. And today’s repository carries 49 Windows task files, up from the 43 the paper analysed, as the authors kept adding to that side benchmark too.

The machinery grew

From 22 October 2024 the project also supported running its virtual machines inside Docker containers, alongside other hosts such as VirtualBox, AWS, Azure, Modal and Daytona. On AWS, the project can now run up to 50 environments at once, so a full evaluation of a model across the whole benchmark finishes in under an hour instead of tying up one machine for a day.

The new scores, with their budgets

At the Verified launch, the leaderboard listed a new set of numbers, and this time every row also carries a step budget. CoACT-1 led at 60.76% with 150 steps allowed (59.93% at 100 steps). Agent S2.5 built on the o3 model reached 56.0% at 100 steps. GTA1 built on o3 reached 53.1%. Claude 4 Sonnet reached 43.9%, but only with 50 steps. UI-TARS-1.5-7B’s best run reached 29.6%. And o3 on its own, with no extra scaffolding, scored 9.1% at 15 steps and 23.0% at 100 steps: the very same model, the very same tasks, two very different numbers.

Why the budget matters

Chapter 9’s cascades explain most of this by themselves: with more steps, an agent that notices a miss has room to try again instead of running out the clock on it. The same o3 model went from 9.1% to 23.0%, 2.53 times its own score, with nothing changed except being allowed 100 steps instead of 15.

Today

As of October 2026, the project’s public leaderboard lists a top entry of 90.19% at 100 steps, dated 25 July 2026: 325.59 of the 361 repaired tasks’ worth of credit, with several other entries also above 80% at the same 100-step budget. Every row on that leaderboard is submitted by the team that built the agent, then checked by the OSWorld team before it is listed.

Against people

The 2025 announcement described its own headline result this way: the best system had reached “84.4% of human capability”. That is 60.76 divided by a rounded 72; divide instead by the paper’s own 72.36 and you get 84.0%, a difference small enough from rounding alone. But look closer at what “human capability” means here. People’s 72.36% was measured exactly once, in 2024, on the original 369 tasks and the original checkers. We found no rerun of that human study on the repaired 361-task set. So a claim like “90.19% now beats people’s 72.36%” is quietly mixing two different task sets, measured a year and a half apart, under two different sets of checkers.

One last pointer, not a destination: the project announced a successor, OSWorld 2.0, on 26 June 2026. This lesson stops here and does not cover it.

worked exampleThe fixed task set
  369 − 8 Google Drive tasks = 361
  impossible tasks: 30 at release, 27 in today’s files
  Windows task files: 11 + 9 + 7 + 16 = 43 in the paper;  11 + 9 + 7 + 22 = 49 today

Scores need their budgets and task sets
  2024    15 steps   369 tasks   GPT-4, text list         12.24%
  2025    15 steps   361 tasks   o3                        9.1%
  2025   100 steps   361 tasks   o3                       23.0%     23.0 ÷ 9.1 = 2.53 times
  2025   100 steps   361 tasks   CoACT-1                  59.93%
  2025   150 steps   361 tasks   CoACT-1                  60.76%  = 219.34 of 361
  2026   100 steps   361 tasks   top leaderboard entry    90.19%  = 325.59 of 361 (self-reported)

“84.4% of human capability”
  60.76 ÷ 72    = 0.844   (people’s 72.36% rounded to 72)
  60.76 ÷ 72.36 = 0.840

pythonresult = {
    "agent": "my-agent-v3",
    "task_set": "OSWorld-Verified (361 tasks)",
    "max_steps": 100,
    "runs_per_task": 1,
    "success_rate": 0.2300,
    "wilson_95": wilson(0.2300, 361),     # (0.190, 0.276): Chapter 7's function
    "measured_by": "self",
}

This is a made-up example agent and a made-up 23.0% score, formatted the way any score on this benchmark should be reported. The Wilson range in the comment is our own run of Chapter 7’s function.

A fivefold improvement, read twice

What you see. A slide says: “AI agents improved fivefold in a year on this benchmark: 12.24% to 60.76%.”

What you measure. Write each score’s budget and task set beside it: 15 steps on 369 tasks, against 150 steps on 361 repaired tasks. Then find one agent measured both ways. o3 alone rose from 9.1% to 23.0%, 2.53 times, just from being allowed 15 steps instead of 100.

What you change. Compare at the same budget on the same task set, and say who measured it. The progress underneath is real. The size of it needs the fine print.

You can now read this paper and explain every part of it: why a real computer, how a task is set up and checked, what the agent sees, what the scores mean and where the agents broke. Let’s lock it in.

The one-paragraph summary

OSWorld runs AI agents on a real Ubuntu computer inside a virtual machine. Each of its 369 tasks starts from a shared snapshot, is set up by a few scripted steps, and is scored by a hand-written checker that reads the finished computer’s files, settings or command output: 1 for done, a fraction for partly done, 0 otherwise, and 1 for answering FAIL to one of the 30 impossible tasks. The agent sees a screenshot, a filtered text list of the screen, or both, and answers in pyautogui code, for at most 15 steps. In 2024 people new to the apps finished 72.36% of the tasks and the best model 12.24%. Most failed runs held a click that missed although the plan was right, and the rest lacked knowledge of the apps. A 2025 repair fixed more than 300 issues, and later agents, with far larger step budgets, report 60.76% and more.

The numbers that matter

The checklist

Reading or building an agent benchmark score, ask these ten questions in order.

1. Does every task start from the same saved state, and is it reset between runs?
2. Does the checker read the finished computer (files, settings, command output), not the route or a screenshot?
3. Is the work saved before the checker reads it?
4. Has every checker been run on a wrong final state (no false pass) and on a second right route (no false fail)?
5. Do text checks have exclude lists, and number checks a tolerance that accepts a correct but rounded answer?
6. What does an agent that always answers FAIL score? What does one that answers DONE at once score? Both should be printed beside every result.
7. Is the overall rate an average over tasks, with each category’s size shown?
8. Does every score carry a 95% range, and does any comparison you make survive it?
9. Are the step budget, the task set, the number of runs and who measured it written beside every score?
10. Was the agent also tried on moved, resized and crowded windows?

The run loop, as code

pythonimport math

def wilson(p, n, z=1.96):
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h

def run_benchmark(env, agent, tasks, max_steps=15):
    rewards = []
    for task in tasks:                              # 369 task files (Chapter 4)
        obs = env.reset(task)                       # revert the snapshot, run setup (Chapter 2)
        for t in range(max_steps):                  # at most 15 steps (Chapter 1)
            action = agent.act(task["instruction"], obs)   # pyautogui code, or DONE / FAIL / WAIT (Chapter 3)
            obs, _, done, _ = env.step(action)      # the reward here is always 0
            if done:
                break
        rewards.append(env.evaluate())              # postconfig, getter, metric: 0 to 1 (Chapter 4)
    sr = sum(rewards) / len(rewards)                # average over tasks, not categories (Chapter 5)
    return sr, wilson(sr, len(rewards))             # always with its range (Chapter 7)

This is the shape of OSWorld’s own loop; the names of the methods are ours.

Where it sits on the path

• Just before it. A lesson on agent environments (coming soon on this path), and before that Prime Agent, which closes the act on agent harnesses.
• Just after it. Tools and sandboxing, then failure modes of distributed systems, a lesson on sandbox fleets (coming soon), long-running tasks, τ²-bench and EnvHarness.
• Coming soon on the same path. GUI grounding and ScreenSpot-Pro, on why the clicks miss and how to measure it on professional screens. Computer-use agents, on the loop, action schemas and context. Agent environments and sandbox fleets. Trajectory data. Agentic RL, DigiRL and UI-TARS-2, on training agents on environments like this one. Prompt injection and AgentDojo, on when the screen itself gives the orders.
• Bridges, already on the site. Agent evaluation (tasks, trials, graders and pass@k), the agent evaluation survey, evaluation statistics, error bars for evals, POMDPs, vision-language models, Qwen2-VL.

The paper

“OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments”, by Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu (University of Hong Kong), Shuyan Zhou (Carnegie Mellon University), Silvio Savarese, Caiming Xiong (Salesforce Research), Victor Zhong (University of Waterloo) and Tao Yu (University of Hong Kong). First posted on arXiv on 11 April 2024 (2404.07972; revised 30 May 2024), published in the NeurIPS 2024 Datasets and Benchmarks Track. Code, data and the project page were released the same day as the paper.

What came after

• Docker support for the virtual machines (October 2024).
• OSWorld-Verified, a repair of more than 300 issues (July 2025).
• The public leaderboard, topping 90.19% at 100 steps as of October 2026, self-reported.
• OSWorld 2.0, announced June 2026, not covered in this lesson.

The limits

• The authors’ own, from Chapter 10: grounding that survives a moved window, memory for past screenshots, agents that explore and reflect, and a measure of safety beyond task success.
• Ours: one run per model, so these numbers carry no run-to-run spread of their own.
• A hand-built set of 369 tasks, not a random sample of all the work people do on a computer.
• People were measured once, in 2024, on the original tasks and checkers, never on the repaired set.
• The main study ran on Ubuntu only; the 43 (now 49) Windows tasks were a smaller side study.

Now press Present or Teach and explain, out loud and from memory, why the AI seeing pixels scored 0 on the Bing task although its plan was right. If you can, you own this paper. Then go back to the computer and hand it to each helper once more.