Computer-Use Agents

Your Downloads folder is full of scanned PDFs with names like scan_0412.pdf. You want each one renamed by the title printed inside it: Invoice March, Lease Renewal, Tax Receipt. You type the request into an AI assistant. A plain chat assistant can only answer in words. It cannot open your folder. So picture a different kind of AI: one that uses the computer the way you do, by looking at the screen and using the mouse and the keyboard.

Start with the looking. A program cannot see the glass of a screen the way your eyes do. What it can do is ask the computer for a picture of everything on the screen at one moment: a screenshot. You have made one yourself if you ever pressed the keys that save your screen as an image.

A picture is made of dots. A screen is a grid of tiny coloured dots called pixels, like the tiles of a mosaic. The screenshots in this lesson are 1,280 pixels wide and 720 tall, the size one vendor's docs suggest as a starting point. That is 921,600 tiny dots in one picture.

Then there is the AI itself. The AI is a program that reads text and pictures and writes text back: the model. You have chatted with one already. The model in this lesson is the same kind of program. The difference is that we hand it pictures of a screen as well as words.

A friend on the phone

Think of helping a friend with their computer over the phone. You cannot see their screen. After every move they make, they send you a photo of it. You say one move at a time, and they do exactly that move, nothing more.

Hold on to that picture, because the whole lesson hangs off it. The model is you, on the phone. The photo is the screenshot. And the friend at the keyboard is the ordinary program around the model that takes the screenshots, hands them to the model, reads its answer and moves the real mouse: the harness.

So the model never touches the mouse. It only ever reads a picture and writes a reply. Everything that happens on the real screen, the harness does. A whole lesson, Harness Engineering, is about this kind of program; you do not need it first.

One thing done to the computer, one click, one key press, one burst of typing, is an action. In the phone call, an action is one move: you name it, your friend makes it.

A model that takes actions one after another toward a goal, instead of only answering, is an agent. A chat assistant is an adviser: it tells you what to do. An agent is a helper: it does the work, one action at a time.

An agent whose only way to act is the screen, the mouse and the keyboard, the way a person uses a computer, is a computer-use agent. It needs no special connection to each app. If a person can do a job with a mouse and a keyboard, this agent can try it.

Its rhythm never changes: look, decide, act, look again, until the task is done. That rhythm is a loop. One action and the screenshot that follows it is one step. One attempt at the whole task, start to finish, is one run.

How a model says where to click

The model cannot point. It has only text, so it writes the point down as two numbers. A point on the screenshot is written as two numbers: x counts pixels from the left edge, y counts pixels down from the top. These are its coordinates, like a grid reference on a map.

So "right click at x 600, y 180" means: 600 pixels in from the left edge, 180 pixels down from the top. The harness reads those two numbers, moves the real mouse there and presses the right button. Chapter 2 shows the exact note a real model sends. Chapter 7 shows what goes wrong when the harness reads the two numbers in the wrong size of picture.

One file, five steps

Now walk one file through the loop, in plain words. To rename scan_0412.pdf by the title inside it, the agent needs five actions:

1. Open the PDF: a double click on its name. The next screenshot shows the title, Invoice March. Reading it needs no action: the model reads it straight from the screenshot.
2. Close it: the keys Ctrl and W pressed together.
3. Open the file's menu: a right click on its name.
4. Choose Rename: a click on that item of the menu.
5. Type the title: Invoice March.pdf, and the typing ends with Enter.

Five actions make five steps. They also make six screenshots: one before the first step, so the model knows where it starts, and one after each step, so it sees what its action did. Press through them below. Then turn on the slip and run the same five steps again.

Look at who does what. The screenshot goes one way, a few words of text come back, and only the harness touches the computer. That split will matter in every chapter that follows.

A whole task is many steps

One file took five steps. Ten files at five steps each is 50 steps, and 51 screenshots. That is not an unusual length for this kind of work. One public test of these agents stops each run after 15 steps in its main experiments, and later papers also report runs given 50. One open-source harness's example allows 100. Real tasks run to tens or hundreds of steps.

worked exampleOne file, five steps (toy)                          screenshot after it shows
  1  double click at (600, 180)   open scan_0412.pdf    the PDF, title "Invoice March"
  2  keys Ctrl+W                  close it              the Downloads list again
  3  right click at (600, 180)    open the file's menu  a menu: Open, Rename, Delete
  4  click at (650, 225)          choose Rename         the name in an edit box
  5  type "Invoice March.pdf"     the new name          scan_0412.pdf is now Invoice March.pdf

Ten files                         10 × 5 = 50 steps, 51 screenshots
If each step goes right 98 times in 100 (illustrative):
  all 50 right                    0.98 × 0.98 × … (50 times) = 0.364     about 36 runs in 100

One slip

A step that goes wrong, the click lands on the neighbour or the keys go to the wrong window, is a slip. On the phone call, a slip is a misheard instruction: your friend clicks the row below the one you meant.

The device showed what one slip does. Step 3 landed one row low. Steps 4 and 5 went exactly as planned, and the task, the goal in words, still failed. The agent renamed the wrong file. It never noticed, because each later step was correct for what it saw.

That is the dangerous part. Nothing later in the run repairs a slip on its own. The model chose Rename because a menu was open, and typed the title because a name box was open. Unless something looks back at what step 3 did, the run carries the slip all the way to the end.

How likely is a slip somewhere in a long run? Suppose each step goes right 98 times in 100. That is a made-up number: nobody has measured one step's chance of success for these agents. A 50-step run goes right only if step 1 goes right, and step 2, and every step after. The chance is 0.98 multiplied by itself 50 times: about 0.36.

About one run in three. That is the 36 green dots in the instrument at the top of the page, with 10 files to rename. Chapter 1 derives why the chances multiply, and where a task tips below a coin flip.

So what decides whether an agent finishes? Four levers, in the order this lesson takes them. First, what the agent can do: its menu of actions. Second, what it sees: the screenshot, and what each one costs. Third, what it remembers over a long run. Fourth, whether it notices a slip and repairs it. Behind all four sits one more question: how the agent itself is built inside.

The slip, as a team meets it

The agent renames ten files and reports that it is done. You open the folder. Nine files are right. One, scan_0415.pdf, still has its old scan name.

You read back through the run. Step 18 was a right click meant for scan_0415.pdf. It landed 40 pixels low, on scan_0416.pdf. The menu that opened belonged to the wrong file. Steps 19 and 20 worked perfectly: they chose Rename and typed Gym Membership into the wrong file. Five steps later the agent renamed that file again, with its own title, and never went back to scan_0415.pdf.

Every later step did its job, and the task still failed, because nothing looked back at step 18. Chapter 6 adds the look.

Here is the road, in order:

1. The loop, written down: the parts of the loop, a score at the end, and why the chances multiply.
2. What the agent can do: the menu of actions, and the exact notes real models send.
3. What the agent sees: what a screenshot costs, small targets, and screens described as text.
4. A memory full of pictures: how a long run fills the model's memory, and how to prune it.
5. One model or a team: two ways to build the agent itself.
6. Check, retry, hand over: the check that turns 36 runs in 100 into 98, and what it costs.
7. Reading a failed run: five broken runs, each measured and fixed.
8. Field Guide: the numbers, a checklist, and the loop as code.

Step 12. The agent has just saved a file called export.pdf, and the screenshot shows it sitting in the Downloads list, finished. It is not finished. The computer is still writing it to the disk, and it is only 60% there. The agent cannot know: the screenshot shows the name, not the writing. It moves on and attaches the half-written file to an email.

Nothing in that step was a slip. The click landed where it was aimed, and the screenshot was honest. The trouble is what a screenshot is. It is a picture of the screen, and the screen is not the whole computer.

What the screenshot cannot show

Everything that is true about the computer at one moment, seen or not, every file, every window, a save still running, is the state. The part of the state the agent actually gets to see, here one screenshot, is an observation.

Back on the phone, the photo your friend sends shows one corner of one room. The state is the whole house. The file still being written, the window hidden behind another, the words sitting in the clipboard: none of them is in the photo. Switch between the two views below.

When the agent never sees the whole state and must remember or guess the rest, the problem is partially observable. You work this way whenever you help someone from photos: you keep a picture of the room in your head, built from every photo so far.

So the agent has two jobs at every step. It must choose an action, and it must keep track of what it cannot see: the earlier screenshots, what it already did, what is probably still happening. The first job is this chapter and the next. The second job, memory, is Chapter 4.

Changes, and a score at the end

How one action changes the state is a transition: cause and effect. A double click opens a window. A key press closes one. Some transitions take time. The save in the opening scene kept changing the state for seconds after the click that started it.

The task gives no score along the way. A score the task gives the run at the very end, 1 if the files are right and 0 if not, is the reward: a pass or fail stamp. The desktop test below also gives part marks on some tasks, a fraction between 0 and 1 at the final step.

Someone has to work out that score. A small program that reads the final state and decides the reward is the checker, like the inspector at the end of a factory line. And a run cannot go on forever: the most steps the harness allows before it stops the run is the step budget, like the time limit on an exam.

This whole set-up, a state you cannot fully see, actions that change it, and a reward at the end, has a formal name: a partially observable Markov decision process, or POMDP. Markov means the next state depends only on the current state and the action, not on the path that led there.

Its five parts are written (S, O, A, T, R): the states, the observations, the actions, the transitions and the reward. You have now met every one of them in plain words, so the name only labels what you already built. microPOMDP builds the idea from scratch with a tiger behind a door, and microMDP is the easier case where the agent sees everything. Neither is needed first.

A real checker, line by line

How do people compare two agents fairly? They give both the same tasks and score both the same way. A fixed set of tasks, each with its own checker, used to compare agents fairly, is a benchmark: the same exam paper for every student.

One public benchmark gives agents 369 real tasks on a real Ubuntu desktop (Ubuntu is a free version of the Linux operating system). Its authors call it OSWorld, and this lesson calls it the desktop test. Here is one real task file from it, shortened only where the dots are. Read it once, then line by line below.

json · one task of the desktop test
{
  "id": "030eeff7-b492-4218-b312-701ec99ee0cc",
  "instruction": "Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?",
  "config": [
    { "type": "launch", "parameters": { "command": ["google-chrome", "--remote-debugging-port=1337"] } },
    { "type": "launch", "parameters": { "command": ["socat", "tcp-listen:9222,fork", "tcp:localhost:1337"] } }
  ],
  "evaluator": {
    "postconfig": [ "… close Chrome with pkill, open it again, wait 3 seconds …" ],
    "func": "exact_match",
    "result":   { "type": "enable_do_not_track" },
    "expected": { "type": "rule", "rules": { "expected": "true" } }
  }
}

instruction is the request exactly as a person would type it. It is the only thing the agent is told about its goal.

config holds two commands the harness runs before the agent starts. The first opens Chrome with a debugging port, a door that lets other programs read the browser's settings. The second starts a small relay program that forwards a second port to that door.

evaluator is the checker. Its postconfig closes Chrome and opens it again, so the setting is read fresh. func: exact_match compares two values. result: enable_do_not_track reads the browser's setting through the debugging door. expected says what it must equal: "true".

Notice what is missing. The agent never sees the checker. It gets the instruction and screenshots; the checker reads the state. The agent cannot talk its way to a reward. Either the setting is on when the run ends, or it is not.

Now the test as a whole, with each number made concrete. Of the 369 tasks, 268 happen inside one app (72.6%), and 101 move data between apps (27.4%). Thirty of them (8.1%) are impossible on purpose: the right answer is to say so instead of trying.

People finished 72.36% of the tasks, and took a median of 111.94 seconds each, about three times as long as on an earlier test of web tasks. In the 2024 paper's runs, with a 15-step budget, the best model finished 12.24%. Read that as runs in 100: people finish about 72, the best model of 2024 about 12. That gap is what the rest of the field has been closing.

The loop, as code

Here is the whole loop as a sketch in Python. The names are ours. The shape is the desktop test's own harness, which waits 2 seconds after every action before it takes the next observation.

python · the loop, from scratch
def run_task(task, model, env, budget=15):
    obs = env.reset(task)                    # start state; obs is the first screenshot
    history = []                             # what the model said and did so far
    for step in range(1, budget + 1):        # the step budget
        action = model(task, history, obs)   # the model sees obs, never the state
        if action == "DONE" or action == "FAIL":
            break                            # the model ends the run itself
        obs = env.step(action)               # act, wait 2 s for the screen to settle, screenshot
        history.append(action)
    return env.checker()                     # 1 or 0, read from the final state

Read it from the top. env.reset puts the computer in the task's starting state and returns the first screenshot. The for loop is the step budget: it runs at most 15 times. model is the call to the AI. It gets the task, the history and the current screenshot, and it returns one action.

If the action is DONE or FAIL, the model has ended the run itself. Otherwise env.step acts, waits for the screen to settle, and returns the next screenshot. After the loop, the checker reads the final state and returns the reward.

One line makes all of this partially observable: the model gets obs, never the state. Everything the model knows about the computer arrives through that one screenshot and the history it is handed.

Why the chances multiply

Now the arithmetic Chapter 0 promised. Start with a coin. Flip a fair coin twice. Heads twice in a row happens one time in four: half of the time the first flip is heads, and half of those times the second flip is heads too. Half of a half: 0.5 × 0.5 = 0.25.

Steps work the same way. If one step goes right 98 times in 100, then two steps both go right 98% of 98% of the time: 0.98 × 0.98 = 0.9604, about 96 runs in 100. Each new step keeps 98% of the runs that were still clean.

That rule needs one condition. If one step's slip does not change the chance of the next one, the steps are independent, like separate coin flips. Then the chance of n clean steps in a row is p multiplied by itself n times, which is written p to the power n.

P(finish)

The chance a run of n steps has no slip.

p

The chance one step goes right (made up here: 0.95, 0.98 or 0.99).

n

The number of steps in the run.

Work it all the way out once, at 98 in 100 per step. Every line below is the line above it multiplied by more factors of 0.98, so the numbers can only fall.

worked exampleChance a run has no slip, if each step goes right 98 times in 100 (illustrative)
  1 step     0.98                                  98.0 runs in 100
  2 steps    0.98 × 0.98             = 0.9604       96.0
  10 steps   0.98^10                 = 0.8171       81.7
  20 steps   0.98^20                 = 0.6676       66.8
  30 steps   0.98^30                 = 0.5455       54.5
  34 steps   0.98^34                 = 0.5031       50.3   still above half
  35 steps   0.98^35                 = 0.4931       49.3   below half from here
  40 steps   0.98^40                 = 0.4457       44.6
  50 steps   0.98^50                 = 0.3642       36.4
  100 steps  0.98^100                = 0.1326       13.3

The shortcut
  expected slips in 50 steps      = 50 × 0.02           = 1.0
  ln 0.98                         = −0.0202
  50 × ln 0.98                    = −1.0101
  e^(−1.0101)                     = 0.3642            the same 36.4
  e^(−1), one expected slip       = 0.3679            close: about 37 in 100

The coin-flip line
  n = ln 0.5 ÷ ln 0.98 = −0.6931 ÷ −0.0202 = 34.3 steps
  at 95 in 100: 13.5 steps     at 99 in 100: 69.0 steps

Small changes in p, at 50 steps
  0.99^50 = 0.6050    0.98^50 = 0.3642    0.95^50 = 0.0769
  and at 15 steps, 0.90^15 = 0.2059

There is a shortcut that lets you do this in your head. Multiply the number of steps by the chance of a slip: 50 × 0.02 = 1. That is the expected number of slips per run, the average over many runs, the way 50 flips of a fair coin give 25 heads on average.

One expected slip does not mean every run slips once. Some runs slip twice, and some never slip at all. How many never slip? The second equation answers it: p to the power n is very close to e to the power of minus the expected slips. Here e is the number 2.718…, the base of natural growth and decay.

pⁿ

The chance of no slip in n steps, as before.

n(1−p)

The expected number of slips per run.

ln p

The natural log of p. For p near 1 it is very close to −(1 − p): ln 0.98 = −0.0202.

e

The number 2.718…; e to the minus 1 is 0.368.

0.98⁵⁰ = e^{50 × (−0.0202)} = e^−1.01 = 0.364: about 36 runs in 100.

So a run that expects one slip comes out clean about 37 times in 100, whatever the step chance and length that produced the one slip. Two expected slips: e to the minus 2, about 14 in 100. Three: about 5 in 100. The shortcut turns any agent's numbers into one quick question: how many slips does a run expect?

Last, the coin-flip line. 0.98³⁴ = 0.503 and 0.98³⁵ = 0.493. So at 98 in 100 per step, a task of 35 steps or more finishes less than half the time. Drag the device below past that line, then change the step chance.

Read the device at 50 steps. At 99 in 100 per step, 60.5 runs in 100 finish. At 98 in 100, 36.4 finish. At 95 in 100, only 7.7. One extra slip per hundred steps, 99 down to 98, cuts the finished runs from 60.5 to 36.4 in 100. Three more, down to 95, cut them to 7.7.

Short tasks are not safe either when steps are weak. At 90 in 100 per step, a 15-step task finishes 20.6 times in 100. The length of the task and the quality of each step trade against each other, and the curve shows exactly how.

Keep one caution in mind. This is a model, not a measurement. Real agents do not slip at a fixed rate, and some slips can be undone. Chapter 6 compares the model with what real agents scored when they were given more steps. The model's job is to show the shape of the danger: long tasks punish small error rates.

Done is not the score

The agent's last message says: Done, all ten files renamed. The checker opens the folder, compares the ten names with the ten titles, and scores the run 0. One file is wrong.

Which one should you believe? The checker. The agent's final message is not the score. A sample harness from one vendor puts it in one line: "A final answer does not prove the task succeeded. Inspect the result before relying on it." The reward comes from the state, read by a checker, never from the agent's own report.

The agent has a spreadsheet open and one job: select the cells from A1 to C5. It can see exactly where they are. But the only actions its harness offers are click and type. A click selects one cell. Typing writes into it. There is no way to press the mouse button, hold it, move, and let go. The task is impossible, and no amount of intelligence fixes it.

That is worth sitting with. The model in this scene is not confused. It knows what to do. The harness simply never gave it the move. The full menu of actions the agent may choose from is its action space, like the moves allowed in a game, and this menu is missing one.

Name the kind of move first. An action on the mouse and keyboard themselves, like a click at a point or a key press, is a low-level action: pressing the buttons yourself. In the phone picture, the action space is the list of moves you are allowed to say to your friend.

Each action earns its place

Every action on a good menu is there because some everyday task fails without it. Walk through them, each with the task that needs it:

• A double click opens a file. (A click to select it and then the Enter key does the same job.)
• A right click opens a file's own menu, where Rename lives.
• A drag, or a press, a move and a release, selects a range of spreadsheet cells.
• Resting the pointer on something without clicking is a hover; the small label that then appears is a tooltip. A hover is how you read what an unlabelled icon does.
• A scroll reaches the bottom of a long page. (The Page Down key does too.)
• Keys save a file with Ctrl+S.
• A triple click selects a whole paragraph. (A drag across it does too.)

Notice the brackets. Some tasks have two routes, so losing one action does not always lose the task. Turn the actions off one at a time below, and watch which tasks die and which survive on a second route.

What the vendors chose

The vendor menus on the device are real. Anthropic's computer toolset has 17 actions, from screenshot and zoom through left_click, left_click_drag, left_mouse_down, left_mouse_up, scroll, type, key, hold_key and wait. OpenAI's computer tool has 9 action types: click, double_click, drag, move, scroll, keypress, type, wait and screenshot. It folds the right click into click, with a button field that says which button to press.

The desktop test from Chapter 1 gives agents 13 action types, plus 3 special ones that pause or end a run: WAIT, FAIL and DONE. Both vendors' menus cover all seven everyday tasks on the device.

Why would a menu keep both a drag and a separate press, move and release? A drag is one action from a start point to an end point. Some apps need the slower version, with the press and the release as actions of their own. Anthropic's docs say exactly this about spreadsheets: use left_mouse_down and left_mouse_up.

worked exampleWhich toy tasks survive which menu (the lesson's own reasoning, not a measurement)
  menu                         actions on   tasks possible   first task that fails, and what it needs
  click and type               2            0 of 7           open a PDF: a double click, or a click and Enter
  click, type and keys         3            3 of 7           rename from the menu: a right click
  OpenAI's 9                   8 of the 10  7 of 7           (none; right click is click with a button field)
  Anthropic's 17               all 10       7 of 7           (none)
Drop one action from the full menu: only 5 actions cannot be dropped
  left click, right click, type   → the rename fails        move (hover) → the tooltip fails
  key                             → saving with Ctrl+S fails
  drag, double click, triple click, press-move-release, scroll → nothing fails: another action covers it

Real menus
  Anthropic computer toolset   17 actions    key repeat 1 to 100, wait up to 300 s
  OpenAI computer tool          9 action types, several per call in an ordered list
  desktop test (computer_13)   13 + WAIT, FAIL, DONE

How a model asks for an action

The model cannot move a mouse, so it writes a short structured note that names an action and its inputs, and the harness carries it out: a tool call. On the phone, a tool call is dictating one move in a fixed form, so your friend cannot mishear it.

The note is written in a plain-text format of names and values inside curly braces, which programs read easily: JSON. Think of a filled-in form: a field name, a colon, a value. (Agents & Tool Use covers tool calls for any tool. This chapter covers the computer's.)

Here is a tool call exactly as one vendor's docs show it, from Anthropic's computer use documentation:

json · the model's note
{
  "type": "tool_use",
  "id": "toolu_01Qg8m3XqC5aRy7tD2eS4jUg",
  "name": "left_click",
  "toolset_name": "computer",
  "input": { "coordinate": [500, 300], "text": "shift" }
}

Read it a line at a time. "type": "tool_use" says this is a tool call, not ordinary text. "id" names this one call, so the harness's answer can point back at it. "name": "left_click" is the action. "toolset_name": "computer" says which set of tools the action came from.

"input" holds the action's inputs. "coordinate": [500, 300] is where to click, in the pixels of the screenshot the harness last sent, with the origin at the top left. The optional "text": "shift" holds the Shift key down during the click.

How did the model know these actions exist? The request that starts the run offers the whole menu in one line:

json · the menu, offered
"tools": [{ "type": "computer_toolset_20260801" }]

That one entry gives the model all 17 actions. There is no field for the screen's size. Coordinates are always in the pixels of the screenshot the harness returned, so the harness alone must know how that picture relates to the real screen. Chapter 7 is about what happens when it forgets.

Several actions in one answer

A model can also ask for more than one action at once. Several actions in one answer, which the harness runs in order, are a batch. On the phone: "click there, then type this, then send me a photo." Here is the harness's answer to a three-action batch, a click, some typing and a screenshot, shortened from the same docs:

json · the harness's answers
[
  { "type": "tool_result", "tool_use_id": "toolu_01HqCF3nJ4Vzr8sTkPZ2wxYA", "toolset_name": "computer",
    "content": [{ "type": "text", "text": "OK" }] },
  { "type": "tool_result", "tool_use_id": "toolu_01Ppr3sZ3TnE9m6VUu4RyH2K", "toolset_name": "computer",
    "content": [{ "type": "text", "text": "OK" }] },
  { "type": "tool_result", "tool_use_id": "toolu_01Xf5W1sD8Q9aBcJ7kLmN2pQ", "toolset_name": "computer",
    "content": [{ "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "iVBORw0KGgo…" } }] }
]

Each action gets one tool_result, matched to its call by tool_use_id. Only a screenshot or a zoom answer carries an image. The others just say OK, which keeps the reply short.

What if the click fails? The docs give a firm rule: run the actions in order and stop at the first failure. The failed action's result carries "is_error": true and a reason. Every later action in the batch gets "is_error": true and exactly the text Not executed: an earlier computer action in this turn failed. The reason is simple. The typing only makes sense if the click before it put the cursor in the right box.

One note on words, so the chapters stay exact. From here on, a step is still one action. One call to the model may now carry several steps.

The other vendor writes the same idea in a different shape. OpenAI's computer_call carries an ordered actions list, here a click at (405, 157) and then the word penguin:

json · the other vendor's call
{ "type": "computer_call", "call_id": "call_002",
  "actions": [ { "type": "click", "button": "left", "x": 405, "y": 157 },
               { "type": "type", "text": "penguin" } ],
  "status": "completed" }

"status": "completed" means the model finished writing the call. It does not mean anything happened on screen yet; the harness still has to run it. The harness answers with a computer_call_output that carries a computer_screenshot, sent at "detail": "original" so no detail is lost:

json · and its answer
{ "type": "computer_call_output", "call_id": "call_002",
  "output": { "type": "computer_screenshot", "image_url": "data:image/png;base64,…", "detail": "original" } }

Same idea, different shape. Always read your own vendor's docs, and never mix the two formats in one harness.

Aim at elements, or hand over a program

Everything so far aims at points. There is another way to ask. One thing on the screen a person can use, a button, a text box, a menu item, is an element. An action aimed at an element by its name or number instead of at a point is an element action: "press the red button", not "press 3 cm from the left".

One research agent tags every element on screen with an integer and lets the model say "click element 14". One browser library offers three such moves, act, observe and extract, and hands back the exact address of each element in the page, so that passwords never reach the model. The trade: an element action is exact when the element exists, and useless when the thing is not an element. Chapter 3 meets a chart that is not.

A third way hands over a whole program. The model writes a short program that does several actions in a row: a code action. On the phone, it is a written recipe instead of spoken moves.

The desktop test lets models write PyAutoGUI code, a Python library that moves the mouse and presses keys, because, in its authors' words, it "saves tokens for describing action space definition in prompting". OpenAI now recommends code execution, a model writing PyAutoGUI or Playwright code, for its newest model, and keeps the structured tool as an alternative.

One research agent argues the other way: "Code blocks can contain multiple sequential actions, leaving the agent with neither control over nor feedback from individual steps." Fewer calls, less feedback. Try both failure cases below. (Code as Agent Harness follows the code side much further.)

None of the three ways wins everywhere. Points survive a slow screen and a drawn menu, at the price of a model call per action. Elements are exact, until the thing on screen is not an element. Code is fast and cheap in calls, and blind between its lines. A good harness picks the shape on purpose, and knows which failure it is choosing to risk.

Refuse before you act

Models sometimes write impossible notes. The docs warn that the model "might make mistakes or hallucinate when outputting specific coordinates". So the harness checks every note before it acts: a known action name, the inputs that action needs, text that is not empty, and a point inside the screenshot.

The vendor's reference harness raises an error for a point off the screen, with the message "Coordinates {x}, {y} are out of bounds". The docs say to report any failed action as a tool_result with "is_error": true. A refused note never touches the screen. The model reads the reason in its next call, and tries again. Here is the whole check, written from scratch:

python · the harness's check before acting
W, H = 1280, 720                                    # the screenshot's size in pixels
NEEDS = {"left_click": ["coordinate"], "right_click": ["coordinate"],
         "double_click": ["coordinate"], "type": ["text"], "key": ["text"],
         "scroll": ["scroll_direction", "scroll_amount"], "screenshot": []}

def check(call):
    name, inp = call["name"], call.get("input", {})
    if name not in NEEDS:
        return f"unknown action {name}"
    for field in NEEDS[name]:
        if field not in inp:
            return f"missing {field}"
    if "text" in NEEDS[name] and not inp["text"]:
        return "empty text"
    if "coordinate" in inp:
        x, y = inp["coordinate"]
        if not (0 <= x < W and 0 <= y < H):
            return f"Coordinates {x}, {y} are out of bounds"   # the reference harness's wording
    return None                                     # None means: safe to run

# reason = check(call); if reason: answer is_error with the reason, and run nothing.

Read the order of the tests. An unknown name fails first, because nothing else about the note can be trusted. Then each required input. Then empty text, because typing nothing is never what the model meant. Then the point: x must be at least 0 and less than the width, and y the same for the height.

Mind the edge. On a screenshot 1,280 pixels wide, the columns are numbered 0 to 1,279. A click at x 1,280 is already off the picture, which is why the test is "less than W", not "at most W".

The click that was never possible

Step 2 of a run asks for a click at x 1,400 on a screenshot 1,280 pixels wide. The harness passes it straight to the mouse. In this toy the point is pushed back to the right edge, the click lands on empty desktop, and the open menu closes. The next three steps act on a menu that is no longer there.

Compare each requested point with the screenshot's size: 1,400 is not less than 1,280. The note was impossible before anything ran. No later step could have saved it, because the damage happened on the screen.

So check every note before acting: refuse it, return is_error with the reason, and let the model try again. The menu stays open, and the model's next note lands on Rename. One cheap test turned a failed run into a single wasted call.

A designer works on a 4K monitor, 3,840 pixels wide and 2,160 tall. At the top of an expense form sits a checkbox labelled Include tax, 16 pixels wide. The agent is asked to tick it. The agent will never see that screen at 3,840 pixels. Before the model sees anything, the harness shrinks the picture, and by the time the model looks, the checkbox is about 5 pixels wide.

Five pixels is smaller than the dot on this letter i. To understand why it shrank, and what it costs to keep it big, we need two things: how big a picture a model accepts, and how a model reads a picture at all.

Shrinking the screen

How many pixels wide and tall a picture is, written width × height, is its resolution: the size of a photo. Shrinking a picture to fewer pixels is downscaling, the way a phone makes a thumbnail of a photo.

Why must the harness shrink? Each model accepts pictures only up to a size. In one vendor's docs, newer models take up to 2,576 pixels on the long edge. Older ones take up to 1,568 pixels on the long edge and about 1.15 million pixels in all. A screenshot over the limit is not shrunk for you: the request is refused.

So the harness downscales every screenshot itself, before it sends it, and keeps the number it scaled by. Chapter 7 needs that number, because the model's clicks come back in the small picture's pixels.

How a model reads a picture

Start with text. A model reads in small pieces called tokens; for text, a token is about a short word or part of a word, like the pieces of a puzzle. Every model has a limit on how many tokens it can read at once, which is Chapter 4's subject. And every token costs time and money.

Pictures are read in pieces too. The model cuts a picture into squares 28 pixels on a side; each square is a patch, and it reads each patch as one token, a visual token. In the phone picture, your friend's photo arrives as a mosaic, and you look at it one tile at a time.

A patch holds 28 × 28 = 784 pixels. Whatever sits inside those 784 pixels, the model takes in as one piece: a whole button, half a word, or a tiny checkbox together with everything around it.

A partial square at the edge still costs a whole token, so the count rounds up in each direction. The rule from the vendor's docs is:

tokens

The visual tokens one screenshot costs the model.

w, h

The screenshot's width and height in pixels.

28

The side of one patch, in pixels.

⌈ ⌉

Round up to the next whole number: a partial patch still costs a token.

Work it for this lesson's screenshots. 1,280 ÷ 28 = 45.71, which rounds up to 46 patches across. 720 ÷ 28 = 25.71, which rounds up to 26 patches down. So one screenshot costs 46 × 26 tokens:

⌈1,280 ÷ 28⌉ × ⌈720 ÷ 28⌉ = 46 × 26 = 1,196 tokens

Is that really the rule the docs use? Check it against the docs' own table of examples. 1,000 × 1,000 costs 36 × 36 = 1,296, and the table says 1,296. 1,920 × 1,080 costs 69 × 39 = 2,691, and the table says 2,691. 2,576 × 1,449 costs 92 × 52 = 4,784, the most a newer model accepts. The rule reproduces every row. The docs' own rule of thumb, "roughly 1,000 to 1,800 input tokens" per screenshot, agrees.

worked exampleOne screenshot, 1,280 × 720, read in 28-pixel patches
  across   1,280 ÷ 28 = 45.71   round up → 46 patches
  down       720 ÷ 28 = 25.71   round up → 26 patches
  tokens   46 × 26              = 1,196

The rule against the docs' own table
  1,000 × 1,000   36 × 36   = 1,296   ✓
  1,920 × 1,080   69 × 39   = 2,691   ✓
  2,576 × 1,449   92 × 52   = 4,784   ✓   the most a newer model accepts
  1,024 ×   768   37 × 28   = 1,036       768 ÷ 28 = 27.43, the last row is a partial patch

The checkbox, from a 3,840-wide screen
  16 × 1,280 ÷ 3,840 = 5.33 pixels      5.33 ÷ 28 = 0.19 of a patch's side
  (5.33 ÷ 28)² = 0.036                  under 4% of one patch's area

A text view instead
  raw accessibility tree  usually over 1,000,000 tokens
  filtered                about 6,000 tokens covers 90% of screens
  1,000,000 ÷ 6,000 ≈ 167 times smaller

Now put the two halves together: the shrinking, and the patches. Pick a screenshot size below and watch both the price and the checkbox.

Small targets

Read the device at 1,280 × 720. The 16-pixel checkbox is now 5.3 pixels wide: 16 × 1,280 ÷ 3,840. That is a fifth of a patch's side, and under 4% of a patch's area. One patch holds the checkbox, a slice of its label and some empty form, and the model takes it all in as one piece.

The same problem shows up in the research. One paper points to "10×10 pixel icons in a 1920×1080 image". Shrunk to 1,280 wide, such an icon is about 6.7 pixels across.

Two fixes work. A bigger screenshot keeps more detail, at more tokens on every step and only up to a ceiling. Or the agent zooms: an action that sends back a close-up of one region at full detail is zoom, like asking your friend for a close-up photo of one corner. One fix did not work: the vendor reports that splitting a screenshot into quadrants, or drawing a coordinate grid on it, did not improve click accuracy.

So what size should you send? Different sources give different starting points. Each row below names its source. Do not average them; each one fits its own models.

Why do they differ? Different models, and different ways of counting a picture's tokens. OpenAI's own per-image formula is not in this lesson's sources, so this chapter's arithmetic uses the 28-pixel rule only. Follow your own vendor's guide.

Screens described as text

Some agents read a description of the screen instead of, or beside, a picture. Computers keep a list of every element on screen, with its kind, name and position, so that screen-reader software can read the screen aloud to blind users. That list is the accessibility tree. In the phone picture: instead of a photo, your friend reads you the table of contents of the screen.

A web browser keeps the same kind of list for a web page; it is called the DOM. An agent that works only inside a browser can read the DOM directly.

The catch is size. On the desktop test, the raw tree is usually over a million tokens, far more than a model can read. So the harness filters it, keeping elements by their kind and whether they can be seen. After filtering, about 6,000 tokens covers 90% of screens: a cut of about 167 times.

The text view comes in more than one shape. The desktop test writes one row per element, separated by tabs: its kind, name, text, position and size. One open-source browser harness writes [index]<tagname attribute=value /> for each element you can use, marks elements new since the last step with *[, and by default lists only what is in view.

One research agent tags every element with an integer and adds the screen's text, read by software that reads the letters out of a picture: OCR, optical character recognition. It adds that text for anything the tree missed. Why would a tree miss anything? Switch the views below, then add a chart.

A part of a window that the app paints itself, a chart or a drawing board, with no elements inside it for the tree to list, is a canvas: a painting on the wall, not a shelf of labelled objects. To the tree, a canvas is one row with no children. Anything written inside it is invisible to a text-only agent.

No source measures how often this happens. But the research agent's fix shows the problem is real: it adds OCR text blocks to the tree "if they do not already exist in the tree". Reading the pixels catches what the list leaves out.

Pictures or text?

Which is better, then? Here is what one study measured, explained before the numbers. The same 369 desktop tasks, the same 15-step budget, and four ways for the model to see the screen: the tree, screenshots alone, both together, or screenshots with numbered marks drawn on them.

In 2024, reading the tree, the best model finished 12.24%. With screenshots alone, the four models tested finished between 2.42% and 5.80%. With both, 12.17% and 11.21%. People finished 72.36%. The same paper found three more things: higher resolution usually helped, more text history helped, and more screenshot history did not.

Then 2025. Agents that see only screenshots, trained for screens, finished 22.7% (a one-model agent) and 27.0% (a team agent) under the same 15-step budget. That is roughly twice the best 2024 tree reader: 22.7 ÷ 12.24 = 1.85, and 27.0 ÷ 12.24 = 2.21. Pictures caught up once models were trained to read screens. Chapter 5 opens both of those agents.

The total no one could read

An agent that reads only the accessibility tree is asked: what does the expense chart say the total is? It answers that it cannot find a total. A person glances at the screen and reads Total: 83 in the chart's title.

Print the tree. The chart is one row: canvas, no name, no text, a position and a size. The words Total: 83 were painted as pixels inside it, so no element holds them. The agent did not misread anything; it was never given the words.

Give the agent the screenshot too, or add the screen's text through OCR as extra rows, the way one research agent does. Now the total is a row the agent can read.

Step 98 of a long run. The agent has renamed sixty files without a fault. The harness sends call 98 and gets back an error instead of an action: the call is too long for the model to read. Nothing went wrong on the screen. The agent ran out of room for its own history.

To see how a run runs out of room, we need one surprising fact about models, one limit, and a little addition.

The model remembers nothing between calls

Here is the surprise for anyone who has chatted with an AI. Each call to the model starts from nothing. The model keeps no memory of the last call at all. A chat only seems to remember, because the program around it sends the whole conversation again, every time you press Enter.

A computer-use harness does the same. One vendor's sample loop appends each answer and each result to a list of messages, and sends the whole list on the next call. The other vendor can keep the history on its own side and lets the harness point at it by name. Either way, the model reads all of it, on every call.

Everything the model can read in one call is its context window, measured in tokens; this lesson uses a window of 200,000 tokens, the size the vendor's own advice is written for. In the phone picture, the window is your desk. It holds only so many photos and notes, and when it is full, nothing more fits.

What one call carries

So what is on the desk at call k? Three things. First, a fixed part: the descriptions of the 17 actions, about 4,500 tokens, plus the task and the system text, which we set at a toy 200. Second, the text of every earlier step: the model's thought, its action and the harness's reply, a toy 800 tokens per step. Third, every screenshot kept, at 1,196 tokens each for 1,280 × 720.

C_k

Tokens in call k.

F

The fixed part: about 4,500 tokens of action descriptions plus a toy 200 of task text.

T

Text per step, a toy 800 tokens.

S

Tokens per screenshot, 1,196 at 1,280 × 720.

m_k

Screenshots kept at call k.

g_k

Screenshots pruned so far, each leaving a 5-token placeholder (toy).

Work it with every screenshot kept, so m_k = k and nothing is pruned. Call 1 carries 4,700 + 800 + 1,196 = 6,696 tokens. Call 50 carries 4,700 + 40,000 + 59,800 = 104,500 tokens, which is 52% of the window. Call 98 carries 4,700 + 78,400 + 117,208:

keep every one: C₉₈ = 4,700 + 78,400 + 117,208 = 200,308 > 200,000

That call does not fit, and the run stops there. Look at which part grows fastest. Each step adds 800 tokens of words and 1,196 of picture, about 2,000 in all, and the pictures are 60% of every step's growth. By step 100 the pictures alone would be 100 × 1,196 = 119,600 tokens, 60% of the whole window.

There is a second limit too. One vendor holds every image in a request to a stricter size limit once the request carries more than 20 images. Then each screenshot must be 2,000 pixels or less per side, or the request must keep 20 or fewer images. A run that keeps every screenshot passes 20 images at step 21.

Play with all three parts below. Change the screenshot size first, then the way of keeping screenshots, and drag along the run.

Keep the last few pictures, and all the words

The obvious fix is to keep only recent screenshots, and it has plenty of precedent. The desktop test's baseline agents saw the 3 most recent observations. One vendor's reference harness keeps the 3 most recent screenshots by default. A one-model agent keeps its last few screenshots inside a 32,000-token limit, and keeps every earlier thought and action as text.

The evidence agrees with them. In the desktop test's study, more text history helped the agents, and more screenshot history did not. Old pictures are not thrown away silently, either. The harness replaces an old screenshot with a short note, [Image omitted]: it prunes the screenshot and leaves a placeholder, like clearing old photos off the desk and leaving a sticky note where each one was.

Now a subtlety that changes the answer. If the start of a call is exactly the same as last time, the provider can reuse the work it already did on that start: a prompt cache. In the phone picture it is a bookmark in a long letter, and it only works if nobody changed the earlier pages.

Dropping the oldest screenshot at every step changes an early part of every call, so the cache must be rebuilt every time. The docs say so plainly: "Dropping a screenshot every turn changes the prefix every turn and invalidates the cache." The prefix is the start of the call. On the device, Keep the last 3 changes the start of 97 of 100 calls.

Prune in batches

The docs' default is a middle way: keep the last three screenshots and prune every 25 turns, "so the prefix stays byte-identical between prune events". The docs say turn for one model call; in this lesson's toy, one turn is one step.

Walk the mechanism. Screenshots pile up to 28. The 29th triggers one prune, which replaces the oldest 25 with placeholders and leaves 4. They pile up again to 28. In a 100-step run the prunes fall at steps 29, 54 and 79: three rebuilds of the cache instead of 97.

The price is that a call can carry up to 28 screenshots. That is over 20, so each screenshot must stay at 2,000 pixels or less per side; 1,280 × 720 does. The docs also say where to put cache markers: one after the system text and the action descriptions, and up to three on the most recent steps. (Context & Streaming, in its chapter on the cache boundary, builds the cache idea in full.)

worked exampleTokens in call k, keeping every screenshot:  4,700 + 800 × k + 1,196 × k
  step 1     4,700 +    800 +   1,196    =   6,696
  step 50    4,700 + 40,000 +  59,800    = 104,500     52% of 200,000
  step 97    4,700 + 77,600 + 116,012    = 198,312     still fits
  step 98    4,700 + 78,400 + 117,208    = 200,308     does not fit: the run stops here
  pictures alone at step 100: 100 × 1,196 = 119,600, 60% of the window
  at 1,920 × 1,080 (2,691 each) the same run stops at step 56

Keeping the last 3 screenshots, every step
  step 100   4,700 + 80,000 + 3 × 1,196 + 97 placeholders × 5 = 88,773     44%
  the start of the call changed on 97 of 100 calls: the cache is rebuilt 97 times

Keep 3, prune 25 at a time
  screenshots 1, 2, … 28, then the 29th triggers a prune: 29 − 25 = 4
  prunes at steps 29, 54, 79 (and 104)        the cache is rebuilt 3 times in 100 steps
  step 100   25 screenshots: 4,700 + 80,000 + 29,900 + 75 placeholders × 5 = 114,975   57%
  the most screenshots in one call: 28, over 20, so each must be at most 2,000 px per side

The vendor's docs add model-specific advice for their newest models about pruning on the client, so read your model's section before you copy this default.

Longer than about 50 actions

Pruning pictures still leaves the words, and the words keep growing: 800 tokens a step in our toy. A rolling set of recent screenshots alone is fine for short tasks, "under ~50 actions", in the vendor blog's words. Beyond that, the blog says to add compaction.

The harness asks the model to write a summary of everything so far, and replaces the old history with it: compaction, like replacing a stack of notes with one page of minutes. The run keeps its story in fewer words, and the window has room again.

Another tool lives outside the window entirely. A note the agent writes to disk and reads back later, like a to-do list, is a memory file. One browser harness keeps a todo.md for tasks of 10 steps or more, and tells the model not to bother for shorter ones. (Agent Memory covers memory files in depth.)

Two harnesses, one long task

A harness keeps every screenshot. A 120-step task fails at step 98 with an error that the request is too long. A second harness, rewritten in a hurry, drops the oldest screenshot at every step. It finishes, but the bill for the run is several times what the team expected.

Log the tokens in each call. The first harness grows by about 2,000 tokens a step, 800 of text and 1,196 of picture, and crosses 200,000 at step 98. The second stays under 90,000 until step 100, but its log shows that the start of the call changed on almost every step, so the cache never helped.

Keep every thought and action as text, keep the last three screenshots, and prune 25 at a time. The calls stay under 126,000 tokens for all 120 steps, and the start changes only at steps 29, 54, 79 and 104. For runs past about 50 actions, add a summary.

Two teams build the same file-renaming agent. The first team trains one model to look at the screenshot and write the whole answer: what it sees, what it will do, and the exact point to click. The second team wires together several parts: one plans the job, one decides the next move in words, and specialists turn words like the Rename item into a point. Both agents get the same 369 tasks. Which one finishes more, and why?

Until now we treated the model as one box that reads a screenshot and writes an action. This chapter opens the box. There are two ways to build what is inside, and the best agents measured so far use both.

One model does everything

Start with the first team's answer. One network that does everything, sees the screen, finds the target, thinks, remembers and writes the action: a native agent model, the one-model design. In the phone picture, it is you alone on the phone, doing every part of the job yourself.

The best-documented example sees only screenshots: no accessibility tree, no page text. Its authors call this "pure-vision": the model relies only on pictures of the interface.

How does one network learn all of that? Showing a model many examples so that its numbers adjust and it gets better at a job is training, the way practice works for a person. This one started from a general model that reads pictures and text, and was trained further on about 50 billion tokens built to teach it screens and actions. It also learns from its own corrected mistakes; the fine print under the device names the method.

It thinks before it acts. A short sentence the model writes about what it sees and plans before it acts is a thought, like thinking aloud. The design writes a thought and then an action at every step. (ReAct introduced thinking before acting.)

A measured twist: when the model drew a single answer, writing thoughts first did slightly worse than acting directly, on all three of the authors' tests. When it drew 16 or 64 answers and the best one was kept, thoughts clearly won. Thinking pays when there is room to choose.

What does it read at step n? The task, every earlier thought and action as text, the last N screenshots, and the current one. From those it writes the thought and the action:

t_n, a_n

The thought and the action the model writes at step n.

task

The instruction.

t₁, a₁ …

Every earlier thought and action, kept as text.

o_n−N … o_n

Only the last N screenshots and the current one.

P( … | … )

The chance the model gives to writing this thought and action, given everything after the bar. (A simplified form of the paper's equation; the paper also pairs each recent screenshot with its thought and action.)

This is Chapter 4's policy built into a model: keep the words, keep a few pictures, inside a 32,000-token limit. Its menu has click, drag, scroll, type, wait and hotkeys, plus two ways to stop. Finished() says the task is done. CallUser() hands the step to a person, for things like a login.

A team of parts

Now the second team's answer. Several parts, each with one job, wired together around general models: a modular framework, the team design. In the phone picture, it is an orchestra with a conductor.

One research team's version has three kinds of part. The part that breaks the task into smaller pieces is the planner; each piece, like "open the first PDF and read its title", is a subgoal. The part that picks the next action for the current subgoal is the worker. A third part judges finished subgoals and writes what it learned into two memories, one for whole tasks and one for subgoals.

This first version reads an accessibility tree whose elements carry integer tags, with OCR text added. And it acts once per step, in its authors' words "so it can observe immediate feedback".

Turning a description such as "the Rename item" into a point on the screenshot is grounding, like finding a street on a map from its name. A part that does only that is a grounding expert.

The team's second version drops the tree. It sees only screenshots and its action history, and the worker routes each target to one of three experts. A visual expert finds things by sight; it is itself a one-model agent, the largest version of the one above. A text expert finds words with OCR. A spreadsheet expert writes cells through the spreadsheet program's own interface instead of clicking.

One more change: rewriting the plan after every finished subgoal, not only after a failure, is proactive replanning, like checking the route at every junction instead of only when you are lost. Send one action through each design below, for three kinds of target.

Why splitting the job can help

Why would a team do better? Here is a toy derivation. A step goes right only if the plan was right, the target was found, and the action was carried out as meant. If those three are independent, their chances multiply, just as Chapter 1's steps did:

p

The chance one step goes right.

p_plan

The chance the plan for this step was right (illustrative).

p_ground

The chance the target was found (illustrative).

p_act

The chance the action was carried out as meant (illustrative).

Take illustrative values: 0.99 for the plan, 0.97 for grounding, 0.995 for the action. Then p = 0.99 × 0.97 × 0.995 = 0.9555, and a 15-step run is clean 0.9555¹⁵ = 50.5% of the time. Now raise only grounding, to 0.99. Then p = 0.9752, and 15 steps are clean 68.6% of the time.

One factor moved by 0.02 and the clean runs rose by 18 in 100. Improving the weakest factor lifts the whole product most. So a team that puts a specialist on its weakest factor gains the most.

worked exampleA step goes right only if all three parts go right (toy values, independent)
  p = p_plan × p_ground × p_act
    = 0.99 × 0.97 × 0.995     = 0.9555     15 steps: 0.9555^15 = 0.505
  raise grounding to 0.99:
    = 0.99 × 0.99 × 0.995     = 0.9752     15 steps: 0.9752^15 = 0.686

Measured on the desktop test, 369 tasks, 15-step budget
  general model alone (tree + screenshot)       11.21%
  first team agent                              20.58%    +9.37 points, 83.6% more
  one-model agent (screenshots only)            22.7%
  second team agent (screenshots only)          27.0%     27.0 ÷ 22.7 = 1.189: 18.9% more
Failed tasks of the first team agent, by error kind (one task can have several)
  adjusting while acting 79.59%   grounding 53.06%   planning 34.69%

What failed, and what scored

Which factor is weakest for real agents? The first team agent's authors read every failed run on part of the desktop test and sorted the errors into three kinds; one run can have several. A wrong plan appeared in 34.69% of failed tasks. A missed target, which is grounding, appeared in 53.06%. A failure to adjust while acting, such as repeating the same action, appeared in 79.59%.

Planning was the smallest problem. Finding things and adjusting were the biggest. That is why the second version spent its effort on grounding experts and on replanning.

Now the scores, with what is compared said first: the same 369 tasks, a 15-step budget. The first team agent finished 20.58%, where the same general model alone, reading the tree and the screenshot, finished 11.21%. That is 9.37 points more, or 83.6% more in relative terms. On a 65-task part of the test, its interface changes alone moved 10.77% to 12.31%, and adding its memory lookups moved it to 20.00%.

The second team version finished 27.0% at 15 steps, against the one-model agent's 22.7%: 18.9% more in relative terms. Each of its new pieces mattered more with more steps. The grounding experts moved 27.69% to 30.77% at 15 steps, and 33.85% to 38.46% at 50. Replanning added 4.62 points at 15 steps and 6.15 at 50.

One more result turns the contest around. As the visual expert inside the team, a smaller version of the one-model agent did better than the general model it replaced: 29.23% against 24.61%. The two designs are not simply rivals.

The plan that never changed

A team agent plans four subgoals at the start of a task. During the second, a dialog the planner never expected appears. The worker keeps trying to finish subgoal 2 as planned, and the run runs out of steps.

Read its trace: the plan never changed after step 1. The planner only replanned after a failure, and the worker never declared one; it just kept trying.

Replan after every finished subgoal, not only after a failure. In the second version's tests, this proactive replanning added 4.62 points at 15 steps and 6.15 at 50.

The agent presses Ctrl+S. A small window pops up: a file with that name already exists; replace it? The agent does not look. Its next three clicks land on the window behind, and nothing happens. Then it writes its final message: Done, the report is saved. It is not.

Every chapter so far tried to make each step less likely to slip. This one accepts that slips will happen, and asks a different question: what if the agent noticed?

Look after every step

The vendor's own docs name the problem directly: "Claude sometimes assumes outcomes of its actions without explicitly checking their results." And they give a fix, a few sentences to add to the model's instructions: "After each step, take a screenshot and carefully evaluate if you have achieved the right outcome. State in one sentence what the screenshot shows and whether the step succeeded. If it didn't, try again."

In the phone picture, it is a rule you give your friend's helper: after every move, look at the photo, and say in one sentence what it shows and whether the move worked. Only then make the next move.

What does a check buy? Derive it. A step can now end right in two ways. It works the first time, with chance p. Or it slips, with chance 1 − p, the check notices, and the step is done again and works.

Two new words name those parts. The share of slips the check notices is its catch rate, written d, like a proofreader who catches 9 typos in 10. Doing the step again after a caught slip is a retry, like redialling a dropped call; a retry goes right with the same chance p.

Add the two ways together, and one step ends right with chance q:

q

The chance one step ends right, check included.

p

The chance one try goes right (illustrative).

1 − p

The chance it slips.

d

The catch rate: the share of slips the check notices.

the last p

The chance the retry goes right.

A run of n steps still needs every step to end right, so it finishes with chance q to the power n. And each caught slip costs one more action, so a step costs 1 + (1 − p) · d actions on average:

P(finish)

The chance a run has no uncaught slip.

n

Steps in the task.

actions per step

One action, plus a retry for each slip the check catches.

Now the worked number, at 98 in 100 per step with a check that catches 9 slips in 10. One step: q = 0.98 + 0.02 × 0.9 × 0.98 = 0.98 + 0.01764 = 0.99764. Fifty steps: 0.99764⁵⁰ = 0.8886. Without the check, 0.98⁵⁰ = 0.3642.

0.98 + 0.02 × 0.9 × 0.98 = 0.99764; 0.99764⁵⁰ = 0.889, against 0.98⁵⁰ = 0.364

With a perfect check, d = 1, the formula tidies up: q = p + (1 − p) · p = 2p − p² = 1 − (1 − p)². In words, a step now fails only if the first try and the retry both slip. At 98 in 100 that is q = 1 − 0.02² = 0.9996, and 0.9996⁵⁰ = 0.9802: the instrument's 98 runs in 100. At 95 in 100 a perfect check turns 7.7 finished runs in 100 into 88.2.

What the check costs

The price is small. Each caught slip costs one more action: 1 + 0.02 × 0.9 = 1.018 actions per step, so a 50-step task takes about 51 actions instead of 50. The check also costs words: one sentence of the model's own text per step, which Chapter 4's history keeps. Against that, 0.8886 − 0.3642 = 0.5244: 52.4 more finished runs in 100.

Two retries help a little more. At 9 slips in 10 caught, the chance per step becomes 0.98 × (1 + 0.018 + 0.018²) = 0.997958, and 50 steps finish 90.3 times in 100. Most of the gain comes from the first retry.

Be plain about the limits of this toy. Real slips are not independent, and real checks miss in ways this model does not capture. A model can look at a screenshot and be wrong about it. The toy shows the size of the lever, not a promise.

Bound the run

A check can also loop: retry, fail, retry, fail, forever. So the harness needs limits that the model cannot argue with. The other vendor's guide puts the rule in one line: "Bound and verify the run. Set step, time, or cost limits, support cancellation, and check the actual outcome instead of relying only on the model's final answer."

The desktop test's main runs stop at 15 steps, and later reports also give agents 50. What do the extra 35 steps buy? That depends on the agent. Extra steps help only an agent that can use them: to finish a long task, or to undo a slip and try again. An agent that goes wrong and never notices gains nothing from more steps.

Here are four real agents measured both ways, explained before you look. The same 369 desktop tasks, screenshots only, once with a 15-step budget and once with 50. A task that reaches the step limit before the agent says done counts as failed.

Read the bars. OpenAI's agent gained 12.9 points, from 19.7% to 32.6%. Claude, used directly, gained 10.5, from 15.5% to 26.0%. The team agent of Chapter 5 gained 7.5, from 27.0% to 34.5%, and solved 29 tasks, 7.9% of the test, only when given 50 steps. The one-model agent gained 1.9, from 22.7% to 24.6%.

The one-model agent's authors framed the same numbers the other way: its 22.7% at 15 steps matched the 22.0% that an earlier version of Claude scored at 50 steps, as measured in their own paper. Both readings are true. Which one matters depends on whether you pay per step.

Can Chapter 1's multiplying model predict these bars? Try it. If every slip were fatal and every task took all 15 steps, a 22.7% score would mean each step goes right about 0.906 of the time (0.227 to the power 1/15). Then a 50-step task would finish 0.906⁵⁰ = 0.7% of the time. But the 50-step score is 24.6%.

The model fails because it mixes up two things. Tasks are not all 15 steps long, and the budget is a limit, not a length: most tasks that finish need fewer steps than the budget allows. The p-to-the-n model is a picture of the danger, not a way to predict a real agent's score.

worked exampleOne step, checked, with one retry (illustrative p and d)
  q = p + (1 − p) × d × p
  p = 0.98, d = 0.9:   q = 0.98 + 0.02 × 0.9 × 0.98 = 0.98 + 0.01764 = 0.99764
  50 steps:            0.99764^50 = 0.8886   → 88.9 runs in 100
  no check:            0.98^50    = 0.3642   → 36.4 runs in 100
  perfect check, d = 1: q = 1 − 0.02² = 0.9996,   0.9996^50 = 0.9802
  at p = 0.95, d = 1:  q = 1 − 0.05² = 0.9975,   0.9975^50 = 0.8824  (no check 0.0769)
  two retries, d = 0.9: q = 0.98 × (1 + 0.018 + 0.018²) = 0.997958,   q^50 = 0.9028

The price
  actions per step     = 1 + (1 − p) × d = 1 + 0.02 × 0.9 = 1.018
  a 50-step task       = 50 × 1.018 = 50.9 actions, about 51
  gain                 = 0.8886 − 0.3642 = 0.5244: 52.4 more finished runs in 100

Real agents, 15 steps against 50 on the desktop test
  OpenAI's agent        19.7 → 32.6   +12.9
  Claude, used directly 15.5 → 26.0   +10.5
  team agent            27.0 → 34.5   +7.5      29 tasks solved only at 50 steps = 7.9%
  one-model agent       22.7 → 24.6   +1.9

Break loops, hand over, confirm

Agents also get stuck in a different way: they repeat themselves. A rule in the harness that notices the agent repeating itself and makes it try something else is a loop breaker, like a friend saying "you've tried that three times". One browser harness tells its model: "if you are on the same URL for 3+ steps without meaningful progress, or the same action fails 2-3 times, try a different approach."

Some steps should not be the agent's at all. The agent stops and asks a person to do a step, such as typing a password: a hand-over, like passing the phone to the owner. The one-model agent has an action for exactly this, CallUser(), "invoked in cases requiring user intervention, such as login or authentication".

The harness asks a person before a consequential action, such as deleting data or paying: a confirmation, the familiar "are you sure?". The other vendor's integration guide sorts actions into three levels.

• A person must take over: the final step of a password change; going past a security warning or a paywall.
• Always confirm: deleting data, changing permissions, CAPTCHAs, installing software, sending or posting to others, money, system settings, medical actions.
• A standing approval can be enough: logging in to a site the user asked for, browser permission prompts, uploads, moving files.

Two sentences from that guide are worth keeping on a card: "The model's request to act is not user permission." And: "Typing sensitive data into a form counts as transmission." Both vendors also advise running agents in an isolated machine with limited access. A later lesson in this path covers attacks hidden in what the agent reads.

Last, the final check is not the model's. One vendor's sample harness says it in one line: "A final answer does not prove the task succeeded. Inspect the result before relying on it." The checker of Chapter 1 reads the state. Your own harness should too, before it tells a person "done".

The window nobody looked at

The agent reports: Done, the report is saved. The checker opens the folder and finds yesterday's report, unchanged. The run scores 0.

Read screenshots 10, 11 and 12 of the trace. All three show the same small window asking to replace the file. The model's thoughts for those steps never mention it; they say the file is saved.

Add the check prompt, so each step begins with one sentence about what the screenshot shows. Treat replacing a file as deleting data, so the harness asks a person first. And before telling anyone done, read the folder yourself.

A teammate sends you a run that failed at step 23 with a one-line note: the model is bad at clicking. You open the run's record. The first two clicks, near the top left of the screen, hit their targets. Every click after that misses, and the misses grow toward the bottom right. A model that is bad at clicking does not miss in a pattern.

This chapter is about reading that record. Most failed runs can be explained from it, and most of the explanations turn out to be the harness's fault, not the model's.

The run's record

The step-by-step record of a run, each screenshot, thought, action and result, with times, is the run's trace. Think of the flight recorder in an aircraft. In the phone picture, it is a recording of the whole call, with every photo kept. Here is one step of a trace, in this lesson's own format, one JSON line per step:

json · one step of a trace
{"step": 4, "screenshot": "run17/step04.png", "size": [1330, 864], "screen": [1512, 982],
 "thought": "The menu is open. I will click Rename.",
 "action": {"name": "left_click", "input": {"coordinate": [665, 432]}},
 "applied_at": [665, 432], "result": "OK", "ms": 2140}

Read its fields. step is the step number. screenshot names the saved picture, and size is that picture's resolution. screen is the real display's resolution. thought and action are what the model wrote. applied_at is the point the harness actually clicked on the real screen. result is what the harness reported back, and ms is the milliseconds since the last step.

applied_at is the field most harnesses forget to record, and it is the one that finds the first bug. Look at this record again: size and screen differ, and applied_at equals the model's coordinate. Something is missing between the two.

The vendor's own table of symptoms

One vendor's docs list four click symptoms, each with its usual cause. Read them as a doctor's chart: what you see, what it usually means, and what to try.

The teammate's run matches the first row exactly: offset in one direction. Here is why that happens, worked through with the docs' own example.

The scale bug, derived

Chapter 3 said the harness shrinks each screenshot and keeps the number it scaled by. The number the harness multiplied the screen's size by to make the screenshot is the scale factor, like the scale printed on a map. The docs give a rule for it, for older models with a long-edge limit L and a total-pixel limit A:

s

The scale factor.

L

The longest edge a model accepts: 1,568 pixels for older models.

A

The most pixels it accepts: about 1,150,000 for older models.

w, h

The screen's width and height.

The docs' example screen is 1,512 × 982, which is 1,484,784 pixels. The long-edge test gives 1,568 ÷ 1,512 = 1.0370. The total-pixel test gives √(1,150,000 ÷ 1,484,784) = 0.8801. The smallest of 1, 1.0370 and 0.8801 is 0.8801, so the screenshot is 1,512 × 0.8801 = 1,330 wide and 982 × 0.8801 = 864 tall.

The model answers in the screenshot's pixels. So before clicking, the harness must turn every point back into the screen's pixels, by dividing by s:

x_shot, y_shot

The point the model wrote, in screenshot pixels.

x_screen, y_screen

Where the harness must click on the real screen.

665 ÷ 0.8801 = 755.6; forgetting it lands 90.6 pixels short

If the harness forgets, every click lands at x instead of x ÷ 0.8801. It falls short by x × (1 ÷ 0.8801 − 1), which is 13.6% of x, and the same share of y. Near the top left, 13.6% of a small number is a few pixels: a click at x 120 is only 16 pixels short. Near the bottom right it is large: a click at x 1,200 is 164 pixels short. That is the teammate's pattern exactly.

One kind of screen doubles the trap. On some screens, such as a Mac's Retina display, each point of the screen's own coordinates is 2 pixels wide in a screenshot: a device pixel ratio of 2, two tiles per grid square. The docs give two ways out: downscale the screenshot by 2 before sending it, or halve the coordinates the model returns. Forget it, and a click meant for the middle of the screen lands near its bottom-right corner.

worked exampleThe docs' scale example
  screen 1,512 × 982 = 1,484,784 pixels
  long edge:   1,568 ÷ 1,512 = 1.0370
  pixel total: √(1,150,000 ÷ 1,484,784) = 0.8801
  s = min(1, 1.0370, 0.8801) = 0.8801      screenshot = 1,330 × 864

Where the clicks should land, and where they land if the harness forgets to divide by s
  model says (x, y)   should click (x ÷ s, y ÷ s)   clicks at (x, y)   short by
  (120, 80)           (136.4, 90.9)                 (120, 80)          (16.4, 10.9)
  (665, 432)          (755.6, 490.9)                (665, 432)         (90.6, 58.9)
  (1,200, 800)        (1,363.5, 909.0)              (1,200, 800)       (163.5, 109.0)
  short by 13.6% of each coordinate: 1 ÷ s − 1 = 0.1363   (s = 0.88007 before rounding)

Retina: the same screen captured at a device pixel ratio of 2
  capture 3,024 × 1,964 → s = 0.4400 → the same 1,330 × 864 screenshot
  model says (665, 432) → ÷ 0.4400 → (1,511.2, 981.7) in capture pixels
  halve for the screen's own points → (755.6, 490.9)
  forget to halve → (1,511, 982): the bottom-right corner of a 1,512 × 982 screen

The fix is one line, in the code below, using the docs' rule. Then open the device and read five broken runs, this one first.

python · the fix, with the docs' scale rule
import math
screen_w, screen_h = 1512, 982

def scale_factor(w, h):                        # older models: 1,568 px long edge, about 1.15 MP
    return min(1.0, 1568 / max(w, h), math.sqrt(1_150_000 / (w * h)))

s = scale_factor(screen_w, screen_h)           # 0.8801
shot_w, shot_h = int(screen_w * s), int(screen_h * s)   # 1330 × 864: send the screenshot at this size

def to_screen(x, y):                           # the model's pixels → the screen's
    return x / s, y / s                        # (665, 432) → (755.6, 490.9)

Four more ways a run breaks

Focus. The window or box that receives your typing has the focus, like whoever holds the microphone. In the device's second run, a notification takes the focus between the click on the name box and the typing, and the typing goes into the notification's reply box. Measure: the screenshot after the typing shows a highlighted ring around the reply box, and the file's name unchanged. Fix: click the target again, check in the next screenshot that it has the focus, then type.

The dialog. A small window that blocks the rest of the app until you answer it is a dialog, a door you must open before going on. The third run is Chapter 6's replace-the-file window. Measure: three identical screenshots in a row, all showing the dialog, while the thoughts never mention it. Fix: the check prompt, and a confirmation before replacing a file.

Too early. A short pause after each action, so the screen finishes changing before the screenshot, is a settle delay, like waiting for a photo to come into focus. The vendor's reference harness waits 2.0 seconds before its screenshot, and the desktop test's harness waits 2 seconds after every action. In the fourth run, a harness with no delay clicks Download report 0.4 seconds after asking for the page, and the page needs 1.5 seconds (both toy numbers). The click hits a loading placeholder. Measure: the trace's times, and a spinner in the screenshot. Fix: settle for 2 seconds, or let the model ask for a wait, which can last up to 300 seconds.

Scrolling forever. In the fifth run the agent scrolls down nine times in an app where the mouse wheel does nothing, and gets nine identical screenshots. The docs name this case: "In applications where scrolling doesn't take effect, keyboard alternatives such as Page Down can help." Measure: compare each screenshot with the one before; zero pixels changed for three steps in a row. Fix: a loop breaker, Chapter 6's three-step rule, and the Page Down key.

One more message to recognise in a trace: an error that reads "Coordinates 1400, 215 are out of bounds" is the harness refusing an impossible point, as the reference harness does. That is Chapter 2's check doing its job. The fix belongs in the model's next step, not in the harness.

The teammate's run, resolved

The teammate's harness resized each screenshot from 1,512 × 982 to 1,330 × 864 and never divided the model's coordinates by 0.8801. The first two clicks were near the top left, where 13.6% of a small number is a few pixels, so they hit. The later clicks were 71 to 164 pixels short across, and 45 to 109 pixels short down. Add one line, x ÷ 0.8801 and y ÷ 0.8801, and the same model, on the same task, clicks every target.

A computer-use agent is a model in a loop: a screenshot in, an action out, a new screenshot, for tens to hundreds of steps. The model only writes text; the harness does everything else. It decides the menu of actions and checks every request, sizes the screenshot and scales the clicks back, keeps the history inside the model's window and its cache, asks for a check after each step, stops the run, hands risky steps to a person, and reads the real result at the end. Small error rates per step become large failure rates per task, so most of the work of building an agent is in that harness.

The numbers

Every number this lesson used, with where it comes from. The step chances in the instrument and in Chapters 0, 1 and 6 are made up to show the idea; everything else below is measured or documented.

The checklist

1. Size the screenshot on purpose: start at 1,280 × 720 on Anthropic models, or follow your own vendor's guide, and keep the scale factor.
2. Divide every coordinate by the scale factor before clicking, and halve it again on a Retina Mac if your capture is twice the screen's points.
3. Check every action before running it: a known name, its required inputs, text that is not empty, a point inside the screenshot.
4. Run a batch in order and stop at the first failure; answer each later action with the halt text.
5. Wait about 2 seconds after each action before the screenshot, and let the model ask for longer waits.
6. Ask for a check after every step: one sentence on what the screenshot shows and whether the step worked.
7. Cap the run in steps, time and cost, and let a person cancel it.
8. Break loops: if the screen has not changed for 3 steps, tell the model and make it try another way.
9. Keep every thought and action as text; keep the last 3 screenshots; prune old ones 25 at a time, or follow your model's own docs.
10. Keep a request to 20 images or fewer, or keep each image at 2,000 pixels or less per side.
11. Hand passwords to a person, and confirm deletes, payments, installs and anything sent to others.
12. Before saying done, read the real result yourself.
13. Record a trace: each screenshot, thought, action, the point actually clicked, the result and the time.

The loop, as code

A complete harness loop for Anthropic's computer toolset, in about 90 lines. Run it only inside a virtual machine or a container with nothing on it you would mind losing. It uses PyAutoGUI to act, sizes screenshots with the docs' rule, refuses impossible points, runs batches in order and stops at the first failure, waits 2 seconds after each action, asks for a check after every step, prunes screenshots in batches, and stops at a step budget.

python · fg_loop.py
import base64, io, math, os, time
import anthropic, pyautogui                      # run this only inside a virtual machine or a container

client = anthropic.Anthropic()
MODEL = os.environ["MODEL"]                       # a model that supports the computer toolset
TOOLS = [{"type": "computer_toolset_20260801"}]   # 17 actions, about 4,500 tokens of descriptions
HALT = "Not executed: an earlier computer action in this turn failed."
CHECK = ("After each step, take a screenshot and carefully evaluate if you have achieved the right "
         "outcome. State in one sentence what the screenshot shows and whether the step succeeded. "
         "If it didn't, try again.")              # the docs' own wording, first three sentences

SW, SH = pyautogui.size()                         # the real screen, in its own points
S = min(1.0, 1568 / max(SW, SH), math.sqrt(1_150_000 / (SW * SH)))   # the docs' rule for older models
W, H = int(SW * S), int(SH * S)                   # the screenshot the model sees, and keeps

def shot():                                       # resize every capture to W x H (this also undoes Retina's 2x)
    buf = io.BytesIO(); pyautogui.screenshot().resize((W, H)).save(buf, format="PNG")
    return {"type": "image", "source": {"type": "base64", "media_type": "image/png",
                                        "data": base64.b64encode(buf.getvalue()).decode()}}

KEYS = {"return": "enter", "page_down": "pagedown", "page_up": "pageup"}

def act(name, inp):                               # one action; raise on anything it must refuse
    if name == "screenshot":
        return [shot()]
    x = y = None
    if "coordinate" in inp:
        x, y = inp["coordinate"]
        if not (0 <= x < W and 0 <= y < H):
            raise ValueError(f"Coordinates {x}, {y} are out of bounds")
        x, y = x / S, y / S                       # scale back up to the screen
    if name == "left_click":     pyautogui.click(x, y)
    elif name == "right_click":  pyautogui.click(x, y, button="right")
    elif name == "double_click": pyautogui.doubleClick(x, y)
    elif name == "type":         pyautogui.write(inp["text"])
    elif name == "key":          pyautogui.hotkey(*[KEYS.get(k.lower(), k.lower()) for k in inp["text"].split("+")])
    elif name == "scroll":
        n = inp["scroll_amount"] * (1 if inp["scroll_direction"] == "up" else -1)
        pyautogui.scroll(n, x, y)
    elif name == "wait":         time.sleep(inp["duration"])
    else:
        raise ValueError(f"unknown action {name}")
    time.sleep(2.0)                               # let the screen settle before anyone looks
    return [{"type": "text", "text": "OK"}]

def images(messages):                             # every screenshot block, oldest first
    found = []
    for m in messages:
        for blk in m["content"] if isinstance(m["content"], list) else []:
            if not isinstance(blk, dict):
                continue
            if blk.get("type") == "image":
                found.append((m["content"], m["content"].index(blk)))
            for i, c in enumerate(blk.get("content", []) if isinstance(blk.get("content"), list) else []):
                if c.get("type") == "image":
                    found.append((blk["content"], i))
    return found

def prune(messages, keep_n=3, interval=25):       # the blog's batch rule: keep 3, prune 25 at a time
    found = images(messages)
    if len(found) <= keep_n + interval:
        return
    for where, i in found[:-keep_n][-interval:]:
        where[i] = {"type": "text", "text": "[Image omitted]"}

def agent(task, budget=50):
    messages = [{"role": "user", "content": [{"type": "text", "text": task}, shot()]}]   # words before the picture
    for _ in range(budget):                       # the step budget, counted in calls
        prune(messages)
        resp = client.messages.create(model=MODEL, max_tokens=4096, system=CHECK,
                                      tools=TOOLS, messages=messages)
        messages.append({"role": "assistant", "content": resp.content})
        calls = [b for b in resp.content if b.type == "tool_use"]
        if not calls:
            return "the model stopped: now check the real result yourself"
        results, failed = [], False
        for b in calls:                           # a batch: in order, and stop at the first failure
            r = {"type": "tool_result", "tool_use_id": b.id, "toolset_name": "computer"}
            if failed:
                r.update(is_error=True, content=HALT)
            else:
                try:
                    r["content"] = act(b.name, b.input)
                except Exception as e:
                    r.update(is_error=True, content=str(e)); failed = True
            results.append(r)
        messages.append({"role": "user", "content": results})
    return "budget spent: stopped by the harness"

A sketch, checked to compile, not run against a live model. Its choices follow the docs: the toolset entry and member names, the result shapes, the halt text, the out-of-bounds message, the scale rule, the Retina note, the check prompt, instruction text before the screenshot, the 2-second settle, the batch prune and max_tokens=4096.

Find each chapter in the code. act is Chapters 2 and 7: the menu, the refusal of impossible points, and the scaling back to the screen. shot and the lines that set S, W and H are Chapters 3 and 7: the screenshot's size and the scale factor. prune is Chapter 4: keep 3, prune 25 at a time. CHECK and the budget are Chapter 6: look after every step, and stop.

Where this lesson sits

This lesson is in Act 4, action space and harness, of the path Computer Use & Multimodal Agents. Before it: Harness Engineering, the program around any model. After it: CS 8803-LLM S07: Agent Harness, workflow memory and SDK design.

Around it in the same act: Agents & Tool Use, ReAct, MCP, Claude Plays Robotics (the same loop with a robot body), The Agent Loop, Context & Streaming, Agent Memory, Code as Agent Harness and Prime Agent.

Coming in this path (not yet published): how a model finds the button (GUI grounding) and the ScreenSpot-Pro benchmark; agent environments, the OSWorld paper in full, and sandbox fleets; trajectory data, agentic reinforcement learning, DigiRL and UI-TARS-2; prompt injection and AgentDojo.

Sources

1. Anthropic, computer use tool docs. platform.claude.com
2. Anthropic, vision docs. platform.claude.com
3. Anthropic, "Best practices for computer and browser use with Claude". claude.com/blog
4. Anthropic, computer-use-demo reference implementation. GitHub
5. OpenAI, computer use guide and integration guide. guide · integration
6. OpenAI, "Computer-Using Agent", 23 January 2025. openai.com
7. OpenAI, openai-cua-sample-app. GitHub
8. Xie et al., "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments", 2024. arXiv · repository
9. Qin et al., "UI-TARS: Pioneering Automated GUI Interaction with Native Agents", 2025. arXiv
10. Agashe et al., "Agent S: An Open Agentic Framework that Uses Computers Like a Human", 2024. arXiv
11. Agashe et al., "Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents", 2025. arXiv
12. browser-use, README and system prompt. GitHub
13. Stagehand, README. GitHub

Go back to the Downloads folder at the top of the page. Set 20 files at 95 in 100 per step: about 1 run in 100 finishes. Switch on Check each step: 78 do. Every control there now has a chapter behind it: the files are Chapter 1's multiplication, the chance per step is what Chapters 2, 3 and 7 improve, and the check is Chapter 6. Then open Teach and draw the loop from memory, or Present it in your own words. Back to the Downloads folder