Trajectory Data

Picture a new assistant at your desk. You say: “Find last month's electricity bill in my email and save it in the Bills folder.” They look at the screen, open the mail program, type in the search box, open the bill, click Download, find the folder. A dozen small moves, each one after a look at the screen.

Now picture a program doing the same job, with nothing but the screen to look at and the mouse and keyboard to act with. How would it learn? The same way your assistant did: by watching someone do the job, many times, and copying.

The loop every agent runs

Start by naming the parts. A program that does a job on a computer for you works in a loop: it looks at the screen, picks one thing to do, does it, and looks again. Such a program is an agent. One that works only through the screen, the mouse and the keyboard, as you do, is a computer-use agent.

What it looks at is a picture of the whole screen at one moment: a screenshot. Think of a photograph of the desk, taken just before the assistant moves. What it does is one thing done to the computer: a click, some typing, a scroll, a key press. That is an action, one move of the assistant's hand.

One look at the screen and one action together make a step, one beat of the job. The bill task above is about a dozen steps. Here is the loop the agent runs, step after step:

Every computer-use agent runs this loop, a few times for a short task and dozens of times for a long one.

What learning means here

The part of the agent that decides is the model: a program that learned its behaviour from examples instead of being written rule by rule. Think of a student, not a rulebook. Nobody typed in “if you see a search box, click it”. The model picked that habit up from examples.

How does it pick it up? Show the model a screen and the task. It guesses an action. Compare its guess with what the person did, and nudge the model a little toward the person's choice. Do that millions of times. Showing a model many examples and nudging it after each one, so that its answers move closer to the right ones, is training. It is an apprentice corrected after every try.

This lesson is about the examples: where they come from, what they cost, and how to tell a good one from a bad one. It is not about the inside of the model. Any model that learns by copying needs examples, and for computer use they are scarce.

Copying people

The most natural example is a person doing the task while a program records every screen and every action. That recording is a demonstration: the chef cooks while the apprentice watches. Learning a job by copying demonstrations is imitation learning. Its plainest form, copying the action the person took on each screen, is behaviour cloning.

Behaviour cloning has one weakness worth knowing now. A copy that drifts off the person's path meets screens no demonstration ever showed, and it never saw how to get back. Chapter 2 shows how one dataset turns people's mistakes into lessons about getting back on track. The site's Imitation Learning lesson covers this drift, and its fixes, in depth.

How we know it learned

A student who memorised last year's answers can still fail this year's exam. So we keep a set of tasks apart, never used in training, and score the model on them. That set is a test: an exam with questions the student has not seen. Its score says how the model does on something new.

There are two natural scores. The share of steps on which the model picks the same action the person did is its step accuracy, like marks per question. The share of whole tasks that finish is the task success rate, like passing the whole exam. They sound alike. They are very different numbers, and the difference is the reason this lesson exists.

A chain of steps

Think of a long task as a chain, with each step one link. If one link breaks, the chain breaks. One wrong click sends the bill to the wrong folder, and the task has failed, even though every other step was right.

Now put numbers on it. If each step is right with chance p, and steps go wrong on their own, a task of n steps finishes only when all n are right: p times p times p, n times. One wrong step can sink the whole task, so a long task is only as strong as its weakest step. This is the weakest link effect.

The numbers are harsh. At 90% right per step, a 5-step task finishes 59 times in 100. At 95%, 77 times. At 99%, 95 times. A desktop task in one large recorded dataset averages 18.6 steps. There, 99% per step finishes 83 tasks in 100, 95% finishes 39, and 90% finishes only 14.

Try it yourself. The circles are the steps of one task, and the squares are 100 tries at the whole task.

Two things stand out as you drag. The curve falls fast as the task gets longer. And the gap between 90% and 99% per step, which sounds small, is the gap between a model that usually fails and one that usually finishes.

Read it backwards

Here is where the rule comes from. Take a task of 2 steps. The first is right with chance p. If it is right, the second is right with chance p too. Both right: p × p. Each extra step multiplies by p again, so for n steps the chance is p × p × … × p, n times, which is p to the power n.

This assumes each step goes right or wrong on its own. Real mistakes cluster: a model that misreads one screen may misread the next. So treat the rule as a first guess. The builders of the phone dataset we meet below made the same assumption when they named the weakest link effect.

S

The task success rate: the chance the whole task finishes.

p

The step accuracy: the chance the model picks the right action on one step.

n

The number of steps in the task. Each step multiplies by p once.

0.90⁵ = 0.59  ·  0.95⁵ = 0.77  ·  0.99⁵ = 0.95

Now turn the question around. Suppose you want 95 tasks in 100 to finish. How often must each step be right? We want p to the power n to equal 0.95, so take the n-th root of both sides: p = 0.95 to the power 1 ÷ n. For 5 steps that is 0.9898, so 98.98% of steps must be right. For 18.6 steps it is 0.9972, or 99.72%.

So the question is not whether a model is good. It is whether it is nearly perfect. Two lines of Python do both sums:

pythonp, n = 0.90, 5
print(p ** n)             # 0.59: the chance the whole task finishes
print(0.95 ** (1 / n))    # 0.9898: the step accuracy needed for 95 tasks in 100

How far is a trained model?

Here is a real case. Picture a 2024 dataset of phone tasks recorded by 20 trained people over a year. Call it the phone set. It holds 15,283 demonstrations across 833 apps, 5.5 steps each on average. Its builders set 13,604 of them aside as its training part, and kept the rest for tests.

The best model its builders trained picked the right action on 71.5% of steps when told only the goal, and on 86.6% when told each step in words. Both sound respectable. Run them through the chain, though, and they are not: the story at the end of this chapter does that sum.

How many demonstrations would it take?

The builders asked the obvious next question: how much more data would it take? They trained models on more and more of their data. Step accuracy rose by about the same amount every time the data grew tenfold. Drawn on a scale of powers of ten, the points lie on a straight line, and the fit is very good.

Then they extended that straight line. For 95% right steps on apps it trained on, it needs about 500,000 demonstrations when each step is told in words, and 1 million when only the goal is given. For 95 in 100 five-step tasks to finish, it needs about 2 million.

There is a second catch. Tested on the same apps it trained on, a model is in domain, like an exam on the textbook's own exercises. Tested on apps it has never seen, it is out of domain, like an exam on new problems. Out of domain the same line asks for far more: 10 million, 60 million and 150 million demonstrations. That is “one to two orders of magnitude” more, in the builders' own words.

The pace of people

Now count the cost in time. Those 15,283 demonstrations took 20 trained people about a year. At that pace, 2 million demonstrations would take 131 years, and 150 million would take 9,815 years.

Count it in money too. Another 2025 dataset of desktop tasks was recorded by 634 part-time annotators on Windows, macOS and Ubuntu. Call it the desktop set. It cost about $1.41 a task in all. At that price, 2 million tasks cost about $2.8 million, and 150 million about $212 million. More people shorten the years, not the dollars.

worked exampleThe weakest link, 5 steps
  90% right per step:  0.90 × 0.90 × 0.90 × 0.90 × 0.90 = 0.590 → 59 tasks in 100
  95% right per step:  0.95 to the power 5             = 0.774 → 77 in 100
  99% right per step:  0.99 to the power 5             = 0.951 → 95 in 100
A desktop task of 18.6 steps (the desktop set's average)
  0.90 → 0.141      0.95 → 0.385      0.99 → 0.829
Backwards: step accuracy for 95 tasks in 100
  5 steps:     0.95 to the power 1/5    = 0.9898 → 98.98%
  18.6 steps:  0.95 to the power 1/18.6 = 0.9972 → 99.72%
The phone set's best model, told only the goal
  71.5% right steps, 5 steps: 0.715 to the power 5 = 0.187 → about 19 tasks in 100
How many demonstrations, and how long people would take
  the phone set's training part:                     13,604 demonstrations
  its pace:                                          15,283 demonstrations a year, 20 people
  2,000,000 (95 in 100 five-step tasks, same apps):  ÷ 15,283 = 131 years; 147 times the training part
  150,000,000 (the same, on new apps):               ÷ 15,283 = 9,815 years
  at about $1.41 a task:  2,000,000 → $2.8 million;  150,000,000 → $212 million

So programs must make them

Examples made by a program instead of by people are synthetic data. This lesson builds three ways to make them:

• replay the how-to pages already on the web (Chapter 3);
• tap around first and write the task afterwards (Chapter 4);
• let a model set itself tasks and check them (Chapter 5).

The shelf at the top of the page showed the catch. Programs are cheap, but they make failures, and they make repeats. So every source needs a check that a run really did its task, and a check for repeats, before a model learns from it. Those two checks are Chapters 6 and 7.

Right on most steps, wrong on most tasks

What the team sees. A team trains a model on 13,604 recorded phone tasks. It picks the right action on 71.5% of steps when told only the goal, and they expect it to finish most short tasks. On five-step tasks it finishes few.

What you measure. Multiply instead of hoping: 0.715 to the power 5 is 0.187. If steps go wrong on their own, fewer than 1 task in 5 finishes. The step score was never the number to watch; the task score is its fifth power.

What you change. Set the goal per step, not per task: 99% right steps for 95 in 100 five-step tasks. The builders' own trend says that takes about 2 million demonstrations, 147 times what they had. So plan for synthetic data, and for the checks it needs.

Here is the road, in order:

1. Anatomy of a trajectory: what one recorded task holds, and how it becomes training examples.
2. People recording tasks: from raw mouse and key signals to clean steps, and what each task costs.
3. Replaying tutorials: the web's how-to pages, followed by an agent while a recorder runs.
4. Explore first, name it after: tap around first, then write the task from what happened.
5. Let the model set the task: four model roles that propose, refine, describe and check.
6. The judge at the door: how often a pass-or-fail check lets failures in, and where.
7. Repeats and leaks: near-copies, and test tasks hiding in the training data.
8. Is the data any good?: what more data, and mixed data, really buy.
9. Field Guide: the checklist, the code, and every number with its source.

In the shelf at the top, People is Chapter 2, Exploring is Chapters 4 and 5, the judge is Chapter 6, and one square is Chapter 1.

A person sits down to do one small job: find the word ‘Travelers’ in an open document and search for it on Google. A recorder runs in the background. Eight steps later the job is done, and the recorder has written a file.

What is in that file? And how does a model learn anything from it?

The recorder's file

Think of a flight recorder. After a flight, its file holds what the crew set out to do, every reading and every control input in order, and how the flight ended. The recorder on the computer writes the same kind of file. The whole recorded run of one task, what was asked, then every screen and action in order, then how it ended, is a trajectory.

A trajectory has three parts. First comes the instruction: the task in words, as a person would ask it, like a job ticket. Then come the steps. Last comes the outcome: whether the run finished the task, the flight's landing.

Instructions come in two sizes. The whole goal, “Search Google for the word ‘Travelers’ from the open document”, is a high-level instruction: the job ticket. The goal of one step, “Click on the Google search box”, is a low-level instruction: the sticky note for that one step. Many datasets store both, so a model can learn from either.

One step, opened up

Each step holds three things. The observation is what the agent saw: the screenshot, sometimes with a text list of what is on the screen. It is what the assistant sees when they look up. The thought is a sentence the agent writes before it acts, saying what it sees and why it acts. It is thinking aloud. The action is what it did.

Two more words, and you can step through a real trajectory. A screen is a grid of dots called pixels, like the tiles of a mosaic, and a click lands on one of them. And a model never learns from a whole trajectory at once. It learns from one moment at a time: the task and everything so far, then the next action. One input and the answer we want the model to give for it is a training example, like one flashcard.

Notice what changed and what did not as you stepped. The instruction stayed the same all the way through. The observation, the thought and the action changed at every step. And step 8 is an action too: the agent must learn to say that it has finished.

Actions are code

Datasets write each action as a line of code that a program could run to repeat it. The desktop set from Chapter 0 uses PyAutoGUI, a Python library that moves the mouse and presses keys from code. Its whole list of actions has twelve entries:

Clicks dominate. They are 60.03% of all actions on Windows, 57.22% on macOS and 63.62% on Ubuntu: about six in ten on every system. Typing, keys and scrolls share the rest.

The phone set speaks a different language. Its list has 8 actions: click, long_press, input_text, scroll, navigate_home, navigate_back, open_app and wait. Every source speaks its own action language, so translating them all into one is part of building a dataset.

Where a click lands

A click's place is two numbers, how far across and how far down: its coordinates, like a map reference. Screens differ in how many pixels they have. A screen 1,280 pixels wide and 720 tall is called 720p. One 1,920 by 1,080 is called 1080p, and one 3,840 by 2,160 is called 4K.

Look at the code for step 5: pyautogui.click(x=0.157, y=0.1229). Those are not pixels. They are fractions of the screen's width and height: 0.157 of the way across, 0.1229 of the way down. Multiply by the screen's size to get the pixel.

Why fractions? The desktop set was recorded on screens from 720p to 4K. A click stored as pixel (301, 133) means the search box on a 1080p screen, and a spot near the top-left corner on a 4K one. A click stored as fractions keeps one meaning on every screen.

x, y

The click as fractions of the screen's width and height, as the record stores it.

W, H

The screen's width and height in pixels.

x_px, y_px

Where the click lands, in pixels from the top-left corner.

0.157 × 1920 = 301  ·  0.1229 × 1080 = 133

So the same pair (0.157, 0.1229) is pixel (301, 133) on a 1,920 by 1,080 screen, and pixel (603, 265) on a 3,840 by 2,160 one: the same spot, the search box, on both. Switch the screen size in the device above and watch the pixel change while the spot stays put.

The record on disk

A plain-text way to write a record with named fields between curly brackets is JSON, like a form with labelled boxes. A file with one JSON record on each line is JSONL. Most agent datasets ship as JSONL, one trajectory per line. Here is one trajectory in the desktop set's format, spread over several lines so you can read it:

json{
  "instruction": "Search Google for the word 'Travelers' from the open document",
  "task_completed": true,
  "alignment_score": 9,
  "efficiency_score": 8,
  "task_difficulty": 3,
  "traj": [
    {"index": 0, "image": "step_0.png",
     "value": {"observation": "A document is open behind a browser window.",
               "thought": "The document is behind the browser.",
               "action": "Click the document window to bring it to the front",
               "code": "pyautogui.click(x=0.45, y=0.12)",
               "last_step_correct": true, "last_step_redundant": false, "reflection": ""}},
    "… seven more steps …"
  ]
}

The scores here are made up; their names are the dataset's.

Read it from the top. instruction is the high-level task. task_completed is the outcome. alignment_score, efficiency_score and task_difficulty are grades the builders added after recording. traj is the list of steps.

Inside each step, image names the screenshot file, and observation describes it in words. thought is the thought. action is the low-level instruction, in words. code is the same action as PyAutoGUI. The two flags, last_step_correct and last_step_redundant, and the reflection are Chapter 2's subject.

What the agent can read besides pixels

Picture the directory board in a building's lobby: a list of every office, with its name and its floor. Computers keep a board like that for the screen. It is a text list of everything on the screen that a program can read: every button, link and box, with its name and its kind. Computers keep it for screen readers, the programs that read the screen aloud for blind people. It is the accessibility tree.

A web page also has its own code, which says what is on the page: its HTML, the building's blueprint. A blueprint says everything, including much that a visitor never needs. The lobby board says only what you can visit. Here are a few lines of an accessibility tree as a web agent sees it, then two actions:

accessibility tree[119] link 'Magento Admin Panel'
[122] menubar '', orientation='horizontal'
click('119')
fill('61', 'Boston')

Read the first line: item number 119 is a link named ‘Magento Admin Panel’. An action names the number: click('119'), or fill('61', 'Boston') to type Boston into item 61. The agent never needs to know where item 119 sits in pixels.

Models read text and pictures in small pieces called tokens, like the words in a telegram, each one paid for. How many tokens an input has sets how much it costs to run, and how much the model must hold in mind. For one open model, a 720p screenshot costs about 1,200 tokens, and the same page's HTML about 4,000: 3.3 times more. That is why many pipelines show the screenshot plus a short tree, not the whole page's code.

From one trajectory to many examples

The model that learns from these records is rarely trained from nothing. It is a model that has already learned a lot, trained further on a smaller set of examples for one job. That is fine-tuning: an experienced cook learning one restaurant's menu.

Now cut the trajectory at every step. Each cut is one training example. Its input is the task, the earlier actions written as text, and the current screenshot. Its answer is this step's thought and action. Here is a real training example from the desktop set, cut at step 5 and shortened:

training examplesystem:    You are a GUI agent. You are given a task and a screenshot of the screen.
           You need to perform a series of pyautogui actions …
history:   # Step 3: ## Action: Press Ctrl + C to copy the selected word 'Travelers'
user:      <screenshot>  # Task Instruction: …  Please generate the next move according to
           the screenshot, task instruction and previous steps
target:    # Step 5: ## Action: Click on the Google search box …
           ## Code: pyautogui.click(x=0.157, y=0.1229)

Read it from the top. The system line tells the model its job: “You are a GUI agent”, an agent that works a graphical screen. The history line carries an earlier action as text: step 3 copied the word. The user line brings the new screenshot and the task. The target is the answer: step 5's action, in words and as code.

Training needs a number that says how far the model's answer was from the one we wanted, like the red ink on a marked test. That number is the loss, and training pushes it down. Here the loss is counted only on the target. The model is marked on what it should write, never on what it was given.

Counting examples

One trajectory with n steps gives n training examples, because each step is one “given all this, do what next?” question. So a dataset's examples are its trajectories times their average number of steps.

The desktop set's 22,625 tasks average 18.6 steps: about 420,825 training examples. The explorer of Chapter 5 kept 94,000 runs of 7.7 steps: about 724,000 steps, close to the 720,000 screenshots it saved. The cutting itself is a short loop:

pythondef unroll(record):
    """One trajectory of n steps -> n training examples (input, answer)."""
    examples, history = [], []
    for step in record["traj"]:
        v = step["value"]
        examples.append({
            "input": {"task": record["instruction"], "previous": list(history), "image": step["image"]},
            "target": f"## Thought: {v['thought']}\n## Action: {v['action']}\n## Code: {v['code']}",
            "train_on_this": v["last_step_correct"] and not v["last_step_redundant"],
        })
        history.append(v["action"])          # this step becomes history for the next one
    return examples

The last field, train_on_this, is Chapter 2's subject: a step the person got wrong stays in the history but is never a target.

Read the loop. For every step, it builds one example from the task, the history so far and this step's screenshot. It sets the target to this step's thought, action and code. Then it adds this step's action to the history, so the next example sees it. Run on a 3-step record, it prints 3 examples.

worked exampleOne trajectory, eight steps (illustrative task; steps 3 and 5 as in the desktop set's own example)
  training examples           = 8, one per step
  example 5:  input  = the task + actions 1 to 4 as text + screenshot 5
              answer = a thought + "Click on the Google search box" + pyautogui.click(x=0.157, y=0.1229)
Fractions to pixels
  1920 × 1080:  x = 0.157 × 1920 = 301.4 → 301      y = 0.1229 × 1080 = 132.7 → 133
  3840 × 2160:  x = 0.157 × 3840 = 602.9 → 603      y = 0.1229 × 2160 = 265.5 → 265
Whole datasets
  desktop set:  22,625 tasks × 18.6 steps = 420,825 training examples
  explorer:     94,000 runs × 7.7 steps   = 723,800 steps; it saved 720,000 screenshots
What the model reads
  one 720p screenshot ≈ 1,200 tokens; the same page as HTML ≈ 4,000: 3.3 times more
  if a model kept all 18.6 screenshots of a task at 720p: 18.6 × 1,200 = 22,320 tokens

0.1229 × 2160 is 265.46, which rounds to 265.

Clicks in the corner

What the team sees. A team merges two datasets into one training file. After training, many of the model's clicks land in the top-left corner of the screen.

What you measure. Plot the x value of every click in the merged file. There are two piles: one between 0 and 1, one in the hundreds. One dataset stored fractions, the other pixels. The model learned both, and when it writes a small number the harness reads it as a pixel: a click at x = 0.157 lands in the screen's first column.

What you change. Convert every click to fractions when a record is loaded, and refuse to write any record whose x or y is outside 0 to 1. (This story is illustrative: it shows the kind of bug the fraction rule prevents.)

A part-time annotator opens her laptop on a Tuesday evening. A small program records her screen, her mouse and her keys while she renames files, edits a slide and books a table. She finishes a task about every six minutes.

Somewhere, 633 other people are doing the same on their own computers. Six months later they have recorded 22,625 tasks. This chapter follows one recording from her laptop to a training file, and adds up the bill.

Three ways people have recorded

A person paid to record or label data is an annotator. Some datasets hire annotators and train them for weeks. Others use crowd workers: people who take small paid jobs from a website, one at a time, with no training beyond the job's instructions.

Three datasets show the range. You have met two of them already. The third is the web crowd set: a 2023 dataset of web tasks recorded by paid crowd workers.

The phone set also has a person write a short instruction for every single step, 3 to 14 words long, beside the whole task's 8 to 34.

Read down the steps column. The desktop tasks are more than twice as long as the web ones. The desktop set wanted it that way, for reasons this chapter comes back to.

What the recorder captures

The desktop set's recorder runs in the background on the annotator's own computer. It saves three streams: a video of the screen, every mouse and keyboard signal, and the accessibility tree from Chapter 1.

The middle stream is very fine-grained. It logs every tiny signal: each mouse move, each button going down or up, each wheel click, each key. Each of these is a raw event, like one frame of a film. A typical task produces thousands of them, far too many to learn from. Nobody wants a model that learns to wiggle the mouse 120 times a second.

Squashing raw events into actions

So the raw events must become the few actions of Chapter 1: click, scroll, write, press, hotkey. One more word first. Keys pressed together, like Ctrl and C, are a hotkey. Then the desktop set's four rules:

1. Keep a mouse move only as the start or end of a click or a drag.
2. Merge wheel clicks in one direction into one scroll with a count.
3. Merge key presses in a row into typed text.
4. Merge a modifier key held with another key into a hotkey.

Squashing the raw events into a few actions is action reduction, like cutting a film down to the shots that matter. Watch it on a short recording of a person renaming a file.

The mouse rule does almost all the work: moving the mouse was 324 of the 363 raw events. Here is the whole reducer as a sketch in Python. It reads raw events in time order and writes PyAutoGUI actions:

pythondef reduce_events(events):
    """events: (kind, value) in time order -> PyAutoGUI-style actions."""
    out, text, wheel, pos, mods = [], "", 0, None, []
    def flush():
        nonlocal text, wheel
        if text:  out.append(f"pyautogui.write('{text}')"); text = ""
        if wheel: out.append(f"pyautogui.scroll({wheel})"); wheel = 0
    for kind, v in events:
        if kind == "move":   pos = v                      # rule 1: remember only where the mouse is
        elif kind == "down": flush()
        elif kind == "up":   out.append(f"pyautogui.click(x={pos[0]:.3f}, y={pos[1]:.3f})")
        elif kind == "wheel":
            if text: flush()
            wheel += v                                    # rule 2: one scroll with a count
        elif kind == "key":
            if wheel: flush()
            if v in ("ctrl", "alt", "shift", "cmd"): mods.append(v)
            elif mods: flush(); out.append("pyautogui.hotkey(" + ", ".join(repr(m) for m in mods + [v]) + ")"); mods = []   # rule 4
            elif len(v) == 1: text += v                   # rule 3: letters become typed text
            else: flush(); out.append(f"pyautogui.press('{v}')")
    flush()
    return out

Keys arrive here as one event per press. On the device's recording it writes the same 8 actions: click, click, scroll(-7), click, press('f2'), hotkey('ctrl', 'a'), write('final.txt'), press('enter').

The output is the action language of Chapter 1: click, scroll, write, press, hotkey. A clean trajectory of 8 steps has come out of 5.5 seconds of signals.

Which screenshot goes with the click?

A recording is a video, so for every step someone must pick the one frame that becomes its screenshot. The one screenshot saved for a step is its keyframe, the photo pinned to that step. The obvious choice is the moment of the click. It is the wrong one.

By the moment of the click, the cursor already sits on the target. Picture a quiz where the answer is pencilled in the margin. A model can learn to click wherever the cursor is, and score well, without ever reading the screen. A picture that already shows the answer is a leak from the future.

The desktop set's rule fixes this. For a click, step back to where the mouse began to move, then back to the last frame where the screen changed. That frame shows the screen as the person saw it before deciding. Try both moments:

As a sketch in code, the rule walks backwards through the frames:

pythondef keyframe_for_click(frames, t_click, mouse_start):
    """frames: list of (t, image). Take the frame before the mouse began to move."""
    i = max(k for k, (t, _) in enumerate(frames) if t <= mouse_start)   # back to the move's start
    while i > 0 and same_picture(frames[i - 1][1], frames[i][1]):
        i -= 1                                                           # back over identical frames
    return frames[i]

same_picture compares two frames; any pixel difference test works for a sketch.

The bill

Now add up what the desktop set cost. Its annotators recorded about ten tasks an hour, so each task took six minutes of a person's time. 22,625 tasks at 10 an hour is 2,262.5 hours of recording.

The builders paid about $20,000 for all of it. Divide by the hours: about $8.84 an hour. Divide by the tasks: about $0.88 a task. Then a model wrote the thoughts and notes for every step afterwards, at $0.60 a task. Together that makes about $1.48. The builders' own total, about $32,000, works out to about $1.41 a task, a little less than the sum, so say “about”.

c

The cost of one recorded task.

P

The pay for all the recording: about $20,000 for the desktop set.

N

The number of tasks recorded: 22,625.

r

The cost of having a model write the reasoning for one task: $0.60.

$20,000 ÷ 22,625 + $0.60 = $0.88 + $0.60 = $1.48 a task (the builders' total comes to about $1.41)

The web crowd set paid $0.80 for each approved task and $0.05 for each task idea, at an estimated $10.10 an hour. Both figures are pay to workers. Neither includes the builders' own time, the tools, or the checking.

worked exampleThe desktop set's bill
  annotator hours  = 22,625 tasks ÷ 10 tasks an hour   = 2,262.5 hours
  pay per hour     = $20,000 ÷ 2,262.5                 = $8.84
  pay per task     = $20,000 ÷ 22,625                  = $0.88
  + reasoning      = $0.60 a task                      → $1.48 a task
  the builders' total, about $32,000 ÷ 22,625          = $1.41 a task (say "about")
  per annotator    = 22,625 ÷ 634                      = 35.7 tasks, about 3.6 hours each
The web crowd set's pay
  $0.80 per approved task + $0.05 per task idea        = $0.85
Checking people's work
  thrown out:  61 ÷ 2,411 = 2.53%      descriptions rewritten: 390 ÷ 2,350 = 16.6%
  desktop uploads kept: 22,625 ÷ 24,095 = 93.9%
The toy recording in the device (illustrative)
  363 raw events → mouse rule: 36 → + scroll rule: 30 → + key rule: 11 → + hotkey rule: 8 actions

Even people's recordings need checking

People make mistakes too. The web crowd set's builders replayed every demonstration by hand. They threw out 61 of 2,411 (2.5%), rewrote the task description in 390 of the rest (16.6%), and cut extra steps from 187.

The desktop set set rules up front. Tasks had to run more than 15 steps, and tasks under 5 steps were rejected. Every task was graded by hand as rejected, ok, good or excellent. In the end 22,625 of the 24,095 uploaded tasks survived: 93.9%.

Mistakes kept on purpose

The desktop set did something surprising: it kept its annotators' mistakes. A model reads each step and marks it correct, wrong or unneeded. Then it writes a short note saying why. A short note written after a step, saying whether it was right and why, is a reflection, like a coach's note on the replay.

A wrong or unneeded step stays in the history but is never a training target. Its reflection stays too, as a lesson in noticing and recovering. Those are the two flags and the note you saw in Chapter 1's record: last_step_correct, last_step_redundant and reflection.

This is the cure for behaviour cloning's drift from Chapter 0. A model that only ever saw perfect runs has never seen what going off the path looks like. A model that saw flagged mistakes and their notes has seen how to get back. The Imitation Learning lesson explains why this drift compounds over a long task.

Actions alone are not enough

What happens if you keep only the screens and the actions, and drop the thoughts and reflections? Trained on the desktop set's screens and actions alone, a model finished just 4.4% of a standard test of desktop tasks. Adding reasoning that reflects on each step lifted a smaller mix from 11.5 to 15.3. Chapter 8 comes back to this: what a dataset carries can matter as much as how big it is.

What people are bad at

How different the examples are from one another is their diversity: a menu with many dishes, not one dish many times. People ask for many different things, but they do them in the ways they always do. Their tasks are varied; their action paths are not.

And people repeat themselves. The phone set holds 15,283 demonstrations of only 14,548 different tasks: 735 of them are repeats. Chapter 7 shows how to find repeats, and why they matter more than they seem.

A model that clicks where the cursor is

What the team sees. A model trained on a team's own recordings scores very well on recorded test clicks. On a live computer it often clicks in the wrong place.

What you measure. Score a lazy baseline on the same test: a program that ignores the screen and clicks wherever the cursor is in the screenshot. It scores almost as well as the model. The screenshots were taken at the moment of each click, so the cursor already sat on the answer.

What you change. Take each click's screenshot from before the mouse began to move, as the desktop set does. Score the lazy baseline again: it should now fail almost every click. (The story is illustrative; the leak and the rule are the desktop set's.)

Search the web for how to add a page break in a document, and you find a page of numbered steps, often with screenshots. Somebody already wrote down the task, the steps and what success looks like.

The web holds millions of such pages. What if an agent simply followed them, on the live website, while a recorder ran?

Recipe cards for a computer

A how-to page that lists the steps for a task, often with screenshots, is a tutorial: a recipe card for a computer. It already holds what a trajectory needs most and people find hardest to invent. It has a real task, worded the way people word tasks, and a path that someone once followed to the end.

So picture a 2024 pipeline that turns how-to pages into recorded runs. Call it the tutorial replayer. It has three jobs. It must find tutorials among billions of pages, turn each one into a task an agent can attempt, and keep only the attempts that worked.

Finding them: a funnel

The search runs as a funnel. Each stage is cheaper per page than the one after it, so the cheap stages throw out most of the pages before the costly ones run:

Each stage is cheap enough to run on everything that reached it: simple rules on billions of pages, a small fast classifier on millions, a costly agent on thousands.

The third stage needs a word. A small, fast program trained to sort text into kinds, here “tutorial” or “not a tutorial”, is a classifier: a mail sorter. It is far too simple to follow a tutorial, but quick enough to read 68.8 million pages.

The first stage is the crudest, so its builders checked it. On 285 pages that people had labelled, the keyword filter caught 92.69% of the real tutorials. A filter that drops a few good pages is fine here, since the web holds millions more.

Tag and rewrite

A tutorial is written for people, not for an agent. So a small, cheap model reads each one and tags it, for example with the platform it is for. Then it rewrites the tutorial as a task: a description, what must be true first, numbered steps, and the expected result.

Tagging shows how much of the funnel is waste. Only 27.5% of the tagged tutorials turn out to be about websites, the only kind this agent can follow. The rest are for desktop programs or phones. Their tagging still costs money, and the price below counts it.

The replay

An agent follows a tutorial's steps on the live website while a recorder runs. That is a replay: cooking from the recipe card with a camera on. The agent sees each page as a screenshot plus its accessibility tree, the numbered list from Chapter 1, and not the full HTML.

Replays are not cheap in tokens. Each step costs about 8,027 tokens and a whole attempt about 86,114. Divide one by the other: about 10.7 steps per attempt. That is fewer than the 12.1 steps of the runs that pass, because failed attempts stop early.

Besides screenshots and video, each run keeps a trace: a recording a browser tool makes of every page, click and network message, which can be played back later. It is a black box you can replay. A trace lets anyone check a run long after the website has changed. Each record ends up with four parts:

1. the task's details: the site, the description, the steps and the expected result;
2. screenshots and video;
3. the replayable trace;
4. the cleaned-up trajectory of observations, thoughts and actions.

The judge

Not every replay works, so someone must decide which runs to keep. Picture a doorkeeper at the door of the shelf from the top of the page. Here the doorkeeper is a second model, the judge. It reads the task, then every thought and action in order. It returns three things: pass or fail, a step-by-step review, and the earliest step where things went wrong.

How good is it? The builders had people check 558 runs, and the judge agreed with them 84.0% of the time. That sounds reassuring. Chapter 6 shows why agreement alone does not say how clean the kept runs are.

The price of one good run

Now the money. The builders priced everything per 1,000 attempts. Replaying 1,000 attempts costs $215.36 in model fees, and judging them $3.10. Tagging costs $0.89 per 1,000 tutorials tagged.

Only 39.9% of replays pass. To get 1,000 that pass, you need 1,000 ÷ 0.399 = 2,506 attempts. So the replay and judging behind 1,000 good runs cost $218.46 ÷ 0.399 = $547.53.

Tagging is cheap, but most tagged tutorials are useless here. Only 27.5% are web tutorials, so tagging costs $0.886 ÷ 0.275 = $3.22 per 1,000 web tutorials. Together: $3.22 + $547.53 = $550.75 per 1,000 good runs, about $0.551 each. Per attempt, before any judging, the same fees come to $219.35 per 1,000, about $0.22 each.

c

The cost of 1,000 good runs.

t

The cost of tagging and rewriting 1,000 tutorials: $0.886.

w

The share of tagged tutorials that are about websites: 0.275.

p

The cost of replaying 1,000 attempts: $215.359.

j

The cost of judging 1,000 attempts: $3.104.

s

The share of attempts the judge passes: 0.399.

0.886 ÷ 0.275 + 218.463 ÷ 0.399 = 3.22 + 547.53 = $550.75 per 1,000 good runs

Carry one rule out of this: the price of a kept run is the price of an attempt divided by the share that passes. Move the share and watch the price of one good run.

The curve is steep at the low end. Halve the pass rate and you double the price of every good run. Every attempt is paid for, passed or not, and the failures ride on the backs of the runs that pass.

pythondef cost_per_1000_good(tag=0.886, web_share=0.275, replay=215.359, judge=3.104, passing=0.399):
    return tag / web_share + (replay + judge) / passing

print(cost_per_1000_good())               # 550.75 dollars per 1,000 good runs
print(cost_per_1000_good(passing=0.52))   # 423.34 with the tutorial's steps

Tutorials help

Is the tutorial doing real work, or would the goal alone do? The builders tried 400 tasks two ways. Given the tutorial's steps, the agent finished 52% of them: 208 usable runs. Given only the goal, it finished about 16%: 63 runs. That is more than three times as many usable runs from the same tasks.

The price follows. At about $0.22 an attempt, a good run with the tutorial's steps costs $0.42; with the goal alone, $1.39. The “With the steps” and “Goal only” chips in the device are exactly these two runs.

Does it teach?

A pile of runs is only worth its price if a model learns from it. Trained on 10,000 replayed runs, an open model got much better at finding elements on web screenshots. On a test of clicking the named element, its score rose from 30.7 to 67.4. People's data still counted, though: Chapter 8 shows what mixing the two did.

Lenient by design

A judge is a set of rules, and rules can be strict or lenient. The tutorial replayer's judge was told, in its own prompt, to be generous. Here are three of its rules, quoted as written:

The grammar is the prompt's own; the leniency is the point.

Read them plainly. More than 8 correct actions counts as success, even if the task did not finish. A forgotten final save still passes. And finishing one of two subtasks is still a success.

Why be lenient? Because a strict judge throws away runs that were nearly right, and those cost money too. But a lenient judge keeps more good runs and lets more half-done runs in. Chapter 6 counts that trade.

worked exampleThe tutorial replayer's bill, per 1,000 good runs
  tag and rewrite:   $0.886 per 1,000 tagged ÷ 0.275 web share   = $3.22
  replay:            $215.359 per 1,000 attempts
  judge:             $3.104 per 1,000 attempts
  replay + judge:    $218.463 ÷ 0.399 pass rate                   = $547.53
  total:             $3.22 + $547.53                              = $550.75 → $0.551 a good run
  per attempt:       $0.886 + $215.359 + $3.104                   = $219.35 per 1,000 → about $0.22
Tokens
  86,114 tokens a task ÷ 8,027 a step = 10.7 steps per attempt (passed runs average 12.1)
  1,000 tasks × 86,114 tokens = 86.1 million tokens for about $215 → about $2.50 a million
With and without the tutorial's steps, 400 tasks
  with the steps:  208 ÷ 400 = 52%        goal only:  63 ÷ 400 = 15.75% (printed as 15.78%)
  gap: 36.2 points; 208 ÷ 63 = 3.3 times as many usable runs
  at $0.219 an attempt:  0.219 × 400 ÷ 208 = $0.42 a good run;  0.219 × 400 ÷ 63 = $1.39
The funnel
  20.8 billion → 68.8 million (0.33%) → 18.8 million (27.3%) → 23,430 replayed → 10,398 kept (44.4%)
  (the funnel's 44.4% and the cost formula's 39.9% are both the builders'; they do not reconcile them)

The booking that could not happen

What you see. The replay of a hotel-booking tutorial fails again and again at the same place: the date picker.

What you measure. Read the judge's report: it names the earliest step that went wrong, picking a date in April. Open the live site: it offers dates from October onwards. The tutorial was written months ago, and its dates have passed.

What you change. Tutorials expire. Before replaying, check every date, price and name in the tutorial against today's site. Rewrite dates as “a date next month”, and drop tutorials whose site has changed shape.

Hand a phone to a toddler. They tap whatever is bright: a menu opens, a switch flips, the screen goes dark. They had no task in mind.

Yet afterwards you could describe exactly what they did: they turned on dark mode. The description is true, because you wrote it after watching.

What goes wrong with tasks written first

The obvious way to make a synthetic trajectory is to write a task first, then ask an agent to try it. Writing the task first, then asking an agent to try it, is task-first synthesis: you order a dish, then someone cooks it.

It fails in two ways. Often the task is impossible on this app: it asks for a setting the app does not have. Or the task is possible, but the agent fails. Either way you have paid for a failure. The tutorial replayer of Chapter 3 avoided this with tutorials, whose tasks are known to be possible. But tutorials exist only where someone wrote one.

Explore first

So turn the order around. Tapping around an app with no task in mind, to see what each button does, is exploration: the toddler with the phone. Picture a 2024 pipeline that taps around apps first and writes the task afterwards. Call it the explore-then-name pipeline.

It walks through the screen's buttons, boxes and lists with simple rules, choosing among click, type and scroll. It asks a model for help only to fill a text box, since rules cannot invent sensible text. Each step is saved as a triple: the screen before, the action, the screen after. It is before and after photos with the move in between.

Name each step

Next, a model writes a one-step instruction for each triple. It sees the action, the before and after screens, and a box drawn around the element that was touched. The real pipeline draws that box in red; this page draws it teal, its colour for actions.

The pipeline's own example shows the idea. If a click opens a dropdown menu, the instruction might be “click the dropdown to display options”. The instruction is true by construction, because the after screen shows the options.

Name the whole walk

Finally, from the run's one-step instructions, the model writes a whole task that they add up to. “Click the dropdown to display options” might become “configure application settings”. Writing the task after the run, from what the run did, is reverse task synthesis: naming a photo after you take it. Try it on a toy phone:

Why it works

The task always describes something that happened, so it is never impossible on this app. The risk does not vanish; it moves. The question is no longer “could this be done?” but “is the description vague or wrong?”, and that can be checked.

Do these tasks read like people's? On phones, its tasks averaged 18.01 words against 18.71 for people's, and its runs 5.60 steps against 5.31. And its action paths were more varied than people's, the weakness Chapter 2 found in human data.

Score, do not discard

Some walks are better than others. A walk that wanders on after finishing its task, or loops, teaches less. So a second model reads each run and gives it a score from 1 to 5. A model that reads a run and gives it a score is a reward model, like a judge holding up score cards. The pipeline's rubric:

1. done perfectly;
2. mostly done;
3. partly done, or ends in a loop or an error;
4. goes wrong early;
5. fails completely, with no useful actions.

It reads the whole task, every action, and the last three screenshots. Then comes the surprise: the scores do not decide who stays. How often training picks an example, compared with the others, is its sampling weight, like how many tickets each name has in a hat. Each run's chance of being drawn for training is its score divided by the total of all scores.

g_i

Run number i.

R_i

Its score from the reward model, 1 to 5.

Σ_k R_k

The total of all the runs' scores.

P(g_i)

The chance that training draws run i next.

scores 5, 4, 4, 3, 2, 1 → 0.263, 0.211, 0.211, 0.158, 0.105, 0.053

Work it through. Six runs get scores 5, 4, 4, 3, 2 and 1 (made up). Their total is 19. Each run's chance is its score over 19: 5/19 = 0.263, 4/19 = 0.211 twice, 3/19 = 0.158, 2/19 = 0.105 and 1/19 = 0.053. The chances add up to 1, as the chances of one draw must.

A score-5 run is drawn five times as often as a score-1 run, and no run is ever thrown away. Why keep the weak ones? In the builders' words, “even incomplete trajectories often contain a valuable exploration” of the screen. A walk that went wrong at step 5 still showed, at steps 1 to 4, what each button does.

Count the steps in play. The six runs have 6, 7, 5, 6, 4 and 3 steps (made up): 31 in all. Keeping only the score-5 run keeps 6 of 31 steps, 19%. Drawing by score keeps all 31 in play, and still favours the good runs.

Here is the whole pipeline as a sketch. Each model. line is one call to a model; everything else is rules:

pythondef explore_then_name(app, model, steps=6):
    triples, screen = [], app.home()
    for _ in range(steps):
        action = app.random_action(screen)        # click, type or scroll, chosen by rules
        after = app.do(action)
        triples.append((screen, action, after))   # before, action, after
        screen = after
    low = [model.describe_step(b, a, c) for b, a, c in triples]   # one instruction per step
    task = model.name_task(low)                   # the whole task the steps add up to
    score = model.score(task, triples)            # 1 to 5
    return task, low, triples, score

The real formats are short. The first line below is the reward model's answer. The second is a phone action as the pipeline writes it. The third line lists its web actions:

formatsReason: <why the run did or did not do the task>  Score: 4
{"action_type": "type", "text": <text_input>, "x": <x_coordinate>, "y": <y_coordinate>}
click [id]     type [id] [content] [press_enter_after=0|1]     stop [answer]

And drawing by score takes two lines of numpy. Run 19,000 draws and the shares land close to the chances:

pythonimport numpy as np
scores = np.array([5, 4, 4, 3, 2, 1])
chances = scores / scores.sum()                       # 0.263 0.211 0.211 0.158 0.105 0.053
picks = np.random.default_rng(0).choice(len(scores), size=19000, p=chances)
print(np.bincount(picks) / 19000)                     # 0.258 0.213 0.213 0.161 0.102 0.053

Does the scorer agree with people?

A reward model is only useful if its scores track what people would say. On 100 runs that people also scored, the reward model's scores rose and fell with theirs: a rank agreement of 0.813 on phone runs and 0.798 on web runs, where 1 is perfect agreement. An open model in its place kept 0.788 on web runs and 0.729 on phone runs, on 200.

Results

A fixed set of test tasks that everyone scores on, so results can be compared, is a benchmark: a standard exam. On a standard test of phone tasks, an open model trained on 1,000 explore-then-name runs finished 17.41% of tasks. Trained on 1,500 task-first runs with extra self-written tasks, it finished 9.82%. That is 7.59 points less: the new data did 1.77 times better.

Another open model went from 5.36% to 16.96%, 3.16 times. For scale, a large commercial model with no training at all scored 23.70%. And more runs helped, then flattened out: the gains slowed as the data grew.

A fairer test

Was the test new to the model? The builders also tested on the phone set from Chapter 2. Only 20 of its 833 apps had been touched while exploring: 2.4%. So most of that test was new to the model.

One more comparison tells the story. Tasks written by people, then carried out by a model, made worse training data than the explore-then-name tasks. And the explore-then-name runs kept over 80% of the value of 1,000 people's demonstrations.

worked exampleExplore first, name after (the explore-then-name pipeline's own example)
  triple:      before = a settings screen,  action = CLICK the dropdown,  after = the options showing
  one step:    "click the dropdown to display options"
  whole task:  "configure application settings"
Drawing by score, six runs (illustrative scores and lengths)
  scores 5, 4, 4, 3, 2, 1                total 19
  chances 0.263, 0.211, 0.211, 0.158, 0.105, 0.053
  steps 6, 7, 5, 6, 4, 3 = 31            only score 5: 1 run, 6 steps (19%)    by score: all 31
On a standard test of phone tasks
  open 7B model: 9.82% (1,500 task-first runs) → 17.41% (1,000 explore-then-name runs)
                 +7.59 points; 17.41 ÷ 9.82 = 1.77 times
  another open model: 5.36% → 16.96%, 3.16 times
  a large commercial model with no training: 23.70%
How new the phone set's test was
  apps touched while exploring: 20 of 833 = 2.4%

Keeping only the perfect runs

What the team does. A team replaces the 1 to 5 weighting with a yes-or-no check that keeps only complete runs. Fewer runs, all complete: it feels cleaner.

What you measure. Test two ways: whole tasks given as a goal, and single steps given as instructions.

What you find. The builders ran exactly this test. Keeping only complete runs helped slightly on whole tasks, and cost ground on single steps.

What you change. Keep the incomplete runs, weighted by their scores. Their early steps still show what each button does, which is what single-step skill is made of.

A program lands on the home page of an online shop it has never seen. Nobody gave it a task. It reads the page and sets itself one: find a men's watch under $50. It clicks Men.

The next page shows categories it did not expect, so it adjusts its plan a little and clicks again. Five clicks later it writes down what it actually did.

Four roles

Chapter 4 explored with rules and named the task at the end. This chapter's method lets a model steer the exploration itself, toward things a person might want. Picture a 2025 pipeline in which a model sets itself tasks on live websites. Call it the explorer.

The explorer works like a small newsroom with four jobs: a writer, an editor, a caption writer and a fact checker. All four are the same kind of model, each given different instructions:

• a proposer reads the home page and suggests a task and a first action;
• a refiner looks at each new page, picks the next action, and rewrites the task to fit what the page shows;
• a summarizer reads the whole run, every action and screenshot, and writes the final task;
• a verifier reads that task, the actions, the screenshots and the last page as text, and says success or failure.

Watch one example being made, role by role, on a toy shop:

Why rewrite as you go

A task proposed on the home page is a guess about a site the model has not seen. Each new page shows what is really there, and the refiner moves the task toward something the site can do. In the device, “find a men's watch under $50” became “open the page of the cheapest men's watch under $50”, because the page offered a price filter and a sort.

Rewriting also adds variety. Across the whole run, 53,000 different first proposals became 94,000 final tasks, 81,000 of them different. Exploring produced more variety than it started with: two runs that began with the same guess often ended in different places.

What the agent sees and writes

The explorer's agent reads each page as a numbered list of elements, like Chapter 1's accessibility tree, and answers with an action in JSON. Here is one entry of the page list, one action, and the shape of the verifier's answer:

formats[24] [INPUT, TYPE=TEXT] [Search Amazon]
{"action": "type", "action_natural_language": "Type the search words", "idx": 24, "value": "men's watch"}
Thoughts: …  Status: 'success'

Read the first line: item 24 is a text box named Search Amazon. The action in the second line names the item by number, says in words what it does, and carries the text to type. The words matter. They become the low-level instruction for that step, for free. On average a page offered 46.3 elements to choose from.

When to stop

A small puzzle a website shows to stop robots is a CAPTCHA. The agent is told to stop when a site shows a CAPTCHA, asks it to log in, or asks for payment. There is nothing to learn behind those doors, and some of them must not be opened by a program.

The price

Every proposing or refining step calls the model once, at about $0.0128 in model fees, covering the text and the screenshot. An average run has 7.7 such steps: 7.7 × $0.0128 = $0.0986. Then comes one call to summarize, $0.02581, and one to verify, $0.02381. One attempt costs $0.0986 + $0.02581 + $0.02381 = $0.148.

Only some attempts pass. If 53.1% pass, each passed run carries the cost of the failed ones too: $0.148 ÷ 0.531 = $0.279, about $0.28. It is the same rule as Chapter 3: the price of a kept run is the price of an attempt divided by the share that passes.

k

The model fee for one proposing or refining step: $0.0128.

n

The number of such steps in the run: 7.7 on average.

a

The fee to summarize the run: $0.02581.

v

The fee to verify it: $0.02381.

c

The cost of one attempt.

s

The share of attempts that pass: 0.531.

c₁

The cost of one passed run.

0.0128 × 7.7 + 0.02581 + 0.02381 = $0.148  ·  ÷ 0.531 = $0.28

The toy run in the device was shorter: 5 steps. At the same prices it cost 5 × $0.0128 + $0.02581 + $0.02381 = $0.1136, which is what its meter shows at the end.

Scale

Now the whole run. The explorer made 175,000 attempts, and 94,000 passed. They covered 49,000 different web addresses, with 720,000 screenshots and 33.3 million page elements. It took 50 hours on 60 copies of the pipeline running side by side. That is 3,000 hours of work in all, about 58 attempts and 31 passed runs per copy per hour.

Compare, with care, the people of Chapter 2. A desktop annotator recorded about ten tasks an hour, of a longer kind. A copy of the explorer made 31 passed runs an hour, around the clock, and 60 copies ran at once.

What it did

Clicks were 57% of its 725,000 actions, scrolls 29% and typing 9%. Going to a new address, choosing from a list and searching the web made up the rest. Scrolls are the suspect ones: a scroll changes what you see, but rarely what the task needs.

So for training, the builders drew 40,000 runs and dropped every run with more than two scrolls. That kept about 30,000, three quarters. Chapter 7 comes back to scrolls: they also make two runs of the same task look different.

Where it breaks

Two more words name the explorer's commonest mistakes. Linking words to a place on the screen, “the search box” to the box's pixels, is grounding: pointing at what you mean. Writing something that sounds right but is not in the evidence is a hallucination, like a witness who adds details they never saw.

The builders sorted the failures they found into five kinds:

1. a click that lands on the wrong thing while refining (a grounding error);
2. a website that does not respond;
3. a summary that adds goals the run never pursued (a hallucination);
4. technical blocks, such as logins and CAPTCHAs;
5. a task left unfinished at the step limit.

The first and the third were the most common. Notice where the third one lives: in the summarizer, the role that writes the very words that go into the training data. Switch the device to “Adds a detail” to see why that hurts.

Here is the explorer's loop as a sketch. It is not a library: each model. call is one call to a model, and the stop rules come first in every step:

pythondef explore(site, model, max_steps=10):
    page = site.open_home()
    task, action = model.propose(page)                     # a task and a first action
    steps = []
    for _ in range(max_steps):
        if page.has_captcha or page.asks_login or page.asks_payment:
            break                                          # the stop rules
        page = site.do(action)
        steps.append((page.screenshot(), action))
        action, task = model.refine(task, steps, page)     # next action, task rewritten to fit
        if action is None:
            break
    final_task = model.summarize(steps)                    # written last, from the whole run
    passed = model.verify(final_task, steps, page.as_text())
    return final_task, steps, passed

Does it teach?

On a live web test, an open model trained on the explorer's data finished 19.3% of tasks. Before training it finished 14.5%, and a large commercial model finished 25.3%. A smaller open model went from 2.4% to 18.1%. The same idea, a model reasoning about what data to make and then criticising it, drives synthetic text data in Simula.

worked exampleOne attempt at the explorer's average prices
  propose and refine:  $0.0128 a step × 7.7 steps     = $0.0986
  summarize:           $0.02581
  verify:              $0.02381
  one attempt:         $0.0986 + $0.02581 + $0.02381  = $0.148
  one passed run:      $0.148 ÷ 0.531                 = $0.279 → $0.28
  (its final table shows 94,000 ÷ 175,000 = 53.7% passing; the cost uses its estimate, 53.1%)
The whole run
  175,000 attempts, 94,000 passed, 49,000 web addresses, 720,000 screenshots
  50 hours × 60 copies = 3,000 copy-hours → 58 attempts and 31 passed runs per copy-hour
  720,000 screenshots ÷ 94,000 runs = 7.66 per run (runs average 7.7 steps)
Variety
  53,000 different proposals → 94,000 final tasks, 81,000 different (13,000 repeats, 13.8%)
Actions
  725,000 in all: click 415,000 (57.2%), scroll 213,000 (29.4%), type 62,000 (8.6%),
  go to an address 26,000 (3.6%), choose from a list 5,000 (0.7%), search the web 4,000 (0.6%)
The device's toy run (illustrative 5 steps, at the explorer's prices)
  5 × $0.0128 + $0.02581 + $0.02381 = $0.1136

The task that said more than the run did

What you see. A model trained on self-set web tasks starts adding conditions nobody asked for, and claims to have checked things it never looked at.

What you measure. Read 50 kept pairs side by side: the final task and the actions. Mark every condition in the task (a price, a colour, free shipping) and look for the step that checked it. Some conditions have no step at all. The summarizer invented them, and the verifier passed the run anyway. (The count of 50 is illustrative; the kind of error is the explorer's own finding.)

What you change. Make the summarizer write only what a step shows. Have the verifier check each condition in the task against a step and the last page. Drop runs where a condition has no step behind it.

You have a hundred thousand recorded runs and nobody to watch them. So a second model watches each run and says pass or fail.

People check 100 of its verdicts. It agreed with them on 81. Is your data 81% clean?

Judges so far

This lesson has already met four judges: the people who replayed the web crowd set by hand (Chapter 2), the tutorial replayer's judge (Chapter 3), the reward model's 1 to 5 scores (Chapter 4), and the explorer's verifier (Chapter 5). Papers also call a judge a verifier or an evaluator; this page keeps saying judge.

Every pass-or-fail judge has the same job: stand at the shelf's door, let runs in or turn them away. This chapter counts how often it gets that wrong, and in which direction.

Four boxes

A verdict can land in one of four ways. Picture a sorting tray with four boxes. A run really did its task, or it did not; the judge passed it, or it did not. Two by two makes four boxes. The four ways a pass-or-fail verdict can land, in a two-by-two table, are a confusion matrix.

Two of the boxes are mistakes, and they are different mistakes. A failure the judge passed is a false pass. A success the judge failed is a false fail. Statisticians say false positive and false negative.

The explorer's builders had people check 100 of its verifier's verdicts. Here is their count, the only full audit of a trajectory judge in this lesson:

Every number this chapter needs comes out of those four boxes. Before any formula, look at them as two piles: what the judge kept, and what it threw away. The device starts on exactly this audit.

Accuracy counts the wrong thing

The obvious score is how often the judge agrees with people. The share of verdicts that agree with people is accuracy. Here it is (39 + 42) ÷ 100 = 81%.

But look at where the 81 right verdicts are. 42 of them are rejections, and rejected runs never reach your data. Accuracy counts the runs you threw away. For a judge at the door of a training set, that is the wrong thing to count.

Precision and recall

Two other numbers ask the questions that matter for data. The first asks how clean the shelf is: of everything the judge kept, the share that really did the task. That is precision. The second asks how much good stuff made it in: of all the runs that really did the task, the share the judge kept. That is recall.

Read them off the boxes. The judge kept 39 + 14 = 53 runs. Precision is 39 ÷ 53 = 73.6%, so 26.4% of the kept runs are failures. There were 39 + 5 = 44 real successes. Recall is 39 ÷ 44 = 88.6%, so 11.4% of the real successes were thrown away.

Scale it up. If the audit holds for the whole set, about 24,800 of the explorer's 94,000 kept runs are failures. An audit of 100 is a small sample, so read that as a rough size, not a count. Still, it is a quarter of the shelf.

How common are real successes?

There is a third number hiding in the boxes, and it changes everything. Picture a crate of apples. How many good apples were in the crate before anyone sorted them? How common real successes are before any judging is the base rate. In the audit it is 44 in 100.

Now derive precision from the judge's two habits. Take 100 attempts, of which a share s really succeed: 100s successes and 100(1 − s) failures. The judge passes a share r of real successes: r × 100s right passes. It passes a share f of real failures: f × 100(1 − s) false passes.

Precision is right passes over all passes: r·s ÷ (r·s + f·(1 − s)). The 100s cancel. With the audit's habits, r = 39 ÷ 44 = 0.886 and f = 14 ÷ 56 = 0.25. At s = 0.44 that gives 0.390 ÷ (0.390 + 0.140) = 0.736, the audit's own precision.

P

Right passes: runs that really succeeded and were kept (39 in the audit).

F

False passes: failures the judge kept (14).

M

False fails: successes the judge threw away (5).

s

The base rate: the share of attempts that really succeed.

r

The judge's recall: the share of real successes it passes (0.886).

f

The share of real failures it passes (0.25).

39 ÷ 53 = 0.736  ·  39 ÷ 44 = 0.886  ·  at s = 0.10: 0.0886 ÷ 0.3136 = 0.283

Now make the task hard: s = 0.10, a task the agent finishes one time in ten. Precision is 0.0886 ÷ (0.0886 + 0.225) = 0.283. The judge's habits did not change. The crate just held fewer good apples, so more of what it passes is bad.

Run the same habits across the range. At s = 0.80, precision is 93.4%. At 0.44, 73.6%. At 0.20, 47.0%. At 0.10, 28.3%. At 0.05, only 15.7%. One honest caveat: this assumes the judge's habits are the same on hard and easy tasks. Nobody has measured that here, so treat it as the arithmetic of a plausible case.

Rolling until you like the dice

Here is a tempting fix for hard tasks. Try each task several times, and keep the first attempt the judge passes. Trying a task several times and keeping only the attempts the judge passes is rejection sampling: rolling until you like the dice.

pythondef rejection_sample(task, agent, judge, tries=4):
    for _ in range(tries):
        run = agent.attempt(task)
        if judge(task, run):
            return run        # the first run the judge passes goes into the data
    return None               # no pass: the task adds nothing

It does get hard tasks into the data. At s = 0.10 the judge passes an attempt with chance 0.886 × 0.10 + 0.25 × 0.90 = 0.314. Trying 4 times instead of once gets the task in 78 times in 100 instead of 31. But each example it brings is still a failure 72 times in 100. More tries change how often a hard task enters the data, not what it brings.

Across a whole dataset the effect is quieter, but it points the same way. Take a made-up mix of easy and hard tasks that averages the audit's 44%. The failures among the kept runs rise from 26.4% with one try, to about 29.5% with four, and 30.7% with sixteen. Retries pull in the hard tasks, and the hard tasks bring the false passes.

Lenient and strict

A lenient judge, like the tutorial replayer's rules in Chapter 3, raises recall and lowers precision. A strict one does the reverse. Try the presets in the first device. The strict judge (made up) keeps 34 of 100 with only 3 failures, 8.8%, but it throws away 13 of the 44 real successes, 29.5%. The lenient judge (made up) keeps 77, with 34 failures among them, 44.2%, and loses just 1 success, 2.3%.

Neither is right in general. Precision protects what the model learns. Recall protects what you paid for. And notice the accuracies: the strict judge is right on 84 of 100, the audited one on 81, the lenient one on 65. Accuracy alone would never tell you which shelf is clean.

Know your judge

The only way to know a judge's precision and recall is to have people check a sample of its verdicts, as the explorer's builders did with 100 runs. Do it per kind of task, because the hard ones are where the judge fails.

Two other ways see past a bare pass or fail. Graded scores, like Chapter 4's reward model, keep weak runs at a low weight instead of a yes-or-no verdict. And a program can check the final state of the computer directly, where one exists: was the file really renamed? The environment lessons coming in this path build those checks.

pythontp, fn, fp, tn = 39, 5, 14, 42            # the explorer's audit of 100 runs
kept = tp + fp
print("precision", tp / kept)             # 0.736: how clean the kept runs are
print("recall", tp / (tp + fn))           # 0.886: how many real successes survived
print("accuracy", (tp + tn) / 100)        # 0.81: counts the rejections too
print("failures in kept", fp / kept)      # 0.264

worked exampleThe explorer's audit: 100 runs, the judge's verdict against people's
                       judge: pass   judge: fail
  really succeeded         39            5          → 44 real successes
  really failed            14           42          → 56 real failures
  accuracy    = (39 + 42) ÷ 100     = 0.81
  kept        = 39 + 14             = 53 of 100
  precision   = 39 ÷ 53             = 0.736 → 26.4% of kept runs are failures
  recall      = 39 ÷ 44             = 0.886 → 5 of 44, 11.4%, of real successes thrown away
  false-pass rate  f = 14 ÷ 56      = 0.25
Scaled to the whole set, if the audit holds
  94,000 kept × 0.264 = about 24,800 failures
Precision at other base rates (same habits, r 0.886 and f 0.25)
  s = 0.80:  0.709 ÷ 0.759 = 0.934 → 6.6% of kept are failures
  s = 0.44:  0.390 ÷ 0.530 = 0.736 → 26.4%
  s = 0.20:  0.177 ÷ 0.377 = 0.470 → 53.0%
  s = 0.10:  0.089 ÷ 0.314 = 0.283 → 71.7%
  s = 0.05:  0.044 ÷ 0.282 = 0.157 → 84.3%
Rejection sampling at s = 0.10
  pass chance per try  q = 0.886 × 0.10 + 0.25 × 0.90 = 0.314
  enters the data:  1 try 31%;  4 tries 1 − 0.6864 = 78%;  16 tries about 100%
Over a made-up mix of tasks averaging 0.44 (illustrative)
  failures among kept:  1 try 26.4%,  4 tries 29.5%,  16 tries 30.7%

More tries made the hard tasks worse

What the team does. Their agent is weak on hard tasks, so they give each hard task 16 tries and keep the first run the judge passes. Hard tasks now fill a large part of the data.

What they see. After training, the model is worse at exactly those tasks.

What you measure. Have people check 50 kept runs from hard tasks. For tasks the agent finished about one time in ten, the judge's habits predict that about 72% of what it passed were failures. The hand check finds most of them failures too. (The story is illustrative; the 72% is the audited habits worked through.)

What you change. On hard tasks, demand more: a stricter judge, two checks that must agree, or a program that checks the final state where one exists, and a hand-checked sample every time. Never let retries alone fill the hard tasks.

Open the explorer's data and read the final tasks. “Find a men's watch under $50.” A few thousand rows later, the same words again.

Of its 94,000 final tasks, 81,000 are different. 13,000 are repeats.

Repeats are everywhere

Every source makes repeats. The explorer has 13,000 repeated tasks in 94,000: 13.8%. The phone set has 15,283 demonstrations of 14,548 different tasks: 735 repeats, 4.8%. Even the tutorial replayer had to remove duplicate pages before it started, and still began from 18.8 million.

Why do repeats hurt? A repeated task is learned too well, while others are learned once. It costs money twice. And a repeat can cross into the test, which is the worst of the three, as the end of this chapter shows.

Near-copies

Exact repeats are easy to find: compare the words or the actions, character by character. The hard ones are almost the same. Picture two photocopies of one page, one with a smudge. Two runs of the same task that differ only by a harmless extra step are near-copies (papers say near-duplicates).

Two runs of the same shopping task can differ only by one scroll in the middle. Compare them as whole lists and they are different lists. To a model they teach the same thing.

Overlapping roof tiles

So compare runs by their pieces instead of as whole lists. Take every pair of actions in a row. Each pair overlaps the next by one action, like roof tiles overlap. Every pair of actions in a row is one shingle. With n actions in a row instead of two, it is an n-gram; this chapter uses pairs.

Here are two runs of the same shopping task. Run a is: Search, type “running shoes”, Filter, Under $50, Item 3, Add to cart. That is 6 actions and 5 pairs. Run b is the same, with a scroll between “Under $50” and “Item 3”: 7 actions and 6 pairs.

The scroll breaks one of a's pairs, (Under $50, Item 3), so that pair is only in a. It makes two new pairs, (Under $50, scroll) and (scroll, Item 3), so those are only in b. The other 4 pairs are in both.

Counting what two runs share

Picture two parties. Count the guests they have in common, then divide by everyone invited to either party. The shingles two runs share, divided by all the different shingles they have between them, is their Jaccard similarity. It runs from 0 (nothing shared) to 1 (identical).

A, B

The sets of shingles of two runs: every pair of actions in a row.

A ∩ B

The shingles both runs have.

A ∪ B

Every different shingle either run has.

| … |

How many there are.

shop a and shop b: 4 shared ÷ 7 in all = 0.57

Two runs count as the same task when their similarity reaches a line you choose. The line at or above which two runs count as the same task is the threshold. Here it is 0.5, a made-up value. Shop a and shop b share 4 pairs out of 7 different pairs: 0.57, so at a threshold of 0.5 they count as the same task. Try it on twelve runs from three made-up sites:

Short runs are fragile

Why does the bank pair score lower than the shop pair, when both differ by one scroll? Because the bank runs are shorter. Count it in general. A run of L actions has L − 1 pairs in a row. Insert one extra step in the middle. It breaks the one pair it lands inside, and makes two new pairs with its neighbours. The new run has L pairs.

The shared pairs are the original L − 1 minus the broken one: L − 2. All the different pairs are the original's L − 1, plus the copy's 2 new ones: L + 1. So J = (L − 2) ÷ (L + 1). That is 0.63 for 7 actions, 0.57 for 6, 0.50 for 5, 0.40 for 4, and 0.25 for 3.

A 4-action run with one extra scroll scores 0.40, below a 0.5 threshold, though it is the same task. So drop moves that change nothing the task cares about, scrolls and waits, before comparing. The explorer went further, as Chapter 5 showed: it dropped every training run with more than two scrolls.

Same site, different task

Similarity is not the same as sharing a website. Shop a (buy running shoes) and shop d (track an order) share no pair at all: J = 0. And two different sites can share a pair. “Share” then “Copy link” appears on a map site and on a news site, giving J = 1 ÷ 6 = 0.17 for those runs. Same site is not same task, and different sites are not always different.

At scale

Here are the three functions a pipeline needs, with the noise removed first:

pythonIGNORE = {"scroll:down", "scroll:up", "wait"}        # moves that change nothing the task cares about

def shingles(run, n=2):
    a = [x for x in run if x not in IGNORE]           # compare what the run did, not how it wandered
    return {tuple(a[i:i + n]) for i in range(len(a) - n + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b)                    # shared over all different

def dedup(runs, threshold=0.5):
    kept, sigs = [], []
    for r in runs:
        s = shingles(r)
        if all(jaccard(s, t) < threshold for t in sigs):
            kept.append(r); sigs.append(s)            # keep the first of each group
    return kept

Read dedup: it walks the runs in order, and keeps a run only if it is below the threshold against every run kept so far. That is exactly what the device does.

It compares each new run with every kept run, and that does not scale. Comparing every pair of a million runs is a trillion comparisons. Two tricks make it fast. MinHash shrinks each set of shingles to a short signature, and locality-sensitive hashing only compares runs whose signatures collide. They are built in Data II: Filtering and Deduplication. Comparing what runs mean, not their exact actions, uses embeddings: SemDeDup and DataComp.

Leaks

A near-copy inside the training data wastes money. A near-copy across the line between training and test does worse. Picture a student who saw the exam paper before the exam. Test tasks that slipped into the training data are contamination.

Dividing the data into a training part and a test part is a split. A split decides what “new” means. If one recording of a task is in training and its twin is in the test, the test partly measures memory. How you draw the line decides what leaks:

Split by run, and most test runs have a twin in training: in expectation, 10 × (1 − 9 ÷ 47) = 8.1 of 10. Split by task, and no test task appears in training, but the test is on familiar apps. Split by app, and the test is on apps the model has never seen: the “out of domain” test of Chapter 0.

What the sources did

The desktop set was split so that no task overlaps the standard desktop test. The explore-then-name pipeline was tested on the phone set, where only 20 of 833 apps had been touched while exploring. The phone set's own out-of-domain tests hold back apps, tasks or whole app categories that training never saw. No paper here measured how much leaking happens in practice; the procedures are the evidence.

A leak check by shared three-word phrases is short. It flags any training run whose task shares a phrase with a test task:

pythondef phrases(text, n=3):
    w = text.lower().split()
    return {tuple(w[i:i + n]) for i in range(len(w) - n + 1)}

test_phrases = set().union(*(phrases(t) for t in test_tasks))
leaks = [r for r in train if phrases(r["instruction"]) & test_phrases]

worked exampleTwo runs of the same shopping task (illustrative runs)
  a: Search · type "running shoes" · Filter · Under $50 · Item 3 · Add to cart     6 actions, 5 pairs
  b: the same, with a scroll between "Under $50" and "Item 3"                      7 actions, 6 pairs
  only in a:  (Under $50, Item 3)
  only in b:  (Under $50, scroll), (scroll, Item 3)
  shared 4, all different 7:  J = 4 ÷ 7 = 0.571 → at a threshold of 0.5, a near-copy
One extra step in a run of L actions
  J = (L − 2) ÷ (L + 1):  L = 3: 0.25   L = 4: 0.40   L = 5: 0.50   L = 6: 0.57   L = 7: 0.63
Repeats in real data
  explorer:   94,000 − 81,000 = 13,000 repeated tasks, 13.8%
  phone set:  15,283 − 14,548 = 735 repeated tasks, 4.8%
The device's toy split (illustrative: 8 apps × 3 tasks × 2 recordings = 48 runs)
  split by run (10 test runs):    most test runs have their twin in training (8.1 of 10 expected)
  split by task (5 tasks, 10 runs): no twins in training; the test runs come from apps seen in training
  split by app (2 apps, 12 runs):   no twins, no apps seen

The score that jumped

What you see. After a big new batch of web runs is added, the model's score on a standard web test jumps far more than its score on anything else.

What you measure. Compare every training task with every test task by shared three-word phrases, and every training site with the test's sites. Many test tasks are in the new batch, word for word or nearly: it was explored on the same sites the test uses.

What you change. Drop training runs that share a phrase with a test task or come from a test site. Report the score on held-out sites only. Keep the check in the pipeline, so the next batch is checked too; Chapter 9's code does both. (The story is illustrative.)

You have money for 10,000 more runs. Should you buy them? Should they come from people, or from a program?

There is only one honest way to answer: train on them, and score a test.

What a thousand more runs buy

Score a model the same way every time. Run it on tasks it never trained on, one step at a time, and count how many single steps it gets right. The share of single steps the model gets right is its step success rate. The test in this chapter is the web crowd set's own test, scored step by step, on websites from areas of life the model never trained on.

The tutorial replayer's builders trained the same model five times: on 20%, 40%, 60%, 80% and all of its kept runs, about 2,080, 4,159, 6,239, 8,318 and 10,398 of them. How a score changes as you add more of the same kind of data is data scaling. Here it is: the step success rate went 39.5, 42.5, 42.8, 44.3, 45.0.

Per doubling, not per thousand

Count what one more step of data bought. The gain per thousand runs is the change in score, divided by the change in the number of runs, times 1,000.

g

The gain per thousand runs.

Δy

The change in score between two points.

Δn

The change in the number of runs between them.

(45.0 − 39.5) ÷ (8,318 ÷ 1,000) = 0.66 points per thousand runs

From 2,080 to 4,159 runs: +3.0 points ÷ 2.079 thousand = 1.44 per thousand. From 8,318 to 10,398: +0.7 ÷ 2.080 = 0.34. The gain per thousand runs shrinks at almost every step.

Now count doublings instead of thousands. 2,080 to 4,159 is one doubling, and so is 4,159 to 8,318. Fit a straight line to the score against the logarithm of the number of runs, through all five points, and it rises about 2.3 points each time the data doubles. That rise is per doubling: the rise in score each time the amount of data doubles. So the next 10,400 runs, one more doubling, would buy about as much as the last 5,200 did, if the line held. The gain per run shrinks; the gain per doubling holds steady.

People's data still led. With people's data alone the model scored 47.7, still 2.7 points above all 10,398 of the replayed runs combined.

Mixing sources

Picture baking bread from only wheat flour, or only rye. A loaf baked from a blend of flours can turn out better than either flour alone. Training on several sources of data at once is a mixture.

In a separate experiment, the tutorial replayer's runs alone scored 42.1, people's runs alone 47.7, both together 52.6. The explorer's runs alone scored 43.0, people's alone 49.5, both together 54.3. Mixing beat people's data alone by 4.9 and 4.8 points. (The replayer's "alone" score appears twice in this chapter, 45.0 in the scaling run above and 42.1 here: they come from two different experiments in the same paper, so compare a number only against others from its own experiment.)

Mixing across systems

The same pattern shows up on desktop computers. Adding 14,000 Windows and macOS runs on top of 7,000 Ubuntu runs lifted a model from 9.8% to 18.5% on a standard test of desktop tasks. More Ubuntu runs alone, 3,000 up to 10,000, improved the average score by 72%. More Windows and macOS runs alone, 3,000 up to 14,000, improved it by 125%. For its largest model, the desktop set's builders also mixed in 8,000 runs made by a strong model, called o3, acting inside Ubuntu on its own.

What the data carries

Take one part of a recipe away and see what it was worth. Taking one part away to see what it was worth is an ablation. Take the reasoning text out of the desktop set's runs, training on nothing but the raw screenshot-and-action pairs, and the data taught almost nothing: 4.4% on a standard test of desktop tasks. Reasoning that reflects on each step, rather than a bare explanation of it, lifted another mix from 11.5 to 15.3 on the same test.

What a pass-or-fail label throws away and lets through matters just as much as how the reasoning is written. The replayer's lenient judge passed half-done runs (Chapter 3). Keeping only complete runs, with no partial credit, cost single-step skill (Chapter 4). People's own mistakes, kept in the record and flagged rather than erased, taught the model to recover from them (Chapter 2). And false passes pile up fastest on exactly the hardest tasks (Chapter 6). None of that shows up as a count of runs. It shows up in what each run is allowed to teach.

Spend by the point

Price each source per kept run, the way Chapters 2, 3 and 5 did, then ask what a thousand more of them buy. Near the end of the replayer's curve, 1,000 more runs at $0.551 each, about $551 in all, bought about 0.34 points. That is a thin return for real money.

There is a better way to spend it than buying more of the same. Instead of adding runs everywhere, find what the agent cannot do yet and make data for that gap on purpose. Targeting new data at a model's weak spot, instead of at its average, is the idea behind TRACE, later in this path.

pythonruns  = [2080, 4159, 6239, 8318, 10398]
score = [39.5, 42.5, 42.8, 44.3, 45.0]
for (n0, y0), (n1, y1) in zip(zip(runs, score), zip(runs[1:], score[1:])):
    print(f"{n0}->{n1}: {(y1 - y0) / ((n1 - n0) / 1000):.2f} points per thousand")
import numpy as np
slope = np.polyfit(np.log2(runs), score, 1)[0]
print(f"about {slope:.1f} points per doubling")     # 2.3

Read the loop: each printed line is one step of the curve, the change in score over the change in runs. The fitted slope collapses all four steps into the one number this chapter keeps coming back to, about 2.3 points every time the data doubles.

worked exampleThe tutorial replayer, more of the same data
  (the web crowd set's own test, step success, websites from new areas of life)
  share          20%      40%      60%      80%     100%
  runs          2,080    4,159    6,239    8,318   10,398    (shares of 10,398; the paper gives shares)
  score          39.5     42.5     42.8     44.3     45.0
  per thousand:     +1.44     +0.14     +0.72     +0.34         whole range: 5.5 ÷ 8.318 = 0.66
  per doubling (straight line in log of runs, fit r² 0.97): about 2.3 points
  people's data alone: 47.7, still 2.7 points ahead
Mixing, each pair within one experiment
  tutorial replayer:  alone 42.1 · people alone 47.7 · both 52.6   (+4.9 over people alone)
  explorer:           alone 43.0 · people alone 49.5 · both 54.3   (+4.8 over people alone)
Desktops
  +14,000 Windows and macOS runs on top of 7,000 Ubuntu runs:  9.8% → 18.5%
  runs without their reasoning text: 4.4%;   reflective reasoning: 11.5 → 15.3
Spending
  1,000 more replayed runs × $0.551 = $551  →  about 0.34 points, near the end of the curve

Three times the data, a lower score

What the team does. A team replaces its people-recorded web runs with a larger batch of replayed tutorials, because the replayed runs cost less per run.

What they see. The step success rate on websites from new areas of life goes down, not up.

What you measure. Train three ways on the same test: the replayed runs alone score 42.1, people's runs alone score 47.7, both together score 52.6.

What you change. Keep the human data, and add the program-made data on top of it, not in place of it. Spend the program-made budget where the human data is thin: new sites, new apps, rare tasks it never covered.

A computer-use agent learns from trajectories: an instruction, then a screenshot, a thought and an action at every step, then how it ended. Long tasks need nearly every step right, which needs far more examples than people can record, so programs make most of them: by replaying how-to pages, by exploring first and naming the task afterwards, or by letting a model set and check its own tasks. Every source costs a price per attempt divided by the share that passes; every source makes failures a judge only partly catches, and repeats that can leak into the test. Checked and mixed with people's data, the program-made runs add points. On their own, they do not replace it.

The checklist

1. Write one schema for every source: instruction; steps of observation, thought, action and code; outcome; flags for wrong and unneeded steps (Chapter 1).
2. Store clicks as fractions of the screen, and refuse any record with an x or y outside 0 to 1 (Chapter 1).
3. Squash human recordings with the four rules, and take each click's screenshot from before the mouse moved (Chapter 2).
4. Keep people's mistakes, flagged: never a training target, always in the history, with the reflection kept (Chapter 2).
5. Price every source per kept run: the price of an attempt divided by the share that passes (Chapters 3 and 5).
6. Prefer runs that start from something real: a tutorial's steps, or an exploration named afterwards, over a task invented first (Chapters 3 and 4).
7. Write the final task last, from the whole run, and check every condition in it against a step (Chapter 5).
8. Audit your judge on at least 100 runs people check; report precision and recall, never accuracy alone (Chapter 6).
9. Audit it again on hard tasks, and never let retries alone fill them (Chapter 6).
10. Remove near-copies with shingles and the Jaccard similarity, after dropping scrolls and waits; set the threshold on pairs checked by hand (Chapter 7).
11. Split by app or site, and check every training task against every test task before you report a score (Chapter 7).
12. Measure points per thousand runs and per doubling, and keep people's data in the mix (Chapter 8).

The numbers, and where each one comes from

The names behind the nicknames

The core, as code

One function carries the whole checklist, once the runs already exist: the judge, the near-copy check, the leak check, and the unrolling into training examples. Each step reports how many runs survive, so you can see exactly where your data goes.

python# trajectory_pipeline.py: from raw runs to a training file (the lesson's checklist as code)
import json

IGNORE = {"scroll down", "scroll up", "wait"}          # moves that change nothing the task cares about

def shingles(actions, n=2):
    a = [x for x in actions if x not in IGNORE]           # compare what the run did, not how it wandered
    return {tuple(a[i:i + n]) for i in range(len(a) - n + 1)} or {tuple(a)}

def jaccard(a, b):
    return len(a & b) / len(a | b)

def words(text, n=3):
    w = text.lower().split()
    return {tuple(w[i:i + n]) for i in range(len(w) - n + 1)}

def build(runs, judge, test_tasks, test_apps, threshold=0.5, min_score=None):
    """runs: dicts with instruction, app, steps. judge(run) -> True/False or a 1 to 5 score."""
    report = {"in": len(runs)}
    # 1. success filter: keep what the judge passes (or weight it by its score)
    verdicts = [judge(r) for r in runs]
    if min_score is None:
        runs = [r for r, v in zip(runs, verdicts) if v]
    else:
        runs = [dict(r, weight=v) for r, v in zip(runs, verdicts) if v >= min_score]
    report["passed judge"] = len(runs)
    # 2. near-copies: keep the first run of each group
    kept, sigs = [], []
    for r in runs:
        s = shingles([a["action"] for a in r["steps"]])
        if all(jaccard(s, t) < threshold for t in sigs):
            kept.append(r); sigs.append(s)
    report["after near-copies"] = len(kept)
    # 3. leaks: drop any run from a test app, or whose instruction shares a 3-word phrase with a test task
    test_phr = set().union(*(words(t) for t in test_tasks)) if test_tasks else set()
    clean = [r for r in kept if r["app"] not in test_apps and not (words(r["instruction"]) & test_phr)]
    report["after leak check"] = len(clean)
    # 4. unroll: one training example per step, skipping steps flagged wrong
    rows = []
    for r in clean:
        history = []
        for st in r["steps"]:
            if st.get("correct", True):
                rows.append({"task": r["instruction"], "previous": list(history), "image": st["image"],
                             "target": st["action"], "weight": r.get("weight", 1)})
            history.append(st["action"])
    report["training examples"] = len(rows)
    return rows, report

if __name__ == "__main__":
    mk = lambda ins, app, acts, ok=True: {"instruction": ins, "app": app, "ok": ok,
          "steps": [{"action": a, "image": f"{app}_{i}.png"} for i, a in enumerate(acts)]}
    runs = [mk("Mute the phone", "settings", ["tap Settings", "tap Sound", "tap Mute"]),
            mk("Mute the phone", "settings", ["tap Settings", "scroll down", "tap Sound", "tap Mute"]),
            mk("Turn on dark mode", "settings", ["tap Settings", "tap Display", "tap Dark mode"], ok=False),
            mk("Set an alarm for 7:00", "clock", ["tap Alarm", "tap Set 7:00"]),
            mk("Save a new note", "notes", ["tap New note", "tap Save"]),
            mk("Set the volume to half", "settings", ["tap Settings", "tap Sound", "drag Volume to 50%"])]
    rows, report = build(runs, judge=lambda r: r["ok"], test_tasks=["Set an alarm for 7:00 tomorrow"], test_apps={"notes"})
    print(json.dumps(report))
    with open("train.jsonl", "w") as f:
        for row in rows: f.write(json.dumps(row) + "\n")

Run it, and six made-up runs go in. The judge turns away the dark-mode run that failed: 5 remain. The second “Mute the phone” differs from the first only by a scroll, so it drops as a near-copy: 4. The alarm task shares a phrase with a test task, and the notes app is held out for the test, so both drop: 2. Their 6 steps become 6 training examples, one JSON line each. On this toy input the function prints {"in": 6, "passed judge": 5, "after near-copies": 4, "after leak check": 2, "training examples": 6}.

Where this sits in the field

This lesson is Act 6 of the path Computer Use and Multimodal Agents: the data act. It comes right after Imitation Learning, which teaches behaviour cloning and why copies drift, and right before Data II: Filtering and Deduplication, which builds the fast near-copy tools this chapter needs at web scale. The act continues with SemDeDup and DataComp, which curates data by meaning instead of exact matches, and Simula, which reasons about what synthetic data to make in the first place. Later, in the training act, TRACE targets the capabilities an agent lacks, instead of adding data everywhere at once.

Coming in this path, built alongside this lesson: Computer-Use Agents: Screenshots In, Clicks Out, the loop this data trains; GUI Grounding, where a click really lands; Agent Environments and Sandbox Fleets, where attempts run and how runs are reset and checked; Agentic RL, with the DigiRL and UI-TARS-2 papers, training by trial and error, where every run needs a judge again; and OSWorld, the standard test of desktop tasks.

Go back to the shelf at the top. Switch People on and off: that is Chapter 2. Exploring is Chapters 4 and 5. The judge sweeping the shelf is Chapter 6, and the red squares it leaves behind are the 26 in 100 that Chapter 6 counted. Then open Present and explain the shelf to someone, or Teach, and draw the four boxes of a judge from memory.