Agentic RL

Picture a phone lying on a desk. Nobody is touching it, yet its screen keeps changing. A small model is driving it. It was given one sentence: search a shopping site for a USB-C cable and open the first result. It taps the address bar, types the shop's address and waits for the page. Then a pop-up slides over the search box, asking it to sign in for deals. The model taps where the search box was. Nothing happens. It taps again, and again, until it runs out of steps.

This lesson is about how to fix that model. Not by writing a rule for pop-ups, but by letting it learn from its own tries. Before we can do that, we need names for the parts of the scene.

What is driving the phone

Think of a new assistant sitting at your phone, reading the screen and pressing buttons for you. A model that uses a computer the way you do, by looking at the screen and tapping, clicking or typing, is an agent. This kind is a computer-use agent. A phone counts as a computer here, and so do a laptop and a web browser.

The agent never sees the phone the way you do. It sees a picture of the whole screen at one moment, a screenshot. It looks at the screenshot and decides what to do.

Then it does one thing to the computer: a tap, a click, a few typed words, a key press. That is one action, like one move in a board game. One look at the screen plus one action is a step. The agent works in steps: look, act, look again at what changed, act again.

The job, given in one sentence, is a task, like a note on a sticky pad. One whole try at a task, from the first screenshot to the end, is an episode, like one attempt at a level in a video game. We use both words for the same thing: "try" when we count them, "episode" when we look at one closely.

An episode cannot run forever. If the agent has not finished after a fixed number of steps, the episode ends anyway: the step limit, like a game clock. In our toy version of the shopping task, the plain path takes 7 steps when nothing goes wrong. That 7 is made up for the picture. The real study this scene is drawn from gave the agent up to 20 steps on shopping tasks.

One step, in and out

Here is what crosses the agent's boundary on one step. Two things go in: the task sentence, and the last two screenshots. One thing comes out: a single line of text that names one action.

one step's outputtap(0.50, 0.22)        # tap the search box: x and y as fractions of the screen, 0 to 1
type("usb c cable")    # type into whatever has focus
press(ENTER)           # one of the special buttons: HOME, BACK, ENTER

The two numbers in tap(0.50, 0.22) say where to tap. They are fractions of the screen's width and height. A 0 is the left or top edge, and a 1 is the right or bottom edge. So 0.50 is the middle across, and 0.22 is a little over a fifth of the way down.

Fractions make the same action work on any screen size, small phone or large. Besides tapping, this agent can type a string of text, slide a finger across the screen, and press three special buttons: HOME, BACK and ENTER.

The exact spelling tap(x, y) is our own made-up example. Every real agent has its own format for actions. Chapter 1 shows a real one, from an agent that drives a desktop computer.

How it learned: by copying

How did this model learn to drive a phone at all? Think of learning to cook by watching videos. Someone records a person doing a task, screen by screen and tap by tap: a demonstration. Collect thousands of them. Then train the model to predict the person's next action from each screen.

A model is a very large set of numbers, its weights, like the settings on a mixing desk. Training changes them a little at a time, so that the model's guess at the person's next action gets closer to what the person really did.

Learning from people's recordings like this is called imitation learning. When the model is a language model, people call it supervised fine-tuning, or SFT. We will just say copying.

Why the copied agent froze

Copying has a blind spot, and our phone fell straight into it. Demonstrations show the path when things go right. People rarely slip on camera, and when they do, the recording seldom shows the way back.

So the copied agent has never seen a screen with that pop-up on it. It has never seen anyone close one. It does what people did at this point in the task: tap the search box. The box is under the pop-up, so the tap lands on the pop-up and nothing happens. The next screen looks just like the last one, so the agent taps again.

Two studies saw exactly this. In a 2024 study of phone agents (DigiRL), the failure that practice cut the most, compared with copying, was "fail to recover from mistakes". In a study of agents on websites (WebRL), copied agents over-used "scroll down" and got trapped in loops.

The arithmetic of surprises

Why does one blind spot matter so much? Count. Suppose that on every step there is a small chance, call it s, that something unexpected appears: a pop-up, a slow page, a moved button. An agent that cannot recover finishes only if no surprise happens on any step.

For one step, the chance that all goes to plan is 1 − s. For two steps in a row it is (1 − s) × (1 − s). For H steps in a row, it is (1 − s) multiplied by itself H times: (1 − s)^H.

Put in illustrative numbers: a 5% chance of a surprise on each step, so s = 0.05. Over 10 steps, 0.95¹⁰ = 0.60. So 4 tries in 10 fail on a surprise alone, however good the agent is at the plain path. Over 20 steps, 0.95²⁰ = 0.36, and now 6 tries in 10 fail.

The 10 is not arbitrary. Good solutions to the tasks on a well-known set of test websites take about 10 steps. Longer tasks make the problem worse, because every extra step is one more chance for a surprise.

An agent that can recover pays one extra step per surprise and carries on. Run both agents on the same task below, first without the pop-up and then with it.

Here are the same numbers written out, line by line. The first block is the toy arithmetic of surprises. The second is the real result we come to in a moment.

worked exampleThe arithmetic of surprises (illustrative: a 5% chance of a surprise on each step)
  chance a step goes as planned         = 1 − 0.05                    = 0.95
  10 steps in a row, no surprise         = 0.9510                   = 0.599
  so, for an agent that cannot recover  = 1 − 0.599                   = 0.401: 4 tries in 10 fail
  20 steps in a row, no surprise         = 0.9520                   = 0.358: 6 tries in 10 fail

What practice bought, on the shopping benchmark's 96 test tasks
  copying only                           = 17.7%  ≈ 17 of 96 tasks
  copying, then practice                 = 67.2%  ≈ 64 or 65 of 96 (the average of two training runs)
  the gain                               = 67.2 − 17.7                 = 49.5 points

Practice

So how could an agent learn the way back? Think of learning to ride a bike. Nobody can show you balance on a video. You ride, you wobble, you fall, and you ride again. Each time, your body keeps whatever worked.

Let the agent try tasks itself, grade every episode, and make whatever worked more likely next time. That is reinforcement learning, RL for short. We will just say practice.

Practice needs a teacher who marks only the final answer. Something must decide whether the task got done. Anything that scores an episode is a grader. A grader written as ordinary code is a checker. A grader that is itself a model is a judge.

The grader's verdict, as a number, is 1 for pass and 0 for fail. That number is the reward: a gold star, or no gold star. In the phone study the grader was a judge. A large model read the final screenshot and the task, and answered "Status: success" or "Status: failure". A success ended the episode with reward 1. On average, this judge disagreed with people only 2.8% of the time.

What practice bought

Did it work? Researchers compare agents on a benchmark: a fixed set of tasks with graders, used to compare agents fairly, like a standard exam.

In the phone study, the same model, with about 1.3 billion weights, was trained two ways. Copying alone finished 17.7% of the shopping benchmark's 96 test tasks, about 17 of them. After practice, it finished 67.2%, about 64 or 65. Large general-purpose models, given the same tasks with careful instructions, stayed under 20%.

A second benchmark uses tasks on five realistic websites that run just for testing: a forum, a code host, a shop's admin panel, a map, and an online shop. There, practice took a model with 8 billion weights from 4.8% to 42.4%. Copying alone had reached 20.6%.

Then the slower lesson: websites change. In the phone study, a copy of the agent frozen after training got worse over the following days as the sites changed. An agent that kept practicing held its success rate.

Why practice is hard

If practice is so good, why is it hard? There are three problems, and the rest of the lesson takes them in turn.

One verdict, many steps. A pass or fail arrives once, after about 10 steps. Which of them earned it? Think of a football team that won: who gets the credit, the striker, the keeper, or the defender who made one quiet tackle? Working out which steps earned the result is called credit assignment.

The right tasks. The instrument at the top showed that a task the agent always passes, or always fails, teaches nothing. Practice has to be aimed.

Slow tries. One step of a large agent's thinking took 4.0 seconds in one report. Then the computer must respond: a page loads, an app opens. Thousands of episodes take serious time and serious machinery.

The instrument, explained

In the instrument at the top, the agent tried one task 8 times. Each row was one episode. The rows that passed were pushed up and the rows that failed were pushed down, by an amount you can now start to read. Here is the road, in order:

1. One score for every step: how eight tries at one task become a push for every word the agent wrote.
2. Which step earned it: a second model that guesses the chance of success on every screen.
3. Tasks at the edge: why the hardest and easiest tasks teach nothing, and what to do about it.
4. Slow computers, fast models: running hundreds of computers at once, and learning from slightly old tries.
5. Keeping training stable: what stops the agent from settling on one answer too early.
6. Reading a flat reward curve: four faults that look the same, and the measurement that tells them apart.
7. What practice bought: the published gains, read honestly.
8. Field Guide: the loop, the checklist, the numbers and their sources.

Each part of the instrument at the top has its chapter. The rows and their pushes are Chapter 1. The same push on every step is Chapter 2. The difficulty chips and Replay are Chapter 3. Rows that finish at different times are Chapter 4. Eight identical rows are Chapters 5 and 6.

Eight copies of the same desktop computer sit side by side, each one simulated inside a bigger machine. On each, the same document is open, and the agent gets the same task: turn the 2 in H2O into a subscript. All eight start at once. Two finish. Six do not. Each episode took about ten steps, so there are about 80 steps on the table. Which of them should the agent do more often?

This is a real task, from a 2025 study of an agent that drives a desktop computer. Before practice, that agent passed it 25% of the time, which is 2 tries in 8. After practice, it passed 62.5% of the time. The "about ten steps" in our scene is illustrative.

Copies of a computer

How can eight tries start from exactly the same screen? Each runs on a whole computer simulated inside another computer, a virtual machine. Think of it as a computer inside a computer. A training run can start hundreds of identical copies, all showing the same screen at the start. For now, eight is enough. Chapter 4 runs hundreds of them at once.

How the agent writes an action

A language model writes text in small pieces, like Scrabble tiles laid down one at a time. Each piece is a word or part of a word: a token.

For each token, the model gives every possible next piece a chance, its probability, and then picks one. The chance of a whole action is the chances of its tokens multiplied together.

The desktop agent writes each step as text. First comes a short thought, then the action. Here is one step, as the agent wrote it:

what the desktop agent writesThought: H2O is selected. The 2 must become a subscript, so I open the Format menu.
Action: click(start_box='(863,149)')

# two more real actions from the same study
hotkey(key='ctrl z')
drag(start_box='(689,493)', end_box='(710,495)')

The numbers are pixel positions on the screen: click(start_box='(863,149)') clicks the point 863 pixels across and 149 pixels down. hotkey presses keys together, here Ctrl and Z to undo. drag presses at one point and releases at another. The action strings are real. The wording of the thought is our own illustration.

The desktop agent can do a few more things: right-click, scroll, and type text. It also has four special actions: wait, finish, fail, and call the user. Every one of them is just text the agent writes.

The chance of an action

Each token was chosen with some probability, so the action as a whole has a probability too: the token chances multiplied together. Take an illustrative action of 12 tokens, each chosen with probability 0.95. The whole action has probability 0.95¹² = 0.54.

A logarithm turns multiplying into adding: log(0.95 × 0.95) is exactly log(0.95) + log(0.95), the same way 10² × 10³ is 10⁵, adding the exponents instead of multiplying the powers. So instead of multiplying twelve small numbers together, which shrinks toward 0 fast and can get too tiny for a computer to hold well, we add twelve logarithms, which shrinks far more gently and stays easy to compute with.

The logarithm of a probability is its log-probability. For our action it is 12 × ln 0.95 = 12 × (−0.0513) = −0.62. A log-probability is never above 0, and the closer to 0, the more likely the text. Every later step in this lesson, the loss, the per-episode push, the ratio between two policies, is built out of adding and subtracting log-probabilities instead of multiplying probabilities, because addition is the easier arithmetic to build training on.

Policy, and how training changes it

Think of a player's habits: in a given position, they are likely to make some moves and unlikely to make others. Seen as a rule for choosing actions, the agent's model is its policy. Given everything on the screen so far, it gives every possible next action a probability. We write it π, the Greek letter pi, for policy.

Training changes the policy by changing the weights. One gradient step moves every weight a tiny amount in the direction that makes a chosen piece of text more likely, or less likely. It is like turning every knob on the mixing desk a hair at once, each in the direction that helps.

The push has a size. A push of 2 moves the weights twice as far as a push of 1, and a push of −1 moves them the other way. That is all of training's machinery this lesson needs. The full derivation of "push up what worked" is in Policy Gradients.

The simplest rule, and its flaw

Here is the simplest rule you could write. After a pass, push up every action of that episode. After a fail, push down. Use the reward itself as the push: 1 for a pass, 0 for a fail.

It has two flaws. First, a fail gets a push of 0, so the agent learns nothing from its mistakes. Second, on an easy task where every try passes, every action is pushed up by the same amount. The agent learns nothing about which actions were better than others.

Compare with a baseline

A bowler who scores 150 is not judged against zero. They are judged against their usual score. A 150 is a great night for someone who usually scores 110, and a poor one for someone who usually scores 200.

Training does the same. What we compare a reward against is the baseline. How much better an episode did than its baseline is its advantage, like beating your own average.

The best baseline for one task is how well the agent usually does on exactly this task, from exactly this screen. We already have it. The 8 episodes of one task, started from the same screen and compared with each other, form a group, like eight runners in one heat. The group's own average is the baseline.

One more adjustment. Some groups are close, and some are far apart. How far the rewards in the group spread from their average is measured as the standard deviation. We will call it the spread, like how scattered the darts are on a board. Dividing by the spread makes an easy task and a hard one push with a similar strength.

Comparing each episode with its own group, and nothing else, is group relative policy optimization, GRPO. It is grading on the curve within one class. It was introduced in a 2024 math-reasoning paper that sampled 64 answers per question. A 2025 desktop-agent study carried it over to whole episodes, with 8 tries per task.

Â_i

The advantage of episode i: the push every token of it gets.

r_i

Episode i's reward: 1 for a pass, 0 for a fail, −1 for an unreadable action.

mean(r)

The group's average reward: the baseline.

std(r)

The group's spread.

Two passes in eight: (1 − 0.25) ÷ 0.433 = +1.73;  (0 − 0.25) ÷ 0.433 = −0.58

Why subtracting the average is allowed

Is it fair to subtract a number from every reward? It is. Subtracting the same number from every episode in a group does not change which episodes end up pushed hardest relative to each other, only how far from zero the pushes land.

Here is the reason in plain words, and it is a different kind of "push" from the one you just computed. The push you just computed (+1.73 for a pass, −0.58 for a fail) moves the agent's weights. A separate, smaller fact is also true: the policy's chances of everything the agent could do at one screen always add up to 1, because it has to pick something. Picture a policy that only ever says one of three things, with chances 0.5, 0.3 and 0.2: these three numbers must always add to 1. If training tried to raise all three chances by the same amount, they could not all actually go up, because the total is stuck at 1. Raising one chance always lowers at least one other.

That is the sense in which adding the same constant to every episode's reward, before computing its push, does not change which actions end up more or less likely than before: it only changes how far from zero the pushes are, not their order. Our group of eight already shows this. Score it as 1 and 0 (what we did) or as 2 and 1 (add 1 to every reward first): either way the average moves with it, so the pass still ends up pushed up and the fail still ends up pushed down by the same +1.73 and −0.58. The baseline subtraction does not add information; it only strips out the part every episode shared, so training sees the part that actually tells passes and fails apart. The full proof is in Alignment II: RLVR and GRPO.

The numbers for our scene

Two passes in eight. The average is (1 + 1) ÷ 8 = 0.25. For rewards that are only 1 or 0, the spread has a short form: √(average × (1 − average)). That is because with a share m of 1s, the average of the squared rewards is also m, so the spread is √(m − m²). Here, √(0.25 × 0.75) = 0.433.

So each pass gets (1 − 0.25) ÷ 0.433 = +1.73, and each fail gets (0 − 0.25) ÷ 0.433 = −0.58. Check that they add up to zero: two passes give 2 × 1.73 = 3.46, and six fails give 6 × 0.58 = 3.46. The group pushes up exactly as much as it pushes down.

Flip the tries in the device below and watch every push change at once, because every episode is measured against the same group.

Many steps, one sequence

So far each episode got one number. But an episode has many steps. Which tokens does the number land on?

To the model, an episode is one long piece of text that it reads and writes. It reads the task, then screenshot 1. It writes thought and action 1. It reads screenshot 2, writes thought and action 2, and so on to the end. A model that reads images turns each screenshot into tokens too, by cutting the picture into small squares and turning each square into a token.

The desktop study does the simplest thing: the episode's one advantage goes onto every token the agent wrote, in every step. A passing episode has all of its thoughts and actions pushed up by +1.73. A failing one has all of them pushed down by −0.58. That is why the hero painted the push back along every step of every row.

Why the screenshots get no push

The screenshots are tokens in the same sequence. Should they be pushed too? No, and the reason takes six short steps.

1. The chance that a whole episode happens is a product: for every step, the chance the agent wrote its text, times the chance the computer showed the next screen.
2. In symbols: P(episode) = π(text₁) × C(screen₂) × π(text₂) × C(screen₃) × …, where π is the agent and C is the computer.
3. Take logarithms, and the product becomes a sum: log P = log π(text₁) + log C(screen₂) + log π(text₂) + …
4. Training changes only the agent's weights. The computer's chances, the C terms, do not depend on those weights at all.
5. So when training asks which way to move the weights, every C term contributes exactly nothing. Only the agent's own tokens are left.
6. In code, a row of 1s and 0s, the loss mask, keeps the agent's tokens and zeroes out the rest.

Two names from that list. The number training tries to make smaller is the loss. A row of 1s and 0s that says which tokens the agent wrote is the loss mask. Think of a highlighter run over only your own lines in a script.

Now the budgets of the desktop study. Everything the model reads before it writes is its context, like the pages open on your desk. The study allowed at most 15 screenshots and about 64,000 tokens of context, on screens 1080 pixels tall, and at most 15 steps per episode. That is about 4,300 tokens per step, screenshot plus text, by our own arithmetic.

The rewards had three values. The study's checker gave 1 if the task was done and 0 if not. And when the agent wrote something that could not be read as an action, it got −1: a format penalty, like a spoiled ballot. The penalty matters because an unreadable action wastes a whole episode, and the group rule then pushes it down harder than a plain fail.

L

The loss: training makes it smaller.

N

How many agent tokens the batch holds.

m_i,t

The mask: 1 if the agent wrote token t of episode i, else 0.

o_i,t

Token t of episode i.

log π(o_i,t)

The log-probability the agent gives that token.

This is a simplified teaching form of our own. The full objective, with a comparison against the older agent that wrote the tokens, returns in Chapter 4.

Here is the whole chapter's arithmetic, line by line, including the group with an unreadable action in it.

worked exampleEight tries at one task: passes on tries 1 and 4
  rewards               = 1, 0, 0, 1, 0, 0, 0, 0
  average               = (1 + 1) ÷ 8                              = 0.25
  spread                = √(0.25 × (1 − 0.25)) = √0.1875           = 0.433
  push for a pass       = (1 − 0.25) ÷ 0.433                       = +1.73
  push for a fail       = (0 − 0.25) ÷ 0.433                       = −0.58
  check                 = 2 × (+1.732) + 6 × (−0.577) = 3.464 − 3.464 = 0

The same group, but try 8 wrote an action the computer could not read
  rewards               = 1, 0, 0, 1, 0, 0, 0, −1
  average               = (1 + 1 − 1) ÷ 8                          = 0.125
  spread                = √(average of the squares − average²)
                        = √(3 ÷ 8 − 0.125²) = √(0.375 − 0.015625) = √0.359375 = 0.599
  push for a pass       = (1 − 0.125) ÷ 0.599                      = +1.46
  push for a plain fail = (0 − 0.125) ÷ 0.599                      = −0.21
  push for try 8        = (−1 − 0.125) ÷ 0.599                     = −1.88

One episode, as one long sequence (the desktop study's limits)
  at most 15 screenshots, about 64,000 tokens of context, at most 15 steps
  64,000 ÷ 15                                                      ≈ 4,300 tokens per step (our arithmetic)
  every token the agent wrote, in all its steps, gets the same push; screenshot tokens get none

One action's chance (illustrative: 12 tokens, each chosen with probability 0.95)
  probability           = 0.9512                                     = 0.540
  log-probability       = 12 × ln 0.95 = 12 × (−0.0513)            = −0.616

The data, as the code sees it

Every finished episode is saved. The saved record of an episode, every screenshot, thought and action plus its reward, is its trajectory, like a flight recorder. Here is one, with field names we made up for illustration:

one trajectory record{
  "task_id": "subscript-h2o",
  "group": 17,
  "episode": 3,
  "policy_version": 41,
  "steps": [
    { "screen": "ep3/step01.png",
      "text": "Thought: H2O is selected ... Action: click(start_box='(863,149)')",
      "agent_tokens": 38 },
    { "screen": "ep3/step02.png",
      "text": "Thought: The Format menu is open. Action: click(start_box='(902,311)')",
      "agent_tokens": 29 }
  ],
  "reward": 1,
  "advantage": 1.73
}

Read it from the top. task_id and group say which task this episode tried and which group it belongs to, so it can be compared with its seven siblings. episode numbers it inside the group.

policy_version says which version of the agent wrote it. Training keeps changing the agent, and Chapter 4 needs to know how old each episode is.

Each entry in steps holds the screenshot's file and the agent's exact text for that step, with how many tokens the agent wrote. The loss mask is built from those counts: 1 for each of the 38 agent tokens, 0 for every screenshot token around them.

reward is the grader's verdict. advantage is filled in last, only after the whole group has finished, because it needs the other seven rewards.

And here is the group rule and the masked loss in a few lines of Python, with each array's shape:

pythonimport numpy as np

rewards = np.array([1, 0, 0, 1, 0, 0, 0, 0], dtype=float)   # one group: 8 episodes of one task
adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6)   # +1.73 for a pass, -0.58 for a fail

# logp: shape (8, T), the log-probability of every token of each episode under the current weights
# mask: shape (8, T), 1 where the agent wrote the token, 0 for screenshot tokens and padding
loss = -(adv[:, None] * mask * logp).sum() / mask.sum()     # every agent token pushed by its episode's advantage
# a gradient step that makes this loss smaller pushes the passes up and the fails down

The shapes: rewards and adv hold 8 numbers, one per episode. logp and mask are 8 rows of T numbers, where T can reach about 64,000 tokens. The adv[:, None] copies each episode's one number along its whole row. Real trainers add two things this sketch lacks: a comparison with the older agent that wrote the tokens (Chapter 4), and a careful choice of how to average (Chapter 5).

One step further, for later. A 2025 variant keeps one comparison number per episode but lets each token carry its own advantage, aimed at agents that act over many turns. Chapter 2 shows the main way agents that use a screen give different steps different credit today.

Replay one of the two winning episodes from Chapter 1, step by step. Steps 1 to 3 are clean: select the text, open the menu, find the effects. At step 4 the agent opens the wrong menu. At step 5 it closes it again. Then it finishes cleanly. The group rule gave all ten steps the same push, +1.73, the wrong menu included.

Fair on average, but slow

Is that wrong? Not exactly. Over many episodes, the wrong menu shows up in passes and fails alike. Its pushes up and its pushes down cancel out in the long run. What is left is the push from the steps that really mattered.

But "in the long run" means many episodes, and every episode costs real seconds on a real computer, as Chapter 4 will show. So here is a better question. Can we tell, inside one episode, which step mattered?

The commentator's line

Watch a football match on television. Many broadcasts now show a line for each team's chance of winning, updated after every play. The line jumps after a goal or a red card. Those jumps tell you which moments mattered, before the match is over.

An agent can have the same line. A second model looks at a screen and guesses the chance that the agent will finish from here. That guess is a value estimate, written V. The model that makes it is the critic.

How the critic learns

The critic learns from finished episodes. Every screenshot of every episode gets a label: how that episode ended. A screenshot from an episode that passed is labelled 1, and one from an episode that failed is labelled 0.

The critic learns to predict that label from the screen and the task. It is a yes-or-no guess, so its output is a chance between 0 and 1. A screen early in a good episode should get a middling guess. A screen with the right dialog open, one click from the end, should get a high one.

Think of a weather forecaster scored on the days they said "90% rain". If it stays dry on such a day, that confident miss should cost them a lot. The usual loss for a yes-or-no guess works the same way: cross-entropy, which punishes a confident wrong guess hardest.

Two studies trained their critic this way, and one measured the choice. In the phone study, training the critic with cross-entropy instead of plain squared error improved results by about 12%.

The critics themselves were different sizes. In the phone study the critic was small: a frozen image reader and a text reader, feeding a two-layer network. In the web study the critic was the copied model itself, with a new output added on top.

Credit from the critic's jumps

Now use the line. For each step, take the change in the critic's guess from the screen before the step to the screen after it: V(after) − V(before). If the guess rose, the step helped. If it fell, the step hurt.

The change in the critic's guess across one step is the one-step advantage. Engineers also call it a temporal difference, or TD, because it is a difference across one tick of time.

In our scene, before step 4 the critic says 0.48: about an even chance. After the wrong menu opens, it says 0.18. That step cost 30 points of chance: its one-step advantage is 0.18 − 0.48 = −0.30. Step 5, closing the menu, wins them back: +0.30.

Two ways to be wrong

Picture two bathroom scales. One always reads 2 kg heavy: it is wrong in the same direction every time. A guess like that is biased. The other is right on average but jumps around from one weighing to the next. A guess like that is noisy. Engineers say it has high variance.

The group rule of Chapter 1 gives every step the episode's real outcome. That is like judging a chess move by who won the game. It is the Monte Carlo estimate, named after the casino, because you learn by playing it out. It is never biased: on average, it gives each step its true worth. But it is noisy. One lucky finish credits every step of that episode, the bad ones too.

The critic's jumps are the other scale. They are steadier, because they look at one step at a time. But where the critic is wrong, its jumps are wrong every time. So the useful answer is a mix of the two, set by a dial between trusting the outcome and trusting the critic: λ, the Greek letter lambda. Think of a mixing slider.

The phone study's mix

Here is how the phone study built each step's credit, in six short steps.

1. Number the steps h = 1, 2, …, H, where H is the last step. Let r be the episode's final reward, 1 or 0.
2. The critic's jump at step h is V(screen after step h) − V(screen before step h).
3. The real outcome's share of the credit at step h is λ^{H − h}. At the last step (h = H) it is λ⁰ = 1, all outcome. One step earlier it is λ, two steps earlier λ², and it fades toward the start.
4. So the credit of step h is the outcome's share times r, plus the rest of the weight times the critic's jump.
5. Written out: A_h = λ^{H − h} r + (1 − λ^{H − h} r) × (V_after − V_before).
6. Keep step h for training only if A_h > 1 ÷ H.

A_h

The credit of step h.

λ

The dial between the real outcome (1) and the critic (0).

H

The last step.

r

The final reward, 1 or 0.

V(s_h)

The critic's guess of success before step h.

V(s_h+1)

Its guess after step h.

Step 4: 0.016 + 0.984 × (0.18 − 0.48) = −0.28, below 0.10, so it is dropped

Read the formula in words. Near the end of the episode, trust the real outcome: the reward is close in time, so it is probably about these steps. Near the start, trust the critic's jumps: the reward is far away, and much could have happened since.

The phone study set λ = 0.5. Then it kept only the steps whose credit was above 1 ÷ H, where H is the step limit. For a 10-step limit, that is 0.10. It trained on the kept steps exactly the way copying trains: make them more likely. The steps that hurt, or barely helped, are simply left out.

Mixing the outcome and the critic's jumps with a dial like λ is a simple form of generalized advantage estimation, GAE. The phone study calls its own version a simplified GAE.

Move the wrong turn in the device below, and slide the dial. Watch how the credit flows from the critic to the outcome as λ rises.

Notice what happens when you move the wrong turn late, to step 8. At λ = 0.5 its credit becomes +0.03. It is no longer negative, yet it is still below 0.10, so it is still dropped. Near the end, the outcome's share is large (0.25 at step 8), and the outcome was a pass, so the real result outweighs the critic's dip. That is the price of trusting the outcome near the end.

Here is the arithmetic in full, for the wrong turn at step 4.

worked exampleA 10-step episode that passes (r = 1, H = 10), with a wrong turn at step 4      (illustrative)
  the critic's guess before each step, V(s1) … V(s10):
    0.15  0.26  0.37  0.48  0.18  0.48  0.59  0.70  0.81  0.92
  steps 1 to 3 and 6 to 9 each raise the guess by 0.11; step 4 drops it by 0.30; step 5 restores it

With λ = 0.5 (the phone study's value)
  outcome share λ^(10−h):  step 1: 0.002   step 4: 0.016   step 8: 0.25   step 9: 0.5   step 10: 1
  step 1:  0.002 + (1 − 0.002) × (0.26 − 0.15)  = 0.002 + 0.998 × 0.11     = +0.112
  step 4:  0.016 + (1 − 0.016) × (0.18 − 0.48)  = 0.016 + 0.984 × (−0.30)  = −0.280
  step 5:  0.031 + (1 − 0.031) × (0.48 − 0.18)  = 0.031 + 0.969 × 0.30     = +0.322
  step 9:  0.5   + (1 − 0.5)   × (0.92 − 0.81)  = 0.5   + 0.5 × 0.11       = +0.555
  step 10: 1     + (1 − 1)     × (anything)                                 = +1.000
  all ten: +0.112  +0.113  +0.117  −0.280  +0.322  +0.166  +0.221  +0.333  +0.555  +1.000
  keep if above 1 ÷ H = 1 ÷ 10 = 0.10: steps 1, 2, 3, 5, 6, 7, 8, 9, 10 kept; step 4 dropped

The two ends of the dial
  λ = 1: every step gets 1 + 0 × (jump) = +1.00, the wrong turn included (the group rule's verdict)
  λ = 0: every step gets its jump (+0.11, or −0.30 at step 4, +0.30 at step 5); step 10 gets +1.00

The web study's mix at the same wrong turn (λ = 0.5)
  0.5 × (0.18 − 0.48) + 0.5 × (1 − 0.48)  = −0.15 + 0.26                       = +0.11

A critic for an agent whose λ grows with length (α = 0.05)
  λ = 1 − 1 ÷ (0.05 × length):  length 100   → 1 − 1 ÷ 5  = 0.80
                                length 1,000 → 1 − 1 ÷ 50 = 0.98

The same thing, as code you can run. The function walks the steps, works out the outcome's share and the critic's jump, and mixes them:

pythondef step_credit(V, r, lam):
    """V: the critic's guess before each of the H steps; r: the final reward (1 or 0)."""
    H = len(V)
    credit = []
    for h in range(1, H + 1):
        share = lam ** (H - h)                       # the real outcome's share at step h
        after = V[h] if h < H else 0.0               # guess after the step (0 once the episode is over)
        jump = after - V[h - 1]                      # the critic's change across step h
        credit.append(share * r + (1 - share * r) * jump)
    return credit

V = [0.15, 0.26, 0.37, 0.48, 0.18, 0.48, 0.59, 0.70, 0.81, 0.92]
A = step_credit(V, r=1, lam=0.5)
keep = [h + 1 for h, a in enumerate(A) if a > 1 / len(V)]   # the 1/H filter
print([round(a, 3) for a in A])   # [0.112, 0.113, 0.117, -0.28, 0.322, 0.166, 0.221, 0.333, 0.555, 1.0]
print(keep)                       # [1, 2, 3, 5, 6, 7, 8, 9, 10]

For a failed episode, r = 0, and the formula reduces to the critic's jumps alone. At the last step the jump is 0 − V(s_H), because the episode is over and nothing is left to win. With these guesses, step_credit(V, r=0, lam=0.5) prints [0.11, 0.11, 0.11, -0.3, 0.3, 0.11, 0.11, 0.11, 0.11, -0.92]: only the last step takes the blame in full.

And here is the critic's own training, as a yes-or-no guess with cross-entropy. The shapes are illustrative:

python# feats: (N, D) one row per screenshot, from a frozen image reader plus the task text
# label: (N,)  1 if that screenshot's episode passed, else 0
p = 1 / (1 + np.exp(-(feats @ w + b)))                        # the critic's guess, between 0 and 1
loss = -np.mean(label * np.log(p) + (1 - label) * np.log(1 - p))   # cross-entropy

A different mix, a different verdict

The web study mixed the same two ingredients differently. Its credit for a step is half the critic's jump, plus half of "final reward minus the critic's guess", also with λ = 0.5.

At the wrong turn in our episode, that gives 0.5 × (−0.30) + 0.5 × (1 − 0.48) = −0.15 + 0.26 = +0.11, not −0.28. Half of its weight looks at the episode's happy ending, so the wrong turn still earns a little credit.

Neither is "the" answer. The choice decides how much a single step can be blamed inside a winning episode. It is a design decision, and it should be made on purpose.

Not a process reward

Be clear about what the critic is not. In these computer tasks, the computer gives no reward until the end. The critic only guesses, from end results it has seen before.

A different idea comes from math reasoning. There, a model is trained to grade each step of working directly: a process reward model, like a teacher who marks every line of a student's working. The credit of a token is then the sum of the step grades that follow it.

The difference matters in practice. A critic learns from end results, which practice gives you for free. A process reward model needs step-by-step grades from somewhere, usually from people.

The critic at scale

GRPO, the group rule, is a simplification of an older method that learns with a critic: proximal policy optimization, PPO. A large 2025 desktop-agent report trained with PPO and a critic. It found that PPO beat GRPO for its agent, with higher reward and less up and down along the way.

Its critic needed care. Early on, the critic's guesses were often negatively related to the real rewards: it was most hopeful about episodes that would fail. Think of a new referee who should watch games for a while before blowing the whistle.

So the team trained the critic first, with the agent frozen, until its guesses were good. That is value pretraining. The critic learned from plain end results (λ = 1) until its loss and its explained variance looked good.

How much of the spread in real outcomes the critic's guesses account for is its explained variance. A value of 1 is perfect. A value of 0 is no better than always guessing the average. Below 0 is worse than that.

The report also used different λ values for the agent and for the critic. And it let the agent's λ grow with the episode's length, as λ = 1 − 1 ÷ (0.05 × length). An episode of length 100 gets 0.80. One of length 1,000 gets 0.98, so a long episode leans more on its real outcome.

What step credit bought

In the phone study, giving each step its own credit made learning 12% more efficient than crediting whole episodes alone. The trained agent also took fewer steps than one trained by a simpler method: copying only its own successful episodes. On general tasks it took about 6.3 and 6.6 steps, against 8.1 and 7.4, over two runs each.

You hand the agent a full set of a few hundred desktop tasks and start practice. A day later the reward curve, the average reward of each update drawn over time, has hardly moved. You open the logs. On most tasks, all eight tries failed. On a few, all eight passed. Not one of those groups changed a single weight.

Groups that cannot teach

Recall the rule from the instrument at the top and from Chapter 1. A group whose rewards are all equal has a spread of 0. Every try scores exactly the group's average, so every push in it is 0. The code adds 0.000001 to the spread, so it gives a clean 0 rather than an error, but a 0 is still a 0.

Those groups are not free. Each one cost a full set of episodes: eight virtual machines, up to 15 steps each, seconds of thinking per step. And they teach nothing. A practice run full of them burns time and goes nowhere.

How often does a group teach?

So the useful question is: for a given task, how often will a group hold at least one pass and one fail? Work it out in five steps. This formula is our own arithmetic, not a number from a paper, and it assumes the tries do not affect each other.

1. One try passes with chance p and fails with chance 1 − p.
2. If the tries do not affect each other, all G fail with chance (1 − p) × (1 − p) × … G times: (1 − p)^G.
3. In the same way, all G pass with chance p^G.
4. A group teaches whenever it is neither, so the share of teaching groups is 1 − p^G − (1 − p)^G.
5. With replay (we come to it below), an all-fail group is rescued, so only all-pass groups are wasted: 1 − p^G.

share(p)

The fraction of groups with at least one pass and one fail: the groups that teach.

p

The chance one try passes.

G

Tries per group.

p^G

The chance that all G pass.

(1 − p)^G

The chance that all G fail.

p = 0.01, G = 8: 1 − 0.01⁸ − 0.99⁸ = 0.077, about 8 groups in 100 teach anything

The shape of that formula is a hill. At p = 0.5 nearly every group teaches: 99.2% of them. At p = 0.1 it falls to 57%, and at p = 0.01 to 7.7%. The other side mirrors it: a task the agent solves 9 times in 10 teaches as rarely as one it solves 1 time in 10. That is exactly what the difficulty chips at the top showed.

What it costs

If only some groups teach, each teaching group costs G ÷ share episodes. At p = 0.01 with 8 tries per group, that is 8 ÷ 0.077 = about 104 episodes for one group that teaches anything.

Now put time on it. One large agent took 4.0 seconds of thinking per step. If episodes run to a 15-step limit, 104 episodes cost 104 × 15 × 4.0 = 6,240 seconds. That is about 1.7 hours of thinking alone, longer than a long meeting, for one push. This combines two published figures; the product is our own arithmetic.

Bigger groups

More tries per group rescue harder tasks. At p = 0.05, a group of 8 teaches 34% of the time, and a group of 16 teaches 56% of the time. But every group then costs twice as many episodes.

Real systems made different choices. The desktop study used 8 tries per task. A 2025 math-reasoning recipe used 16. The original math paper that introduced group scoring used 64.

The device below puts a whole pool of tasks on the hill. Choose which tasks to practice on and how many tries go in each group, and read how much practice teaches.

Every fix below does one thing: it moves practice toward the middle of that hill.

Fix 1: choose tasks before practice

The desktop study did the simplest thing first. It ran its starting agent 16 times on every task of a 369-task desktop benchmark. It kept only the tasks the agent solved at least once: 128 tasks, about a third of them.

Training on those 128 gave clearly larger spreads inside groups than training on the full set. The full set gave, in the study's words, "flatter reward distributions". Flat rewards mean equal rewards, and equal rewards mean no push.

The 16-try test is generous on purpose. A task the agent solves only 1 time in 20 still passes the test with chance 1 − 0.95¹⁶ = 56%. So many tasks at the hard edge stay in.

Fix 2: drop groups that cannot teach, while training

A coach chooses drills: which ones, and in what order. The mix and order of tasks the agent practices on is its curriculum.

A good coach also skips drills the team always gets right, or never gets right. The training version is to sample extra groups, throw away every group whose rewards are all equal, and keep going until the batch is full. That is dynamic sampling.

Why does this matter more as training goes well? Because the better the agent gets, the more groups pass on every try. The share of useless groups grows, and the useful part of each batch shrinks, just when the agent is ready to learn harder things.

Dynamic sampling's own evidence comes from math reasoning, not from agents that use a screen. In a 2025 math recipe it was the last of five fixes, and it took the score on a hard math exam from 42 to 50 points. Agent builders borrowed it.

Fix 3: replay a saved success

Think of a highlight reel: the best moments, kept so you can watch them again. A store of past successes, kept per task, is a replay buffer.

The desktop study stores each task's successful episodes. When a whole group for a task fails, it swaps one failure for a stored success of that same task. Now the group has a pass and seven fails again, and it teaches.

That changes the arithmetic. With replay, an all-fail group is rescued, so the share of teaching groups becomes 1 − p^G, by our own arithmetic. Only all-pass groups remain useless. This is the Replay switch on the instrument at the top.

It worked. With replay, the average reward at the end of training rose from 0.65 to 0.75. The replay run pulled ahead from around update 30 and stayed ahead.

Fix 4: write new tasks at the right difficulty

A good tutor writes new exercises from your mistakes. The training version is to use the tasks the agent failed as seeds for new, similar tasks, and to keep only those at the right difficulty. That is a self-evolving curriculum.

The web study did exactly this. It took tasks the agent had failed and asked a large model to write new, similar ones. Its critic scored each new task at its starting screen, and the study kept only those with an estimated chance of success between 0.05 and 0.75. The large model then removed tasks that could not be done at all.

Each round added 500 new tasks, for 8 rounds. Without this curriculum, practice "progresses more slowly and reaches a lower performance ceiling", in the study's words.

The phone study did something related. A second critic scores whole tasks: how likely is the agent to solve this one? Training then put first the successes on tasks the agent usually fails. That sped learning by about 25%.

The same edge, for old data

The web study also replays old successes, but not all of them. Think of how hard a sentence is for you to predict: a cliché is easy, a sentence in a language you barely know is hard. How surprised the current agent is by an action, as a number from 1 (not surprised at all) upward, is its perplexity.

Perplexity is just a probability turned upside down: if the agent currently gives an action a chance p, its perplexity for that action is 1 ÷ p. An action it is almost sure about, p close to 1, has perplexity close to 1. An action it thinks is unlikely, a small p, has a much larger perplexity. The study kept only old actions with perplexity between 1 ÷ 0.95 = 1.05 and 1 ÷ 0.5 = 2.0: neither too easy nor too hard. It also capped old data at twice the size of the new data.

A one-round test backs the band. Keeping only the middle band gave 31.5% success. Keeping all old data gave 29.1%, only the easy ones 27.9%, and only the hard ones 23.0%. The edge matters for old data too.

Here is every number of this chapter, in one block:

worked exampleShare of groups that teach, G = 8 tries per task                 (our arithmetic)
  p = 0.01:  1 − 0.018 − 0.998  = 1 − 0.000 − 0.923  = 0.077   (about 8 groups in 100)
  p = 0.05:  1 − 0.058 − 0.958  = 1 − 0.000 − 0.663  = 0.337
  p = 0.10:  1 − 0.108 − 0.908  = 1 − 0.000 − 0.430  = 0.570
  p = 0.25:                                           = 0.900
  p = 0.50:  1 − 0.58 − 0.58    = 1 − 0.004 − 0.004  = 0.992
  p = 0.90:  the mirror of 0.10                       = 0.570
  p = 0.99:  the mirror of 0.01                       = 0.077

Episodes spent per group that teaches, G = 8:  8 ÷ share
  p = 0.01: 8 ÷ 0.077 = 104      p = 0.10: 8 ÷ 0.570 = 14.0      p = 0.50: 8 ÷ 0.992 = 8.1
  at p = 0.01, worst case: 104 episodes × 15 steps × 4.0 s = 6,240 s ≈ 1.7 hours of thinking

Group size, at p = 0.05:   G = 4: 0.185     G = 8: 0.337     G = 16: 0.560
Group size, at p = 0.10:   G = 4: 0.344     G = 8: 0.570     G = 16: 0.815

With replay (G = 8): 1 − p8   p = 0.01 → 1.000   p = 0.90 → 0.570   p = 0.99 → 0.077

Choosing tasks before practice (the desktop study)
  16 tries per task on 369 tasks; keep a task if it passed at least once
  kept: 128 of 369 = 35%
  chance a task with p = 0.05 survives the 16-try test: 1 − 0.9516 = 0.56

The web study's band for old actions (perplexity)
  1 ÷ 0.95 = 1.053      1 ÷ 0.5 = 2.0

And the four fixes as one sampling loop. Every name in it is defined above:

pythondef fill_batch(tasks, run_group, replay, batch_groups=32, G=8):
    batch = []
    while len(batch) < batch_groups:
        task = tasks.sample()                        # fix 1: tasks were pre-filtered to the solvable ones
        group = run_group(task, G)                   # G episodes of this task, in parallel
        rewards = [ep.reward for ep in group]
        if max(rewards) <= 0 and task.id in replay:  # fix 3: all failed, swap in a saved success
            group[0] = replay[task.id]
            rewards[0] = 1.0
        for ep in group:
            if ep.reward == 1.0:
                replay[task.id] = ep                 # remember this task's latest success
        if min(rewards) == max(rewards):             # fix 2: all equal, nothing to compare
            continue                                 # drop it and sample another (dynamic sampling)
        batch.append(group)
    return batch                                     # fix 4 (new tasks) refills `tasks` between rounds

Read it in order. A task is drawn from a pool that fix 1 already filtered. Its group runs. If every try failed and a saved success exists, one try is swapped for it (fix 3), and every new success is saved for later. If the rewards are still all equal, the group is dropped and another is drawn (fix 2). Between rounds, fix 4 refills the pool with new tasks near the edge.

Time one step with a stopwatch. The agent reads the screenshot and writes its thought and its action: a few seconds. Then the computer has to respond: a page loads, an app opens, a file saves, a few more seconds. Now multiply. Eight tries of each task, dozens of tasks per batch, up to 15 steps per episode. Most episodes finish in a minute. One, stuck behind a slow page, might take ten: a made-up worst case, not a measured number.

Where the time goes

Start with the agent's half of a step. One large agent wrote 29.6 tokens a second. That is about 34 thousandths of a second per token, quicker than a blink. A whole round of thinking, from screenshot in to action out, took 4.0 seconds, because a round holds many tokens.

The same report tried storing the agent's weights with fewer binary digits each, so the chips have less data to move. The agent then wrote 47 tokens a second and took 2.5 seconds per round. The price was a small loss of skill: its success on a desktop benchmark slipped from 47.5% to 44.4%.

Then add the computer's own time, which no faster model removes. Pages load at the speed of the network. Apps open at the speed of the disk. A 15-step episode spends 60 seconds on thinking alone, by our own arithmetic (15 × 4.0), before the computer's share is even counted.

The machinery

Think of a practice room that tidies itself after every session. The computer the agent acts on, plus the code that resets it and runs the grader, is the environment. For desktops it is usually a virtual machine. For phones, it is a simulated phone, an emulator.

The agent's model runs on graphics chips, GPUs, which do the arithmetic of a large model quickly. Rather than give every environment its own copy of the model, one program holds the model on the GPUs and answers every environment's request: the inference server. It is one chef cooking for many tables.

The desktop study ran 256 virtual computers and one inference server. The server collected screenshots from all of them and answered them in batches. The phone study ran up to 64 phone emulators at once, on 128 processor cores. A large 2025 report ran several thousand virtual machines, handed out and taken back by a manager service that answered thousands of requests a second.

Things break at that scale, so the plumbing must expect it. The phone study gave each emulator its own process, so one crash could not take down the rest. It reset an emulator whenever it hit an error it could not recover from.

More computers: faster practice, slower batches

The desktop study timed this, and the numbers are worth reading slowly. With 8 computers, one batch of episodes took 3 minutes, and a full pass through the training tasks took over 6 hours. With 256 computers, one batch took 19 minutes, but a full pass took about 1.2 hours.

So each batch got slower while the whole job got five times faster. A full pass is 128 tasks × 8 tries = 1,024 episodes. Over 6 hours that is over 21 seconds of clock time per episode. In 1.2 hours it is 4.2 seconds per episode. Both figures are our own arithmetic from the study's numbers.

The study gave two reasons. The inference server works far more efficiently on big batches: one trip through the model answers many environments at once. And the computers' slow moments overlap instead of adding up: while one waits for a page, the others keep working.

The phone study saw the same effect. Spread across machines, it collected 1.74 episodes a minute. With every emulator on one machine, it collected only 0.74.

Waiting for the slowest

There are two ways to run a batch. Think of a school bus that waits for the last child, against a taxi rank where each car leaves when it is full. Start a batch of episodes together and wait for all of them: that is synchronous. Keep every computer busy, and update whenever enough episodes are done: that is asynchronous.

The trouble with the bus is the last child. The few episodes that take far longer than the rest are the long tail, like the last shopper at closing time. A synchronous batch is exactly as slow as its slowest episode, and every other computer sits idle until it ends.

A large 2025 report named both problems. Running episodes batch by batch, in its words, gets "bottlenecked by complex long-tail problems". And it creates drift away from the current agent, which we come to next. Its answer is a pool. Training starts as soon as enough episodes are complete, and unfinished ones keep running into the next round.

Run both in the device below. Make the slowest task slower and watch what happens to each.

Waiting is simple and wasteful. The pool is busy and efficient. But look at the little tags on the pool's episodes: many were written by an older version of the agent than the one that trains on them. That is the price.

The price: stale episodes

Each update makes a new version of the policy. An episode that an older version produced is stale, like yesterday's newspaper. In the pool, an episode can start under version 3 and finish after version 5 is live.

Learning from episodes that a different policy produced is off-policy learning, like learning to drive from someone else's dashcam. The footage is useful, but it shows what they did, not what you would do.

Even a synchronous run has this problem. One big batch is usually split into smaller pieces, with one update made from each: mini-batches. Every mini-batch after the first trains on episodes from an older version. A 2025 math recipe made 16 updates from each batch of episodes. A 2025 paper on sequence ratios, which Chapter 5 returns to, split each batch into 4 mini-batches.

That same sequence-ratio paper says this plainly: the episodes come from an old policy, not the one being trained. In its words, "This also explains the necessity of the clipping mechanism". We build both halves of that sentence now.

The fix: the importance ratio

How do you learn correctly from someone else's dashcam? You reweight it. Here is the argument in five steps.

1. We want the average push the new agent would get: each action's chance under the new agent, times its push, added up.
2. But the episodes were written by the old agent, so each action shows up as often as the old agent chose it, not the new one.
3. Fix the count: multiply each action's push by how much more (or less) often the new agent would choose it. That factor is new chance ÷ old chance.
4. Old chance × (new ÷ old) × push = new chance × push, for every action. So the reweighted average over the old agent's episodes equals the new agent's average exactly.
5. For a whole episode the chance is a product over tokens, so in practice each token gets its own ratio, computed from log-probabilities: ratio = e^{(new log-probability − old log-probability)}.

The new probability divided by the old one is the importance ratio, like an exchange rate between two currencies. Try it on one screen with two possible actions, using illustrative chances. The old agent clicked with chance 0.30 and typed with chance 0.70. The new agent would click with 0.45 and type with 0.55. Clicking was pushed by +1.73 and typing by −0.58.

The new agent's average push is 0.45 × 1.73 + 0.55 × (−0.58) = 0.4595. The ratios are 0.45 ÷ 0.30 = 1.500 for clicking and 0.55 ÷ 0.70 = 0.786 for typing. Reweight the old agent's tries: 0.30 × 1.500 × 1.73 + 0.70 × 0.786 × (−0.58) = 0.4595. Exactly the same.

That has a concrete cost in the plumbing. The inference server must return each token's log-probability along with the text, and the trajectory must store it, tagged with the version. That is why Chapter 1's record had a policy_version field. Training then recomputes the same tokens' log-probabilities under the new weights and takes the difference.

ρ_t

The importance ratio of token t (rho, the Greek letter r).

π_new(o_t)

The chance the current agent gives token t.

π_old(o_t)

The chance the agent that wrote it gave it.

Why the ratio needs a cap

The ratio has a weak spot. Suppose the old agent gave a token a tiny chance, and the new one gives it a large chance. The ratio is then huge, and one stale token can dominate a whole update. So the objective caps it.

Think of a speed limiter on a car. Cap the ratio inside a band, from 1 − ε to 1 + ε: that is clipping. The band is the clip band, and ε (epsilon) sets its width.

Read the rule in words. For a token being pushed up, the gain stops growing once the ratio passes 1 + ε. For a token being pushed down, the penalty stops growing once the ratio falls below 1 − ε. Past the cap the token still counts, but its gradient is zero: this update stops moving it.

The two edges can differ. The desktop study used 0.2 below and 0.3 above. A math recipe used 0.2 and 0.28. Chapter 5 says why the top is wider.

J_t

What token t counts toward the update.

Â

Its episode's advantage.

clip(…)

The ratio, held inside the band.

ε_low

How far below 1 the band reaches.

ε_high

How far above 1 the band reaches.

ρ = 1.50, Â = +1.73, ε_high = 0.28: min(2.60, 2.21) = 2.21, and the gradient is 0

The rule always takes the smaller of the two values. That small detail does a lot. Drag the ratio below and watch where the line goes flat.

Now try a push down with the ratio at 1.50. The token grew more likely even though it was pushed down. Its plain value is 1.50 × (−0.58) = −0.87, and its capped value is 1.28 × (−0.58) = −0.74. The rule takes the smaller, −0.87, so the full penalty applies and the token still learns. The cap never softens a push down on a token that grew.

Here are this chapter's numbers in one block:

worked exampleWhere one step's time goes (one large agent)
  thinking speed            29.6 tokens per second → 1,000 ÷ 29.6 = 33.8 ms per token
  one round of thinking     4.0 s
  with lighter weights      47 tokens per second, 2.5 s per round; benchmark success 47.5% → 44.4%
  a 15-step episode         15 × 4.0 s = 60 s of thinking, before the computer's own time   (our arithmetic)

More computers (the desktop study: 128 tasks × 8 tries = 1,024 episodes per full pass)
  8 computers     batch 3 min,  full pass over 6 h   → over 6 × 3,600 ÷ 1,024 = 21 s per episode
  256 computers   batch 19 min, full pass about 1.2 h → 1.2 × 3,600 ÷ 1,024 = 4.2 s per episode
  speed-up of a full pass   6 ÷ 1.2 = 5 times, although each batch took 19 ÷ 3 = 6.3 times longer
The phone study
  1.74 episodes per minute across machines, 0.74 on one machine: 1.74 ÷ 0.74 = 2.35 times

Why reweighting works (one screen, two possible actions; chances illustrative)
  old agent: click 0.30, type 0.70        new agent: click 0.45, type 0.55
  pushes:    click +1.73, type −0.58
  the new agent's average push:       0.45 × 1.73 + 0.55 × (−0.58)       = 0.7785 − 0.3190 = 0.4595
  ratios:    click 0.45 ÷ 0.30 = 1.500           type 0.55 ÷ 0.70 = 0.786
  the old agent's tries, reweighted:  0.30 × 1.500 × 1.73 + 0.70 × 0.786 × (−0.58) = 0.7785 − 0.3190 = 0.4595

The clip, with 0.2 below and 0.28 above
  a token pushed up (+1.73) whose ratio is 1.50:
      plain  1.50 × 1.73 = 2.595      capped  1.28 × 1.73 = 2.214      take the smaller: 2.214, gradient 0
  a token pushed down (−0.58) whose chance fell from 0.50 to 0.30 (ratio 0.60):
      plain  0.60 × (−0.58) = −0.348   capped  0.80 × (−0.58) = −0.464  take the smaller: −0.464, gradient 0

And the clipped objective, per token, as code. It also returns the share of tokens the clip has taken over, which is worth logging on every update:

pythonimport numpy as np
EPS_LOW, EPS_HIGH = 0.2, 0.28

def clipped_objective(logp_new, logp_old, adv, mask):
    """All arrays (B, T). adv is the episode's advantage repeated along its tokens."""
    ratio = np.exp(logp_new - logp_old)                     # new chance ÷ old chance, token by token
    capped = np.clip(ratio, 1 - EPS_LOW, 1 + EPS_HIGH)
    per_token = np.minimum(ratio * adv, capped * adv)       # take the smaller: the cap only ever holds back
    clipped = ((adv > 0) & (ratio > 1 + EPS_HIGH)) | ((adv < 0) & (ratio < 1 - EPS_LOW))
    share_clipped = (clipped * mask).sum() / mask.sum()     # log this every update
    return (per_token * mask).sum() / mask.sum(), share_clipped   # maximize the first (minimize its negative)

Engines disagree slightly

One more wrinkle. The inference server and the training code run the exact same weights, but on different hardware paths and in a different order of operations, and floating-point arithmetic does not always give the exact same answer when the same numbers are added or multiplied in a different order. So the inference server and the training code compute slightly different probabilities for the identical model and the identical tokens, and even a fresh episode's ratio is not exactly 1. Chapter 5 shows that this per-token wobble tends to cancel out once you average it over an episode's many tokens, which is the reasoning the 2025 sequence-ratio paper uses to argue that one ratio per episode, instead of one per token, tolerates these small gaps. Then the server's own numbers can be used directly, which helps exactly in pooled, many-turn training.

The reward climbs for two days, then stops. You open one group from the latest batch. All eight episodes are the same, click for click: the same menu, the same field, the same mistake at step 6. The agent has stopped trying anything new. And as the instrument at the top showed, eight identical tries teach nothing.

A number for "trying new things"

Picture a shuffled deck of cards against a stacked one. Draw from the shuffled deck and anything can come up. Draw from the stacked deck and you know what you will get. How spread out the policy's choices are is its entropy. It is high when the agent would try many different actions, and near 0 when it always picks the same one.

Put numbers on two possible actions, by our own arithmetic. At 50/50 the entropy is 0.69, the most two actions can have. At 90/10 it is 0.33. At 99/1 it is 0.06: the deck is nearly stacked.

H

The entropy of one choice.

π(a)

The chance the agent gives action a.

∑_a

Add up over every action it could take.

90 / 10: −(0.9 ln 0.9 + 0.1 ln 0.1) = 0.095 + 0.230 = 0.33

Entropy falling to almost nothing early in training is entropy collapse, like a chess player who only ever opens with the same move. A 2025 math recipe saw it plainly: under the standard method, entropy fell fast, and the answers in some groups became nearly identical.

For a group method, that is fatal. Identical tries get identical rewards. Identical rewards mean a spread of zero, and a spread of zero means every push is zero. Collapse switches learning off.

Why the clip band causes it

Use Chapter 4's band, 0.2 on both sides. Take an action the agent picks 1 time in 100, a chance of 0.01, and push it up as hard as the band allows. Its ratio can reach 1.2 at most, so its chance can rise to 0.01 × 1.2 = 0.012 in one update.

Now take an action the agent picks 9 times in 10. The same band would let it rise to 0.9 × 1.2 = 1.08. A chance cannot pass 1, so in practice nothing holds it back.

So the band slows rare actions and barely touches common ones. The math recipe also measured which tokens hit the top of the band: mostly rare ones, with chances under 0.2 on average. Rare actions are exactly where new behaviour comes from. A band that holds them back pushes the agent toward what it already does.

Count it in the device below. How many updates does a rare action need before the agent takes it as often as a coin flip?

Clip-Higher

The fix follows from the picture: widen the top and leave the bottom alone. A band whose top is wider than its bottom is called Clip-Higher. The math recipe kept 0.2 below and raised the top to 0.28. A rare action at 0.01 then needs 16 updates, not 22, to reach a coin flip, in the best case.

It kept the bottom where it was on purpose. Widening the bottom lets updates push tokens all the way to zero, which also kills exploration: an action the agent never takes can never be found again.

Agent builders borrowed it. The desktop study used 0.2 below and 0.3 above. The large 2025 desktop report also used separate top and bottom edges, following the same recipe, though it gives no numbers for them.

The leash: KL, which means three different things

Keep a photo of where you started. A frozen copy of the agent from before practice is the reference model. A number for how different two policies' choices are is the KL divergence, after Kullback and Leibler, who defined it. Think of how far two versions of a recipe have drifted apart.

Papers use KL to keep the agent on a leash near its starting point. But "KL" means three different things in these papers, and it is worth keeping them apart.

A penalty in the loss. The original group method subtracts β × KL from its objective. Here β (beta, the Greek letter b) sets how strong the leash is, and that method used β = 0.04. It estimates KL token by token with a formula that is never negative: q − ln q − 1, where q is the reference chance divided by the current chance. At q = 0.8 that is 0.023; at q = 0.5 it is 0.193. Times 0.04, the penalties are 0.0009 and 0.0077: a gentle leash.

KL

How far the agent has drifted from the reference, for one token.

q

The reference chance divided by the current chance.

π_ref

The reference model's chance for the token.

π

The current agent's chance for it.

A target to match. The web study did something different. First, notice what β × log(new ÷ reference) measures: log(new ÷ reference) is 0 when the agent's chance for an action has not moved from the reference at all, and grows the further the new chance drifts from it, in either direction. The web study trained the agent to make that number match each action's advantage, a squared-error fit, the same kind of fit you would use to draw a line through scattered points. Match it to a big positive advantage, and the log-ratio is pulled up too, so the action is allowed to drift further from the reference. Match it to an advantage near zero, and the log-ratio is pulled back toward zero, so the action stays close to where it started. The agent still moves, in other words, but only about as far as its own results justify: a leash with some give in it, rather than the fixed-length leash of a flat penalty. It tried β from 0.01 to 10 in one round. At 0.01 the leash was too weak and performance fell. At 1 or more it hurt, unless replay was on.

No leash. The math recipe and the desktop study removed the KL term altogether. Their argument: an agent that reasons or acts over many steps is supposed to move far from where it started.

When should you keep it? The web study argued its leash kept past knowledge. Without it, a plain method "quickly overfits the current phase's data", in its words. So the choice depends on whether you want the agent to remember or to roam.

Long episodes get quiet pushes

Here is a quieter way training goes wrong. The usual average works episode by episode: average each episode's tokens first, then average the episodes. So each episode counts the same, and each token of a long episode counts less.

Take two episodes in one group: a pass with 100 agent tokens, pushed by +1, and a looping fail with 1,000 tokens, pushed by −1. Each episode weighs a half. A pass token weighs 1/2 × 1/100 = 0.005. A fail token weighs 1/2 × 1/1,000 = 0.0005: ten times quieter. A long, looping failure, the most important thing to push down, is pushed down gently.

The fix is to average over every agent token in the batch, not episode by episode: a token-level loss. Now every token weighs 1/1,100 = 0.00091, and the long fail carries 1,000/1,100 = 91% of the push. The math recipe saw the episode-by-episode average let entropy and length creep up in an unhealthy way.

Episodes cut off by the limit

An episode cut off by the step limit is truncated, like a sentence cut off mid-word. It did not fail at its task: it ran out of time. Scoring it as a plain fail punishes steps that may have been fine.

The math recipe found that masking the loss of cut-off answers steadied training. It also added a gentle penalty that grows as an answer nears the limit. Agent builders face the same choice with step limits. The large 2025 desktop report saw agents learn to exploit larger step budgets, and suggested putting the budget into the reward. The desktop study refused one shortcut outright: replacing the final action with "fail" when the limit is hit, which games the reward on tasks that really are impossible.

One ratio per token is mostly noise

Back to Chapter 4's ratio. Each token's ratio is computed from a single sample, one token drawn once. The 2025 sequence-ratio paper argues that such a ratio "fails to perform the intended distribution-correction role", and adds noise that builds up over long sequences.

Its rule is short: "the unit of optimization objective should match the unit of reward". The reward is per episode, so use one ratio per episode. Here is how to build it, in five steps.

1. The chance of a whole episode's text is the product of its token chances, so the episode's ratio is the product of its token ratios.
2. A product of hundreds of numbers near 1 can still land far from 1, and it grows with length. So take the n-th root: the geometric mean, s = (ρ₁ × ρ₂ × … × ρ_n)^1/n.
3. With logarithms, that is the plain average of the log-ratios, put back through e: s = e^{mean of log ρ}.
4. Noise in single tokens' log-ratios averages out over many tokens, so s stays close to 1 unless the agent really moved.
5. Clip s with one band per episode. Every token of the episode then shares one fate.

Two names from that list. To multiply n numbers and then take the n-th root is the geometric mean. One ratio for the whole episode, the geometric mean of its token ratios, is a sequence-level ratio: one exchange rate for the whole trip.

s_i

Episode i's one ratio.

y_i

The agent's text in episode i.

|y_i|

How many tokens it has.

40 tokens, log-ratios averaging ln 1.0002: s = 1.0002, inside 0.9997 to 1.0004

Because it is an average over many tokens, the sequence-level ratio sits very close to 1, so its band is tiny: 0.0003 below and 0.0004 above. The same paper used 0.2 and 0.27 for its per-token version. In its math and code tests, the sequence version clipped far more, about 15% of tokens against 0.13%, and still trained more efficiently. See why in the device below.

Per token, chance alone decides which tokens lose their push: the ones that happened to draw a high ratio. As one episode, the noise cancels, and the band judges only how far the agent really moved.

One more reason: experts that move

Think of a hospital that routes each patient to two specialists out of dozens. A model whose layers hold many small sub-networks, called experts, and send each token to only a few of them is a mixture of experts.

In one such model, about 10% of the experts a given sample used changed after each update. So the per-token ratios jumped around: the same token went through different experts before and after. The per-token method needed an extra trick to cope; the per-episode ratio did not. Large agents are often built this way. The large 2025 desktop agent uses one with 23 billion active weights out of 230 billion.

Which method, then?

Here is the honest state of things for agents that use a screen. The large 2025 desktop report found that PPO with a pretrained critic beat GRPO (Chapter 2). The desktop study used GRPO with Clip-Higher and no KL. Nobody has settled it.

What the three agent studies share is smaller and easier to copy. They sampled their tries at temperature 1.0. A setting that sharpens (below 1) or flattens (above 1) the agent's chances before it picks a token is its temperature, like how adventurous a cook is with a recipe. At 1.0, the agent's own randomness is left as it is, not turned down.

Here are the chapter's numbers together:

worked exampleEntropy of a choice between two actions (our arithmetic)
  50 / 50:   −(0.5 ln 0.5 + 0.5 ln 0.5)        = 0.693
  90 / 10:   −(0.9 ln 0.9 + 0.1 ln 0.1)        = 0.095 + 0.230 = 0.325
  99 / 1:    −(0.99 ln 0.99 + 0.01 ln 0.01)    = 0.010 + 0.046 = 0.056

How far one update can lift an action (the top of the band)
  band top 0.2:    0.01 × 1.2  = 0.012        0.9 × 1.2  = 1.08 (above 1: no real limit)
  band top 0.28:   0.01 × 1.28 = 0.0128
  updates to lift 0.01 to 0.5, if every update hit the top (our arithmetic):
      0.2:  ln(0.5 ÷ 0.01) ÷ ln 1.2  = 3.912 ÷ 0.182 = 21.5 → 22 updates
      0.28: ln(0.5 ÷ 0.01) ÷ ln 1.28 = 3.912 ÷ 0.247 = 15.9 → 16 updates

The KL estimator, per token (β = 0.04)
  KL ≈ q − ln q − 1, where q = reference chance ÷ current chance
  q = 0.8:   0.8 − ln 0.8 − 1 = 0.8 + 0.223 − 1  = 0.023     penalty 0.04 × 0.023 = 0.0009
  q = 0.5:   0.5 − ln 0.5 − 1 = 0.5 + 0.693 − 1  = 0.193     penalty 0.04 × 0.193 = 0.0077
  never negative: it is 0 only when q = 1

Two episodes in one group: a pass with 100 agent tokens (+1), a looping fail with 1,000 (−1)   (our arithmetic)
  episode by episode: each episode weighs 1/2
      a pass token weighs  1/2 × 1/100   = 0.005
      a fail token weighs  1/2 × 1/1,000 = 0.0005: ten times quieter
  token-level: every token weighs 1/1,100 = 0.00091; the long fail now carries 1,000/1,100 = 91% of the push

One ratio per token, or one per episode (40 tokens; the model moved very little)  (illustrative)
  per-token noise 0.15: token ratios from 0.72 to 1.34; 2 of 40 above 1.28 lose their push at random
  per episode: s = e^(mean of the 40 log-ratios) = 1.0002, inside 0.9997 to 1.0004: all 40 learn together

And the two code changes this chapter asks for. First, the two averages side by side:

python# per_token: (B, T) what each token counts; mask: (B, T) 1 for the agent's tokens
per_episode = (per_token * mask).sum(1) / mask.sum(1)       # average inside each episode first
loss_by_episode = -per_episode.mean()                       # then over episodes: long episodes' tokens count less
loss_by_token = -(per_token * mask).sum() / mask.sum()      # one average over every agent token in the batch

Second, one ratio per episode, clipped with its own tiny band:

pythonlog_ratio = (logp_new - logp_old) * mask                    # (B, T)
s = np.exp(log_ratio.sum(1) / mask.sum(1))                  # (B,) the geometric mean of each episode's token ratios
s_capped = np.clip(s, 1 - 3e-4, 1 + 4e-4)                   # the per-episode band
per_episode = np.minimum(s * adv, s_capped * adv)           # adv: (B,) one advantage per episode

Monday morning. Four practice runs went over the weekend, each on a different team's agent. All four reward curves are flat. They look alike on the dashboard. They are four different faults, and each one has a different fix.

Why a curve goes flat

Think of a fitness tracker's weekly line: one number per week, and you watch it climb or not. The average reward per update, drawn over the course of training, is the reward curve. It is the first thing everyone looks at, and the least informative thing to look at alone.

Recall the rule from the instrument at the top and from Chapter 1. If every group's rewards are equal, every push is zero. And if the rewards carry no information about what the agent did, the pushes point nowhere. Every fault below is one of those two. The trick is a cheap measurement for each, checked in the right order.

Fault 1: the computer cannot be won

Sometimes no episode can pass, whatever the agent does. The virtual machine's saved image is missing the app the tasks need. Or its reset leaves the last episode's mess behind, so every new episode starts from a broken screen. Every episode fails because no episode can pass.

The measurement is a test key that must open the lock. Script a solution yourself, step by step, and run it in the environment. An episode you scripted yourself and know solves the task is a known-good episode. Then check the final state with your own eyes, and check what the grader says.

If the known-good episode does not reach the goal, stop. No training setting can help. You will also see the signature in the logs: rewards are exactly 0, and every group's spread is 0. The phone study's infrastructure shows the everyday version of this fault: emulators that crash and must be reset.

Fault 2: the grader is unreliable

Now suppose the computer can be won, but the grader is often wrong. How would you know? Mark the teacher's marking. People mark 100 finished episodes pass or fail by hand: the hand labels. The share on which the grader says the same is its agreement.

Why does a bad grader flatten the curve? Work it out in five steps. This is our own arithmetic.

1. Let t be the share of episodes that truly succeed, and q the chance the grader gets any one episode wrong.
2. A true success is scored 1 with chance 1 − q. A true failure is scored 1, wrongly, with chance q.
3. So the measured reward is t × (1 − q) + (1 − t) × q.
4. Rearranged: measured = q + (1 − 2q) × t.
5. The real improvement is multiplied by 1 − 2q. At q = 0.44 that factor is 0.12: an eighth of the improvement survives, buried under the noise.

measured

The reward the curve shows.

q

The chance the grader gets one episode wrong.

t

The share of episodes that truly succeed.

q = 0.44: 0.44 + 0.12 × 0.20 = 0.464, and 0.44 + 0.12 × 0.40 = 0.488

Read that line slowly. The agent's true success doubled, from 20% to 40%. The curve moved from 0.464 to 0.488: a rise of 0.024 that disappears in the wobble of any real run. A grader that agrees with people 56% of the time is barely better than a coin, and its curve looks flat even while the agent improves.

For contrast, here are healthy graders. The phone study's judge disagreed with people only 2.8% of the time on average, so its factor is 1 − 2 × 0.028 = 0.944, by our arithmetic. Almost all of an improvement shows. The web study trained its own judge, which agreed with hand labels on 79.4% of 100 episodes and on 80.8% of its test set. That beat every large general model it was compared with. Its factor, assuming its errors fall equally on passes and fails, is about 0.59.

The fix is a better grader. Check the final state directly where you can: the file exists, the setting changed. Give a judge the task's reference answer. Show it more of the episode. The large 2025 desktop report's judge read the full text history plus the last five screenshots.

Fault 3: the tasks are out of reach

Here everything works: the environment can be won, the grader is right, and the agent still tries different things. But most groups fail on every try. This is Chapter 3's arithmetic. With a pass chance of 0.01 and 8 tries, 92.3% of groups are all-fail, by our arithmetic.

The measurement is the share of all-equal groups, split into all-fail and all-pass. The desktop study plotted the spread of rewards inside groups and saw it stay low on the full task set. The fix is Chapter 3: choose tasks, drop useless groups, replay successes, and write new tasks at the edge.

Fault 4: the tries have collapsed

This one has a different shape. The reward rose, then stopped. Entropy fell toward zero. And the share of all-equal groups climbs as entropy falls, because eight copies of the same try get the same reward.

The measurement is entropy per token, and the number of different first actions in each group. The fix is Chapter 5: Clip-Higher, temperature 1.0, a token-level loss.

Four runs, four faults. Reveal one measurement at a time in the device below, and see which one singles out each run.

More instruments worth logging

A few more numbers earn a place on the dashboard, one sentence each. If you use a critic, log its explained variance, as Chapter 2 showed. Log the share of clipped tokens and the age of each episode, as Chapter 4 showed. And log the number of steps per episode: in the large 2025 desktop report, steps per task fell while reward rose, a healthy sign that the agent was getting more direct.

Reward hacking

A flat curve is not the only symptom to fear. A curve that climbs for the wrong reason is worse, because it looks like success. Think of a student who learns the marker, not the subject. Getting reward without doing the task is reward hacking.

The desktop study gives a clean example. When an episode hits the step limit, a harness can swap the agent's final action for "fail". On a task that really is impossible, "fail" is the right answer, so the swap earns reward without any skill. The study measured with that swap forbidden. The rule that follows: read episodes, not just curves.

The order matters

Check the four faults in this order, cheapest and most total first:

Collapse must be checked before out-of-reach, because a collapsed agent also produces all-equal groups. If you look at the all-equal share first, a collapsed run looks like a run whose tasks are too hard, and you would fix the wrong thing.

The same order, as code. The thresholds are our own rules of thumb, not numbers from a paper:

pythondef diagnose(run):
    if run.known_good_score < 1.0:          # 1. can the computer be won at all?
        return "environment"
    if run.grader_agreement < 0.70:         # 2. does the grader agree with people?
        return "grader"
    if run.entropy_end < 0.2 * run.entropy_start:   # 3. have the tries collapsed? (check before 4)
        return "collapse"
    if run.all_equal_share_end > 0.80:      # 4. are the tasks out of reach?
        return "out of reach"
    return "healthy"

And the chapter's numbers in one block:

worked exampleA grader that agrees with people 56% of the time: q = 1 − 0.56 = 0.44          (our arithmetic)
  measured = 0.44 + (1 − 2 × 0.44) × t = 0.44 + 0.12 × t
  true 20% → 0.44 + 0.12 × 0.20 = 0.464
  true 40% → 0.44 + 0.12 × 0.40 = 0.488       a doubling shows up as +0.024

The phone study's judge: 2.8% disagreement, q = 0.028
  factor 1 − 2 × 0.028 = 0.944:  true 20% → 0.217,  true 40% → 0.406

All-equal groups when tasks are out of reach (G = 8)                            (our arithmetic)
  p = 0.01:  0.998 + 0.018 = 0.923 → 92% of groups teach nothing

The four runs at the end of training (illustrative)
  run   reward   known-good   agreement   all-equal groups   entropy
  A     0.00     0            93%         100%               1.10
  B     0.46     1            56%         35%                1.10
  C     0.03     1            93%         92%                1.10
  D     0.38     1            93%         90%                0.03

Two sentences describe the same result. One says practice raised the agent's score on a desktop benchmark from 43.0% to 47.5%. The other says it improved by nearly 10.5%. Both are true. Which one tells you more?

Points and percent

A score in percent can change in two ways, and both get called "percent". The plain difference between two percentages is measured in points. That difference divided by the starting value is the relative gain.

For the desktop result: 47.5 − 43.0 = 4.5 points. And 4.5 ÷ 43.0 = 10.5% relative. The "nearly 10.5%" was relative; the honest headline is 4.5 points.

The same report described another gain, from 64.6% to 73.3% on a phone benchmark, as "over 8.7%". That one is in points: 73.3 − 64.6 = 8.7. As a relative gain it is 8.7 ÷ 64.6 = 13.5%. One report, two gains, two different meanings of "percent".

So here is the rule. Always ask which one a number is, and prefer points. Points can be added and compared across results; relative gains grow large whenever the starting score is small.

One task's worth

Tasks held back from practice and used only to measure are the test set, like an exam the student never saw. A score is a count of tasks passed, divided by the number of tasks in the test set.

So one task is worth 100 ÷ n points, where n is the number of test tasks. On a 96-task test set, one task is worth 1.04 points. On 165 tasks it is 0.61, and on 369 tasks it is 0.27. A gain of 1 point means about one task on the first and almost four on the last.

Read the phone study through that lens. Its 17.7% is about 17 of 96 tasks. Its 67.2% is about 64 or 65 of them: the figure is the average of two training runs, so it need not be a whole number of tasks.

Counting noise

A poll of 1,000 people comes with a margin of error, because another 1,000 people would answer a little differently. A score has the same kind of wobble. How much a score would wobble if you measured on another set of tasks of the same size is its standard error. We will call it the counting noise. Derive it in four steps; the arithmetic is ours.

1. Each task is a coin that lands pass with chance p. One flip's result, 1 or 0, has a spread of √(p (1 − p)) around its average p: the same formula the group of eight used back in Chapter 1, now applied to a single coin.
2. Flip n of these coins, independently, and add up the results. Independent wobbles do not point the same way every time, so they do not add directly, but their squares do: n independent coins give a summed squared-spread of n × p (1 − p). Taking the square root, the total number of passes across n tasks has a spread of √(n p (1 − p)) passes.
3. The score is passes ÷ n, so dividing that spread by n gives √(n p (1 − p)) ÷ n = √(p (1 − p) ÷ n).
4. Multiply by 100 for points. Halving the noise takes four times as many tasks.

At 67.2% on 96 tasks: √(0.672 × 0.328 ÷ 96) = 0.048, about ±4.8 points. At 29.9% on 369 tasks: about ±2.4. So on a 96-task test set, a 3-point gain is smaller than the counting noise of one score. On 369 tasks it is a little larger.

This is a different noise from the "±" a paper prints after repeating training. The phone study's 67.2 ± 1.5 is the spread over two training runs on the same 96 tasks. It says how much training itself wobbles, not how much the choice of tasks does. Both matter. The full story of error bars is in The Statistics of Evaluation.

Same rules, or no comparison

Three traps make the same result look bigger or smaller than it is.

Step limits. The desktop study measured every agent with the same 15-step limit. A score measured at a larger limit is not comparable, because a longer clock gives the agent more chances to recover.

Scoring rules. The same study also reported a stricter version, which forbids swapping the last action for "fail" at the limit (Chapter 6). Its numbers are lower: 23.8% against 29.9% for the same agent. A comparison must use the same rule on both sides.

Practiced-on tasks. Tasks the agent practiced on are in-domain. Held-out tasks it never saw are out-of-domain, like homework problems against exam problems. On 32 tasks it practiced on, plain group scoring raised the desktop study's starting agent from 43.8% to 68.8%, and the replay version to 81.25%. On 96 held-out tasks, plain group scoring slipped from 55.2% to 52.08%, and replay reached 56.3%. Practice can memorize.

The results, now that you can read them

Here are the published results, with each benchmark named once. The phone study measured on Android in the Wild's shopping and general tasks, 96 test tasks each. The web study measured on WebArena-Lite, 165 test tasks across five sites. The desktop study measured on OSWorld, 369 tasks. The large 2025 report measured on OSWorld, on AndroidWorld (116 tasks) and on Online-Mind2Web (300 tasks).

Read it in groups. The phone rows are the biggest gains on the page: +49.5 points on shopping and +57.3 on general tasks. They come from a small model, and the copied and prompted models stayed far below. With 96 tasks, the counting noise is about ±4 to ±5 points, so these gains are many times larger than the noise.

The website rows show the same pattern at 8 and 70 billion weights: practice roughly doubles the copied model's score. The earlier phone-study method, run on websites, reached 30.3%, well short of the web study's own 42.4%.

The desktop rows are smaller and more honest about it: +2.5 points for plain group practice and +6.4 for replay, on 369 tasks, with the stricter scoring telling the same story. The large agent's rows are gains on top of an already strong agent: +4.5, +8.7 and +4.5 points.

Read any of them three ways in the device below.

Practice against offline shortcuts

There is a cheaper alternative to practice: collect episodes once, then learn from that fixed data without any new tries. The desktop study compared both on the same collected episodes. Replay practice reached 27.3% and plain group practice 26.0%. Two methods that learn from preferences between fixed episodes reached 24.6% and 22.4%, and copying only the successful episodes reached 21.8%.

The ordering suggests that new tries help: both practice methods come first. But read it with the counting noise. Assuming the same 369-task set this study's main results use elsewhere, the noise at that size is about ±2.3 points by our arithmetic, so neighbours such as 27.3 and 26.0 blur together. The gap from the top to the bottom, 5.5 points, is the part worth trusting.

Who measured it

One last check: who ran the measurement? Agents from large vendors appear in the large 2025 report's own comparison table. Those numbers were measured by that report's authors, not by the vendors. The Field Guide lists them with that label, and this chapter leaves them out.

Here are the chapter's numbers in one block:

worked examplePoints and percent
  desktop: 47.5 − 43.0 = 4.5 points;   4.5 ÷ 43.0 = 10.5% relative
  phone:   73.3 − 64.6 = 8.7 points;   8.7 ÷ 64.6 = 13.5% relative

One task's worth = 100 ÷ n
  96 tasks: 1.04 points    116 tasks: 0.86    165 tasks: 0.61    300 tasks: 0.33    369 tasks: 0.27

Counting noise = √(p (1 − p) ÷ n), in points                                   (our arithmetic)
  67.2% on 96:    √(0.672 × 0.328 ÷ 96)  = √0.002296 = 0.048 → ±4.8
  42.4% on 165:   √(0.424 × 0.576 ÷ 165) = √0.001480 = 0.038 → ±3.8
  29.9% on 369:   √(0.299 × 0.701 ÷ 369) = √0.000568 = 0.024 → ±2.4

In tasks
  phone shopping, 96 tasks:  17.7% ≈ 17 tasks → 67.2% ≈ 64.5 tasks (two runs averaged): +47.5 tasks
  websites, 165 tasks:       20.6% = 34 tasks → 42.4% = 70 tasks: +36 tasks
  desktop, 369 tasks:        23.5% ≈ 87 tasks → 29.9% ≈ 110 tasks: about +24 tasks

Practice teaches an agent that uses a computer from its own episodes, scored only pass or fail at the end. Run each task several times and score each episode against its own group. Push every word the agent wrote by that one number. A critic can share the credit more finely. Keep practice where the agent sometimes wins. Run many computers at once, and correct for slightly old episodes with a capped ratio. Keep the tries different enough to compare. And measure before you tune.

The checklist

Run it in order the first time you set up practice for a new agent, and again whenever the tasks, the grader or the fleet change.

1. Run a known-good episode through the environment and the grader. It must score 1 before any training starts.
2. Hand-label 100 finished episodes. Measure the grader's agreement; aim far above a coin flip.
3. Run the starting agent several times on every task. Keep the tasks it solves at least once.
4. Pick tries per group (8 is a common choice) and log the share of all-equal groups every update.
5. Score inside each group, add 0.000001 to the spread, and drop or rescue all-equal groups.
6. Mask the loss to the agent's own tokens. Screenshots get no push.
7. Store each token's log-probability and the agent's version with every trajectory.
8. Clip the ratio with a wider top than bottom. Log the clipped share and every episode's age.
9. Sample at temperature 1.0. Watch entropy and steps per episode, not only reward.
10. Measure on held-out tasks at a fixed step limit. Report points, the task count and the counting noise.

The numbers, and where each one comes from

Every number here was checked against the paper it comes from. The ids in the last column point at our fact sheet for this lesson.

The core loop as code

Here is the whole lesson as one sketch. It runs on your own objects: parse_action, server, env, policy, pool and tasks are yours to supply. Real systems add sharding, retries and logging; the logic is all here.

python# agentic_rl.py: the core loop of group-relative practice for a computer-use agent (a sketch).
# Real systems add sharding, retries and logging; the logic is all here.
import numpy as np

G = 8                        # episodes per task in one group
MAX_STEPS = 15               # step limit per episode
EPS_LOW, EPS_HIGH = 0.2, 0.28
MAX_LAG = 2                  # drop episodes written more than 2 updates ago (our choice)

def run_episode(server, env, task):
    """Screenshot in, one action out, until the agent stops or hits the limit."""
    obs = env.reset(task)                          # a fresh virtual machine at the task's start
    ep = {"task": task.id, "version": server.version, "steps": [], "reward": 0.0}
    for _ in range(MAX_STEPS):
        out = server.act(task.instruction, ep["steps"], obs)   # thought + action text, with log-probs
        ep["steps"].append({"screen": obs.ref, "text": out.text, "logp_old": out.token_logps})
        action = parse_action(out.text)            # e.g. click(start_box='(863,149)')
        if action is None:                         # unreadable: a format penalty, and stop
            ep["reward"] = -1.0
            return ep
        if action.kind in ("FINISH", "FAIL"):
            break
        obs = env.step(action)                     # seconds: the slow part of every step
    ep["reward"] = 1.0 if task.check(env) else 0.0 # the checker reads the machine's final state
    return ep

def group_advantages(group, replay):
    r = np.array([ep["reward"] for ep in group])
    if r.max() <= 0 and replay.get(group[0]["task"]):           # all failed: swap in a saved success
        group[0] = replay[group[0]["task"]]
        r[0] = 1.0
    if r.std() == 0:                               # all equal: nothing to compare, skip the group
        return None
    for ep in group:
        if ep["reward"] == 1.0:
            replay[ep["task"]] = ep                # remember the latest success for this task
    return (r - r.mean()) / (r.std() + 1e-6)       # one number per episode

def loss(policy, batch):
    """Clipped objective over every token the agent wrote; screenshots carry no loss."""
    total, n_tokens = 0.0, 0
    for ep, adv in batch:
        for step in ep["steps"]:
            logp_new = policy.token_logps(step)    # same tokens, current weights
            ratio = np.exp(logp_new - step["logp_old"])
            clipped = np.clip(ratio, 1 - EPS_LOW, 1 + EPS_HIGH)
            total += np.minimum(ratio * adv, clipped * adv).sum()
            n_tokens += len(ratio)
    return -total / max(n_tokens, 1)               # token-level average, then minimize

def train(server, policy, pool, tasks, replay, updates):
    for _ in range(updates):
        batch = []
        while len(batch) < 32:                     # 32 episodes = 4 groups of 8
            task = tasks.sample()
            group = pool.collect(task, G)          # filled by many environments in parallel
            group = [ep for ep in group if server.version - ep["version"] <= MAX_LAG]
            if len(group) < G:
                continue                           # too stale: run this task again
            adv = group_advantages(group, replay)
            if adv is not None:
                batch += list(zip(group, adv))
        policy.step(loss(policy, batch))           # one gradient update
        server.load(policy)                        # environments keep running; new episodes use it

One episode. run_episode is Chapters 0 and 1. A fresh virtual machine shows the task's first screen. The agent writes a thought and an action, and the server returns the log-probability of every token it wrote. The trajectory keeps the screen, the text and those log-probabilities, tagged with the agent's version. An unreadable action ends the episode with the format penalty; otherwise the checker reads the final state.

One group. group_advantages is Chapters 1 and 3. If all eight failed and a saved success exists, one is swapped in. A group whose rewards are still all equal is skipped. Every success is remembered for later. The rest get one number each: reward minus the average, divided by the spread.

The loss. loss is Chapters 4 and 5. Each token's ratio compares the current weights with the version that wrote it. The clip band holds stale tokens back. The average runs over every agent token in the batch, so long failures are pushed down properly. Screenshots never enter it.

The training loop. train is Chapter 4's pool. Many environments fill it in parallel. Episodes more than MAX_LAG = 2 versions old are dropped; that cap is our own design choice for the sketch, not a number from these papers. After each update the server loads the new weights, and the environments keep running.

Where this lesson sits

This lesson sits in Act 8, training, of the path Computer Use and Multimodal Agents. Just before it: RL for Robot Foundation Models (VLAs), the same trial-and-error idea for robot policies. Next in the path, coming soon: a full walkthrough of the DigiRL paper, then of the UI-TARS-2 report. After those: Self-Play RL I: Absolute Zero, where a model writes its own practice tasks. Also coming in this path: GUI grounding (how a model finds the button), the computer-use agent loop, agent environments and the OSWorld benchmark, sandbox fleets, trajectory data, and prompt injection.

Keep going

• Policy Gradients: the full derivation of "push up what worked".
• Alignment II: RLVR and GRPO: group scoring for language models with checkable answers.
• RL for LLMs: GRPO, DAPO and R1-Zero: every stability fix of Chapter 5, in its home setting.
• DAPO: the math recipe behind Clip-Higher and dynamic sampling, paper by paper.
• PPO: the clipped objective and the critic, from the original paper.
• Exploration in Reinforcement Learning: what to do when an agent never finds a reward at all.
• TRACE: building practice environments around an agent's weak spots.
• The Statistics of Evaluation: counting noise and error bars in full.

Sources

• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. UC Berkeley, UIUC, Google DeepMind, 2024. arXiv 2406.11896 (preprint). Code: github.com/DigiRL-agent/digirl.
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. Tsinghua, Zhipu AI. ICLR 2025. arXiv 2411.02337. Code: github.com/THUDM/WebRL.
• ARPO: End-to-End Policy Optimization for GUI Agents with Experience Replay. CUHK, SmartMore, HKUST, 2025. arXiv 2505.16282 (preprint). Code: github.com/dvlab-research/ARPO.
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning. ByteDance Seed, 2025. arXiv 2509.02544.
• Group Sequence Policy Optimization (GSPO). Qwen Team, Alibaba, 2025. arXiv 2507.18071.
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale. ByteDance Seed, Tsinghua AIR, HKU, 2025. arXiv 2503.14476.
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). DeepSeek-AI, 2024. arXiv 2402.03300.

The large 2025 report's own comparison table also lists vendor agents (for example Claude-4-Sonnet at 43.9% and OpenAI CUA-o3 at 42.9% on OSWorld). Those were measured by that report's authors, not by the vendors.

Go back to the eight tries at the top. Set the task to 1 in 10 and press Try 8 more times until a group fails on every try. Now you know why nothing moves, and three ways to fix it. Then explain it out loud with Present, or test yourself with Teach.