UI-TARS-2

Chapter 0

Copying Is Not Enough

Watch an AI use a computer for many steps, and see why it must practise as well as study.

Someone types one sentence to an AI: “Play the Hex game, try to get more score.” The AI has never seen the rules of Hex. So it opens a web browser, searches for them, and reads two pages. Then it switches to the game and places pieces, one click at a time. Later the person types again: “Great! Now build a web page about your playing experience.” The AI opens a window for typing commands and writes the page as code.

Nothing in that story is a chat. The AI did not answer in words. It used a computer, the same way you would, for many minutes. That is a different job from answering questions, and this lesson is about how such an AI is trained to do it well.

An AI with hands

A program that works toward a goal by acting on the world is an agent. Think of an assistant who has hands as well as a voice. Here the agent is an AI model, and its world is a computer.

It works the computer the way you do. It uses the windows, buttons and menus you click, which engineers call the graphical user interface, or GUI. So it is a GUI agent. It has no secret back door into the apps. If a person cannot click it, the agent cannot click it either.

Here is how it works, one move at a time. The agent sees a screenshot, a picture of the whole screen. It answers with an action: one click, one typed word, one key press. The computer changes, and the agent sees the next screenshot.

Look, think, act: that cycle is one step, like one move in a board game. One try at a task, from the first screenshot to the last action, is an attempt, like one round of golf. And the job written in plain words, plus a computer set up for it, is a task.

look

a screenshot of the whole screen

↓

think

a short note on what to do next

↓

act

one click, one typed word or one key press

↓

the computer changes

a new screenshot

↻ next step

One step. An attempt repeats it until the agent says the task is done, or until it runs out of steps.

Three kinds of screen in one attempt

The Hex attempt used three kinds of screen. It read web pages in a browser. It clicked pieces into place in the game. At the end it typed commands into a window where you type commands instead of clicking: a terminal. The report behind this lesson calls that mix a hybrid: clicking, plus the files and the terminal of the same computer.

Count the moves and the attempt is long: dozens of steps. Each step is one turn of a back and forth: the agent acts, and the computer answers. An attempt made of many steps, each one a turn of the back and forth between agent and computer, is multi-turn. It is a conversation, not a single question.

Here is the hard part. Nothing tells the agent how it is doing along the way. No teacher says “good click” at step 12. The only judgement comes after the last step: did the attempt do the job or not? Step through one long attempt and feel how far apart the actions and that judgement are.

Follow one long attempt

One attempt at the Hex task, slowed down. The big panel is what the agent sees; under it are the agent's thought and its action. The row of dots is every step of the attempt. Press Play, or step through it one action at a time.

The task and the switch to building a web page come from the report's Figure 1. The 34 steps, the thoughts and the click positions are made up to show the idea. Clicks are written in the action format of the released UI-TARS-1.5 prompt; the terminal commands use a made-up format, since the report does not print its own. The 4 seconds per step is the time the report measured per step before it compressed the model (Chapter 10).

Thirty-four steps, two minutes and sixteen seconds of computer time, and one grade at the very end. Hold on to that shape. Almost everything hard about training this agent comes from it.

The first way to learn: studying

How do you teach an AI to do this at all? The obvious way is to show it. Record a person doing a task: every screen and every click. A recording of a person doing a task, every screen and every click, is a demonstration.

Now ask the person to say why they made each click, and write it next to the click. Recordings of tasks done right, step by step with the reasoning, are worked examples. Think of a maths textbook with the solutions printed. Training the model to reproduce them, step by step, is studying worked examples.

Studying works, up to a point. The agent in this lesson finished 64.6 of every 100 tasks on a standard set of phone tasks after studying. That is good. It is also roughly where studying stops helping much, for two reasons.

Why studying runs out

The first reason is cost. Recordings of people using computers, with their reasoning written out, are slow and costly to make. So there are few of them, far fewer than the pages of text a language model reads. The report lists this as the first of its four problems: not enough data.

The second reason goes deeper. A worked example shows the right path. It never shows what happens after a wrong step, because the person did not take one. So an agent that only studies has never seen the screen that follows its own mistake. Its first wrong click puts it somewhere no recording went, and it has nothing to go on.

The second way to learn: practice

Think about how people learn to drive. Reading the manual and watching an instructor is studying. It helps. But you really learn to drive by driving yourself, while an examiner watches and marks you pass or fail at the end.

An agent can learn the same way. Let the agent try tasks itself, grade each attempt, and make what worked more likely. That is practice, and its formal name is reinforcement learning, or RL. The grade an attempt gets, often just 1 for pass and 0 for fail, is its reward.

Practice fixes both problems of studying. It makes its own data: every attempt is a new recording, and no person has to make it. And every attempt follows the agent's own path, mistakes included. The agent sees the screens after its wrong clicks, and the reward tells it which paths ended well.

Here is the kind of gap practice fills. Picture an agent that has studied thousands of recordings of people saving contacts on a phone. On a real phone, a permission box pops up that no recording ever showed. The agent taps where the Save button was in every recording. The tap lands on the box's edge, nothing happens, and the agent taps again, and again, until it runs out of steps.

How would you find a failure like that? Read the agent's failed attempts one by one, and mark the step where each one went wrong. In this picture, that step is often a screen that no worked example contained: a box, a menu or a page the recordings never showed.

What would you change? More recordings help only if they happen to show that box. Instead, let the agent practise: try such tasks itself, many times, on screens like these, and learn from the reward at the end of each attempt.

That pop-up agent is made up, to show the shape of the problem. The report does not say why its own agent's failed attempts failed. It does report what practice bought. Its agent finished 64.6 of every 100 phone tasks after studying, and 73.3 after practice.

What practice bought

To compare agents fairly, people give them all the same test. A fixed list of test tasks, each with an automatic check, used to compare agents, is a benchmark. It is like a driving test route that every learner drives. Its score is the share of tasks finished, out of 100.

So “73.3 on the phone benchmark” means this: of every 100 phone tasks, the agent finished 73.3, as judged by each task's own check. A rise from 64.6 to 73.3 is 8.7 points: the score went up by 8.7 on the scale of 100.

The report gives three such before and after pairs. On phone tasks the score rose from 64.6 to 73.3. On desktop tasks it rose from 43.0 to 47.5. On live websites it rose from 83.7 to 88.2. Here is the arithmetic, together with the price of one attempt at the Hex task in computer time.

worked exampleOne attempt at the Hex task (steps illustrative, seconds per step real)
  steps            = 6 in the browser + 22 in the game + 6 in the terminal = 34
  computer time    = 34 × 4 s = 136 s, about 2 minutes 16 seconds
  rewards received = 1, at the very end

What practice added to studying (after studying → after practice)
  phone tasks      64.6 → 73.3    73.3 − 64.6 = 8.7 points
  desktop tasks    43.0 → 47.5    47.5 − 43.0 = 4.5 points
  live websites    83.7 → 88.2    88.2 − 83.7 = 4.5 points

Notice one more thing, because Chapter 9 comes back to it. Most of that practice was on websites. The phone and desktop scores rose anyway.

Studying shows the right path. Practice shows where your own steps lead. An agent that only studies has never seen the screen after its own wrong click. Practice puts it there, thousands of times, and grades how each attempt ended.

Why practice on computers is hard

If practice works so well, why is it not the whole story? Because practice on real computers is hard in four ways. The report names all four, and it answers each one with one piece of its system.

The problem	The report's piece	Chapter
Worked examples are scarce and costly	a data flywheel that turns every attempt into training data	3
One reward arrives after hundreds of steps, and training is unstable	a steadier version of the standard practice method, with tricks for long attempts	5, 7
Clicking alone is clumsy for files and code	one sandbox where the browser, the terminal and tools share one disk	2
Practice needs thousands of computers that crash, stall and must be reset	one platform that runs them all, and a trainer that never waits for the slowest	2, 6

The agent in this lesson is UI-TARS-2. It starts from a large model that its makers had already trained on huge amounts of data, and it adds those four pieces. A technical report describes how, and this lesson walks through it piece by piece. Each chapter explains one part of the instrument at the top of the page.

Look, think, act: what the agent sees, what it writes, and how text becomes a click.
A thousand computers to practise on: sealed practice computers that share one disk and get reset after a crash.
The data flywheel: every finished attempt becomes training data.
Tasks and graders: practice tasks hard enough to teach, and automatic pass or fail.
Spreading one reward: which of thousands of words earned the pass.
Never wait for the slowest: keeping every computer busy.
Small, safe updates: changing the agent a little at a time.
One agent from many: blending specialists into one.
What practice bought: the scores, read the right way.
Inside the training run: the curves, and a speed trade.
The map: numbers, checklist, code, and what comes next.

An agent has studied thousands of worked examples and finishes 64.6 of every 100 phone tasks. What does practice give it that more studying cannot?

More recordings of people doing the same phone tasks, each with their reasoning written out beside the clicks. Screens that follow its own actions, mistakes included, and a grade for how each attempt ended. A way to skip the grader, because the agent can see on the screen for itself whether it did well. A shortcut past long attempts, since it learns only from the one-step tasks, which are cheaper to run.

Chapter 1

Look, Think, Act

Follow one step from a screenshot to a click, and see what the agent keeps in mind between steps.

Freeze the Hex attempt at step 4. A page of search results fills the screen, and the agent wants to open the club page that explains the rules. It cannot move a mouse. It can only write text. So how does a sentence become a click in exactly the right place?

This chapter follows that one step from start to finish: what the agent receives, what it writes, and how a separate program turns its words into a click. Then it looks at what the agent keeps in mind from one step to the next.

What arrives: a grid of dots

A screen is a grid of tiny coloured dots, pixels, like the tiles of a mosaic. A common laptop screen is 1920 pixels across and 1080 down. That is 1920 × 1080 = 2,073,600 pixels. Each pixel is three numbers, how much red, green and blue it holds, so one screenshot is 6,220,800 numbers.

A language model does not read pixels. Language models read and write text in small pieces, a word or part of a word each, called tokens. So before the model can look at a screenshot, something has to turn six million numbers into something token-shaped.

How a model reads a picture

Picture a translator on the phone, describing a photo to a friend who cannot see it. The friend never sees the photo, only the description. A model that reads screens works the same way.

It is a language model with a second part in front of it, a vision encoder, that turns a picture into something the language part can read. Together they are a vision-language model. The vision encoder cuts the picture into small squares, patches, like cutting a photo into stamps. It turns each patch into a list of numbers that the language part reads like extra words.

Both parts are made of parameters: the adjustable numbers inside a model, set during training, that hold everything it knows. Think of the knobs on a mixing desk. Training turns the knobs; a trained model is one particular setting of all of them.

UI-TARS-2 is big. Its vision encoder has 532 million parameters. Its language part has 230 billion, organised as a Mixture-of-Experts: a model split into many smaller expert networks, where each token uses only a few. A hospital works like this: each patient sees two specialists, not every doctor in the building. So only 23 billion parameters work on each token, 10% of the whole. The model starts from Seed1.6, a model its makers had already trained, and Chapter 3 explains what that earlier training was.

Resizing to whole squares

The vision encoder needs a picture whose sides are a whole number of squares. A 1920-pixel side is not. So the code first resizes the screenshot, stretching it a little so each side is a whole number of squares.

In the released UI-TARS-1.5 code the sides become multiples of 28 pixels, and the total stays between 78,400 and 12,845,056 pixels. A 1920 × 1080 screenshot becomes 1932 × 1092: a grid of 69 × 39 = 2,691 squares. The report does not say what size UI-TARS-2 itself uses. We use the released UI-TARS-1.5 numbers because they are public.

pythondef snap(w, h, f=28):                    # the released UI-TARS-1.5 rule, for images inside the budget
    return round(w / f) * f, round(h / f) * f
print(snap(1920, 1080), snap(1080, 2400), snap(1280, 720))
# (1932, 1092) (1092, 2408) (1288, 728)

The released code also shrinks an image that is bigger than the budget and grows one that is smaller; inside the budget it only snaps. A phone screenshot of 1080 × 2400 becomes 1092 × 2408.

What the model writes

Now the model has the screenshot, the task and its notes so far. Here is everything it writes back at step 4, exactly as text:

model outputThought: The club page explains the rules. I'll open it.
Action: click(point='<point>197 525</point>')

Read it line by line. The first line is the reasoning, in plain language. Before each action the model writes a short thought: what it sees, and what it means to do about it.

The second line is the action: a function name and its arguments, written as text. Here the function is click and its one argument is a point, two numbers between point tags. The model does not run this function. It only writes it down.

The released UI-TARS-1.5 prompt gives the model a fixed menu of such functions. The report does not print UI-TARS-2's own menu, so this is the published one:

Action	What it does
`click(point='<point>x y</point>')`	one left click at a point
`left_double(...)`	a double click
`right_single(...)`	a right click
`drag(start_point=..., end_point=...)`	press, move and release
`hotkey(key='ctrl c')`	a key combination: split keys with a space, at most 3 keys
`type(content='xxx')`	type some text
`scroll(point=..., direction='down')`	scroll down, up, right or left at a point
`wait()`	sleeps 5 seconds, then takes a new screenshot
`finished(content='xxx')`	the task is done, with a short report
`long_press(...)`, `open_app(app_name='')`	phones only: press and hold; open an app by name
`press_home()`, `press_back()`	phones only: the Home and Back buttons

From text to a click

Look at a city laid out on a grid. Any house has an address: so many blocks east, so many blocks south. A position on the screen works the same way. Written as two numbers, pixels from the left edge and pixels from the top, it is the position's coordinates.

Here is the catch. The numbers 197 and 525 are coordinates on the 1932 × 1092 image the model saw, not on the real screen. Something has to translate them.

That something is the harness: the program that drives the computer for the agent. It takes the screenshots, sends them to the model, and carries out the actions it writes. Think of the reins between a rider and a horse. The model decides; the harness pulls.

The translation is a proportion. Here is the derivation, one step at a time:

The code snaps each side to the nearest multiple of 28: new side = round(side ÷ 28) × 28, for images inside the pixel budget.
The model answers in image coordinates, (x_img, y_img).
A point on the screen sits at the same fraction of the width as on the image: x_screen ÷ W = x_img ÷ w_img. So x_screen = x_img × W ÷ w_img, and the code keeps the whole-number part.
If the harness skips step 3, it clicks at (x_img, y_img) on the screen. The miss is x_img − x_screen = x_screen × (w_img ÷ W − 1). It grows with the target's distance from the top-left corner, and with how much the image was resized.

For step 4: 197 ÷ 1932 × 1920 = 195.77, kept as 195, and 525 ÷ 1092 × 1080 = 519.23, kept as 519. The harness clicks at (195, 519). In code, reading the model's two lines and mapping the point back takes six lines:

pythonimport re
SCREEN_W, SCREEN_H = 1920, 1080          # the real screen
IMG_W, IMG_H = 1932, 1092                # the image the model saw (sides snapped to multiples of 28)

out = """Thought: The club page explains the rules. I'll open it.
Action: click(point='<point>197 525</point>')"""

thought = out.split("Action:")[0].removeprefix("Thought:").strip()
x, y = map(int, re.search(r"<point>(\d+) (\d+)</point>", out).groups())
sx, sy = int(x / IMG_W * SCREEN_W), int(y / IMG_H * SCREEN_H)   # map back to the screen
print(thought, (x, y), "->", (sx, sy))   # The club page explains the rules. I'll open it. (197, 525) -> (195, 519)

With the snap, skipping the mapping costs almost nothing: (197, 525) is 2 pixels right and 6 low, still on the link. But a harness is free to send the model a smaller image, to save time. Then the mapping matters a great deal. Choose a size and switch the mapping off.

From screenshot to click

Left: the image the model saw, with the point it wrote. Right: the real screen, with the click the harness made. Choose how the harness sized the image, then switch the mapping off.

The 28-pixel snap, the pixel budget and the (197, 525) example come from the released UI-TARS-1.5 code and its README. The report does not say what image size UI-TARS-2 sees, so the three shrink options are a made-up harness, there to show what a skipped mapping costs. The page is a toy.

Here is how that bug looks from the outside. A team wraps the agent in its own harness. To save time, the harness halves every screenshot before sending it. In testing, the agent's thoughts are right every time (“open the club page”), but its clicks land in empty space, up and to the left of every target.

What do you measure? Log the target and the click for 50 steps. The misses are not random. Each one is half the target's distance from the top-left corner. A link at (195, 519) is missed by 277 pixels; a button at the bottom right is missed by far more.

What do you change? Map every click back with the size the model actually saw: x × 1,920 ÷ 960, and y × 1,080 ÷ 540. Every click lands. The harness here is made up; the arithmetic is exact.

worked exampleOne screenshot
  pixels             = 1920 × 1080 = 2,073,600, three numbers each = 6,220,800 numbers

Snap to multiples of 28 (released UI-TARS-1.5 code)
  width              = 1920 ÷ 28 = 68.57 → 69 squares → 69 × 28 = 1,932 px
  height             = 1080 ÷ 28 = 38.57 → 39 squares → 39 × 28 = 1,092 px
  grid               = 69 × 39 = 2,691 squares
  pixel budget       = 100 × 28 × 28 = 78,400 up to 16,384 × 28 × 28 = 12,845,056

Map the click back
  x                  = 197 ÷ 1,932 × 1,920 = 195.77 → 195
  y                  = 525 ÷ 1,092 × 1,080 = 519.23 → 519
  skipping it        = (197, 525): 2 px right, 6 px low, still on the link

A harness that halves the image first (illustrative)
  the model writes   = (98, 260) for a target at (195, 519)
  mapped back        = 98 ÷ 960 × 1,920 = 196;  260 ÷ 540 × 1,080 = 520  → on target
  skipping it        = miss of (97, 259) → √(97² + 259²) = √76,490 = 277 px

Model size
  in use per token   = 23 billion of 230 billion parameters = 10%

A click is a sentence. The model never touches the mouse. It writes two numbers as text, measured on the picture it saw, and a separate program turns them into a click on the real screen.

One step, named

After the click, the computer changes and the harness takes a new screenshot. What comes back from the computer after the action, usually the new screenshot, is the observation.

Thought, action, observation: this rhythm of reasoning, acting and observing has a name from an earlier paper, ReAct. UI-TARS-2 follows it at every step. The full recording of one attempt, every thought, action and observation in order, is a trajectory. Think of the flight recorder in an aeroplane: it keeps everything, in order, for whoever needs to study the flight later. The report writes a trajectory like this:

τ = { (t₀, a₀, o₀), (t₁, a₁, o₁), …, (t_T, a_T, o_T) }

τ: The trajectory (tau): one attempt, step by step.
t_k: The thought at step k.
a_k: The action at step k.
o_k: The observation after it, usually the next screenshot.
T: The last step of the attempt.

The report writes t both for thoughts and for counting steps; this page counts steps with k, so the two never mix.

What the agent keeps in mind

At step 50, what does the agent know about steps 1 to 49? Only what you send it. The most text and images a model can read at once is its context window, like the size of a desk. Everything the model uses for the next step must fit on the desk at the same time.

Screenshots fill the desk fast. If one step kept in full costs a few thousand tokens, a long attempt would push the earliest steps off the edge, or overflow the desk entirely. So the report keeps two kinds of memory. The last few steps stay in full: that is working memory. Everything older becomes a short summary: that is episodic memory. It is how you keep notes: today's notes in full, last week's as a line in a diary.

M_k = ( W_k, E_k )

M_k: Everything the agent remembers at step k.
W_k: Working memory: the most recent steps, in full.
E_k: Episodic memory: short summaries of what came before, its goals and outcomes.

How many recent steps stay in full is a number the report calls N, and it does not publish it. The device below makes up every size, so you can see the shape of the problem: a step in full is heavy, and a summary is light.

worked exampleMemory (every size illustrative: a full step 2,800 tokens, a summary 40, a window of 32,000)
  keep everything    = 50 steps × 2,800 tokens            = 140,000 tokens = 4.4 windows of 32,000
  recent 5 in full   = 5 × 2,800 + 45 × 40 = 14,000 + 1,800 = 15,800 tokens = 49% of the window
  last step that fits, everything = 32,000 ÷ 2,800 = 11.4 → step 11
  last step that fits, N = 5      = 5 + (32,000 − 14,000) ÷ 40 = 5 + 450 → step 455

What fits in the agent's memory

The long box is the context window. Each step kept in full needs room for a screenshot and its text; an older step kept as a summary needs a few words. Slide the attempt forward and see what still fits.

steps so far50

recent steps kept in full (N)5

Every size here is made up: the report gives no token counts, no window size and no value for N. Keeping recent steps in full and older ones as summaries is the report's design (its equations 2 and 3). The report's own judge reads only the last five screenshots of an attempt, to fit its context window.

With every step in full, this made-up desk overflows after step 11. With five steps in full and the rest as summaries, the same desk holds 455 steps. That is the whole reason for two kinds of memory.

The report has a real echo of the same limit. Its own judge, which Chapter 4 meets, reads only the last five screenshots of an attempt, “to fit within the context window”.

The agent's rule for the next step

Now put the pieces together. The agent's rule for choosing what to do next, given what it has seen, is its policy. For a model, the policy is the probabilities it gives each possible next token. The report writes the whole rule in one line:

P( t_n, a_n | instruction, W_n, o_n, E_n )

P: The probability the model gives.
t_n, a_n: The next thought and action.
|: “Given”: everything to the right is what the model reads.
instruction: The task in plain words.
W_n: The last N steps in full.
o_n: The screenshot it sees now.
E_n: The summaries of everything older.

At step 50 with N = 5 (illustrative): 5 steps in full + 45 summaries + this screenshot + the instruction

Two families of action, one model

The full menu of actions the agent may use is its action space, like the keys on a keyboard. UI-TARS-2's action space has two families. The first is the GUI menu above: clicks, keys, typing and scrolling. Games reuse exactly the same clicks and keys.

The second family skips the screen. A set of functions a program can call is a software development kit, or SDK. The agent's ready-made calls, such as running a terminal command, are SDK functions: a toolbox beside the keyboard. They include calls to outside tools through a common standard for calling outside tools, the Model Context Protocol, or MCP.

One last design choice sits under all of this. One model that sees, plans, remembers and acts by itself is a native agent. Separate programs for each job, glued together with hand-written rules, form a modular pipeline. Think of one cook who does everything, against a kitchen of specialists who follow a rulebook.

The report argues for the single cook. Hand-written rules are brittle and hard to scale: every new app needs new rules. One model trained end to end can keep learning from data instead. The price is that it needs a great deal of data, and data is the subject of Chapter 3.

The model saw a 1932 × 1092 image of a 1920 × 1080 screen and wrote a click at (197, 525). Where should the harness click on the screen?

At (197, 525), because the numbers the model writes are already measured in screen pixels. At (198, 530), scaling each number by the size the model saw divided by the real size. At (195, 519), scaling each number by the real size divided by the size the model saw. At (28, 28), because the model can only point at whole squares of the 28-pixel grid.

Chapter 2

A Thousand Computers to Practise On

Build the practice computers: sealed, shared between browser and terminal, and reset when they crash.

Practice means letting an agent loose on a computer. It will click Delete. It will fill the disk with downloads. It will leave a game running and walk away. Now do that on thousands of computers at once, all day, and have each one clean and ready for the next attempt within seconds.

That is the job of this chapter. Before any learning can happen, someone has to build the practice computers, hand them out, take them back and wipe them. It is less glamorous than the learning, and just as necessary.

A sealed computer

Think of a sandpit in a playground. Children dig, build and wreck, and at the end of the day someone rakes it smooth for tomorrow. A sealed practice computer the agent can wreck, which is reset before the next attempt, is a sandbox.

There are two common ways to build one. The first is a whole computer imitated by software on a bigger computer, with its own operating system: a virtual machine, or VM. A flight simulator is the picture: everything the pilot touches behaves like a real cockpit, but nothing can crash for real. The operating system is the base software that runs a computer, such as Windows, Ubuntu or Android.

The second is lighter. A sealed box that shares the big computer's core but has its own files and programs is a container. A virtual machine is a whole house; a container is a flat in a building that shares the plumbing.

Either way, the agent meets a real operating system with real apps. Nothing it touches is a mock-up. When it opens a spreadsheet, a real spreadsheet program opens. That matters, because the skills it learns must work on real computers afterwards.

The report's computers

For clicking, the report runs a cloud of virtual machines with Windows, Ubuntu and Android. The agent's actions reach them through two tools. The first is a Python library that moves the mouse and presses keys, PyAutoGUI. The second is a tool that sends taps and swipes to an Android phone, the Android Debug Bridge, or ADB.

One SDK (Chapter 1's word: a kit of functions a program can call) covers a session's whole life. It books a computer, sets it up for the task, lets the agent act, collects screenshots and screen recordings, and runs the task's check at the end. The same SDK serves three users: people recording worked examples, benchmark runs, and practice. So all three see the same kind of computer.

The report describes these stages but prints no code; every name below is made up. Read it as the shape of one attempt's life:

pythonvm = fleet.book(os="ubuntu", lease_minutes=10)       # a computer and a session ID, with a due date
vm.setup(task.setup_steps)                           # open the right apps and files for this task
shot = vm.screenshot()
for k in range(task.step_budget):
    thought, action = agent.step(task.instruction, memory, shot)
    vm.do(action)                                    # a click, a key, or a terminal command
    shot = vm.screenshot()                           # the observation for the next step
    if action.startswith("finished"):
        break
reward = task.check(vm)                              # the task's own check reads the final state
fleet.release(vm)                                    # reset and return the computer

Thousands, and their bookkeeping

How many of these computers does practice use? The report says “several thousand”, run by one manager program that handles “several thousand” requests a second. Requests handled per second is called QPS. The report gives no exact numbers, so neither do we.

With thousands of computers and thousands of attempts, you need bookkeeping. One attempt's booking of one computer is a session. Its ticket number is the session ID, like a cloakroom ticket: hand it over and you get your coat back. The session ID ties a task to the computer running it, so a log line, a screenshot or a crash can be traced to the right attempt.

People also need to see what the agents are doing. Anyone on the team can watch a session live through a way to watch and control a remote screen live: VNC. (The report also names RTC streaming, another way to send a live screen.)

How many computers you need

Picture a restaurant that seats 3,000 diners an hour, and each diner stays 2 minutes. How many tables are in use at any moment? Each minute 50 diners arrive. Each one stays 2 minutes. So at any moment, the diners from the last 2 minutes are seated: 100 of them, at 100 tables.

A practice fleet works the same way. Finished attempts per hour is the fleet's throughput. The question is how many computers that throughput needs. Here is the reasoning, in four steps:

Let λ (lambda) be the attempts started per second, and W the seconds each attempt keeps its computer.
Pick any moment. The attempts still running then are exactly the ones that started in the last W seconds.
In W seconds, λ × W attempts start. So, on average, λ × W computers are busy at any moment.
Turned around, the throughput a fleet of L computers can reach is λ = L ÷ W.

Computers busy at once equals attempts started per second times seconds each attempt keeps its computer. That is Little's law, the same rule the restaurant follows:

L = λ × W

L: Computers busy at once.
λ: Attempts started per second.
W: Seconds each attempt keeps its computer.

0.833 × 120 = 100 computers (3,000 attempts an hour, 30 steps of 4 s each)

This is why a slow step matters twice. It makes each attempt longer, and so the same throughput needs more computers. A 30-second reset between attempts counts too: it holds the computer just as surely as the attempt does.

worked exampleLittle's law: computers busy = attempts started per second × seconds each one holds a computer
  target            = 3,000 attempts an hour (illustrative) = 3,000 ÷ 3,600 = 0.833 a second
  one attempt       = 30 steps (illustrative) × 4 s          = 120 s
  computers busy    = 0.833 × 120                             = 100
  plus a 30 s reset = 0.833 × (120 + 30)                      = 125
  turned around     = 3,000 computers (illustrative) ÷ 150 s × 3,600 = 72,000 attempts an hour

Leaked computers (illustrative: 40 computers, 2-minute attempts, 1 in 200 crash silently)
  attempts per computer in 8 h = 8 × 60 ÷ 2         = 240
  still working     = 40 × 0.995^240 = 40 × 0.300    = 12.0
  throughput        = 12.0 × 30 an hour              = 360, down from 40 × 30 = 1,200
  with 10-minute leases: average hold = 2 + 0.005 × 10 = 2.05 min → 40 × 60 ÷ 2.05 = 1,171 an hour

Two hands (illustrative task, real seconds per step)
  clicks only       = 11 steps × 4 s = 44 s
  clicks + terminal = 6 steps × 4 s  = 24 s

When an attempt never comes back

Here is a failure that every fleet meets sooner or later. A practice run starts at 9:00 with 40 computers finishing 1,200 attempts an hour. Nothing about the agent changes all day. By 17:00 the fleet finishes 360 an hour.

What do you measure? List every session and how long it has been open. Attempts take 2 minutes, yet 28 sessions have been open for hours. Each one is an attempt that crashed without ever reporting “done”, still holding its computer. Nothing will ever release it, because the release step comes after the end of an attempt, and this attempt never ends.

What do you change? Every booking gets a due date. A booking with a due date, where the computer is taken back if the attempt has not finished by then, is a lease, like a library loan. With 10-minute leases, the fleet holds at about 1,171 attempts an hour, with about 1 computer stuck at any moment. That is the report's design: leases release computers after success or failure, and overdue sessions are taken back. The numbers here are a toy.

Keep the fleet healthy

Forty practice computers through a working day. Now and then an attempt crashes without reporting back. Without leases its computer stays booked forever. Slide through the day, then switch leases on.

hours into the day8 h

A toy fleet: 40 computers, 2-minute attempts, a 1 in 200 chance that an attempt crashes without saying so, and 10-minute leases are all made up. Session IDs and taking back overdue sessions with leases are the report's design; its real fleet had several thousand computers.

Notice the shape of the warm line. It does not drop off a cliff; it sags a little every hour. That is why this failure is easy to miss: no single hour looks broken.

Two hands on one disk

Some tasks are clumsy by clicking alone. Say the task is to download a shop's sales file and say how many orders it lists. By clicking, the agent must open the file in a spreadsheet, jump to the last row and read the row number off the screen. A person at a terminal would type one short command.

So the report's sandbox gives the GUI and a terminal one disk that the browser and the terminal both see: a shared file system. Picture one desk that two people work at: whatever one puts down, the other can pick up. In the report's words, the browser downloads a file and the shell processes it at once, inside the same box. The report's name for clicking plus these extra functions is GUI-SDK.

The sandbox goes further. Services started from the terminal show up at proxy web addresses, so the agent can open a page it has just built. The browser also offers VNC, a remote code editor, Jupyter notebooks and terminal views. That is how the last part of the Hex attempt works: the agent writes a page in the terminal and opens it in the browser.

One sandbox, two hands

The task: download the shop's sales file and say how many orders it lists. Choose what the agent may use. Then switch the shared disk off and watch the same plan fail.

A made-up task and made-up step counts, at the report's 4 seconds per step. The disk shared by the browser and the terminal is the report's design. Its measured payoff is in Chapter 9: on a hard English web-search test, adding these functions lifted the score from 7.0 to 29.6.

Here are the two hands as the agent would issue them. The terminal line is real shell, and the output is what the standard tools print; the sales file and its 1,284 rows are made up.

shell$ tail -n +2 ~/Downloads/sales.csv | wc -l
1284
$ # without a shared disk:
$ tail -n +2 ~/Downloads/sales.csv | wc -l
tail: cannot open '/home/agent/Downloads/sales.csv' for reading: No such file or directory
0

tail -n +2 prints the file from line 2 on, skipping the header row; wc -l counts the lines it is given.

A sandbox for games

Games get their own sandbox. The report's practice games are mini-games written for the web, in HTML5 and WebGL, so they run in a browser. The sandbox runs them in a container with several Chrome browsers inside it. Browsers start and stop as demand changes, and a browser that crashes is restarted.

Two more parts keep the fleet tidy. A saved state of a page you can return to is a snapshot, like a save point in a video game (the report calls it checkpointing). And a garbage collector clears away sessions that have gone idle.

To drive the browsers, the sandbox speaks a remote-control socket built into the Chrome browser, the Chrome DevTools Protocol. It also works with a popular library that drives browsers through it, Playwright. Screenshots are drawn with the graphics card, so they are fast. The contract is simple: actions go in, and observations come straight back, a screenshot plus the game's score and level. Here are the two sides, as the report's Figure 2 draws them:

The client side (what the trainer calls)

run an action: a click or a key
take an observation: a screenshot, the score, the level
control time: speed up or pause the game's clock
save and restore snapshots

The container side (what runs the games)

a page manager: create and delete pages, track each page's state
a browser manager: several Chrome browsers, restarted after a crash
an event handler for actions and results
a garbage collector for idle sessions

The fake clock

One problem is special to games. The agent spends about 4 seconds on each step, thinking. A live game keeps running during those 4 seconds: the snake moves, the timer runs out. A person never has to think that long between moves.

So the sandbox replaces the page's own timer functions, and game time can run faster or stop while the agent thinks. That is a fake clock, like a referee who stops the match clock during an injury. The report says this makes practice faster and every run repeatable, without changing the game's own logic.

Practice is a building problem before it is a learning problem. Every attempt needs a clean computer, a check at the end and a way back from a crash. The report built one platform for all of it, so recording examples, practice and testing all run on the same kind of computer.

A practice run's throughput falls from 1,200 attempts an hour to 360 over eight hours, while the agent itself has not changed. What should you check first?

Whether the agent now thinks longer per step, since 4 seconds a step may have stretched to 13 over the day. Whether the grader fails more attempts, since a failed attempt does not count toward the hourly throughput. Whether crashed attempts are still holding computers, because nothing takes back a session that never reports done. Whether the browser and the terminal share a disk, since a missing file would make every attempt fail.

Chapter 3

The Data Flywheel

Turn every finished attempt into training data, the passes and the failures both.

By the end of a practice day, ten thousand attempts have finished. Each one is a complete recording: every screenshot, every thought, every click, and a pass or a fail. Some are excellent. Most are not. What should happen to the ones that failed?

The report's answer is a loop it calls a data flywheel. To see why it is built the way it is, we first need the three ways a model learns from data, and where this model's data comes from in the first place.

Three ways a model learns from data

Think of how a person becomes an expert. First they read a library: thousands of books, without a test. Then they work through a workbook with the answers in the back. Finally they sit real exams and learn from the marks. A model goes through the same three stages.

Reading huge amounts of text and predicting each next token is pre-training. Every large language model starts this way. More of the same on new material, here material about using computers, is continual pre-training, or CT. In this lesson the CT material is called background reading: the model reads it and predicts it, but nobody marks whether it copies it well.

Training on worked examples, so the model reproduces each step, is supervised fine-tuning, or SFT. This is the workbook. It is Chapter 0's “studying worked examples”, now with its formal name.

Practice, from Chapter 0, is the exam. Its formal name is RL. The report's training has exactly these three stages: CT, then SFT, then RL.

Where it starts

A saved copy of a model's parameters is a checkpoint. The checkpoint you start from is the base model. UI-TARS-2 starts from the pre-trained checkpoint of Seed1.6, a model its makers had already trained, and it also uses all of the data that model was trained on after pre-training.

Computer-use data is only one ingredient. It is a small part of what CT reads, and a much larger part of what SFT studies. Both stages also mix in general chat and reasoning data, so the model does not forget how to talk and think while it learns to click.

The first data, before the model can make any of its own, is the cold start. For CT, the report starts from everything UI-TARS and UI-TARS-1.5 were trained on: computer tutorials collected from the internet, open-source recordings of agent attempts, and the company's own recordings.

People who think out loud

Some data has to come from people. People paid to record and label tasks are annotators, and their work is annotation. The report tells a short story about how hard good annotation is.

In a first trial, the team recorded people's clicks and asked them to add their reasoning afterwards. It failed. Nobody could remember what they had been thinking at each click, so the reasoning they wrote was a guess.

So the team switched to a cooking-show method. Saying your thoughts out loud while you work, recorded together with the screen, is think-aloud. The recording tool ran on the annotators' own computers while they used them normally. Their speech went through software that turns speech into text, automatic speech recognition (ASR). Then a language model tidied the text and lined it up with the clicks.

The report adds care at every stage. Both experts and novices recorded, so the data shows smooth paths and clumsy ones. Apps were chosen by how widely people use them. Tasks were graded for difficulty by their number of steps, the pages they cross, what they need first, and the errors they must handle. Every recording was checked that it runs, de-duplicated, and reviewed by two annotators.

People who correct the agent

Recordings of experts have a blind spot, the same one Chapter 0 found. Data from situations the agent itself gets into is on-policy. Data from someone else's path, such as an expert's, is off-policy. Learning from your own driving mistakes is on-policy; watching the instructor's perfect drive is off-policy. An expert's perfect recording never visits the screens the agent's own mistakes lead to.

So the report's second tool mixes the two. At every step, the newest model proposes its next thought and action. The annotator accepts it, or overrides it with a better one. The computer then carries out whichever was chosen. Every recording made this way follows a path the agent itself would take, with a person stepping in only where the agent would go wrong. The tool streams the computer's screen to the annotator and draws the proposed click on it, and the model and the task list are refreshed regularly.

The flywheel

Now the loop. Let the model make several attempts and keep them sorted by a check: that is rejection sampling (the report writes RFT). A check that says pass (1) or fail (0) for each new sample is the validator, written V(s).

A loop where the model makes data and the data makes a better model is a data flywheel. A real flywheel is a heavy wheel that, once spinning, keeps itself going. One pass around the loop is a round. Each round works like this:

The newest model makes new attempts, by rejection sampling or through the annotators who correct it.
The validator checks each one.
Passes go to the worked-examples pile, for SFT. Failures go to the background-reading pile, for CT.
The model is trained again on the grown piles: CT first, then SFT, then RL. SFT and RL run more often than CT.

The report writes the routing as two lines, one per pile:

D_SFT^(r+1) = D_SFT^(r) ∪ D_RFT,high^(r)

D_CT^(r+1) = D_CT^(r) ∪ D_RFT,low^(r)

D_SFT^(r): The worked-examples pile after round r.
∪: Add these samples to the pile.
D_RFT,high^(r): The round's new samples that the validator passed, V(s) = 1.
D_CT^(r): The background-reading pile after round r.
D_RFT,low^(r): The new samples the validator failed, V(s) = 0.

The report counts rounds with t; this page uses r, because t already means a thought.

The bookkeeping follows from those two lines. Here it is in five steps:

Round r makes n new samples. The validator splits them: n_pass with V = 1 and n_fail with V = 0, and n_pass + n_fail = n.
The SFT pile grows by n_pass; the CT pile grows by n_fail. Together the piles grow by exactly n: nothing is lost.
With a pass share p_r, n_pass = p_r × n.
After R rounds the SFT pile has grown by n × (p₁ + … + p_R), and the CT pile by n × R minus that.
The flywheel's claim is only that p_r rises with r.

The report publishes no pile sizes, no pass rates and no number of rounds. So the worked example below makes up four rounds of 10,000 attempts each, with a pass share that climbs. Only the routing rule is the report's.

worked exampleFour rounds of 10,000 new attempts, with a made-up pass share each round
  round 1   p = 0.30   to SFT 0.30 × 10,000 = 3,000   to CT 7,000   piles: SFT  3,000  CT  7,000
  round 2   p = 0.45   to SFT 0.45 × 10,000 = 4,500   to CT 5,500   piles: SFT  7,500  CT 12,500
  round 3   p = 0.60   to SFT 0.60 × 10,000 = 6,000   to CT 4,000   piles: SFT 13,500  CT 16,500
  round 4   p = 0.70   to SFT 0.70 × 10,000 = 7,000   to CT 3,000   piles: SFT 20,500  CT 19,500
  check     20,500 + 19,500 = 40,000 = 4 × 10,000: nothing thrown away

A lenient validator that also passes 1 in 4 failures (illustrative)
  round 1   SFT gets 3,000 + 0.25 × 7,000 = 3,000 + 1,750 = 4,750, of which 1,750 are failures = 36.8%
  round 2   SFT gets 4,500 + 0.25 × 5,500 = 4,500 + 1,375 = 5,875   junk 1,375 ÷ 5,875 = 23.4%
  round 3   SFT gets 6,000 + 0.25 × 4,000 = 6,000 + 1,000 = 7,000   junk 1,000 ÷ 7,000 = 14.3%
  round 4   SFT gets 7,000 + 0.25 × 3,000 = 7,000 +   750 = 7,750   junk   750 ÷ 7,750 =  9.7%
  after 4   SFT 4,750 + 5,875 + 7,000 + 7,750 = 25,375, of which 4,875 are failures = 19.2%

Spin the flywheel

Each spin is one round: the newest model makes 10,000 attempts, and the validator sends each one to a pile. Green dots are passes, going to the worked examples; the rest go to the background reading. Spin four rounds, then try a lenient validator.

All counts are made up: the report publishes no data sizes, no pass rates and no number of rounds. The two outlets, passes to fine-tuning and failures to continual pre-training, and the claim that the pass share rises each round, are the report's (its Section 2.3).

Why not bin the failures?

Most data pipelines are filters: keep the good, throw away the rest. The report's flywheel has two outlets instead. Its claim is plain: every sample is reused at a stage where it fits, so no data is wasted.

The report does not say more about why failures belong in background reading. Here is one way to see it, and it is our reading, not the report's. SFT tells the model: do exactly this. Only passes belong there. CT only asks the model to read and predict. A failed attempt still shows real screens and what real clicks did to them, and that is worth reading even when it is not worth copying.

Why the wheel speeds up

A better model passes more often. So the next round has more passes to study, which makes a better model again. The report states this as one inequality:

P( V(s) = 1 | r ) > P( V(s) = 1 | r − 1 )

P( V(s) = 1 ): The chance that a new sample passes the validator.
|: “Given”, here: in which round.
r: The round. The chance in round r is higher than in the round before.

made-up shares: 0.30 < 0.45 < 0.60 < 0.70

That rising share is why the worked-examples pile grows faster each round in the device: 3,000, then 4,500, then 6,000, then 7,000 new examples.

When the validator is too kind

The flywheel is only as good as its validator. Here is how a weak one shows up. After a round, 4,750 new worked examples join the pile. The next model trains on them, and it is no better on a fixed test set.

What do you measure? Take 200 of the new worked examples at random and check them by hand. About 37% of them are failed attempts the validator passed. The model was trained to reproduce them, step by step.

What do you change? Make the validator stricter, and send what it rejects to the background reading instead. A failed attempt is useful to read, and harmful to copy. Switch the device to the lenient validator to see the same numbers: 1,750 of round 1's 4,750 new examples are mistakes.

In code, the routing rule is four lines. Every sample lands somewhere:

pythondef route(samples, validator, sft_pile, ct_pile):
    for s in samples:
        if validator(s) == 1:
            sft_pile.append(s)        # a pass: study it and reproduce it
        else:
            ct_pile.append(s)         # a failure: read it, never copy it
    return sft_pile, ct_pile          # every sample lands somewhere: no data is wasted

What is a sample, concretely? The report does not print its record format, so here is an illustrative layout of one attempt as a record, in the common text format for records, JSON:

json{
  "task": "Play the Hex game, try to get more score.",
  "steps": [
    {"thought": "I don't know the rules of Hex. I'll search the web first.",
     "action": "click(point='<point>640 60</point>')",
     "screenshot": "step_000.png"},
    {"thought": "Type the search.",
     "action": "type(content='hex game rules strategy')",
     "screenshot": "step_001.png"}
  ],
  "round": 3,
  "source": "rejection_sampling",
  "validator": 1
}

Read it field by field. task is the instruction in plain words. steps is the trajectory: each entry holds a thought, an action and the screenshot that followed it. round says which turn of the flywheel made it, and source says how: by rejection sampling, or by an annotator correcting the agent. Last, validator holds the verdict, and that one number decides the record's pile.

Two outlets, not one filter. A filter keeps the good and bins the rest. The flywheel keeps everything and decides only where each attempt goes: passes to the worked examples, failures to the background reading.

In round 3 the newest model makes 10,000 attempts and the validator passes 6,000 of them. With the report's routing, what happens to the 4,000 failures?

They are deleted, because studying a failure would teach the agent to repeat its own mistakes. They join the worked examples as well, so the agent sees every situation at least one time. They go back to annotators, who redo each one correctly by hand before it is used for training. They join the background reading for continual pre-training, so that no attempt is wasted.

Chapter 4

Tasks and Graders

Make tasks worth practising, and decide automatically whether each attempt passed.

Here is one practice question from the report: “Discovered by a representative from the Music And Cabaret talent agency, this group had a founding lineup, initially under another name, that included members from Dreghorn and Irvine, plus a lead guitarist and drummer. The lead vocalist joined after being recommended by a founding member who saw them perform with a Kilmaurs-based band, and their lead guitarist left to form another ensemble before late 1975. Which record label did this group sign with?” No person wrote it. A pipeline built it to be hard to look up, and easy to grade.

Practice needs two things this chapter builds. It needs tasks: lots of them, hard enough to teach something, each with a known right outcome. And it needs a grader for every task, one that decides pass or fail with no person in the loop.

Rewards a program can check

Think of an answer key in the back of a maths book. Anyone can check an answer against it, instantly, without being a mathematician. A reward a program can check without a person is a verifiable reward. Practice with such rewards is RL with verifiable rewards, or RLVR.

The report builds three practice worlds around rewards like this. The first is finding facts by reading web pages; the report calls it GUI-Browsing. The second is getting things done on websites, called GUI-General. The third is browser games. Each world needs its own way to make tasks, and its own grader.

Hard questions on purpose

Fact questions are easy to grade: the answer is a name or a date. The trick is making them hard to answer. The report uses two recipes.

The first recipe starts from an encyclopedia page about a person, a band or a place. A pipeline lists the facts on the page. A language model scores how much each fact gives away. The most telling facts are dropped, and the rest are blurred into vaguer words. Rewriting the question so its most telling clues are gone is obfuscation. A good crossword clue works the same way: every word is true, and none of them hands you the answer. Read the band question again with that in mind: every clue is specific, and no single clue names the band.

The second recipe follows a link from one page to another. The thing on the second page becomes the answer, and the step repeats several times. Then the chain is fused into one question. A question whose answer needs several lookups in a chain is multi-hop, like a treasure hunt where each clue leads to the next.

Then a filter throws out the easy ones. Any question a model can answer from memory, or with a single search, is dropped. What is left needs real searching.

No search engine allowed

There is one more twist. The agent gets no direct line to a search engine that returns results as text, which engineers call a search API. It must find the answer the way a person at a browser does: by reading screenshots of pages, clicking links and scrolling. That is what makes these tasks practice for a GUI agent, rather than for a search bot.

Things to do on websites

The second world is doing things on websites: filling a form, changing a setting, finding a product. The report starts from public lists of websites. It drops broken sites, sites behind a login, static sites and casual-game sites. A vision-language model lists each remaining site's core functions, and tasks are written for those functions, each on a single page and each checkable. The set covers 690 websites.

Games

The third world is browser games: public mini-games written in HTML5 and WebGL, plus new games written by a language model so that their state can be read. Each game gets a few lines of JavaScript that read the game's own numbers while it runs, its runtime variables: the score, the level, the lives left. That script is a verifier script. Think of it as the scoreboard.

Every verifier writes its result in one layout, using a common text format for records with nested names and values: JSON. The record holds a reward, a flag that says whether the attempt is over, and details such as the game's version and a short fingerprint computed from the game's state, so that a changed record can be spotted. That fingerprint is a checksum, like the seal on an envelope.

The report says only that a short script reads runtime variables such as score, level and lives, and writes a reward, a done flag and details like the game version and a checksum. So the names below are illustrative:

javascript// verifier for one game page (made-up names)
function verify(game) {
  const cleared = game.level > game.startLevel;          // did the agent clear the level?
  return {
    reward: cleared ? 1 : 0,                             // pass or fail
    done: cleared || game.lives === 0,                   // the attempt is over
    meta: { game_version: game.version, checksum: hashState(game) }
  };
}

Three kinds of grader

Each world gets the grader it can support. In games, the verifier script reads the numbers and gives pass (1) or fail (0). No judgement is needed.

Fact questions have a right answer, but the agent may phrase it differently from the answer key: a label's full name, its short name, or a whole sentence around it. A plain string match would fail all three. So a language model compares the agent's answer with the right one, the reference answer. That is LLM-as-judge. During training, this judge was GPT-4o.

Tasks on websites are the hardest to grade. “Change the delivery address” has no single right string to compare with. So the report uses a model that reads a whole attempt and decides whether it succeeded: an outcome reward model, or ORM. Picture an examiner who watches the recording of your driving test afterwards. A generative ORM writes its verdict as text.

The report's ORM is UI-TARS-2 itself, trained for the job with targeted annotation and single-turn RL. It reads the full text of the attempt, every thought and action, but only the last five screenshots, to fit its context window. Again with made-up names, the judge's input looks like this:

pythonjudge_input = {
    "instruction": task.instruction,
    "history": [s.thought + "\n" + s.action for s in attempt.steps],  # the full text of the attempt
    "screenshots": [s.screenshot for s in attempt.steps[-5:]],          # only the last five screenshots
}
verdict = judge.generate(judging_prompt(judge_input))   # the model writes its verdict as text
reward = 1 if "success" in verdict.lower() else 0      # read as pass or fail

How good is the judge?

A judge can be wrong, so the team tested theirs. They collected 300 attempts that people had labelled pass or fail. As a judge, UI-TARS-2 scored an F1 of 83.8, and its mistakes leaned toward passing failures. To read that sentence we need four words.

Picture a fishing net. Two questions matter: how much of the catch is fish, and how many of the fish in the water it caught. A judge has the same two questions. A failed attempt the judge calls a pass is a false pass (engineers say false positive). A good attempt it calls a fail is a false fail (false negative).

Of the attempts the judge passed, the share that really succeeded is its precision: how much of the catch is fish. Of the attempts that really succeeded, the share the judge passed is its recall: how many fish it caught. One number that is high only when both are is F1. Here is where its formula comes from, in four steps:

Count four boxes. TP: real successes the judge passed. FP: failures it passed, the false passes. FN: successes it failed, the false fails. TN: failures it failed.
precision = TP ÷ (TP + FP), and recall = TP ÷ (TP + FN).
F1 is their harmonic mean: 2 × precision × recall ÷ (precision + recall). A harmonic mean is pulled down hard by the smaller of the two numbers.
Substitute and simplify: F1 = 2 TP ÷ (2 TP + FP + FN). So F1 never uses TN: it asks only how well the judge finds the successes without passing failures.

precision = TP ÷ (TP + FP) recall = TP ÷ (TP + FN)

F1 = 2 × precision × recall ÷ (precision + recall) = 2 TP ÷ (2 TP + FP + FN)

TP: Real successes the judge passed.
FP: Failures it passed: false passes.
FN: Successes it failed: false fails.

2 × 124 ÷ (2 × 124 + 34 + 14) = 248 ÷ 296 = 0.838

The report gives the F1 and the direction of the errors, not the four counts. So the worked example uses one made-up split of the 300 attempts that gives exactly that F1, with more false passes than false fails:

worked exampleThe report's judge test: 300 attempts labelled by people, F1 83.8, more false passes
One split of the 300 that gives exactly that (illustrative)
  real successes 138:  passed 124 (TP)   failed 14 (FN)
  real failures  162:  passed  34 (FP)   failed 128 (TN)        124 + 14 + 34 + 128 = 300
  precision   = 124 ÷ (124 + 34) = 124 ÷ 158 = 0.785
  recall      = 124 ÷ (124 + 14) = 124 ÷ 138 = 0.899
  F1          = 2 × 124 ÷ (2 × 124 + 34 + 14) = 248 ÷ 296 = 0.838
  passed attempts that really failed = 34 ÷ 158 = 21.5%
  plain accuracy (not F1) = (124 + 128) ÷ 300 = 252 ÷ 300 = 84.0%

A real judge does not just say pass or fail. Inside, it has a confidence, and someone picks the line above which it says pass. Move that line and watch precision and recall trade against each other.

Grade the judge

300 attempts that people labelled. Top row: attempts that really succeeded. Bottom row: attempts that really failed. Each dot sits at the judge's confidence that the attempt succeeded. Move the pass line and watch the four counts.

pass if the judge's confidence is at least0.50

The 300 attempts and the F1 of 83.8 are the report's: it tested its judge on 300 attempts that people had labelled, and found more false passes than false fails. How the 300 split, and every confidence score, are made up so that the default line reproduces that F1.

Drag the line down and the judge passes nearly every real success, but precision falls: at 0.20 it passes 115 failures. Drag it up and precision climbs to 100% at 0.90, but the judge now fails 116 of the 138 real successes. In this toy, the best F1 sits right at 0.50.

Why a judge with false passes still works

A judge that passes failures sounds dangerous: the agent is rewarded for attempts that did not do the job. The report gives an argument for why practice still worked, and it is an argument, not a measurement. In a falsely passed attempt, most of the steps were usually right. Rewarding those right steps helps more than rewarding the few wrong ones hurts.

The bigger danger is a different one. Finding a way to get the reward without doing the task is reward hacking, like a student who learns the marker's habits instead of the subject. The team read rewards by hand during training, and the report says it saw no substantial reward hacking.

Steering with small extra rewards

The reward is mainly about the final outcome: did the attempt do the job? Adding small extra rewards to steer behaviour is reward shaping. The report uses two kinds in some settings. A bonus for writing actions in the right form is a format reward. A cost for running too long is a length penalty. Together they guard against two bad habits: quitting too early, and never stopping.

Here is what the first habit looks like. Midway through practice the reward climbs fast and the number of steps per attempt drops. It looks like a breakthrough. The test score does not move.

What do you measure? Read 50 recent attempts the judge passed. Many end with the agent declaring the task finished on the second or third screen, before the work was done, and the judge believed it.

What do you change? Add a cost for stopping before the task's final state is reached, raise the judge's pass threshold, and keep reading samples of passed attempts by hand. The report uses format rewards and length penalties in some settings for exactly this: to stop the agent quitting early, or never stopping. This story is illustrative; the guard rails are the report's.

A grader is a model of the task, and it can be wrong. The report's own judge passes some failures. Practice still worked, by the report's account, because a falsely passed attempt is mostly right steps. A judge that passed mostly wrong steps would teach the wrong thing.

The judge passes 158 attempts, and 34 of them really failed. Of the 138 attempts that really succeeded, it passed 124. What is its precision?

About 90%, because it passed 124 of the 138 attempts that really succeeded. About 78%, because 124 of the 158 attempts it passed really succeeded. About 84%, because that is the share of all 300 attempts it judged correctly. About 22%, because 34 of the 158 attempts it passed had really failed.

Chapter 5

Spreading One Reward

Pass a single reward back over thousands of tokens, and see why a critic must learn first.

An attempt ends after 30 steps. Along the way the agent wrote about 2,000 tokens of thoughts and actions. Then the grader says one thing: pass. One number, 1, for 2,000 tokens. Which of them earned it?

Those sizes are made up, but the shape is real: long attempts, one reward at the end. This chapter is about turning that one number into a lesson for every token. It is the hardest idea in the lesson, so we take it one step at a time.

What practice changes

The model writes one token at a time. For each next token it gives every possible token a probability, then picks one at random by those odds. That is sampling, like rolling a weighted die. A token with probability 0.7 comes up seven times in ten.

That list of probabilities is the policy from Chapter 1. Training nudges the parameters, and so the probabilities. The simplest practice rule follows directly. After a pass, raise the probability of every token the agent wrote. After a fail, lower them. Over many attempts, tokens that tend to appear in passing attempts become more likely. (Policy gradients builds this rule from scratch.)

Better than expected

That simple rule has a flaw. On an easy task, almost every attempt passes. Raising everything after every pass teaches nothing new: the agent already does this. What matters is whether an attempt did better than expected.

Think of betting on a football match. Backing the favourite and winning is no news; the bookmaker expected it. Backing the outsider and winning is big news. What you expected to earn from here is a baseline. How much better than expected things turned out is the advantage.

Here is the arithmetic, with made-up numbers. The agent passes 9 of 10 attempts at a task, so it expects 0.9. A pass has advantage 1 − 0.9 = 0.1: small news. A fail has advantage 0 − 0.9 = −0.9: big news. The update pushes in proportion to the advantage, so the rare fail moves the agent nine times as hard as a pass.

A critic for every token

One baseline per task is coarse. Halfway through an attempt the agent may already be stuck, or nearly done. The expectation should change as the attempt goes on.

So a second network watches. It looks at any point in an attempt and guesses the reward to come, its value. That network is the critic, like a commentator calling the odds mid-match. Everything the attempt earns from this point to the end is the return, and the critic's job is to guess the return before it arrives. The report trains with a critic, inside the update method that Chapter 7 names.

The surprise at each token

Now the key move. Compare the critic's guess before a token with its guess after. If the guess went up, that token looked like good news. The reward just received, plus the critic's guess from the next point, minus its guess from this point, is the TD error: the surprise at that token.

δ_k = r_k + γ V(s_k+1) − V(s_k)

δ_k: The surprise at token k (delta).
r_k: The reward that arrives at token k: 0 everywhere except the last token.
γ: The discount (gamma); 1 on this page.
V(s_k): The critic's guess of the reward to come, from token k.
s_k: Everything written and seen up to token k.

A factor that shrinks rewards further in the future is the discount, γ (gamma). It matters for tasks that never end. In this lesson γ = 1, so it drops out and every reward counts in full.

Adding up the surprises

One surprise only looks one token ahead. To credit a token with what happened later, add up the surprises ahead of it, each one shrunk by a factor λ (lambda) per token of distance. That is generalized advantage estimation, or GAE. Think of an echo that fades: the reward at the end rings back through the attempt, quieter at every token.

Â_k = Σ_j=0 (γλ)^j δ_k+j

Â_k: The advantage of token k: how much better than expected.
λ: How much each later surprise still counts, per token of distance.
Σ_j=0: Add the terms for j = 0, 1, 2 and on, to the end of the attempt.

Here is the derivation, from the definition to the two ends of the dial:

The TD error at token k is δ_k = r_k + γ V(s_k+1) − V(s_k). With γ = 1, δ_k = r_k + V(s_k+1) − V(s_k).
GAE adds future surprises with a fading weight: Â_k = δ_k + λ δ_k+1 + λ² δ_k+2 + …, up to the last token.
The same sum, computed from the back in one pass: Â_last = δ_last, then Â_k = δ_k + λ Â_k+1.
At λ = 0: Â_k = δ_k, only the next surprise. The token trusts the critic completely.
At λ = 1: Â_k = Σ_j≥k (r_j + V(s_j+1) − V(s_j)). The V terms cancel in pairs: each + V(s_j+1) meets − V(s_j+1) in the next term. What is left is the final reward + V(after the end) − V(s_k), and nothing is expected after the end. So Â_k = final reward − V(s_k): the real outcome minus the guess.

That last case has a name. Using the real final reward and ignoring the critic's middle guesses, λ = 1, is the Monte Carlo return. So λ is a dial between two kinds of error. A steady error in one direction, from trusting a critic that is wrong, is bias: a bathroom scale that always reads 2 kg heavy. A wobble from attempt to attempt, from trusting one noisy outcome, is variance: a scale that jumps around. Low λ leans on the critic and risks bias. High λ leans on the outcome and risks variance.

Work a tiny case by hand. Four tokens, rewards 0, 0, 0, 1, and the critic's guesses 0.5, 0.6, 0.7, 0.8, with 0 expected after the end. The surprises are 0.1, 0.1, 0.1 and 0.2:

worked exampleA baseline, by hand (illustrative): the agent passes 9 of 10 attempts at a task
  advantage of a pass = 1 − 0.9 = 0.1      advantage of a fail = 0 − 0.9 = −0.9

A 4-token toy (illustrative): rewards 0, 0, 0, 1;  critic guesses V = 0.5, 0.6, 0.7, 0.8;  0 after the end;  γ = 1
  surprises δ   = 0 + 0.6 − 0.5 = 0.1;   0 + 0.7 − 0.6 = 0.1;   0 + 0.8 − 0.7 = 0.1;   1 + 0 − 0.8 = 0.2
  λ = 0         A = 0.1, 0.1, 0.1, 0.2               only the next surprise
  λ = 1         A = 1 − 0.5, 1 − 0.6, 1 − 0.7, 1 − 0.8 = 0.5, 0.4, 0.3, 0.2   the real reward minus the guess
  λ = 0.95      A3 = 0.2
                A2 = 0.1 + 0.95 × 0.2     = 0.1 + 0.19     = 0.29
                A1 = 0.1 + 0.95 × 0.29    = 0.1 + 0.2755   = 0.3755
                A0 = 0.1 + 0.95 × 0.3755  = 0.1 + 0.356725 = 0.4567

Long attempts
  0.95^100 = 0.0059: after 100 tokens the reward's weight is under 1%
  half-life at 0.95 = ln 0.5 ÷ ln 0.95 = −0.693 ÷ −0.0513 = 13.5 tokens;  reach = 1 ÷ 0.05 = 20 tokens
  length-adaptive λ = 1 − 1 ÷ (0.05 × l):  l = 100 → 0.80 · 1,000 → 0.98 · 2,000 → 0.99 · 10,000 → 0.998
  reach = 0.05 × l:                        5 · 50 · 100 · 500 tokens
  at l = 2,000:  0.99^100 = 0.366 (against 0.006);  half-life 69 tokens
  l = 20 gives λ = 0, l = 10 gives λ = −1: the formula is for long attempts only

In code, the backward pass of step 3 is a single loop. This is the realization of everything above; the Code Lab below asks you to build it yourself:

pythonimport numpy as np
def td_errors(reward, V, V_next):
    return reward + V_next - V                    # the surprise at each token (gamma = 1)

def gae(delta, lam):
    A = np.zeros_like(delta); run = 0.0
    for k in range(len(delta) - 1, -1, -1):       # from the last token back to the first
        run = delta[k] + lam * run
        A[k] = run
    return A

print(gae(td_errors(np.array([0, 0, 0, 1.]), np.array([.5, .6, .7, .8]), np.array([.6, .7, .8, 0.])), 0.95))
# [0.456725 0.3755   0.29     0.2     ]

Long attempts break a fixed λ

With a single reward at the end, the reward's weight at distance d from the end is λ^d. Add up those weights over every distance and you get 1 ÷ (1 − λ). That sum says how many tokens back the reward's influence stays strong, about 1 ÷ (1 − λ). Call it the reward's reach.

A common choice is λ = 0.95. Its reach is 1 ÷ 0.05 = 20 tokens. The weight halves every 14 tokens, and after 100 tokens it is 0.95¹⁰⁰ = 0.006. In a 2,000-token attempt, then, almost every token hears only the critic's guesses, never the real reward. If the critic is wrong, those tokens learn the critic's mistake.

How far back does the reward reach?

The row is one attempt, token by token; the green dot at the right is the reward. The purple curve is how much of that reward each earlier token feels. Change the attempt's length, then switch to the length-adaptive λ.

attempt length2,000 tokens

λ = 0.95, the weight 0.95 to the power 100 ≈ 0.006 and the length-adaptive formula with α = 0.05 come from VAPO, the method the report follows; the report sets α = 0.05. The curves are exact arithmetic; the attempt lengths are examples.

Length-adaptive GAE

The fix is to let λ grow with the length of the sequence. The report uses λ = 1 − 1 ÷ (α × l), with α = 0.05. That is length-adaptive GAE. Where does the formula come from? Ask for a reach that is a fixed share of the attempt, α × l. Then 1 ÷ (1 − λ) = α × l, and solving for λ gives exactly λ = 1 − 1 ÷ (α l).

λ_policy = 1 − 1 ÷ (α l)

λ_policy: The λ used for the agent's own update.
α: A fixed fraction (alpha): 0.05.
l: The length of the sequence, in tokens.

l = 2,000: 1 − 1 ÷ 100 = 0.99, reach 100 tokens

So the reach is always 5% of the attempt: 5 tokens of a 100-token attempt, 50 of 1,000, 100 of 2,000, 500 of 10,000. At l = 2,000, the reward's weight 100 tokens back is 0.99¹⁰⁰ = 0.37, against 0.006 with a fixed 0.95.

One caution, in plain words. Below 20 tokens the formula gives 0 or less: l = 20 gives λ = 0, and l = 10 gives λ = −1, which makes no sense. It is meant for long sequences only. The report takes the formula and α = 0.05 from VAPO, an earlier method for training reasoning models.

The critic gets its own λ

The critic and the agent use GAE for different jobs. The critic must learn the real long-range return, so it should not shrink the reward on its way back. The agent's update prefers steadier advantages. So the two get different λ values: the critic and the policy use different λ values, which is decoupled GAE.

In VC-PPO, the method the report follows here, the critic uses λ = 1.0 and the policy 0.95. The report sets its two values to differ but does not print its own critic value.

The backwards critic

Everything above trusts the critic at least a little. So what happens when the critic is bad? Picture practice starting from a fresh critic. The reward climbs, but slowly, and some updates seem to push the agent backwards. That symptom is illustrative; the measurement and the fix below are the report's.

What do you measure? Compare the critic's guesses with the rewards the attempts really got. One number for that is how much of the spread in real rewards the critic's guesses account for, from 0 (no better than guessing the average) to 1 (perfect): explained variance. In the report's early runs it was worse than useless. The critic's guesses were negatively correlated with the real rewards: it expected success where attempts failed, and failure where they passed. Every advantage built on those guesses pointed the wrong way.

What do you change? Train the critic first. That is value pretraining, like a commentator who studies old match tapes before the season. Freeze the agent after studying, let it make many attempts, and fit the critic to their real outcomes with λ = 1, the plain Monte Carlo return. Stop when the critic's error is low and its explained variance high. Then start practice from that critic. In the report's comparison, rewards with a pretrained critic stayed higher throughout training.

The critic is half the learner. Every advantage is the real reward measured against the critic's guess. A critic that guesses backwards turns good tokens into bad news, which is why the report trains it before practice starts.

With λ fixed at 0.95, how much weight does the final reward carry 100 tokens before the end of an attempt?

About 0.95, since each token keeps 95% of the weight of the token right after it. About 0.05, since 1 − 0.95 is the share of the weight lost at each token. Exactly 1, since every token of a passed attempt gets the same reward of 1. About 0.006, since 0.95 is multiplied by itself 100 times on the way back.

Chapter 6

Never Wait for the Slowest

Keep every practice computer busy while long attempts finish, and count what that costs.

Eight practice computers start eight attempts at the same moment. Within a minute, seven have finished. The eighth has landed on a hard page and keeps scrolling. Three minutes later it is still going, and the learner, which wants all eight before it changes anything, waits. So do the other seven computers.

You have seen this already: it is the instrument at the top of the page. This chapter names what you saw there, and explains the report's fix and its price.

Names for what the hero showed

Running one attempt with the current agent to collect training data is a rollout. It is the training system's word for an attempt. The group of finished attempts used for one update is a batch. In the hero, “All 8 are done” made a batch of 8; “Any 4 are done” made batches of 4.

The usual way: everyone waits

Think of a tour bus that leaves only when every passenger is back on board. One slow shopper holds up forty people. Everyone starts together and waits for everyone: that is synchronous.

Most training code collects rollouts this way: start a group, wait for the whole group, update, repeat. How long does one round take? Here is the arithmetic, step by step:

Eight attempts of lengths L₁ to L₈ steps start together. A step takes 4 seconds.
The round ends when the longest attempt ends: T_round = 4 s × max(L_i).
The work done is 4 s × (L₁ + … + L₈) computer-seconds. The time booked is 8 × T_round.
So the busy share is (L₁ + … + L₈) ÷ (8 × max L_i), which is the mean length divided by the longest.
In a pool, a computer starts its next attempt at once, so none waits; an update comes every time B attempts finish. The cost moves from waiting to something else, which this chapter ends on.

T_round = 4 s × max(L₁, …, L₈)

busy share = (L₁ + … + L₈) ÷ (8 × max L)

L_i: Steps in attempt i.
max: The longest of them.
4 s: One step, the report's measured time per step.
T_round: How long one synchronous round lasts.

120 ÷ 384 = 31% busy

If every attempt took the same time, the busy share would be 100%. They do not. Picture a supermarket queue where most people have a basket and one has a huge trolley. Most attempts are short and a few are very long: that is a long tail. In the toy list behind the hero, the median attempt is 11 steps, the mean is 15.4, and the longest is 60. With a long tail, the longest is several times the mean, so most booked time is spent waiting.

The report names this problem directly. Batch rollouts get stuck on the long-tail attempts. That slows training, and, the report adds, it pushes the training data away from the current agent. The second half of that sentence is where this chapter ends.

The model as a server

Running a trained model to get an answer is inference. In the report, inference is a service. A program that answers requests from many others is a server, like a restaurant kitchen that cooks each order as it arrives instead of waiting for every table to order.

The agent's model runs as an asynchronous server: each part runs at its own pace and nobody waits for anybody. Each sandbox runs its own little loop. It sends its latest screen to the server and gets back the next thought and action, whenever it is ready. The report names two gains. The agent-loop code can change without touching the model serving, and inference itself runs more efficiently.

The partially filled pool

Now the fix for waiting. A waiting area where finished attempts collect is the rollout pool. The number of finished attempts that triggers an update is the minimum batch size. Once the pool holds that many, the learner updates. Unfinished attempts are not cut short: they keep running and join a later update. That is the hero's “Any 4 are done”. (The report likens its pool to one in Kimi-Researcher, an earlier agent system.)

Why not simply cut the slow attempts off and start them again later? Because a sandbox remembers. A sandbox that keeps everything between steps, the open windows, the files, the logged-in page, is stateful. It is like a chess game you cannot restart from the middle. An attempt keeps its computer until it ends, so the only choice is whether everyone else waits for it.

The whole machine

Put the pieces together and you get the report's Figure 6. Here it is in plain words first. One server runs the agent's model (the policy server). One runs the graders (the reward server). One runs the sandboxes for clicking, games, tools and code (the environment server). Finished attempts flow into the pool, and the minimum batch size triggers each update:

environment server

thousands of sandboxes: clicking, games, tools, codeeach sends its screen, gets a thought and an action back

⇅

policy server

the agent's model, answering asynchronously

↓ a finished attempt

reward server

the graders of Chapter 4: pass or fail

↓

rollout pool

finished attempts wait hereunfinished ones keep running

↓ once it holds the minimum batch size

update

the learner changes the agent; the policy server gets the new version

The device below is the hero's simulation drawn as a timeline, over 10 minutes. Each lane is one computer.

Rollouts on a timeline

Eight computers across 10 minutes. Blue bars are attempts, hatching is waiting, and each purple line is an update. Switch to the pool, then change how many finished attempts trigger an update.

update when this many are done4

A toy fleet of 8 computers with made-up attempt lengths (most 6 to 16 steps, a few 25 to 60) at the report's 4 seconds per step. The pool that updates once enough attempts have finished, and keeps unfinished ones for later, is the report's design, which it likens to Kimi-Researcher's; it names no batch size.

The price: older versions

Look closely at the pool's timeline. A long attempt may start with version 3 of the agent. Say the learner updates four times while it runs. When the attempt finally joins the pool, it describes what version 3 did, but the learner is now at version 7.

How many updates old the version that started an attempt is: that is its lag. An attempt with lag above zero is stale, like yesterday's newspaper: still informative, a little out of date. In the hero's 20-minute shift with “update at 4”, 141 of the 152 attempts learned from were stale. The average lag was 1.65 updates, and the worst was 7.

The minimum batch size is a dial between waiting and staleness. Set it to 1 and the learner updates after every attempt: 79 updates in 10 minutes, but the worst lag is 30. Set it to 8 and the worst lag is 3, with 9 updates. Chapter 7 shows how the update itself copes with stale data. The report adds no extra correction beyond that.

worked exampleRound one, all 8 together (lengths illustrative, 4 s per step real)
  lengths       = 9, 12, 7, 15, 10, 48, 8, 11 steps
  round time    = 48 × 4 s = 192 s
  work done     = 9 + 12 + 7 + 15 + 10 + 48 + 8 + 11 = 120 steps, of 8 × 48 = 384 booked
  busy share    = 120 ÷ 384 = 31%;  idle 1 − 0.31 = 69%

A 20-minute shift (the hero's numbers)
  all 8 together     6 updates, 48 attempts learned from, idle 65%
  pool, update at 4  38 updates, 152 learned from (155 finished), idle 0%
                     141 of 152 started by an older version (93%), mean lag 1.65, worst 7

The pool's dial, over 10 minutes (the device above)
  update at 1 → 79 updates, worst lag 30     update at 4 → 19 updates, worst lag 7
  update at 8 →  9 updates, worst lag 3      all 8 together → 3 updates, idle 62.5%

Here is the failure that the pool fixes, told the way you would meet it. A practice run uses eight computers per group and waits for each whole group. The fleet's dashboard shows bursts of work and long flat waits, and the learner updates only a few times an hour.

What do you measure? Log every attempt's length. Most take 6 to 16 steps; a few take 25 to 60. Each round lasts exactly as long as its longest attempt, and the computers sit idle 65% of the shift.

What do you change? Let finished attempts gather in a pool and update when 4 are ready; keep the long ones running for a later update. Idle time drops to zero, and updates go from 6 to 38 in the same 20 minutes. Then watch the new number this creates: how old the data is.

The pool's loop is short enough to write from scratch. It keeps a clock of when each computer's attempt will finish, pops the next one, and updates whenever enough have gathered. It reproduces the hero's 38 updates and 141 stale attempts from the same list of lengths:

pythonimport heapq
STEP = 4                                                     # seconds per step

def run_pool(lengths, computers=8, min_batch=4, shift=1200):
    version, pool, updates = 0, [], []
    clock = [(L * STEP, c, 0) for c, L in zip(range(computers), lengths)]  # (finish time, computer, start version)
    heapq.heapify(clock); nxt = computers
    while clock:
        t, c, v0 = heapq.heappop(clock)                      # the next attempt to finish
        if t > shift:
            break
        pool.append(v0)                                      # it joins the pool
        if len(pool) >= min_batch:                           # enough finished: update now
            updates.append([version - v for v in pool])      # the lag of each attempt used
            version += 1
            pool = []
        L = lengths[nxt % len(lengths)]; nxt += 1
        heapq.heappush(clock, (t + L * STEP, c, version))    # this computer starts again at once
    return updates

Waiting is the most expensive thing in practice. A group of attempts takes as long as its slowest member, and the slowest is often four or five times the typical one. Learning as attempts finish keeps every computer working. The price is data from an older version, which the update in Chapter 7 is built to handle.

Eight attempts of 9, 12, 7, 15, 10, 48, 8 and 11 steps start together, at 4 seconds a step, and the learner waits for all eight. What share of the computers' time in this round goes to waiting?

About 13%, because only one attempt in the eight is the slow one. About 69%, because 120 of the 384 booked steps are used for work. None, because every computer is running an attempt when the round starts. About 31%, because each computer works for 31% of the round on average.

Chapter 7

Small, Safe Updates

Change the agent a little at a time, even when its data comes from an older version.

The pool now holds four finished attempts, and three of them were started two versions ago. The learner wants to make the tokens of the passing attempts more likely. How hard may it push? Push too little and nothing changes. Push too hard and the agent turns into something the data no longer describes.

Chapter 5 gave every token an advantage: how much better than expected it turned out. This chapter is about the update itself: how the learner turns advantages into a change of the agent, and why it limits the size of every change.

How an update moves

Picture walking uphill in fog. You cannot see the top. You can feel which way the ground slopes under your feet, so you take one careful pace that way, then feel again.

Training works the same way. The number the learner tries to raise is its objective. The direction that would raise the objective fastest is the gradient: the slope under your feet. How far to move along it per update is the learning rate: the length of your pace. One update is one pace.

Why big paces break

Here is the danger. The attempts in the pool describe the version of the agent that wrote them. They say: this is what that version did, and this is how it went. If one update changes the agent a lot, those attempts no longer say what the new agent would do. The next update then builds on a wrong picture, and the one after on a worse one. Training can collapse.

So the learner needs a way to measure how far the agent has moved on each token, and a rule that stops it moving too far in one update.

The importance ratio

The measure is a ratio. Every token in the pool carries the probability that its writer gave it. The current agent's probability for that same token, divided by the probability the version that wrote it gave, is the importance ratio, r.

Work one by hand. The writer gave a token probability 0.04. The current agent gives it 0.07. Then r = 0.07 ÷ 0.04 = 1.75: the current agent is 75% more likely to write that token than its writer was. For fresh data, at the first update, the agent has not moved yet, so r = 1 for every token.

In code, the ratio is computed in logs, because probabilities of long token strings are tiny and logs keep them stable: r = exp(log p_now − log p_writer).

The report's objective

Now the update rule. Think of a speed limiter on a car. Cap the ratio inside a band, the clip window, so no update pushes too far. That is clipping. The band's lower and upper widths are ε_low and ε_high (epsilon): the ratio may go from 1 − ε_low to 1 + ε_high. This whole update rule, the ratio, the clip and a min, is proximal policy optimization, or PPO. The report writes it as its equation 4:

J(θ) = E_{(q,a)∼D, o_≤t∼π_θold} [ min( r_t(θ) Â_t, clip( r_t(θ), 1 − ε_low, 1 + ε_high ) Â_t ) ]

r_t(θ) = π_θ( o_t | q, o_<t ) ÷ π_θold( o_t | q, o_<t )

J(θ): The objective the update raises.
E: The average over the batch. The subscript says what it averages over: (q, a) ∼ D, a task and its answer drawn from the practice data, and the tokens the old version wrote for it.
q: The prompt: the task and everything shown so far.
o_t: The t-th token the agent wrote.
o_<t: The tokens before it.
π_θ: The agent now; θ (theta) is its parameters.
π_θold: The version that wrote the data.
r_t: The importance ratio for token t.
Â_t: The token's advantage, from Chapter 5.
clip: Cap a number inside a range.
ε_low, ε_high: How far below and above 1 the ratio may go before the cap.
1 + ε_high: The top of the clip window.
min: Keep the smaller of the two.

Here t counts tokens, as in the report; it is a different counter from the steps of Chapter 1.

r = 1.5, Â = +1, window 0.8 to 1.28: min(1.5, 1.28) = 1.28

Why this shape? Here is the reasoning, in five steps:

The objective for one token is r × Â. Raising a good token's probability (r above 1, Â positive) raises it.
Unlimited, the learner would push r as high as it could on one batch: the big pace that breaks things.
So cap the ratio: clip(r, 1 − ε_low, 1 + ε_high) × Â.
Take the smaller of the plain term and the capped term. For Â > 0, once r passes 1 + ε_high the term goes flat, so that token's gradient is zero: it gives no further push. For Â < 0, once r falls below 1 − ε_low, it goes flat again. In the other direction, a good token getting less likely or a bad one getting more likely, the plain term is the smaller one, so the full penalty stays.
Average over every token of every attempt in the batch. That is J(θ), equation 4.

Walk the min with numbers

Take the window 0.80 to 1.28 and four tokens. A passing token at r = 1.5 is capped at 1.28: min(1.5, 1.28) = 1.28, and it gets no further push up. A passing token at r = 1.1 is inside the window: min(1.1, 1.1) = 1.1.

Now two failing tokens, with Â = −1. At r = 1.5 the token moved the wrong way, toward a failing choice. The terms are −1.5 and −1.28, the min keeps −1.5, and the whole penalty stays. At r = 0.5 the token has already been pushed well down. The terms are −0.5 and −0.8, the min keeps −0.8, and it gets no further push down.

So the min caps only how much credit an update takes for moving a token the way the advantage wants. When the capped term wins, that token gives no push at all. The device draws exactly this, and below it, how the ratios of a real batch spread out.

Clip the ratio

Top: what one token adds to the objective, for every value of its ratio. Bottom: how the ratios of a batch spread when the data is older. Choose a passing or a failing attempt and a clip window, then age the data.

data is this many updates old3

Equation 4 is the report's. The widths 0.2 and 0.28 are VAPO's: the report uses separate lower and upper widths but does not print its own. How fast the ratios spread as data ages is a made-up random walk of 0.08 per update, the same one the Code Lab below uses.

Clip-higher

There is one more choice: how wide each side of the window is. To see why it matters, we need one more measure. How spread out the agent's choices are, high when it is unsure and tries many things, is its entropy. Trying things it is not sure of is exploration, and an agent with low entropy has stopped exploring.

As a number, entropy is H = −Σ p ln p over the agent's choices. Choices of 0.7, 0.2 and 0.1 give 0.80. Choices of 0.98, 0.01 and 0.01 give 0.11. The surer the agent, the lower the number.

Now take a rare good token, one the writer gave probability 0.04. With a symmetric window of 0.2 on each side, the objective stops rewarding its rise once it reaches 0.04 × 1.20 = 0.048. Raise the upper width to ε_high = 0.28 and the push continues to 0.04 × 1.28 = 0.0512. Making the upper width larger than the lower one is clip-higher. It lets rare good choices grow faster, which keeps the agent exploring.

The lower width stays small on purpose, so tokens are not pushed to nothing too early. The report uses separate widths, as DAPO recommends, but does not print its own values. The 0.2 and 0.28 on this page are VAPO's.

Stale data, wider ratios

Now connect this to Chapter 6. The ratio is exactly how PPO learns from an older version's attempts: it divides by the writer's probability, whoever the writer was. The older the data, the further the current agent has moved since, so the ratios spread out and more tokens hit the cap.

The share of tokens whose push is cut off by the clip is the clip fraction. In a toy where the log of the ratio drifts by 0.08 per update, data 1 update old has 95% of its ratios between 0.85 and 1.17. Data 3 updates old spreads from 0.76 to 1.31. Then 9.4% of passing tokens are capped at 1.20 (3.7% at 1.28), and 5.4% of failing tokens are capped at 0.80. The report adds no separate staleness correction beyond the ratio and the clip.

Why PPO and not GRPO

PPO needs the critic of Chapter 5. A popular alternative drops it. Instead of a critic, it compares each attempt with the average reward of a group of attempts at the same task, like grading on the curve. That is group relative policy optimization, or GRPO.

The report tried both in early runs on web fact-finding and web tasks. PPO kept higher rewards with less wobble, so the team chose PPO. The report shows the curves, not a reason. Here is the structural difference, without claiming it is the cause: a critic gives every token its own baseline, while a group average gives the whole attempt one. (PPO, GRPO and DAPO and the DAPO paper build both from the ground up; PPO covers the original clipped objective.)

When the updates come too fast

Here is how the dial of Chapter 6 can bite. To squeeze more updates out of the fleet, a team sets the pool to update after every single finished attempt. For a while the reward climbs. Then it falls off a cliff, and the agent's entropy plunges with it.

What do you measure? Log the clip fraction at every update, and the lag of the data it used. With an update per attempt, the average lag reaches about 6 updates and the worst 30, and the clip fraction climbs update after update. Most of each batch is capped and gives no push, while the rest pulls hard on data from a much older agent.

What do you change? Raise the minimum batch size back to 4, so the average lag drops below 2. Keep ε_low small and use clip-higher for the upper side. Watch the clip fraction settle. The lags here are the toy fleet's from Chapter 6; the cliff is a story, not a measurement from the report.

worked exampleThe ratio
  writer gave the token 0.04; the current agent gives it 0.07  →  r = 0.07 ÷ 0.04 = 1.75

The window
  symmetric 0.2 / 0.2:            [1 − 0.2, 1 + 0.2]  = [0.80, 1.20]
  clip-higher 0.2 / 0.28 (VAPO):  [1 − 0.2, 1 + 0.28] = [0.80, 1.28]

Four tokens through min(r × Â, clip(r) × Â), window [0.80, 1.28]
  r = 1.5, Â = +1:   min( 1.50,  1.28) =  1.28    capped: no further push up
  r = 1.1, Â = +1:   min( 1.10,  1.10) =  1.10    inside the window
  r = 1.5, Â = −1:   min(−1.50, −1.28) = −1.50    full penalty kept
  r = 0.5, Â = −1:   min(−0.50, −0.80) = −0.80    capped: no further push down

Where the cap stops pushing a rare good token that the writer gave 0.04
  symmetric:   0.04 × 1.20 = 0.048        clip-higher: 0.04 × 1.28 = 0.0512

Entropy H = −Σ p ln p
  (0.7, 0.2, 0.1):     0.7 × 0.357 + 0.2 × 1.609 + 0.1 × 2.303 = 0.250 + 0.322 + 0.230 = 0.80
  (0.98, 0.01, 0.01):  0.98 × 0.020 + 2 × 0.01 × 4.605        = 0.020 + 0.092         = 0.11

Stale data (toy: log-ratio drifts 0.08 per update)
  σ after k updates = 0.08 × √k:   k = 3 → 0.08 × 1.732 = 0.139
  95% of ratios in e^(±1.96 × 0.139) = e^(±0.272) = 0.76 to 1.31
  passing tokens above 1.20: 9.4%    above 1.28: 3.7%    failing tokens below 0.80: 5.4%

And the update itself, in code: two lines for the ratio and the clipped objective, checked on the same four tokens.

pythonimport numpy as np
def ratio_of(logp_new, logp_old):
    return np.exp(logp_new - logp_old)                      # r = p_now / p_writer, via logs

def ppo_objective(r, A, eps_low=0.2, eps_high=0.28):
    return np.minimum(r * A, np.clip(r, 1 - eps_low, 1 + eps_high) * A)

print(ppo_objective(np.array([1.5, 1.1, 1.5, 0.5]), np.array([1., 1., -1., -1.])))
# [ 1.28  1.1  -1.5  -0.8 ]

The ratio is the memory of who wrote the data. Every token keeps the probability its writer gave it. Dividing by that number is how PPO learns from an older version's attempts, and the clip is how it refuses to learn too much from any of them at once.

The clip window is 0.8 to 1.28. A token from a failing attempt (advantage −1) now has a ratio of 1.5. What does the objective give it?

−1.5, because the min keeps the full penalty for a token that moved the wrong way. −1.28, because the ratio is capped at the top of the window, as for a passing attempt. 0, because any token whose ratio falls outside the window is dropped from the update. −0.8, because a failing attempt's ratio is capped at the bottom of the window.

Chapter 8

One Agent From Many

Blend specialists into one agent by averaging their parameters, and see when that is enough.

The team now has four agents. One searches the web for facts. One gets things done on websites. One plays browser games. One works the terminal. Each is good in its own world. A user wants one agent that does all four.

This chapter is about how the report turns several practised agents into one, and about the other road it tried.

Why not practise everything at once?

The obvious plan is one big practice run over every world. The report explains why it did not do that. The worlds differ in their actions and screens, in how long attempts run, and in how costly each attempt is. A game attempt and a web-search attempt look nothing alike to the learner. The report says this makes practising all of them together in one big run unstable, and too expensive to compute.

Specialists

So the report practises one world at a time. An agent practised in one world only is a specialist. Every specialist starts from the same checkpoint: the model after studying worked examples. Then each runs its own practice: fact-finding on the web, tasks on websites, games, the terminal and tool functions, and more.

That leaves several sets of parameters, each good at one thing. How do you get one agent out of them?

Merging

Think of mixing paints. Half blue and half yellow gives green, and every drop of the mix is the same green. Average the parameters of several specialists, with weights that add up to 1: that is merging, or parameter interpolation. The result is one new model. It runs once, like any other model.

Here is the arithmetic, in four steps:

Every specialist k has parameters θ^(k): a long list of numbers, in the same order for every specialist.
Pick weights α_k ≥ 0 that add up to 1.
The merged model's i-th parameter is Σ α_k θ_i^(k): the same weighted average, number by number.
For two specialists, θ = (1 − α) θ^A + α θ^B. As α goes from 0 to 1, θ walks the straight segment from A to B.

θ^(merge) = Σ_k α_k θ^(k), with Σ_k α_k = 1 and α_k ≥ 0, k ∈ {GUI-Browsing, GUI-General, Game, GUI-SDK, …}

θ^(merge): The merged agent's parameters.
θ^(k): Specialist k's parameters.
α_k: How much of specialist k goes into the blend. The weights add up to 1, and none is negative.
Σ_k: Add over all the specialists.
k: One practice world: web fact-finding, web tasks, games, terminal and tools.

0.5 × 0.80 + 0.5 × 0.60 = 0.70 (the first parameter of a 50/50 blend, below)

worked exampleThree parameters, two specialists (all values illustrative)
  θ_browse = (0.80, −0.20, 1.10)        θ_game = (0.60, 0.40, 0.90)
  α = 0.5 / 0.5:   (0.5 × 0.80 + 0.5 × 0.60,  0.5 × −0.20 + 0.5 × 0.40,  0.5 × 1.10 + 0.5 × 0.90)
                 = (0.40 + 0.30, −0.10 + 0.20, 0.55 + 0.45) = (0.70, 0.10, 1.00)
  α = 0.25 / 0.75: (0.25 × 0.80 + 0.75 × 0.60, 0.25 × −0.20 + 0.75 × 0.40, 0.25 × 1.10 + 0.75 × 0.90)
                 = (0.20 + 0.45, −0.05 + 0.30, 0.275 + 0.675) = (0.65, 0.25, 0.95)

The toy map of the device below (scores out of 100)
  browsing specialist:  browsing 100.0, games 75.4
  game specialist:      browsing 75.4,  games 100.0
  the shared start:     browsing 75.2,  games 75.2
  α = 0.5 blend:        browsing 93.2,  games 93.2
  both stay ≥ 90 for α from 0.40 to 0.60

Hybrid RL's budget
  same total batch as a one-interface run → each interface gets half the data,
  and the hybrid still beat the click-only agent on click-only tasks

Why an average can work at all

Averaging two good models does not obviously give a good model. Two different solutions to one problem can average into nonsense. So why does the report expect it to work here? It needs one more word first.

Training pushes down a number that measures how wrong the model is: its loss. A good model sits low in the loss, like a ball resting at the bottom of a valley. Every setting of the parameters is a point on a huge landscape, and training rolls the model downhill.

Now picture two hikers who set off from the same hut. They walk in slightly different directions, but they both stay in the same valley, and the straight path between them stays low too. Models fine-tuned from one checkpoint stay in one low valley of the loss, so the straight line between them stays low as well. That is linear mode connectivity. The report relies on it, and cites earlier work for it. The device shows it in two dimensions.

Blend two specialists

A made-up model with only two parameters, so it fits on a page. The rings show where each world's score is high. Both specialists started from the grey dot. Slide the weight and move the blend along the straight line between them.

weight on the game specialist (α)0.50

A made-up two-parameter world: real agents have hundreds of billions of parameters, and the report does not publish its weights. Starting every specialist from one checkpoint and averaging them with weights that add up to 1 is the report's method (its equation 5).

In the toy, each specialist scores 100 in its own world and 75.4 in the other. The 50/50 blend scores 93.2 in both, and both scores stay at 90 or above for α from 0.40 to 0.60. Slide past that and one skill fades. That is the trade every weighting makes.

What merging bought

The report's claim is qualitative. On tasks that need several worlds at once, the merged model performs almost as well as the best specialist in each world, with no extra training. The report gives no table for this. Its conclusion adds a fair caveat: specialists can still beat the merged model on their own single benchmarks. The merged model is the balanced one.

In code, merging is a weighted sum over named parameters. Every specialist shares the same names, because they all grew from the same starting checkpoint:

pythonimport numpy as np
def merge(specialists, alphas):
    assert abs(sum(alphas) - 1) < 1e-9 and min(alphas) >= 0     # weights add to 1, none negative
    names = specialists[0].keys()                                # every specialist has the same parameter names
    return {n: sum(a * s[n] for a, s in zip(alphas, specialists)) for n in names}

browse = {"w": np.array([0.80, -0.20, 1.10])}
game   = {"w": np.array([0.60,  0.40, 0.90])}
print(merge([browse, game], [0.5, 0.5])["w"])                    # [0.7 0.1 1. ]

The other road: one agent, both interfaces

The report also tried the harder road, on a small scale. One agent practising tasks that can be solved either by clicking or with the extra functions, in the same run, is hybrid RL. The test task was fact-finding on the web, which can be done both ways.

The hybrid run used the same total batch as the single-interface runs, so each interface got only half the data. It still beat the click-only agent on click-only tasks. And its shared critic explained more of the variance in rewards, so it was a better judge of how attempts would go. The report's summary is a trade: averaging is cheap, and hybrid practice transfers more.

When the blend comes out lopsided

Here is how the trade shows up in practice. The merged agent goes out with equal weights on four specialists. Users report it is good at web tasks and weak at games.

What do you measure? Score the merged agent on each world's own test, next to each specialist. On games it sits well below the game specialist, while on web tasks it nearly matches the web specialist.

What do you change? Raise the game specialist's weight and re-test every world, because every weight change trades one world against the others. If no weighting holds every world up, practise the worlds that must share skills together, as the report's hybrid run did, and pay for the extra training. This story is illustrative; both remedies are the report's.

A shared start makes averaging possible. Two models fine-tuned from one checkpoint stay close enough that the straight line between them stays good. That is why the report can practise the worlds apart and still ship one agent.

Why does the report start every specialist from the same fine-tuned checkpoint before merging them?

So the merged agent can call each specialist in turn and keep whichever answer it likes best. So the specialists need no practice of their own, since the checkpoint already knows every world. So their parameters stay in one low valley, and an average of them is still a good model. So the merging weights can take any values at all, negative ones included, without any harm.

Chapter 9

What Practice Bought

Read the report's scores the right way: in points, against people, and against other agents.

A score of 47.5 on a desktop test means the agent finished 47.5 of every 100 tasks, each checked by a script. That one number hides a lot: which agents it beat, by how much, on which kind of task, and whether a gain is in points or in percent.

Four tests, read in plain words first

Four benchmarks score UI-TARS-2 by clicking alone, or by clicking plus its extra functions. A desktop test of 369 tasks across Ubuntu, Windows and macOS is OSWorld. A Windows only test of over 150 tasks is WindowsAgentArena. A phone test of 116 tasks across 20 apps, run on a virtual phone with each task's details changed from run to run, is AndroidWorld. And 300 tasks spread across 136 live websites make up Online-Mind2Web. On all four of these, UI-TARS-2 may also reach for the terminal and the tool calls from Chapter 1, not only click.

The scores

On these four tests UI-TARS-2 scores 47.5, 50.6, 73.3 and 88.2. Against its own last version, UI-TARS-1.5, which scored 42.5, 42.1, 64.2 and 75.8, that is 5.0, 8.5, 9.1 and 12.4 points higher. Against the very first UI-TARS, which has only a desktop and a phone number on record, it is 24.6 to 47.5 on the desktop and 44.6 to 73.3 on the phone. The report's own Table 1 lists other companies' agents on some of the same tests too: Claude-4-Sonnet reaches 43.9 on the desktop test, and OpenAI's CUA-o3 reaches 42.9 on the desktop, 52.5 on the phone and 71.0 on live websites.

Pick a test

Pick a test. Each bar is one agent's score out of 100, as the report's Table 1 lists it. Teal is UI-TARS-2, warm its earlier versions, grey other companies' agents. Then show the score UI-TARS-2 had before practice.

Every score is from the report's Table 1. Other companies' agents are shown as the report lists them; we did not check them against their makers. † marks results that used the extra terminal and tool functions. The report does not say how many steps its desktop-test run allowed.

Points, not percent

The gain divided by where you started, written as a percent, is the relative change. The report's own summary writes that practice improved the desktop test “by nearly 10.5%” and the phone test “by over 8.7%”. Set side by side, those read like the same kind of number. They are not.

The desktop test moved from 43.0 to 47.5. That gain is 47.5 − 43.0 = 4.5 points, and 4.5 ÷ 43.0 is 10.5% relative. The phone test moved from 64.6 to 73.3. That gain is 8.7 points, and 8.7 ÷ 64.6 is 13.5% relative. The report's “10.5%” is the desktop gain's relative change, written as a percent; its “8.7” for the phone is already in points, not a percent at all. Two different units, read as if they were one.

Here is where that gets a reader into trouble. Someone skims only the two headline numbers, 10.5 and 8.7, and plans a demo around a ten point gain on the desktop test. What do you measure instead? Subtract. 47.5 − 43.0 is 4.5 points on the desktop, and 73.3 − 64.6 is 8.7 points on the phone. What do you change? Write both numbers every time a gain is reported, the points and the word points, next to the relative percent if you give one too: “+4.5 points (10.5% relative)” cannot be misread. A gain without its unit is not a result.

Practice transferred

Most of what UI-TARS-2 practised was on websites: fact finding and getting things done in a browser. A desktop test and a phone test look nothing like that, so a test unlike anything the agent practised earns a name of its own, out-of-distribution. And yet after practice the desktop score rose 4.5 points and the phone score rose 8.7, even though the agent spent almost none of its practice on either kind of computer. Whatever practice taught it did not stay inside the browser.

Beyond clicking

With its extra functions switched on, UI-TARS-2 is also scored on three harder tests. Terminal-Bench asks it to work a command line; it scores 45.3, counted on 75 of the 80 tasks in the version the report used (five did not run inside the report's own sandbox). SWE-Bench Verified asks it to fix real software bugs; it scores 68.7. BrowseComp asks hard questions that need long chains of web search; with the extra functions on, it scores 29.6 in English and 50.5 in Chinese, against 7.0 and 32.1 by clicking alone, gains of 22.6 and 18.4 points.

None of these is a test UI-TARS-2 wins outright. The report's own Table 1 lists Claude-4-Sonnet higher on fixing bugs, at 72.7 against 68.7, with Claude-4-Opus close behind at 72.5. On hard English web search it lists OpenAI's o3 well ahead, at 49.7 against 29.6. Reading every number against the report's own table, rather than against a one-line summary, is what keeps a comparison honest.

Games against people

Fifteen browser mini games score UI-TARS-2 against a person playing the same game, from screenshots only, with clicks, keys and scrolling and nothing else. Divide a game's score by a human's score on that same game, multiply by 100, and the result is a normalized score: 100 means human level. On 2048, UI-TARS-2 scores 932.40 against a human's 1,024.31, which works out to 91.0.

Averaged across all 15 games, UI-TARS-2 reaches 59.8 (the report's own table prints 59.77), against 44.27 before practice, 24.73 for OpenAI's CUA and 21.61 for Claude Computer Use. That is 2.4 and 2.8 times those two agents, 35.0 and 38.2 points ahead. It lands close to human on Infinity-Loop (92.7), Tiles-master (82.7), Snake-solver (76.5) and Merge-and-double (75.2), and above human on Shapes (108.9). Practice is not a free win everywhere: three games score lower than before practice, 2048 (968.00 down to 932.40), Gem-11 (84.90 down to 63.90) and Tiles-master (3.20 down to 3.10).

Fifteen games against a human

Each bar is UI-TARS-2's score on one browser game, as a share of a human's score: 100 means human level. Switch to the averages to compare it with other agents.

Each bar is the report's Table 2 score for that game divided by the human's score, times 100; the average is the report's 59.77. Practice scored below the before-practice model on three games: 2048, Gem-11 and Tiles-master. The other agents' averages are as the report's Table 2 lists them.

Games it never practised

LMGame-Bench is a separate set of six classic games the report uses to check whether skill carries over to games UI-TARS-2 never trained on. Averaged over three runs, with how much the result moved between runs written as the standard deviation, ±, it reaches 117.1 ± 1.9 on 2048, within 9% of OpenAI o3's 128.2. On Candy Crush it scores 163.2, ahead of o3's 106.0 and behind Gemini-2.5 Pro's 177.3. On Super Mario Bros it reaches 1,783.2 against o3's 1,955.0. It is weaker on games that reward long planning: 16.0 on Tetris against o3's 31.0, and 0.3 on Sokoban against o3's 2.0.

worked examplePoints, not percent
  desktop   43.0 → 47.5:  47.5 − 43.0 = 4.5 points;   4.5 ÷ 43.0 = 10.5% relative
  phone     64.6 → 73.3:  73.3 − 64.6 = 8.7 points;   8.7 ÷ 64.6 = 13.5% relative
  websites  83.7 → 88.2:  88.2 − 83.7 = 4.5 points;   4.5 ÷ 83.7 =  5.4% relative

UI-TARS-1.5 → UI-TARS-2, in points
  desktop +5.0 · Windows +8.5 · phone +9.1 · websites +12.4

The extra functions on hard web search
  English 7.0 → 29.6 = +22.6 points        Chinese 32.1 → 50.5 = +18.4 points

Normalized game score
  2048       932.40 ÷ 1,024.31 × 100 = 91.0       Hex-frvr 2,389 ÷ 5,276.85 × 100 = 45.3
  mean of the 15 games = 59.77
  59.77 ÷ 24.73 = 2.42 (OpenAI CUA)            59.77 ÷ 21.61 = 2.77 (Claude Computer Use)
  59.77 − 24.73 = 35.0 points                   59.77 − 21.61 = 38.2 points
  after practice − before practice = 59.77 − 44.27 = 15.5 points
  2048 before practice: 968.00 ÷ 1,024.31 × 100 = 94.5, so practice cost it 3.5 points here

Practice transferred. Most of the practice tasks were on websites, yet the desktop score rose 4.5 points and the phone score 8.7. Skills practised in one place showed up in places the agent never practised.

The desktop test score rose from 43.0 to 47.5 after practice. Which sentence is correct?

It rose by 10.5 points, which the report rounds to nearly 10.5%. It rose by 4.5%, which is 10.5 points on the 100-point scale. It rose by 47.5%, because the score is a share out of 100. It rose by 4.5 points, which is 10.5% of the starting score.

Chapter 10

Inside the Training Run

Read the curves of a live practice run, and trade a little accuracy for a lot of speed.

A practice run is in its second day. No one can watch thousands of attempts. So the team watches a few lines on a dashboard instead: the reward, how spread out the agent's choices are, how much it writes before each action, and how many steps it takes.

Training curves

A line of one measurement plotted against updates is a training curve, like a patient's chart on a hospital wall. One note before the curves themselves: the report's own figures label their x axis “Step”. On this page that axis is updates, since on this page a step always means one look, one thought and one action (Chapter 1).

Reward rises everywhere

Across roughly 80 updates of practice on web fact finding and on web tasks, the reward climbs. Across roughly 350 updates on the 15 browser games, the average reward climbs too. Two very different worlds, the same basic shape: more practice, more reward.

Entropy often rises instead of falling

In many reasoning-model training runs, entropy falls as practice goes on: the model settles on what works and stops trying other things. In the report's web and game runs it often did the opposite. Entropy rose, which means the agent kept trying new ways to click and scroll even while its reward climbed. With the terminal and tool functions switched on, the pattern flips again: entropy fell steadily across about 250 updates while the score rose.

Here is where that number, by itself, misleads a reader. Thirty updates into a run, the entropy curve is climbing. Someone watching the dashboard calls it: the agent is falling apart, stop the run. What do you check first? Put the reward curve next to it. If the reward is climbing too, the agent is trying more things and winning more often, exactly the pairing the report saw in its own web and game runs. If the reward is flat or falling while entropy climbs, that is the moment to look at the clip fraction and the data's lag from Chapter 7, and at the judge from Chapter 4. What you change follows from which story the two curves tell together: nothing, in the first case; fresher data or a stricter judge, in the second.

Thoughts get shorter, except when the game gets harder

How many tokens the agent writes in its thought before each action is its think length. On web tasks it fell steadily through practice: once the right click is obvious, a long explanation stops paying for itself. On games it does something different. It rises whenever practice moves to a harder level, then falls again as the agent masters that level, because the games themselves get harder step by step as training goes on.

Fewer steps to finish, yet more steps still help

On web tasks the number of steps a task takes falls as the reward rises: the agent reaches the goal more directly as it practises. The most steps an attempt may take is its step budget, and the report suggests building a step budget straight into the reward for runs where steps per task grow without helping. At the same time, scoring higher by allowing more steps at test time is inference-time scaling, and the report sees it working in both directions at once. On the 15 games the score keeps rising in a staircase as the step budget grows toward about 1,000, while OpenAI's CUA and Claude 3.7 flatten out well before that; on the desktop test the score keeps rising too, up to about 100 steps, even though practice had already taught the agent to finish most tasks in fewer steps than that.

Read the training dashboard

Pick a curve. Each one is drawn in the shape the report's figures show; the heights are made up, so read the direction, not the numbers.

Shapes only. The directions, and where the report's own axes end, come from its Figures 7 to 11 and 14; the heights here are made up, and none should be read as the report's values.

Per-game patterns

Not every game learns the same way. Some reach the human reference during practice: 2048, Infinity-Loop, Emoji-sort-master, Tiles-master and Shapes. Some start from almost nothing and climb from there: Free-the-key and Yarn-untangle. And some stall, or slip and recover only a little: Gem-11 and Hex-frvr.

A speed trade

Storing the model's numbers with fewer bits is quantization, the same idea as rounding every price to the nearest pound. The model as it is normally stored, every number at its usual size, runs at full precision. One particular recipe stores the model's weights in 4 bits and its activations, the numbers flowing between layers, in 8 bits: W4A8. The report measured the trade directly. Tokens per second, how fast the model writes, rises from 29.6 to 47, 1.59 times faster. Latency, how long one step takes from screenshot to action, falls from 4.0 to 2.5 seconds, 37.5% less. And the desktop-test score falls from 47.5 to 44.4, 3.1 points lower.

Faster or better

The same agent stored two ways. Set how many steps an attempt takes and compare the time per attempt and the desktop-test score.

steps per attempt50

Tokens per second, seconds per step and both desktop-test scores are the report's measurements (Section 3.3). Time per attempt and attempts per hour are our arithmetic from them.

worked exampleW4A8: 4-bit weights, 8-bit activations
  speed      47 ÷ 29.6 = 1.59 times the tokens per second
  latency    (2.5 − 4.0) ÷ 4.0 = −37.5% per step
  accuracy   47.5 − 44.4 = 3.1 points lower on the desktop test

A 50-step attempt (our arithmetic from the report's numbers)
  full precision   50 × 4.0 s = 200 s  →  3,600 ÷ 200 = 18.0 attempts an hour per computer
  W4A8             50 × 2.5 s = 125 s  →  3,600 ÷ 125 = 28.8 attempts an hour
  saved            200 − 125 = 75 s per attempt

Practice made attempts shorter, yet more steps still help. Training cut the steps the agent needs on web tasks, but at test time a bigger step budget still raised its score, on games up to about 1,000 steps. Fewer wasted steps and more chances do not conflict.

W4A8 cuts a step from 4.0 to 2.5 seconds and costs 3.1 points on the desktop test. How much time does it save on a 50-step attempt?

1.5 seconds, the saving on a single step of the attempt. 75 seconds, since each of the 50 steps saves 1.5 seconds. 37.5 seconds, the 37.5% cut applied to a 100-second attempt. 155 seconds, the 3.1 points multiplied by the 50 steps.

Chapter 11

The Map

Carry the numbers, the checklist and the whole loop as code, and find where this sits in the path.

UI-TARS-2 is a native GUI agent: one model that reads screenshots, writes a thought and an action as text, and works through clicks, keys, a terminal and tools inside one sandbox. It starts from a large pre-trained model and grows through rounds of reading, studying worked examples and practice, where a validator sends every new attempt either to the worked examples or to the background reading. Practice uses verifiable rewards where it can get them and the agent itself as a judge where it cannot, and updates the agent with PPO, helped by a critic trained first, a λ that grows with length, and a clip window raised at the top. Thousands of sandboxes feed a pool that updates as soon as enough attempts finish. Specialists practised one world each are merged by averaging their parameters. The result: 47.5 on a desktop test, 50.6 on Windows, 73.3 on a phone test, 88.2 on live websites, and 59.8% of human level across fifteen games.

The numbers that matter

Quantity	Value	Why it matters
Model	532M vision encoder + Mixture-of-Experts language model, 23B of 230B parameters per token	10% of the model works on each token
Base	the pre-trained Seed1.6 checkpoint and its later training data	practice starts from a strong reader
One step	4.0 s at full precision, 2.5 s with W4A8	sets fleet size and practice speed
Released click format	a Thought line and an Action line, `click(point='<point>x y</point>')` (UI-TARS-1.5)	a click is text, mapped back to the screen
Fleet	“several thousand” virtual machines, a manager at “several thousand” requests a second	practice is a systems problem
Web task set	690 websites	where most practice happened
Judge	F1 83.8 on 300 labelled attempts, more false passes than false fails	an imperfect judge still worked
Length-adaptive GAE	λ = 1 − 1 ÷ (0.05 × l)	the reward's reach is 5% of the attempt
Clip	separate lower and upper widths; VAPO uses 0.2 and 0.28	rare good tokens can still grow
Desktop / Windows / phone / websites	47.5 / 50.6 / 73.3 / 88.2	+5.0 / +8.5 / +9.1 / +12.4 points over UI-TARS-1.5
Practice over studying	+4.5 desktop, +8.7 phone, +4.5 websites (points)	gains even where it never practised
With extra functions	Terminal-Bench 45.3 (75 of 80 tasks), SWE-Bench Verified 68.7, BrowseComp 29.6 English / 50.5 Chinese	clicking alone gets 7.0 / 32.1 on BrowseComp
Games	59.8% of human on 15 games; 2.4× OpenAI CUA, 2.8× Claude Computer Use	Shapes 108.9 beats the human
W4A8	1.59× tokens per second, −37.5% per step, −3.1 desktop points	a speed trade, measured

The checklist

Ten steps for anyone building their own practice loop, in the order the report's own choices suggest:

Map every click back with the size the model saw; log the target and the click for a sample of steps.
Give every session a lease, and alert on sessions open longer than any attempt should run.
Put the browser and the terminal on one disk if tasks mix clicking and files.
Route every new attempt: passes to worked examples, failures to reading. Audit a sample of each round's passes by hand.
Measure your judge on attempts people labelled; report precision, recall and F1, not only accuracy.
Train the critic before practice and check its explained variance; make λ grow with attempt length.
Do not wait for the slowest attempt: update from a pool, and log every attempt's lag.
Watch the clip fraction against the lag; raise the minimum batch size if both climb.
Read entropy next to reward, never alone.
Report gains in points, with the starting score, and attribute other agents' numbers to their source.

The whole loop, as code

Names here are made up; each line points to the chapter that explains it.

python# One flywheel round with practice inside it (made-up names; chapter numbers in comments)
def practice(agent, critic, tasks, fleet, min_batch=4, eps_low=0.2, eps_high=0.28, alpha=0.05):
    pool = []
    for attempt in fleet.run_forever(agent, tasks):          # Ch 2 sandboxes, Ch 6 never wait
        attempt.reward = grade(attempt)                        # Ch 4: script, judge with reference, or ORM
        pool.append(attempt)
        if len(pool) < min_batch:
            continue
        for a in pool:                                         # Ch 5: spread one reward back
            lam = 1 - 1 / (alpha * len(a.tokens))
            a.adv = gae(td_errors(a.rewards, critic(a), critic.next(a)), lam)
        agent.raise_objective(pool, eps_low, eps_high)         # Ch 7: ratio, clip, min (equation 4); 0.2 / 0.28 are VAPO's
        critic.fit(pool, lam=1.0)                              # Ch 5: decoupled, the critic's own λ (1.0 in VC-PPO)
        pool = []

def flywheel_round(model, data):
    model = train(model, data.ct)                              # Ch 3: reading
    model = train(model, data.sft)                             # Ch 3: studying worked examples
    critic = pretrain_critic(model, lam=1.0)                   # Ch 5: the critic learns first
    model = practice(model, critic, data.tasks, fleet)         # Ch 4 to 7
    for s in model.sample_attempts(data.tasks):                # Ch 3: rejection sampling
        (data.sft if validator(s) == 1 else data.ct).append(s)
    return model, data

sft_model = train(train(base, data.ct), data.sft)                            # one shared starting checkpoint
critic0 = pretrain_critic(sft_model, lam=1.0)
specialists = [practice(sft_model, critic0, world_tasks[w], fleet) for w in worlds]   # Ch 8: one practice run per world
agent = merge(specialists, alphas)                                           # Ch 8: equation 5, weights add to 1

The limits, in the report's own terms

No joint practice across every world at once; merging specialists afterwards stands in for it.
The judge passes too many failures: more false passes than false fails, even at F1 83.8.
Weaker on games that reward long planning, such as Tetris and Sokoban, and stalled on Gem-11 and Hex-frvr.
Practice lowered three game scores below the before-practice model: 2048, Gem-11 and Tiles-master.
Terminal-Bench is scored on 75 of the 80 tasks in the version the report used.
Steps per task can grow unless the reward includes a step budget.
W4A8 buys 1.59× the writing speed for 3.1 desktop points.
The weights are not public: the only open checkpoint in the repository is UI-TARS-1.5-7B.

The paper

“UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning”, ByteDance Seed, a long alphabetical author list, correspondence Yujia Qin and Guang Shi; arXiv 2509.02544, first posted 2 September 2025, revised 5 September 2025. Code and desktop app: github.com/bytedance/ui-tars and github.com/bytedance/UI-TARS-desktop; a demo lives at seed-tars.com/showcase/ui-tars-2. Borrowed parts, named plainly: ReAct for the look, think, act loop; VAPO and VC-PPO for length-adaptive and decoupled GAE and for value pretraining (arXiv 2504.05118 and 2503.01491); DAPO for clip-higher, taught chapter by chapter in this site's DAPO Veanor; Kimi-Researcher for the idea of a partially filled pool; GPT-4o as the fact-question judge during training; and linear mode connectivity, cited by the report, for why merging works.

Where this sits in the path

This lesson sits in Act 8, training, of the path “Computer Use & Multimodal Agents”. Just before it: PPO, GRPO and clip-higher, and RL for robot foundation models, with two lessons coming soon to this path, Agentic RL (training a GUI agent by trial and error) and DigiRL (practice on real Android phones). Just after it: self-play RL, Absolute Zero. Also coming soon to this path: Computer-Use Agents (the screenshot-in, click-out loop), GUI Grounding (how a model finds the button), Agent Environments and OSWorld (tasks, resets and checkers), Sandbox Fleets (a thousand desktops that never leak), and Trajectory Data (where attempts come from).

Keep going

PPO, GRPO and clip-higher, from the ground up.
DAPO: the clip-higher paper, chapter by chapter.
PPO: the original clipped objective.
ReAct: the look, think, act loop this agent's steps follow.
Policy gradients and actor-critic methods: the rule under practice, and where critics come from.
MCP: the tool standard the agent's extra functions can call.
Vision-language models and Qwen2-VL: how a model reads a screenshot at all.
Agent evaluation and evaluation statistics: reading agent scores with error bars.
EnvHarness: making practice environments push back.

Practice at this scale is three systems in one. A data system that wastes no attempt, a grading system that is trusted but checked, and a rollout system that never waits. The learning rule, PPO with a few careful changes, is the smallest of the three.

Now press Teach and explain, out loud and from memory, why a trainer that waits for every attempt wastes most of its computers, and what the pool costs in return. Then go back to the computers and switch between the two ways of learning one more time.