DigiRL

Picture a phone on a desk. A message arrives: Go to costco.com, search for "bose soundsport free" and select the first entry. Nobody touches the phone. A program looks at the screen, taps the browser, types the address and presses Enter.

Then a box pops up over the page: Sign in for member prices. The program taps where the search box used to be. Nothing happens. It taps again, and again, until it has used every move it was allowed.

That program learned only by copying people. A program that only ever copied people has never seen its own mistakes, so it has no idea how to get out of one. A program that practises, and is told each time whether it succeeded, can learn the way back. That difference is this whole paper.

The words for the scene

Let's name the parts, because we use them all the way through. A program that uses the phone by itself is an agent. It looks at the screen and decides what to press, like a remote hand on the glass.

The one sentence that says what to do is the task, like a shopping list with one line. A picture of everything on the screen at one moment is a screenshot, a photo of the phone. One thing the agent does is an action: a tap, a swipe, typing some text, or pressing Back, Home or Enter. One screenshot in and one action out is one step, like one turn in a board game.

Done well, the costco task takes eight actions, so eight steps:

1. Tap the Chrome icon on the home screen.
2. Tap the address bar of the new tab.
3. Type "costco.com".
4. Press Enter, and the shop's home page opens.
5. Tap the search box.
6. Type "bose soundsport free".
7. Press Enter, and a list of results appears.
8. Tap the first result, and the product page opens. Done.

No attempt may run forever. Each one has a step budget: the most steps one attempt may take, like a fixed number of moves in a puzzle. The paper allows 20 steps for shopping tasks like this one. It allows 10 for general questions such as "What's on the menu at Burger King?". When the budget runs out, the attempt has failed.

Two ways to build an agent, before practice

The first way is to ask a giant. A vision-language model (VLM) reads a picture and some text together and writes text back. Think of a friend describing a photo to you over the phone. The largest of these models have read much of the web.

You can give such a model instructions in words, without training it any further. That is called prompting. It is like briefing a contractor instead of training an employee: fast, but the contractor only knows what they already knew.

The paper prompted two of the largest commercial models of the time (GPT-4V and Gemini 1.5 Pro). It even drew numbered boxes on each screenshot, so a model could answer "press box 7" instead of naming a position. They finished under 20% of the tasks they were tested on. Letting them look up notes about each app raised that test score by at most 5 points.

Their typical failure is telling. Given "Go to newegg.com, search for alienware area 51, and select the first entry", one of these agents typed "alien area 51 ebay" into Google's search box. Then it decided it had made progress. A model that has read the web has not practised using a phone.

The second way is to copy people. A demonstration is a recording of a person doing a task: every screenshot they saw and every action they took, like a cooking video. Imitation learning trains the agent to pick, on each recorded screen, the action the person picked. Think of tracing over someone's handwriting.

In practice you start from a model that already exists and train it further on the demonstrations. Taking a model that exists and training it further on labelled examples is supervised fine-tuning (SFT): extra lessons for a student who can already read. The recordings here come from a large public collection of people using Android phones: 715,000 recorded attempts over 30,000 different instructions.

A small model trained this way finished 17.7% of the shopping tasks it was tested on, and 14.6% of the general questions. It is the agent labelled Copies people on the phone at the top of the page.

Why copying breaks

Copying has a blind spot. The people who recorded demonstrations mostly did the task well. So the recordings are full of good screens and nearly empty of bad ones.

When the agent lands on a screen that no person in the recordings ever had to handle, it is off the demonstrations. It is a learner driver who only ever practised on empty roads, now facing a roundabout at rush hour. The driver has no example to copy.

So one mistake makes the next one more likely. A wrong tap leads to an unfamiliar screen. On that screen the copier has nothing good to imitate, so its next action is more likely wrong too. This is called compounding errors: one wrong turn in a city you do not know leads to streets you know even less. (Imitation learning covers compounding errors and their classic fix in depth.)

Here is the arithmetic, with toy numbers that are illustrative, not the paper's. Say each screen brings a surprise 1 time in 10. Say the copier taps the wrong thing 10.5 times in 100 on a clean screen. A step goes cleanly only if neither happens, and the two chances multiply:

0.9 × 0.895 = 0.8055 per step;  0.8055⁸ = 0.177 (eight clean steps in a row)

A copier that can never get back on track needs all eight steps clean, one after another. Multiply 0.8055 by itself eight times and you get 0.1772, about 17.7%. That is the copier's measured shopping score: the toy is tuned to land there. Even with no surprises at all, the copier finishes only 0.895⁸ = 0.412 of the time, because its own wrong taps strand it too.

An agent that can find its way back after a slip does not need eight clean steps in a row. It needs to recover more often than it slips. Drag the slider and watch how differently the three agents age as surprises get more common.

Read the gap at 20%. The copier falls to 6.9%, while DigiRL still finishes 46.3%. The difference is not a sharper eye or a bigger brain. It is one extra move: the way back. In the toy, the two learners find it 58% and 65% of the time, and the copier never does.

A phone in the wild is not a recording

On a live phone, surprises are not rare. The world is stochastic: the same tap does not always give the same next screen, like rolling a die. The paper names three sources of surprise.

First, distractors: pop-up ads, login requests, and search results that come back in a different order. Second, glitches: pages that load only halfway, and sites that block you for a while. Third, the apps and sites themselves change, and that one needs its own word.

The paper tells one copier's story. Asked "what's on the menu of In-n-Out", the agent switched on voice input by accident. It never left that page. It tapped until its steps ran out.

The third source is slower. Apps and websites change from week to week, like a shop that rearranges its shelves. We call such a world non-stationary. The paper measured it. The authors trained an agent on attempts from 1 to 3 June. Then they either froze it, or let it keep learning on fresh attempts from 7 to 11 June.

The frozen agent got gradually worse. The one that kept learning held its level. Chapter 8 lets you scrub through those days yourself.

So the skill that matters most on a live phone is recovery: getting back on track after something went wrong, like finding the road again after a wrong turn. Recordings of people doing a task well cannot teach it, because they almost never show a mistake.

The fix: practice with a score

How do people learn a video game? They play. They lose, notice what went wrong, and try something else next time. Learning by trial and error from a score is reinforcement learning (RL): try, get judged, do more of what worked.

An agent that practises on its own phone sees its own mistakes. The score tells it which of its escapes worked. That is exactly the experience the recordings lack.

Practice raises four problems, and they are this lesson's road. Who gives the score, when nothing inside a shopping site says "task complete"? (Chapter 2.) An attempt has many actions and one score: which actions earned it? (Chapters 3 to 5.) Which tasks are worth practising? (Chapter 6.) And how do you practise fast enough on real software? (Chapter 8.)

DigiRL answers all four problems. It takes the same small model the copier started from. That model has about 1.3 billion parameters: the adjustable numbers inside a network, like the knobs on a giant mixing desk.

DigiRL trains that model in two stages. First it learns offline: from a pile of earlier attempts, without touching a phone, like studying recordings of old matches. Then it learns online: while it practises on a fleet of simulated phones that browse the live web, like playing matches between lessons. The paper calls the two stages together offline-to-online reinforcement learning.

A large model acts as the judge and scores every attempt. On the shopping test, DigiRL finishes 67.2% of the tasks, up from the copier's 17.7%. That is 49.5 points more, with the same model. On the general questions it finishes 71.9%.

67.2 − 17.7 = 49.5 points (same 1.3-billion-parameter model, shopping test)

The failure, read like an engineer

The symptom. The copier looks fine when you replay the recordings in a lab, and poor on a live phone.

The measurement. Sort its failed attempts by what went wrong. The paper did this. The largest group was "fails to recover from a mistake".

The fix. Give the agent experience of its own mistakes. Only practice can provide it.

Here is the road, in order. Each chapter explains one part of the phone at the top of the page.

1. Screen in, tap out: what the agent sees, and how a line of text becomes a tap.
2. Who says it worked?: the judge behind the green tick.
3. Scores for screens: one score at the end, and a value for every screen.
4. Copy the good parts: what "copies its wins" means, and why it is not enough.
5. Which tap earned it?: grading every step. This is the heart of the paper.
6. Practise at the edge: which tasks are worth learning from.
7. Teaching the critics: how the two critics learn their scores.
8. Offline, then online: where practice attempts come from, and why practice never stops.
9. What it buys: the numbers behind the tally at the top.
10. The map: the numbers, the code, the limits, and where to go next.

Freeze the costco task at step 8. The phone shows a list of results. The first one says Bose SoundSport Free wireless headphones. In the next fraction of a second, a picture and a sentence go into the agent, and a tap comes out. What exactly happens in between?

Here is the short answer. Every step, the agent turns one picture and one sentence into one line of text. A small program then turns that text into a finger on the glass. This chapter follows that path, one hop at a time, with the real sizes and the real strings.

The phone is a program

DigiRL never touches a physical phone. It uses an emulator: a program that pretends to be a phone on an ordinary server, running the real Android software. Think of a flight simulator, but for a phone. The apps are real, the web pages are real, and the internet connection is real. Only the hardware is pretend.

The screen is made of pixels, the single dots that make up a picture, like the tiles of a mosaic. This emulated phone's screen is 1,080 pixels wide and 2,280 tall. That is 2,462,400 pixels in all.

Every attempt starts from a fresh copy of a saved phone, at the home screen. Why not real phones? You can run dozens of emulators on rented servers, and reset each one to a fresh copy at any time. Practice needs thousands of attempts, so that speed matters a great deal (Chapter 8 has the fleet's numbers).

What the agent may do

The full list of moves the agent is allowed to make is its action space, like the legal moves of a game. Here it has three kinds of move. It can touch the screen at a position. It can type a string of text. And it can press one of three special buttons: Home, Back or Enter.

A position is not written in pixels. It is written as normalized coordinates: a fraction of the screen, from 0 at the left or top edge to 1 at the right or bottom edge. You say "halfway across" instead of "540 dots across". The same fraction works on any screen size.

A touch comes in two kinds. A tap presses and lifts at one spot, like a poke. A swipe presses at one spot and lifts at another, like a drag. The model writes both the same way: as two points, where the finger pressed and where it lifted.

Inside the model's text, each move has a name. A touch is DUAL_POINT, named for its two points. Typing is TYPE. The three buttons are PRESS_HOME, PRESS_BACK and PRESS_ENTER. Two more names end the attempt from the agent's side: STATUS_TASK_COMPLETE ("I am done") and TASK_IMPOSSIBLE ("this cannot be done").

The action string, byte for byte

The line of text the model writes to name its move is the action string, like a written order slip. Here is the one for tapping the first result, exactly as the released code formats it:

action stringAction Decision: "action_type": "DUAL_POINT", "touch_point": "[0.6200, 0.5000]", "lift_point": "[0.6200, 0.5000]", "typed_text": ""

Read it field by field. The model may write other text first, such as notes to itself. The environment ignores all of it and reads only what follows the marker Action Decision: .

"action_type": "DUAL_POINT" says this move is a touch. "touch_point" is where the finger presses, and "lift_point" is where it lifts. Here they are the same point, so this touch is a tap. "typed_text" is empty, because a tap types nothing. Each number has exactly four decimals.

Now the trap. The two numbers in each point are written y first, then x. So [0.6200, 0.5000] means 62% of the way down the screen and 50% of the way across. The released code flips each pair to (x, y) before it uses it. Forget the flip, and every tap goes somewhere else.

From the string to a finger

Turning fractions into pixels is one multiplication per number. The x fraction multiplies the screen's width, and the y fraction multiplies its height:

x = 0.5000 × 1,080 = 540.0  ·  y = 0.6200 × 2,280 = 1,413.6 (y is written first)

The code passes these numbers on unrounded, so the tap goes to x 540.0, y 1,413.6. That is on the first result, which fills the band from 53% to 71% of the way down our mock page.

Next, tap or swipe? The rule in the code is simple. Measure the squared distance between where the finger pressed and where it lifted. If it is under 10 square pixels, the phone gets a tap. Otherwise it gets a swipe.

(540.0 − 540.0)² + (1,413.6 − 1,413.6)² = 0 < 10 → tap

Under 10 square pixels means under √10 = 3.16 pixels apart. So a finger that drifts 3 pixels still taps, and one that drifts 4 swipes. A real swipe is far bigger. Pressing at [0.8000, 0.5000] and lifting at [0.3000, 0.5000] moves the finger from y 1,824.0 up to y 684.0. That is 1,140 pixels, and 1,140² = 1,299,600 square pixels: a swipe, by a mile.

The three buttons need no position. Home is Android key code 3 and Enter is key code 66; Back has its own call. The program that performs all of this is the driver: it turns an action string into a real touch on the emulator, like the hand inside a puppet. Here the driver is Appium with its UiAutomator2 helper, one per emulator.

Try the whole path yourself. Drag the crosshair to choose a point, and the device writes the string the model would write for it. Then it decodes that string the way the driver does.

Notice that drift is measured after the round trip through text. A 4-pixel drift becomes a lift point rounded to four decimals, and the decoded points land 4.1 pixels apart: 16.8 square pixels, a swipe. The rule sees only the string.

What the agent sees

The screenshot is 2,462,400 pixels, with three colour numbers each. No small model reads that many numbers directly. So the first thing that happens to a screenshot is compression.

An image encoder is a network that turns a picture into a short list of numbers that sums it up, like a fingerprint of the picture. That list is a feature vector. Here each screenshot goes through an encoder borrowed from another model (BLIP-2's vision part) and comes out as one list of 1,408 numbers.

This encoder is frozen: its numbers are never changed during training, like a lens bolted in place. It holds 1.1 billion parameters, most of the agent's 1.3 billion. Training changes only the rest of the agent. The authors kept the encoder frozen for every method they trained.

Say plainly what this costs. One fingerprint per screen squeezes tiny text and exact positions into 1,408 numbers. There are no image pieces the model could point at. When the agent names a position, it writes the coordinates as text, from what the fingerprint and the task suggest.

What the agent reads

The other input is text. A language model is a network that reads text and writes text, one small piece at a time, like autocomplete that keeps going. Each small piece is a token: a word or part of a word, like one tile in a word game.

The text the model reads before it writes is the prompt, like the question on an exam paper. DigiRL's prompt is short. It holds the last action, the task, and an end mark:

Previous Actions: "action_type": "DUAL_POINT", "touch_point": "[0.1200, 0.5000]", "lift_point": "[0.1200, 0.5000]", "typed_text": "" Goal: Go to costco.com, search for "bose soundsport free" and select the first entry</s>

The prompt at step 6, just after the agent tapped the search box. Only the single last action is included.

The </s> at the end is the model's end-of-text mark. Only the last action is included, never the full history. So the agent's memory of what it did is one action long.

What the agent knows at a step is its state. One screenshot cannot show everything: is that page still loading, or broken? So the paper uses the last two screenshots as the state, like the last two frames of a film. In the released code, the critics of Chapter 5 read both fingerprints side by side, 2 × 1,408 = 2,816 numbers. The agent itself reads the current fingerprint plus the last action in its prompt.

Inside the agent

The agent's decision rule is its policy: given the screen and the task, which action to take, and with what chance. Think of a person's habits. DigiRL's policy is a model called AutoUI-Base: a 200-million-parameter language model (T5) plus the frozen 1.1-billion-parameter image encoder. That makes about 1.3 billion parameters in all.

How does a language model look at a picture? Through attention, the model's way of looking things up. Each position asks a question and pulls in whatever matches from elsewhere, like looking up an index. Here each word position looks up the screenshot's fingerprint. Then a gate decides how much of what it found to blend in:

h

The language model's numbers at one word position.

v

What that position picked up from the screenshot's fingerprint, by attention.

[ h ; v ]

The two lists placed end to end.

W

A learned matrix: trained numbers that mix the joined list.

σ

The squash: it turns any number into one between 0 and 1.

g

The gate: how much picture to blend in at this position.

⊙

Multiply number by number, position by position.

If g is 0, the position ignores the picture. If g is 1, the position is replaced by what it saw. In between, it keeps some of each. Training sets W, so the model learns, position by position, how much the picture should count.

Then the model writes, one token at a time, up to 128 new tokens per step. At each token it does not simply take the likeliest choice. It samples from its chances, at a sampling temperature of 1.0. The temperature sets how much chance the policy gives to less likely tokens, like how adventurous a menu choice is. At 1.0 the agent uses exactly the chances it learned, so the same agent can succeed on one try and fail on the next.

Two bugs worth knowing

The symptom. Every tap lands somewhere plausible but wrong, as if mirrored across the screen's diagonal.

The measurement. Log the decoded pixel for each step, and draw it on the saved screenshot.

The cause. The string is [y, x], and your harness read it as (x, y). Then [0.62, 0.50] lands at x = 0.62 × 1,080 = 669.6 and y = 0.50 × 2,280 = 1,140.0. That is √(129.6² + 273.6²) = 302.7 pixels from the intended spot, inside the ad banner on our mock page.

The fix. Flip the pair, as the released code does. Then check the drawing again.

A second story comes from the emulator. Sometimes it returns a blank white screenshot before the page has drawn. The released code counts the near-white pixels. If more than 2,300,000 of the 2,462,400 are near-white, that is 93.4% of the screen, it waits 5 seconds and tries again, up to 5 times. Without this guard, the agent would act on a blank screen and the judge would score a blank page.

The step, as code

Here is our sketch of the decoding. The released code does the same in two places: one function parses the model's text, and another multiplies by the screen size and calls the driver.

pythonimport json

def decode(action_str, W=1080, H=2280):
    d = json.loads("{" + action_str + "}")
    ty, tx = json.loads(d["touch_point"])      # the string is [y, x]
    ly, lx = json.loads(d["lift_point"])
    touch, lift = (tx * W, ty * H), (lx * W, ly * H)
    dist2 = (touch[0] - lift[0]) ** 2 + (touch[1] - lift[1]) ** 2
    return d["action_type"], ("tap" if dist2 < 10 else "swipe"), touch, lift

decode('"action_type": "DUAL_POINT", "touch_point": "[0.6200, 0.5000]", '
       '"lift_point": "[0.6200, 0.5000]", "typed_text": ""')
# ('DUAL_POINT', 'tap', (540.0, 1413.6), (540.0, 1413.6))

After each step, the environment hands back an observation, a small dictionary. Here is its shape, with each key explained below:

observation{
  "prompt":        "Previous Actions: ... Goal: Go to costco.com, ...</s>",
  "image_feature": tensor[1408],                     # the screenshot's fingerprint
  "task":          "Go to costco.com, search for ... and select the first entry",
  "image_path":    ".../<image_id>_<step>.png",         # the saved screenshot
  "video_path":    "..."                                # the screen recording
}

prompt is the text the policy reads next. image_feature is the 1,408-number fingerprint of the new screenshot. task is the task sentence on its own, which the judge needs. image_path points to the saved picture, named by attempt and step, so you can replay any attempt later. video_path points to a recording of the screen.

The agent has tapped the first result. A product page is open: headphones, a price, an Add to Cart button. Did it succeed? You know at a glance. The program does not. Nothing inside the phone says task complete, and the shop's website certainly does not.

Here is DigiRL's answer in plain words. It shows the final screen and the task to a large model, and asks one question: did this work? The one-word answer is the agent's only score.

Why a score is hard to get

Practice needs a score for every attempt, and there are thousands of attempts over hundreds of different tasks. The tasks run on live websites that change from week to week. Whatever gives the score must cope with all of that.

Earlier systems wrote a small checking program for each task. "Is the product page of item 4471 open?" That works in a sealed test world. It needs access to the inside of the emulator, and a person to write every single check. Neither scales to the live web.

DigiRL wants one judge for every task. That judge must look only at what any user could see: the screen. It may not peek at the phone's internals.

The score itself

The score an agent gets is its reward, like a gold star. In DigiRL the reward is 1 if the judge says the task is done, and 0 otherwise. A success also ends the attempt at once. If the judge never says success, the attempt ends when the step budget runs out, with a reward of 0.

So the reward comes once, at the end, and never in between. A reward like that is called a sparse reward, like an exam marked only on the final answer. Typing the right address earns nothing by itself. Only the finished task does.

The judge

The judge is formally called the evaluator: a large model that is shown the task and the final screenshot, and answers success or failure. It is a referee. DigiRL uses a large commercial vision-language model (Gemini 1.5 Pro) for this job.

Nobody trains this judge. Instead, its prompt includes a few example screenshots that people labelled success or failure: 9 examples for the general questions and 11 for the shopping tasks. Showing a model a few solved examples inside its prompt is called few-shot prompting, like worked examples printed at the top of an exam.

The prompt asks for a fixed three-line answer. Here is the format, with the shopping task filled in:

Q: What should I expect to see on the screenshot if I've <repeat the task>?
A: I should expect to see <first expectation, then what's in the given screenshot.>
Status: success or failure (don't return anything else)

The judge's answer format, from the paper's Appendix G. The judge fills in all three lines itself.

Read it line by line. In the Q line, the judge first restates the goal in its own words. In the A line, it says what success would look like, then what it actually sees in the screenshot. In the Status line, it gives the verdict: one word, success or failure.

Why this order? The verdict comes last, so it rests on an expectation the judge has already written down. A referee who first says what a goal looks like, then looks at the replay, makes fewer snap calls. The code reads the verdict with the pattern Status:\s*(\w+). If a call to the large model fails, the code retries it, up to 5 times.

What actually gets sent

The full screenshot is 1,080 × 2,280 pixels. Before sending, the code shrinks it 4 times on each side:

1,080 ÷ 4 = 270  ·  2,280 ÷ 4 = 570  ·  270 × 570 = 153,900 pixels (16× fewer than 2,462,400)

Calls to a commercial model cost time and money, so two guards skip calls that cannot matter. The first guard compares the new screen with the one before. It computes the mean squared difference: for each pixel, the change in brightness, squared, then the average over the whole screen. If that average is below 0.001 × 255² = 65.025, the screen hardly changed. The judge is not asked, and the reward is 0.

What does 65.025 mean? Its square root is 8.06, so the average pixel must change by more than about 8 brightness levels out of 255, or 3.2%. Two examples, both illustrative, make it concrete.

A blinking cursor changes 2,000 pixels by 200 levels each. The mean squared difference is (2,000 ÷ 2,462,400) × 200² = 0.000812 × 40,000 = 32.5. That is under 65.025, so the call is skipped. A pop-up covering a quarter of the screen, changing it by 100 levels, gives 0.25 × 100² = 2,500. That is far above, so the judge is asked.

32.5 < 65.025 → skipped  ·  2,500 > 65.025 → judged

The second guard is a small memory. The code remembers up to 5 recent screens, and skips any screen it has already judged. Try the judge on four final screens below.

How good is the judge?

A judge is only useful if it agrees with people. The paper measured this. People labelled the same attempts by hand, and the authors counted the disagreements. How often the judge disagrees with a person looking at the same screen is its error rate, like a referee's mistakes per hundred calls.

The judge's average error rate is 2.8%. On a test of 96 attempts, that is 96 × 0.028 = 2.69, about 3 attempts misjudged. The success rates it gives each agent are within 3 points of people's, for almost every agent tested.

It is more precise on the shopping tasks than on the general questions. The shopping tasks use narrow language: a site, a search, the first entry. "What's a good restaurant in New York?" has many right answers, and a judge can argue about any of them.

Two kinds of wrong call

A judge can be wrong in two directions. A false success is the judge saying success on an attempt that failed. A false failure is the judge saying failure on one that worked. Both are a referee's wrong call, but they do different damage.

The symptom. Success during training climbs, but when a person checks the final pages, many look wrong.

The measurement. Label a sample of final screenshots by hand and count the disagreements. That is how the paper measured its 2.8% (Figure 8).

The cause. Some attempts stopped on a page that only looks right, such as the right product on the wrong site. The judge scored them 1. The learning step treated them as worth copying, so the agent learned to stop there.

The fix. Add that confusing kind of screen to the judge's labelled examples, then measure again.

Now the asymmetry. A false failure wastes one good attempt: the agent just does not learn from it. A false success is worse, because it teaches the wrong habit. Every later chapter copies what the reward marks as good.

The judge, as code

Here is our sketch, modelled on the released evaluator with the names simplified:

pythonimport re, numpy as np

def judge(task, prev_png, png, ask_vlm):
    img, prev = load(png), load(prev_png)                # float arrays, 1080 x 2280 x 3
    if np.mean((img - prev) ** 2) < 0.001 * 255 ** 2:   # nothing changed: 65.025
        return 0                                         # no call, no reward
    small = shrink(img, 4)                               # 270 x 570
    reply = ask_vlm(FEW_SHOT_EXAMPLES, small, PROMPT.format(task=task))
    status = re.search(r"Status:\s*(\w+)", reply).group(1)
    return 1 if status.lower() == "success" else 0       # 1 also ends the attempt

Walk it through. load turns each saved screenshot into an array of numbers. The first test is the change guard: no change, no call, reward 0. shrink cuts each side by 4. ask_vlm sends the labelled examples, the small screenshot and the filled-in prompt to the large model. The last two lines find the word after Status: and turn it into 1 or 0.

The agent finished the costco task in 9 steps, and the judge said success. Nine actions, one score. Was the second tap good? The fifth? Most were. One was a detour. The score arrives once, at the end, like an exam marked only on the final answer.

Here is the idea in plain words. To learn from one score at the end, the agent needs a running guess, on every screen, of how likely it is to finish from there. A tap is good if it raises that guess. This chapter builds that guess, and gives it its proper names.

Attempts, records and returns

One attempt at one task, from the home screen to the end, is an episode, like one game. Its record is the trajectory: every state, every action and every reward, in order, like the replay of that game. A fresh attempt by the current agent is called a roll-out; we come back to that word below.

The total reward of an episode is its return, the final score of the game. In DigiRL every reward is 0 except possibly the last one. So the return is always 1 (the judge said success) or 0 (it did not).

Recall the policy from Chapter 1: the agent's decision rule. Now we write it as chances. We write it π(a | s, c): for the current state s and task c, the chance of each action a. The agent samples at temperature 1.0, so it really does pick at random by those chances. That is why the same agent can succeed one time and fail the next, from the same screen.

The rulebook

Reinforcement learning describes a trial-and-error task in a standard way, like the rulebook of a game. That description is a Markov decision process (MDP). The paper writes DigiRL's in one line:

M

The whole rulebook: the six parts below.

S

Every possible state: here, pairs of screenshots (the last two).

A

Every possible action: the action space of Chapter 1.

T

How an action changes the state. The phone and the web decide this, with some randomness: a tap may bring a pop-up.

μ₀

Where attempts start: the home screen, and a task c drawn from the task list.

R

The reward: the judge's 0 or 1 at the end.

H

The step budget: 10 or 20.

The step budget, as a symbol, is the horizon H: the move limit. H = 10 for the general questions and H = 20 for the shopping tasks. One honest caveat about S: one screenshot cannot show everything. Is that page still loading, or broken? That is why the state is the last two screenshots, not one. (Partially observed decision processes covers the general idea; Markov decision processes covers the rulebook itself.)

Value is a chance

Now the central quantity. Stand on some screen at step h, and pick an action a_h. How good is that choice? Add up all the rewards from now to the end of the attempt, and average over everything that could happen:

Q^π

The chance of finishing if you take a_h now, then act as the policy π usually does.

s_h, a_h

The state and the action at step h.

c

The task sentence.

Σ r

The rewards from now to the end. Here, that is just the final 0 or 1.

E_π

The average over the randomness of the phone and of the policy's own choices.

Why is Q a chance? Every reward in the sum is 0, except possibly the last. So the sum is just the final reward: 1 or 0. The average of something that is 1 or 0 is the chance that it is 1. So Q is the chance of finishing the task, if you take this action now.

The agent does not always take one particular action. Average Q over the policy's own choices, and you get the value function V: the chance of finishing from this screen, if the agent keeps acting as it usually does. Think of a weather forecast, but for success. Q is the same forecast for one specific choice, so we call it the Q function.

V^π

The chance of finishing from this screen, acting as usual.

A^π

The advantage: how much better this action is than usual.

The last line is the one this whole lesson runs on. The advantage A is how much better an action is than the agent's usual choice: Q minus V. It is beating your own average. A positive advantage means "this was better than I usually do here". A negative one means "worse".

One screen, worked out

Take the results screen of the costco task. Say the policy taps the first result half the time, taps an ad 1 time in 10, and scrolls 4 times in 10. Say the chances of finishing after each are 0.95, 0.20 and 0.40. All five numbers are illustrative. The value of the screen is the policy-weighted average of the three Q values:

V = 0.5 × 0.95 + 0.1 × 0.20 + 0.4 × 0.40 = 0.475 + 0.020 + 0.160 = 0.655

So from this screen, acting as usual, the agent finishes about two times in three. Now the three advantages. Tapping the first result: 0.95 − 0.655 = +0.295. Tapping the ad: 0.20 − 0.655 = −0.455. Scrolling: 0.40 − 0.655 = −0.255.

V = 0.475 + 0.020 + 0.160 = 0.655  ·  A(ad) = 0.20 − 0.655 = −0.455 (worse than usual)

One check is worth doing by hand. Weight each advantage by how often the policy takes that action, and add: 0.5 × 0.295 + 0.1 × (−0.455) + 0.4 × (−0.255) = 0.1475 − 0.0455 − 0.1020 = 0.000. The policy's own average beats itself by exactly zero. That is true on every screen, because V is that average.

Estimating by trying

Where do numbers like 0.655 come from? Nobody hands them to you. The simplest way to estimate a chance is to try many times and count, like flipping a coin a hundred times to check it is fair. That is a Monte Carlo estimate.

Here, one try means playing the attempt out to the end from a chosen screen, with the current policy. That is a roll-out, like playing a chess position out to the end. A single roll-out ends in 1 or 0. It can never say 0.655. Only the average of many can.

How fast does the average settle? How much an estimate jumps around from one batch of tries to the next is its variance, like a wobbly bathroom scale. For a chance V estimated from n tries, the typical error is √(V(1 − V) ÷ n). With V = 0.655, the top part is √(0.655 × 0.345) = √0.225975 = 0.475.

n = 1: ±0.475  ·  n = 16: 0.475 ÷ 4 = ±0.119  ·  n = 100: 0.475 ÷ 10 = ±0.048

Sixteen tries still leave you 0.12 either way. A hundred tries get you within 0.05. Roll it out yourself and watch the estimate wander toward the truth.

Start from "After the first result" and the estimate drifts to about 0.95. Start from "After the ad" and it drifts to about 0.20. The gap between each of those and 0.655 is exactly the advantage you computed by hand. Monte Carlo can measure advantages too, just slowly.

Why trying alone is too expensive here

In a board game, a roll-out takes microseconds. On a live phone it takes real minutes. The paper's whole fleet of simulated phones collected 1.74 attempts a minute (Chapter 8 has the fleet). So 100 full attempts take 100 ÷ 1.74 = 57 minutes. And that would estimate the value of just one screen.

The world's randomness makes it worse. A pop-up or a slow page can sink an attempt for reasons that have nothing to do with the action you are testing. So each roll-out is noisier still. The paper names this directly: the Monte Carlo estimate has high variance because of the environment's randomness.

So DigiRL does something smarter as well. It trains a network to predict V for any screen it has seen something like. Then it does not need a hundred fresh roll-outs per screen. Chapter 5 shows how that network grades each step, and Chapter 7 shows how it is trained.

Which action earned it?

Step back and see what we now have. The final reward belongs to the whole attempt. Working out which actions earned it is called credit assignment, like deciding which players earned the goal.

The advantage is the tool for that job. It says which taps beat the agent's own usual choice, screen by screen. Chapters 4 to 6 use exactly this to choose what to copy. (Policy gradients shows the classic way to learn from advantages, which we meet again in Chapter 4.)

A value that lies

The symptom. The predicted value says 0.9 on a page from which the agent rarely finishes.

The measurement. Collect the screens where the predictor says about 0.9. Count how often attempts from those screens actually finished. A trustworthy predictor's 0.9 screens finish about 9 times in 10. This is a calibration check, and it works for any forecaster.

The fix. Retrain the predictor on fresher attempts, and with the right loss (Chapter 7).

Roll-outs, as code

pythonimport random

def rollout_from_results(rng):                 # one roll-out from the results screen (toy chances)
    a = rng.choices(["first", "ad", "scroll"], weights=[0.5, 0.1, 0.4])[0]
    q = {"first": 0.95, "ad": 0.20, "scroll": 0.40}[a]
    return 1 if rng.random() < q else 0

def mc_value(n, seed=0):
    rng = random.Random(seed)
    wins = sum(rollout_from_results(rng) for _ in range(n))
    v = wins / n
    return v, (v * (1 - v) / n) ** 0.5         # the estimate and its typical error

rollout_from_results is one roll-out: pick an action by the policy's chances, then succeed with that action's Q. mc_value runs n of them, counts the wins, and returns the average with its typical error. With a real agent, each call to rollout_from_results would be minutes on a phone.

After a few days of practice, the agent has a pile of a thousand recorded attempts. Some succeeded. Most did not. Every attempt is a list of screens and taps, with one score at the end. What is the simplest useful thing you could do with the pile?

Here is the simplest answer: copy the taps that helped, and ignore the rest. The only real questions are how to measure "helped", and how strongly to prefer it. This chapter answers both, and ends with the rule DigiRL actually uses.

Fresh attempts, or a pile?

The classic method is the policy gradient. Make fresh attempts with the current agent. Nudge the agent toward the actions that scored well. Then throw those attempts away and make new ones. Practise, adjust, forget. (Policy gradients derives it from scratch.)

Learning only from fresh attempts by the current agent is called on-policy learning, like practising only today's drills. Learning from a pile of older attempts too is off-policy learning, like also replaying old matches. On a live phone, attempts are slow: the paper's whole fleet of simulated phones collected under two a minute. Throwing them away after one use is expensive.

So DigiRL keeps a pile and learns from it again and again. The stored pile of past attempts is the replay buffer, written ν (the Greek letter nu), like a diary of every attempt. DigiRL's holds up to 5,000 attempts. The authors estimate their approach needs about a thousand attempts, where on-policy reinforcement learning on a simpler set of web tasks needed about a million. Their phrase is "1000x more sample efficient".

Copying, as a formula

Training needs a number to make small. That number is the loss, like the penalty in a game. For copying, the loss of one recorded step is −log π(a | s, c): minus the log of the chance the policy gives to the recorded action.

Why that form? If the policy already gives the recorded action a high chance, the log is near 0, so the loss is small. If it gives a low chance, the log is very negative, so the loss is large. Training lowers the loss, and that raises the chance. With numbers: a chance of 0.20 has a loss of −ln 0.20 = 1.609. Raise the chance to 0.30, and the loss falls to −ln 0.30 = 1.204.

−ln 0.20 = 1.609 → −ln 0.30 = 1.204 (a higher chance for the recorded action, a smaller loss)

Training that raises the chance the policy gives to chosen actions is called maximum likelihood (MLE), like turning up the volume on one answer. Plain imitation learning is maximum likelihood on every recorded step. The question is which steps to choose.

Weight each copy

Chapter 3 gave us a grade for each step: its advantage. A natural idea is to copy every step, but let good steps count more. That is advantage-weighted regression (AWR). Copy past actions, but give each one a weight that grows fast with its advantage, like a teacher who grades homework on a curve. It is the paper's Equation 4.1:

π

The policy being trained.

E_ν

The average over the replay buffer ν, the pile of past attempts.

log π(a | s, c)

The log of the chance the policy gives the recorded action.

A(s, a, c)

That action's advantage, from Chapter 3.

β

The sharpness: how fast the weight grows with the advantage.

arg max_π

Choose the policy that makes the average largest.

Read the exponential in plain words. A positive advantage makes the weight bigger than 1. A negative advantage makes it smaller than 1, but never negative. The knob β sets how fast the weight grows, like the steepness of a ramp. The paper calls β only "a positive parameter". We call it the sharpness, and keep the word temperature for sampling.

With numbers, at β = 0.5: an advantage of +0.3 gives e^0.6 = 1.822, and −0.4 gives e^−0.8 = 0.449. A good step counts four times as much as a bad one. Now shrink β to 0.05. The same two steps give e⁶ = 403.4 and e⁻⁸ = 0.000335. A ratio of over a million.

The sharpness is touchy

Take eight recorded steps with these advantages, all illustrative. The table shows each step's weight at three settings of β:

At β = 1 the weights run from 1.649 to 0.638. Every step is copied, the best a little more. At β = 0.05, one step weighs 22,026 and the next 403. The best step takes 98.1% of all the weight. Training would copy one lucky tap and almost nothing else.

How do you measure "almost nothing else"? Use the effective sample size: how many equally weighted examples a weighted set is worth. Think of a jury where one juror's vote counts for ten. The formula is (Σw)² ÷ Σw². Eight equal weights give 8. One huge weight and seven tiny ones give about 1.

β = 2: 7.84  ·  β = 1: 7.39  ·  β = 0.2: 2.75  ·  β = 0.05: 1.04 (effective size, out of 8)

So β has two failure modes. Too large, and the weights are nearly even: you are back to plain imitation, mistakes and all. Too small, and one lucky step drowns out the rest. Slide it yourself.

Keep or drop

DigiRL sidesteps the knob entirely. Instead of a smooth weight, each step gets a hard filter: weight 1 if its advantage passes a threshold, weight 0 if not, nothing in between. It is a sieve. Then the agent copies whatever is left with plain maximum likelihood. That is the paper's Equation 4.2:

L(π)

The loss to make small.

filter(ν)

Only the steps from the buffer that passed the keep-or-drop test.

−

The minus sign: making the loss small makes the log chance large.

On our eight steps, with a threshold of 0.1, three steps pass: +0.50, +0.30 and +0.15. Each gets a third of the weight. The effective size is exactly 3. There is no β to tune, so there is no way to set it wrong.

Does throwing away the smooth weights cost anything? The paper checked with an ablation: remove or change one part of the method, and measure what changes, like taking one ingredient out of a recipe. The hard filter matched the best tuned exponential weighting, and it beat plain AWR. The paper chose it to avoid tuning β.

The simplest sieve: copy your wins

There is an even simpler hard filter, with no advantages at all. Keep the whole attempts that succeeded, and copy every action in them. That is Filtered BC, short for filtered behavior cloning. It is like keeping only the games you won and replaying them move by move.

In the released code, Filtered BC keeps the attempts whose reward is at or above the 90th percentile of all rewards. Rewards are 0 or 1, so when more than 1 attempt in 10 succeeds, that means every success. Simple as it is, it works well. Trained offline then online, it lifts the copier from 17.7% to 57.8% on the shopping test. It is the agent labelled Copies its wins at the top of the page.

Filtered BC's blind spot

A winning attempt can contain a bad step. A losing attempt can contain good steps. Filtered BC sees neither, because it grades whole attempts.

Chapter 5 follows one attempt in detail: six steps to answer "What's on the menu at Burger King?", one of them an accidental tap on voice input. Filtered BC copies all six, the mistake included. And if that same attempt had failed, it would throw away its two good opening steps with the rest.

The symptom. The agent finishes tasks, but slowly, with detours that look like habits.

The measurement. Compare attempt lengths. DigiRL's attempts end up shorter than Filtered BC's: about 6.5 steps against 7.7 on the general questions (Chapter 9, Table 4).

The fix. Grade each step, not each attempt. That is Chapter 5.

The weights, as code

pythonimport numpy as np
A = np.array([0.50, 0.30, 0.15, 0.05, 0.00, -0.10, -0.30, -0.45])

def awr_weights(A, beta):  return np.exp(A / beta)            # Eq. 4.1's weight
def hard_filter(A, t=0.1): return (A > t).astype(float)        # Eq. 4.2: keep or drop
def eff_size(w):           return w.sum() ** 2 / (w ** 2).sum()

for beta in (1.0, 0.2, 0.05):
    w = awr_weights(A, beta)
    print(beta, round(w.max() / w.sum(), 3), round(eff_size(w), 2))
# 1.0 0.194 7.39
# 0.2 0.554 2.75
# 0.05 0.981 1.04
print(eff_size(hard_filter(A)))                                # 3.0

Each line matches a formula above. awr_weights is the exponential of Eq. 4.1. hard_filter is the sieve of Eq. 4.2. eff_size is the jury count. The printed columns are β, the best step's share of the weight, and the effective size.

One more failure, the mirror of the blind spot. The symptom: the training loss drops fast, but success on fresh attempts falls. The measurement: compute the effective size of each training batch's weights; it sits near 1. The fix: raise β, or switch to the hard filter.

Task: What's on the menu at Burger King? Step 1: the agent taps Chrome. Step 2: it taps the search bar. Step 3: its finger lands on the microphone, and voice input opens. Step 4: it presses Back. Step 5: it types burger king menu. Step 6: it presses Enter, the menu appears, and the judge says success.

Six steps, one success. Step 3 was a mistake. How would a program know? This attempt is our own example, built on a task from the paper's list. Its step 3 echoes the paper's real story of an agent stuck in voice input (Chapter 0).

Here is DigiRL's answer in plain words. Grade each step with two opinions. The ending says: this attempt worked, and this step was close to the end. The critic says: this tap made success more likely, or less. Trust the ending near the end, and the critic further back.

Two ways to grade a step

The first way uses only the ending. That is the Monte Carlo idea of Chapter 3: every step of a success looks good, and every step of a failure looks bad. It is honest, because the ending really happened. But one attempt is one random draw. A pop-up can sink a good attempt, and luck can save a bad one. The paper calls this estimate high in variance.

The second way uses a critic: a network that predicts the value of a screen, like a coach with an opinion about every position. DigiRL's step critic is the step-level value function, written V_step. It looks at a screen and guesses the chance of finishing from it.

With a critic, you can grade one step on its own. Check the scoreboard before and after the play. Take the critic's value after the action, add any reward, and subtract the value before. That is the one-step estimate: V_step(s_h+1) + r_h − V_step(s_h). Opening voice input drops the critic's value from 0.55 to 0.20, so the step grades −0.35.

The one-step estimate is steady, because it does not wait for one random ending. Its weakness is bias: an error that does not average away, because the critic itself is wrong. Think of a bathroom scale that always reads 2 kg light. Weigh yourself a hundred times, and you are still 2 kg off.

Blend the two: Equation 4.3

DigiRL uses both, in the paper's Equation 4.3:

A_step

The step's grade.

λ

How fast the ending's credit fades as you walk back from the end. DigiRL uses 0.5.

H − h

How many steps before the end of this attempt the step was taken. Here H is the step where this attempt ended; the code counts it that way.

r(s_H, a_H, c)

The judge's final verdict, 0 or 1.

V_step

The critic's value of a screen.

r(s_h, a_h, c)

The reward right after this step: 0, except at the very end.

s_h+1

The screen this step led to.

It looks heavy, but one renaming makes it simple. The term λ^H−h r(s_H) appears twice. Call it m, the ending's credit. Call the one-step estimate td. Then the whole equation is one line:

m

The ending's credit: λ^H−h times the final verdict.

td

The critic's one-step guess: V_step(s_h+1) + r_h − V_step(s_h).

Read the blend. A share m of the grade comes from the ending, and the remaining share, 1 − m, from the critic. Three cases show how it behaves.

1. A failed attempt. The verdict r is 0, so m = 0 for every step. The grade is td: the critic alone.
2. The last step of a success. Here H − h = 0, so m = λ⁰ × 1 = 1. The grade is 1: the ending alone.
3. Steps in between. With λ = 0.5, the credit halves with each step back from the end: 1, 0.5, 0.25, 0.125, 0.0625, 0.03125. The echo of a shout, halving each time.

That last case is the design. A success is strong evidence about the step just before it, and weak evidence about a step taken long before. So near the end the ending speaks, and further back the critic takes over.

Which steps to keep

A grade is not yet a decision. DigiRL keeps a step for copying if its grade beats 1/H:

1/H

The threshold: 0.1 for a 10-step task (general questions), 0.05 for a 20-step task (shopping).

Why 1/H? The paper just states it. Here is an intuition, and it is ours, not the paper's. With H steps to finish, each step must carry about one H-th of the way. A step that moves the agent less than that is not clearly progress.

The Burger King attempt, graded

Now give the critic some illustrative values for the seven screens: home 0.30, Chrome open 0.42, search bar focused 0.55, voice input open 0.20, back at the search bar 0.55, query typed 0.60. After the last step the attempt is over, so the last value is 0. The general questions have H = 10, so the threshold is 0.1. And λ = 0.5.

Five steps are kept, and the voice-input tap is dropped. Look at how step 3 lost. It sits three steps from the end, so the ending's credit is only 0.125. The critic says −0.35, and it gets 87.5% of the vote. The verdict is −0.18.

step 3: 0.125 + 0.875 × (−0.35) = −0.18 < 0.1 → dropped (the ending's credit is small this far back; the critic's verdict wins)

Step 4 is just as interesting. Pressing Back is the way back from a mistake. Filtered BC would copy it only as part of a whole winning attempt. Here it earns its place on its own: the critic sees the value jump from 0.20 back to 0.55. This is how DigiRL learns recovery.

Why λ = 0.5

Try the two extremes. At λ = 1, the credit never fades: m = 1 at every step, so every step of a success grades 1. All six are kept, the microphone tap included. That is exactly Filtered BC.

At λ = 0, only the critic speaks (except at the last step, where 0⁰ = 1). The grades become 0.12, 0.13, −0.35, 0.35, 0.05 and 1.0. Now step 5, the typed query, grades 0.05 and is dropped. This critic underrates the screen after typing: 0.60, barely above the 0.55 before it. A biased critic loses a good step.

At λ = 0.5, the ending lends step 5 half its credit, and the right five steps survive. Slide λ yourself, flip the verdict, and drag the critic's values.

The family it belongs to

Eq. 4.3 is built from two estimates so that each covers the other's weak spot. The ending has variance but no bias; the critic has bias but little variance. That design is called doubly robust, like wearing a belt and braces. The paper reports that mixing the high-variance and the high-bias estimates gave "a sweet-spot".

It is also a pared-down form of a standard recipe, generalized advantage estimation (GAE). GAE mixes estimates that look 1, 2, 3 and more steps ahead, like listening to every echo of a shout. With no rewards between the start and the end, the middle echoes add nothing new. So DigiRL keeps only the shortest look (one step) and the longest (to the end). (The GAE paper lesson builds the full recipe.)

Did grading steps help? The paper's ablation says step-level grades made learning about 12% more efficient than grading whole attempts.

Failures still teach

Now flip the ending. Same first three steps, but the agent never leaves voice input and runs out its 10 steps. Say the critic's values along the way are 0.30, 0.42, 0.55, 0.20, 0.20, 0.18, 0.18, 0.15, 0.15, 0.12, and then 0.10. The verdict is 0, so every m is 0 and the grades are the critic's alone.

0.12, 0.13, −0.35, 0.00, −0.02, 0.00, −0.03, 0.00, −0.03, −0.02 → steps 1 and 2 kept

The two good opening taps pass the 0.1 bar, and everything after is dropped. A failed attempt still donates its good opening. Filtered BC would have thrown the whole attempt away.

What the released code does

The released code computes the ending's credit as a discounted sum of rewards with a discount of 0.5. For a reward only at the end, that is exactly m. For a 4-step success it gives [0.125, 0.25, 0.5, 1.0]. The code stores this number with every step as mc_return.

Its step filter is a single line: grade = nv − v − 0.05 + reward + mc_return, clamped to between 0 and 1, kept when above 0. Here nv is the critic's value of the next screen and v of this screen. The fixed margin 0.05 equals the shopping threshold 1/20, and mc_return is the ending's credit.

On our six steps it gives 0.1012, 0.1425, −0.275, 0.55, 0.50 and 1.35 before clamping. The same five steps are kept. Be honest about what this is: the code's sum differs in form from Eq. 4.3. This lesson teaches the paper's equation, and notes that the released code approximates it.

A critic that loves dead ends

The symptom. The agent starts opening menus that lead nowhere.

The measurement. Plot V_step along failed attempts. It rises on screens that never lead to success, so the one-step grade passes the steps that reach them.

The fix. Retrain the critic on fresh attempts with the right loss, and score with the lower of two critics. Both are in Chapter 7.

Eq. 4.3, as code

pythonimport numpy as np

def step_advantage(V, r, lam=0.5):
    T = len(r); h = np.arange(1, T + 1)
    m = lam ** (T - h) * r[-1]          # the ending's credit
    td = V[1:] + r - V[:-1]             # the critic's one-step guess
    return m + (1 - m) * td             # Eq. 4.3

V = np.array([0.30, 0.42, 0.55, 0.20, 0.55, 0.60, 0.0])
r = np.array([0, 0, 0, 0, 0, 1.0])
A = step_advantage(V, r)                # [0.1475 0.184375 -0.18125 0.5125 0.525 1.0]
keep = A > 1 / 10                       # five kept; step 3 dropped

V has one more entry than r: the value of every screen, including the one after the last step. T - h counts back from this attempt's own last step. V[1:] + r - V[:-1] computes all six one-step guesses at once. The last line is the 1/H test.

Three tasks from the paper's own picture of this idea. For each, a critic guesses how often the agent wins it. Go to walmart.com: 8 times in 10. Go to ebay.com, search for "asus zenbook": 1 time in 10. Go to costco.com, search for "bose soundsport free", and select the first entry: 1 time in 100.

Today the agent won the first two and failed the third. Which attempt teaches it the most?

Here is the idea in plain words. A win on a task you rarely win is news. A win on a task you always win is not. So rank whole attempts by how surprising their result was, and train on the most surprising wins.

Not all practice is equal

The task list mixes very easy and very hard tasks. The shopping tasks come in three difficulty levels, like the rungs of a ladder. Level 1 is "go to the site". Level 2 is "go to the site and search". Level 3 is "go, search, and select the first entry".

The authors found that collecting more data on tasks the agent could already do hurt how fast it learned. An attempt at walmart.com that succeeds for the hundredth time costs the same minutes as any other attempt. It teaches almost nothing.

Step grades (Chapter 5) cannot fix this. Inside an easy win, every step looks good, so every step is kept. The problem is one level up: which whole attempts deserve the agent's limited training time?

A critic for whole tasks

DigiRL trains a second critic that reads only the task sentence. It guesses how often the agent currently solves that task, like a teacher's guess of how hard an exam question is. This is the instruction-level value function, written V_instruct(c). In this lesson we also call it the task critic.

The task critic never sees a screenshot. It reads the task text through a text network (RoBERTa) and outputs one chance. Chapter 7 shows its exact shape and how it is trained.

How surprising was this result?

With a task critic, scoring an attempt is one subtraction. It is the paper's Equation 4.4:

A_instruct

How surprising this attempt's result was.

Σ r

The rewards from step h to the end, which is just the final verdict.

V_instruct(c)

The task critic's guess of how often the agent solves task c.

r(s_H, a_H, c)

The final verdict, 0 or 1.

Why does the second line equal the first? Rewards come only at the end of an attempt. So the sum of rewards from any step to the end is just the final reward. The score is the result minus what the critic expected: a surprise.

Now the three tasks. The easy win scores 1 − 0.8 = 0.2. The medium win scores 1 − 0.10 = 0.90. The hard failure scores 0 − 0.01 = −0.01. The medium win is the most surprising result by far.

1 − 0.10 = 0.90 (a medium win: kept)  ·  1 − 0.8 = 0.2 (an easy win: dropped)

Look at the hard failure too. It scores almost exactly zero. The agent expected to fail, and it failed. Nothing new happened, so there is nothing to learn yet.

Keep the top 10%

DigiRL scores every attempt in the replay buffer this way. Then it keeps only the best fraction p by score, like an honour roll. This is the top-p filter. The paper does not state p; the released code keeps the top 10%, the attempts at or above the 90th percentile of the scores.

Inside the kept attempts, the step grades of Chapter 5 decide which steps survive. The agent copies those steps with plain maximum likelihood (Eq. 4.2). So DigiRL filters twice: first attempts, then steps. The paper compares the first filter to replaying the most informative experience first.

Here is a worked buffer, the same one Lab 2 uses as a test (illustrative). Say easy tasks are already solved 95% of the time, medium tasks 30% and hard tasks 2%. The buffer holds 30 attempts:

• 10 easy wins, each scoring 1 − 0.95 = 0.05;
• 5 medium wins, each scoring 1 − 0.30 = 0.70;
• 5 medium failures, each scoring 0 − 0.30 = −0.30;
• 10 hard failures, each scoring 0 − 0.02 = −0.02.

You might expect exactly 3 attempts to survive, 10% of 30, but the real rule keeps every attempt at or above the 90th-percentile score, not a fixed count. Here that score is 0.70, and five attempts tie there, so all five medium wins stay, and not one easy win.

An automatic curriculum

An order of practice from easy to hard is a curriculum, like the years of school. Nobody writes one for DigiRL. It falls out of Eq. 4.4.

Two words before you try it. One nudge of the policy's or a critic's numbers during training is an update, like one stride. We never call it a step: a step is one screenshot in and one action out. One cycle of practice and training, from fresh attempts to a new policy, is a round, like one training week.

At first the agent wins mostly easy tasks. Medium and hard wins are too rare to fill the top 10% on their own, so the cut falls low enough to let easy wins in, and easy wins are the most numerous. As easy tasks become routine, their value nears 1 and their wins score near 0, below the cut. Now medium wins are both surprising and common enough to fill the kept set. Later the hard wins take over. The toy below runs this for 40 rounds.

Ranked by r − V, the kept set walks up the ladder. After 5 rounds it is about 42% easy wins, 44% medium and 14% hard. After 20 rounds the easy wins are gone, and about 68% of the kept attempts are hard wins. Hard tasks now succeed about 53% of the time.

Keep every win instead, and easy wins keep crowding the kept set: still over half of it after 20 rounds. Hard tasks crawl to about 27%. Easy tasks improve at the same rate either way, about 96.5% at round 20, because every win includes going to the site.

The authors saw the same order in the real agent. Success on level 1 rose first, then level 2, then level 3. Their ablation measured the payoff: the curriculum made learning about 25% faster.

One exception. In the offline phase (Chapter 8), DigiRL skips this filter and trains on every successful attempt in the pile. Offline data is all it has at that point, so it uses the whole of it.

A stale task critic

The symptom. The agent keeps training on a task it has already mastered.

The measurement. Compare V_instruct with the task's recent success rate in the buffer. In our illustration the critic still says 0.10 where the agent now wins 8 times in 10. So every easy win scores 0.9, and easy wins flood the kept set.

The fix. Retrain the task critic every round, on the raw buffer, before filtering. DigiRL does exactly this: 5 updates of the task critic every round.

Eq. 4.4, as code

pythonimport numpy as np
r = np.array([1.0, 1.0, 0.0]);  V = np.array([0.8, 0.10, 0.01])
A = r - V                                   # [0.2, 0.9, -0.01]   Eq. 4.4
keep = A >= np.quantile(A, 1 - 0.10)        # the top 10%

The released code does the same in two lines:

released codebaselines = trajectory_rewards - baselines
cutoff = np.quantile(baselines, 1 - 0.1)

Read them word by word. trajectory_rewards holds each attempt's final verdict. baselines first holds the task critic's value for each attempt's task. After the first line it holds the scores of Eq. 4.4, which is confusing but harmless. np.quantile(..., 1 - 0.1) finds the 90th percentile, and attempts at or above that cutoff are kept.

Both filters lean on two critics: one that scores screens and one that scores tasks. Someone has to train them, from nothing but the judge's 0 or 1 at the end of each attempt.

Picture a critic that, after a lucky first day, says 0.99 on a screen from which the agent really finishes only half the time. Every step graded against that critic is graded wrong. How fast can training talk it down?

Here is DigiRL's answer in plain words. Train each critic as a forecaster answering one question: will this attempt end in success? Score it the way weather forecasters are scored: harshly for a confident miss.

What the critics learn from

Every step of every attempt gets one label: the attempt's final verdict. If the attempt succeeded, every screen in it is labelled 1. If it failed, every screen is labelled 0. In the code the label is "did this step's credit exceed 0": class 1 or class 0.

One label per screen is a noisy teacher. A good screen in a failed attempt is labelled 0. But a forecaster sees many attempts through similar screens. Averaged over all of them, the labels say how often attempts from such a screen succeed. That average is exactly the value of Chapter 3.

Two ways to score a forecast

A network cannot output a chance directly. Its last layer outputs a raw number, called a logit: the dial before the gauge. A squash then turns the logit into a chance between 0 and 1, like a gauge whose needle can never leave 0 to 1. For one logit the squash is the sigmoid, σ(z) = 1 ÷ (1 + e^−z). The code uses two logits and a softmax; for two classes the two give the same result.

Recall the loss from Chapter 4: the number training tries to make small. The obvious loss for a forecast is squared error: (guess minus outcome) squared, like the distance from the bullseye, squared. The other is cross-entropy: minus the log of the chance you gave to what actually happened. Cross-entropy punishes a confident miss very hard.

Compare them on single outcomes. The critic says 0.99 and the attempt fails. Cross-entropy is −ln 0.01 = 4.605. Squared error is 0.99² = 0.9801. Now the critic says 0.99 and the attempt succeeds. Cross-entropy is −ln 0.99 = 0.01005, and squared error 0.01² = 0.0001. Both reward a good call; only cross-entropy screams at a confident wrong one.

DigiRL trains both critics with cross-entropy. These are the paper's Equations 4.5 and 4.6:

L

The loss to make small.

r(s_H, a_H, c)

The attempt's final verdict, 0 or 1. It switches on exactly one of the two terms.

V_instruct(c)

The task critic's chance of success for task c. (The paper prints this one as V^traj in Eq. 4.5; it is the same task critic.)

V_step(s_h, a_h, c)

The step critic's chance of success for this step's screen.

log

The natural log: near 0 for a chance near 1, very negative for a chance near 0.

E_ν

The average over the replay buffer.

Read it for one attempt. If it succeeded, r = 1 and the loss is −log V: small when V is near 1. If it failed, r = 0 and the loss is −log(1 − V): small when V is near 0. Each attempt pushes the forecast toward what happened.

Which way to nudge, and how hard

Training changes the logit z, not the chance directly. What matters is the gradient: which way, and how hard, to nudge a number to make the loss smaller. Think of the slope under your feet. A steep slope means a big nudge; a flat one means almost none.

Here is what each loss does to the logit, without working through the calculus behind it. Cross-entropy's pull is simply V − r: the gap between the forecast and what actually happened. Squared error's pull is that same gap, times an extra factor that depends on V itself:

z

The logit: the raw number before the squash.

V(1 − V)

The squash's slope. It is tiny near 0 and near 1: the source of saturation.

That extra factor is the whole story. The squash is flat near 0 and near 1, so nudging the logit there barely moves the chance. This is saturation, like pushing a door that is already against the wall. Squared error inherits that flatness, since its pull carries the extra factor. Cross-entropy's pull does not carry it: the factor cancels out against the squash's own slope, leaving only the plain gap V − r.

The numbers

Put the confident critic in. It says V = 0.99, which means a logit of z = ln(0.99 ÷ 0.01) = 4.595. The true chance from this screen is 0.5. So on average the outcome r is 0.5, and we can use the expected pull.

V − p = 0.99 − 0.50 = 0.49  ·  2(V − p)·V(1 − V) = 2 × 0.49 × 0.0099 = 0.0097 (50 times weaker)

Cross-entropy pulls 0.49. Squared error pulls 0.0097, about 50 times weaker (50.5, to be exact). Near the middle the gap shrinks. At V = 0.5 with a truth of 0.8, the pulls are −0.30 and 2 × (−0.30) × 0.25 = −0.15: only twice. At V = 0.9 with a truth of 0.2, they are 0.70 and 0.126: 5.6 times.

Now run training with a step size of 1 on those expected pulls, starting from 0.99. Cross-entropy brings the critic below 0.60 after 12 updates. Squared error needs 124. Try other starting points and other truths below.

The paper's reason, and its evidence

The paper gives two reasons. First, it cites recent findings that transformer-based networks are better trained with cross-entropy losses than with squared error. Second, its own ablation: training the critics with cross-entropy instead of regression improved results by about 12%.

One more thing makes cross-entropy natural here. A network trained this way is a classifier: it says how likely "success" is. For a return that is always 0 or 1, that chance is exactly the expected return, which is the value. The critic's output means what Chapter 3 needs it to mean.

The critics, shape by shape

Now the real networks, read from the released code. The step critic reads two things. The prompt text goes through a text network, RoBERTa, which returns 768 numbers. The two screenshot fingerprints, side by side, give 2 × 1,408 = 2,816 numbers. Placed end to end, that is 768 + 2,816 = 3,584 numbers per step.

Those 3,584 numbers go into an MLP: a small stack of layers, each one a multiply, an add, and "keep the positives". Think of a short assembly line. The "keep the positives" part is called ReLU: it replaces every negative number with 0.

step critic, one headinput   [128, 3584]     768 from RoBERTa + 2,816 from the two screenshots
Linear  3,584 → 768      [128, 768]     3,584 × 768 + 768   = 2,753,280 numbers
ReLU                    [128, 768]
Linear  768 → 768        [128, 768]     768 × 768 + 768     =   590,592
ReLU                    [128, 768]
Linear  768 → 2          [128, 2]       768 × 2 + 2         =     1,538
softmax, take class 1   [128]          the value: a chance of success per step
                                       one head: 3,345,410  ·  two heads: 6,690,820

Shapes for one batch, a group of 128 steps trained together in a single update (Table 6's batch size). The parameter counts are our arithmetic from the layer sizes in critic.py.

Read it top to bottom. Each row of 3,584 numbers shrinks to 768, then stays at 768, then becomes 2 logits. The softmax turns the 2 logits into two chances that add to 1. The chance of class 1, "success", is the value. The paper calls this a two-layer MLP; the code has three linear layers.

The task critic is much smaller. It reads only the task text through RoBERTa and adds one linear layer, 768 → 2: just 1,538 numbers on top. Both critics reuse the frozen image encoder of Chapter 1, so their image side is never trained.

Two critics, and trust the gloomier

The step critic actually has two heads, trained side by side on the same data. When the step filter needs a value, it takes the lower of the two. This is a double critic, like asking two coaches and believing the gloomier one.

Why? Here is our intuition, not the paper's words. An over-optimistic critic passes bad steps, because it inflates the value after a step. Two critics trained from different starting numbers make different mistakes. The lower of two guesses is less likely to be wildly optimistic.

A saturated critic

The symptom. After an early run of wins, the critic outputs about 0.99 almost everywhere. The one-step grades are all near 0, and the step filter keeps almost nothing except last steps.

The measurement. Draw a histogram of the critic's outputs: a spike at 0.99. Then check its calibration on held-out attempts, as in Chapter 3.

The fix. Use cross-entropy. In the device's toy, the saturated critic needed 124 small updates under squared error to fall below 0.6, and 12 under cross-entropy.

The critic, as code

Two last words for the code. The learning rate is how big each update is, like stride length. A batch is how many examples one update uses, like how many opinions you gather before you move.

pythonimport torch, torch.nn as nn, torch.nn.functional as F

class StepCritic(nn.Module):
    def __init__(self, text_encoder):                    # roberta-base, 768 numbers out
        super().__init__()
        self.text = text_encoder
        self.heads = nn.ModuleList([nn.Sequential(
            nn.Linear(768 + 2816, 768), nn.ReLU(),
            nn.Linear(768, 768), nn.ReLU(),
            nn.Linear(768, 2)) for _ in range(2)])       # two heads: the double critic
    def forward(self, prompt_tokens, img2816):
        x = torch.cat([self.text(prompt_tokens).pooler_output, img2816], -1)   # [B, 3584]
        return [head(x) for head in self.heads]          # two [B, 2] logit pairs

def critic_loss(logits, mc_return):                       # Eq. 4.6
    return F.cross_entropy(logits, (mc_return > 0).long())

def value(critic, prompt_tokens, img2816):               # what the step filter uses
    v1, v2 = [F.softmax(l, -1)[:, 1] for l in critic(prompt_tokens, img2816)]
    return torch.minimum(v1, v2)                          # [B]

This is a sketch with the released code's shapes (critic.py, trainer.py), not the code itself. StepCritic builds the two heads. forward joins the text and the screenshots into 3,584 numbers and runs both heads. critic_loss is Eq. 4.6 written as a two-class cross-entropy, with the label "did this step's credit exceed 0". value softmaxes each head and keeps the lower chance.

Day one: the agent has never practised. Where does its first pile of attempts come from? Day ten: the shop has redesigned its home page and moved the search box. What happens to an agent that stopped learning on day one?

Here is DigiRL's answer in plain words. Start from a pile of the copier's own attempts, and learn from it without touching a phone. Then keep practising on a fleet of phones, so the agent always learns from the web as it is today.

The first pile

Chapter 0 named the two stages. Now we define them properly. Offline reinforcement learning learns from a fixed pile of earlier attempts, without making any new ones, like studying recordings of old matches. The pile here is the copier's own attempts at the training tasks, scored by the judge.

The paper used separate piles for its two kinds of run. For the general questions: 608 attempts to start the offline-then-online runs, and 1,552 for the offline-only runs. For the shopping tasks: 528 and 1,296. The training tasks number 545 general questions and 438 shopping tasks. The tests use the first 96 tasks of each test list.

Why two pile sizes? To keep the budgets fair. Offline-only runs use about 1,500 attempts. Offline-then-online runs start from about 500 and add about 1,000 online. That gives 608 + 1,000 = 1,608 against 1,552 for the general questions, and 528 + 1,000 = 1,528 against 1,296 for shopping. Roughly the same number of attempts either way.

In the offline phase, DigiRL trains on every successful attempt in the pile. It skips the task filter of Chapter 6, to make full use of the little data it has.

Online, one iteration at a time

Online reinforcement learning learns while it practises: make fresh attempts with the current agent, learn from them, repeat. Offline-to-online means doing the offline phase first, then switching to online practice, starting from where offline learning left off.

The online phase runs in iterations. An iteration is the formal name for a round (Chapter 6): one cycle of practice and training. Here is one iteration, in order:

1. Practise. Collect 16 fresh attempts across the emulators. Each is a roll-out from the home screen with the current policy, sampling at temperature 1.0.
2. Store. Add them to the replay buffer, which holds up to 5,000 attempts.
3. Teach the critics. Train the task critic (5 updates) and the step critic (5 updates) on the raw buffer, unfiltered.
4. Filter attempts. Keep the top 10% of attempts by Eq. 4.4.
5. Filter steps. Inside them, keep the steps that grade above 1/H by Eq. 4.3.
6. Copy. Train the policy on the kept steps with 20 updates of plain maximum likelihood (Eq. 4.2).
7. Ship. Send the new policy to the machines that run the phones.

The paper's Table 6 lists every setting. Here are the ones not yet covered, each in one line:

One new idea is in that table. The gradient-norm cap says no single nudge may be longer than 0.01, like a speed limit for each update. It stops one strange batch from yanking the policy far off course.

The authors note that they tuned Filtered BC and DigiRL with the same care. The only settings DigiRL adds are the critics' training settings and λ. Everything else is shared, which makes the comparison in Chapter 9 fair.

What flows into the buffer

Each step of each attempt becomes one record. Here is one, with the real field names from the released code and made-up values:

one transition record{
  "observation": "Previous Actions: \"action_type\": \"DUAL_POINT\", \"touch_point\": \"[0.1200, 0.5000]\", \"lift_point\": \"[0.1200, 0.5000]\", \"typed_text\": \"\" Goal: Go to costco.com, search for \"bose soundsport free\" and select the first entry</s>",
  "image_features": "float32[1408]",
  "action": "\"action_type\": \"TYPE\", ...",
  "reward": 0,
  "done": false,
  "next_observation": "Previous Actions: ... Goal: ...</s>",
  "next_image_features": "float32[1408]",
  "task": "Go to costco.com, search for \"bose soundsport free\" and select the first entry",
  "image_path": ".../<image_id>_<step>.png",
  "video_path": "...",
  "trajectory_reward": 1,
  "mc_return": 0.125
}

This is step 6 of a 9-step success. Step 5 tapped the search box, so that tap is the previous action in observation, the prompt of Chapter 1. This step types the query; its action string is shortened here, since only the tap format is shown in full in this lesson.

image_features and next_image_features are the fingerprints before and after the step. reward is 0 and done is false, because the attempt goes on. trajectory_reward is the whole attempt's final verdict, written onto every step once the attempt ends: 1.

mc_return is the ending's credit of Chapter 5. This step is three steps before the end, so it is 0.5³ = 0.125. Later the code stacks each fingerprint with the one before it, so the critics see 2,816 numbers. And the attempts are split 80/20 into a training buffer and a checking buffer.

The fleet

Practice is the slow part, so DigiRL spreads it over many machines. Think of a kitchen with one head chef and many line cooks. The host is the one machine with a graphics chip, a GPU, which does the training: the head chef. The workers are ordinary machines with only general-purpose processors, CPUs, which run the phones: the line cooks.

The host holds the current policy and sends a copy to every worker. Each worker runs several emulators and collects attempts. The host gathers them all, trains, and sends out the new policy. Each emulator starts every attempt from a fresh clone of a saved phone and has its own driver. An emulator that hits an error it cannot recover from is reset.

The machines were cloud machines, each with one Tesla T4 GPU and 16 CPUs. The policy and DigiRL's training fit in 12 GB of GPU memory. How fast attempts come in is the throughput: finished attempts per minute, like plates out of the kitchen per minute.

The paper measured it. One big server running everything managed 0.74 attempts a minute with 64 CPUs. The distributed fleet managed 0.99 at 32 CPUs and 1.74 at 64. With no communication or error-handling cost at all, the ceiling would be 3.55 at 128 CPUs. The fleet runs up to 64 emulators at once, with near-linear speedup. Pick a setup:

Turn the rates into hours. A thousand online attempts is the online budget of one run:

• one big server, 64 CPUs: 1,000 ÷ 0.74 = 1,351 minutes, about 22.5 hours;
• the fleet, 32 CPUs: 1,000 ÷ 0.99 = 1,010 minutes, about 16.8 hours;
• the fleet, 64 CPUs: 1,000 ÷ 1.74 = 575 minutes, about 9.6 hours;
• the ceiling, 128 CPUs: 1,000 ÷ 3.55 = 282 minutes, about 4.7 hours.

1,000 ÷ 1.74 = 575 min ≈ 9.6 h (distributed, 64 CPUs)  ·  1,000 ÷ 0.74 = 1,351 min ≈ 22.5 h (one server)

At 64 CPUs, spreading the phones over machines is 1.74 ÷ 0.74 = 2.35 times faster. The ceiling is still 3.55 ÷ 1.74 = 2.04 times above what the fleet achieved; communication and error handling eat the rest. Running 64 emulators on 128 CPUs gives each emulator 2 CPUs.

The paper's learning curves run to 960 attempts. At 16 attempts per iteration, that is 960 ÷ 16 = 60 iterations. At 1.74 attempts a minute, it is about 9.2 hours of practice. A thousand attempts is a working day, not a long weekend.

Why practice never stops

Chapter 0 showed that the web is non-stationary: sites change from week to week. The paper's experiment makes that concrete. The authors trained a policy on attempts from 1 to 3 June. From 7 to 11 June, one copy was frozen, and another kept learning on fresh attempts. The frozen policy got gradually worse. The learning one held its level.

The same drift hurt the offline phase too. By the time the online runs happened, the offline pile was already out of date. Attempts recorded on a version of the web that no longer exists are stale data, like last year's map. The paper says plainly that this made learning harder.

What online practice buys

Compare DigiRL offline-only with DigiRL offline-then-online, on the tests. General questions: 62.8% offline, 71.9% offline then online. Shopping tasks: 45.8% offline, 67.2% offline then online. Online practice adds 9.1 and 21.4 points.

What about skipping the offline phase entirely? The camera-ready version of the paper adds that offline-then-online converges faster than online learning from scratch, and ends no worse. The offline pile is a cheap head start; the online practice does the rest.

A sick emulator

The symptom. A burst of failed attempts that end after one or two steps, all from one worker.

The measurement. Count blank screenshots per worker. The code calls a screenshot blank when more than 2,300,000 of its 2,462,400 pixels are near-white.

The fix. Retry the screenshot, up to 5 times, 5 seconds apart, and reset any emulator that keeps failing. Never let one sick phone fill the buffer with false failures: the critics would learn that perfectly good screens lead nowhere.

The host loop, as code

pythonpolicy = load_sft_copier()                                    # the copier, the starting point
buffer = load_offline_attempts()                              # e.g. 608 general-question attempts
train_offline(policy, successes_only(buffer))                 # offline phase: every success

for it in range(n_iterations):                                # online phase
    attempts = workers.collect(policy, n=16, temperature=1.0) # 16 fresh attempts per iteration
    buffer.add(stack_last_two_frames(attempts))               # 1408 + 1408 = 2816 numbers
    task_critic.fit(buffer, updates=5)                        # Eq. 4.5
    step_critic.fit(buffer, updates=5)                        # Eq. 4.6
    kept = top_fraction(buffer, score=r_minus_v_instruct, p=0.10)          # Eq. 4.4
    steps = [s for s in kept if step_advantage(s) > 1 / H]               # Eq. 4.3
    policy.fit_max_likelihood(steps, updates=20, batch=128, lr=3e-3)      # Eq. 4.2
    workers.send(policy)                                      # the host ships the new weights

This is our sketch of the released training loop, with the equations named. The first three lines are the offline phase. The loop is one iteration per pass, in exactly the order of the list above. A saved copy of a model's numbers at one moment is a checkpoint, like a save file in a game; the workers load the host's latest checkpoint before each batch of practice.

Ninety-six shopping tasks the agent never practised on. Ninety-six general questions it never practised on. The judge scores each attempt. How many does each kind of agent finish?

Here is the answer in plain words. On tasks it never practised, the same small model goes from 17.7% to 67.2% on shopping, and from 14.6% to 71.9% on general questions. Most of the gain is the way back from mistakes. This chapter reads the evidence behind those numbers, including the parts that are less flattering.

How the test works

A fixed set of tasks that everyone tests on is a benchmark, like a standard exam. The tasks used for practice and the tasks kept aside for the exam form a train and test split: homework versus the final. An agent that only memorised its homework would do well on train and poorly on test.

The paper tests on the first 96 tasks of each test list, for general questions and for shopping. The judge of Chapter 2 scores every attempt. The offline runs were repeated 3 times, and the offline-then-online runs 2 times. Each result is reported as a mean and a standard deviation (written ±): how far the result of a repeated run typically lands from the average, like the spread of darts around the middle.

The table, row by row

The paper's Table 1 compares four families of agents. Meet them before you see the bars.

1. Prompted large models. GPT-4V and Gemini 1.5 Pro, used in two ways. Set-of-Marks draws numbered boxes on the screenshot, so a model can answer "press box 7" instead of giving coordinates: numbered stickers on the buttons. AppAgent lets the model use notes about each app.
2. Copiers. CogAgent, with 18 billion parameters, and AutoUI, with 1.3 billion, both trained on people's recordings. AutoUI is the starting point of DigiRL.
3. Offline. Filtered BC and DigiRL, each trained only on the offline pile.
4. Offline, then online. Filtered BC and DigiRL again, with online practice added.

In this lesson the two task sets are "the general questions" and "the shopping tasks"; the bench uses the dataset's own labels, General and Web Shopping. Pick a task set and a split, then tap any row to compare it with DigiRL.

The comparisons that matter

Each of these is a difference between two Table 1 cells:

• Shopping test, the copier to DigiRL: 67.2 − 17.7 = 49.5 points, with the same 1.3-billion-parameter model.
• General test, the copier to DigiRL: 71.9 − 14.6 = 57.3 points. The best prompted agent there (AppAgent with GPT-4V, 17.7%) to DigiRL: 71.9 − 17.7 = 54.2 points.
• Shopping test, CogAgent to DigiRL: 67.2 − 38.5 = 28.7 points.
• Offline then online, Filtered BC to DigiRL: 67.2 − 57.8 = 9.4 points on the shopping test, and 71.9 − 61.5 = 10.4 points on the general test.

67.2 − 17.7 = 49.5 points (same 1.3B model, shopping test)

Notice what does not win. CogAgent is 18 ÷ 1.3, about 14 times larger than AutoUI, and reaches 38.5% on the shopping test. The prompted giants stay under 20% on the test split. Size and reading the web do not substitute for practice on the phone itself.

The honest reading

Offline-only DigiRL does not beat Filtered BC everywhere. On the general training tasks it scores 46.9% against Filtered BC's 51.7%. On the shopping test they tie at 45.8%. On a fixed, stale pile, DigiRL's extra machinery buys little; offline, it even skips the task filter (Chapter 8).

The gap opens online. There, the filters choose what to learn from fresh data that the agent itself produced, including its own mistakes and escapes. That is where DigiRL pulls 9 to 10 points ahead of the strongest prior method.

How fast it learns

The paper also plots success against the number of attempts, up to 960. DigiRL improves faster with fewer attempts than Filtered BC. The two runs of each method were started at least two days apart. Collecting attempts is the slow part, so attempts track wall-clock time: a faster curve here is a faster day in the lab.

Where the gain comes from

The authors sorted failed attempts into three kinds: fails to recover from a mistake, gets stuck midway, and arrives at the wrong goal. (The appendix splits these into six finer kinds.) Practice cut "fails to recover" the most, and most consistently.

Looking at the attempts themselves, they found a habit. DigiRL learned to go back to a screen with a high value, often the start screen, and try again. That is Chapter 5's Back step, kept because the critic saw the value jump.

Its attempts are also shorter. Averaged over all attempts in two runs, DigiRL takes about 6.5 steps on the general questions against 7.7 for Filtered BC. On shopping it takes about 11.1 against 13.4.

(6.31 + 6.64) ÷ 2 = 6.475 vs (8.08 + 7.36) ÷ 2 = 7.72  ·  (11.35 + 10.86) ÷ 2 = 11.105 vs (12.05 + 14.72) ÷ 2 = 13.385

What each piece is worth

Recall the ablation of Chapter 4: remove or change one part, and measure what changes. The paper ran four on the shopping tasks:

• training the critics with cross-entropy instead of squared error: about 12% better (Chapter 7);
• grading individual steps instead of whole attempts: about 12% more efficient (Chapter 5);
• the automatic curriculum: about 25% faster learning (Chapter 6);
• the hard filter matched the best tuned exponential weighting, and plain AWR did worse (Chapter 4).

One more. On the general questions, the authors tried a step budget of 20 instead of 10. The longer budget made both methods learn faster. DigiRL stayed about 100 attempts ahead of Filtered BC either way.

Check the judge

Every number above was scored by the judge of Chapter 2. If the judge favoured one method, every row would move. So the paper checked: success rates scored by the judge and by people differ by under 3 points for almost every agent. A bias would have to hide inside those 3 points.

Reading a paper's numbers

The symptom. You quote a gain from the paper's text, and a reader cannot find it in the table.

The measurement. Put every number you quote next to its table cell. Three sentences in DigiRL's text restate Table 1 with numbers that do not match it, and one computes a relative gain from a training-set cell.

The fix. Cite the table. This lesson uses Table 1 throughout, and repeats none of those sentences. (Evaluation statistics covers reading ± and repeated runs; agent evaluation covers judging agents.)

You can now read the DigiRL paper and explain each design decision. Why a judge instead of hand-written checks. Why one fingerprint per screen. Why keep-or-drop instead of weights. Why blend the ending with a critic. Why rank by surprise. Why cross-entropy. Why keep practising. Let's lock it in.

The one-paragraph summary

DigiRL trains a 1.3B phone agent (AutoUI-Base: a 200M T5 language model and a frozen 1.1B BLIP-2 image encoder that turns each screenshot into 1,408 numbers) with offline, then offline-to-online reinforcement learning on Android emulators browsing the live web. A few-shot Gemini 1.5 Pro judge gives a 0 or 1 at the end of each attempt, with 2.8% average error against people. From a replay buffer, a task critic keeps the top 10% of attempts by reward minus the task's value (Eq. 4.4). A step critic keeps the steps whose blended grade beats 1/H (Eq. 4.3, λ = 0.5). The policy copies what is kept by maximum likelihood (Eq. 4.2), and both critics are trained with cross-entropy. On 96 held-out shopping tasks, success rises from 17.7% to 67.2%; on general questions, from 14.6% to 71.9%.

The numbers that matter

If you build a DigiRL-style trainer

1. Decode actions exactly as the model writes them: [y, x], then scale to the real screen.
2. Measure your judge against people on a sample before you train, and know its error rate.
3. Skip the judge when the screen did not change, and never send a blank screenshot.
4. Store every step with the attempt's final verdict and its discounted credit.
5. Retrain both critics every iteration on the raw buffer, with cross-entropy.
6. Keep the top 10% of attempts by reward minus task value, then the steps above 1/H.
7. Copy what is kept with plain maximum likelihood; there is no sharpness knob to tune.
8. Keep practising online; the web will change under a frozen policy.
9. Watch attempt lengths and failure kinds, not only the success rate.
10. Quote the results table, never a restatement.

One iteration, as code

pythonimport numpy as np

def step_advantage(V, r, lam=0.5):                 # Eq. 4.3, for one attempt
    T = len(r); h = np.arange(1, T + 1)
    m = lam ** (T - h) * r[-1]                           # the ending's credit
    return m + (1 - m) * (V[1:] + r - V[:-1])           # blended with the critic's one-step guess

def one_iteration(policy, task_critic, step_critic, buffer, workers, H):
    buffer.add(workers.collect(policy, n=16, temperature=1.0))   # practise: 16 fresh attempts
    task_critic.fit(buffer, updates=5)                    # Eq. 4.5, cross-entropy
    step_critic.fit(buffer, updates=5)                    # Eq. 4.6, cross-entropy, two heads

    r = np.array([a.reward for a in buffer])                # each attempt's verdict, 0 or 1
    v = np.array([task_critic(a.task) for a in buffer])
    A = r - v                                               # Eq. 4.4: how surprising
    kept = [a for a, s in zip(buffer, A) if s >= np.quantile(A, 0.9)]   # top 10%

    steps = []
    for a in kept:                                         # inside each kept attempt
        V = step_critic.values(a.screens)                   # lower of two heads, plus a final 0
        grades = step_advantage(V, a.rewards)
        steps += [s for s, g in zip(a.steps, grades) if g > 1 / H]

    policy.fit_max_likelihood(steps, updates=20, batch=128, lr=3e-3)   # Eq. 4.2
    workers.send(policy)                                    # ship the new checkpoint

Our sketch, assembled from the chapters' pieces with the equations named; the released code is organised differently but runs the same steps in the same order.

Where it sits in the field

• The judge extends Pan et al. 2024, "Autonomous Evaluation and Refinement of Digital Agents" (arXiv 2404.06474), which also introduced the Filtered BC baseline used here.
• The learning step builds on advantage-weighted regression (arXiv 1910.00177), then replaces its exponential weights with a hard filter.
• The step grade is a pared-down generalized advantage estimation: only the one-step and the to-the-end estimates.
• The tasks come from Android in the Wild (arXiv 2307.10088): 715,000 recorded attempts over 30,000 instructions. DigiRL uses its General and Web Shopping subsets.
• The starting policy is AutoUI (arXiv 2309.11436). The large copier baseline is CogAgent (arXiv 2312.08914). The prompted baselines use AppAgent (arXiv 2312.13771) and Set-of-Marks.

Where it sits on the path

This paper is in Act 8, training, of the path Computer Use & Multimodal Agents. Just before it: CS224R: RL for Robot Foundation Models, which covers offline-to-online learning for large policies, then Agentic RL for GUI agents (coming soon, a Gleam in this series). Just after it: UI-TARS-2 (coming soon), the next paper on this path, which trains a GUI agent with multi-turn reinforcement learning. Then CS 8803-LLM Session 13: Self-Play RL I: Absolute Zero.

Other lessons of this series are on their way: GUI grounding, OSWorld, sandbox fleets and trajectory data.

The limits, in the authors' words and ours

• Trained and tested only on the two Android in the Wild task sets, for lack of compute, not on everything a phone can do.
• The score is a commercial model's opinion: 2.8% error on average, and every tested result is judge-scored.
• Emulators on the live web, not physical phones; sites with captchas and the add-to-cart step were removed.
• The offline pile was stale by the time of the online runs.
• The algorithm is deliberately simple, meant as a base for later work.
• The paper's text restates some results inconsistently; the table is the source.

Provenance

"DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning." Hao Bai and Yifei Zhou (equal contribution), Mert Cemri, Jiayi Pan, Alane Suhr and Sergey Levine (UC Berkeley; Bai also UIUC), and Aviral Kumar (Google DeepMind; CMU in the camera-ready). arXiv 2406.11896, posted 14 June 2024. Published at NeurIPS 2024 as a poster. Code and project page under the Apache 2.0 licence: github.com/DigiRL-agent/digirl and digirl-agent.github.io. Four checkpoints are released, one per method and task set; the repository's README lists its own re-evaluation scores for them, which this lesson does not quote.

Keep going

• Imitation learning: why copying compounds errors.
• Policy gradients: the on-policy alternative.
• Actor-critic: critics and one-step estimates.
• GAE: the general advantage recipe.
• PPO: the on-policy workhorse DigiRL chose not to use.
• Reward learning: where scores come from.
• Agent evaluation: judging agents.

Go back to the phone at the top. Throw a pop-up at the copier, then at DigiRL. You now know every piece of what happens next: what the agent sees, who scores it, which of its steps it learned from, and why it practised. Then open Teach and explain Eq. 4.3 on the whiteboard, or Present the whole paper in five minutes.