GUI Grounding

You are editing a drawing on a big 4K monitor, and you ask an AI assistant to save your work. It takes a picture of the screen, thinks for a second, and the mouse pointer jumps to the toolbar. It clicks. Nothing saves. The click landed on Export, the button next to Save.

The assistant read your request correctly. It knew which button you meant. It still missed, by about one button's width. This lesson is about that miss: where it comes from, how to measure it, and how to stop it.

An assistant with a mouse

Think of a helper on a video call. They can see your screen, but they cannot touch it. So they tell you where to click: "the second button from the left, near the top." A chat assistant that has been given a computer works the same way, except that its words move a real mouse.

A model that works a computer the way you do, looking at a picture of the screen and answering with a click or a key press, is a computer-use agent. Inside, it is a language model, the same kind of model as a chat assistant. The difference is what it answers with: an action instead of a paragraph.

Its life is a loop. It looks at the screen, decides what to do, acts, and then looks again to see what changed. A task like "save this drawing as a PDF" can take a dozen turns of that loop. This lesson zooms in on one turn: one look and one click. A later lesson in this series follows the whole loop.

What a screen is made of

Look closely at any screen and you find tiles, like a mosaic. A screen is a grid of tiny coloured dots. Each dot is a pixel. Every letter, every picture and every button on the screen is made of those dots.

How many pixels a screen has across and down is its resolution, like the number of tiles across and down a mosaic. A screen 3,840 pixels across and 2,160 down is often called 4K. Multiply the two sides: 3,840 × 2,160 = 8,294,400 pixels, about 8.3 million.

Each pixel is itself three numbers: how much red, how much green and how much blue it shows. So one 4K picture is 8,294,400 × 3 = 24,883,200 numbers. Engineers write pixels as px, so a 4K screen is 3,840 × 2,160 px.

A copy of every pixel on the screen at one moment is a screenshot: a photo of the screen, taken from the inside. It is the only thing the assistant ever sees of your computer. If you want to see those 24.9 million numbers laid out in memory, the Pixel Buffers lesson opens them up.

From words to a place

The windows, buttons and menus you click are the graphical user interface, or GUI. It is the dashboard of a car: the part you see and touch, on top of machinery you never look at.

When you say "click Save", you name a thing. A mouse needs a place. Think of asking a friend for the salt: the friend points at it, and the word becomes a spot on the table. Turning words ("the Save button") into a place on the screen is called grounding: tying the words to the ground they stand for. This whole lesson is about grounding.

A place on the screen is two numbers, like a seat in a theatre: which seat along the row, and which row from the front. How far a place is from the left edge is called x. How far it is down from the top edge is called y. Both are counted in pixels. Together they are the place's coordinates, written (x, y).

One warning before we go on. On a school graph, y grows as you go up. On a screen, y grows as you go down: the top row of pixels is y = 0. Every number in this lesson counts that way.

The button the model should click is the target: the bullseye. In our scene the target is Save. It is a square 32 pixels on a side, and its centre sits 1,200 pixels from the left edge and 180 pixels down: (1,200, 180).

Who moves the mouse

The model itself only reads and writes. It cannot take a screenshot, and it cannot move a mouse. Picture a navigator who can only look at photos and talk, sitting next to a driver with their hands on the wheel. The ordinary program around the model, which takes the screenshot, sends it, reads the answer and moves the real mouse, is the harness. It is the driver.

The harness does not send the full screenshot. 8.3 million pixels is a lot for a model to read, and reading costs time and memory. The agent also sends a fresh screenshot at every turn of its loop, so that cost comes back on every step. So the harness sends a smaller copy of the screenshot, about a third as wide, like the thumbnail of a photo. Chapter 1 shows exactly how much smaller, and why.

One click, end to end

Now follow one click all the way, with real numbers. Save is the 32 × 32 px button centred at (1,200, 180) on the 3,840 × 2,160 screen. The harness shrinks the screenshot to a copy 1,316 × 728 px. Every length across shrinks by 1,316 ÷ 3,840 = 0.3427. Every length down shrinks by 728 ÷ 2,160 = 0.3370.

So in the copy, Save's centre is at 1,200 × 0.3427 = 411.25 across and 180 × 0.3370 = 60.67 down. Save itself is now about 11 pixels wide. The model looks at the copy and writes its answer as plain text: click(411, 61). That format is made up for now. Chapter 2 shows the real ones.

The harness reads the two numbers and carries them back to the screen. It multiplies 411 by 3,840 ÷ 1,316, which gives 1,199.3. It multiplies 61 by 2,160 ÷ 728, which gives 181.0. The mouse clicks (1,199, 181). That is 1.4 pixels from the centre of Save, well inside the button. This time, the click lands.

Step through that trip yourself. Each press of Next step adds one stage, in the order the harness runs them.

Here is the same trip as arithmetic, one line per step. Two numbers in it, 0.3427 and 0.3370, come back in every chapter.

worked exampleThe screen                 3,840 × 2,160 px = 8,294,400 pixels (× 3 colours = 24,883,200 numbers)
Save button                32 × 32 px, centre (1,200, 180), box x 1,184 to 1,216, y 164 to 196    (illustrative)
The copy the model sees    1,316 × 728 px
  across                   1,316 ÷ 3,840 = 0.3427
  down                     728 ÷ 2,160   = 0.3370
Save in the copy           centre 1,200 × 0.3427 = 411.25,  180 × 0.3370 = 60.67;  about 11 × 11 px
The model's answer (text)  click(411, 61)                                           (format illustrative)
Back to the screen         411 × 3,840 ÷ 1,316 = 1,199.3 → 1,199
                           61 × 2,160 ÷ 728   = 181.0  → 181
The click                  (1,199, 181): 1.4 px from the centre, inside 1,184 to 1,216 and 164 to 196: on Save
If the harness forgets     it clicks (411, 61): 789 px left and 119 px up, 798 px away

Going down into the copy, every length is multiplied by 0.3427 across and 0.3370 down. Coming back, the harness divides by the same two numbers. That is the same as multiplying by 3,840 ÷ 1,316 = 2.918 across and 2,160 ÷ 728 = 2.967 down.

The model writes whole numbers, so its answer can be off by up to half a pixel of the copy. Multiplied back up by 2.918, that is at most about 1.5 pixels on the screen. That is small, and harmless for a 32 px button. Keep it in mind: Chapter 2 meets answer formats whose rounding is not harmless at all.

Here is the whole turn as the harness runs it, written in Python. The function names, like take_screenshot, are placeholders for whatever your own system calls them.

python# One turn of the loop, as the harness runs it (names like take_screenshot are placeholders)
shot = take_screenshot()                         # 3840 x 2160 pixels
copy = shot.resize((1316, 728))                  # the smaller copy the model is sent
text = model.ask("Click the Save button.", copy) # the model answers with text: "click(411, 61)"
ax, ay = parse_click(text)                       # the two numbers: 411, 61
x = round(ax * 3840 / 1316)                      # back to the screen: 1199
y = round(ay * 2160 / 728)                       #                     181
mouse_click(x, y)                                # the real mouse moves and clicks

Every chapter from here opens one of these lines: the resize (Chapter 1), the answer and the two multiplications (Chapter 2), a different kind of answer (Chapter 3), the model behind model.ask (Chapters 4 and 5), and what to do when the click misses (Chapter 6).

Three places a click can go wrong

The trip worked. So why did the assistant in the opening scene click Export? Each step of the trip is a suspect, and each has a number that makes it real.

The copy. In the copy, Save is 11 px wide. Chapter 1 shows that the model reads the copy in squares 28 px wide, so Save is less than half of one square. The model can tell roughly where Save is, but not exactly where inside its square.

The numbers. If the harness forgets that 411 means a place in the copy, it clicks x = 411 on the real screen. That spot is 789 px to the left of Save and 119 px above it. Chapter 2 hunts this bug and its quieter cousins.

The model itself. The model learned to point from millions of examples (Chapter 4), and how well it points is measured on tests (Chapter 5). Even with a perfect copy and perfect arithmetic, a model can choose the wrong button.

The miss on the designer's monitor

Why is the laptop's Save twice as wide in its copy? With the same settings, the harness shrinks a 1,920 × 1,080 laptop screen to the very same 1,316 × 728 copy. That is a squeeze of 0.6854, half as hard as the 4K screen's 0.3427. So the same 32 px button is 32 × 0.6854 = 21.9 px in the laptop's copy, and 32 × 0.3427 = 11.0 px in the 4K copy. The bigger screen made the button smaller.

Back to the toolbar at the top

The instrument at the top of the page is this trip, run forty times. The dots are forty tries at Save. "Squeezed" is the 1,316 × 728 copy you just followed. The blue squares are the squares the model reads, drawn at the size each one covers on your screen.

On a big screen with small buttons, squeezing makes buttons narrower than one square, and that is where the clicks start to miss. Professional software, the kind people use for design, engineering and science, is full of such buttons. On a test built from real professional screens, the best model of its time found the right target 18.9% of the time: fewer than one time in five. The rest of this lesson builds every piece of that sentence.

Here is the road, in order:

1. From screen to tokens: how a model cuts a screenshot into squares, and how many squares a screen costs.
2. Coordinates are text: how a click becomes numbers and back, and the bug that sends it to the wrong corner.
3. Numbers instead of places: handing the model a numbered list to pick from, and what breaks when the list is empty or crowded.
4. Where the examples come from: how web pages label millions of buttons for free.
5. Measuring grounding: the tests, the scores, and why professional screens break them.
6. Debugging a miss: reading where the misses land, and zooming in.
7. Field Guide: the checklist, the numbers and where each one came from.

The harness holds a 4K screenshot: 8.3 million pixels, 24.9 million numbers. The model has to read it before it can answer. But a language model reads text, one small piece at a time. How does a picture become something it can read, and why does the harness shrink it first?

How a language model reads

You read this sentence one word at a time, left to right. A language model does the same, with smaller pieces. A language model reads text in small pieces, about a word or part of a word each. Each piece is a token. A short request like "Click the Save button" is a handful of tokens.

Inside the model, each token becomes a long list of numbers that describes it. The model reads the whole row of lists, in order, and then writes its answer one token at a time. That is all it can read: a row of tokens.

So to read a picture, the model needs the picture as tokens too. The blue squares in the hero at the top of the page are exactly those picture tokens, drawn at the size each one covers on your screen. Here is how a picture is cut into them.

Cut, describe, merge

Think of a sheet of stamps: one picture, perforated into small squares you can tear apart. The model first cuts the picture into small squares, 14 pixels on a side. Each square is a patch. One patch holds 14 × 14 = 196 pixels, and each pixel is three numbers, so one patch is 196 × 3 = 588 numbers.

Next, each patch gets a description. Picture a translator who looks at one stamp and sums it up in a few words: "blue sky, a white edge at the bottom." The part of the model that turns each patch into a list of numbers describing it is the vision encoder. Its output for each patch is one list, the same kind of list a word becomes.

Many patches are needed to cover a screen, and every one becomes a token the language model must read. So one family of models saves work. Think of four tiles grouted into one bigger tile. It joins every 2 × 2 block of neighbouring patches into one. That join is a merge, and after it one token covers 28 × 28 pixels. The merge cuts the number of tokens to a quarter.

Last, the picture needs edges, the way a quote needs quotation marks. Two extra tokens, one at each end, mark where the picture starts and stops. Without them the model could not tell where your words end and the picture begins.

Run the steps on the smallest picture the paper behind this patch-and-merge method uses as its example: 224 × 224 pixels. Cut: 224 ÷ 14 = 16 patches across and 16 down, so 16 × 16 = 256 patches. Merge: 256 ÷ 4 = 64 tokens, each 28 × 28 pixels. Mark: 64 + 2 = 66 tokens. The Vision Transformer lesson builds the encoder itself, if you want to see inside it.

Look at the teal outline in the device. The Save button covers parts of eight tokens. No token is "the Save button"; each token is a square of picture, and Save is spread across several. That will matter as soon as buttons get small.

What a whole screen costs

Now count a whole screen. The squares must fit the picture exactly, so the model first rounds each side to the nearest multiple of 28. Then the token count is the number of squares down times the number across.

Take a laptop screen, 1,920 × 1,080. Down: 1,080 ÷ 28 = 38.57, which rounds to 39 squares, so the height becomes 39 × 28 = 1,092. Across: 1,920 ÷ 28 = 68.57, which rounds to 69, so the width becomes 1,932. Tokens: 39 × 69 = 2,691.

Now the 4K screen. Down: 2,160 ÷ 28 = 77.14, which rounds to 77, a height of 2,156. Across: 3,840 ÷ 28 = 137.14, which rounds to 137, a width of 3,836. Tokens: 77 × 137 = 10,549. The 4K screen costs about four times the laptop's tokens, because it has four times the pixels.

Why the harness shrinks it

More tokens take more memory and more time. An agent also sends a fresh screenshot at every step of a task, so screenshots pile up in what the model must read. Ten steps on a 4K screen is over 100,000 picture tokens.

That is why the people who built one open model suggest capping each picture at 256 to 1,280 tokens, in their words "to balance speed and memory usage". The same model accepts anything from 4 to 16,384 tokens per picture. Most harnesses pick a cap well below the top.

A few words for what happens next. Making a smaller copy of a picture is downscaling it, like photocopying a page at 34%. The most tokens the harness lets one picture use is its token budget. Think of a suitcase weight limit: you pack less, or you squeeze.

How much smaller the copy is, as a share of the screen, is its scale, like the scale on a map: 0.34 means about a third as wide. The shape of a picture, its width against its height, is its aspect ratio. A TV is wide; a phone held upright is tall. The shrink must keep the aspect ratio. Otherwise circles become ovals and text leans.

The resize rule

Picture a tailor who shrinks a pattern to fit less cloth without changing its shape. The rule that picks the copy's size from the screen's size and the budget is the resize rule. It is short, and we can build it in four steps.

Step 1: the budget in pixels. Each token is 28 × 28 = 784 pixels, so a budget of B tokens allows B × 784 pixels. For B = 1,280 that is 1,003,520 pixels.

Step 2: does it fit? Round each side to a multiple of 28, as above. If the rounded picture has no more pixels than the budget, keep it. Nothing shrinks.

Step 3: if not, shrink both sides by the same factor. Call the factor β (the Greek letter beta). Dividing both sides by β keeps the shape. We want (h ÷ β) × (w ÷ β) to equal the budget, so β × β = h × w ÷ budget, and β is the square root of that.

Step 4: round each shrunk side down to a multiple of 28. Down, not to the nearest, so the copy can never go over the budget. Then count the tokens. (The real code also grows a tiny picture up to a minimum of 4 tokens; screens never need that.)

h, w

the screenshot's height and width, in pixels

B

the token budget: the most tokens one picture may use

P_max

the budget in pixels: B tokens of 28 × 28 pixels each

β

how many times smaller each side must get (1 means no shrink)

h̄, w̄

the copy's height and width, whole multiples of 28

⌊ ⌋

round down to a whole number

N

the number of picture tokens the model reads (plus 2 markers)

Run it on the 4K screen at three budgets. Every line below is one step of the rule.

worked exampleA 4K screen under a 1,280-token budget
  P_max   = 1,280 × 28 × 28                 = 1,003,520 pixels
  h × w   = 2,160 × 3,840                   = 8,294,400 pixels: too many, so shrink
  β       = √(8,294,400 ÷ 1,003,520) = √8.2653 = 2.8749
  height  = 2,160 ÷ 2.8749 = 751.32 → ÷ 28 = 26.83 → down to 26 → 26 × 28 = 728
  width   = 3,840 ÷ 2.8749 = 1,335.68 → ÷ 28 = 47.70 → down to 47 → 47 × 28 = 1,316
  tokens  = 26 × 47 = 1,222   (1,224 with the 2 markers; 4,888 patches before the merge)
  scale   = 1,316 ÷ 3,840 = 0.3427 across,  728 ÷ 2,160 = 0.3370 down

The same screen at the default budget (16,384 tokens: P_max = 12,845,056)
  2,160 ÷ 28 = 77.14 → 77 → 2,156;  3,840 ÷ 28 = 137.14 → 137 → 3,836
  2,156 × 3,836 = 8,270,416 ≤ 12,845,056: no shrink;  tokens = 77 × 137 = 10,549

At 4,096 tokens (P_max = 3,211,264)
  β = √(8,294,400 ÷ 3,211,264) = √2.5829 = 1.6071
  2,160 ÷ 1.6071 = 1,344.0 → 48 × 28 = 1,344;  3,840 ÷ 1.6071 = 2,389.3 → 85 × 28 = 2,380
  tokens = 48 × 85 = 4,080;  scale 0.6198 across

What happens to a 32 px icon, and to the 76 px average target
  budget 16,384:  32 × 0.9990 = 32.0 px = 1.14 tokens    76 → 75.9 px = 2.71 tokens
  budget  4,096:  32 × 0.6198 = 19.8 px = 0.71 tokens    76 → 47.1 px = 1.68 tokens
  budget  1,280:  32 × 0.3427 = 11.0 px = 0.39 tokens    76 → 26.0 px = 0.93 tokens

Two facts in that block surprise most people. First, the default budget keeps a 4K screen whole: 8.3 million pixels fit under its 12.8 million. The squeeze happens only when someone sets a smaller budget, which is common, because tokens cost time and money.

Second, under a fixed budget the screen's size stops mattering. A 1,920 × 1,080 laptop screen under the 1,280 budget also becomes 1,316 × 728, at a scale of 0.6854. Every 16:9 screen ends up as the same copy. A bigger screen is simply squeezed harder.

Here is the rule as code, the same steps in the same order. It is a faithful copy of the open model's own function, with plain comments. The outputs printed under it are the three rows of the worked example.

pythonimport math

def smart_resize(h, w, factor=28, max_tokens=16384, min_tokens=4):
    """The resize rule: pick the copy's size from the screen's size and the budget."""
    max_pixels = max_tokens * factor * factor         # the budget, in pixels
    min_pixels = min_tokens * factor * factor
    hb = max(factor, round(h / factor) * factor)      # round each side to a multiple of 28
    wb = max(factor, round(w / factor) * factor)
    if hb * wb > max_pixels:                          # too big: shrink both sides by beta
        beta = math.sqrt(h * w / max_pixels)
        hb = math.floor(h / beta / factor) * factor   # round DOWN, so we stay in budget
        wb = math.floor(w / beta / factor) * factor
    elif hb * wb < min_pixels:                        # too small: grow both sides
        beta = math.sqrt(min_pixels / (h * w))
        hb = math.ceil(h * beta / factor) * factor
        wb = math.ceil(w * beta / factor) * factor
    return hb, wb

for budget in (16384, 4096, 1280):
    h, w = smart_resize(2160, 3840, max_tokens=budget)
    print(budget, w, h, (h // 28) * (w // 28))
# 16384 3836 2156 10549
# 4096 2380 1344 4080
# 1280 1316 728 1222

Read it line by line. The first two lines of the body turn the budgets into pixels. The two round lines make each side a whole number of squares. The if branch is steps 3 and 4: one factor for both sides, then round down. The elif branch grows tiny pictures, which a screen never triggers. The loop prints the three rows you worked by hand.

What the squeeze does to a button

Now the hero's question has numbers. A 32 px icon on a 4K screen is 32 × 0.3427 = 11.0 px in the 1,280-token copy. A token is 28 px wide, so the icon is 11.0 ÷ 28 = 0.39 of a token.

Chapter 5 meets a test built from professional apps. Its average target is 76 px across on a 4K screen. In the same copy it is 26 px: 0.93 of a token. At the default budget the same two targets are 1.14 and 2.71 tokens wide.

What does "under one token" mean for the model? When a button is narrower than one token, the token that holds it also holds part of its neighbour and part of the bar around it. The model can still tell roughly where the button is. But it has nothing finer than a square to point with, and the square is wider than the button.

Older answers: one square, or tiles

Not every model resizes this way. Older models stretch every picture into the same square, for example 224 × 224 pixels. That is a fixed resolution, like forcing every photo into the same square frame.

Do the damage on a 4K screen. Across, 224 ÷ 3,840 keeps 5.8% of the width. Down, 224 ÷ 2,160 keeps 10.4% of the height. The 32 px icon becomes 1.9 × 3.3 pixels: a smear inside a single 14 px patch, squashed sideways because the width shrank harder than the height.

One well-known screen model kept a 224 × 224 view, 256 patches, and added a second, sharper 1,120 × 1,120 view, 6,400 patches, just to read screens. Even there the icon is 9.3 × 16.6 pixels, still squashed. Try both squares in the device above.

The other answer is to cut the screen up. Cutting a big screenshot into several smaller tiles and reading each one is tiling, like reading a big map one page of the atlas at a time. One grounding model reads up to 36 tiles of 224 px, about 1,344 × 1,344 pixels at most. It even drops the usual small overview picture, because 224 px is too small to be useful for screens.

Another model reads up to 6 tiles of 448 px plus a small thumbnail. A third splits a phone screen into two halves, chosen by its shape. The model this lesson follows does neither. It accepts any size and spends more tokens on bigger pictures, which is why its budget is the knob that matters.

The icons that went missing

The model's answer comes back as plain text: (411, 61). Nine characters: two brackets, a comma, a space and five digits. The model wrote them the same way it writes any word. Your harness now has to turn those characters into a mouse position on a 3,840-pixel screen. What do they mean, and what can go wrong on the way?

Where (0, 0) is

Every place needs a starting point. In a theatre, you count seats along the row from one end, and rows back from the stage. On a screen, you count from the top-left corner. The top-left corner, (0, 0), is the origin. From there, x grows to the right and y grows down.

Every tool in this lesson uses this origin, from the open models to the vendor tools. That sounds obvious until you meet a chart library or a 3-D tool that puts y = 0 at the bottom. If a click lands mirrored top to bottom, check the origin first.

Three ways to say "there"

A model can answer "where is Save?" in three ways. It can give a point, a box, or the name of an element from a list.

Think of a map. You can stick a pin in a town, or draw a circle around it. A point is one place: two numbers. A box is a rectangle: four numbers, its left, top, right and bottom edges. Engineers call it a bounding box. Save's box on our screen runs from x = 1,184 to 1,216 and from y = 164 to 196.

A mouse cannot click a rectangle, so when a model answers with a box, the harness clicks the box's centre: halfway between left and right, halfway between top and bottom. The tests in Chapter 5 score box answers the same way. The third way, a name from a list, needs no coordinates at all. It is the whole of Chapter 3.

Numbers are just text

There is no special channel for coordinates. The model writes digits as tokens, exactly like words. One early grounding model said so outright: it treats numbers "as natural languages", and writes a click like click (0.49, 0.40). An older approach added 1,000 new tokens to the model's vocabulary instead, one token per position across the picture.

Writing digits as text is simple, and it works with any language model. But it has one catch. The digits carry no unit of their own. "411" could be pixels of the copy, pixels of the screen, thousandths, or nothing at all. The model and the harness must agree on the unit in advance, and the harness must never forget it.

Three units, one target

Here is the same Save button, centred at (1,200, 180) on the 3,840 × 2,160 screen, written in the units real models use.

Pixels of the copy. Some models answer in pixels of the exact picture they were sent. For our 1,316 × 728 copy, that is (411, 61). One vendor's computer-use tool works this way too: it answers in pixels of whatever screenshot the harness sent it.

Thousandths. "A third of the way along the street" works for any street, long or short. Coordinates written as a share of the width and height, not in pixels, are normalized. One common choice counts in thousandths: 0 is the left edge, 1,000 the right edge, like a ruler with 1,000 marks stretched to fit any screen. Save is at 1,200 ÷ 3,840 × 1,000 = 312.5, written 313, and 180 ÷ 2,160 × 1,000 = 83.3, written 83. One model counts 000 to 999 instead.

Shares with two decimals. The same share, written as a decimal between 0 and 1 with two digits: (0.31, 0.08). One agent even writes its clicks as code, pyautogui.click(x=0.6756, y=0.4), with decimals like these in its examples.

A box in thousandths. Save's box, written as two corners: (308, 76), (317, 91). Its centre is (312.5, 83.5) in thousandths.

The round trip

Think of changing dollars into euros and back: if you use the wrong rate on the way back, you lose money. Screen to copy, copy to answer, answer back to screen: that is the round trip. Coming back is the reverse of going down.

Going down, every x is multiplied by the copy's width over the screen's width. Coming back, the harness multiplies by the opposite fraction: the screen's width over the copy's width. Screen width over copy width, 3,840 ÷ 1,316 = 2.918 here, is the scale-back factor. It is the map's scale read in reverse.

Down has its own factor: 2,160 ÷ 728 = 2.967. Why not the same number? Because the resize rule rounds each side on its own, to a whole number of squares. The copy is 0.3427 as wide as the screen but only 0.3370 as tall, so its shape is a hair wider than the screen's. Two directions, two factors.

For thousandths the trip back is simpler: x = n × 3,840 ÷ 1,000. And there is a bonus. Normalized units never depend on the copy. 313 thousandths is the same place on the screen and in any copy of it, whatever its size. The copy's size drops out of the arithmetic.

x, y

where the mouse clicks, in screen pixels

a, b

the model's answer in pixels of the copy

n, m

the model's answer in thousandths

W, H

the screen's width and height in pixels

w̄, h̄

the copy's width and height

Try the trip with every unit. The model in this device is perfect: it always knows Save's exact centre. So any error you see comes from the unit or from the harness, never from the model.

Here is the trip back as code. It parses the model's text and applies the right formula for the unit.

pythonimport re

W, H = 3840, 2160                     # the real screen
w_copy, h_copy = 1316, 728            # the copy the model was sent (Chapter 1)

def parse_point(text):
    """Pull the first two numbers out of the model's answer, e.g. 'click(411, 61)'."""
    nums = re.findall(r"-?\d+(?:\.\d+)?", text)
    if len(nums) < 2:
        return None                   # no point: count it as a miss, never guess
    return float(nums[0]), float(nums[1])

def copy_pixels_to_screen(a, b):
    return round(a * W / w_copy), round(b * H / h_copy)    # its own factor for each direction

def thousandths_to_screen(n, m):
    return round(n * W / 1000), round(m * H / 1000)

print(copy_pixels_to_screen(*parse_point("click(411, 61)")))   # (1199, 181)
print(thousandths_to_screen(*parse_point("(313, 83)")))         # (1202, 179)

Line by line. The regular expression takes the first two numbers in the text, with or without decimals, and ignores everything else. An answer with no numbers is a miss, never a guess: the professional-screens test in Chapter 5 counts it wrong too. Each direction gets its own factor. The two prints are the first two rows of the table above.

Steps and rounding

Picture a staircase. You can stand on a step, but never between two steps. A coordinate format is the same: it can only write certain places. The smallest change a format can write is its step. Snapping to the nearest step costs at most half a step: the rounding error.

Work the steps on our 3,840 × 2,160 screen. Thousandths step 3.84 px across and 2.16 px down, so a click is at most 1.92 and 1.08 px off. Pixels of the copy step 2.918 and 2.967 screen pixels, so at most 1.46 and 1.48 px off. Both are harmless for a 32 px button.

Two decimals are another story. They step 38.4 px across and 21.6 px down. Half a step is 19.2 px across and 10.8 down. Nineteen pixels is more than half of a 32 px button. So a model that knows exactly where Save is can still miss it, purely because of the format.

Move Save to x = 1,210 and watch it happen. The exact share is 1,210 ÷ 3,840 = 0.3151, written 0.32. Back on the screen that is 0.32 × 3,840 = 1,228.8, which rounds to 1,229: 19 px right of the centre, and off the button. Press the x = 1,210 chip in the device, then pick Two decimals.

e_max

the most a click can be off from rounding alone

W

the screen's width in pixels (use H and its own k for down)

k

how many steps the format has across the width: w̄ for copy pixels, 1,000 for thousandths, 100 for two decimals

Here are all the formats side by side, every step of the arithmetic written out.

worked exampleSave: 32 × 32 px, centre (1,200, 180), on a 3,840 × 2,160 screen; the copy is 1,316 × 728

Pixels of the copy
  down     1,200 × 1,316 ÷ 3,840 = 411.25 → 411       180 × 728 ÷ 2,160 = 60.67 → 61
  back     411 × 2.9179 = 1,199.3 → 1,199             61 × 2.9670 = 181.0 → 181
  click    (1,199, 181): 1.4 px from the centre        step 2.918 × 2.967 px, at most 1.46 × 1.48 off
Thousandths
  down     1,200 ÷ 3,840 × 1,000 = 312.5 → 313        180 ÷ 2,160 × 1,000 = 83.3 → 83
  back     313 × 3.84 = 1,201.9 → 1,202               83 × 2.16 = 179.3 → 179
  click    (1,202, 179): 2.2 px off                     step 3.84 × 2.16 px, at most 1.92 × 1.08 off
Two decimals
  down     1,200 ÷ 3,840 = 0.3125 → 0.31              180 ÷ 2,160 = 0.0833 → 0.08
  back     0.31 × 3,840 = 1,190.4 → 1,190             0.08 × 2,160 = 172.8 → 173
  click    (1,190, 173): 12.2 px off, still on Save    step 38.4 × 21.6 px, at most 19.2 × 10.8 off
  at x = 1,210: 0.3151 → 0.32 → 1,228.8 → 1,229: 19 px right, off the 32 px button
A box in thousandths
  (1,184 ÷ 3.84, 164 ÷ 2.16) → (308, 76);  (1,216 ÷ 3.84, 196 ÷ 2.16) → (317, 91)
  centre (312.5, 83.5) → (1,200, 180): 0 px off here, because two roundings partly cancel

The scale bug (the copy's numbers clicked as screen pixels)
  target x   600 → answer 206   → click 206:   394 px short
  target x 1,920 → answer 658   → click 658:   1,262 px short
  target x 3,500 → answer 1,199 → click 1,199: 2,301 px short       miss = 0.657 × x
One factor for both directions (2.918 used for y)
  y = 180 → 61 × 2.918 = 178.0: 2 px    y = 1,080 → 364 × 2.918 = 1,062.1: 18 px    y = 2,100 → 708 × 2.918 = 2,065.9: 34 px
Two pixels per point
  1,512 × 982 points → 3,024 × 1,964 px screenshot; answer (2,400, 360) → halve → (1,200, 180) points

The scale bug

Think of reading a map's centimetres as kilometres: every distance comes out wrong by the same factor. Clicking the copy's numbers on the full screen, without scaling back, is the scale bug. The harness clicks (411, 61) as if 411 were a screen pixel, and the mouse lands about a third of the way to Save.

Its fingerprint is that the miss grows with the distance from the top-left corner. A target at x = 600 is missed by 394 px. A target at 3,500 is missed by 2,301 px. A target near the origin is missed by almost nothing, which is why a quick test on a button in the top-left corner can pass.

The miss is always the same share of the position: 1 − 1,316 ÷ 3,840 = 0.657. One vendor's own troubleshooting guide lists this bug as "clicks consistently offset in one direction", and gives the fix in one line: "Scale each coordinate by the ratio of your screen size to your screenshot size".

miss_bug

how far left of the target the click lands when the harness skips the scale-back

x

the target's position across the screen, in pixels

w̄ ÷ W

the copy's width over the screen's: 0.3427 here, so the miss is 0.657 of x

Here is the check, as you would run it on a real click log. It fits a straight line to miss against position.

pythonimport numpy as np
# x of each target, and how far left of it each click landed (from your click log)
slope = np.polyfit(target_x, target_x - click_x, 1)[0]
print(round(slope, 3))   # about 0.657 on a 3,840 px screen with a 1,316 px copy: the scale bug

A miss that is a fixed share of the position, 0.657 = 1 − 1,316 ÷ 3,840, is the scale bug. A slope near 0 means the misses are not the harness's fault. Chapter 6 turns this check into a habit.

Two quieter bugs

One factor for both directions. A harness that uses 2.918 for y as well as x is wrong by a little near the top and by more further down. At y = 180 the click is 2 px off. In the middle of the screen, at y = 1,080, it is 18 px off. Near the bottom, at y = 2,100, it is 34 px off: more than a whole 32 px button. This bug hides in testing, because test targets are often near the top. The Code Lab below catches it.

Two pixels per point. Until now this lesson has treated the screen's pixels and the mouse's units as the same thing. On some laptops they are not. Think of a ruler marked in both inches and half-inches: the same length, counted on two different scales. Some laptop screens pack two physical pixels into every point, the unit the mouse and the operating system use to place things on screen. That 2 is the device pixel ratio. A laptop screen of 1,512 × 982 points gives a screenshot of 3,024 × 1,964 pixels.

If the model answers (2,400, 360) in screenshot pixels, the mouse, which moves in points, needs (1,200, 180). So the harness must halve the answer before the mouse moves, or halve the screenshot before it is sent. One vendor's documentation says exactly that for Apple's Retina screens.

An agent is booking a train ticket in a browser. This time the harness sends more than a screenshot. It also sends a short numbered list: [1] link Home, [2] field From, [3] field To, [4] button Search. The request says "search for trains". The model answers with one character: 4. No coordinates, no rounding, no scale-back. The harness looks up item 4, finds its box, and clicks the centre.

That is a different way to ground. Chapters 1 and 2 asked the model where. This chapter asks it which. It removes some problems completely, and it adds two new ones.

Where the list comes from on a web page

Think of a restaurant menu: each item is something you can point at and order. Each thing on a page you can see or click is an element: a link, a button, a text field, a picture.

A web page is written in a text language, HTML. The browser reads it into a tree of elements, the DOM (document object model). Think of an architect's plan of a building, with every room drawn to size. The browser knows the box of every element, because it drew them. Here is one button, and what the browser knows about it:

html<button id="search" class="primary">Search</button>
<!-- the browser knows its box: getBoundingClientRect() → x 2620, y 640, width 160, height 48 -->

The harness can walk this tree, keep the elements a person could click, number them, and send the list along with the screenshot. An element's number in the list is its ID, like a dish's number on the menu. The model answers with an ID, and the harness clicks the centre of that element's box.

Where the list comes from in a desktop app

Desktop apps are not web pages, but most of them keep a similar list for a good reason. Blind people use programs called screen readers, which read the screen aloud. To do that, every app must say what is on its screen. Desktop apps keep a similar list so that screen readers can read the screen aloud to blind users: the accessibility tree. It is the braille label next to each button in a lift.

Each entry in the accessibility tree has a role ("push button"), a name ("Save") and a box. Windows, macOS and Linux each offer a library that lets a program read it. One large effort to collect training examples used exactly those libraries on all three systems, exploring apps automatically (the Field Guide names it). So the same trick works on the desktop: number the entries, send the list, read back an ID.

Why a list is attractive

With a list, the model no longer has to place a click at all. Chapters 1 and 2 stop mattering. A 3-pixel icon is as easy to name as a 300-pixel one, as long as it is on the list and has a name. There is no squeeze, no unit, no rounding. The harness clicks the element's true centre, taken from the app itself.

Compare the same toolbar in three kinds of app. Look at what each one hands back.

Here is how a harness builds the web list, in JavaScript that runs inside the page. It is short.

javascriptconst items = [...document.querySelectorAll('a, button, input, select, textarea, [role="button"]')]
  .map(el => ({ el, r: el.getBoundingClientRect() }))
  .filter(({ r }) => r.width > 0 && r.height > 0);            // skip hidden elements
const list = items.map(({ el, r }, i) =>
  `[${i + 1}] ${el.tagName.toLowerCase()} "${(el.innerText || el.title || el.placeholder || '').trim()}"` +
  ` x ${Math.round(r.x)} y ${Math.round(r.y)} w ${Math.round(r.width)} h ${Math.round(r.height)}`);
// the model answers "3"; the harness clicks the centre of items[2]

Line by line. The first line collects every element a person might click: links, buttons, inputs and anything marked as a button. The filter drops elements with no size, which are hidden. Each list line carries the ID, the kind of element, its visible name and its box. The comment is the whole click step. This code is a sketch: real harnesses also check for elements covered by others.

When the list is empty

Now the drawing app in the device. Think of a painting of a control panel, not a control panel. Some apps draw everything themselves, as pixels inside one element: drawing programs, games, remote desktops. That one element is a canvas.

The page's tree then holds one entry, the canvas, and the tools painted inside it are in no list at all. An ID scheme has nothing to offer the model. This is reasoning, not a measured result: a scheme that picks from a list needs a list, and some apps cannot give one.

There is a second, quieter gap. An icon button whose accessibility name is empty is on the list, but it cannot be told apart from its neighbours. Entry 5 in the desktop list is one. If the target were that icon, the list would not help.

Marks: a list made from pixels

When the app gives no list, the harness can draw one. Everything you send the model, text and pictures, is the prompt. Think of numbered stickers on the boxes in a warehouse: a worker only has to read out a number. Draw a numbered label on every clickable region of the screenshot and let the model answer with a number. That is Set-of-Mark prompting; the labels are marks.

The method's first test was on ordinary photos, not screens. A strong chat model was asked to write the box of a described object as coordinates. It scored 25.7. With numbered marks drawn on the photo and the same question asked as "which number?", it scored 86.4: 3.4 times as high. Pointing with numbers was hard for that model; picking a number was easy.

Who draws the marks: a screen parser

Something has to find the clickable regions first. Think of a sniffer dog that finds every bag but does not know whose it is. A smaller model trained to draw a box around every clickable thing is a detector.

Then the text. Think of a fast reader who reads every sign in a street but does not know what any shop sells. Software that reads the text in a picture is OCR, for optical character recognition. A detector, OCR and a short description of each icon, put together, make a screen parser: a stocktaker who labels every item on a shelf.

One open screen parser shows the recipe. It trained its detector on 67,000 screenshots whose boxes came from web pages, for free, the way Chapter 4 describes. It wrote 7,000 icon descriptions to train a small model that describes icons. Then it merges the detector's boxes with OCR's, dropping any box that overlaps another by more than 90%.

The result on a test of ordinary screenshots: the same strong chat model went from 16.2% alone to 73.0% with the parser's marks and descriptions, 4.5 times as high. A better detector alone was worth 4.3 points of that, 68.7% to 73.0%.

Two new ways to fail

A list removes rounding and scaling, but it adds two failures of its own. Think of a party where some guests never got an invitation: they cannot come, however welcome they are. The share of real buttons the detector finds is its recall.

The target is not on the list. If the detector misses the target, the right answer is not among the marks. No model, however good, can pick a number that was never drawn.

The model picks the wrong number. On a crowded screen, marks sit close together and the model confuses them. On the parser's own test of 112 tasks, the chat model picked the right mark 62.0% of the time when the screen had more than 40 boxes. When each mark also carried a short description of what it is, that rose to 90.0%.

Marks also cover the very pixels they label. On dense screens, marks placed at the centres of their regions collide; the original method moves marks around to avoid overlaps. A crowded toolbar is the worst case for all of this.

Here is a simple way to see both failures at once. It is this lesson's own model, not a measured result. A click by mark lands only if two things both happen: the target is on the list, with probability r (the recall), and the model picks its number, with probability p. If the two are independent, success = r × p.

r

recall: the share of real buttons the detector finds and marks

p

the chance the model picks the right mark, which falls as the screen gets more crowded

At 90% recall and more than 40 marks: 0.90 × 0.620 = 55.8% of clicks land without descriptions, and 0.90 × 0.900 = 81.0% with them. Coordinates have no r at all, because nothing can be left off a list that does not exist. But their p falls with the target's size, as Chapters 1 and 5 show. IDs trade that for a p that falls with crowding.

Every number the device uses, in one place:

worked examplePicking the right mark (measured: one chat model, a 112-task test)
  fewer than 10 boxes   0.913 without descriptions   1.000 with
  10 to 40 boxes        0.692                         0.949
  more than 40 boxes    0.620                         0.900
  all tasks             0.705                         0.938
Found × picked (this lesson's simple model; recall 90% is illustrative)
  more than 40 marks:  0.90 × 0.620 = 0.558 → 55.8% of clicks land
  with descriptions:   0.90 × 0.900 = 0.810 → 81.0%
  recall 70%, >40:     0.70 × 0.620 = 0.434;  with descriptions 0.70 × 0.900 = 0.630
Ordinary screenshots (measured)
  chat model alone 16.2%  →  with parser marks and descriptions 73.0%   (4.5 times)
  better detector:  68.7% → 73.0%  (+4.3 points)

Choosing

So which should an agent use? Lists win when the app gives a full, named list, as most web pages do: exact clicks, and no worries about resolution. Marks win when there is no list but the screen is not too crowded. Coordinates are the only option that needs nothing from the app, which is why the rest of this lesson keeps working on them.

Many agents combine the three: a list when there is one, marks when a parser can find the targets, and pixels when nothing else reaches. What matters is knowing, for your own app, which of the failures above you are exposed to.

The eyedropper that was never marked

52 of 60 is a recall of 87%, a made-up number for this story. The point holds at any recall below 100%: whatever the detector misses, no model can pick.

To teach a model that "Sign in" means this spot, you need examples: a screenshot, the words, and the right place to click. You need millions of them. If a person drew each answer by hand at one every ten seconds, a million would take about 2,800 hours: well over a year of full-time work. So who labels them? Mostly, nobody. The pages label themselves.

The arithmetic: 1,000,000 × 10 seconds ÷ 3,600 = 2,777.8 hours. At 2,000 working hours a year, that is 1.39 years. The ten-second pace is made up; the point is the scale.

What one example is

Think of a flashcard: a question on the front, the answer on the back. A training example is a screenshot, an instruction and the right answer. The right answer is the example's label. For grounding, the label is a point or a box, written in one of the units of Chapter 2.

Think of practising with an answer key: try, check, adjust, try again. Showing a model millions of examples and nudging it toward each right answer is training. Every number inside the model moves a tiny bit after each example, in the direction that makes the right answer a little more likely next time. After millions of examples, the model points well at things that look like what it practised on, and no better than that.

Pages label themselves

Here is the trick that makes millions of examples cheap. Chapter 3 showed that the browser knows the box of every element on a web page. So a program can visit pages, take a screenshot, and read the right answers straight out of the browser. Think of a postman walking every street on the round, one house after another. Visiting web pages automatically, one after another, is a crawl.

The recipe has five steps:

1. Crawl a page and render it in a real browser, at a real screen size.
2. Take a screenshot: that is the picture the model will see.
3. Ask the browser for every element's box.
4. Keep the elements with visible text or a title. That text becomes the instruction.
5. Write the box's centre as the label, in the model's unit.

One early grounding model did exactly this on about 300,000 web pages. With examples from phone screens and ordinary picture questions added, its whole training mix came to about 1 million examples. Nobody drew a single label.

Work one example in full. A 1,920 × 1,080 page has a "Sign in" button. The browser reports its box: x 890, y 412, width 100, height 40. So its left edge is 890, its right edge 890 + 100 = 990, its top 412 and its bottom 412 + 40 = 452.

The centre is halfway between the edges: (890 + 990) ÷ 2 = 940 across, and (412 + 452) ÷ 2 = 432 down. As shares of the page, 940 ÷ 1,920 = 0.4896 and 432 ÷ 1,080 = 0.4000. Written with two decimals, the unit that early model used, the label is click (0.49, 0.40), and the instruction is the button's own text, "Sign in".

worked exampleA 1,920 × 1,080 page. The "Sign in" button's box from the browser: x 890, y 412, width 100, height 40
  left 890, right 990, top 412, bottom 452
  centre        (890 + 990) ÷ 2 = 940         (412 + 452) ÷ 2 = 432
  share         940 ÷ 1,920 = 0.4896          432 ÷ 1,080 = 0.4000
  two decimals  0.49                           0.40
  label         "Sign in"  →  click (0.49, 0.40)
  back on the page  0.49 × 1,920 = 940.8: 0.8 px from the true centre (at most 9.6 across, 5.4 down)
Other elements on the same page (illustrative)
  link "Pricing"   box 1,180, 28, 84 × 24   → centre (1,222, 40)  → (0.64, 0.04)
  icon "Search"    box 1,832, 24, 32 × 32   → centre (1,848, 40)  → (0.96, 0.04)   (its name comes from its title)
  field "Email"    box 760, 330, 400 × 48   → centre (960, 354)   → (0.50, 0.33)

Notice the last line of the Sign in example. The label itself is rounded. Two decimals on a 1,920-pixel page step 19.2 px, so a label can be up to half a step off: 0.005 × 1,920 = 9.6 px across and 0.005 × 1,080 = 5.4 px down. The model learns from these labels, so it can never be more precise than they are. A coarse unit in the training data puts a floor under every click the model will ever make.

Try the recipe on a whole page. Some of its elements give honest labels, and two of them do not.

Words people actually use

There is a catch in "the instruction is the button's own text". If every example asks for "Sign in" by the words printed on it, the model learns to read, not to understand. People do not talk that way. They say "the button that logs me in", or "the one under the email box".

Think of giving directions in a hotel: "the red door at the end of the hall". The words that pick out one element are a referring expression. One project writes three kinds for every element: by how it looks ("the filled button in the middle of the page"), by where it is ("the button under the email field"), and by what it does ("the button that logs you in"). Switch between them in the device above.

Think of practice questions written by a machine rather than a teacher. Examples a program makes, instead of a person, are synthetic. That project's set is 10 million elements on 1.3 million screenshots. Of those, 9 million elements on 773,000 screenshots came from its synthetic web pipeline: about 90% of everything it learned from.

Beyond the web

Desktop apps give the same free labels through the accessibility tree of Chapter 3. A larger project explored apps on Windows, macOS and Linux automatically, reading each screen's tree as it went. It added Android phones and the web: over 13 million elements on over 2.3 million screenshots. It also cut its web pages into full 1,920 × 1,080 screenshots, so the model practised on whole screens, not small crops.

The projects count different things: examples, elements, or question-answer pairs. So read this table with its units, not just its numbers.

Labels that lie

The browser knows every element's box, including boxes that do not match the picture. Some elements are hidden: they exist in the page but are not drawn. Some sit behind a pop-up, like a cookie banner. Some are empty containers with nothing in them. Keep their labels and the model learns to click on nothing, or on the banner.

So the big projects throw a lot away. The larger project above kept 1.6 million of its 3.7 million web screenshots after cleaning, and 7.7 million of 37 million elements. That is about one element in five: it threw away about four fifths of what it crawled.

Even a test set can carry bad labels. About 11.32% of the most used grounding test's labels turned out to be wrong, roughly 144 of its 1,272 instructions, and were fixed in a second version. A model scored on wrong labels is scored wrong, in either direction.

Here is the crawl as code, with the checks that keep the worst labels out. It uses Playwright, a library that drives a real browser; the page address is a placeholder.

pythonfrom playwright.sync_api import sync_playwright

examples = []
with sync_playwright() as p:
    page = p.chromium.launch().new_page(viewport={"width": 1920, "height": 1080})
    page.goto("https://example.com")
    page.screenshot(path="page.png")                       # the picture the model will see
    for el in page.query_selector_all("a, button, input, [title]"):
        box = el.bounding_box()                            # x, y, width, height, or None if hidden
        if not box or box["width"] == 0 or not el.is_visible():
            continue                                       # a label that would lie: drop it
        text = (el.inner_text() or el.get_attribute("title") or "").strip()
        if not text:
            continue                                       # nothing to ask for it by
        cx, cy = box["x"] + box["width"] / 2, box["y"] + box["height"] / 2
        examples.append({"image": "page.png", "ask": text,
                         "label": f"click ({cx / 1920:.2f}, {cy / 1080:.2f})"})

Line by line. Open a browser at the page's real size. Take the screenshot. Walk the links, buttons, inputs and anything with a title. Drop elements with no box, or that are not visible. Take the visible text, or the title, as the instruction. Write the centre as a share with two decimals.

One check is missing: this code does not catch an element covered by another one, like Subscribe under the banner. Asking the browser which element sits at the centre point does. In the page that is document.elementFromPoint(cx, cy): if it returns a different element, the label would lie.

From pointing to doing

A single click is only one step of a task. Think of a flight recorder that keeps every moment of one trip. A recording of every step of a whole task, screenshot and action at each step, is a trajectory.

One agent trained in two stages. First it learned grounding alone, from about 1.036 million single-screenshot examples. Then it learned to plan several steps, from 35,000 trajectories. Pointing came first, because planning is useless if the clicks miss. A later lesson in this series opens trajectory data up.

Training across platforms also needs a shared set of actions. A phone app is tapped and a desktop app is clicked, and a model shown both may learn two words for one thing. One project mapped every platform's actions onto three basic ones, click, type and scroll, so that "tap" and "click" stopped fighting.

What the data cannot give

Almost all of this data is web pages and phones, at ordinary sizes. Professional software on big screens is rare in it: design tools, engineering programs, science software. Chapter 5 shows the price. The early web-trained model that scored 53.4% on a test of ordinary screenshots scored 1.1% on professional screens. The pages labelled themselves, but only the pages that were crawled.

The model that clicked the corner

Two reports land on your desk about the same model. One says it finds the right button 82% of the time. The other says 19%. Nobody made a mistake. The two tests asked the same question about very different screens.

This chapter is about reading those numbers. What does a grounding test count, how does it decide a click is right, and what makes one test four times harder than another for the very same model?

What a test is

Think of a standard exam: the same questions for everyone, marked the same way, so two students' scores can be compared. A fixed set of test questions with known answers, that everyone scores on the same way, is a benchmark.

A grounding benchmark is a pile of screenshots. Each one comes with an instruction ("open the layers panel") and the target's box, drawn by a person who knows the app. The model sees the screenshot and the instruction, and answers with a click. The benchmark then checks the click against the box.

The scoring rule

Think of darts: a dart counts only if it lands inside the ring. The share of test questions where the click lands inside the target's box is the accuracy. The rule is exact: the click's x must lie between the box's left and right edges, and its y between the top and bottom edges. A click exactly on an edge counts.

Two more rules matter. A model that answers with a box is scored at its box's centre, as in Chapter 2. And an answer the scorer cannot read, with no numbers in it or numbers in the wrong form, counts as wrong. Here is the professional test's rule as code, the same logic as its evaluation script.

pythondef score(click, box):
    """The test's rule: right only if the click is inside the target's box (edges count)."""
    if click is None:
        return "wrong_format"                 # an answer the scorer could not read
    x, y = click
    left, top, right, bottom = box
    return "correct" if (left <= x <= right) and (top <= y <= bottom) else "wrong"

def box_to_click(x1, y1, x2, y2):
    """A model that answers with a box is scored at the box's centre."""
    return ((x1 + x2) / 2, (y1 + y2) / 2)

print(score((1199, 181), (1184, 164, 1216, 196)))   # correct: Save at x = 1,200
print(score((1229, 173), (1194, 164, 1226, 196)))   # wrong: Save at x = 1,210, clicked 19 px right
print(score(None, (1184, 164, 1216, 196)))          # wrong_format

Line by line. Unreadable answers are a separate kind of wrong, so you can count them apart. The box test is four comparisons, and a click on an edge counts. A box answer is turned into its centre before it is scored. The three prints are Chapter 2's copy-pixel click on Save, Chapter 2's two-decimal click when Save sits at x = 1,210 (its box runs from 1,194 to 1,226), and an answer with no numbers.

Notice what the rule does not reward. A click 1 px outside the box scores the same as a click on the far side of the screen. A near miss and a wild miss are both just "wrong". Chapter 6 shows why you should keep the distance in your own logs.

The ordinary test

The first widely used grounding test has over 600 screenshots of phones, desktops and web pages, with about 1,270 instructions. Each target is marked as one of two kinds. A target with words on it is a text target; a picture with no words, an icon target. Think of a door with its name painted on it, against a door with only a symbol.

This test is called ScreenSpot. Many of its screenshots are cropped to part of a screen; one example in its successor's paper is 960 × 540 pixels, a quarter of a laptop screen. That matters, because cropping makes every target bigger relative to the picture.

Think of how much of a photo one face fills: in a portrait, most of it; in a crowd shot, a speck. How much of the screenshot the target covers, as a share, is its relative area. On ScreenSpot the average target covers 2.01% of the screenshot. A second version of the test fixed the about 11.32% of its labels that were wrong.

The professional test

A later test set out to measure the screens professionals actually use. It has 1,581 instructions, each on its own full-screen screenshot. They come from 23 professional applications, in programming, creative tools, engineering drawing, science and office work, plus everyday system tasks. They span Windows, macOS and Linux.

Experts with at least five years of experience with each app drew the boxes. Most screens are larger than 1,920 × 1,080. Monitor scaling was turned off, so the screenshot has every physical pixel, and two-monitor setups were captured as one wide image. The most common screens are 2,560 × 1,440 (32.4% of the test) and 3,840 × 2,160 (17.0%).

This test is called ScreenSpot-Pro. Its average target covers 0.07% of the screenshot: 2.01 ÷ 0.07 = 28.7, about 29 times less than on ScreenSpot. About 62% of its targets are text, the rest icons. It tests grounding only, one instruction and one click, not planning a whole task.

From share to tokens

Why is 0.07% so brutal? Connect it to Chapter 1's tokens. Take a square target that covers a share f of a W × H screen. Its area is f × W × H pixels, so its side is the square root of that. On a 4K screen: √(0.0007 × 8,294,400) = √5,806 = 76.2 px. That is the 76 px button the hero starts at.

The copy shrinks every length by its scale k, so in the copy the side is 76.2 × k pixels, and in tokens it is 76.2 × k ÷ 28. Now a small piece of algebra removes the screen, in three steps.

Step 1: the copy's area, counted two ways. Each token is 28 × 28 = 784 pixels, so the copy's N tokens cover 784 × N pixels of the copy. That same area is also the screen's area, W × H, shrunk by k in each direction, so it is k × k × W × H. Two counts of the same area are equal: k × k × W × H is about 784 × N.

Step 2: fold k into the square root. The target's width in tokens is t = √(f × W × H) × k ÷ 28. A factor multiplied outside a square root is the same as its square multiplied inside: k = √(k × k), so t = √(f × W × H) × √(k × k) ÷ 28 = √(f × k × k × W × H) ÷ 28.

Step 3: swap in Step 1's equality, then cancel. Step 1 said k × k × W × H is about 784 × N, so put that straight in: t = √(f × 784 × N) ÷ 28. Since √784 = 28, pull it out of the square root and it cancels the 28 on the bottom, leaving t ≈ √(f × N). The screen's own size, W and H, is gone.

t

the target's width in the model's view, in tokens (for a square target)

f

the target's relative area: its share of the screenshot

N

how many tokens the model reads for the whole screenshot

The surprise is what is missing. The screen's resolution cancelled out. Under a fixed budget, a target's width in tokens depends only on its share of the screen and on the number of tokens. That is why the professional test reports targets as shares: the share is what decides how the model sees them.

Under a 1,280-token budget, N = 1,222: t = √(0.0007 × 1,222) = √0.855 = 0.92 of a token. Chapter 1 measured the same target straight across, with the copy's exact width, and got 0.93. The hundredth of difference is the copy's shape, a hair wider than the screen's; the rule averages the squeeze across and down. Under the default budget the same target is 2.72 tokens. ScreenSpot's average target, 2.01%, is √(0.0201 × 1,222) = 4.96 tokens even under the small budget.

worked exampleThe average professional target: f = 0.07% = 0.0007
  on 3,840 × 2,160:  √(0.0007 × 8,294,400) = √5,806 = 76.2 px a side
  on 2,560 × 1,440:  √(0.0007 × 3,686,400) = √2,580 = 50.8 px a side
  in tokens, any 16:9 screen:
    N = 1,222  (1,280 budget):   t = √(0.0007 × 1,222)  = √0.855 = 0.92   (0.93 measured across)
    N = 4,080  (4,096 budget):   t = √(0.0007 × 4,080)  = √2.856 = 1.69
    N = 10,549 (4K, default):    t = √(0.0007 × 10,549) = √7.384 = 2.72
The average ordinary target: f = 2.01% = 0.0201
    N = 1,222:  t = √(0.0201 × 1,222) = √24.56 = 4.96
Ratio of shares: 2.01 ÷ 0.07 = 28.7: about 29 times smaller by area, √28.7 = 5.4 times narrower
Same best model, two tests: 84.12% (ScreenSpot-v2) against 18.9% (ScreenSpot-Pro): 4.5 times lower

Now see what that does to accuracy. The device puts the target's share on a ruler where each mark is ten times smaller than the last, and converts it into tokens on a second ruler above.

Score a run yourself

Before the real results, score a run with your own hands. The Code Lab below holds a made-up test of 600 targets on a 4K screen, and a made-up model that aims at each target with a wobble of half a token. You write the exact scoring rule above, measure each target in the model's own tokens, and sort the results by size.

The results

Now the real numbers, and you know what they show. On the professional test, the best model of its time scored 18.9%. On text targets it scored 28.1%; on icons, 4.0%: seven times lower. A general chat model, not trained for grounding, scored 0.8%.

The same models had scored far higher on the ordinary test. The device puts five of them side by side, each scored on both tests. The model names are on the bars, because those are the names you will meet in papers and leaderboards.

The kind of software matters too. Here is the same best model on the professional test, by the kind of app the screenshot came from. Engineering drawing software, full of tiny tool icons on huge screens, is the hardest.

The shape behind it

The professional test's authors plotted accuracy against relative area for several models, from targets covering a fifth of the screenshot down to targets covering 0.015% of it. Their summary is one sentence: "There is a universal decrease in accuracy as the target bounding box size becomes smaller."

Every model they plotted falls the same way; only the height of each curve differs. The curve in the first device of this chapter is an illustrative stand-in for that shape. It is pinned to two measured points and makes up everything between and beyond them. Use it to feel the shape, never to read off a value.

Eighty-two percent on paper

The team and its "one time in ten" are made up. The number is in line with the measured 10.3% for engineering drawing software above, but it is not a measurement of any team.

Your agent passes every test on the team's laptops. Then a designer runs it on a 5K monitor, 5,120 × 2,880 pixels, and it misses the eyedropper tool four times in a row. You have a log of every click and every target box. Where do you start?

You now know three places a click can go wrong: the copy, the numbers, and the model. Each one leaves a different fingerprint in a click log. This chapter teaches you to read them, and then fixes the commonest one for a fraction of the cost.

First, rule out the harness

Plot every click as one dot. Across: where the target was on the screen. Up and down: how far left or right of the target the click landed. Then look at the shape before you look at anything else. There are three shapes, and each points at a different cause.

A straight line through zero. The miss is a fixed share of the position. That is the harness: it skipped or botched the scale-back of Chapter 2. On a 3,840 px screen with a 1,316 px copy, the line's slope is −0.657, the same 0.657 you met there.

A flat band around zero, about the same width everywhere. The model found the right area but could not place the click inside a small target. That is resolution, Chapter 1's problem: the target is narrower than the squares the model reads.

Far, scattered misses on top of a band. The model chose a different element altogether. Neither more pixels nor better scaling fixes that. Better examples (Chapter 4), clearer instructions, or a list to pick from (Chapter 3) might.

One vendor's troubleshooting table names the first two the same way. It lists "Clicks consistently offset in one direction" for the scaling bug, and "Clicks land in the right area but miss the target" for a small target or detail lost when a large screen was downscaled. An early grounding paper saw the band in its own model: "incorrect predictions mostly occur near the target bounding box".

Then sort the misses

Once the harness is cleared, find out which clicks miss. Think of sorting laundry before you count the socks. Sorting results into groups and scoring each group is bucketing.

Three buckets are worth keeping always. Size in tokens, √(share × tokens) from Chapter 5. Text against icon. And the kind of software. The professional test shows how far apart they sit for one model: text targets 28.1% against icons 4.0%, office apps 27.4% against engineering drawing 10.3%. An overall score of 18.9% hides all of that. The Code Lab in Chapter 5 built the size bucket on a made-up run.

The 5K screen in numbers

Back to the designer. Run the resize rule for the 5K screen. Under a 1,280-token budget, 5,120 × 2,880 becomes the same 1,316 × 728 copy as every other 16:9 screen, a scale of 0.2570. The 32 px eyedropper becomes 32 × 0.2570 = 8.2 px in the copy: 0.29 of a token.

The obvious fixes cost a lot. Raise the budget to 4,096 tokens and the icon becomes 0.53 of a token, for 4,080 tokens per screenshot. Use the default budget and it becomes 1.06 tokens, for 16,320 tokens per screenshot, at every step of every task. That is buying sharpness everywhere to see one small icon.

Zoom, then click

There is a cheaper way, and it is how you would do it yourself. Cutting a rectangle out of a screenshot is a crop, like cropping a photo. Guess, crop around the guess, look again at full size, click: that is zoom, then click. You do the same with a map: find the street on the overview, then open the street view.

Why does it work? The first look at the squeezed screen does not need to land on the icon, only near it. And misses are usually near: that is the flat band above. Now crop 1,024 × 576 pixels around the first guess. That crop fits under the 1,280-token budget whole: the resize rule turns it into 1,036 × 588 = 37 × 21 = 777 tokens, with no shrink.

So the second look sees the icon at full size: 32 px = 1.14 tokens. The two looks cost 1,222 + 777 = 1,999 tokens, against 16,320 for sending the 5K screen whole at the default budget. A full-size view of the place that matters, for about an eighth of the tokens.

pythondef zoom_then_click(screen, instruction, model, crop_w=1024):
    """Look at the squeezed screen, crop around the first guess, look again, click."""
    W, H = screen.size                                   # e.g. 3840 x 2160
    gx, gy = model.point(instruction, screen.resized())  # first guess, already scaled back to the screen
    crop_h = crop_w * H // W                             # keep the screen's shape: 576 for 1024
    left = min(max(gx - crop_w // 2, 0), W - crop_w)     # keep the crop on the screen
    top = min(max(gy - crop_h // 2, 0), H - crop_h)
    crop = screen.crop((left, top, left + crop_w, top + crop_h))
    cx, cy = model.point(instruction, crop)              # second look: the crop fits the budget whole
    return left + cx, top + cy                           # back to full-screen pixels

Line by line. The first look is the ordinary squeezed screenshot, scaled back as in Chapter 2; model.point is a placeholder for "ask, then scale back". The crop keeps the screen's shape, 1,024 by 576, and is pushed back inside the screen near the edges. The second answer is in the crop's own pixels, so the harness adds the crop's top-left corner to return to screen pixels. That last addition is the step people forget. One vendor's tool does it for you: after its zoom, every coordinate stays in full-screen pixels.

Too small, too big

The zoom can fail in two ways. If the crop is too small, the first guess may leave the target outside it, and the second look searches the wrong place. If the crop is too big, the budget squeezes it again, and the second look is no sharper than the first. Somewhere between, a crop is best. Here is the trade, built from the hero's own toy model.

Step 1. The first guess misses by a random amount in each direction, with a typical size σ₁ (the standard deviation of a bell curve).

Step 2. A crop c wide and 9c ÷ 16 tall, centred on the guess, holds the target when the miss is less than half the crop in each direction. For a bell-shaped miss, that chance comes from erf: the share of a bell curve that falls within a given distance of its middle. Below, erf always takes a distance already divided by σ × √2. Treat that √2 as part of how erf is built, the same way dividing by 28 is part of how a token is built: it is baked into the formula, not something you derive here.

Step 3. The crop is shown at a scale k: 1 if its pixels fit the budget, smaller if not. k = √(budget pixels ÷ crop pixels) when it does not fit, exactly as in Chapter 1's step 3.

Step 4. The second click wobbles by half a token of that view: 14 ÷ k screen pixels. The chance it lands on a square button of side S is the erf share in each direction, multiplied.

Step 5. The click lands only if both happen. The first chance grows with the crop. The second stays flat until the crop outgrows the budget, then falls. Their product peaks in between.

P

the chance that the final click lands

P_in

the chance the target falls inside the crop around the first guess

c

the crop's width in pixels

σ₁

how far off the first guess typically is

P₂

the chance the second click lands, looking at the crop

k

the crop's scale under the budget: 1 when it fits

In this toy, with a first guess about 150 px off and a 40 px button, a 512 px crop holds the button in 64.6% of tries, and a 1,024 px crop in 96.5%. A 2,048 px crop always holds it, but the budget squeezes it to 0.65 of its size, and only 40.6% of clicks land, against 69.2% with the 1,024 px crop. Try it.

Here is the formula worked by hand for three crops, next to the measured results. The simulation in the device also keeps each crop on the screen near the edges, which helps a little: it gives 69.2% at 1,024 px where the formula gives 67.7%.

worked exampleToy model: a 40 px button, a first guess about 150 px off, a 1,280-token budget (P_max = 1,003,520 pixels)
  crop 1,024 × 576:   1,024 × 576 = 589,824 ≤ 1,003,520  →  k = 1, shown at full size
    P_in  = erf(512 ÷ 212.1) × erf(288 ÷ 212.1) = erf(2.414) × erf(1.358) = 0.9994 × 0.9451 = 0.9445
    σ2    = 14 ÷ 1 = 14 px;   P2 = erf(40 ÷ 39.6)² = erf(1.010)² = 0.8469² = 0.7172
    P     = 0.9445 × 0.7172 = 0.677
  crop   512 × 288:   P_in = 0.9121 × 0.6629 = 0.6047;  P2 = 0.7172;  P = 0.434
  crop 2,048 × 1,152: 2,048 × 1,152 = 2,359,296 > 1,003,520  →  k = √(1,003,520 ÷ 2,359,296) = √0.4253 = 0.6522
    σ2 = 14 ÷ 0.6522 = 21.5 px;  P2 = 0.4206;  P_in ≈ 1.000;  P = 0.421
  no zoom (the first guess is the click): erf(40 ÷ 424.3)² = 0.011
Token cost of a zoom on 4K or 5K under the 1,280 budget
  first look 1,222 + crop 1,036 × 588 = 37 × 21 = 777  →  1,999 tokens
Measured on the professional test (best model of its time)
  no zoom 18.9%  →  crop 1,024 px, ground again: 40.2%  (+21.3 points, 2.1 times)
  crops 512 / 768 / 1,024 / 1,280 px: 25.1% / 34.2% / 40.2% / 40.1%
  planner-guided search: 48.1%  (+29.2 points, 2.5 times)

The numbers inside erf are the half-widths divided by σ₁ × √2: 150 × 1.414 = 212.1. For the second click the half-width is 20 px and σ₂ × √2 = 14 × 1.414 = 19.8, so 20 ÷ 19.8 = 1.010.

What the measurements say

The toy is not a model of any real system; it only shows the shape. The real test agrees with the shape. On the professional test, cropping around the first guess and grounding again took the best model of its time from 18.9% to 40.2% with a 1,024 px crop. That is 21.3 points, 2.1 times as high, with no extra training.

The crop size mattered the way the toy says. With crops of 512, 768, 1,024 and 1,280 px, that model scored 25.1%, 34.2%, 40.2% and 40.1%. A second model peaked earlier, at 768 px (28.8%), and fell to 26.3% at 1,280. The authors' explanation is the trade above, in their words: "when images are too small, crucial context is lost, whereas images that are too large exceed the model's processing capacity".

Search with a planner

A crop around one guess trusts that guess. A smarter search asks for help. Think of a friend who says "try the top-right corner". A second model that suggests where to look next is a planner.

The same paper's search method works like this. A planner model says where the target probably is, and which neighbouring elements usually sit next to it: a "new" button near a "delete" button, say. The grounding model's guesses score those candidate areas. The search crops the best area and repeats inside it, until a piece is at most 1,280 px across. Then it grounds there, and the planner checks the answer.

The result: 48.1%, two and a half times the 18.9% it started from, still without training the grounding model at all. Each part earns its keep. Without the repeated search inside crops, 41.9%. Without the planner's guesses about neighbouring elements, 46.4%. With a plain vote instead of its scoring of candidate areas, 46.8%. The price is several model calls per click instead of one.

Smaller fixes worth trying

A few cheap habits help too. One vendor's documentation says that putting the instruction text before the screenshot in the prompt improves click accuracy; that is its claim, from its own testing. Keep every coordinate in full-screen pixels after a zoom, so no crop offset is ever lost. And keep the aspect ratio when you resize, so a round icon stays round. Newer models trained on more high-resolution screens also do better; the Field Guide lists two, with dates.

The eyedropper, solved

The band width is the toy's: on 5K under the 1,280 budget, half a token is 14 ÷ 0.2570 = 54.5 screen pixels, about 50. The designer and the eyedropper are made up.

A model that clicks for you sees a smaller copy of your screen, read in squares 28 pixels wide. It answers with two numbers as text, in a unit you must know, and your harness carries them back to the screen with one factor for each direction. Under a fixed budget, a target's width in squares is about the square root of its share of the screen times the number of squares, so small buttons on big screens fall under one square and the clicks start to miss. Lists of elements avoid the arithmetic but need a list. Zooming in around a first guess rescues most of the rest.

Everything that decides whether a click lands can be written down before you ship: the copy's size and both factors, the answer's unit and its step, the targets' width in tokens, the list or no list, and the zoom. This chapter holds all of it in one place.

The numbers

The checklist

1. Measure your screen in pixels, not points; if the device pixel ratio is 2, halve the answer or the screenshot.
2. Run the resize rule for your screen and budget, and log the copy's size with every screenshot.
3. Keep both scale-back factors, across and down, and never reuse one for the other.
4. Write down the model's coordinate unit and its step on your screen.
5. If the model answers with a box, click its centre.
6. Work out your smallest targets' width in tokens, √(share × tokens); under one token, expect misses.
7. Put the instruction text before the screenshot (a vendor's documented advice).
8. If the app gives a full, named element list, consider IDs; check for unnamed entries and canvas areas.
9. If you use marks, measure the detector's recall on your own screens, and add short descriptions.
10. For small targets, zoom: crop about 1,024 px around the first guess, and map the answer back to screen pixels.
11. Evaluate on screens like yours and report accuracy by size in tokens, text against icon, and kind of software.
12. Log every click with its target box, and plot miss against position before you change anything.

The round trip as code

Here is the whole lesson's arithmetic in one small module. It keeps the screen size, the copy size and both factors together, so they can never drift apart.

pythonimport math

def smart_resize(h, w, factor=28, max_tokens=1280, min_tokens=4):
    """The resize rule (Chapter 1): pick the copy's size from the screen and the budget."""
    max_px, min_px = max_tokens * factor * factor, min_tokens * factor * factor
    hb, wb = max(factor, round(h / factor) * factor), max(factor, round(w / factor) * factor)
    if hb * wb > max_px:
        beta = math.sqrt(h * w / max_px)
        hb, wb = math.floor(h / beta / factor) * factor, math.floor(w / beta / factor) * factor
    elif hb * wb < min_px:
        beta = math.sqrt(min_px / (h * w))
        hb, wb = math.ceil(h * beta / factor) * factor, math.ceil(w * beta / factor) * factor
    return hb, wb

class Screen:
    def __init__(self, W, H, max_tokens=1280, pixels_per_point=1):
        self.W, self.H, self.ppp = W, H, pixels_per_point
        self.h, self.w = smart_resize(H, W, max_tokens=max_tokens)
        self.sx, self.sy = W / self.w, H / self.h          # two scale-back factors, never one
        self.tokens = (self.h // 28) * (self.w // 28)

    def from_copy_pixels(self, a, b):                       # answers in pixels of the copy
        return self._to_points(a * self.sx, b * self.sy)

    def from_thousandths(self, n, m):                       # answers in thousandths
        return self._to_points(n * self.W / 1000, m * self.H / 1000)

    def from_box(self, x1, y1, x2, y2, unit="copy"):        # a box: click its centre
        cx, cy = (x1 + x2) / 2, (y1 + y2) / 2
        return self.from_copy_pixels(cx, cy) if unit == "copy" else self.from_thousandths(cx, cy)

    def width_in_tokens(self, share):                       # Chapter 5: t = sqrt(share x tokens)
        return math.sqrt(share * self.tokens)

    def _to_points(self, x, y):                             # screen pixels to the points the mouse uses
        return round(x / self.ppp), round(y / self.ppp)

s = Screen(3840, 2160)
print(s.w, s.h, s.tokens)                 # 1316 728 1222
print(s.from_copy_pixels(411, 61))        # (1199, 181)
print(s.from_thousandths(313, 83))        # (1202, 179)
print(round(s.width_in_tokens(0.0007), 2))  # 0.92

Read it once more, line by line. The resize rule is Chapter 1's. The class keeps the screen size, the copy size and both factors together. Each from_ method is one unit from Chapter 2. from_box clicks the centre, which is also Chapter 5's scoring rule. width_in_tokens is Chapter 5's formula; it prints 0.92, the root rule's value, where Chapter 1 measured 0.93 straight across. _to_points divides by the device pixel ratio last, so a Retina screen is handled in one place.

One vendor's computer-use tool shows what a real action looks like on the wire. These two lines follow the shape its documentation gives:

json{ "name": "left_click", "input": { "coordinate": [512, 742] } }
{ "name": "zoom", "input": { "region": [688, 0, 1712, 576] } }

The first, the documentation's own example, asks the harness to click at x = 512, y = 742, in the pixels of the screenshot the harness last sent. The second uses our own illustrative values: a 1,024 × 576 region around Save at (1,200, 180), shifted down to stay on the screen. It asks for that region at full resolution. The documentation says that coordinates stay in full-screenshot pixels even after a zoom, and that a harness which shrinks screenshots must scale coordinates back up.

Where it sits

This lesson is in Act 2 of the path Computer Use and Multimodal Agents: how an agent perceives a screen. Just before it: LocateAnything (writing many boxes at once), Molmo and PixMo (a model trained to point), and Grounding and Spatial Reasoning. Next in the path: a full walkthrough of the professional-screens paper, ScreenSpot-Pro (coming soon), then Act 3, Video Understanding.

Later in the path, lessons are coming soon on the whole computer-use loop (Computer-Use Agents), the environments agents train in (Agent Environments, and the OSWorld benchmark), trajectory data (Trajectory Data), and a full agent trained on screens (UI-TARS-2).

Keep going

• Pixel Buffers: what a screenshot is in memory.
• Vision Transformer: the patch encoder, built from zero.
• microVLM: how a vision encoder meets a language model.
• Qwen2-VL: the dynamic-resolution model whose resize rule this lesson uses.
• microCLIP: where many vision encoders come from.
• Agents and Tool Use: the loop around a single action.

Sources

Now press Present or Teach and explain one click out loud, from the screen to the copy, the answer, the two factors and back to the mouse. Then go back to the toolbar at the top, shrink the button below one token, and switch to Zoom first: you now know every step between the screen and the click.