ScreenSpot-Pro

A programmer has her code editor open on a big monitor. She asks her AI assistant: "Refresh the file explorer." The assistant takes a picture of the screen, 2,560 dots wide and 1,664 tall. Somewhere in it is a little circular arrow, 30 dots wide and 36 tall. The assistant has to say exactly where to click. Most of the time, it misses.

Finding a small thing on a big screen is hard, for the same reason a single word is hard to find on a whole newspaper page. The word is not small. The page is huge.

This lesson is about that one problem: pointing at the right spot on a big, busy screen. It follows one paper that measured how badly AI models do at it on the screens professionals really use, and found a cheap way to do much better. First, the words we need.

An assistant that uses your computer

A picture of everything on the screen at one moment is a screenshot: a photo of the screen, taken by the computer itself. A screen is a grid of tiny coloured dots, like the stones of a mosaic. Each dot is one pixel (written px). The programmer's screen is 2,560 pixels across and 1,664 down.

Now picture a friend helping you over a video call. They can see your screen, and they say: "Click the little arrow at the top of the left panel." A program that does that job by itself is a computer-use agent. It works a computer the way you do: it looks at the screen and answers with a click or a key press.

An agent repeats one small step, over and over. It takes a screenshot. It decides what to do. It acts: it moves the mouse and clicks, or it types. Then it takes a new screenshot to see what changed. This lesson is about the very first thing inside that step, before any click: finding where to click.

Pointing is a job of its own

The windows, menus and buttons you click are the graphical user interface, or GUI. Think of the dashboard of a car: everything you can press, turn or read, laid out on one panel. On a computer, the dashboard is drawn in pixels.

Before an agent can press a button, it must find it. Picture a friend at a party. You say "the woman in the green coat", and your friend points at her. Your words named something; the finger tied those words to one spot in the room. Finding the thing on the screen that a sentence names, and pointing at it, is called grounding. On a GUI it is GUI grounding.

Some models do nothing but this job. A model whose one job is grounding is a grounder. It is a pointer, not a talker: it does not chat or plan, it only says where. The sentence it gets, such as "Refresh the file explorer.", is the instruction. The button that sentence names is the target: the bullseye.

Inside the box, or wrong

How do we know whether a grounder got it right? The people who made the test drew a tight rectangle around each target, like the frame a photo app draws around one face in a group photo. They wrote it as four numbers in pixels: its left, top, right and bottom edges. That rectangle is the target's bounding box, or just its box.

The rule is blunt. A click inside the box is right. A click outside it is wrong, even by one pixel. There is no partial credit; Chapter 3 shows the exact rule, line by line. Over many test questions, the share of questions where the click lands inside the target's box is the accuracy. It is a dart score: hits out of throws.

How small is small?

Now the numbers that make this hard. The refresh button's box is [473, 183, 503, 219]: from 473 to 503 pixels across, and from 183 to 219 down. So the button is 30 pixels wide and 36 tall, and it covers 30 × 36 = 1,080 pixels.

The whole screen is 2,560 × 1,664 = 4,259,840 pixels. So the button covers 1,080 ÷ 4,259,840 of the screen: 0.0254%. Put another way, it is one part in 3,944 of the picture.

How much of the screenshot the target covers, as a share, is its relative area. A face that fills a passport photo has a large relative area. The same face in a photo of a stadium crowd has a tiny one, although the face did not shrink at all. The refresh button is a face in the crowd.

Below is that whole screen, shrunk to fit your screen. Before any model tries, you try.

On a phone the button is drawn about 4 by 5 pixels; on a desktop, about twice that. Either way it is a speck. After Look closer it is 2.5 times bigger, because the window you see is 1,024 pixels of the screen wide instead of 2,560, and 2,560 ÷ 1,024 = 2.5.

Notice what changed and what did not. The button is still 30 by 36 pixels on the programmer's screen. What changed is how much of the picture in front of you it fills. Leaning in made the target a bigger share of what you were looking at. Hold that thought: it is the whole fix, and Chapter 6 builds it properly.

Two tests: a paragraph and a page

Every field needs a fair way to compare models. A fixed set of test questions with known answers, scored the same way for every model, is a benchmark. It is a standard exam: everyone sits the same paper, and the marks mean the same thing for everyone.

The best-known exam for grounding was built from screenshots cut down to the region around the target, for example to 960 by 540 pixels. In that older test the average target covered 2.01% of the picture. On a 960 by 540 picture, 2.01% is about a 102-pixel square: easy to see.

The new test keeps the whole screen, at the size the professional really used, mostly larger than 1,920 by 1,080. Its average target covers 0.07% of the screenshot. On a 2,560 by 1,440 screen, 0.07% is about a 51-pixel square. As a share of the picture, the targets got 2.01 ÷ 0.07 = 28.7 times smaller.

The older test is called ScreenSpot; the one this paper builds is ScreenSpot-Pro. From here on we call them the ordinary test and the professional test.

Here is the picture to carry through the whole lesson. The ordinary test hands the model a paragraph cut out of a newspaper and asks it to find one word. The professional test hands it the whole newspaper page and asks for the same word. The word did not change size. The page around it grew.

The same model, two scores

So how much does that matter? The best model in the paper clicks correctly on 83.3% of the ordinary test's tasks. On the professional test, the same model clicks correctly on 18.9%. Same model, same kind of instruction: 83.3 ÷ 18.9 = 4.4 times lower. Four clicks in five now miss.

The paper then does three things. First, it builds the professional test from 1,581 real tasks (Chapters 2 to 4). Second, it asks why models fail (Chapter 5). Third, it wins much of the loss back without training any model, simply by looking closer: 40.2% with a crop around the model's own first guess, and 48.1% when a second model first says where to look (Chapters 6 and 7). The instrument at the top of this page lets you feel all three.

worked exampleThe refresh button, a real task from the test
  button = (503 − 473) × (219 − 183) = 30 × 36            = 1,080 px²
  screen = 2,560 × 1,664                                  = 4,259,840 px²
  share  = 1,080 ÷ 4,259,840                              = 0.0254%  (1 part in 3,944)

The professional test's average target
  0.07% of a 2,560 × 1,440 screen = 2,580 px²            ≈ a 51 × 51 px square

The ordinary test's average target
  2.01% of a 960 × 540 picture    = 10,420 px²           ≈ a 102 × 102 px square
  2.01 ÷ 0.07                                             = 28.7 times the share

The same model on both tests
  83.3% ÷ 18.9%                                           = 4.4 times lower

On a 390-pixel phone, the whole 2,560-pixel screen is drawn at 390 ÷ 2,560 = 0.152:
  the button becomes 30 × 0.152 by 36 × 0.152             = 4.6 × 5.5 px
  (the device's canvas is a little narrower than the phone: about 4.0 × 4.8)

Here is the road, in order. Each chapter explains one part of the instrument at the top.

1. How a model clicks: how a screenshot becomes tiles a model reads, and how its answer becomes a click.
2. Building a harder test: experts, real work screens, and how small the targets are. This is the hero's size slider.
3. One click, right or wrong: the scoring rule, and why only the centre counts. These are the hero's green and red dots.
4. Eleven models, one collapse: the results table, model by model and app by app.
5. Why small targets break grounders: shrinking, aim and clutter.
6. Look closer without a planner: three ways to crop and look again, and the best crop size. This is the hero's second button.
7. Let a planner say where to look: a search guided by words. This is the hero's third button.
8. Building a grounding stack: choosing a method by accuracy and cost, and debugging misses.
9. Map: the numbers, the checklist, the paper's limits and what came after.

Follow the screenshot from Chapter 0 on its trip. The assistant's program grabs the screen: 2,560 by 1,664 coloured dots. It sends that picture and the sentence "Refresh the file explorer." to the model. A moment later, a short line of text comes back. The program reads it and moves the mouse.

Two questions hide in that trip. What does the model actually see in that picture? And what is in that line of text? Each answer has a step where a small button can get lost, and this chapter finds both.

A picture is a grid of numbers

Each pixel stores one colour as three numbers: how much red, how much green and how much blue, each from 0 to 255. A screenshot is just a long list of those numbers, row after row. The computer does not know there is a button in it. It only has the numbers.

How many pixels a screen has across and down is its resolution. A million pixels is a megapixel. The programmer's screen has 2,560 × 1,664 = 4,259,840 pixels: about 4.3 megapixels. That is normal for the professional test. Half of its screenshots are 4.1 megapixels or larger.

If you want every byte of this, the Pixel Buffers lesson follows one image from a file on disk to the numbers a model reads.

A model that reads pictures

A model that reads a picture and text together, and answers in text, is a vision-language model. Think of a reader who can look at a photo while reading its caption. Every grounder in this lesson is one of these.

Models do not take in a whole page at once. They read in small pieces called tokens. For text, a token is about a word or part of a word. For a picture, a token is a small square of it. You read a sentence word by word; the model reads a picture square by square.

So the model never sees the picture the way you do. It sees the stones of a mosaic, one by one, each with a note of where it sits. A button that fills several stones is easy to make out. A button smaller than one stone is a smudge inside it.

Tiles of 28 pixels

How big is one stone? In the model family used here, each picture token covers a square of 28 by 28 pixels, which we call a tile. That family includes the best model in the paper, which is built on it.

Count the tiles it takes to cover the most common screen in the test, 2,560 by 1,440. Across: 2,560 ÷ 28 = 91.4, so 92 tiles, because a part tile still needs a whole token. Down: 1,440 ÷ 28 = 51.4, so 52 tiles. That is 92 × 52 = 4,784 tiles.

Now find the average target in that grid. The professional test's average target is about a 51-pixel square. 51 ÷ 28 = 1.8, so it is about 1.8 tiles across. Its 2,580 pixels fill 2,580 ÷ 784 = 3.3 tiles. Three tiles out of 4,784: that is the whole button, as the model reads it.

A limit on tiles

Every tile costs memory and time, so the family sets a limit. The most tiles a picture may use, 16,384 by default in this family, is its token cap. Think of a suitcase weight limit: pack too much, and something has to come out.

The cap allows 16,384 × 784 = 12,845,056 pixels: 12.8 megapixels. A 2,560 by 1,440 screen is 3.7 megapixels and fits easily. A 3,840 by 2,160 screen is 8.3 megapixels, 10,764 tiles, and still fits. A 5,120 by 2,880 screen is 14.7 megapixels, 18,849 tiles, and does not. That picture is shrunk before the model reads it. Try the screens below.

Shrinking the picture

Making a smaller copy of the picture before the model reads it is shrinking it; engineers say downscaling. Think of a photocopier set to half size. Everything is still on the page, but every detail is smaller, and the smallest details blur into their neighbours.

The best model in the paper reads a 2,560 by 1,440 screen at full size. Its harness keeps the family's default cap, and 4,784 tiles fit well under 16,384. So for that model, on most of the test, nothing is shrunk.

Other grounders shrink much harder. The paper's own code feeds one of them a copy 1,344 pixels wide. On a 2,560-pixel screen that is 1,344 ÷ 2,560 = 0.525 of the size on each side. The 51-pixel average target becomes about 27 pixels, less than one tile. The 30-pixel refresh icon becomes about 16. One older model reads every picture at just 448 by 448 pixels.

The answer is a line of text

Now the trip back. A vision-language model answers in text, so the model cannot move the mouse. It writes where to click. The best model in the paper writes a box, and wraps it in two special tokens, <|box_start|> and <|box_end|>. These two tokens mark where the box starts and stops, the way quotation marks surround a quote.

Inside the markers are four numbers. They are not pixels. The model writes positions in thousandths of the picture: 0 is the left or top edge, and 1,000 is the right or bottom edge. Positions written as a share of the picture like this are normalized. Think of a ruler with 1,000 marks that stretches to fit any screen.

Why not plain pixels? Because the model never sees your screen's pixels. It sees tiles, maybe of a shrunk copy. A share of the picture means the same thing on a phone, a laptop and a 5K monitor, so one answer works for any screen size.

Back to a pixel

Something has to turn that line of text into a mouse click. The ordinary program around the model does it. It takes the screenshot, sends it, reads the answer and moves the mouse, and it is called the harness. Think of the model as a navigator who can only look at photos and talk. The harness is the pair of hands on the wheel.

The harness does three things with the answer. It divides each number by 1,000, which turns thousandths into shares. It multiplies each share by the screen's width or height, which turns shares into pixels. Then it clicks the centre of the box. On this screen one thousandth is 2.56 pixels across and 1.66 pixels down. So the 30-pixel button is only about 12 thousandths wide: the model must get its numbers right to within a handful of marks.

Here are the same three steps as a small Python function. The first line inside it looks for the pattern "(number,number),(number,number)" in the answer and pulls out the four numbers. If there is no such pattern, the function returns nothing: there is no box, so there is nothing to click. The next line divides each number by 1,000. The last line takes the centre of the box and multiplies it by the screen's width and height.

pythonimport re
def answer_to_click(answer, width, height):
    m = re.search(r"\((\d+),(\d+)\),\((\d+),(\d+)\)", answer)   # the four numbers inside the markers
    if m is None:
        return None                                             # no box: nothing to click
    x1, y1, x2, y2 = (int(v) / 1000 for v in m.groups())        # thousandths -> share of the picture
    return ((x1 + x2) / 2 * width, (y1 + y2) / 2 * height)      # the centre, in screen pixels

answer_to_click("<|box_start|>(185,110),(196,132)<|box_end|>", 2560, 1664)   # (487.68, 201.34)

The paper's evaluation code does the same division for this model in one line, pred_bbox = [pos / 1000 for pos in [x1, y1, x2, y2]], and then takes the box's centre as the click. Every number below follows that recipe.

worked exampleTiles for a 2,560 × 1,440 screen (28 × 28 px each)
  across: 2,560 ÷ 28 = 91.4 → 92      down: 1,440 ÷ 28 = 51.4 → 52
  92 × 52 = 4,784 tiles   (cap: 16,384 tiles = 12,845,056 px)
  the average target, 51 × 51 px: 51 ÷ 28 = 1.8 tiles across, 2,580 ÷ 784 = 3.3 tiles

A 5,120 × 2,880 screen: 183 × 103 = 18,849 tiles > 16,384 → shrunk
  each side × √(12,845,056 ÷ 14,745,600) = × 0.933 → about 4,779 × 2,688

Shrunk to 1,344 wide (how the paper's code feeds UGround)
  2,560 → 1,344: × 0.525    51 px target → 27 px    30 × 36 icon → 16 × 19 px

One answer for the refresh task (illustrative answer, real format)
  <|box_start|>(185,110),(196,132)<|box_end|>
  ÷ 1,000: 0.185, 0.110, 0.196, 0.132
  × screen: 0.185 × 2,560 = 473.6   0.110 × 1,664 = 183.0   0.196 × 2,560 = 501.8   0.132 × 1,664 = 219.6
  centre: ((473.6 + 501.8) ÷ 2, (183.0 + 219.6) ÷ 2) = (487.7, 201.3)
  true box: x 473 to 503, y 183 to 219 → inside

A scientist who has used the plotting program Origin for at least five years is cleaning up a chart. She presses a shortcut key. A small tool she can barely notice freezes her screen, lets her drag a box around the tiny button she was about to press, and asks her to type what she meant to do. She types "disable masking", presses Enter, and goes back to work. That button is 23 by 19 pixels.

That moment, repeated 1,581 times by experts in 26 sets of programs, is the professional test. This chapter follows how it was built, and then measures how small its targets really are. The scientist is our picture of the process; the tool, the task and the button's box are real.

Experts, doing their own work

Who should write a test of professional software? People who use it every day. The people who recorded the tasks, each with at least five years in the app, are the annotators. Think of a chef writing a recipe rather than a food critic describing a meal: the chef knows what real work looks like, and which button does what.

The authors asked the experts to do their regular work, not to stage a tidy demo. That one choice shapes everything else. A real working screen has a half-finished document, a second window, a panel of tools pulled out to the side. The test keeps all of it.

To catch those screens without getting in the way, the authors built a silent capture tool. The expert opens it with a shortcut key. It lays the frozen screenshot over the screen, so the expert drags the box around the target and types the instruction right there, in place. Then the work goes on. Nothing is cropped or tidied afterwards.

Whole screens, at full size

The authors preferred screens larger than 1,920 by 1,080, because professionals use big monitors. They also switched off one operating-system setting. The setting that draws everything bigger on a high-resolution monitor is monitor scaling: reading glasses for the whole screen. At a 200% setting, for example, a 2,560-pixel screen draws every button twice as wide and fits a quarter as much. With scaling off, the screenshot shows the screen at its true density, which is exactly the hard case the test is about.

Some experts work across two monitors. The tool captured both as one wide picture. That is why the test has screenshots of 3,840 by 1,080 and 5,120 by 1,440: two ordinary screens side by side.

The most common size is 2,560 by 1,440, in 32.4% of the tasks. Next come 3,840 by 2,160 (17.0%), 3,840 by 1,080 (11.4%) and 3,456 by 2,234 (10.2%). Only 19 screenshots, 1.2%, are a plain 1,920 by 1,080. The median screenshot is 4.1 megapixels, about the size of the programmer's 4.3-megapixel screen in Chapter 0.

What the test covers

The test has 1,581 tasks, and each one has its own screenshot. They come from 23 professional apps, plus everyday tasks on three operating systems. The paper sorts the apps into six categories, like the sections of a library: development, creative, CAD (computer-aided design, the programs engineers draw parts and circuits in), scientific, office, and the operating system itself.

By operating system, 927 tasks were recorded on Windows, 604 on macOS and 50 on Linux. Both ways of counting add up to the same 1,581.

Two reviewers, one target

A test is only fair if every question has exactly one right answer. So each task was reviewed by at least two annotators. Where an instruction could fit two elements, it was rewritten until only one fits. "Click Save" on a screen with two Save buttons becomes an instruction that names which one.

The boxes were checked too. Each box is drawn tight around the part you can actually click, leaving out empty space around it. That matters for scoring: a loose box would let near misses count as hits, and the test would look easier than it is.

Text or icon

Some targets have words on them, like a "Save" button. Others are pictures only, like the refresh arrow. The test labels every target as one or the other, with a strict rule. A target is an icon target only if it shows no words. If any words label it, it is a text target, even when it also has an icon. Think of road signs: a sign with a picture of a deer against a sign that spells out the town's name.

The test has 977 text targets and 604 icon targets. 977 ÷ 1,581 = 61.8%, so about 62% of the targets are text and 38% are icons. Keep the split in mind: Chapter 4 shows that icons are far harder.

One record, line by line

Each task is stored as one record: a short list of named fields, like one row of a spreadsheet. The records are written in a format called JSON: plain text where each field is a name in quotes, a colon, then the value. Here is the record for the refresh task from Chapter 0, exactly as the released files store it.

json{"img_filename": "vscode_mac/screenshot_2024-12-03_15-15-02.png",
 "bbox": [473, 183, 503, 219],
 "instruction": "Refresh the file explorer.",
 "instruction_cn": "刷新文件资源管理器。",
 "id": "vscode_macos_0",
 "application": "vscode",
 "platform": "macos",
 "img_size": [2560, 1664],
 "ui_type": "icon",
 "group": "Dev"}

Read it top to bottom. img_filename says which screenshot the task uses; its name carries the capture date, 3 December 2024. bbox is the bounding box: left, top, right and bottom, in pixels. instruction is the English sentence the model gets. instruction_cn is the same instruction in Chinese; every task has one, translated by GPT-4 and checked by the authors.

The rest describe the task. id names it. application and platform say which app and which operating system. img_size is the screenshot's width and height. ui_type says text or icon, by the rule above. group is the category: "Dev" is development. Two of these fields are all you need to measure a target's relative area: bbox and img_size.

How small, measured

The paper reports one number for target size: the average target covers 0.07% of the screenshot. We can do more, because the records are public. Computing the share for all 1,581 released records gives a mean of 0.065%, which the paper rounds to 0.07%.

An average can hide a lot, so look at the middle too. The size that half the targets are smaller than is the median, like the middle runner in a race. The median target covers 0.036% of the screen, about half the mean. A few large targets pull the mean up; most targets are smaller still.

In pixels, the median box is 57 wide and 25 tall: a short word of text. On the most common screen, 57 × 25 = 1,425 pixels is 0.039% of 2,560 × 1,440. And 84% of all targets cover less than 0.1% of the screen.

Icons are smaller than text targets. The median icon covers 0.014% of the screen; the median text target covers 0.065%. So the icons are both harder to understand and harder to see.

Sizes this spread out do not fit on an ordinary ruler. The chart below uses a ruler where each mark is ten times the one before, so tiny and large sizes fit on one chart. That ruler is a log scale: it counts 0.001, 0.01, 0.1, 1 and 10 in equal steps.

Look at where the ordinary test's average sits: 2.01%, far to the right of almost every bar. 99.2% of the professional test's targets are smaller than 0.5% of the screen. A target as large as the ordinary test's average almost never occurs here.

You can reproduce the headline numbers yourself from the released files. The short program below reads all 26 annotation files, computes each target's share of its screen, sorts the shares and prints the count, the mean and the median.

pythonimport json, glob
rows = [r for f in glob.glob("annotations/*.json") for r in json.load(open(f))]   # 26 files, 1,581 records
share = [(r["bbox"][2] - r["bbox"][0]) * (r["bbox"][3] - r["bbox"][1])
         / (r["img_size"][0] * r["img_size"][1]) for r in rows]
share.sort()
print(len(rows), sum(share) / len(share), share[len(share) // 2])   # 1581, about 0.00065, about 0.000355

The program prints shares as fractions, not percents: 0.00065 is 0.065%, and 0.000355 is 0.0355%.

worked exampleOne record
  bbox [473, 183, 503, 219] → 30 × 36 px = 1,080 px²
  img_size [2560, 1664]     → 4,259,840 px²
  share = 0.0254%  (below the median, 0.0355%)

The "disable masking" icon in Origin
  [998, 2078, 1021, 2097] → 23 × 19 px = 437 px²

The whole test
  341 + 299 + 261 + 254 + 230 + 196 = 1,581 tasks
  927 Windows + 604 macOS + 50 Linux = 1,581
  977 text + 604 icon: 977 ÷ 1,581 = 61.8% text

The median box on the most common screen
  57 × 25 = 1,425 px² ÷ (2,560 × 1,440 = 3,686,400) = 0.039%

How small, over all 1,581 targets
  mean 0.065% (the paper says 0.07%)   median 0.0355%
  under 0.01%: 16.4%   under 0.05%: 58.8%   under 0.1%: 83.6%   under 0.5%: 99.2%

The model has answered. Its box sits mostly on the refresh button, but its right half hangs over the next icon. Is that right? Half right? The test has to decide, the same way, for 1,581 answers from each of eleven models.

A scoring rule sounds like a dull detail. It is not. The rule decides what "18.9%" means, and a scorer you write yourself must match it exactly, or your numbers will not compare with anyone else's. This chapter takes the test's rule apart, one line at a time.

The answer key

The box the experts drew is the ground truth: the answer key. Every model is checked against it, and nothing else.

The test asks one yes-or-no question per task: is the model's point inside the true box? We call it the point-in-box rule. Think of darts. A dart counts where its tip lands, not where it was aimed, and not how close it came. There is no overlap score and no partial credit. Each answer is right or wrong.

A box counts as its centre

Some models answer with a point. Others, like the best model in the paper, answer with a box. The test turns a box into one point: its centre. That matches what happens on a real computer. The harness of Chapter 1 clicks the centre of the box, and the mouse lands on one pixel, not on a rectangle.

Two consequences follow, and both surprise people. A box that covers half the button, but whose centre is off the button, is wrong. A box that is far too big, but centred on the button, is right. Only the centre matters, so only the centre is checked.

Shares, not pixels

Before it checks anything, the scoring code divides the true box by the screen's width and height. So both the box and the point become shares of the picture, from 0 to 1. The model's point already arrives as a share: the code divided the thousandths by 1,000.

For the refresh task on its 2,560 by 1,664 screen, the true box becomes: left 473 ÷ 2,560 = 0.18477; top 183 ÷ 1,664 = 0.10998; right 503 ÷ 2,560 = 0.19648; bottom 219 ÷ 1,664 = 0.13161. The model's centre from Chapter 1 is (0.1905, 0.1210). It lies between 0.18477 and 0.19648 across, and between 0.10998 and 0.13161 down. Correct.

The edge counts

What if a point falls exactly on the edge of the box? The code compares with "less than or equal to" on both sides, so a point on the edge counts as inside. It almost never happens with real answers. But if you write your own scorer with a strict "less than", a handful of results will flip, and your numbers will not match the paper's.

No answer is still an answer

Sometimes a model writes a sentence instead of a box: "I think it is the arrow near the top." There is no point to check. An answer the harness cannot read as a box or a point is marked wrong_format, and it counts as wrong. Think of an exam answer written in the wrong place on the sheet: the examiner cannot mark it right.

The important part is that a wrong_format answer stays in the total. Accuracy is still correct answers divided by all 1,581 tasks. A model that refuses, rambles or breaks its own format is penalised, not excused. Try each case below.

Same input, same answer

A language model writes its answer one token at a time. At each step it has a list of possible next tokens, each with a chance. It could pick one at random, weighted by those chances, and then the same question could get a different answer each time. Or it could always pick the top one.

The test uses the second way. The model always picks its single most likely next token, so the same input always gives the same answer. This is greedy decoding, like always tapping the top suggestion on your phone keyboard. It makes every score in the paper repeatable: run the test again and you get the same number.

Counting every task once

The paper reports three accuracies for each model: on text targets, on icon targets, and overall. The overall number counts every task once and divides by all 1,581. That is a micro-average. The other way would be to average the text rate and the icon rate as equals, which is a macro-average. Think of grading a class by every answer, against averaging the marks of two groups.

The two ways give different numbers when the groups differ in size. The best model gets 28.1% of the 977 text tasks right: about 274.5. It gets 4.0% of the 604 icon tasks: about 24.2. Together that is 298.7 out of 1,581 = 18.89%, reported as 18.9%. A plain mean of the two rates, (28.1 + 4.0) ÷ 2, would say 16.05%. Icons are fewer, so they weigh less in the micro-average.

The evaluation code, line by line

Here is the heart of the test's evaluation script. It is short enough to read whole.

python# from the test's evaluation script, eval_screenspot_pro.py
bbox = [bbox[0] / img_size[0], bbox[1] / img_size[1], bbox[2] / img_size[0], bbox[3] / img_size[1]]
click_point = response["point"]  # may be none
if click_point is None:
    return "wrong_format"
# Check if the predicted point falls in the ground truth box
if (bbox[0] <= click_point[0] <= bbox[2]) and (bbox[1] <= click_point[1] <= bbox[3]):
    return "correct"
else:
    return "wrong"

The first line turns the true box into shares of the screen: left and right are divided by the width, top and bottom by the height. The second line takes the point that the model's adapter produced, already as shares; for a model that answers with a box, the adapter has already taken the box's centre. If the adapter found no point, the task returns wrong_format. The long condition is the point-in-box rule, with "less than or equal to" on both sides of both axes. Anything else is wrong. At the end, the script divides the number correct by the number of tasks, every task counted once.

worked exampleThe refresh task, one answer (illustrative answer)
  answer  <|box_start|>(185,110),(196,132)<|box_end|>
  ÷ 1,000 → (0.185, 0.110), (0.196, 0.132)
  centre  → x = (0.185 + 0.196) ÷ 2 = 0.1905     y = (0.110 + 0.132) ÷ 2 = 0.1210
  true box ÷ screen: 473 ÷ 2,560 = 0.18477   183 ÷ 1,664 = 0.10998
                     503 ÷ 2,560 = 0.19648   219 ÷ 1,664 = 0.13161
  0.18477 ≤ 0.1905 ≤ 0.19648 and 0.10998 ≤ 0.1210 ≤ 0.13161 → correct

A box hanging off to the right: (185,110),(230,132)
  centre x = (0.185 + 0.230) ÷ 2 = 0.2075 > 0.19648 → wrong, though its left part covers the button

Accuracy, counted task by task (micro-average)
  text: 28.1% × 977 = 274.5      icon: 4.0% × 604 = 24.2
  (274.5 + 24.2) ÷ 1,581 = 18.89% → reported 18.9%
  a plain mean of the two rates: (28.1 + 4.0) ÷ 2 = 16.05%

You must choose a grounder for an agent that will work in a CAD program next month. You have a shortlist, and one number for each from the ordinary test: 83%, 77%, 76%, 54%. The paper ran the same models on the professional test. Before you look: how many of them do you think land more than half of their clicks?

Hold your guess. This chapter reads the paper's results the way you would read them to make that choice: first the collapse, model by model, then where the best model fails, app by app.

Eleven models, three kinds

The paper tested eleven models of three kinds. Some are grounders, built for exactly this job. Some are general vision-language models, which read pictures and answer questions about them. One is a general chat model.

Model names carry a size. The numbers a model learned in training are its parameters, like the knobs on a huge mixing desk. "7B" in a name means 7 billion of them. Bigger is not always better here, as the table will show.

The best model in the paper is called OS-Atlas-7B. It is the Qwen2-VL model of Chapter 1, trained a little more to point. Training an existing model a little more on a narrower job is fine-tuning, like a pianist learning one new piece: the skill was there, the piece is new.

At the other end sits GPT-4o. A model trained to chat about anything, not to point, is a general-purpose model. Think of a well-read guide who has never held a laser pointer. It can describe a screen in detail, and still not say where to click.

Same model, two tests

A public table of models' scores on a benchmark is a leaderboard, like a league table. The authors keep one for both tests, so we can put each model's two scores side by side. Make your guess first.

None of them lands even a fifth of its clicks. The best model falls from 83.3% to 18.9%. SeeClick falls from 53.9% to 1.1%, 49 times lower. ShowUI-2B, a small model that scores 77.3% on the ordinary test, more than three of the 7B models, ends at 7.7%.

The whole table

Now the expert view: the paper's own results table for all eleven models. "Overall" is the micro-average of Chapter 3: every task counted once. "Text" and "Icon" are the two kinds of target from Chapter 2. "Kind" is this lesson's plain grouping, not a column of the paper.

Read it from the top. Only three models pass 10%: OS-Atlas-7B at 18.9%, UGround-7B at 16.5% and Aria-UI at 11.3%. Everyone else is below one click in ten.

Look at Qwen2-VL-7B, 1.6%. It is the model OS-Atlas-7B was fine-tuned from. Same starting point, 17.3 points apart: knowing how to read a picture is not the same skill as knowing where to click in it.

Now the promise from earlier: bigger is not always better. CogAgent-18B, the largest model in the table, scores 7.7%, the same as ShowUI-2B, a model nine times smaller. Inside one family, size does help: OS-Atlas-4B scores 3.7% and OS-Atlas-7B 18.9%. So size matters less than what a model was trained to do. A small model trained to point can match a large one that was not.

The general vision-language models sit near the bottom: MiniCPM-V at 3.0%, Qwen2-VL-7B at 1.6% and the older QwenVL-7B at 0.1%. They can read a screenshot and describe it. On a professional screen they almost never put the click inside the box.

GPT-4o scores 0.8%: about 13 correct answers out of 1,581. Remember that number. In Chapter 7, this same model drives the best search in the paper, without ever having to point.

Now compare the last two columns. For every model, icons are harder than text. The best model finds 28.1% of text targets and 4.0% of icons: 28.1 ÷ 4.0 = 7.0 times fewer. Three models score 0.0% on icons.

Where the best model fails

Split the best model's score by category and the spread is wide. Office leads with 27.4%, then Scientific 24.4%, Creative 17.9%, Development 17.7% and the operating system 16.8%. CAD is last, at 10.3%. The lowest single cell is not in CAD, though: it is Development icons, at 1.4%.

By single app, the range is wider still. At the bottom are AutoCAD at 2.9%, about 1 of its 34 tasks, Inventor at 2.9% (2 of 70), Illustrator at 3.2% (1 of 31) and SolidWorks at 3.9% (3 of 77). At the top are EViews at 54.0% (27 of 50), Word at 44.0% (37 of 84) and VMware Fusion at 34.1% (14 of 41). Three of the four worst apps are CAD programs.

Read single-app numbers with care, because the counts are small. AutoCAD has only 34 tasks. One more correct answer would move it from 2.9% to 2 ÷ 34 = 5.9%. A category pools hundreds of tasks, so its score is steadier: CAD's 10.3% rests on 261 tasks. When you compare two apps, compare their counts too.

One more split: language. Every task also has a Chinese instruction. With those, the best model drops 2.1 points, from 18.9% to 16.8%. UGround drops 8.8 points, from 16.5% to 7.7%. So a grounder's score also depends on the language it is spoken to in, and by different amounts for different models. If your agent's users write in another language, measure in that language.

worked exampleSame model, two tests (ordinary from the authors' leaderboard)
  OS-Atlas-7B   83.3 → 18.9   ÷ 4.4      UGround-7B  76.3 → 16.5   ÷ 4.6
  ShowUI-2B     77.3 →  7.7   ÷ 10.0     SeeClick-7B 53.9 →  1.1   ÷ 49
  CogAgent-18B  52.8 →  7.7   ÷ 6.9      Qwen2-VL-7B 39.8 →  1.6   ÷ 24.9
  GPT-4o        20.1 →  0.8   ÷ 25.1

Text against icon, best model
  28.1 ÷ 4.0 = 7.0 times

Correct answers, from rates × task counts
  GPT-4o: 0.008 × 1,581 ≈ 13      EViews: 0.540 × 50 = 27      Word: 0.440 × 84 ≈ 37
  AutoCAD: 0.029 × 34 ≈ 1         SolidWorks: 0.039 × 77 ≈ 3

The 26 app cells, weighted by their task counts, give 18.91% overall
  (the tables agree with each other)

Give the same model the same instruction twice. First, a small cut-out of the file explorer, 600 pixels wide: it clicks the refresh arrow. Then the whole 2,560-pixel screen, with the code, the terminal and a second window open: it clicks a folder icon three rows down. Nothing about the button changed. Everything around it did.

That scene is made up, but the pattern is what the paper's tables show. This chapter asks why. The answer is not one thing. Three causes stack on top of each other, and each one needs a different fix.

Three reasons, in the paper's words

The paper names three reasons, and they are worth reading slowly. First, professional apps are complex, so they need high resolution, and high resolution goes beyond what today's models handle well. Second, higher resolution makes each target a smaller share of the screen, and models do worse on smaller shares. Third, professionals keep extra documents and tools open, and that crowds the screen.

In short: the picture is too big, the target is too small a part of it, and the rest of the picture is busy. We take them one at a time.

Reason 1: the picture gets shrunk

Chapter 1 showed the first one. A model that reads a copy 1,344 pixels wide turns a 2,560-pixel screen's 51-pixel average target into about 27 pixels. The 30 by 36 refresh icon becomes about 16 by 19. A model that reads every picture at 448 by 448 is worse still: 448 ÷ 2,560 = 0.175, so the 30-pixel icon becomes about 5 pixels wide. That is a smudge inside one tile.

But shrinking is not the whole story, and this is the key fact of the chapter. The best model in the paper reads a 2,560 by 1,440 screen at full size: its 4,784 tiles fit well under its 16,384 cap. Every pixel of the button reaches it. And it still scores only 18.9%. Something else is going on.

Reason 2: aim and share

Throw darts at a board. Even a good player does not hit the same spot twice; the darts land in a little cloud around the aim point. How far a model's clicks scatter around the spot it means is its spread, like a dart player's grouping. A tight grouping hits small targets; a loose one does not.

The paper shows the effect of target size directly, on the ordinary test. For each of four grounders it checked, accuracy falls as the target's share of the picture falls, all the way from 20% of the picture down to 0.015%. The paper prints the curves' shape but not their values, so we use the shape only.

Why that shape? Suppose a model's aim wobbles by a fixed share of the picture it looks at: say 2% of its side, as in the toy at the top of this page. On a small picture, 2% is a few pixels. On a huge screen, 2% is dozens of pixels. The button stays 30 pixels. So the bigger the picture, the more of the cloud falls outside the button. We can work out exactly how much.

The chance of a hit, derived

Start with one direction, left and right. The model means to click the target's centre, but each click lands a little off, by an error e. Most errors are small and a few are big. A scatter that is dense in the middle and thins out evenly on both sides is a bell curve, like a pile of sand poured from one spot. Its width is the spread, σ: about two thirds of clicks land within σ of where the model meant.

The click is inside a target w pixels wide when the error is less than half the width either way: from −w/2 to +w/2. So the chance of a hit along x is the share of the bell curve between those two limits. That share has a standard name: erf, the error function. It turns "how far out" into "what share of the pile":

erf

The share of a bell curve within a given distance of its middle, measured in units where the spread is √2. erf(0) = 0; erf(1) = 0.84; erf(2) = 0.995.

w

The target's width in the pixels of the picture the model reads (after any shrinking).

σ

The spread of the clicks along one axis, in the same pixels: about two thirds of clicks land within σ of where the model meant.

Where does 2√2 come from? The 2 is because we go out half the width, w/2, on each side. The √2 comes from how erf is defined: it measures distance in units of √2 σ. Neither changes the idea. The wider the target compared with the spread, the closer the share gets to 1.

Now add the up-and-down direction. Left-right and up-down errors are independent: a click that is off to the left is no more likely to be off upward. When two things are independent, the chance that both happen is the product of their chances. The click must be inside on both axes, so:

P_hit

The chance that one click lands inside the box. Left-right and up-down errors are independent here, so the two shares multiply.

h

The target's height in the same pixels.

Look at what sits inside each erf: only w/σ and h/σ. That is the whole story in two ratios. Double the picture, with the button and the spread both twice as large in pixels, and nothing changes. What hurts is a target that shrinks compared with the spread. And if the spread grows with the picture, as we supposed, then w/σ and h/σ fall exactly as the target's share of the picture falls. Accuracy follows the share: the shape the paper saw.

Try it. The cloud below is 200 clicks with the same spread. Shrink the button and count the green dots, then shrink the whole picture to 1,344 pixels wide.

With a 12-pixel spread, the average target catches 93% of clicks at full size and 54% after shrinking. The refresh icon catches 68% and then 28%. The ordinary test's average target catches every click. The numbers below are worked by hand.

erf(51 ÷ 33.94)² = 0.966² = 0.93 (the average target, full size, σ = 12)

worked exampleSpread σ = 12 pixels in the model's picture (illustrative); 2√2 × 12 = 33.94
  average target at full size, 51 × 51:   erf(51 ÷ 33.94)²   = erf(1.503)²             = 0.966² = 0.93
  shrunk to 1,344 wide (× 0.525), 26.8:   erf(26.8 ÷ 33.94)² = erf(0.790)²             = 0.736² = 0.54
  refresh icon at full size, 30 × 36:     erf(0.884) × erf(1.061) = 0.789 × 0.866            = 0.68
  refresh icon shrunk, 15.75 × 18.9:      erf(0.464) × erf(0.557) = 0.488 × 0.569            = 0.28
  ordinary-test average, 102 × 102:       erf(3.005)²                                        = 1.00

Shrinking for other models
  448 ÷ 2,560 = 0.175:  the 30-pixel icon becomes 30 × 0.175 = 5.3 pixels wide
  1,344 ÷ 3,840 = 0.35: on a 3,840-pixel screen, everything shrinks to 0.35 of its size

Reason 3: clutter

The third reason is the busy screen. Professionals keep extra documents and tools open: a second window, a reference PDF, a pulled-out panel of tools. Two monitors become one wide picture. Every one of those adds more things that look like buttons.

Another element that looks like the target, such as a second refresh arrow in another window, is a lookalike: a twin in a crowd. The more lookalikes, the more ways a guess can go wrong, even with a steady aim. Chapter 7 shows a real case from the paper: a simple crop-and-look-again method was pulled to a file tab in a code editor window behind the one that mattered.

Why icons are hardest

Icons get hit by everything at once. They are smaller: the median icon covers 0.014% of the screen, against 0.065% for the median text target. And they are harder to understand. The paper gives three reasons.

1. Too many functions to label. Professional apps have so many functions that they cannot all get words. The paper points at the toolbar of Origin, the plotting program from Chapter 2, as an example.
2. Familiarity is assumed. These apps expect their users to know the icons and buttons already, so they do not explain them on screen.
3. Meanings belong to one profession. An icon in a CAD program or a sound editor carries a meaning that rarely appears on the web pages most models learn from.

So a text target gives the model words to match against the instruction. An icon target gives it a tiny picture with a specialist meaning. That is why the best model finds 28.1% of text targets and only 4.0% of icons.

You cannot read a word on a newspaper page held at arm's length. So you glance, guess where the word is, and lean in. The paper's simplest fixes do exactly that, with no new training: the same model, asked twice.

Chapter 5 ended on one rule: accuracy follows the share. If the target is too small a part of the picture, make the picture smaller around the target. This chapter builds three ways to do that, measures each, and finds that the simplest one wins.

Shrink the search, not the picture

The part of the screen the model must look through is the search area: the page you scan for a word. On the professional test, the search area starts as the whole screen. Every method in this chapter makes it smaller.

The tool for that is the oldest one in photo editing. Cutting a rectangle out of the screenshot is a crop, and the rectangle is the window. A crop is not a shrink. Shrinking keeps the whole screen and throws away detail. Cropping keeps every pixel of the window and throws away the rest of the screen. The button stays 30 pixels wide, but now it is a far bigger share of what the model sees.

Each time the harness sends a picture to the grounder and waits for an answer is one grounder call: one question to the expert. Calls cost time and money, so we count them for every method. All three methods here use the same grounder, OS-Atlas-7B, exactly as it is. Only the harness changes.

Iterative Zooming: quarters

The first method plays twenty questions with the screen. Ground on the whole screen to get a first guess. Split the screen into four quarters, and keep the quarter that holds the guess. Ground again inside that quarter, split it into four, keep one, and repeat. This is Iterative Zooming.

The paper runs three rounds. On a 2,560 by 1,440 screen the patch goes from 2,560 by 1,440 to 1,280 by 720, to 640 by 360, to 320 by 180: a sixty-fourth of the screen. That is 1 call on the whole screen plus 1 call per round: 4 grounder calls. It lifts the best model from 18.9% to 31.0%.

Iterative Narrowing: halves around the guess

The second method fixes something odd about quarters. The quarters are fixed in place, so a target that sits on the line between two quarters gets cut in two. Instead, cut a window half as wide and half as tall, centred on the guess, and repeat. This is Iterative Narrowing: closing in on a word by halving the page around your finger.

After three rounds the window is also 320 by 180 pixels, with the same 4 grounder calls. It scores 31.9%, a little better than Zooming's 31.0%. The paper's explanation is exactly the centring: its windows follow the guess, so a target near the screen's middle lines is not split.

ReGround: glance, then lean in

The third method is the simplest. Guess once on the whole screen. Crop a fixed window around the guess, say 1,024 by 1,024 pixels. Ground again inside it, and click there. That is ReGround: glance, then lean in. It takes just 2 grounder calls.

And it is the best of the three, by far: 40.2%, from 18.9% looking once. On text targets it reaches 55.7%; on icons, 15.1%. It helps in every category: Development 37.5%, Creative 38.1%, CAD 33.3%, Scientific 37.8%, Office 59.1% and the operating system 37.8%. Step through all three methods below.

Why one big crop can beat three small steps

The paper reports the scores; the reasons below are ours, and the device shows them. Every round of quartering or halving is another decision that can go wrong. And a target that one round cuts away can never come back: every later round only looks inside the last patch.

The last patch is also small: 320 by 180 pixels, with little around the target. ReGround makes only one decision, and it keeps a 1,024-pixel window, about 28% of the screen. So a slightly wrong first guess still leaves the target inside the window, with its surroundings. In the device, a first guess 665 pixels off kills both iterative methods at round 1, and ReGround still recovers.

How big a window?

ReGround has one knob: the window's size. The paper tried four sizes with two grounders. With OS-Atlas-7B: 512 pixels gives 25.1%, 768 gives 34.2%, 1,024 gives 40.2%, and 1,280 gives 40.1%. With UGround-7B: 27.0%, 28.8%, 28.2% and 26.3%.

Each grounder has a peak. The crop size where accuracy peaks is the sweet spot, like the distance at which a page is easiest to read. For OS-Atlas-7B it is 1,024 pixels; for UGround it is 768. The paper gives the reason in one sentence: "when images are too small, crucial context is lost, whereas images that are too large exceed the model's processing capacity."

As a share of a 2,560 by 1,440 screen, the four windows cover 7.1%, 16.0%, 28.4% and 44.4%. Pick one below and see where it sits.

The trade-off, derived

Why should there be a peak at all? Think of ReGround's success as two things that must both go right. First, the window must hold the target. Second, the look inside the window must hit it. For one task:

P_ReGround

The chance one task ends with a click inside the true box.

P(window holds)

Rises as the window grows: a wrong first guess still leaves the target inside.

P(second look hits)

Falls as the window grows: the target becomes a smaller share again, and a very large window exceeds what the model handles well.

This product is our way of thinking about it, not an equation from the paper. The first factor rises with the window: a bigger window forgives a worse first guess. The second factor falls: in a bigger window the target is a smaller share again, which is Chapter 5's problem coming back. Multiply a rising curve by a falling one and you get a hill. The top of the hill is the sweet spot.

Where the top sits depends on the grounder. A grounder with a wild first guess needs a bigger window to catch the target. A grounder that needs more surroundings to recognise things also wants a bigger window. A grounder that struggles with large pictures wants a smaller one. That is why OS-Atlas-7B and UGround peak at different sizes, and why you must sweep the size for your own grounder.

worked exampleIterative Zooming on 2,560 × 1,440, three rounds of 2 × 2
  2,560 × 1,440 → 1,280 × 720 (1/4) → 640 × 360 (1/16) → 320 × 180 (1/64)
  grounder calls: 1 + 3 = 4                                  score: 31.0%

Iterative Narrowing: half width, half height, centred on the guess
  same sizes, 320 × 180 after three rounds; 4 calls          score: 31.9%

ReGround with a 1,024 × 1,024 window
  window share: 1,048,576 ÷ 3,686,400 = 28.4% of the screen
  the average target, 2,580 px², becomes 2,580 ÷ 1,048,576 = 0.246% of what the model sees:
  3.5 times its share of the whole screen
  grounder calls: 2                                          score: 18.9% → 40.2%  (+21.3 points)

The device's wrong first guess, (1,220, 760), against the target's centre, (1,700, 300)
  √((1,700 − 1,220)² + (300 − 760)²) = √(230,400 + 211,600) = 665 px
  ReGround's window: x 708 to 1,732, y 248 to 1,272 → still holds x 1,680 to 1,720, y 280 to 320

Crop size sweep (ReGround)
  OS-Atlas-7B: 512 → 25.1   768 → 34.2   1,024 → 40.2   1,280 → 40.1
  UGround-7B:  512 → 27.0   768 → 28.8   1,024 → 28.2   1,280 → 26.3
  OS-Atlas-7B loses 15.1 points going from 1,024 down to 512

You ask a colleague who knows the program: "Where is Delete?" She does not point at the screen. She says: "In the File Explorer window, in the top action bar, next to New and Copy." Now you know where to look, though she never touched the mouse.

That is the idea behind the paper's best method. Chapter 6 had one weakness: when the first guess lands in the wrong place entirely, no crop around it can help. A colleague who names the right drawer fixes exactly that. This chapter builds the search step by step, then measures what each step is worth.

Screens are nested

A GUI is built like a set of drawers. Buttons sit inside toolbars, toolbars inside panels, panels inside windows. Someone who knows the app can name the right drawer in words, even if they could not point at the button inside it.

A second model that does not point, but says in words where to look, is a planner. In the paper the planner is GPT-4o: the very model that pointed correctly only 0.8% of the time in Chapter 4. It is bad at pointing and good at knowing where things usually are. The search uses only the second skill.

Step 1: where might it be?

The planner gets the screenshot and the instruction, and writes a short answer with tags in it. It wraps the target in <element> tags, the regions to search in <area> tags, and nearby buttons in <neighbor> tags. Think of labels on a map: the house, the street, and the shop next door. The paper calls this step position inference.

It lists the areas most likely first. If the target is not on the screen at all, it may write "No target". Here is an excerpt of the real prompt, and the planner's real first answer for the task "delete file or folder":

planner prompt and answerI want to identify a UI element that best matches my instruction. Please help me determine which
region(s) of the screenshot to focus on and list the UI elements that might appear next to the target.
If the target does not exist in the screenshot, please output "No target".
...
3. Use the following XML tags to describe items in the screenshot:
- <element>: Wrap a specific UI element.
- <area>: Describe an area of the UI containing multiple elements.
- <neighbor>: Describe a UI element that may appear around the target.

The <element>delete button</element> is most likely to be found in the <area>File Explorer window</area>,
next to the <neighbor>New folder button</neighbor>.

Read the answer tag by tag. The element, "delete button", is the target in the planner's own words. The area, "File Explorer window", is the drawer to open first. The neighbour, "New folder button", is a landmark: something easy to find that usually sits near the target. The prompt also tells the planner to make every reference specific, clear and unique. Vague phrases such as "Other icons" or "window" are not allowed, because they would match half the screen.

Step 2: words into boxes

Words cannot be cropped. So the grounder, OS-Atlas-7B again, turns every tagged phrase into a box on the screen. Pointing at "File Explorer window" is far easier than pointing at a 50-pixel icon: the window is huge. Each box is a place the search might look next.

Every region the search might look inside next is a candidate area: a drawer you might open. A box around a small element is too tight to search inside, so the search grows it first. Growing a small box outward so it becomes a region worth searching is dilation, like drawing a wider circle around a pin on a map.

Step 3: score the candidates

Now there are several candidate areas. Which one to open first? The paper lets the boxes vote. Every box the grounder found casts a vote for each candidate area, and the votes add up to the area's score. Think of darts on a board: a dart near the bullseye scores more than one near the rim, and a dart off the board scores nothing.

To weigh a vote, first find where it falls inside the area, as a share from 0 to 1 on each side. This is the paper's Equation 2:

x, y

The centre of one voting box: a box the grounder found for one of the planner's phrases.

x₁, y₁, x₂, y₂

The left, top, right and bottom edges of the candidate area being scored.

x′, y′

Where the vote falls inside the area, as a share from 0 to 1 on each side. The area's centre is (0.5, 0.5).

Then the weight. It is a bell centred on the area's middle: 1 at the centre, falling toward the edges, and 0 outside. A setting decides how fast a vote loses weight away from the centre; it is called sigma (σ), and the paper fixes it at 0.3 in every experiment. This is the paper's Equation 1:

s

The vote's weight for this area: 1 at the centre, falling toward the edges, 0 outside. An area's score is the sum of s over every voting box.

σ

How fast the weight falls away from the centre. The paper uses 0.3 in every experiment. (This σ is a share of the area, not the click spread of Chapter 5.)

centre (0.5, 0.5): s = 1.000
x′ = 0.25, y′ = 0.5: s = exp(−0.0625 / 0.18) = 0.707
corner (0, 0): s = exp(−0.5 / 0.18) = 0.062 (2σ² = 2 × 0.3² = 0.18)

Why a bell, and not a plain count of the boxes inside each area? The paper gives two reasons. It works like human visual attention, which is strongest at the middle of where you look. And it stops big areas from winning just by being big. A huge area would hold every vote, but its votes would mostly sit far from its centre, so each would count little. Drag a vote below, then compare two areas.

Here are the two areas worked by hand, in a made-up layout. Area A is the top action bar, a thin strip. Area B is the whole file list, a big pane that contains A. Five boxes vote: the delete button, the New button, the Copy button, the action bar itself, and the file list. A scores 3.418. B holds every vote, yet scores only 2.115, because four of its five votes sit near its top edge. So A is searched first. Widen the bell to σ = 0.6 and the scores become 3.83 and 3.90: a wide bell lets the big area win, which is the bias the paper wanted to avoid.

worked exampleTwo candidate areas on one level of a search (positions illustrative, σ = 0.3)
  A: the top action bar   x 600 to 1,400, y 80 to 140
  B: the file list        x 0 to 1,500,   y 60 to 1,300
Five votes: the centres of the five boxes the grounder found
  delete button (1,040, 108)   New button (760, 112)   Copy button (880, 104)
  top action bar (1,000, 110)  file list (750, 680)

Area A                     x′      y′      s
  delete button          0.550   0.467   0.980
  New button             0.200   0.533   0.603
  Copy button            0.350   0.400   0.835
  top action bar         0.500   0.500   1.000
  file list              0.188  10.000   0     (outside A)
  score                                  3.418

Area B
  delete button          0.693   0.039   0.249
  New button             0.507   0.042   0.312
  Copy button            0.587   0.035   0.289
  top action bar         0.667   0.040   0.265
  file list              0.500   0.500   1.000
  score                                  2.115

A is searched first: B holds every vote, but they sit near its top edge, so each counts little.
With σ = 0.6 the scores become 3.83 and 3.90: a wide bell lets the big area win.

Step 4: drop duplicates

Dilation often makes candidates that overlap a lot: two boxes around nearly the same toolbar. Searching both would waste calls. When two candidates overlap a lot, the search keeps only the higher-scoring one. This is non-maximum suppression: two people claim the same seat, and the one with the ticket stays.

Step 5: search inside, again and again

Now the search crops the best candidate and runs the same steps inside it: ask the planner, box the phrases, score, drop duplicates, crop again. Searching inside a candidate with the same steps, again and again, is recursion, like a set of Russian dolls. How many times it has gone inside is the depth.

When a patch is 1,280 pixels or smaller, the search stops nesting. The grounder clicks directly inside it. Then the planner does one last job: it looks at the click, drawn inside a red box, and says whether it is the target. We call it the check: a second pair of eyes. Its answer is a small JSON record:

json{"result": "is_target" | "target_elsewhere" | "target_not_found",
 "new_instruction": "..." or null}

On anything but is_target, the search moves on to the next candidate, and the planner may rewrite the instruction for it. The search stops when the planner says the target is found, or at a depth limit that the paper does not publish. The whole method, steps 1 to 5, is the paper's planner search, called ScreenSeekeR. Follow one real search below.

In the paper's own example, ReGround fails this task: its first look grounds a file tab in a code editor window in the background, and its crop never reaches the explorer. The planner search instead opens the File Explorer window, then the top action bar, and finds the button.

What it scores

ScreenSeekeR lifts the best model to 48.1%, with no additional training. From 18.9%, that is 29.2 points more, and 48.1 ÷ 18.9 = 2.5 times the accuracy: a 154% gain. On text targets it reaches 64.1%; on icons, 22.4%. By category: Development 49.8%, Creative 41.9%, CAD 37.9%, Scientific 47.2%, Office 64.3% and the operating system 52.0%.

The striking part is the planner. GPT-4o scores 0.8% when it must point by itself, and it powers the whole search. The paper's lesson: a model that understands screenshots well can help grounding a great deal, even if it was never trained to ground.

What each part buys

How do you know which part of a method matters? Take one part out and measure what is lost. That is an ablation, like pulling one ingredient from a recipe to taste the difference. The paper ran three.

1. No recursive search. Stop after the planner's first answer, without searching inside: 41.9%, a loss of 6.2 points. Nesting is the biggest part.
2. No neighbours. The planner names no landmarks: 46.4%, a loss of 1.7 points.
3. Majority vote. Count votes plainly instead of the bell-shaped score: 46.8%, a loss of 1.3 points.

The algorithm, in order

Now you can read the paper's Algorithm 1. One warning first: the printed listing has two lines in the wrong order. It scores the candidates before it removes the duplicates it scores. The version below puts the steps in the order the text describes: infer, ground, dilate, drop duplicates, score, sort, recurse. D_MAX stands for the depth limit, whose value the paper does not publish.

pythonS_MIN = 1280                                       # call the grounder directly at or below this size (px)
def seek(image, instruction, depth=0):
    if depth >= D_MAX or max(image.width, image.height) <= S_MIN:
        click = grounder_click(image, instruction)            # one grounder call
        verdict = planner_check(image, click, instruction)    # is_target / target_elsewhere / target_not_found
        return click if verdict["result"] == "is_target" else None
    phrases = planner_where(image, instruction)               # <element>, <area>, <neighbor> phrases
    boxes = [grounder_box(image, p) for p in phrases]         # one grounder call per phrase
    candidates = nms(dilate(boxes))                           # grow small boxes, drop overlapping ones
    candidates.sort(key=lambda c: -sum(vote(center(b), c) for b in boxes))   # Equations 1 and 2
    for c in candidates:                                      # most promising area first
        click = seek(image.crop(c), instruction, depth + 1)
        if click is not None:
            return to_parent(click, c)                        # add the crop's offset back
    return None

Line by line: the first branch is the bottom of the dolls. When the patch is small enough, or the search is deep enough, the grounder clicks and the planner checks. Otherwise the planner names phrases, the grounder boxes each one, the boxes are grown and de-duplicated, and the candidates are sorted by their summed votes. The loop then searches inside each candidate in turn, most promising first. A found click is passed back up, with each crop's offset added, until it is in screen pixels.

Your agent starts work in a CAD program next month. On your own screenshots it picks the right toolbar icon about 1 time in 20. You have a week. Where do you start?

That situation is made up, but it is close to the paper's CAD icon cell, 4.7%. This chapter turns the paper into a plan: measure first, pick a method by its accuracy and its cost, tune it, and know what the paper did not test.

Measure on your own screens first

Do not start with a fix. Start with a test that looks like your agent's job. Record tasks on your own screens in the test's record format from Chapter 2: the screenshot, the box, the instruction, the screen size, text or icon. Compute each target's relative area. Score with the point-in-box rule from Chapter 3.

Then split the results before you read them. Sorting results into groups, by size, by text or icon, by app, and scoring each group is bucketing, like sorting laundry before counting socks. One overall number hides where the misses live. Buckets show it: often the misses are almost all small icons, or almost all one app.

pythonfrom collections import defaultdict
def bucket(records, results):
    # records: tasks in the test's JSON format; results: "correct", "wrong" or "wrong_format" for each
    groups = defaultdict(lambda: [0, 0])                        # group name -> [correct, total]
    for r, res in zip(records, results):
        w, h = r["img_size"]; x1, y1, x2, y2 = r["bbox"]
        share = (x2 - x1) * (y2 - y1) / (w * h) * 100          # % of the screen
        size = "under 0.01%" if share < 0.01 else "under 0.1%" if share < 0.1 else "0.1% or more"
        for key in ("all", "size " + size, r["ui_type"], r["application"]):
            groups[key][0] += res == "correct"                   # wrong_format counts as a miss
            groups[key][1] += 1
    for key, (ok, n) in sorted(groups.items()):
        print(f"{key:24s} {ok:5d} / {n:5d} = {ok / n:6.1%}")

Every task lands in four buckets at once: all tasks, its size band, text or icon, and its app. A wrong_format answer counts as a miss in each.

How many tasks do you need? Count what one task is worth. In a bucket of 120 icon tasks, each task is 1 ÷ 120 = 0.8 points of accuracy. A change that wins two more tasks moves the score 1.7 points, which is easy to mistake for noise. Record enough tasks in the buckets you care about before you trust a difference between two methods.

The method ladder, with its cost

The paper gives you a ladder of methods, all with the same grounder. Each rung is more accurate and costs more. How many times a method asks a model per click is its cost in model calls: how many phone calls one question takes.

1. Look once: 1 grounder call, 18.9%.
2. ReGround: 2 grounder calls, 40.2%.
3. Iterative Zooming and Narrowing: 4 grounder calls each, 31.0% and 31.9%: more calls than ReGround, for less.
4. ScreenSeekeR: one planner call and one grounder call per phrase at every level, then a final click and a check: 48.1%.

One rung is never worth taking. Iterative Zooming and Narrowing cost twice as many calls as ReGround and score 8.3 to 9.2 points less. On this evidence there is no budget at which they are the right choice. That leaves a short ladder: look once, ReGround, then the planner search.

Cost also multiplies, because an agent clicks many times per task. Take a made-up task of 20 clicks. Looking once costs 20 grounder calls. ReGround costs 40. A planner search at about 14 calls per click costs about 280 model calls, many of them to a large planner. Whether that is worth 7.9 more points per click depends on what a wrong click costs in your app.

The paper reports no call counts or times for any method. The counts above come from each method's steps, so they are estimates. For ScreenSeekeR the count depends on how many phrases the planner writes and how deep the search goes. Set those yourself below.

What the gains are made of

Split the gains by kind of target. ReGround adds 27.6 points on text (28.1% to 55.7%) and 11.1 on icons (4.0% to 15.1%). ScreenSeekeR adds 36.0 on text (to 64.1%) and 18.4 on icons (to 22.4%). Icons stay hard: even the best method finds fewer than one in four. By category, the search reaches 64.3% in Office and 37.9% in CAD.

Now look at where most of the gain comes from. ScreenSeekeR gains 29.2 points over looking once. ReGround already gets 21.3 of them, about 73%, with just two grounder calls and no planner. The planner adds the last 7.9 points. In the paper's own example, that is the case it fixes: a first guess that landed in the wrong window entirely, where no crop can help.

And tune the window for your grounder. The sweet spot moved from 1,024 pixels for OS-Atlas-7B to 768 for UGround-7B, and the wrong size cost the best model up to 15.1 points. Sweep 512, 768, 1,024 and 1,280 on your own buckets before you ship.

worked exampleAccuracy and calls, all with the same grounder
  look once            1 call     18.9%
  ReGround             2 calls    40.2%   +21.3
  Iterative Zooming    4 calls    31.0%   +12.1
  Iterative Narrowing  4 calls    31.9%   +13.0
  ScreenSeekeR         many       48.1%   +29.2   (ReGround's share of that gain: 21.3 ÷ 29.2 = 73%)

ScreenSeekeR's calls, structural estimate (illustrative)
  per level: 1 planner call + 1 grounder call per phrase; at the end: 1 grounder click + 1 check
  3 levels × (1 + 3 phrases) + 2 = 14 calls    (the Chapter 7 example: 3 + 9 + 1 + 1 = 14)

Gains by target kind
  ReGround:     text 28.1 → 55.7 (+27.6)   icon 4.0 → 15.1 (+11.1)
  ScreenSeekeR: text 28.1 → 64.1 (+36.0)   icon 4.0 → 22.4 (+18.4)

What the paper did not measure

A good engineer reads the limits as carefully as the results. This paper tests grounding only: finding one element for one instruction. It does not test whole tasks that plan and act over many steps, and the authors say why: to avoid legal risk from the licences of the commercial apps.

It reports no call counts and no times for any method, so cost is yours to measure. It does not publish the search's depth limit. Every multi-round method uses one grounder, OS-Atlas-7B, and one planner, GPT-4o, so we do not know how the methods rank with other models. And its printed Algorithm 1 has two lines in the wrong order, which Chapter 7 corrected.

What came after

The test became a standard. Newer grounders listed on the authors' public leaderboard score far higher on the same 1,581 tasks; many of those entries are reported by their own authors. Two of the listed names mention zooming in and cascades, the idea this paper tested, though the leaderboard does not say how they work. The Map lists them with their numbers.

You can now read the ScreenSpot-Pro paper and explain every number in it: why the targets are 0.07% of the screen, how a click becomes right or wrong, why the best model falls from 83.3% to 18.9%, and how looking closer wins back more than half of the gap. This chapter locks it in.

The paper in one paragraph

ScreenSpot-Pro is a test of GUI grounding on the screens professionals really use: 1,581 tasks recorded by experts in 23 professional apps and on 3 operating systems, on whole screens mostly above 1,920 by 1,080. Its targets cover 0.07% of the screen on average, 28.7 times less than the older test's. Eleven models were tested; the best scored 18.9%, against 83.3% on the older test, and icons were 7 times harder than text. Looking closer recovers much of the loss without training: one crop around the first guess lifts the best model to 40.2% with two calls, and a planner that names areas in words lifts it to 48.1%.

The numbers that matter

The checklist for a grounding stack

1. Measure the relative area of targets on your own screens; compare it with 0.07% and 2.01%.
2. Score with the point-in-box rule: the centre of a box, edges inside, unreadable answers wrong and kept in the total.
3. Bucket by size, by text or icon, and by app before you change anything.
4. Log the picture size and tile count the model actually receives; know your token cap and any resize.
5. Convert thousandths to pixels with the real width and height, and click the centre.
6. Decode greedily, so scores repeat.
7. Try ReGround first; sweep 512, 768, 1,024 and 1,280 pixel windows for your grounder.
8. For each miss, check whether the window held the target at all.
9. Add a planner search where icons and clutter dominate; log its phrases and cap its depth.
10. Report model calls per click next to accuracy.

The core, as code

Four small functions carry the whole paper. to_screen turns a box in thousandths of a crop into a click on the screen: it adds the crop's corner back, the step every cropping method needs. correct is the test's point-in-box rule from Chapter 3. reground is Chapter 6's best method: one look, one window kept on the screen, one more look. vote is Equations 1 and 2 from Chapter 7.

pythonimport math

def to_screen(box, crop):                      # a box in thousandths of a crop -> a click in screen pixels
    x0, y0, w, h = crop
    x1, y1, x2, y2 = (v / 1000 for v in box)
    return (x0 + (x1 + x2) / 2 * w, y0 + (y1 + y2) / 2 * h)

def correct(click, true_box, screen_w, screen_h):          # the test's point-in-box rule
    if click is None:
        return False                                       # wrong_format still counts against you
    x, y = click[0] / screen_w, click[1] / screen_h
    l, t, r, b = true_box[0] / screen_w, true_box[1] / screen_h, true_box[2] / screen_w, true_box[3] / screen_h
    return l <= x <= r and t <= y <= b

def reground(screen, instruction, L=1024):                 # two grounder calls
    fx, fy = to_screen(grounder(screen, instruction), (0, 0, screen.w, screen.h))
    x0 = min(max(fx - L / 2, 0), screen.w - L)
    y0 = min(max(fy - L / 2, 0), screen.h - L)
    window = (x0, y0, L, L)
    return to_screen(grounder(screen.crop(window), instruction), window)

def vote(point, area, sigma=0.3):                          # Equations 1 and 2 of the paper
    (x, y), (x1, y1, x2, y2) = point, area
    xp, yp = (x - x1) / (x2 - x1), (y - y1) / (y2 - y1)
    if not (0 <= xp <= 1 and 0 <= yp <= 1):
        return 0.0
    return math.exp(-((xp - 0.5) ** 2 + (yp - 0.5) ** 2) / (2 * sigma ** 2))

Where it sits in the path

This paper closes Act 2 of the Computer Use and Multimodal Agents path, on perception and GUI grounding. Before it comes LocateAnything, a model that writes a whole box in one step, and the GUI Grounding lesson, coming soon. After it, Act 3 begins with Video Understanding. Three lessons of this series build on this one and are coming soon: Computer-Use Agents, OSWorld, and UI-TARS-2.

Where it sits in the field

• Seeing pictures as tokens. Qwen2-VL is the model family whose 28-pixel tiles and flexible picture sizes Chapter 1 uses. Pixel Buffers follows every step from an image file to the numbers a model reads.
• Pointing as text. Molmo and PixMo is a model that answers by pointing at pixels. Grounding and Spatial Reasoning is the deep-dive on how models tie words to regions.
• The search ideas it builds on. V* guided visual search (arXiv 2312.14135), Iterative Narrowing (Nguyen, 2024, arXiv 2411.13591), and Divide, Conquer and Combine (arXiv 2408.15556).
• The models it tests. SeeClick, UGround, OS-Atlas, ShowUI, Aria-UI, CogAgent, Qwen-VL, Qwen2-VL, MiniCPM-V and GPT-4o.

What came after

The test became a standard yardstick. These are the scores of later models as listed on the official leaderboard, last updated on 10 September 2026; many entries are self-reported. Qwen2.5-VL-7B 26.8%, UI-TARS-72B 38.1%, GTA1-7B 55.5%, UI-TARS-1.5 61.6%, Holo1.5-72B 63.3%, GTA1-32B 63.6%, Holo2-235B-A22B (agentic) 78.5%, Indeed-UI-32B-zoomin 82.7%, and GroundCascade 85.1%, the top entry. Two general agents are also listed: "Claude (Computer Use)" at 17.1% and "GPT5-minimal (resized)" at 18.5%.

These are the leaderboard's numbers, not the paper's, and several were reported by their own authors. Two of the top names mention zooming in and cascades, the idea this paper tested; the leaderboard does not say how they work.

The paper

"ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use", by Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang and Tat-Seng Chua (National University of Singapore, East China Normal University, Hong Kong Baptist University). It is arXiv 2504.07981, April 2025; the PDF notes a first upload on 3 January 2025. The authors' leaderboard cites a 2025 workshop version. The data is released under the MIT licence; the screenshots were captured in November and December 2024, and nine annotators are credited by name.

The limits, in the authors' words and ours

• Grounding only. The test leaves out "agent planning and execution tasks like those in OSWorld", because of licensing risk with the commercial apps.
• No cost numbers. The paper reports no call counts or times for any method.
• An unpublished setting. The search's depth limit is not given.
• One grounder, one planner. Every multi-round method uses OS-Atlas-7B, and the search uses GPT-4o.
• One test. The zoom methods were measured only on the professional test.

Keep going

• Qwen2-VL: the model family under the best grounder, and its tiles.
• Molmo and PixMo: a model that points by writing coordinates.
• Pixel Buffers: from raw bytes to the tokens a model reads.
• LocateAnything: writing a whole box in one step.
• Grounding and Spatial Reasoning: the deep-dive on words and regions.
• Video Understanding: the start of Act 3.

Now press Present or Teach and explain, out loud and from memory, why a crop around the first guess more than doubles the accuracy, and why too small a crop hurts. If you can, you own this paper. Then go back to the button, shrink it to the professional test's average, and let the planner search.