LocateAnything: Parallel
Box Decoding

Chapter 1: The Key Insight

Here is the move. A box's four coordinates are not a sentence to be spelled out — they are an atomic unit that should be decoded all at once, in a single parallel step.

Three ways to decode a box, side by side

Why "box-aligned" is the whole trick

Generic multi-token prediction asks the model to fill in random masked positions. For a token stream like <box> 130 647 911 832 </box> <ref> cat </ref>, a random chunk might span the </box> boundary, mixing coordinate tokens with the next object's category. The model is forced to learn "what token usually follows what," capturing spurious co-occurrence rather than geometry.

By contrast, PBD always masks exactly the coordinates inside one box. The model's job becomes well-posed: "given the visual features and the prefix, fill in the four numbers that form THIS box." The supervision matches the structure of the answer. Result: both higher throughput and higher accuracy — usually a tradeoff, here a free lunch, because the structure was being thrown away before.

Chapter 2: Background — How VLMs Localize

Before we can appreciate PBD, we need to understand how a VLM turns "where is the dog?" into a rectangle. The answer: it reuses the same next-token machinery it uses for text.

Coordinates as tokens

Continuous pixel coordinates are first normalized to a fixed range [0, 1000] (so the scheme is resolution-independent), then discretized into integer bins. Each bin becomes a special token in the vocabulary, e.g. <313>. A box is then the sequence <box> <313> <638> <345> <663> </box> — the model emits these just like words.

This is elegant: detection becomes "language modeling over a coordinate vocabulary." But it inherits language modeling's serial bottleneck.

The autoregressive factorization

Let the output be a sequence of blocks B = (b₁, b₂, ..., b_N). Conditioned on visual features Z and a text query E, every generative localizer factorizes the joint probability the same way:

Reading it symbol by symbol:

• B — the full structured output (all the boxes and labels).
• b_i — the i-th unit. In NTP this is a single token; in PBD it is a whole box.
• b_<i — everything decoded before unit i (the "history" / prefix). The model can look back at it.
• Z — visual tokens from the image encoder; E — the encoded text query. Both are the fixed "context" every step attends to.
• ∏ — the product reflects the chain rule of probability: the whole sequence's likelihood is the product of each unit's likelihood given the past.

The only thing PBD changes in this formula is the granularity of b_i. NTP makes b_i a single token (so N is large). PBD makes b_i an entire 6-token box block (so N is ~6x smaller). Fewer factors in the product = fewer sequential steps.

The architecture LocateAnything builds on

LocateAnything is a native-resolution VLM:

• Vision encoder: Moon-ViT, producing visual tokens Z = Encoder(I) at the image's native resolution (no downscaling that would blur small objects).
• Projector: an MLP bridging vision features into the language model's embedding space.
• Language decoder: Qwen2.5, which consumes Z + the query and emits the block sequence.

Native resolution matters for localization: a 12x12 pixel traffic sign needs its high-frequency detail preserved, or the predicted box will be off by tens of pixels.

Chapter 3: The Block Format

For parallel decoding to work, every step must produce a tensor of the same shape. You cannot batch and parallelize variable-length outputs cleanly. So LocateAnything reorganizes the entire output into fixed-length blocks of constant length L = 6.

Why exactly 6?

A bounding box needs four quantized coordinates plus two structural tokens that bracket it: <box> x1 y1 x2 y2 </box>. That is 6 slots. Any block type that uses fewer than 6 meaningful tokens pads the rest with a <null> token, guaranteeing uniform tensor shapes for parallel decoding.

The four functional block types

How a query becomes a block sequence

Query: "find the hot dog." The model emits: a Semantic block carrying <ref> hot dog </ref>, then a Box block <box> 313 638 345 663 </box>, then an End block. Three blocks = three decoding steps in Fast Mode, versus ~12 token steps under NTP.

Chapter 4: PBD Showcase

This is the heart of the paper. Below you can decode the same box three different ways — NTP, generic MTP, and PBD — and watch the decode steps, the wall-clock cost, and whether the output stays coherent. This reconstructs Figure 2 of the paper as an interactive race.

Reading the throughput numbers

Let T = single forward-pass latency, N = number of objects. The decode-step counts are:

That ~4.8x reduction in steps is where the paper's headline "up to 2.5x throughput" comes from after accounting for the larger per-step compute of predicting 6 tokens at once and KV-cache management.

Why the box stays coherent under PBD

Inside a PBD block, the four coordinate tokens attend to each other bidirectionally (next chapter). So when the model decides x1=313, the same step's prediction of x2 can "see" that decision and ensure x2 > x1. Generic MTP lacks this guarantee because its chunk doesn't respect the box boundary — half the chunk might belong to the next object entirely.

Chapter 5: The Attention Mask

The single cleverest engineering piece. To train both an NTP stream (for causal reasoning) and an MTP stream (for parallel box prediction) in one forward pass, LocateAnything builds a specialized attention mask with three distinct behaviors.

The three rules, decoded

1. Causal attention for NTP. The shared context (visual tokens x_vis + query x_q) and the NTP stream x_ntp use a standard causal mask: each token sees only earlier tokens. Crucially, they are blocked from attending to the MTP block stream x_blk — otherwise the NTP stream could cheat by peeking at the parallel answer (data leakage). This preserves the original VLM language ability and matches how a normal KV cache works at inference.
2. Causal flow across blocks. Within the MTP stream, block i can attend to the shared context and all previously committed blocks b_<i, but not future blocks. This lets the model learn dependencies between boxes — e.g., "I already placed a box here, don't duplicate it" — mitigating duplicate or missing detections.
3. Bidirectional intra-block attention. The tokens inside one block attend to each other fully (bidirectionally). This is what lets the four coordinates be resolved simultaneously while respecting their joint geometry (x2 > x1, y2 > y1).

Why isolate the streams?

If the NTP tokens could attend to the masked MTP block, the model would learn to copy the answer instead of reasoning toward it. At inference the MTP block doesn't exist yet (you're predicting it), so the cheat-path would create a train/test mismatch. The strict isolation guarantees the NTP behavior at training time is identical to inference time — which is exactly why the standard causal KV cache still works.

Chapter 6: Joint Training Design

Parallelizing the output directly during training risks destroying the model's inherited causal reasoning. The fix: a dual-formulation training strategy that optimizes the same ground truth in two formats at once.

One concatenated input, two views of the same target

A single sequence is built by concatenation:

where ⊕ is concatenation. Defining each piece:

• x_vis — visual tokens (shared context).
• x_q — the text query (shared context).
• x_ntp — the target written as a normal token-by-token sequence.
• x_blk — the same target re-written as fixed-length blocks.

So x_ntp and x_blk are literally the identical answer in two encodings: token-level and block-level.

How x_blk is constructed (the masking recipe)

Traverse x_ntp left to right; split and pad it according to the L=6 block rules. Then, within each block, keep the first token as a prediction anchor and replace all subsequent tokens with [mask]. The model must predict all masked tokens in the block in one step.

The objective

Both streams are supervised with cross-entropy, and the losses simply add:

• L_ntp — keeps the causal reasoning / language faculties sharp (this is the gradient that preserves the pretrained VLM behavior).
• L_mtp — teaches box-aligned parallel prediction (this is the gradient that unlocks throughput).

Because both are computed in the same forward pass over x_all (the mask keeps them isolated), training cost barely grows while the model becomes bilingual: it can decode serially (Slow Mode) or in parallel (Fast Mode) from the same weights.

Working code: the block masking core

def build_block_targets(ntp_tokens, L=6, MASK="[mask]", NULL="<null>"):
    # ntp_tokens: the flat target sequence, e.g.
    # ["<box>","<313>","<638>","<345>","<663>","</box>", ...]
    blocks = []
    for i in range(0, len(ntp_tokens), L):
        chunk = ntp_tokens[i:i+L]
        while len(chunk) < L:          # pad to constant length
            chunk.append(NULL)
        anchor = chunk[0]              # keep first token as context
        masked = [anchor] + [MASK]*(L-1)  # mask the rest
        target = chunk                  # supervise against the real tokens
        blocks.append((masked, target))
    return blocks
# L=1 collapses to plain next-token prediction.

Chapter 7: Hybrid Mode — Knowing When to Slow Down

Parallel decoding has an exploration-exploitation dilemma in hard scenes. LocateAnything names two failure patterns and builds a fallback for both.

The two failure modes

• Format Irregularity. In scenes with many instances across categories, the model hesitates at a category boundary — should it keep predicting the current class or switch? The parallel block comes out malformed, mixing structural and coordinate tokens, e.g. <box><211></ref><911><887></box>.
• Spatial Ambiguity. When objects sit in a dense regular grid (rows/columns), the parallel prediction blurs boundaries and outputs a coordinate between two objects, giving a low-IoU box.

The ambiguity trigger (exact thresholds from the paper)

During Fast Mode, the model continuously validates syntactic integrity and monitors spatial confidence. A fallback is triggered when both conditions hold simultaneously:

Reading condition (2): if the five most likely coordinate tokens are {410, 415, 418, 470, 495}, the spread is 495 - 410 = 85 > 80 — the model is torn between two spatially distant answers. Combined with low top-1 confidence, that's a red flag.

The fallback mechanism

On trigger, the compromised block is discarded, generation reverts to the last verified prefix, and that one problematic block is re-decoded with token-by-token NTP (higher precision). Once the block is finished, the model switches back to Fast Mode. Surgical, localized re-decoding — not a full restart.

Three on-demand modes

Hybrid preserves most of the speed gains while keeping outputs robust — only the rare problematic block pays the NTP tax.

Chapter 8: The Data Engine

A fast decoder is useless without precise supervision. LocateAnything pairs PBD with LocateAnything-Data: 12M unique images, 138M natural-language queries, and 785M annotated bounding boxes across six task families.

The six task families

Why detection dominates the mixture

Detection supplies 83% of the bounding boxes despite being 67% of queries — because one detection query ("find all objects") returns many boxes, while a referring query returns one. Dense box supervision is what teaches the model precise coordinate regression, so the authors deliberately weight it heavily. The other five tasks add diversity of intent: the same coordinate-prediction skill, exercised through UI, text, and language-referring lenses.

Chapter 9: Experiments

The headline: LocateAnything-3B advances the speed-accuracy frontier — higher throughput and higher localization quality, especially at strict IoU thresholds.

Earlier VLMs cluster at low throughput (~1 box/sec) with modest mean F1. LocateAnything sits to the upper-right: ~12.7 boxes/sec while improving mean F1 — the rare move that pushes both axes the good way.

The numbers that matter (from Table 1)

BPS = Boxes Per Second (decoding throughput). LocateAnything is ~2.5x the throughput of the next-fastest VLM (Rex-Omni-3B at 5.0) and ~12x faster than the dense 8B models — while topping mean F1.

Why high-IoU quality improves the most

The gains concentrate at strict IoU thresholds (e.g., F1@0.95), which demand pixel-precise boxes. This is exactly where the box-aligned, jointly-decoded coordinates pay off: predicting x1,y1,x2,y2 together with bidirectional intra-block attention keeps the rectangle internally consistent, so it snaps tighter to the object than four independently-sampled tokens would.

Breadth of evaluation

The paper evaluates across layout grounding, long-tail detection (LVIS has 1000+ rare categories), and GUI grounding — a deliberately diverse battery to show PBD isn't tuned to one benchmark. Outperforming SOTA "by a large margin" across all three is the evidence that box-aligned decoding is a general principle, not a COCO-specific trick.

Chapter 10: Connections

LocateAnything sits at the intersection of three lineages: generative detection in VLMs, multi-token / semi-autoregressive decoding, and large-scale grounding data.

What's genuinely new here

• Structure-aware parallelism. Prior MTP/diffusion-LM work is structure-agnostic (random chunks/masks). PBD's contribution is recognizing that geometric objects are the right unit, resolving both NTP's fragmentation and MTP's arbitrary chunking.
• Localized fallback. The Hybrid mode re-decodes only the offending block, not the whole sequence — a cheap reliability insurance policy.
• Dual-stream single-pass training. One attention mask trains causal-NTP and parallel-MTP simultaneously, with NTP=L1 as a clean special case.

The cheat sheet

Open questions

• Beyond boxes: can PBD extend to masks, polygons, or 3D boxes (more coordinates per atomic unit)?
• Inter-block parallelism: blocks are still semi-autoregressive across each other. Could independent objects be decoded fully in parallel?
• Adaptive block size: L=6 is fixed; could the model choose L per object (a point needs 2 coords, a 3D box needs more)?