Molmo2: Open Video VLMs
That Point and Track

Chapter 1: The Key Insight

How do you make a language model — a thing that emits text tokens — point at a pixel? The clever move, inherited from the original Molmo, is to make the point itself be text. The model doesn't have a special "pointing head." It just learns to emit a string like x=43.2 y=61.0 and you read the coordinates back out.

Molmo2 extends this 2D image idea into the temporal dimension. A video point gains a timestamp and an object ID. Now a single text string can say: "at t=2.5s, at pixel (43, 61), this is object #2". String together many such points and you have a track — the same object across many frames. Count the distinct IDs and you have counting. One unified output format does pointing, tracking, and counting.

Three things the model produces

The whole system in one breath

Molmo2 is: sample frames from the video → a ViT encodes each frame into patch features → a connector pools and projects them into visual tokens → an LLM reads those tokens interleaved with timestamps and the question → it generates either prose or a coordinate string. We'll build each piece, and along the way meet the three training tricks (bidirectional attention, packing, token weighting) that make 16k-token video sequences trainable at all.

Chapter 2: The Point Format — Coordinates as Tokens

Let's get concrete about what the model actually writes. For the query "Point to the waterfalls," Molmo2 emits something like:

Every symbol earns its place. x, y are normalized to [0, 100] — a percentage of width and height, so they don't depend on the resolution of the crop the model happened to see. t is the timestamp in seconds (for an image set it's an image index instead). id is an integer that is unique per distinct object — this is the magic that turns a pile of points into tracks and counts.

How tracking and counting fall out for free

Look at the id column above. Rows with id=1 at t=0.5 and t=2.5 are the same waterfall at two moments — connect them and you have a track. The number of distinct ids (here, 2) is the count. Points are sorted by time, then x, then y, so the output is deterministic and easy to score.

Try it below: place objects across frames and watch the exact text string the model would have to produce. Notice how adding a frame for an existing id extends a track, while a brand-new id bumps the count.

Chapter 3: The Architecture — From Frames to Tokens

Molmo2 deliberately uses a boring architecture: a pre-trained ViT + a pre-trained LLM glued by a connector. The novelty is in the data and training, not the blocks. But to understand the training tricks you must see exactly how a video becomes tokens. Let's trace the tensor shapes.

Step 1 — Frame sampling

A 10-minute video has ~18,000 frames. You cannot feed them all. Molmo2 samples at S = 2 fps with a cap of F = 128 frames (or 384 in the long-context stage). If the video is longer than F/S seconds, it uniformly subsamples down to F frames — and always keeps the last frame, because video players freeze on the final frame, so it often matters to the user.

Step 2 — ViT encodes each frame

Each sampled frame is treated like a single image crop (downscaled to fit), run through the ViT, producing patch features. For an image, Molmo2 also tiles it into up to K overlapping crops for higher resolution (K=8 train, K=24 inference); for video frames it uses one crop each to save compute.

Step 3 — The connector pools patches into tokens

This is where the token count gets controlled. The connector takes patch features and pools spatial windows into single vectors using a tiny multi-head attention layer (the mean of the patches is the query). The pooling window size is the key dial:

• Images: 2×2 patch windows → one token each (more detail per image).
• Video frames: 3×3 patch windows → one token each — coarser, because you have many frames and must keep the total token count survivable.

Same connector weights for both — the only difference is the window size. After pooling, a shared MLP projects each pooled vector into the LLM's token space.

Step 4 — Interleave with text and feed the LLM

The visual tokens enter the LLM interleaved with text timestamps (for video) or image indices (for multi-image), plus frame-start tokens and, when available, subtitles placed as text after the frames. Explore the full pipeline below — drag the frame count and pooling window and watch the final LLM token count, the number that decides whether training fits in memory.

Chapter 4: Bidirectional Attention on Vision Tokens

Here is a subtle but high-impact choice. An LLM is causal by default: token t may only attend to tokens before it. That makes sense for generated text — you can't peek at words you haven't written. But it makes no sense for the image tokens, which are all given to the model at once, up front.

Think about it: the token for the top-left patch of frame 5 has no reason to be blind to the bottom-right patch of frame 5 — they are the same frame, presented simultaneously. Forcing causal masking on vision tokens cripples the model's spatial understanding for no benefit.

Trace the data flow: what changes in the mask

The attention mask is a N×N matrix of 0/1 (1 = allowed to attend). For a sequence of [visual tokens | text tokens]:

• Causal baseline: strictly lower-triangular — every token, visual or text, sees only the past. Visual token #50 can't see visual token #51.
• Molmo2: the visual×visual sub-block is fully filled (bidirectional). The text part stays causal. Text can see all the vision; vision never sees the text it will help generate.

Toggle the mode below and watch the mask change. The newly-lit cells in the upper-left are exactly the "let vision see vision" gains.

Chapter 5: Packing & Message Trees — the 15× trick

This is the engineering heart of the paper, and the SHOWCASE. The problem: training examples range from a few hundred tokens (a short text question) to 16,000+ (a long video with subtitles). If you batch them naively, you must pad every example up to the longest one — wasting enormous compute on padding tokens that teach the model nothing.

Two ideas fix this, and they compose:

Idea 1 — Sequence packing

Instead of one example per row, concatenate several short examples into one long sequence until you hit the max length (16,384). No padding. The catch: example A's tokens must not attend to example B's tokens — they're unrelated. So you build a block-diagonal attention mask: each example is its own island.

Idea 2 — Message trees

A single video usually has many annotations — a caption, three Q&A pairs, a pointing task. Re-encoding the expensive video tokens once per annotation would be insane. So Molmo2 encodes the video once as the root, and each annotation becomes a branch hanging off it. The branches share the video but are masked so they can't see each other (otherwise answer #2 could cheat off answer #1).

The combined attention mask — build it yourself

This is Figure 3 of the paper, made interactive. The full mask must enforce three rules at once: (1) different packed examples are isolated; (2) within one example, the shared visual tokens are seen by all branches; (3) different branches (QA pairs) of the same example can't cross-attend. Toggle each rule and watch the mask — and the wasted-padding meter — respond. Then try turning packing off to feel the pain.

Chapter 6: Token Weighting — balancing the loss

Now a problem that's pure loss-function design. Molmo2 trains on wildly different output lengths: a multiple-choice answer is 1 token ("B"); a dense video caption is 4,000+ tokens. The training loss sums over output tokens. So even if you sample captions rarely, their 4,000 tokens each drown out the single-token MCQ examples — the model gets fantastic at captioning and forgets how to answer short questions.

The fix: weight each example's loss

Molmo2 multiplies each example's loss contribution by a weight chosen to balance long vs short. The recipe:

• Video captions: fixed weight 0.1 (heavily down-weighted — they're long and dense).
• Pointing: fixed weight 0.2 (also long, dense coordinate outputs).
• Everything else: weight √(4n) / n per token — equivalently a per-example factor that scales like √(4n), where n is the number of answer tokens. (The paper writes the heuristic as √(4n).)

The √ is the important shape. A linear weight (1/n) would make every example contribute exactly equally regardless of length — but then a 1-token MCQ gets the same total say as a 100-token answer, over-weighting trivia. A square-root sits between "sum all tokens" (length-dominated) and "one vote per example" (length-blind): longer outputs still matter more, but sub-linearly.

# per-token cross-entropy, then a per-example weight
def weighted_loss(logits, targets, task, n_answer_tokens):
    # logits: [B, T, V], targets: [B, T], mask: which tokens are "answer"
    tok_loss = F.cross_entropy(
        logits.view(-1, logits.size(-1)),
        targets.view(-1), reduction='none')        # [B*T]
    n = n_answer_tokens                                  # tokens in THIS example's answer
    if   task == 'video_caption': w = 0.1
    elif task == 'pointing':      w = 0.2
    else:                          w = math.sqrt(4 * n) / n  # sub-linear: per-token weight
    return (tok_loss * w).sum()                       # total ∝ sqrt(4n), not n

Chapter 7: The Data Engine — no closed teacher

The architecture is intentionally ordinary; the data is the contribution. Molmo2 builds nine new datasets (five human-annotated, four synthetic) plus two curated from academic sources — all without ever querying a proprietary model for answers. Let's see the cleverest pipelines, because each one is a lesson in getting quality data cheaply.

Dense captions: speak, don't type

Humans type lazy captions but speak rich ones. So annotators narrate each clip aloud; Whisper transcribes; an LLM rewrites for coherence; then frame-level details from the original Molmo are merged in so nothing is missed. Result: an average of 924 words per video — versus 547 in LLaVA-Video and 280 in ShareGPT4Video. Density is the moat.

Long QA: human-and-LLM loop, not distillation

To avoid distilling, the answer isn't taken from a closed model. An LLM proposes an initial answer conditioned on a Molmo2 caption; the human annotator then accepts or iteratively refines it via dialogue. The human is the authority; the LLM is a drafting assistant. Counting questions are deliberately removed from QA — because counting should be answered by pointing, not prose.

Grounding data: convert tracks into points

For pointing, annotators find the frame where an object first appears, then click its exact location (frames at 2 fps). For curated data, existing segmentation masks become points by sampling a Gaussian around the mask center. For tracking, they repurpose Ref-VOS datasets and run SAM-2 to turn bounding boxes into masks, then mask-centers into point tracks.

Chapter 8: Results — where open catches up

Three claims to verify against the tables. First, on standard video understanding, Molmo2-8B leads the open-weights-and-data field and challenges proprietary models on counting and captioning. Second, on grounding, it doesn't just lead open models — it beats Gemini 3 Pro on video pointing (38.4 vs 20.0 F1) and tracking (56.2 vs 41.1 J&F). Third, it's honest about its weakness: long (10+ min) videos, where it lags the best closed systems for lack of open long-video training data.

Two ablations that justify the tricks

The paper's training innovations aren't free decoration — bidirectional vision attention and token weighting each "improve performance" in ablations, and packing+message-trees deliver the 15× throughput that made training the full mixture feasible at all. The architecture is ordinary because the data and training discipline carry the weight.

Compare models below across the key axes. Toggle between the general video score, counting, and the grounding metrics where the open/closed gap actually inverts.

Chapter 9: Connections & Cheat Sheet

Molmo2 sits at the confluence of three lineages: the ViT+connector+LLM design lineage (LLaVA, original Molmo), the pointing-as-text paradigm (PixMo, Molmo), and the open-data movement (OLMo, Tulu). Its move is to push pointing into time and to release everything.

How the pieces depend on each other

The cheat sheet

Implement the point parser yourself

import re
# Parse Molmo2's point string back into structured tracks
def parse_points(text):
    # each point: t= id= x= y=
    pts = re.findall(r"t=([\d.]+) id=(\d+) x=([\d.]+) y=([\d.]+)", text)
    tracks = {}
    for t, oid, x, y in pts:
        tracks.setdefault(int(oid), []).append(
            (float(t), float(x), float(y)))     # [(t,x,y), ...] per id
    count = len(tracks)                                  # distinct ids = the count
    return tracks, count
# tracks[2] = [(0.5, 78.1, 55.4), (2.5, 79.0, 54.0)]  ← object 2's path

Related lessons

• Vision-Language foundations — how ViT + connector + LLM stitch into a VLM (the chassis Molmo2 reuses).
• Grounding & spatial reasoning — the 2D pointing paradigm Molmo2 extends into time.
• SAM / segmentation — used here to convert masks ↔ points for the tracking data.
• Attention masks & packing — the LLM-training mechanics behind the 15× trick.