Molmo & PixMo:
Open VLMs Without Distillation

Chapter 1: The Key Insight

If detailed captions are impossible to type, change the modality. Molmo's central data trick is almost embarrassingly simple: don't ask people to type — ask them to talk.

Two beautiful side effects fall out of this choice:

• An anti-cheating receipt. The annotator's audio recording is itself proof that a human looked at the image and described it — you cannot fake a 75-second monologue by pasting GPT-4V text. The audio is a built-in authenticity certificate.
• Length and detail for free. PixMo-Cap captions average 196 words. Compare: COCO captions average 11 words, Localized Narratives 37. That is a 5–18× jump in descriptive density, and it came from changing one verb (type → speak).

The second pillar: pointing

The other half of the insight attacks a different weakness. VLMs are bad at grounding — connecting a word to a specific place in the image — and at counting. Molmo's fix is to collect data where annotators don't describe, they point: click directly on the pixels that answer the question.

The whole paper in one breath

Chapter 2: Background — How a VLM Is Built

Before the tricks, let's nail down the standard VLM template Molmo uses, because everything downstream is a tweak to it. A modern VLM has four parts, and the magic is that three of them already exist as pretrained components — you are mostly learning how to glue them.

The data flow, with shapes

This is the part people skip — and it's the part that makes the model concrete. Let's trace one image of size 336×336 through the encoder (ViT-L/14, so 14-pixel patches):

So the image becomes a sequence of "visual word" tokens that get prepended to the text tokens. The LLM doesn't know or care that some tokens came from pixels — it just attends over the whole sequence.

What Molmo keeps standard (on purpose)

The authors deliberately resist exotic architecture. They use OpenAI's CLIP ViT-L/14 336px as the encoder and swap LLMs across scales: OLMoE-1B-7B (a mixture-of-experts, their most efficient), OLMo-7B, Qwen2-7B (→ Molmo-7B-D), and Qwen2-72B (their best). Same data and recipe for all — only learning rates differ. The message: hold the architecture fixed, vary the data, and watch the data do the work.

Chapter 3: Overlapping Multi-crop

Here is a real headache. Most ViTs only accept square images at a fixed, fairly low resolution — say 336×336. But the interesting tasks (reading a street sign, parsing a chart) need high resolution. If you shrink a 1500×1000 photo down to 336×336, the text becomes an unreadable blur. So how do you feed a high-res image into a low-res encoder?

The standard answer (used by many VLMs): tile the image into multiple square crops. Each crop is encoded independently by the ViT at full 336px, and you also include the whole image resized to 336 as a low-resolution "overview." More crops = more effective resolution.

Molmo's fix: let crops overlap

Allow adjacent crops to share a margin of pixels. Now a patch near a seam appears inside a crop where its neighbors are also visible, so the ViT can encode it with context. The bike's brand name is now fully inside at least one crop. This is the difference shown in the paper's Figure 3, and it is the simulation below.

Chapter 4: The Connector (SHOWCASE)

The connector is the bridge between the visual world and the language world, and Molmo makes three specific choices that the ablations show genuinely matter. Let's build it piece by piece, then drive it interactively.

Choice 1: which ViT layers?

You might assume you take the ViT's final layer output as the image features. Molmo instead concatenates features from two layers — the third-to-last and the tenth-from-last. Why? Different depths capture different things: later layers hold abstract semantics, earlier layers hold finer spatial/texture detail. Concatenating gives the LLM both. It's a small, free win (the ablation calls it "slightly better").

Choice 2: pool 2×2 windows by attention, not concatenation

576 patches per crop × many crops would blow the token budget. So we pool. The naive option: take each 2×2 block of 4 patches and just stack them into one fat vector (concatenation). Molmo instead pools each 2×2 window with a small multi-headed attention layer, where the query is the mean of the four patches and the keys/values are the four patches themselves.

Choice 3: project with an MLP

Finally a small MLP maps the pooled vectors into the LLM's embedding dimension. Now each pooled window is a single "visual token" the LLM reads alongside words.

Arranging the tokens

Order matters because the LLM is a sequence model. Molmo lays out: low-res overview tokens first, then high-res crop tokens in row-major (left-to-right, top-to-bottom) order. Special tokens mark the start/end of the low- and high-res blocks, and row-end tokens mark row transitions — so the LLM can recover 2D structure from a 1D sequence.

Chapter 5: PixMo-Cap — Talking Captions

Now the data engine in detail, because this is the paper's true contribution. PixMo-Cap is the pre-training dataset: 712k images, each turned into a 200-word caption — with no VLM in the loop.

The collection pipeline, step by step

The numbers that prove it worked

And the killer ablation (Table 3b): a Molmo trained on PixMo-Cap's human transcripts (cap-F1 54.1) matches one trained on GPT-4o captions of the same images (cap-F1 52.9). Human speech, cleaned by a text LLM, is as good as distilling GPT-4o — without the distillation.

Chapter 6: Pointing & Counting (SHOWCASE)

PixMo-Points is Molmo's most novel idea. Instead of describing where something is in words ("the bus is on the left"), the model emits 2D coordinates — it points at the pixels. This single capability unlocks grounding, counting, and visual explanation.

The output format

Molmo predicts points as plain-text coordinates normalized to [0, 100] — not pixel values, not special tokens. A point to Mt. Rainier looks literally like text the LLM generates:

Normalizing to 0–100 means the coordinates are resolution-independent — the same "63.5" means "63.5% across" whether the image is 400px or 4000px wide. The ablation (Table 4d) found plain-text coordinates beat dedicated special tokens (89.4 vs 85.8): the LLM is already great at generating numbers as text, so don't fight it.

Chapter 7: The Training Recipe

Molmo's training is deliberately simple — and a couple of details quietly save a lot of compute. There are just two stages.

Stage 1: Pre-training on captions (all params)

Train all parameters (ViT, connector, LLM) on PixMo-Cap to generate either the clean caption or one of the raw transcripts. The prompt specifies the style, and 90% of the time includes a length hint ("write ~200 words") — which the ablation (Table 2e) shows is a better pre-training task than plain captioning, because it teaches the model to control output length.

Stage 2: Fine-tuning on the mix

Fine-tune on a blend of all PixMo datasets plus standard academic datasets (VQA v2.0, TextVQA, ChartQA, DocVQA, AI2D, …). Datasets are sampled proportional to the square root of their size (so giant synthetic sets don't drown small ones), with pointing data up-weighted because "pointing tasks learn more slowly than QA tasks."

The multi-annotation trick (a real efficiency win)

Many images have several annotations (VQA v2.0 has multiple Q&A pairs per image). Encoding the image once per annotation wastes the expensive vision forward pass. Molmo packs all annotations for an image into one long sequence and uses an attention mask so each annotation's tokens attend to the image tokens and to themselves — but not to other annotations.

They train for 4 epochs with AdamW, cosine LR decaying to 10% of peak, with gradient clipping applied separately to LM, ViT, and connector (because their LR scales differ wildly).

Chapter 8: The Results

Two yardsticks: academic benchmarks (11 datasets) and a human-preference Elo (15k prompts, ~870 annotators, 325k pairwise ratings). The headline: the data-first recipe pays off.

Where Molmo wins, and where it doesn't

• Counting (CountBenchQA, PixMo-Count): Molmo leads all models, proprietary included — the pointing chain-of-thought is the differentiator.
• Natural-image VQA & RealWorldQA: matches or beats everyone — the dense captions taught broad scene understanding.
• OCR tasks (Chart/Doc/Info/Text-VQA): beats most open models, but trails Qwen2-VL slightly.
• Reasoning (MMMU, MathVista): Molmo lags — the training mix lacks advanced-reasoning data. An honest, instructive limitation: data-first means you only get the skills your data targets.

Chapter 9: Ablations — What Actually Mattered

The ablations are the most scientifically valuable part of the paper, because they isolate which choices earned the performance. Two cheap proxy metrics drive them: cap-F1 (precision/recall of post-pretraining captions — a fast read on broad image understanding) and 11-avg (the benchmark suite average). Let's read the key findings as design lessons.

Chapter 10: Connections & Code

Molmo's lasting lesson: in 2024, the architecture of a VLM is nearly a solved, commodity template — the differentiator is data you can trust. And the way to get trustworthy data is to engineer the annotation task so humans naturally produce quality (talk, don't type; click, don't box).

The core insight as runnable Python

Two ideas you can implement today. First, the attention-pooling connector (mean-query over each 2×2 window):

import torch, torch.nn as nn

class MolmoConnector(nn.Module):
    """Pool each 2x2 ViT patch window with mean-query attention, then project."""
    def __init__(self, d_vit=1024, d_llm=4096, heads=8):
        super().__init__()
        # concat 3rd-to-last + 10th-from-last ViT layers -> 2*d_vit
        self.attn = nn.MultiheadAttention(2*d_vit, heads, batch_first=True)
        self.mlp  = nn.Sequential(nn.Linear(2*d_vit, d_llm), nn.GELU(),
                                  nn.Linear(d_llm, d_llm))

    def forward(self, patches):           # patches: [B, 24, 24, 2*d_vit]
        B, H, W, C = patches.shape
        # group into non-overlapping 2x2 windows -> [B, 144, 4, C]
        win = patches.reshape(B, H//2, 2, W//2, 2, C)
        win = win.permute(0,1,3,2,4,5).reshape(B, (H//2)*(W//2), 4, C)
        q = win.mean(dim=2, keepdim=True)          # mean of 4 patches = query
        N = win.shape[1]
        out, _ = self.attn(q.reshape(B*N,1,C),     # 1 query per window
                           win.reshape(B*N,4,C),   # 4 keys/values
                           win.reshape(B*N,4,C))
        out = out.reshape(B, N, C)                  # [B, 144, C] vision tokens
        return self.mlp(out)                        # [B, 144, d_llm]

Second, the count-by-pointing decode — the model emits points, you count them:

import re
# Molmo emits points as plain text, coords normalized to [0,100]:
out = ('<points x1="12.0" y1="30.5" x2="48.2" y2="29.0" '
       'x3="80.1" y3="31.5" alt="table">tables</points>')

pts = re.findall(r'x\d+="([\d.]+)" y\d+="([\d.]+)"', out)
pts = [(float(x), float(y)) for x, y in pts]
count = len(pts)                  # count = length of the point list
# de-normalize to a 640x480 image:
pixels = [(x/100*640, y/100*480) for x, y in pts]
print(count, pixels)              # 3  [(76.8,146.4),(308.5,139.2),(512.6,151.2)]

The cheat sheet

Where to go next in this collection

• LLaVA-1.5 — the open VLM template Molmo refines; see how the connector evolved.
• Qwen2-VL — the open-weights model Molmo trails on OCR but beats on counting.
• LLaVA-OneVision — another route to a strong open VLM via data scaling.
• MMMU — the reasoning benchmark where Molmo's data-first recipe shows its limit.
• Boxer (3D lifting) & π-0.7 — pointing as actionable grounding for agents and robots.