Kosmos-2

Kosmos-2 teaches a language model to answer with places as well as words. It writes a box as two tokens from a grid of 1,024 location tokens, learned from 91 million grounded images.

Learn how a plain next-word model learns to point: how a box becomes two words, how words link to boxes, and where the training data comes from.

Pick an object, then drag the grid finer and coarser. Watch the box the model would write snap to the grid, and watch the overlap with the true box fall as the grid gets coarse. Then we build, piece by piece, the whole recipe from the paper.

You need a rough idea of what a language model does (it predicts the next word) and what a bounding box is (a rectangle around an object). We build the rest from zero.

Based on “Kosmos-2: Grounding Multimodal Large Language Models to the World” by Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma and Furu Wei (Microsoft Research, 2023), arXiv 2306.14824. It is an assigned reading of MIT MMAI, Multimodal AI (Spring 2026), and belongs to the lecture on interactions and fusion models. Every idea is re-taught here in our own words and devices.

Snap a box to the grid

32 × 32

The scene and its three boxes are a toy made for this page, in a 224 × 224 pixel image like the paper’s. The paper uses 32 bins per side (1,024 location tokens, 7 pixels per bin); the other grid sizes are ours. The encoding is the paper’s: each corner becomes the bin that holds it, and decoding returns the centre of that bin. We number the bins row by row, which is our reading of the paper’s examples.

Chapter 0

A Model That Can Only Talk About the Picture

See why a caption cannot point, and what changes when a model can answer with a place

Picture a photo of a field with five cows. You hand it to a friend and say, “Look at that cow.” Your friend asks, “Which one?” You could say “the one on the right.” Two cows are on the right. You could say “the front one on the right, beside the fence.” That works, but it took a whole sentence to do what one pointing finger does in a moment.

A language model has the same problem, and it is worse. A multimodal large language model (MLLM) is a language model that can also read images. You give it a photo and a question, and it writes words back. Before Kosmos-2, those words were the only thing it could write. It could say “there are five cows.” It could not show you which one it meant.

Two gaps: no finger going in, no finger coming out

The paper names both gaps. The first is on the way in. You cannot point at part of the image when you ask a question. You must describe the part in words, and words can be ambiguous. The paper calls the fix referring: you refer to an image region by its box, and the model understands that region.

The second gap is on the way out. When the model answers, it cannot mark where its answer lives in the picture. The paper calls the fix grounding: the model links a word in its answer to a place in the image. If it writes “a campfire”, it also says where the campfire is. A grounded answer is checkable. You can see the box and decide whether the model looked at the right thing.

The place is given as a bounding box: the smallest rectangle that holds an object, written as the position of its top-left corner and its bottom-right corner. Two corners, four numbers. That is all the paper needs from the image side.

Why earlier methods fall short

You might ask why we need a new model. Object detectors already draw boxes. The trouble is that a detector is a separate machine with a fixed list of object names. It does not read free text and it does not talk. Earlier language-and-vision models used one in a side pipeline. The paper points at one such model, GRILL, as one that “relies on an attached detector”. Every extra part is another thing to train, tune and keep in step.

The paper’s other target is the general MLLMs of the day: Flamingo, Kosmos-1 and others. They understand images and follow instructions, but, in the paper’s words, they “can only provide text output”. Neither family gives you one model that reads text, reads images, takes a box in and writes a box out.

The paper’s bet: a box is just more words

Kosmos-2 makes one simple move. It turns a box into a short run of special words, and it puts those words into the same stream as the ordinary text. The model keeps its one job, predicting the next token. It never gets a new output head and never calls a detector. To write a box, it writes two tokens. To read a box, it reads two tokens.

That one move gives the model four new skills with no new machinery. It can find the thing a phrase describes. It can name the thing inside a box. It can caption an image and mark every object it mentions. It can answer a question and point at the evidence. The rest of this lesson builds that move, piece by piece, so that you could write it yourself.

First, feel the gap. Choose each way of saying which cow you mean, and watch how many cows the phrase still fits.

Say which cow

Five toy cows stand in a field. The target is the teal-ringed cow. Pick a way to refer to it. The cows that the phrase still fits light up in warm colour.

Illustrative: the field, the cows and which cows each phrase fits are a toy made for this page. The paper’s point is the same: a short text phrase can fit several objects, while a box fits one.

A box costs two location tokens, and it fits one cow. The best sentence cost seven words and still could not promise that. Pointing is cheaper and sharper than describing. The model needs a way to point, and that is what the next chapters build.

What you will be able to do

By the end you should be able to rebuild the recipe on a whiteboard. You will know how an image enters the stream, how a box becomes tokens, what the training sequence looks like, how 91 million training examples were made without a human drawing one box, how the model is trained and prompted, and what the tables really show.

  1. How a language model reads a picture: patches, an encoder, a resampler, and a short row of embeddings.
  2. A box is two words: the 32 × 32 grid, 1,024 location tokens, and decoding back.
  3. Linking words to boxes: the markdown-link format, special tokens and what the loss reads.
  4. The GRIT data: noun chunks, a detector, a threshold, and referring expressions.
  5. Training: one next-token objective over four data sources, then instruction tuning.
  6. Prompts and scoring: grounding, referring, captioning and IoU.
  7. Results: grounding and referring against finetuned specialists.
  8. Cost and limits: what grounding did to captioning, question answering and language skills.
  9. Field guide: numbers, terms, checklist and the core code.
What is the main reason to let a model answer with a box instead of only words?

Chapter 1

How a Language Model Reads a Picture

Follow an image from pixels to a short row of embeddings that sits in the text stream

Hand a photo to a program that only reads numbers. A photo is already numbers: a grid of pixels, each with a red, a green and a blue value. But a language model does not want a million raw pixels. It wants a short row of vectors, the same kind it makes from words. So the first job is to turn a picture into that short row. Kosmos-2 inherits this part whole from its parent, Kosmos-1, and does not change it.

Background: what a language model reads

A token is a piece of text the model counts as one unit: a word, part of a word, or a mark. Each token becomes an embedding: a list of numbers, a vector, that stands for the token. For Kosmos-2 each vector has 2,048 numbers, because the language model’s hidden size is 2,048. The model is a causal language model: it reads left to right, and each position may look only at earlier positions. Its one job is next-token prediction: given the tokens so far, give a probability to every possible next token. (Our transformer lesson builds this from zero.)

The trick of a multimodal model is simple. If an image can be turned into vectors of the right width, you can drop them into the row between the word vectors. The model then reads the image the way it reads text, because to the model both are just vectors in a row.

From pixels to patches

The paper resizes every image to 224 × 224 pixels. The first step cuts the image into square patches, small tiles, 14 × 14 pixels each. This is the idea from the vision transformer (see our ViT lesson). The count follows from the sizes.

Npatch
How many patches the image becomes.
H / p
Image height in pixels divided by the patch side p. With H = 224 and p = 14 this is 16 rows of patches.
W / p
The same for the width: 224 / 14 = 16 columns.

16 × 16 = 256 patches, each 14 × 14 × 3 = 588 numbers

The vision part of the model is a vision encoder, a transformer with 24 layers and a hidden size of 1,024 (the paper copies this from Kosmos-1). It turns the 256 patches into 256 vectors of 1,024 numbers. Each vector now knows about its own tile and, through attention, about the others.

The resampler squeezes 256 down to 64

Two hundred fifty-six vectors per image is a lot to push through a language model. Attention compares every position with every earlier one, so cost grows with the square of the row length. The paper therefore adds a resampler, a small attention block with a fixed number of slots. Think of it as a panel of 64 reporters. Each reporter looks over all 256 patch vectors and writes one summary. Whatever the image size, the output is always 64 summaries. Table 7 of the paper lists the “image embedding number” as 64.

The data flow, with shapes, is this. Where the paper gives a number, we use it. The last width is our reading: the summaries must match the language model’s width of 2,048 to sit in its row.

  1. Image: 224 × 224 × 3 pixels.
  2. Patches: 256 × 588, then the vision encoder gives 256 × 1024.
  3. Resampler: 64 × 2048 image embeddings.
  4. Row for the language model: <s> <image> [64 embeddings] </image> text…

The markers <image> and </image> are special tokens that tell the model where the picture starts and stops. After them comes the text. Because of the causal rule, every text token can look back at the 64 image embeddings, but the image embeddings never see the text that follows. The picture is read first, then the words.

Try it. Change the image size and watch what changes and what does not.

Squeeze patches into 64 slots

Left: the image cut into 14-pixel patches. Right: the 64 image embeddings the language model receives. Drag the image side and compare the two counts.

224 px

The paper only uses 224 × 224 images, a 14-pixel patch and 64 image embeddings; other sizes are ours, to show the resampler’s fixed output. The attention-pair counts are our arithmetic for causal attention, n(n + 1)/2 pairs for n positions, on a toy row of 30 text tokens and 4 marker tokens.

At 224 pixels there are 256 patches and 4 patches feed each embedding. Raise the side to 448 and the patch count climbs to 1,024, yet the language model still sees 64 embeddings. That is the whole point of a fixed number of slots.

What the shortcut buys, and what it can cost

Suppose you skipped the resampler and fed all 256 patch vectors. The row for a 30-token caption would grow from 98 positions to 290. One attention layer would then compare 42,195 pairs instead of 4,851, about 8.7 times more. That is the cost side. The symptom you would see in training is a slower step and a smaller batch that fits in memory.

The price of the shortcut is detail. Sixty-four summaries must carry everything the language model will ever know about the picture, and a small object can be lost in them. The paper uses a 224-pixel image, which is low, and this will matter again when we meet the limits in Chapter 8. The fix in later work is to raise the resolution or the slot count; this paper keeps both small.

One more fact to keep. In Kosmos-2 the vision encoder and resampler are not frozen. The paper says it updates all parameters during training. The image side learns together with the language side.

An image grows from 224 to 448 pixels on a side. What happens to the number of image embeddings the language model reads?

Chapter 2

A Box Is Two Words From a 1,024-Word Grid

Turn the corners of a rectangle into location tokens, and decode them back

Think of the board game Battleship. You never say “the square 214 pixels from the left.” You say “B7.” A grid turns a position into a name. Kosmos-2 does the same to an image. It lays a grid over the picture, gives every cell its own name, and writes a box by naming two cells: the cell that holds the top-left corner and the cell that holds the bottom-right corner.

The names are location tokens: new words added to the model’s vocabulary, one per grid cell. To the language model they are as ordinary as “cat” or “the”. It predicts them with the same softmax it uses for words. That is the unified modelling the paper aims for.

Why not just write the numbers?

A box is four numbers. The model could write “94 30 212 200” as digits. That would work in principle, but it takes many tokens, and the digits mix with ordinary text numbers. A grid gives each corner exactly one token, so a box is two tokens. The paper follows an earlier idea (Pix2seq, which writes object boxes as discrete tokens) and gives this reason: it wants to “encode with text tokens in a unified way”.

The grid, derived

Take an image of width W and height H. Cut each side into P equal segments. You get P × P cells, called bins, and each bin is (W/P) × (H/P) pixels. The paper sets P = 32 and uses 224 × 224 images. So each bin is 224 / 32 = 7 pixels on a side, and there are 32 × 32 = 1,024 location tokens.

To encode a corner at pixel (x, y), find which bin holds it. To decode a token, return the centre of its bin. The paper says it uses “the coordinates of the center pixel of each bin” to determine boxes. The formulas below are ours. The paper does not state the numbering order, so we number bins row by row, which fits its examples (every token it shows is below 1,024).

id
The number in the token name, <loc{id}>, from 0 to 1,023.
b
Bin side in pixels: W / P, which is 7 here.
⌊y / b⌋
The row of the bin: divide the pixel height by 7 and drop the fraction. A corner at y = 30 is in row 4.
P
Bins per side, 32. Each full row holds 32 tokens, so we multiply the row by 32.
⌊x / b⌋
The column of the bin, found the same way from the pixel’s x position.
x̂
The decoded x position: the centre of the column, half a bin from its left edge.
col
The column index recovered from the token: id mod 32. The row is the id divided by 32.

A worked example from the paper

Figure 1 of the paper shows the sentence “It sits next to a campfire” with the box tokens <loc4> <loc1007> for “a campfire”. Let us decode them with the formulas, step by step. This decoding is our own arithmetic on the paper’s tokens.

  1. First token, 4: row = 4 ÷ 32 = 0, column = 4 mod 32 = 4. Centre: x = (4 + 0.5) × 7 = 31.5, y = (0 + 0.5) × 7 = 3.5.
  2. Second token, 1007: row = 1007 ÷ 32 = 31 (31 × 32 = 992), column = 1007 − 992 = 15. Centre: x = (15 + 0.5) × 7 = 108.5, y = (31 + 0.5) × 7 = 220.5.
  3. Box: from (31.5, 3.5) to (108.5, 220.5) in a 224 × 224 image: a tall box, 77 pixels wide, on the left. That fits a campfire standing beside a snowman.

The same method gives “It” the box from (87.5, 10.5) to (220.5, 185.5): tokens 44 and 863, the snowman on the right. Notice that the two boxes overlap a little in x, from 87.5 to 108.5. Boxes are allowed to overlap.

Several boxes for one phrase are joined with a <delim> token. A single box is wrapped as <box> <loc_a> <loc_b> </box>. Now play with it. Move the four corners and read the tokens.

Encode and decode a box

The grid has 32 × 32 bins of 7 pixels. Drag the corners, or pick a preset. The two purple cells hold your corners. The warm box is what you get after decoding.

94
30
212
200

The scene is our toy, not the paper’s. The grid size (32 × 32), the 7 pixel bin, the 224 pixel image and the centre-of-bin rule are the paper’s; the row-by-row numbering is our reading of it.

The failure mode: small objects and coarse bins

Every decoded corner is the centre of its bin. A true corner can sit anywhere in the bin, so a decoded corner is off by up to half a bin: 3.5 pixels on the paper’s grid. For a big object this is nothing. For a tiny one it is a lot. Take a 20-pixel box from (10, 10) to (30, 30). It decodes to (10.5, 10.5) to (31.5, 31.5). The overlap score (IoU, Chapter 6) is 380.25 / 460.75 = 0.82. A 7-pixel object from (10, 10) to (17, 17) decodes to (10.5, 10.5) to (17.5, 17.5), and the score drops to 0.76. Pick the “tiny” preset above to see it.

The fix is a finer grid, and the cost is a bigger vocabulary and a harder learning problem. A 112 × 112 grid would give 2-pixel bins but 12,544 location tokens, each needing its own embedding learned from data. The paper picks 32 × 32 and moves on. The hero at the top lets you drag along that trade.

One more fact is easy to miss. Nothing in the name <loc5> tells the model it sits left of <loc6> or above <loc37>. The paper initialises the new location embeddings randomly and updates all parameters. The model must learn the geometry of the grid from the grounded data alone.

On the paper’s 32 × 32 grid over a 224 × 224 image, what is the largest error a single decoded corner can have in one direction?

Chapter 3

Linking Words to Boxes Like a Markdown Link

Wrap a phrase and its box in markers, and see which tokens the loss reads

On a web page, a blue word is a link. You click it and you land somewhere else. The word and its target travel together, written in a form like [text](address). Kosmos-2 borrows this idea. The text is a phrase from the caption, and the address is a box. The paper says the data format serves as a “hyperlink” that connects the image regions to the caption.

The word “hyperlink” matters for one practical reason. The model never has to be told, through a separate channel, which words go with which boxes. The link is in the text stream itself, right next to the words. A model that reads a stream left to right can read the link as easily as it reads a comma.

The markers

A special token is a vocabulary item that carries no meaning of its own. It marks structure, like a bracket in code. Kosmos-2 uses these:

TokenWhat it marks
<s> and </s>Start and end of the whole sequence.
<image> and </image>Start and end of the 64 image embeddings.
<grounding>A switch that tells the model to ground the text it writes.
<p> and </p>Start and end of a text span, the “link text”.
<box> and </box>Start and end of the location tokens, the “address”.
<delim>A separator when one span has several boxes.

For one span and one box, the unit is <p> text span </p><box><loc_a><loc_b></box>. For several boxes, the box part becomes <box><loc><loc><delim><loc><loc></box>.

The full sequence, built step by step

The paper’s own Figure 1 example is “It sits next to a campfire”. Its training sequence is built in five small moves, and you can watch them below. We count one token per word, which is a simplification; a real tokenizer may split some words.

Build the sequence

Press a step. Each step adds the tokens that are new in that step. The last step marks, in green, the tokens the loss reads.

The caption, its two spans and the five location tokens come from the paper’s Figure 1 (“It” is <loc44><loc863>, “a campfire” is <loc4><loc1007>). One token per word is our simplification. The paper says the loss covers text and location tokens; treating the markers the same way, and leaving out <s>, which nothing predicts, is our assumption.

Read the order carefully. The image embeddings come first, then <grounding>, then a caption in which each linked phrase is followed by its box. The phrase comes before the box. Because the model is causal, that order has a meaning. When the model writes the box, it has already read the phrase and the image, so the phrase can steer the box. It is grounding: from words to a place.

One format, two directions

Here is the clever part. In a causal model, whatever you put first is the input, and whatever comes after is what the model fills in. The unit <p> span </p><box> … can be cut at two places. Stop after the span, and the model must write the box: that is grounding. Write the span and the box, then ask for words after them, and the model must write a description of the box: that is referring. The paper’s prompt for that is <p> It </p><box><loc1><loc2></box> is. Same format, no new head. Chapter 6 uses this.

The switch: why <grounding> exists

A plain caption such as “a snowman next to a campfire” has no boxes. If the model had only seen grounded captions, it might add boxes everywhere. If it had also seen plain captions, it would need a clue about which kind to write. The paper adds the <grounding> token for that: “In order to tell the model when to ground text output to the visual world, we prepend the ‘<grounding>’ token to the grounded caption during training.” At evaluation the prompt ends with it, such as <s><image> … </image><grounding>, to ask for locations.

What the loss reads

The training loss is the usual next-token loss. The paper states that it “only considers discrete tokens, such as text tokens and location tokens”. The 64 image embeddings are continuous vectors, not vocabulary items, so there is nothing to predict at those positions. In the example, the row has 87 positions. Sixty-four are image embeddings with no loss. The other 23 are discrete tokens, and 22 of them are predicted (the first one, <s>, is only ever an input).

This also fixes the symptom to watch for in practice. If the markers were missing, you would see captions with phrases but no boxes, or boxes with no phrase to say what they mean. The measurement is simple: check that every <p> has a matching </p> and a <box> right behind it. The fix is to build the sequence with one function, like the one in the Field Guide.

Why can the same <p> span </p><box> … </box> format train both grounding (words to box) and referring (box to words)?

Chapter 4

Building 91 Million Grounded Pairs Without Hand Labels

Run the two-step GRIT pipeline on one caption and watch a confidence threshold decide what survives

Now the model has a format, but a format needs examples. To teach grounding, you need many images where a phrase in the caption is tied to a box. Drawing those boxes by hand for millions of images is out of reach. Existing datasets are small: Visual Genome has 108,077 images, and Flickr Entities has 31,783. The paper’s answer is to build the data with machines. The result is GRIT, a dataset of Grounded Image-Text pairs, with about 91 million images, 115 million text spans and 137 million boxes.

The raw material is cheap and huge: image and caption pairs scraped from the web. The paper takes a subset of two such collections, COYO-700M and LAION-2B. A caption here is just the alt text or description that sits next to an image online. The pipeline adds the missing part, the boxes, in two steps.

Step 1: find the things the caption names

The first tool is spaCy, a language-parsing library. It cuts a caption into noun chunks: a noun with the small words that go with it. From “a dog in a field of flowers” it extracts three chunks: “a dog”, “a field” and “flowers”. The paper drops abstract nouns such as “time”, “love” and “freedom”, because nobody can draw a box around them.

The second tool is a pretrained grounding model, GLIP. You give it an image and a list of phrases. It returns boxes, each with a confidence score between 0 and 1. The score says how sure the detector is that the box really shows the phrase. Two more rules clean the result:

  1. Non-maximum suppression (NMS): when two boxes overlap heavily, keep the better one and drop the other, “even if they are not for the same noun chunk”.
  2. Confidence cut: keep only boxes with a score above 0.65.
  3. Empty means discard: if no box survives, throw the whole image and caption pair away.

That is a form of quality control with one dial. A higher threshold gives cleaner boxes and fewer pairs. A lower one gives more pairs and more wrong boxes.

Step 2: grow the chunks into full descriptions

A bare chunk such as “a dog” is short. People often point at things with longer phrases: “a dog in a field of flowers”. The model should learn to ground those too. So step 2 builds referring expressions: noun phrases rich enough to pick out one object.

The trick uses the sentence’s dependency tree. This is a diagram in which each word points to the word it modifies. In our caption, “in” hangs off “dog”, “field” hangs off “in”, and “flowers” hangs off “of”. The paper expands a chunk by “recursively traversing its children in the dependency tree and concatenating children tokens with the noun chunk.” From “a dog” it walks to the children and gets “a dog in a field of flowers”. From “a field” it gets “a field of flowers”. It does not expand across conjuncts, such as “and” lists.

Many expressions now contain others. The pipeline keeps the longest and drops the ones it contains. Here it keeps “a dog in a field of flowers” and drops “a field of flowers” and “flowers”. The expression takes the box of its own head noun, the dog. The paper’s Figure 3 lists the dog box as [290, 371, 605, 750] and the field box as [0, 264, 919, 921].

Run it. Pick a step and move the confidence threshold. The scores are a toy.

Run the GRIT pipeline

The caption is “a dog in a field of flowers”. Pick a step to see it, then slide the teal threshold. Bars that reach the line keep their box.

0.65

The caption, its three chunks, the expansions, the 0.65 cut and the keep-the-longest rule follow the paper’s Section 2 and Figure 3. The detector scores (0.69, 0.91, 0.77) are invented for this page so you can see the cut move; the paper does not list scores.

Slide the threshold from 0.65 to 0.72. The dog’s score of 0.69 now falls short. The “a dog in a field of flowers” expression has no dog box to use, so the pair that survives is “a field of flowers” with the field’s box. Slide above 0.91 and nothing survives, so the pair is discarded.

What GRIT is, and what it is not

GRIT is big. Compare its images with Visual Genome’s: 90,614,680 against 108,077, about 838 times as many. The average text span in GRIT has 4.7 words. The paper’s Table 1 lists RefCOCO at 3.61 words, RefCOCO+ at 3.53 and RefCOCOg at 8.43. So GRIT’s spans are longer than the first two and much shorter than the last.

GRIT is also machine-made. Every box is a detector’s guess that cleared a threshold. The paper reports no accuracy figure for the labels and no human audit. This is a limit, and it is worth saying plainly: any systematic blind spot of the detector is now in the training data. The model learns what GLIP found, not what a person would draw. The scale is the paper’s bet that quantity and variety outweigh this noise.

In GRIT step 1, a caption’s noun chunks all get detector scores at or below 0.65. What does the pipeline do with that image and caption pair?

Chapter 5

One Objective, Four Kinds of Data

Mix text, captions, grounded pairs and interleaved pages into one batch, then follow the learning rate

Think of a stew with four ingredients in one pot. You stir it all together, taste it a few thousand times, and adjust. Kosmos-2 training works like that. There is one pot, one objective and one optimiser. The four ingredients are different kinds of data. The model never switches mode. It always reads a sequence of tokens and predicts the next one.

The recipe

The objective is the one we met in Chapter 1: next-token prediction. At each position, the model gives a probability to every token in the vocabulary, and the loss is the negative log of the probability it gave to the true next token. Chapter 3 told us which positions count: the discrete ones. Words, markers and location tokens are all scored in the same way.

The paper starts from Kosmos-1’s weights. Only the 1,024 location embeddings are new, and they start random. Then it trains all parameters, the vision encoder, resampler and language model together. The total is about 1.6 billion trainable parameters. The language model is a 24-layer MAGNETO transformer (a variant that adds extra normalisation for stable training) with hidden size 2,048, 32 attention heads and a 8,192-wide feed-forward layer.

Each training step draws a batch from four sources. The paper gives token counts for the batch: 185K from text corpora, 215K from original and grounded image-caption pairs, and 19K from interleaved image-text pages (web pages where pictures sit among paragraphs). These add up to 419K tokens per step. With 60,000 steps, the model sees about 25 billion tokens: 60,000 × 419K is 25.1 billion. The run uses 256 V100 GPUs and takes about one day.

The same batch, in sequences

Appendix Table 7 counts the batch in sequences instead: 93 text sequences, 1,117 image-caption sequences, 1,117 grounded sequences, and 47 interleaved ones. Divide tokens by sequences and you can guess how long each kind is. This is our arithmetic, with the paper’s rounded totals:

Nearly half of the sequences in the pretraining batch, 1,117 of 2,374, are the new grounded pairs. The rest keep the old skills alive. The paper does not run an ablation on this mix, so why it was chosen is a fair guess rather than a result. The sensible reading is that language data guards language skills, and that the grounded half supplies the new skill.

The learning rate: warm up, then glide down

The learning rate is the size of each update step. The paper warms it up for 375 steps to a peak of 2e-4, then lowers it in a straight line to zero at step 60,000. A warmup keeps early updates small, while the new random location embeddings are still noise. The line below makes the schedule exact.

lr(s)
The learning rate used at training step s.
ηpeak
The highest rate: 2e-4 for pretraining. Instruction tuning uses 1e-5.
s / sw
The warmup ramp, climbing from 0 to 1 over the first sw = 375 steps.
(S − s) / (S − sw)
The linear decay, falling from 1 to 0 between step 375 and the last step S = 60,000.
ηpeak
The same peak, which scales the decay.

Step 100: 2e-4 × 100 / 375 = 5.3e-5  ·  Step 30,000: 2e-4 × 30,000 / 59,625 = 1.0e-4

The other optimiser settings are AdamW with betas of 0.9 and 0.98, weight decay 0.01 and dropout 0.1. These are standard.

Then instruction tuning

After pretraining, the paper runs a second, shorter stage called instruction tuning. It teaches the model to follow human-style requests. This stage lasts 10,000 steps with the same 375 warmup steps and a much lower peak rate of 1e-5. It mixes language-only instructions (Unnatural Instructions and FLANv2), vision-language instructions (LLaVA-Instruct) and a new kind: grounded instruction data built from GRIT.

The grounded instructions are made from expression and box pairs. One kind asks for the box: the input is <p> expression </p> and the model writes the location tokens. The other asks for the words: <p> It </p><box><loc1><loc2></box> is and the model writes the expression. Appendix Table 9 gives six templates for the second kind, such as “What is <p> it </p>…? It is {expression}.”

Mix the batch, scrub the schedule

Top: one batch, split by data source, in sequences. Bottom: the learning rate over training. Pick a stage and drag the step.

30,000

Batch sizes, steps, warmup and peak rates are from the paper’s Tables 7 and 8. The straight-line decay to zero is the paper’s schedule. For instruction tuning the paper does not say whether the rate decays to zero; we draw it the same way as pretraining, which is an assumption.

Watch how the shares change. Pretraining is 47% grounded sequences. Instruction tuning is 73% grounded data by sequence count (1,404 of 1,917), because the grounded instructions share a pool with the grounded pairs. After the second stage the model is a Kosmos-2 you can talk to.

Which statement about the Kosmos-2 training loss is correct?

Chapter 6

Grounding, Referring and Captioning Are Just Prompts

Write the prompt for each task, read the box that comes back, and score it with IoU

You have a trained model. How do you ask it for anything? You do not call a different function for each job. You write the beginning of a sequence and let the model continue it. The only thing that changes between tasks is the beginning you write. This is the payoff of the single token stream.

The prompts, task by task

Every prompt starts the same way: <s><image> Image Embedding </image>. For grounding tasks it ends the prefix with <grounding> so the model knows to write locations. Here is what comes after, for each task in the paper.

TaskPrompt after the imageThe model writes
Phrase groundingA man in a blue hard hat and <p>orange safety vest</p>A box, <box><loc><loc></box>
Referring expression comprehension<p>A man in a blue hard hat and orange safety vest</p>A box
Referring expression generation<p> It </p><box><loc><loc></box> isA description of that box
Image captioningAn image ofA caption
Visual question answeringQuestion: {question} Answer:An answer
Grounded captioningDescribe this image in detail:A caption with linked boxes

Phrase grounding has one subtle design choice. The paper does not give the model the phrase alone. It gives the phrase together with the words before it. In a photo with several men, the context “a man in a blue hard hat and” helps the model see which vest is meant. This follows the benchmark’s own captions. The paper explains it as a way to “reduce ambiguity”.

Referring expression generation can also use examples. In the few-shot setting the prompt first holds two or four solved examples: an image, a box and a description. Then comes the new box. The model picks up the pattern from the examples without any weight update. That is in-context learning, and Table 4 will show it helps a little.

The decoding rules matter for reproduction. The paper uses greedy search (always pick the most probable next token) for referring expression generation and visual question answering, and beam search with a beam of 5 (keep the five best partial sentences) for image captioning. All of these use 224 × 224 images.

Reading the box back

The model’s reply is text. To use it, you pull out the location tokens and decode them with the Chapter 2 rule. In code that is a short function (the version in the Field Guide handles every case). The key line is a pattern match for <loc(\d+)>, then the row-and-column arithmetic you already know.

How a box is scored: IoU

Suppose the model writes a box. Is it right? The test is intersection over union (IoU): the area where the predicted and true boxes overlap, divided by the area they cover together. It runs from 0 (no overlap) to 1 (the same box).

IoU(A, B)
The overlap score for boxes A and B.
area(A ∩ B)
Area of the shared rectangle: its width times its height, or 0 if the boxes do not touch.
area(A ∪ B)
Area of A plus area of B minus the shared area, so the overlap is not counted twice.

A = (0,0,10,10), B = (5,5,15,15): 25 / (100 + 100 − 25) = 0.14  ·  B = (2,0,12,10): 80 / 120 = 0.67

The paper counts a box as correct when IoU is greater than 0.5. It adds a strict rule: if the model writes a location sequence that cannot be decoded, such as <box><loc1></box> with one token instead of two, the answer counts as a miss. For phrase grounding it reports recall at k (R@1, R@5, R@10): the share of phrases for which any of the model’s first k boxes hits. If the model wrote fewer than k boxes, all of them are used. For referring expression comprehension it scores only the first generated box and reports accuracy.

Score a predicted box

The dashed blue box is the truth for “a snowman”. The warm box is the model’s guess. Move it and watch the overlap. The green area is the intersection.

0 px
0 px
100%

Illustrative: the true box is our toy snowman, (94, 30) to (212, 200) in a 224 × 224 image. The IoU formula, the greater-than-0.5 bar and the rule that an undecodable output is a miss are the paper’s.

Try the “a little off” preset. A box shifted by only 20 pixels can still pass. A box at half the size cannot, because even when it sits fully inside the truth its IoU is 0.25 (it covers a quarter of the area). This is why the 0.5 bar is generous to shifts and harsh to the wrong scale.

The symptom of a model that has not learned its tokens is a pile of unreadable boxes. The measurement is the share of outputs that fail to decode, which the benchmark counts against the model. The fix is more grounded data and training. The paper does not report this share, so it is a number to measure if you reproduce the work.

For referring expression comprehension, Kosmos-2 writes a first box with IoU 0.48 against the truth. How is that answer scored?

Chapter 7

Reading the Grounding and Referring Tables

Compare a zero-shot model with finetuned specialists, benchmark by benchmark

A table of results is a claim in numbers. Before we read the paper’s claim, we need the vocabulary. This chapter walks through the three tables that test the new skills: Table 2 (phrase grounding), Table 3 (referring expression comprehension) and Table 4 (referring expression generation).

Zero-shot against finetuned

A finetuned model was trained further on the training split of the benchmark it is tested on. A zero-shot model was not. It meets the benchmark cold. The paper marks each row with a tick or a cross in a “Zero-shot” column. Almost every competitor is finetuned. Kosmos-2 is not. So when Kosmos-2 scores below a finetuned model, it is not a clean loss: the finetuned model had an advantage. And when it scores above a zero-shot model, the comparison is fair.

Table 2: phrase grounding on Flickr30k Entities

The task is to find the box for each phrase of a caption. The score is recall at k: the share of phrases for which one of the model’s first k boxes has IoU above 0.5.

At R@1 on the test split, Kosmos-2 scores 78.7. The earlier zero-shot model GRILL scores 18.9, which is a gap of 59.8 points. The finetuned VisualBERT scores 71.3, so Kosmos-2 beats it by 7.4 points, on val and on test. But the strongest finetuned models are ahead: MDETR 84.3, GLIP 87.1 and FIBER 87.4. Those lead by 5.6, 8.4 and 8.7 points.

Look at the other columns. Kosmos-2 scores 78.7, 80.1 and 80.1 at R@1, R@5 and R@10. The detectors climb steeply, from 87.1 to 98.1 for GLIP. The reason is the design. Detectors work from many proposals and rank them, so more guesses help. Kosmos-2 writes the box it believes in, usually one, so extra guesses add little. The paper puts it this way: the model has “no prior designs (e.g., object queries or proposals)” and so gets “similar results among R@1, R@5, and R@10”. A plain reading: R@1 is the fair column for this model, and R@10 flatters the detectors.

Table 3: referring expression comprehension

Here the model gets one long description and must return one box. The score is accuracy: how often the first box has IoU above 0.5. The three datasets differ. RefCOCO and RefCOCO+ came from a two-player game and use short phrases. RefCOCO+ bans words about position, such as “on the left”. RefCOCOg has longer expressions and allows position words.

The results are mixed, and it is worth reading them straight. On RefCOCOg test, Kosmos-2 reaches 61.65, far above the zero-shot GRILL at 47.5 (+14.2). But the finetuned systems lead by a wide margin: FIBER 87.32, OFA 85.20, MDETR 83.31. On RefCOCO val the gap is bigger still: Kosmos-2 52.32 against FIBER 90.68, a gap of 38.4 points. The paper calls this “slightly lower”. That word undersells it. The paper’s explanation is the short expressions in RefCOCO and RefCOCO+, such as “left bottom”, which fit a web-caption-trained model poorly. It lists understanding “more types of human expressions” as future work.

Table 4: referring expression generation

This is the reverse direction: a box goes in and a description comes out. Two scores are used. METEOR matches words between the model’s sentence and human sentences, allowing stems and synonyms. CIDEr compares the sentence with several human references and gives more weight to words that are distinctive for this image. Higher is better for both.

Zero-shot, Kosmos-2 gets CIDEr 60.3, which beats the finetuned SLR (59.2) by 1.1. The better finetuned SLR with reranking reaches 66.2. With two examples in the prompt Kosmos-2 climbs to 62.2, and with four to 62.3. That is in-context learning, with no weight update. On METEOR it stays below SLR (12.2 zero-shot, 14.1 with four examples, against 15.4).

Choose a benchmark below. The warm bar is Kosmos-2. Blue bars are finetuned models. Grey bars are other zero-shot models.

Compare on a benchmark

Pick a table and column. Bars show the paper’s numbers on one shared 0 to 100 scale.

Numbers are copied from the paper’s Tables 2, 3 and 4 (Flickr30k Entities test R@1 and R@10, RefCOCO val accuracy, RefCOCOg test accuracy, RefCOCOg CIDEr). Not every row exists on every benchmark. VisionLLM, for example, reports only RefCOCO val here.

How to read these tables with care

The paper reports one number per cell. There are no error bars, no repeated runs, and no ablation that removes GRIT to show its effect. The table shows what Kosmos-2 can do and where it stands. It does not prove which part of the recipe matters. The paper also evaluates all images at 224 × 224 pixels, a low resolution for small objects. The fair headline is modest and still real: a single next-token model, never finetuned on these benchmarks, gets within sight of specialists on phrase grounding, and it beats earlier zero-shot systems clearly.

On Flickr30k Entities, Kosmos-2 scores 78.7, 80.1 and 80.1 at R@1, R@5 and R@10, while GLIP climbs from 87.1 to 98.1. What is the best explanation?

Chapter 8

What Grounding Cost the Old Skills, and What It Cannot Do

Check captioning, question answering and language tasks against Kosmos-1, then list the limits

Teach a student to draw maps and you might worry that their handwriting gets worse. The same worry applies here. The model spent part of its training on a new skill. Did it lose some of the old ones? The paper tests this in two places: perception-language tasks (Table 5) and plain language tasks (Table 6). The answer is mostly no, with some exceptions.

Table 5: captioning and question answering

Two zero-shot tests carry over from Kosmos-1. Image captioning asks the model to describe a picture, here on the Flickr30k test set, scored with CIDEr. The prompt is “An image of” and the decoder is a beam search of width 5. Visual question answering (VQA) asks a question about an image, here on VQAv2 test-dev, with the prompt “Question: {question} Answer:” and greedy decoding. For a fair comparison, the paper reports both models without instruction tuning.

Captioning rises slightly, from 65.2 to 66.7 CIDEr. VQA falls slightly, from 46.7 to 45.6 accuracy. The paper calls this “comparable overall performance.” Both models are also well above Flamingo-3B and Flamingo-9B on captioning (60.6 and 61.5), per the same table.

Table 6: eight language tasks

Here Kosmos-2 is compared with Kosmos-1 and with a plain text-only language model trained on the same text with the same set-up (the paper takes both baselines from the Kosmos-1 paper). The tasks cover story endings, sentence completion, pronoun resolution, commonsense physics and three SuperGLUE tests. Drag the toggle below to compare against each baseline and read the change per task.

What changed, task by task

Each bar is Kosmos-2 minus the baseline, in score points. Right of the line means Kosmos-2 gained. Left means it lost.

Scores are from the paper’s Tables 5 and 6, all zero-shot and without instruction tuning. Table 5 has no text-only baseline, so the captioning and VQA bars show only against Kosmos-1. Differences are our subtraction.

Against Kosmos-1, five of the eight language tasks move by less than one point. Two rise: BoolQ gains 5.6 and COPA gains 4.0. One falls hard: CB drops 14.2, from 44.6 to 30.4. CB is one of the smaller SuperGLUE sets, so big swings there are common (this is our note, not the paper’s). Averaged over the eight tasks the change is −0.65 points. Without CB it is +1.29. The paper’s summary is fair: “similar performance” on five tasks, “a decrease” on CB, and “improvement” on BoolQ and COPA.

Against the text-only model the picture is a little weaker. Kosmos-2 trails on seven of the eight tasks: by 0.3 to 2.5 points on six of them, and by 8.9 on CB. It is ahead only on BoolQ. This is the honest cost of sharing capacity between pictures and words, though the paper does not isolate the cause.

Failure mode: a new skill can crowd out an old one

The symptom is a score that drops on a task you did not train for. The measurement is a table like Table 6, run before and after the change. The fix the paper uses is to keep language-only data in every batch (185K of 419K tokens). That does not erase the CB drop. A reader who builds on this work should run the same table for their own model.

The limits, listed plainly

  1. Small pictures. Images are 224 × 224 and a bin is 7 pixels, so tiny objects get coarse boxes (Chapter 2: 0.76 IoU for a 7-pixel object).
  2. Boxes only. A rectangle cannot describe a thin or curved shape. The model has no masks and no points.
  3. Machine labels. Every GRIT box is a detector guess above 0.65. The paper gives no accuracy audit of the labels.
  4. Short human phrases. On RefCOCO val the model scores 52.32, far below the finetuned 90.68. The paper names this as future work.
  5. No ablations. The paper never removes GRIT, the grid size or the language mix to show what each part buys.
  6. Big claims, no test. The abstract speaks of embodied AI and general intelligence. The paper runs no embodied experiment.
  7. Responsible use. The ethics statement limits the model to academic and research use and forbids making unsuitable material.

None of these makes the idea less useful. They tell you what to test if you build on it: smaller objects, other box styles, label quality and a before-and-after language table.

Which statement about Table 6 is supported by the paper’s numbers?

Chapter 9

Field Guide

Keep the numbers, the terms, the checklist and the core code on one page

Kosmos-2 is a language model that learned to point. It turns a box into two location tokens from a 32 × 32 grid, links each phrase to its box with <p> and <box> markers, and trains on GRIT, 91 million web images whose phrases were boxed by a parser and a detector. One next-token loss trains words and places together. Every task is then a prompt prefix. The result is competitive zero-shot grounding, a small cost on the older skills, and clear limits on small objects and short human phrases.

The numbers on one page

QuantityValueWhere
Image size and patch224 × 224, 14 × 14 patches, 256 patchesSection 3.3
Image embeddings64Table 7
Location tokens32 × 32 = 1,024, 7 px binsSection 3.3
Parametersabout 1.6 billionSection 3.3
GRIT91M images, 115M spans, 137M boxes, 4.7 words per spanSection 2, Table 1
Detector cutconfidence above 0.65Section 2
Batch419K tokens: 185K text, 215K captions, 19K interleavedSection 3.3
Schedule60K steps, 375 warmup, peak 2e-4, about 25B tokensSection 3.3, Table 7
Compute256 V100 GPUs, about one daySection 3.3
Flickr30k R@1 (test)78.7 (GRILL 18.9, FIBER 87.4)Table 2
RefCOCOg test61.65 (GRILL 47.5, FIBER 87.32)Table 3
RefCOCOg CIDEr60.3 zero-shot, 62.3 with 4 examplesTable 4
Hit ruleIoU greater than 0.5; undecodable output is a missSection 4.1

Terms

TermPlain meaning
GroundingLinking words the model writes to places in the image.
ReferringPointing at a region of the image, with a box, as part of the input.
Location tokenA vocabulary word that names one cell of the 32 × 32 grid.
BinOne grid cell, 7 × 7 pixels on a 224 pixel image.
ResamplerAn attention block with fixed slots that squeezes patch vectors into 64 image embeddings.
GRITThe machine-made set of grounded image-text pairs.
Zero-shotTested without finetuning on that benchmark.
IoU, R@kOverlap score of two boxes; share of phrases hit within the first k boxes.

The core as code

This sketch is ours, written from the paper’s description. It encodes a box, builds a grounded sequence, parses a reply and scores it. The bin numbering is our reading of the paper.

pythonimport re
P, SIDE = 32, 224            # bins per side, image side in pixels
BIN = SIDE / P               # 7 pixels per bin

def corner_to_token(x, y):
    col = min(int(x // BIN), P - 1)
    row = min(int(y // BIN), P - 1)
    return row * P + col         # 0 .. 1023

def token_to_corner(t):
    row, col = divmod(t, P)
    return (col + 0.5) * BIN, (row + 0.5) * BIN   # centre of the bin

def box_text(b):
    a, z = corner_to_token(b[0], b[1]), corner_to_token(b[2], b[3])
    return f"<box><loc{a}><loc{z}></box>"

def grounded_sequence(parts):      # [("It", box), ("sits next to", None), ...]
    out = ["<s>", "<image>", "[64 image embeddings]", "</image>", "<grounding>"]
    for text, box in parts:
        out.append(f"<p> {text} </p>" + box_text(box) if box else text)
    return " ".join(out + ["</s>"])

def parse_boxes(reply):
    boxes = []
    for m in re.finditer(r"<box>((?:<loc\d+>|<delim>)+)</box>", reply):
        for one in m.group(1).split("<delim>"):
            t = [int(v) for v in re.findall(r"<loc(\d+)>", one)]
            if len(t) != 2:
                boxes.append(None); continue      # undecodable: counts as a miss
            (x1, y1), (x2, y2) = token_to_corner(t[0]), token_to_corner(t[1])
            boxes.append((x1, y1, x2, y2))
    return boxes

def iou(a, b):
    iw = min(a[2], b[2]) - max(a[0], b[0]); ih = min(a[3], b[3]) - max(a[1], b[1])
    inter = max(iw, 0) * max(ih, 0)
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)

def hit(box, truth):                  # the paper's rule
    return box is not None and iou(box, truth) > 0.5

A checklist for rebuilding it

Tick what you could now do from memory. This is for you; nothing is saved.

Can you rebuild it?

A private self-check. It does not save or send anything.

Keep going

The takeaway. Do not add a new output head for every new kind of answer. Find a way to write the answer as tokens, put it in the same stream as the words, and let one next-token loss do the rest. For Kosmos-2 that way is a grid of 1,024 location tokens and a markdown-style link.

Credit: Kosmos-2 is the work of Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma and Furu Wei at Microsoft Research. This lesson re-teaches it for MIT MMAI, Multimodal AI (Spring 2026, Paul Liang). Now press Teach and draw the pipeline from a box to two tokens and back from memory. Then go back to the grid at the top, set it to 4 × 4 and to 32 × 32, and explain the overlap you see.

You want a model to output a new kind of answer, such as a box. Which choice matches the Kosmos-2 approach?
Kosmos-2: Grounding Multimodal Large Language Models to the World
Read the paper · Code and models · MIT MMAI schedule · The lecture this paper belongs to