Kosmos-2 teaches a language model to answer with places as well as words. It writes a box as two tokens from a grid of 1,024 location tokens, learned from 91 million grounded images.
Learn how a plain next-word model learns to point: how a box becomes two words, how words link to boxes, and where the training data comes from.
Pick an object, then drag the grid finer and coarser. Watch the box the model would write snap to the grid, and watch the overlap with the true box fall as the grid gets coarse. Then we build, piece by piece, the whole recipe from the paper.
You need a rough idea of what a language model does (it predicts the next word) and what a bounding box is (a rectangle around an object). We build the rest from zero.
Based on “Kosmos-2: Grounding Multimodal Large Language Models to the World” by Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma and Furu Wei (Microsoft Research, 2023), arXiv 2306.14824. It is an assigned reading of MIT MMAI, Multimodal AI (Spring 2026), and belongs to the lecture on interactions and fusion models. Every idea is re-taught here in our own words and devices.
The scene and its three boxes are a toy made for this page, in a 224 × 224 pixel image like the paper’s. The paper uses 32 bins per side (1,024 location tokens, 7 pixels per bin); the other grid sizes are ours. The encoding is the paper’s: each corner becomes the bin that holds it, and decoding returns the centre of that bin. We number the bins row by row, which is our reading of the paper’s examples.
Chapter 0
See why a caption cannot point, and what changes when a model can answer with a place
Picture a photo of a field with five cows. You hand it to a friend and say, “Look at that cow.” Your friend asks, “Which one?” You could say “the one on the right.” Two cows are on the right. You could say “the front one on the right, beside the fence.” That works, but it took a whole sentence to do what one pointing finger does in a moment.
A language model has the same problem, and it is worse. A multimodal large language model (MLLM) is a language model that can also read images. You give it a photo and a question, and it writes words back. Before Kosmos-2, those words were the only thing it could write. It could say “there are five cows.” It could not show you which one it meant.
The paper names both gaps. The first is on the way in. You cannot point at part of the image when you ask a question. You must describe the part in words, and words can be ambiguous. The paper calls the fix referring: you refer to an image region by its box, and the model understands that region.
The second gap is on the way out. When the model answers, it cannot mark where its answer lives in the picture. The paper calls the fix grounding: the model links a word in its answer to a place in the image. If it writes “a campfire”, it also says where the campfire is. A grounded answer is checkable. You can see the box and decide whether the model looked at the right thing.
The place is given as a bounding box: the smallest rectangle that holds an object, written as the position of its top-left corner and its bottom-right corner. Two corners, four numbers. That is all the paper needs from the image side.
You might ask why we need a new model. Object detectors already draw boxes. The trouble is that a detector is a separate machine with a fixed list of object names. It does not read free text and it does not talk. Earlier language-and-vision models used one in a side pipeline. The paper points at one such model, GRILL, as one that “relies on an attached detector”. Every extra part is another thing to train, tune and keep in step.
The paper’s other target is the general MLLMs of the day: Flamingo, Kosmos-1 and others. They understand images and follow instructions, but, in the paper’s words, they “can only provide text output”. Neither family gives you one model that reads text, reads images, takes a box in and writes a box out.
Kosmos-2 makes one simple move. It turns a box into a short run of special words, and it puts those words into the same stream as the ordinary text. The model keeps its one job, predicting the next token. It never gets a new output head and never calls a detector. To write a box, it writes two tokens. To read a box, it reads two tokens.
That one move gives the model four new skills with no new machinery. It can find the thing a phrase describes. It can name the thing inside a box. It can caption an image and mark every object it mentions. It can answer a question and point at the evidence. The rest of this lesson builds that move, piece by piece, so that you could write it yourself.
First, feel the gap. Choose each way of saying which cow you mean, and watch how many cows the phrase still fits.
Five toy cows stand in a field. The target is the teal-ringed cow. Pick a way to refer to it. The cows that the phrase still fits light up in warm colour.
Illustrative: the field, the cows and which cows each phrase fits are a toy made for this page. The paper’s point is the same: a short text phrase can fit several objects, while a box fits one.
A box costs two location tokens, and it fits one cow. The best sentence cost seven words and still could not promise that. Pointing is cheaper and sharper than describing. The model needs a way to point, and that is what the next chapters build.
By the end you should be able to rebuild the recipe on a whiteboard. You will know how an image enters the stream, how a box becomes tokens, what the training sequence looks like, how 91 million training examples were made without a human drawing one box, how the model is trained and prompted, and what the tables really show.
Chapter 1
Follow an image from pixels to a short row of embeddings that sits in the text stream
Hand a photo to a program that only reads numbers. A photo is already numbers: a grid of pixels, each with a red, a green and a blue value. But a language model does not want a million raw pixels. It wants a short row of vectors, the same kind it makes from words. So the first job is to turn a picture into that short row. Kosmos-2 inherits this part whole from its parent, Kosmos-1, and does not change it.
A token is a piece of text the model counts as one unit: a word, part of a word, or a mark. Each token becomes an embedding: a list of numbers, a vector, that stands for the token. For Kosmos-2 each vector has 2,048 numbers, because the language model’s hidden size is 2,048. The model is a causal language model: it reads left to right, and each position may look only at earlier positions. Its one job is next-token prediction: given the tokens so far, give a probability to every possible next token. (Our transformer lesson builds this from zero.)
The trick of a multimodal model is simple. If an image can be turned into vectors of the right width, you can drop them into the row between the word vectors. The model then reads the image the way it reads text, because to the model both are just vectors in a row.
The paper resizes every image to 224 × 224 pixels. The first step cuts the image into square patches, small tiles, 14 × 14 pixels each. This is the idea from the vision transformer (see our ViT lesson). The count follows from the sizes.
16 × 16 = 256 patches, each 14 × 14 × 3 = 588 numbers
The vision part of the model is a vision encoder, a transformer with 24 layers and a hidden size of 1,024 (the paper copies this from Kosmos-1). It turns the 256 patches into 256 vectors of 1,024 numbers. Each vector now knows about its own tile and, through attention, about the others.
Two hundred fifty-six vectors per image is a lot to push through a language model. Attention compares every position with every earlier one, so cost grows with the square of the row length. The paper therefore adds a resampler, a small attention block with a fixed number of slots. Think of it as a panel of 64 reporters. Each reporter looks over all 256 patch vectors and writes one summary. Whatever the image size, the output is always 64 summaries. Table 7 of the paper lists the “image embedding number” as 64.
The data flow, with shapes, is this. Where the paper gives a number, we use it. The last width is our reading: the summaries must match the language model’s width of 2,048 to sit in its row.
<s> <image> [64 embeddings] </image> text…The markers <image> and </image> are special tokens that tell the model where the picture starts and stops. After them comes the text. Because of the causal rule, every text token can look back at the 64 image embeddings, but the image embeddings never see the text that follows. The picture is read first, then the words.
Try it. Change the image size and watch what changes and what does not.
Left: the image cut into 14-pixel patches. Right: the 64 image embeddings the language model receives. Drag the image side and compare the two counts.
The paper only uses 224 × 224 images, a 14-pixel patch and 64 image embeddings; other sizes are ours, to show the resampler’s fixed output. The attention-pair counts are our arithmetic for causal attention, n(n + 1)/2 pairs for n positions, on a toy row of 30 text tokens and 4 marker tokens.
At 224 pixels there are 256 patches and 4 patches feed each embedding. Raise the side to 448 and the patch count climbs to 1,024, yet the language model still sees 64 embeddings. That is the whole point of a fixed number of slots.
Suppose you skipped the resampler and fed all 256 patch vectors. The row for a 30-token caption would grow from 98 positions to 290. One attention layer would then compare 42,195 pairs instead of 4,851, about 8.7 times more. That is the cost side. The symptom you would see in training is a slower step and a smaller batch that fits in memory.
The price of the shortcut is detail. Sixty-four summaries must carry everything the language model will ever know about the picture, and a small object can be lost in them. The paper uses a 224-pixel image, which is low, and this will matter again when we meet the limits in Chapter 8. The fix in later work is to raise the resolution or the slot count; this paper keeps both small.
One more fact to keep. In Kosmos-2 the vision encoder and resampler are not frozen. The paper says it updates all parameters during training. The image side learns together with the language side.
Chapter 2
Turn the corners of a rectangle into location tokens, and decode them back
Think of the board game Battleship. You never say “the square 214 pixels from the left.” You say “B7.” A grid turns a position into a name. Kosmos-2 does the same to an image. It lays a grid over the picture, gives every cell its own name, and writes a box by naming two cells: the cell that holds the top-left corner and the cell that holds the bottom-right corner.
The names are location tokens: new words added to the model’s vocabulary, one per grid cell. To the language model they are as ordinary as “cat” or “the”. It predicts them with the same softmax it uses for words. That is the unified modelling the paper aims for.
A box is four numbers. The model could write “94 30 212 200” as digits. That would work in principle, but it takes many tokens, and the digits mix with ordinary text numbers. A grid gives each corner exactly one token, so a box is two tokens. The paper follows an earlier idea (Pix2seq, which writes object boxes as discrete tokens) and gives this reason: it wants to “encode with text tokens in a unified way”.
Take an image of width W and height H. Cut each side into P equal segments. You get P × P cells, called bins, and each bin is (W/P) × (H/P) pixels. The paper sets P = 32 and uses 224 × 224 images. So each bin is 224 / 32 = 7 pixels on a side, and there are 32 × 32 = 1,024 location tokens.
To encode a corner at pixel (x, y), find which bin holds it. To decode a token, return the centre of its bin. The paper says it uses “the coordinates of the center pixel of each bin” to determine boxes. The formulas below are ours. The paper does not state the numbering order, so we number bins row by row, which fits its examples (every token it shows is below 1,024).
<loc{id}>, from 0 to 1,023.Figure 1 of the paper shows the sentence “It sits next to a campfire” with the box tokens <loc4> <loc1007> for “a campfire”. Let us decode them with the formulas, step by step. This decoding is our own arithmetic on the paper’s tokens.
The same method gives “It” the box from (87.5, 10.5) to (220.5, 185.5): tokens 44 and 863, the snowman on the right. Notice that the two boxes overlap a little in x, from 87.5 to 108.5. Boxes are allowed to overlap.
Several boxes for one phrase are joined with a <delim> token. A single box is wrapped as <box> <loc_a> <loc_b> </box>. Now play with it. Move the four corners and read the tokens.
The grid has 32 × 32 bins of 7 pixels. Drag the corners, or pick a preset. The two purple cells hold your corners. The warm box is what you get after decoding.
The scene is our toy, not the paper’s. The grid size (32 × 32), the 7 pixel bin, the 224 pixel image and the centre-of-bin rule are the paper’s; the row-by-row numbering is our reading of it.
Every decoded corner is the centre of its bin. A true corner can sit anywhere in the bin, so a decoded corner is off by up to half a bin: 3.5 pixels on the paper’s grid. For a big object this is nothing. For a tiny one it is a lot. Take a 20-pixel box from (10, 10) to (30, 30). It decodes to (10.5, 10.5) to (31.5, 31.5). The overlap score (IoU, Chapter 6) is 380.25 / 460.75 = 0.82. A 7-pixel object from (10, 10) to (17, 17) decodes to (10.5, 10.5) to (17.5, 17.5), and the score drops to 0.76. Pick the “tiny” preset above to see it.
The fix is a finer grid, and the cost is a bigger vocabulary and a harder learning problem. A 112 × 112 grid would give 2-pixel bins but 12,544 location tokens, each needing its own embedding learned from data. The paper picks 32 × 32 and moves on. The hero at the top lets you drag along that trade.
One more fact is easy to miss. Nothing in the name <loc5> tells the model it sits left of <loc6> or above <loc37>. The paper initialises the new location embeddings randomly and updates all parameters. The model must learn the geometry of the grid from the grounded data alone.
Chapter 3
Wrap a phrase and its box in markers, and see which tokens the loss reads
On a web page, a blue word is a link. You click it and you land somewhere else. The word and its target travel together, written in a form like [text](address). Kosmos-2 borrows this idea. The text is a phrase from the caption, and the address is a box. The paper says the data format serves as a “hyperlink” that connects the image regions to the caption.
The word “hyperlink” matters for one practical reason. The model never has to be told, through a separate channel, which words go with which boxes. The link is in the text stream itself, right next to the words. A model that reads a stream left to right can read the link as easily as it reads a comma.
A special token is a vocabulary item that carries no meaning of its own. It marks structure, like a bracket in code. Kosmos-2 uses these:
| Token | What it marks |
|---|---|
<s> and </s> | Start and end of the whole sequence. |
<image> and </image> | Start and end of the 64 image embeddings. |
<grounding> | A switch that tells the model to ground the text it writes. |
<p> and </p> | Start and end of a text span, the “link text”. |
<box> and </box> | Start and end of the location tokens, the “address”. |
<delim> | A separator when one span has several boxes. |
For one span and one box, the unit is <p> text span </p><box><loc_a><loc_b></box>. For several boxes, the box part becomes <box><loc><loc><delim><loc><loc></box>.
The paper’s own Figure 1 example is “It sits next to a campfire”. Its training sequence is built in five small moves, and you can watch them below. We count one token per word, which is a simplification; a real tokenizer may split some words.
Press a step. Each step adds the tokens that are new in that step. The last step marks, in green, the tokens the loss reads.
The caption, its two spans and the five location tokens come from the paper’s Figure 1 (“It” is <loc44><loc863>, “a campfire” is <loc4><loc1007>). One token per word is our simplification. The paper says the loss covers text and location tokens; treating the markers the same way, and leaving out <s>, which nothing predicts, is our assumption.
Read the order carefully. The image embeddings come first, then <grounding>, then a caption in which each linked phrase is followed by its box. The phrase comes before the box. Because the model is causal, that order has a meaning. When the model writes the box, it has already read the phrase and the image, so the phrase can steer the box. It is grounding: from words to a place.
Here is the clever part. In a causal model, whatever you put first is the input, and whatever comes after is what the model fills in. The unit <p> span </p><box> … can be cut at two places. Stop after the span, and the model must write the box: that is grounding. Write the span and the box, then ask for words after them, and the model must write a description of the box: that is referring. The paper’s prompt for that is <p> It </p><box><loc1><loc2></box> is. Same format, no new head. Chapter 6 uses this.
A plain caption such as “a snowman next to a campfire” has no boxes. If the model had only seen grounded captions, it might add boxes everywhere. If it had also seen plain captions, it would need a clue about which kind to write. The paper adds the <grounding> token for that: “In order to tell the model when to ground text output to the visual world, we prepend the ‘<grounding>’ token to the grounded caption during training.” At evaluation the prompt ends with it, such as <s><image> … </image><grounding>, to ask for locations.
The training loss is the usual next-token loss. The paper states that it “only considers discrete tokens, such as text tokens and location tokens”. The 64 image embeddings are continuous vectors, not vocabulary items, so there is nothing to predict at those positions. In the example, the row has 87 positions. Sixty-four are image embeddings with no loss. The other 23 are discrete tokens, and 22 of them are predicted (the first one, <s>, is only ever an input).
This also fixes the symptom to watch for in practice. If the markers were missing, you would see captions with phrases but no boxes, or boxes with no phrase to say what they mean. The measurement is simple: check that every <p> has a matching </p> and a <box> right behind it. The fix is to build the sequence with one function, like the one in the Field Guide.
<p> span </p><box> … </box> format train both grounding (words to box) and referring (box to words)?Chapter 4
Run the two-step GRIT pipeline on one caption and watch a confidence threshold decide what survives
Now the model has a format, but a format needs examples. To teach grounding, you need many images where a phrase in the caption is tied to a box. Drawing those boxes by hand for millions of images is out of reach. Existing datasets are small: Visual Genome has 108,077 images, and Flickr Entities has 31,783. The paper’s answer is to build the data with machines. The result is GRIT, a dataset of Grounded Image-Text pairs, with about 91 million images, 115 million text spans and 137 million boxes.
The raw material is cheap and huge: image and caption pairs scraped from the web. The paper takes a subset of two such collections, COYO-700M and LAION-2B. A caption here is just the alt text or description that sits next to an image online. The pipeline adds the missing part, the boxes, in two steps.
The first tool is spaCy, a language-parsing library. It cuts a caption into noun chunks: a noun with the small words that go with it. From “a dog in a field of flowers” it extracts three chunks: “a dog”, “a field” and “flowers”. The paper drops abstract nouns such as “time”, “love” and “freedom”, because nobody can draw a box around them.
The second tool is a pretrained grounding model, GLIP. You give it an image and a list of phrases. It returns boxes, each with a confidence score between 0 and 1. The score says how sure the detector is that the box really shows the phrase. Two more rules clean the result:
That is a form of quality control with one dial. A higher threshold gives cleaner boxes and fewer pairs. A lower one gives more pairs and more wrong boxes.
A bare chunk such as “a dog” is short. People often point at things with longer phrases: “a dog in a field of flowers”. The model should learn to ground those too. So step 2 builds referring expressions: noun phrases rich enough to pick out one object.
The trick uses the sentence’s dependency tree. This is a diagram in which each word points to the word it modifies. In our caption, “in” hangs off “dog”, “field” hangs off “in”, and “flowers” hangs off “of”. The paper expands a chunk by “recursively traversing its children in the dependency tree and concatenating children tokens with the noun chunk.” From “a dog” it walks to the children and gets “a dog in a field of flowers”. From “a field” it gets “a field of flowers”. It does not expand across conjuncts, such as “and” lists.
Many expressions now contain others. The pipeline keeps the longest and drops the ones it contains. Here it keeps “a dog in a field of flowers” and drops “a field of flowers” and “flowers”. The expression takes the box of its own head noun, the dog. The paper’s Figure 3 lists the dog box as [290, 371, 605, 750] and the field box as [0, 264, 919, 921].
Run it. Pick a step and move the confidence threshold. The scores are a toy.
The caption is “a dog in a field of flowers”. Pick a step to see it, then slide the teal threshold. Bars that reach the line keep their box.
The caption, its three chunks, the expansions, the 0.65 cut and the keep-the-longest rule follow the paper’s Section 2 and Figure 3. The detector scores (0.69, 0.91, 0.77) are invented for this page so you can see the cut move; the paper does not list scores.
Slide the threshold from 0.65 to 0.72. The dog’s score of 0.69 now falls short. The “a dog in a field of flowers” expression has no dog box to use, so the pair that survives is “a field of flowers” with the field’s box. Slide above 0.91 and nothing survives, so the pair is discarded.
GRIT is big. Compare its images with Visual Genome’s: 90,614,680 against 108,077, about 838 times as many. The average text span in GRIT has 4.7 words. The paper’s Table 1 lists RefCOCO at 3.61 words, RefCOCO+ at 3.53 and RefCOCOg at 8.43. So GRIT’s spans are longer than the first two and much shorter than the last.
GRIT is also machine-made. Every box is a detector’s guess that cleared a threshold. The paper reports no accuracy figure for the labels and no human audit. This is a limit, and it is worth saying plainly: any systematic blind spot of the detector is now in the training data. The model learns what GLIP found, not what a person would draw. The scale is the paper’s bet that quantity and variety outweigh this noise.
Chapter 5
Mix text, captions, grounded pairs and interleaved pages into one batch, then follow the learning rate
Think of a stew with four ingredients in one pot. You stir it all together, taste it a few thousand times, and adjust. Kosmos-2 training works like that. There is one pot, one objective and one optimiser. The four ingredients are different kinds of data. The model never switches mode. It always reads a sequence of tokens and predicts the next one.
The objective is the one we met in Chapter 1: next-token prediction. At each position, the model gives a probability to every token in the vocabulary, and the loss is the negative log of the probability it gave to the true next token. Chapter 3 told us which positions count: the discrete ones. Words, markers and location tokens are all scored in the same way.
The paper starts from Kosmos-1’s weights. Only the 1,024 location embeddings are new, and they start random. Then it trains all parameters, the vision encoder, resampler and language model together. The total is about 1.6 billion trainable parameters. The language model is a 24-layer MAGNETO transformer (a variant that adds extra normalisation for stable training) with hidden size 2,048, 32 attention heads and a 8,192-wide feed-forward layer.
Each training step draws a batch from four sources. The paper gives token counts for the batch: 185K from text corpora, 215K from original and grounded image-caption pairs, and 19K from interleaved image-text pages (web pages where pictures sit among paragraphs). These add up to 419K tokens per step. With 60,000 steps, the model sees about 25 billion tokens: 60,000 × 419K is 25.1 billion. The run uses 256 V100 GPUs and takes about one day.
Appendix Table 7 counts the batch in sequences instead: 93 text sequences, 1,117 image-caption sequences, 1,117 grounded sequences, and 47 interleaved ones. Divide tokens by sequences and you can guess how long each kind is. This is our arithmetic, with the paper’s rounded totals:
Nearly half of the sequences in the pretraining batch, 1,117 of 2,374, are the new grounded pairs. The rest keep the old skills alive. The paper does not run an ablation on this mix, so why it was chosen is a fair guess rather than a result. The sensible reading is that language data guards language skills, and that the grounded half supplies the new skill.
The learning rate is the size of each update step. The paper warms it up for 375 steps to a peak of 2e-4, then lowers it in a straight line to zero at step 60,000. A warmup keeps early updates small, while the new random location embeddings are still noise. The line below makes the schedule exact.
Step 100: 2e-4 × 100 / 375 = 5.3e-5 · Step 30,000: 2e-4 × 30,000 / 59,625 = 1.0e-4
The other optimiser settings are AdamW with betas of 0.9 and 0.98, weight decay 0.01 and dropout 0.1. These are standard.
After pretraining, the paper runs a second, shorter stage called instruction tuning. It teaches the model to follow human-style requests. This stage lasts 10,000 steps with the same 375 warmup steps and a much lower peak rate of 1e-5. It mixes language-only instructions (Unnatural Instructions and FLANv2), vision-language instructions (LLaVA-Instruct) and a new kind: grounded instruction data built from GRIT.
The grounded instructions are made from expression and box pairs. One kind asks for the box: the input is <p> expression </p> and the model writes the location tokens. The other asks for the words: <p> It </p><box><loc1><loc2></box> is and the model writes the expression. Appendix Table 9 gives six templates for the second kind, such as “What is <p> it </p>…? It is {expression}.”
Top: one batch, split by data source, in sequences. Bottom: the learning rate over training. Pick a stage and drag the step.
Batch sizes, steps, warmup and peak rates are from the paper’s Tables 7 and 8. The straight-line decay to zero is the paper’s schedule. For instruction tuning the paper does not say whether the rate decays to zero; we draw it the same way as pretraining, which is an assumption.
Watch how the shares change. Pretraining is 47% grounded sequences. Instruction tuning is 73% grounded data by sequence count (1,404 of 1,917), because the grounded instructions share a pool with the grounded pairs. After the second stage the model is a Kosmos-2 you can talk to.
Chapter 6
Write the prompt for each task, read the box that comes back, and score it with IoU
You have a trained model. How do you ask it for anything? You do not call a different function for each job. You write the beginning of a sequence and let the model continue it. The only thing that changes between tasks is the beginning you write. This is the payoff of the single token stream.
Every prompt starts the same way: <s><image> Image Embedding </image>. For grounding tasks it ends the prefix with <grounding> so the model knows to write locations. Here is what comes after, for each task in the paper.
| Task | Prompt after the image | The model writes |
|---|---|---|
| Phrase grounding | A man in a blue hard hat and <p>orange safety vest</p> | A box, <box><loc><loc></box> |
| Referring expression comprehension | <p>A man in a blue hard hat and orange safety vest</p> | A box |
| Referring expression generation | <p> It </p><box><loc><loc></box> is | A description of that box |
| Image captioning | An image of | A caption |
| Visual question answering | Question: {question} Answer: | An answer |
| Grounded captioning | Describe this image in detail: | A caption with linked boxes |
Phrase grounding has one subtle design choice. The paper does not give the model the phrase alone. It gives the phrase together with the words before it. In a photo with several men, the context “a man in a blue hard hat and” helps the model see which vest is meant. This follows the benchmark’s own captions. The paper explains it as a way to “reduce ambiguity”.
Referring expression generation can also use examples. In the few-shot setting the prompt first holds two or four solved examples: an image, a box and a description. Then comes the new box. The model picks up the pattern from the examples without any weight update. That is in-context learning, and Table 4 will show it helps a little.
The decoding rules matter for reproduction. The paper uses greedy search (always pick the most probable next token) for referring expression generation and visual question answering, and beam search with a beam of 5 (keep the five best partial sentences) for image captioning. All of these use 224 × 224 images.
The model’s reply is text. To use it, you pull out the location tokens and decode them with the Chapter 2 rule. In code that is a short function (the version in the Field Guide handles every case). The key line is a pattern match for <loc(\d+)>, then the row-and-column arithmetic you already know.
Suppose the model writes a box. Is it right? The test is intersection over union (IoU): the area where the predicted and true boxes overlap, divided by the area they cover together. It runs from 0 (no overlap) to 1 (the same box).
A = (0,0,10,10), B = (5,5,15,15): 25 / (100 + 100 − 25) = 0.14 · B = (2,0,12,10): 80 / 120 = 0.67
The paper counts a box as correct when IoU is greater than 0.5. It adds a strict rule: if the model writes a location sequence that cannot be decoded, such as <box><loc1></box> with one token instead of two, the answer counts as a miss. For phrase grounding it reports recall at k (R@1, R@5, R@10): the share of phrases for which any of the model’s first k boxes hits. If the model wrote fewer than k boxes, all of them are used. For referring expression comprehension it scores only the first generated box and reports accuracy.
The dashed blue box is the truth for “a snowman”. The warm box is the model’s guess. Move it and watch the overlap. The green area is the intersection.
Illustrative: the true box is our toy snowman, (94, 30) to (212, 200) in a 224 × 224 image. The IoU formula, the greater-than-0.5 bar and the rule that an undecodable output is a miss are the paper’s.
Try the “a little off” preset. A box shifted by only 20 pixels can still pass. A box at half the size cannot, because even when it sits fully inside the truth its IoU is 0.25 (it covers a quarter of the area). This is why the 0.5 bar is generous to shifts and harsh to the wrong scale.
The symptom of a model that has not learned its tokens is a pile of unreadable boxes. The measurement is the share of outputs that fail to decode, which the benchmark counts against the model. The fix is more grounded data and training. The paper does not report this share, so it is a number to measure if you reproduce the work.
Chapter 7
Compare a zero-shot model with finetuned specialists, benchmark by benchmark
A table of results is a claim in numbers. Before we read the paper’s claim, we need the vocabulary. This chapter walks through the three tables that test the new skills: Table 2 (phrase grounding), Table 3 (referring expression comprehension) and Table 4 (referring expression generation).
A finetuned model was trained further on the training split of the benchmark it is tested on. A zero-shot model was not. It meets the benchmark cold. The paper marks each row with a tick or a cross in a “Zero-shot” column. Almost every competitor is finetuned. Kosmos-2 is not. So when Kosmos-2 scores below a finetuned model, it is not a clean loss: the finetuned model had an advantage. And when it scores above a zero-shot model, the comparison is fair.
The task is to find the box for each phrase of a caption. The score is recall at k: the share of phrases for which one of the model’s first k boxes has IoU above 0.5.
At R@1 on the test split, Kosmos-2 scores 78.7. The earlier zero-shot model GRILL scores 18.9, which is a gap of 59.8 points. The finetuned VisualBERT scores 71.3, so Kosmos-2 beats it by 7.4 points, on val and on test. But the strongest finetuned models are ahead: MDETR 84.3, GLIP 87.1 and FIBER 87.4. Those lead by 5.6, 8.4 and 8.7 points.
Look at the other columns. Kosmos-2 scores 78.7, 80.1 and 80.1 at R@1, R@5 and R@10. The detectors climb steeply, from 87.1 to 98.1 for GLIP. The reason is the design. Detectors work from many proposals and rank them, so more guesses help. Kosmos-2 writes the box it believes in, usually one, so extra guesses add little. The paper puts it this way: the model has “no prior designs (e.g., object queries or proposals)” and so gets “similar results among R@1, R@5, and R@10”. A plain reading: R@1 is the fair column for this model, and R@10 flatters the detectors.
Here the model gets one long description and must return one box. The score is accuracy: how often the first box has IoU above 0.5. The three datasets differ. RefCOCO and RefCOCO+ came from a two-player game and use short phrases. RefCOCO+ bans words about position, such as “on the left”. RefCOCOg has longer expressions and allows position words.
The results are mixed, and it is worth reading them straight. On RefCOCOg test, Kosmos-2 reaches 61.65, far above the zero-shot GRILL at 47.5 (+14.2). But the finetuned systems lead by a wide margin: FIBER 87.32, OFA 85.20, MDETR 83.31. On RefCOCO val the gap is bigger still: Kosmos-2 52.32 against FIBER 90.68, a gap of 38.4 points. The paper calls this “slightly lower”. That word undersells it. The paper’s explanation is the short expressions in RefCOCO and RefCOCO+, such as “left bottom”, which fit a web-caption-trained model poorly. It lists understanding “more types of human expressions” as future work.
This is the reverse direction: a box goes in and a description comes out. Two scores are used. METEOR matches words between the model’s sentence and human sentences, allowing stems and synonyms. CIDEr compares the sentence with several human references and gives more weight to words that are distinctive for this image. Higher is better for both.
Zero-shot, Kosmos-2 gets CIDEr 60.3, which beats the finetuned SLR (59.2) by 1.1. The better finetuned SLR with reranking reaches 66.2. With two examples in the prompt Kosmos-2 climbs to 62.2, and with four to 62.3. That is in-context learning, with no weight update. On METEOR it stays below SLR (12.2 zero-shot, 14.1 with four examples, against 15.4).
Choose a benchmark below. The warm bar is Kosmos-2. Blue bars are finetuned models. Grey bars are other zero-shot models.
Pick a table and column. Bars show the paper’s numbers on one shared 0 to 100 scale.
Numbers are copied from the paper’s Tables 2, 3 and 4 (Flickr30k Entities test R@1 and R@10, RefCOCO val accuracy, RefCOCOg test accuracy, RefCOCOg CIDEr). Not every row exists on every benchmark. VisionLLM, for example, reports only RefCOCO val here.
The paper reports one number per cell. There are no error bars, no repeated runs, and no ablation that removes GRIT to show its effect. The table shows what Kosmos-2 can do and where it stands. It does not prove which part of the recipe matters. The paper also evaluates all images at 224 × 224 pixels, a low resolution for small objects. The fair headline is modest and still real: a single next-token model, never finetuned on these benchmarks, gets within sight of specialists on phrase grounding, and it beats earlier zero-shot systems clearly.
Chapter 8
Check captioning, question answering and language tasks against Kosmos-1, then list the limits
Teach a student to draw maps and you might worry that their handwriting gets worse. The same worry applies here. The model spent part of its training on a new skill. Did it lose some of the old ones? The paper tests this in two places: perception-language tasks (Table 5) and plain language tasks (Table 6). The answer is mostly no, with some exceptions.
Two zero-shot tests carry over from Kosmos-1. Image captioning asks the model to describe a picture, here on the Flickr30k test set, scored with CIDEr. The prompt is “An image of” and the decoder is a beam search of width 5. Visual question answering (VQA) asks a question about an image, here on VQAv2 test-dev, with the prompt “Question: {question} Answer:” and greedy decoding. For a fair comparison, the paper reports both models without instruction tuning.
Captioning rises slightly, from 65.2 to 66.7 CIDEr. VQA falls slightly, from 46.7 to 45.6 accuracy. The paper calls this “comparable overall performance.” Both models are also well above Flamingo-3B and Flamingo-9B on captioning (60.6 and 61.5), per the same table.
Here Kosmos-2 is compared with Kosmos-1 and with a plain text-only language model trained on the same text with the same set-up (the paper takes both baselines from the Kosmos-1 paper). The tasks cover story endings, sentence completion, pronoun resolution, commonsense physics and three SuperGLUE tests. Drag the toggle below to compare against each baseline and read the change per task.
Each bar is Kosmos-2 minus the baseline, in score points. Right of the line means Kosmos-2 gained. Left means it lost.
Scores are from the paper’s Tables 5 and 6, all zero-shot and without instruction tuning. Table 5 has no text-only baseline, so the captioning and VQA bars show only against Kosmos-1. Differences are our subtraction.
Against Kosmos-1, five of the eight language tasks move by less than one point. Two rise: BoolQ gains 5.6 and COPA gains 4.0. One falls hard: CB drops 14.2, from 44.6 to 30.4. CB is one of the smaller SuperGLUE sets, so big swings there are common (this is our note, not the paper’s). Averaged over the eight tasks the change is −0.65 points. Without CB it is +1.29. The paper’s summary is fair: “similar performance” on five tasks, “a decrease” on CB, and “improvement” on BoolQ and COPA.
Against the text-only model the picture is a little weaker. Kosmos-2 trails on seven of the eight tasks: by 0.3 to 2.5 points on six of them, and by 8.9 on CB. It is ahead only on BoolQ. This is the honest cost of sharing capacity between pictures and words, though the paper does not isolate the cause.
The symptom is a score that drops on a task you did not train for. The measurement is a table like Table 6, run before and after the change. The fix the paper uses is to keep language-only data in every batch (185K of 419K tokens). That does not erase the CB drop. A reader who builds on this work should run the same table for their own model.
None of these makes the idea less useful. They tell you what to test if you build on it: smaller objects, other box styles, label quality and a before-and-after language table.
Chapter 9
Keep the numbers, the terms, the checklist and the core code on one page
Kosmos-2 is a language model that learned to point. It turns a box into two location tokens from a 32 × 32 grid, links each phrase to its box with <p> and <box> markers, and trains on GRIT, 91 million web images whose phrases were boxed by a parser and a detector. One next-token loss trains words and places together. Every task is then a prompt prefix. The result is competitive zero-shot grounding, a small cost on the older skills, and clear limits on small objects and short human phrases.
| Quantity | Value | Where |
|---|---|---|
| Image size and patch | 224 × 224, 14 × 14 patches, 256 patches | Section 3.3 |
| Image embeddings | 64 | Table 7 |
| Location tokens | 32 × 32 = 1,024, 7 px bins | Section 3.3 |
| Parameters | about 1.6 billion | Section 3.3 |
| GRIT | 91M images, 115M spans, 137M boxes, 4.7 words per span | Section 2, Table 1 |
| Detector cut | confidence above 0.65 | Section 2 |
| Batch | 419K tokens: 185K text, 215K captions, 19K interleaved | Section 3.3 |
| Schedule | 60K steps, 375 warmup, peak 2e-4, about 25B tokens | Section 3.3, Table 7 |
| Compute | 256 V100 GPUs, about one day | Section 3.3 |
| Flickr30k R@1 (test) | 78.7 (GRILL 18.9, FIBER 87.4) | Table 2 |
| RefCOCOg test | 61.65 (GRILL 47.5, FIBER 87.32) | Table 3 |
| RefCOCOg CIDEr | 60.3 zero-shot, 62.3 with 4 examples | Table 4 |
| Hit rule | IoU greater than 0.5; undecodable output is a miss | Section 4.1 |
| Term | Plain meaning |
|---|---|
| Grounding | Linking words the model writes to places in the image. |
| Referring | Pointing at a region of the image, with a box, as part of the input. |
| Location token | A vocabulary word that names one cell of the 32 × 32 grid. |
| Bin | One grid cell, 7 × 7 pixels on a 224 pixel image. |
| Resampler | An attention block with fixed slots that squeezes patch vectors into 64 image embeddings. |
| GRIT | The machine-made set of grounded image-text pairs. |
| Zero-shot | Tested without finetuning on that benchmark. |
| IoU, R@k | Overlap score of two boxes; share of phrases hit within the first k boxes. |
This sketch is ours, written from the paper’s description. It encodes a box, builds a grounded sequence, parses a reply and scores it. The bin numbering is our reading of the paper.
pythonimport re P, SIDE = 32, 224 # bins per side, image side in pixels BIN = SIDE / P # 7 pixels per bin def corner_to_token(x, y): col = min(int(x // BIN), P - 1) row = min(int(y // BIN), P - 1) return row * P + col # 0 .. 1023 def token_to_corner(t): row, col = divmod(t, P) return (col + 0.5) * BIN, (row + 0.5) * BIN # centre of the bin def box_text(b): a, z = corner_to_token(b[0], b[1]), corner_to_token(b[2], b[3]) return f"<box><loc{a}><loc{z}></box>" def grounded_sequence(parts): # [("It", box), ("sits next to", None), ...] out = ["<s>", "<image>", "[64 image embeddings]", "</image>", "<grounding>"] for text, box in parts: out.append(f"<p> {text} </p>" + box_text(box) if box else text) return " ".join(out + ["</s>"]) def parse_boxes(reply): boxes = [] for m in re.finditer(r"<box>((?:<loc\d+>|<delim>)+)</box>", reply): for one in m.group(1).split("<delim>"): t = [int(v) for v in re.findall(r"<loc(\d+)>", one)] if len(t) != 2: boxes.append(None); continue # undecodable: counts as a miss (x1, y1), (x2, y2) = token_to_corner(t[0]), token_to_corner(t[1]) boxes.append((x1, y1, x2, y2)) return boxes def iou(a, b): iw = min(a[2], b[2]) - max(a[0], b[0]); ih = min(a[3], b[3]) - max(a[1], b[1]) inter = max(iw, 0) * max(ih, 0) area = lambda r: (r[2] - r[0]) * (r[3] - r[1]) return inter / (area(a) + area(b) - inter) def hit(box, truth): # the paper's rule return box is not None and iou(box, truth) > 0.5
Tick what you could now do from memory. This is for you; nothing is saved.
A private self-check. It does not save or send anything.
Credit: Kosmos-2 is the work of Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma and Furu Wei at Microsoft Research. This lesson re-teaches it for MIT MMAI, Multimodal AI (Spring 2026, Paul Liang). Now press Teach and draw the pipeline from a box to two tokens and back from memory. Then go back to the grid at the top, set it to 4 × 4 and to 32 × 32, and explain the overlap you see.