MM1: What Matters When a Language Model Learns to See

Apple tested every choice in a seeing language model. Bigger pictures won: 224 to 336 pixels added about 3 points. A new connector between eye and brain added almost nothing.

Learn how a model reads pictures and words together, which design choices move the score, and how a small test setup turns into a 30-billion-parameter model.

Pick one change and spend it. Watch the picture the model sees, and the gain the paper measured. Then we build the whole recipe, piece by piece: the eye, the connector, the three kinds of data, the mix, the scaling rules and the tests.

You need a rough idea of what a language model does (it predicts the next word) and what a pixel is. We build the rest from zero.

Paper: McKinzie, Gan, Fauconnier, Dodge, Zhang, Dufter, Shah, Du, Peng and colleagues at Apple, “MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training” (arXiv 2403.09611). Assigned reading for MIT MMAI, Multimodal AI (Paul Liang), with Lecture 6, Interactions and Fusion Models. Every idea is re-taught here in our own words and devices.

Spend one change

The gains are the paper’s (Section 3.2, Table 1 and Figure 4): about 3 points for 224 to 336 pixels, usually under 1 point for ViT-L to ViT-H, over 1 point in few-shot tests for VeCap, and no clear gain for a new connector type. Bar lengths are drawn from those words, not from a table cell. The picture is a toy scene, illustrative.

Chapter 0

Why a Recipe for Seeing Models Is Worth a Paper

See why building a model that reads pictures and words is mostly a pile of small choices, and why the paper tests them one at a time

You take a photo of a table with two beers on it. You take a second photo of the bar’s menu. You ask a model: “How much should I pay for all the beer on the table, according to the price on the menu?” Nobody types the prices. The model must look at both pictures, read the numbers, count the glasses and do the sum.

The paper’s Figure 2 shows this exact test. The paper’s largest model, MM1-30B-Chat, answers 12. Asked to explain, it says there are 2 beers on the table, each costs 6 on the menu, and 2 times 6 is 12. Two rival models in the same figure fail. Emu-Chat-37B answers 15.99. LLaVA-NeXT-34B hedges, assumes two of each of four beers, and estimates 44. The example comes from the VILA paper. The question is not about the sum. It is about what the model had to be built from to see the right things.

A language model that can see

A language model reads text and predicts what comes next. It knows nothing about pixels. A multimodal large language model (MLLM) is a language model that has been given a way to read pictures too. It takes in images and text, and it writes text. The paper’s name for the idea is simple: the language model still does the talking, and a second part does the looking.

Three parts must work together. An image encoder turns a picture into a grid of number lists (think of a translator who describes each small square of the picture in numbers). A connector reshapes those lists so the language model can accept them as if they were words. The language model then reads the picture’s numbers and the text together. Chapter 1 follows the data through all three.

The problem: nobody wrote the recipe down

By early 2024 many such models existed. The paper sorts them into two groups. Closed models can be used, but nothing is known about their data, design or training. Open models release their weights (the learned numbers) and describe the data and settings. Yet almost none of them explain how the builders arrived at those choices, and that matters most for pre-training, the long first stage where the model learns from huge piles of data before anyone teaches it a task.

So a team starting from zero faces a wall of questions. Which image encoder? How many numbers per picture? How should the connector work? What data? How much text-only data, and how much of each kind? Every wrong guess costs weeks of compute. Earlier papers did study some of these, but the paper says they leave gaps: no optimisation details, no ablations that justify each choice, or a goal of copying one closed model.

The method: change one thing at a time

The paper’s answer is an ablation study. You take a working small model, change exactly one part, and measure the score again. If the score moves, that part matters. If it does not, you can stop worrying about it. The idea is the one a cook uses when testing a recipe: change the salt, keep the rest, taste.

The paper’s small base model has four fixed parts. The image encoder is a ViT-L/14 trained with a CLIP loss on 336 by 336 pixel images. The connector is a C-Abstractor that outputs 144 image tokens. The data is 45% captioned images, 45% interleaved image-text documents and 10% text-only. The language model is a 1.2 billion parameter decoder. Chapters 2 to 5 break each of those words open.

The tests are captioning and question answering on eight benchmarks, scored with no examples (zero-shot) and with 4 or 8 worked examples in the prompt (few-shot). Few-shot is the key test. It shows whether the model can use examples in its own prompt.

What the paper found, in one list

The authors rank what matters. On the modelling side, the order is: image resolution, then the encoder’s loss and size, then the encoder’s training data. How the picture is plugged into the language model matters little. On the data side, captioned images lift zero-shot scores, while interleaved documents and text-only text carry the few-shot and text skills. The hero at the top of this page is a small version of this list. Try the guess below before you read on.

Guess, then reveal

Pick your guess

Four design choices are listed. Choose the one you think mattered most for the score, then read the paper’s order. Your pick is outlined in teal.

Pick a guess to reveal the order.

The order is the paper’s own statement in Section 1 (image resolution, then visual encoder loss and capacity, then visual encoder pre-training data; little evidence that the connector design matters). The rungs show rank only. They do not show sizes.

The rest of the paper turns this ranking into a recipe, then scales the recipe from 1.2 billion to 30 billion parameters (and to a 64 billion parameter mixture of experts). The final models beat published pre-trained models at few-shot captioning and question answering, and after fine-tuning they are competitive on 12 benchmarks. This lesson follows the paper’s order, with one mechanism in each chapter.

The road

  1. The anatomy of a seeing model: how a picture becomes tokens, with shapes.
  2. The image encoder: contrastive and reconstructive training, and why pixels win.
  3. The connector: three ways to plug the eye into the brain, and why it hardly matters.
  4. Three kinds of data: captions, interleaved documents, text-only.
  5. The mix and the packing: 45, 45, 10, and a mask that stops leaks.
  6. Scaling up: a learning-rate rule fitted in log space, and mixture of experts.
  7. Pre-training results and fine-tuning: what the tables say, and how resolution grows.
  8. In-context reasoning: few-shot chain of thought on a token budget.
  9. The field guide: numbers, checklist, code and where to go next.
The authors want to know whether a new connector design helps. What is the cleanest way to find out?

Chapter 1

The Anatomy of a Seeing Model

Follow one photo from pixels to tokens, and count how much room it takes in the language model’s reading window

Imagine you must describe a photo to a friend over the phone, and your friend only understands a numbered list of words. You cut the photo into small squares. You describe each square with a list of numbers. Then you shorten that long list of squares into a smaller list, because your friend can only hold so much in mind. This is, almost exactly, how a seeing model works.

The language model has one job it was trained for: read a row of tokens (small pieces of text such as a word or part of a word, each stored as a list of numbers) and predict the next one. To let it “see”, the paper turns each picture into a few more rows of the same kind of numbers and places them in the row. The language model never learns a new kind of input. It just reads a row that now contains picture tokens as well as word tokens. (For a full lesson on how text becomes tokens, see tokenization.)

Step 1: cut the picture into patches

A pixel is one dot of colour in a picture, stored as three numbers (red, green, blue). A 378 by 378 picture has 378 × 378 = 142,884 pixels, far too many to feed in one at a time. The image encoder is a vision transformer, or ViT, which cuts the picture into square patches and treats each patch as one unit. (Our vision transformer lesson builds one from zero.)

The paper’s encoders use 14 by 14 pixel patches, written ViT-/14. A 378 pixel side holds 378 / 14 = 27 patches across, so the picture becomes 27 × 27 = 729 patches. The encoder reads all 729 patches together and returns one list of numbers for each patch, with every list now holding some knowledge of the whole picture. In shapes: the input is [3, 378, 378] and the output is [729, D], where D is the encoder’s width.

Step 2: shrink the patches to a few image tokens

729 patches is still a lot for the language model. Every token it reads costs memory and time, and a prompt with several pictures multiplies the cost. So a vision-language connector maps the [729, D] list down to a fixed number of image tokens, here 144, each already shaped like a word token: [144, d], where d is the language model’s width. This does two jobs at once. It reduces the count, and it translates the numbers into the language model’s own space. Chapter 3 opens the box.

Step 3: place the tokens in the row

The language model is a decoder-only transformer, the kind that reads a row left to right and predicts the next token. Text goes through the usual tokenizer. Each picture’s 144 tokens are dropped into the row at the spot where the picture belongs. A row can look like: some words, picture one, more words, picture two, a question. The paper fixes the sequence length (the longest row the model reads at once) at 4,096 positions, with up to 16 images in one sequence.

That gives a budget. Let L be the sequence length, k the number of images, m the tokens per image and t the number of text tokens. The row must fit:

k
The number of images in the row. The paper allows up to 16.
m
Image tokens each picture takes after the connector. The final model uses 144.
t
Text tokens left over. This is what the document, caption or question can use.
L
The sequence length, 4,096 in pre-training.

16 × 144 = 2,304 image tokens, so 4,096 − 2,304 = 1,792 tokens of text remain

With 64 tokens per image instead, 16 pictures take only 1,024 positions and 3,072 remain for text. This tension, more image detail against fewer positions for everything else, is the heart of the next three chapters. The paper’s batch has 512 sequences of 4,096 positions, which it describes as roughly 1 million text tokens and 1 million image tokens per batch. Play with the budget below.

Count the tokens

Left: the picture cut into patches. Middle: the connector squeezes them into image tokens. Right: the 4,096-position row the language model reads. Drag the picture side and the number of images, and pick 64 or 144 tokens per image.

336 px
4

Patch size 14, sequence length 4,096 and up to 16 images come from the paper (Sections 3.1, 4 and Appendix B.1); 64 and 144 are the two token counts it compares. The patch grid is drawn at most 24 patches wide to stay legible. Counts are exact arithmetic.

The number that jumps out is the cost of resolution. Double the side from 336 to 672 and the patch count grows four times, from 576 to 2,304. That is why the connector has to shrink the list, and why a bigger picture can still be worth it when the shrinking is done well. The paper notes later that a 672 pixel picture gives exactly 2,304 patches.

What goes in, what comes out

At test time the model gets a prompt and writes text. The paper’s evaluation prompts show it plainly. For captioning the input row is the picture tokens followed by the words “A photo of”, and the model continues the sentence. For question answering the row is the picture tokens, then “Question: {question} Short answer:”, and the model writes the answer. For few-shot, you put several such picture-and-answer pairs first and then the real question. Nothing new is trained at test time; the examples sit in the row.

So the whole system is a pipeline with three stages and two seams. The seam between the encoder and the connector is a list of 729 patch vectors. The seam between the connector and the language model is 144 image tokens. Each seam is a design choice, and each choice trades detail for cost.

A row holds 4,096 positions. A request has 8 images at 144 tokens each. How many positions are left for text?

Chapter 2

The Image Encoder: Teach the Eye Before the Brain

Compare two ways to train the encoder and read the paper’s Table 1 to see which knob moves the score most

Before the language model reads a single picture, the image encoder must already know how to look. Think of hiring a guide for a museum tour. A guide who has studied thousands of captioned paintings can say what each one is about. A guide who has only copied paintings stroke by stroke knows every detail but may not know the names. The paper tests both kinds of guide.

The encoder is trained before MM1 and is then plugged in. Table 1 compares encoders by plugging each into the same MM1 setup (with a 2.9 billion parameter language model, a little bigger than usual so it can use the larger encoders) and scoring the result. The Data column of that table tells you what the encoder itself was trained on, not what MM1 was trained on.

Way 1: contrastive training (CLIP)

A contrastive loss teaches by matching. Take a batch of N pictures, each with its own caption. The encoder turns each picture into a vector, and a text encoder turns each caption into a vector. A good model puts a picture and its own caption close together, and puts the picture far from the other N − 1 captions. (Our CLIP lesson builds this from zero.)

Write the closeness of picture i and caption j as a score sij (a dot product of the two vectors). Divide by a temperature τ, a small number that sharpens the contest. Then turn the row of scores for picture i into probabilities with a softmax and ask: how much probability lands on the true caption?

Li
The loss for picture i. Zero means all the probability sits on the true caption. Large means the model prefers a wrong caption.
esii/τ
The weight of the true pair. The diagonal of the score grid.
∑ esij/τ
The weight of all N captions, true and false together. It turns the weights into shares.

Here is a worked case with τ = 0.1 and N = 3 (toy numbers, illustrative). A well-trained row has scores 0.8, 0.3 and 0.1. Divide by 0.1 to get 8, 3 and 1. The weights are e8 = 2,981, e3 = 20.1 and e1 = 2.7, which sum to 3,004. The true caption gets 2,981 / 3,004 = 99.2%, so the loss is −log(0.992) = 0.0076. A confused row has scores 0.3, 0.4 and 0.1 for the true caption first. The logits are 3, 4 and 1, the weights 20.1, 54.6 and 2.7 sum to 77.4, the true share is 26.0% and the loss is 1.35. Training pushes every row toward the first case.

The paper’s main encoders are CLIP-style. They are trained on DFN-5B (5 billion image-text pairs filtered by a data filtering network, so only well-matched pairs remain) and sometimes on VeCap-300M (300 million pairs whose captions were rewritten by a model to be richer, called synthetic captions).

Way 2: reconstructive training (AIM)

A reconstructive loss teaches by rebuilding. Hide part of the picture and make the model predict it. The paper uses AIM, which cuts the picture into patches in order and asks the model to predict each patch from the ones before it, much like a language model predicts the next word. The motive is clear: a contrastive model only has to match whole pictures to captions, and the paper notes that CLIP-style models struggle on dense prediction (tasks that need detail everywhere). Captioning and question answering need detail. A reconstructive loss “explicitly captures all parts of an image”.

Reading Table 1

The table has ten encoder rows, with scores at 0, 4 and 8 shots, averaged over the benchmarks. The paper’s summary is the “encoder lesson”: image resolution has the highest impact, then model size, then training data composition. The device below lets you check each claim against the table’s own cells. Pick a comparison and read the three bars for each side.

Read Table 1

Each choice compares two encoders that differ in one thing. The hollow bars are the first encoder, the solid bars the second. Read the gap on each of the three tests.

All scores are cells of Table 1 (average over eight captioning and question-answering benchmarks, 2.9B language model). Resolution: ViT-L, DFN-5B+VeCap, 224 vs 336. Size: ViT-L vs ViT-H, 336, DFN-5B+VeCap. Data: ViT-H, 224, DFN-5B vs DFN-5B+VeCap. Loss: AIM 600M vs CLIP ViT-L, both 224. The paper summarises that last gap as 0.3 to 1.5 points; these two rows differ by 0.3, 2.1 and 1.5.

Resolution wins on every test, by 3.0, 3.7 and 3.8 points. Doubling the encoder from ViT-L to ViT-H adds only 0.6, 0.2 and 0.3. Synthetic captions help few-shot (3.0 and 2.2 points) and leave zero-shot unchanged at 37.5. And contrastive training beats the reconstructive AIM. The paper warns that the AIM result is not final: AIM had seen less than half of the training data, and only 20 of its 24 layers were used at inference. The open OpenAI CLIP encoder (ViT-L, 336) scores 39.3, 62.2 and 66.1, on par with the paper’s own encoder of the same size.

Why does resolution matter so much? A picture of a menu, a street sign or a chart holds its meaning in fine detail. At 224 pixels a letter is a few pixels wide. At 336 it is clearer. The hero at the top showed it: a coarse grid loses the thin line, a fine grid keeps it. This is the paper’s best single guide for a builder: if you have a fixed budget, spend it on resolution before you spend it on a bigger encoder.

The final recipe, therefore, uses a ViT-H trained with the CLIP loss on DFN-5B, at 378 by 378 pixels, the last row of Table 1 (40.9, 62.5 and 66.4). It was motivated, in the paper’s words, by the importance of resolution.

You can afford one change to the encoder. According to Table 1, which gives the largest gain on all three tests?

Chapter 3

The Connector: Three Ways to Plug the Eye Into the Brain

Open the box between encoder and language model, build three designs, and see why the count of tokens matters more than the design

You have 576 small descriptions of a photo, laid out in a 24 by 24 grid. The language model will read only 144 of them. Which 144 numbers should it get? You could average neighbours. You could let 144 “questions” each pick what they care about. Or you could blend neighbours with a small convolution first. These three ideas are the three connectors the paper tests.

The connector has two jobs, and the paper states both. First, the encoder gives a grid of vectors, but the language model reads a row, so the spatial layout must become a sequence. Second, the vectors live in the encoder’s number space, and the language model expects vectors that look like its own word embeddings, so they must be mapped across. And it must hit a target count: the paper tries 64 or 144 image tokens.

Design 1: average pooling

Average pooling splits the grid into equal blocks and replaces each block by the mean of its vectors. Then a linear projection (one matrix multiply) maps each mean into the language model’s space. The paper follows Emu2 and uses an output grid of n by n with n = 8 or 12. From a 24 by 24 grid, n = 12 means each token averages a 2 by 2 block of 4 patches, and n = 8 means a 3 by 3 block of 9. In shapes: [576, D] to [144, D] to [144, d]. The design is fixed, with no learned weights besides the projection. It cannot choose what to keep.

Design 2: attention pooling

Attention pooling lets the model choose. It keeps k learnable query vectors, one for each output token. Each query looks at all patch vectors, scores how relevant each is, and returns a weighted blend. This is cross-attention, the same mechanism a transformer uses to compare words. With patch vectors X of shape [P, D] and learned queries Q of shape [k, d]:

O
The k output tokens, shape [k, d]. These go to the language model.
Q
The k learnable queries. Each one is a trained “question” about the picture. They do not depend on the picture.
K
One key per patch, from the patch vectors. A query scores how well each key matches it.
V
One value per patch. The output is the score-weighted mix of values.

Say patch A scores 2.0 against a query and patch B scores 0.0, and P = 2. The weights are e2 / (e2 + 1) = 88% for A and 12% for B, so the output is 0.88 · VA + 0.12 · VB. Every output token can draw from every patch. That gives freedom, and it costs the layout: nothing says which patch is left of which, unless the vectors already carry position.

Design 3: the C-Abstractor

The C-Abstractor comes from Honeybee. It is a ResNet block (a small stack of convolutions, each of which mixes a patch with its neighbours, plus a shortcut that adds the input back) combined with adaptive pooling, which shrinks the grid to the target size. Because convolutions look at neighbours, each output token keeps local information about a small region of the picture. The Honeybee authors argued this beats the other two, and that is the claim the MM1 authors test.

Try each design on the 24 by 24 patch grid below. Pick an output token and see which patches feed it.

Trace one image token

The grid is the 576 patches of a 336 pixel picture. Blue shading shows how much each patch feeds the chosen output token. Choose a design and a token count, then drag the token number.

66 of 144

Illustrative. Block sizes for pooling are exact arithmetic (24 / 12 = 2, 24 / 8 = 3). The attention weights are drawn from a fixed random seed, because a trained model’s weights are not in the paper. The C-Abstractor footprint is drawn as a block plus a one-patch fringe to show local mixing; its true receptive field depends on its trained layers.

Notice what the picture teaches. Pooling is blind and even: every token covers the same size block. Attention pooling is flexible and scattered, with a few strong patches and a long faint tail. The C-Abstractor stays local but can look a little past its block. Each has a plausible story for why it should win.

The result: the plug hardly matters

Figure 4 of the paper compares them at two resolutions (224 and 336) and two token counts (64 and 144). The lesson is stated in one line: the number of visual tokens and the image resolution matter most, while the type of connector has little effect. Zero-shot and few-shot scores rise when you raise tokens or resolution. The different designs do not give conclusively different results, which the authors note is contrary to what Honeybee reported. After fine-tuning, all three designs end up very close at 336 pixels and 144 tokens (Appendix Figure 10).

That fits the story of Chapter 1. The budget k · m + t ≤ L is what bites. A connector can only pass on what the encoder already found, and the encoder’s detail comes from resolution. So the paper picks the C-Abstractor at 144 tokens, because the number of visual tokens is of highest importance and, in their words, the actual architecture seems to matter less.

The team wants more image detail for the same model. Based on Figure 4, which change is most likely to help?

Chapter 4

Three Kinds of Data, Three Different Skills

Compare captions, interleaved web pages and plain text, and see which skill each one builds

Open a recipe page. It has a title, a paragraph about walnuts, a photo of stuffed mushrooms, three more paragraphs, another photo and a list of steps. The photos and the words belong together, but the words do not describe each photo in one neat sentence. Now open a photo-sharing site. Each picture has a short label: “a dog running on a beach.” These are two very different kinds of data. A third kind has no pictures at all: a book chapter, a code file, a news article.

MM1 learns from all three. This chapter explains what each kind is, what it teaches, and what the paper found when it changed their shares.

Image-caption data

Caption data is a list of pairs: one picture and one short text that describes it. The text is short and closely related to the picture. The paper uses about 2 billion such pairs from public sets (CC3M, CC12M, HQIPT-204M, COYO) and a large internal web set. A second set, VeCap, has 300 million pairs whose captions were rewritten by a model to be richer. That is the synthetic data, and it is only about 7% of all caption data. As a training row it looks like: [144 image tokens][caption words].

Interleaved image-text data

Interleaved data is a whole web document, with pictures kept at the exact places where they appeared in the text. The paper builds one internal set of 500 million documents (1 billion images, 500 billion text tokens), following the OBELICS recipe, and the paper’s Table 2 lists 600 million documents in total. As a training row it looks like: [words][image][words][image][words], with up to 16 images. The text is far longer and more varied, and it is less tightly tied to any one picture.

The cleaning is worth noting, because the model can only be as good as its data. The authors start from 3 billion HTML pages. They keep the main text and the image tags, then drop any page with no images or with more than 30. They remove images that are corrupted, have an aspect ratio under 1:2 or over 2:1, are under 100 pixels or over 10,000 pixels, or whose web address contains words like logo, button, icon, plugin or widget. They remove images whose address or MD5 hash (a short fingerprint of the file) appears more than 10 times in the whole set, and when an image repeats on one page they keep only the first.

Text-only data

Text-only data is 2 trillion tokens of web pages, code, social media, books, encyclopedias and math. The paper’s text pipeline starts from 150 billion English HTML files, removes boilerplate (menus, footers), filters out documents that are too short, contain profanity or are low quality, and removes duplicates by exact hash and by locality-sensitive hashing (LSH), which finds near-copies. That leaves 3 billion pages. The text-only data is there to protect the language skills of the language model underneath.

Why the kinds teach different things

Think about how the model is tested. In a zero-shot test, you give one picture and one prompt, such as “A photo of”, and the model must write a caption. That is exactly what caption data trains. In a few-shot test, the prompt holds several examples first. For example: picture one, “Question: what colour is the bus? Short answer: red”, picture two, another question and answer, then picture three and a new question. This prompt is a row with several pictures and related text. It looks like a web page. So interleaved data teaches the model to use earlier pictures and words as context for what comes next.

The paper’s Data Lesson 1 puts numbers on this. As the share of caption data rises, zero-shot performance rises steadily from 25.8% to 39.3%. But for 4-shot and 8-shot tests, you need at least half the data to be interleaved to stay above 58% and 61%. Without any interleaved data, few-shot scores fall to 43.7% (4-shot) and 45% (8-shot). A note on fairness: 3 of the 8 test sets are captioning tasks, and the paper says this tilts zero-shot results toward caption data. Interleaved data still lifts few-shot scores on those same captioning tests, and it also helps text-only performance, since interleaved pages contain long text.

Turn the data dial

The dial sets the share of caption and interleaved image data. The bars show zero-, 4- and 8-shot scores. Solid bars are numbers the paper states. Dashed bars are only a bound or a range, because the paper states no single number there.

From Data Lesson 1 and Figure 5a: caption-only reads 39.3 zero-shot, 43.7 at 4-shot and 45 at 8-shot; all-interleaved reads 25.8 zero-shot; any mix with at least 50% interleaved keeps 4-shot above 58 and 8-shot above 61. The 50% zero-shot score is not stated as a single number, so it is drawn as a range between the two ends. These are averages over the paper’s eight benchmarks.

Read the picture as a trade. Caption data buys a good first answer. Interleaved data buys the ability to learn from the prompt. No single kind is enough, which sets up Chapter 5: how to mix them.

Why does interleaved data help few-shot prompts so much?

Chapter 5

The Mix and the Packing: 45, 45, 10, and a Mask That Stops Leaks

Work out why text-only data stays in the recipe, how the 5:5:1 ratio becomes 45, 45, 10, and how many short examples share one row without peeking at each other

A cook who adds only spice makes a hot dish and a bland one. A cook who adds only salt does the same. The right dish is a ratio. Chapter 4 showed what each kind of data teaches. This chapter asks how much of each to use, and how to serve it to the model in a form that does not waste time or leak.

Text-only data keeps the brain sharp

The language model under MM1 was first trained on text alone, so it is already good at language. If MM1 now trains only on pictures with short captions, it can slowly forget how to handle long text. Text-only data in the mix works like practice scales for a pianist who is also learning a new piece. The paper’s Data Lesson 2 says text-only data helps few-shot and text-only performance. Mixed with caption data, it lifts few-shot scores, because long text trains the model to use several examples as context. Mixed with interleaved data, it causes a small drop on the picture tasks, but in both cases the text-only score (the paper’s TextCore set of eight text benchmarks, such as ARC, HellaSwag and TriviaQA) goes up.

The ratio: 5, 5 and 1

Data Lesson 3 asks how much text-only data to add. The paper tries several ratios between image data (caption plus interleaved) and text-only data. It finds that a caption : interleaved : text ratio of 5 : 5 : 1 gives strong multimodal scores while keeping text scores comparable. To turn a ratio into shares, add the parts and divide:

pi
The share of training rows drawn from source i.
ri
The ratio part for source i: 5 for captions, 5 for interleaved, 1 for text.
∑ rj
The total of all ratio parts, here 5 + 5 + 1 = 11.

5 / 11 = 45.5%, 5 / 11 = 45.5%, 1 / 11 = 9.1%, rounded to 45%, 45%, 10%

That rounding is exactly the final recipe: 45% interleaved image-text documents, 45% image-text pairs and 10% text-only. Each training sequence is drawn from one of the three sources with these probabilities.

Synthetic captions: small share, large gain

Data Lesson 4 looks at VeCap. It is of higher quality but small, only about 7% of the caption data. Adding it gives a non-trivial boost to few-shot performance of 2.4% and 4% absolute. A tiny share of better data can pay more than a large share of average data. This echoes Chapter 2, where adding VeCap to the encoder’s training also lifted few-shot scores.

Packing: many short examples in one row

Training rows are 4,096 positions long. A caption pair is short, because caption text is brief. If you gave each example its own row, most positions would be padding that does no work. Packing places several examples one after another in a single row. For the paper, each row is sampled from one of the three sources, and image-text pairs or interleaved documents are packed along the sequence.

Packing has a hidden hazard. The decoder uses causal self-attention: each token may look at every earlier token in the row, so that it can predict the next one. If caption B follows caption A in the same row, the tokens of B can look at A. That is a leak. The words of B’s caption would be predicted with help from an unrelated picture. The fix is an attention mask: a grid of allowed and blocked pairs. Let doc(i) be the example that position i belongs to. Then:

Mij
Whether position i may attend to position j. 1 means allowed.
j ≤ i
The causal rule: look back only, never forward.
doc(i) = doc(j)
The packing rule: look only inside your own example. This is what the paper adds.

The paper says this change was critical for keeping strong few-shot performance when packing image-text pairs. Try it below on a toy row.

Pack a row, then block the leaks

A toy row holds three examples, drawn in three tints. Each cell shows whether the token on the left may attend to the token on top. Switch the document mask on and off, and move the cursor through the row.

9

Toy row of 16 positions, with example lengths 5, 6 and 5, illustrative. A real MM1 row has 4,096 positions. The mask rule is the paper’s (Appendix B.1: self-attention masks that prevent tokens from attending across example boundaries). Red cells are cross-example attention, which is a leak.

With the mask off, the 9th token (in the middle example) may read 9 tokens, and 5 of them belong to another example. With the mask on, it reads only 4, all from its own example. The cost of a mask is nothing in arithmetic, since masked cells are simply never used.

One fixed snapshot of the data

One more engineering detail. The authors perform the sampling and mixing once, offline, and store it as a fixed snapshot. Every model in the paper (except the mixture ablations themselves) trains on the same examples in the same order. They found this critical for reproducibility, because early tests showed that different random seeds in the input pipeline could change the resulting models by a non-negligible amount. If you want to compare two choices, you must give them identical data.

A row packs three captioned pictures. What does the document mask change?

Chapter 6

Scaling Up Without a Search: One Fitted Line and a Router

Derive the paper’s learning-rate rule, check it against Table 7, then see how a mixture of experts grows the model without growing its cost per token

Suppose you have found a good recipe at 1.2 billion parameters. Now you want 30 billion. Training one 30 billion parameter model takes a very large amount of compute, and you cannot afford to train it many times with different settings to find the best one. You need a rule that tells you the right setting before you start.

The setting that matters most is the learning rate, the size of each step the optimiser takes when it adjusts the weights. Too high and training blows up. Too low and it crawls. The best value depends on the model size: bigger models usually want smaller steps. The paper finds the rule from small models and stretches it to big ones.

Fit a line in log space

The authors run a grid search at four small sizes: 9 million, 85 million, 302 million and 1.2 billion non-embedding parameters (the weights in the transformer layers, not in the lookup table for words). At each size, they try many learning rates and keep the ones that reach close to the best 8-shot score. Plotted on log axes, those best learning rates fall roughly on a straight line (Figure 6). A straight line in log space is a power law, and the paper writes the fit as a linear regression of ln η on ln N:

η
The predicted best peak learning rate, the highest value in the schedule.
N
The number of non-embedding parameters of the language model.

Take the exponential of the right side and you get a power law, η = e−0.5535 · N−0.4214 = 0.575 · N−0.4214. Each time N grows ten times, η shrinks by a factor of 100.4214 = 2.64.

Check it for N = 3 × 1010. First, ln(3 × 1010) = 24.12. Second, 0.4214 × 24.12 = 10.17. Third, add 0.5535 to get 10.72. Fourth, η = e−10.72 = 2.2 × 10−5. That is exactly the value the paper uses for MM1-30B. The same arithmetic at N = 1.2 × 109 gives 8.6 × 10−5, matching Table 7.

One honest detail: the authors fit against the downstream 8-shot average, not the validation loss. They found, as in earlier work, that validation loss was not strongly correlated with downstream task performance. The rule extends the line from 1.2 billion to 30 billion, 25 times further than any point it was fitted on. Drag across the line below.

Extrapolate the learning rate

Both axes are logarithmic. The teal dots are the four sizes where the paper ran its grid search. The warm dots are the sizes the paper trained, using the fitted line. Drag the size to read the rule.

2.0B

The line is the paper’s Equation 1, drawn exactly. The dots on the line are computed from that equation; the paper’s own measured grid-search optima (Figure 6) are scattered around it and are not tabulated, so they are not plotted. Table 7 lists 8.6e-5, 5.9e-5, 4.2e-5 and 2.2e-5 for 1.2B, 2.9B, 6.4B and 30B.

The paper tried a similar search for weight decay (a small pull that shrinks the weights each step to keep them tame). In the end it found a simple rule that worked: weight decay equals 0.1 times the peak learning rate. (Table 7 lists predicted decay values of 5.0e-6 for 1.2B and 1.3e-6 for 30B, a little below 0.1 times the predicted learning rate; the text and the table do not fully agree, so treat the table as the record of what was run.) The learning-rate schedule is a cosine decay with a 2,000 step linear warm-up, falling to 10% of the peak over 200,000 steps. Gradients are clipped at norm 1, the optimiser is AdamW, and for the 30B model a z-loss term of 1e-4 helps training stay stable. All models train fully unfrozen (nothing is held fixed) at batch size 512 and sequence length 4,096, for about 400 billion tokens.

Mixture of experts: more parameters, same cost per token

The second way to scale is a mixture of experts (MoE). (See our mixture-of-experts lesson for the full build.) A transformer layer has a feed-forward block, the part that does most of the arithmetic. An MoE layer replaces that one block with E parallel blocks called experts, plus a small router. For each token, the router picks the best few experts, and only those run. The model has many more total parameters, but each token uses only a few of them. The paper’s phrase is: it scales the total number of parameters while keeping the activated parameters constant.

MM1 uses top-2 gating. The router scores all E experts for a token, keeps the two highest and mixes their outputs:

y
The layer’s output for one token.
∑ over top-2
Only the two chosen experts contribute. The other E − 2 are skipped.
ge
The gate weight, the router’s softmax score for expert e, rescaled so the two weights add to 1.

Two extra loss terms keep the router honest. A load-balance loss (coefficient 0.01) discourages sending every token to the same few experts, which would leave the rest idle. A router z-loss (coefficient 0.001) keeps the router’s scores from growing large, which stabilises training. The paper builds two such models. The 3B-MoE has 64 experts and replaces a dense layer with a sparse one every 2 layers, giving 64 billion parameters in total. The 7B-MoE has 32 experts, with a sparse layer every 4 layers, for 47 billion in total. Only the language decoder is converted. The encoder and connector stay as they were, and training reuses the dense model’s hyperparameters, data and number of tokens.

In the next chapter you will see that these MoE models beat their dense twins on almost every benchmark, which is the strongest sign in the paper that MoE is worth pursuing.

Why does the paper fit its learning-rate line on small models and extrapolate?

Chapter 7

Reading the Results: Pre-training Tables and Fine-tuning

Check which claims Table 3 supports cell by cell, then follow the model through supervised fine-tuning, high resolution and the comparison with rivals

A paper’s claim is only as good as its table. This chapter reads the two main tables the way a careful reviewer would: which cells carry the claim, which cells do not, and what the footnotes warn about. Table 3 covers the pre-trained models, the ones that have only done the long first stage of training. Table 4 covers the fine-tuned chat models.

Table 3: the pre-trained models, few-shot

Table 3 compares MM1 at three sizes (3B, 7B, 30B) with the few published pre-trained models that report few-shot scores, always against larger models. The columns are three captioning tasks (COCO, NoCaps, TextCaps) and four question-answering tasks (VQAv2, TextVQA, VizWiz, OKVQA). Higher is better in every column. Rows come in shots: 0, 8 and, at 30B, 16. The paper says MM1 beats all published pre-trained models in few-shot performance.

You can test that for yourself. The device below shows the paper’s 8-shot cells for MM1 and for the best rival model that reported each task. A dash in the paper means the rival never reported that task, so that column is left out of the comparison.

Check Table 3

Pick a model size. For each of the seven tasks, the solid bar is MM1 and the hollow bar is the best 8-shot score any rival reported. Bars are scaled per task.

All cells are the 8-shot rows of Table 3. Rivals: 3B, Flamingo-3B; 7B, IDEFICS-9B, Flamingo-9B and Emu2-14B; 30B, IDEFICS-80B, Flamingo-80B and Emu2-37B. The paper marks IDEFICS’s COCO score (an asterisk) because IDEFICS trained on data that includes COCO. Where no rival reported a task, no hollow bar is drawn.

At 30B, MM1 leads in six of seven tasks. The one exception is VizWiz at 8 shots, where Emu2-37B scores 54.7 against MM1’s 49.9. The paper’s own words are careful: on VQAv2, TextVQA and OKVQA, MM1 is “comparable” to Emu2. Look at the cells: TextVQA is 49.4 against 49.3, a tie in practice. At 16 shots MM1 reaches 57.9 on VizWiz against Emu2’s 57.0. Read the claim as “best or tied almost everywhere”, not “best everywhere”.

Two footnotes are worth keeping. Some rivals’ zero-shot prompt includes two text-only demonstrations, which MM1’s does not. And the paper says it does not hill-climb zero-shot numbers, because they mostly show how well the training mix matches the test format (Chapter 4). That is why this chapter reads the 8-shot rows.

Supervised fine-tuning: from base model to chat model

Supervised fine-tuning (SFT) is the second training stage. The pre-trained model gets instruction-following practice on a smaller, curated set of examples: a question, a picture, a good answer. The paper follows LLaVA-1.5 and LLaVA-NeXT and collects about 1.45 million examples from three groups. The first group is instruction-response pairs generated by GPT-4 and GPT-4V (conversations, complex reasoning, detailed descriptions). The second group is academic datasets turned into instruction format: natural-image question answering, text-rich images, and document and chart understanding. The third is text-only instruction data, so the model can still follow instructions without a picture. Both the image encoder and the language model stay unfrozen. The run is 10,000 steps at batch size 256, with the Adafactor optimiser.

Models after SFT carry the suffix “-Chat”. Figure 7c shows an important point about the first stage: when the same pre-training run is fine-tuned at earlier and later checkpoints, the final score keeps rising with more pre-training. Pre-training is not a formality. The lessons also transfer: the authors report that the connector type has a negligible impact after SFT, as it did before.

Going to high resolution

Chapter 2 showed that pixels matter. In SFT the paper pushes resolution higher with two tools. Positional embedding interpolation stretches the encoder’s position codes so it accepts a larger grid, then fine-tunes to adapt. This supports 448, 560 and 672 pixel inputs. A 672 pixel picture has 48 × 48 = 2,304 patches, and attention across more than 2,000 tokens is expensive. Sub-image decomposition (following SPHINX) avoids that: resize the picture to 1344 by 1344, cut it into four 672 pixel tiles, add a 672 pixel downsample of the whole picture as an overview, and feed all five as separate images. Five images at 144 tokens each is 720 tokens per image. With position interpolation on each tile, the setup reaches 1792 pixels.

Figure 7b gives the payoff. Compared with a 336 pixel baseline, 1344 by 1344 gives a 15% relative increase in the SFT average. At 1792 the average falls slightly, which the authors guess is due to resizing artefacts, since many test images are smaller than that. The default for Table 4 is 1344.

Table 4: the chat models against the field

Table 4 compares chat models on 12 benchmarks. The paper’s reading: on average MM1-3B-Chat and MM1-7B-Chat beat every listed model of the same size, with strength on VQAv2, TextVQA, ScienceQA, and the newer MMMU and MathVista. The two MoE models are better than their dense twins on almost every benchmark. At 30B, MM1-30B-Chat beats Emu2-Chat-37B and CogVLM-30B on TextVQA, SEED and MMMU, and is competitive with LLaVA-NeXT-34B. But there is a price difference. LLaVA-NeXT spends 2,880 tokens on each image and MM1 spends 720, so LLaVA-NeXT cannot easily take several images or few-shot prompts. This difference is the subject of Chapter 8.

MM1-30B scores 49.4 on TextVQA at 8 shots and Emu2-37B scores 49.3. What is the careful reading?

Chapter 8

In-Context Reasoning on a Token Budget

See why pre-training gives the model the power to learn from examples in its prompt, and how mixed resolution fits eight examples where four used to fit

You show a friend three photos of kitchen scales, each with the weight written next to it, then a fourth photo with no number, and say “your turn.” Your friend has never seen your task, yet answers correctly. They learned it from three examples, inside one conversation. That skill is in-context learning, and the paper’s opening figure shows MM1 doing it: counting objects, reading signs, naming furniture weights, adding numbers, all from a few worked pictures and a custom format.

Where the skill comes from

Chapter 4 showed that interleaved web pages look like few-shot prompts, so pre-training on them teaches the model to use earlier pictures and words as context. The paper’s strongest evidence is this: its fine-tuning data (Chapter 7) holds only single-image examples. Yet MM1-30B-Chat still reasons across several images. The ability was not taught in SFT. It was kept from pre-training.

Few-shot chain of thought

A chain of thought is a short written line of reasoning before the final answer. In a few-shot chain-of-thought prompt, each worked example shows the reasoning as well as the answer, and the model copies the style for the new question. Here is the shape of such a prompt, in the paper’s Figure 2 style:

prompt row
[img: bar photo] [img: menu]   # the picture tokens, 144 each
User: How much should I pay for all the beer on the table,
      according to the price on the menu?
Assistant: 12
User: Explain why.
Assistant: There are 2 beers on the table, and each beer
      costs 6 according to the price on the menu. So, 2*6=12.

The paper tests this on MathVista, a benchmark of maths questions about pictures. MM1-30B-Chat scores 39.4 with no examples. With four examples that include chain-of-thought text, it scores 41.9, which is 2.5 points higher. More examples help. The obstacle is space.

The token budget

The best MM1 chat model decomposes each picture into 5 sub-images (Chapter 7), and each sub-image costs m = 144 tokens. So one picture costs 5 × 144 = 720 tokens. K worked examples with one picture each cost K × 720. Four examples cost 2,880 tokens on pictures alone, before a single word of text. A fixed context fills up fast. The paper’s fix is mixed resolution in-context examples: only the last N examples get the full five-tile treatment, and the earlier K − N examples are encoded at low resolution, as one 144 token image each. The paper sets N = 3.

Tfull
Picture tokens when every example uses all five tiles.
Tmixed
Picture tokens when only the last N examples use five tiles.
N
How many of the latest examples keep high resolution. The paper uses 3.

K = 8, m = 144, N = 3: 3 × 720 + 5 × 144 = 2,160 + 720 = 2,880 tokens

That 2,880 is exactly what four full examples cost. So mixed resolution buys eight examples in the space of four. The paper reports the payoff: 8-shot chain of thought on MathVista reaches 44.4, up from 41.9 for 4-shot. The paper gives no reason for N = 3 and does not report other values, so treat 3 as a choice, not a law.

Fit more examples in the prompt

The bars count picture tokens only. Drag the number of in-context examples. The warm bar is full resolution, the teal bar is mixed resolution with the last 3 at full. The dashed line marks the cost of 4 full examples.

8

Token arithmetic from Appendix C.5: 144 tokens per tile, 5 tiles per full-resolution image, N = 3. It counts the example pictures only, not the question picture or any text. The MathVista scores (39.4 at 0 shots, 41.9 at 4 shots, 44.4 at 8 shots with mixed resolution) are the paper’s.

The failure mode: the prompt that does not fit

Here is the symptom. You build an 8-shot prompt with every picture at full resolution. The request fails, or the system silently cuts the first examples to make room. Your model then answers as if it had seen fewer examples, and its score falls back toward the 4-shot level.

Here is the measurement. Count the picture tokens before you send anything. Eight full examples cost 8 × 720 = 5,760 tokens, more than the whole 4,096-position row the model was pre-trained on. The paper does not state the chat model’s own limit, so the safe habit is to count first and compare with the limit you actually have.

Here is the fix. Send the earlier examples at low resolution, as the paper does. The count drops to 2,880, and no example is cut. The price is detail in those earlier pictures. If the early examples only show the pattern (the format of the question and the style of the reasoning), that price is small. If each example hides a fine detail the model must learn, keep it at full resolution and use fewer examples instead.

The same trade, one more time

This chapter is the budget of Chapter 1 returning. The row has fixed room. Detail per picture, the number of pictures, and the amount of text all compete for it. The paper’s whole line of results says: protect resolution first, then count tokens, and let the connector type fall where it may. When the budget gets tight, as with few-shot prompts, drop detail from the pictures that matter least rather than drop examples.

Compare it with a rival that spends 2,880 tokens on each picture. That model needs 2,880 tokens for one image, so a few-shot prompt of four examples would use more than 11,000. MM1’s smaller per-image cost is why it supports multi-image reasoning and few-shot prompting, as the paper points out.

With K = 6 examples, N = 3 and 144 tokens per tile, how many picture tokens does the mixed-resolution prompt use?

Chapter 9

The Field Guide: MM1 on One Page

Keep the numbers, the checklist and the core code, and see where the paper sits in the field and where to go next

MM1 is a recipe paper. A language model gets a picture encoder and a connector. The authors changed one part at a time at 1.2 billion parameters and learned four things. Pixels beat parameters in the encoder. The count of image tokens beats the connector’s design. A mix of captions, interleaved pages and text keeps both zero-shot and few-shot skills. And a learning-rate rule fitted on small models steers the big ones. They then scaled to 30 billion dense and 64 billion mixture of experts parameters, fine-tuned, and reached state-of-the-art few-shot pre-training scores and competitive chat scores.

The numbers to remember

PartThe paper’s valueWhere
Image encoderViT-H, CLIP loss, DFN-5B, 378 × 378 pixelsSection 4, Table 1
Patches per picture27 × 27 = 729 (patch size 14)Chapter 1 arithmetic
ConnectorC-Abstractor, 144 image tokensSection 4, Figure 4
Encoder lesson224 to 336 pixels adds about 3 points; ViT-L to ViT-H adds under 1; VeCap adds over 1 (few-shot)Section 3.2
Data mix45% interleaved, 45% image-text pairs, 10% text-only (5 : 5 : 1)Sections 3.3 and 4
Data lesson 1Zero-shot 25.8 to 39.3 as captions rise; at least 50% interleaved keeps 8-shot above 61Figure 5a
Synthetic captionsAbout 7% of caption data; +2.4 and +4 points few-shotData lesson 4
Training row4,096 positions, up to 16 images, 144 tokens each; batch 512; 200,000 steps, about 400B tokensSection 4, Appendix B.1
Learning rateη = exp(−0.4214 ln N − 0.5535); 2.2e-5 at 30BEquation 1, Table 7
Mixture of experts3B-MoE: 64 experts, 64B total; 7B-MoE: 32 experts, 47B total; top-2 gateSection 4
Fine-tuningAbout 1.45M examples; 10,000 steps; batch 256Section 5, Appendix B.2
High resolution1344 px = 4 tiles + overview = 720 tokens; +15% relative over 336 pxSection 5.1, Figure 7b
Few-shot chain of thoughtMathVista 39.4 (0-shot), 41.9 (4-shot), 44.4 (8-shot, mixed resolution)Section 5.1, Appendix C.5

The terms, in one line each

TermPlain meaning
MLLMA language model that also reads pictures and writes text.
Patch, image tokenA square of the picture, then one of the few vectors the language model reads for it.
ConnectorThe part that shrinks patch vectors to image tokens in the language model’s space.
Contrastive, reconstructiveTrain by matching pictures to captions, or by rebuilding the picture itself.
Interleaved dataWeb pages with pictures kept in place between paragraphs.
Few-shotThe prompt holds worked examples before the real question.
Packing and document maskSeveral examples share one row, and tokens may not look across examples.
SFTSupervised fine-tuning on curated instruction examples.

A checklist for your own multimodal recipe

The core, as code

python
import math, numpy as np

# 1. The row budget: k images of m tokens leave this many text positions.
def text_room(L=4096, k=16, m=144): return L - k * m   # 1792

# 2. Average-pool connector: [grid*grid, D] patches to [out*out, d] image tokens.
def avg_pool_connector(patches, grid, out, W):       # W: [D, d]
    D = patches.shape[1]
    x = patches.reshape(grid, grid, D)
    s = grid // out                                     # block side: 24 // 12 = 2
    x = x.reshape(out, s, out, s, D).mean(axis=(1, 3))   # [out, out, D]
    return x.reshape(out * out, D) @ W                # [out*out, d]

# 3. Document mask for packed rows: causal AND same example.
def doc_mask(doc_ids):                                # doc_ids: np.array [n]
    n = len(doc_ids)
    causal = np.tril(np.ones((n, n), dtype=bool))
    same = doc_ids[:, None] == doc_ids[None, :]
    return causal & same                                # True = may attend

# 4. Learning-rate rule (Equation 1) and the weight-decay rule from the text.
def peak_lr(n): return math.exp(-0.4214 * math.log(n) - 0.5535)  # 3e10 -> 2.2e-5
def weight_decay(n): return 0.1 * peak_lr(n)

# 5. Pick the source of each training row: 5 : 5 : 1.
SOURCES = ["caption", "interleaved", "text"]
SHARES = np.array([5, 5, 1]) / 11
def pick_source(rng): return rng.choice(SOURCES, p=SHARES)

# 6. Top-2 gate for one token: logits [E] to the two experts and their weights.
def top2_gate(logits):
    idx = np.argsort(logits)[-2:]
    w = np.exp(logits[idx] - logits[idx].max()); w /= w.sum()
    return idx, w

# 7. Picture tokens for K in-context examples, last N at full resolution.
def example_tokens(K, N=3, m=144, tiles=5):
    full = min(K, N)
    return full * tiles * m + (K - full) * m              # K=8 -> 2880

Limits and open questions

Read the lessons with their limits in mind. Most ablations run at one small size, 1.2 billion parameters, from one base configuration, so a choice that does not matter there may matter at 30 billion. The authors call the AIM comparison inconclusive, since AIM saw less than half the data. The two biggest datasets (Web Image-Text-1B and the internal interleaved set) and the in-house starting weights are not public, so the exact recipe cannot be rerun. The weight-decay numbers in the text and in Table 7 do not fully agree. The 1792 pixel drop is a guess, and N = 3 is untested. The paper also leaves the MoE models as future work, saying only that they show great potential. And scores are on benchmarks: the paper does not study safety, bias or hallucination.

Where it sits in the field

Before MM1, the paper names VILA, Emu2 and IDEFICS as the nearest studies. VILA studied pre-training components without optimisation details. Emu2 gave training details without ablations. IDEFICS set out to replicate Flamingo. MM1 aims to give both: the settings and the ablations. Its architecture family is the same as Kosmos-1, a decoder-only model reading visual tokens from an image encoder. In this course the paper pairs with the lecture on fusion models, where it serves as a worked case of fusion in one sequence: images become tokens and share a row with the words.

Keep going

The takeaway. A seeing model is a language model plus an eye. The eye needs pixels, the connector needs tokens, and the mix needs interleaved pages. Change one thing at a time, and let the results rank your effort.

Credit. This lesson teaches the paper “MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training” by Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Peter Grasch, Alexander Toshev, Yinfei Yang and their colleagues at Apple. Read it at arxiv.org/abs/2403.09611. The numbers here come from that paper; toy numbers are labelled.

Now press Present or Teach and explain from memory why pixels beat parameters, what the connector does, and how the data mix and the mask protect few-shot skill. Then go back to the dial at the top and spend one change again.

MM1: What Matters When a Language Model Learns to See
Read the paper · MIT MMAI course · Lecture 6 · Back to Gleams