A frozen multimodal model can steer an image painter. Hand it a few learnable queries, and the painter draws what the model worked out, not just the words you typed.
Learn how a small set of learnable queries turns a language model that only reads pictures into the brain behind a picture-making diffusion model, with no change to the language model at all.
Pick a prompt, then switch between Last layer embeddings and MetaQueries. Watch what the painter is given to read, and how the two benchmark scores move. Then we build, piece by piece, the queries, the connector, the one loss that trains them, and the evidence.
You need a rough idea of what a neural network is and that a language model reads tokens (small pieces of text). We build the rest from zero.
Based on Transfer between Modalities with MetaQueries by Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou and Saining Xie (Meta and New York University, 2025). A reading for Lecture 8, Large Multimodal Models of MIT MMAI, Multimodal AI. Every idea is re-taught here in our own words and devices.
The two score pairs are the paper’s Table 9, for the same Qwen2.5-VL 3B backbone and Sana-1.6B decoder: WiScore on WISE 0.48 against 0.55, and CommonsenseT2I 52.83 against 57.67. The three prompts come from the paper’s Figure 9. The words in the painter’s strip are a schematic: the real condition is a table of numbers, and the answer in quotes is what a person would say the model works out.
Chapter 0
See why one model that must both understand and paint is hard to train, and what a small bridge between two finished models changes
Type this into an image generator: “The national flag of the country where Yellowstone National Park is located.” You never named the country. A person reads the riddle, thinks “Yellowstone is in the United States”, and pictures the stars and stripes. Many image models cannot do that step. They paint a flag, but often the wrong one.
Meanwhile, a multimodal LLM (MLLM: a language model that can also read images) answers the same riddle in one line. It knows where Yellowstone is. So one specialist knows the answer and cannot paint. The other can paint and does not know the answer.
The paper splits the work by what comes out. Understanding means reading text and pictures and writing text: captions, answers, reasoning. Generation means writing pixels: a new picture. A unified model is one system that does both, so that what it understands can guide what it paints, and the reverse.
Today these jobs belong to two different kinds of network. An MLLM is autoregressive: it writes one token after another, each token chosen after reading all the earlier ones. A diffusion model makes a picture by starting from pure noise and removing a little noise at each step, until a clean image is left. Chapter 1 explains both from zero.
Most earlier unified models take an MLLM and keep training it so that it also produces images. Some make it regress image features. Some make it predict discrete image tokens. Some add a diffusion loss on top of the language loss. The paper lists a dozen such systems.
Every one of them changes the weights of the language model. That brings three costs, which the paper names directly. The architecture gets complicated. The data and the two losses must be balanced by hand. The training runs in several stages. Push too hard on painting, and the model gets worse at reading. Push too hard on reading, and the painting suffers.
The authors choose a different path, and they state it as a motto: render unto diffusion what is generative, and unto LLMs what is understanding. Keep a strong MLLM exactly as it is. Keep a strong diffusion model. Train only a small bridge that carries the first one’s thinking into the second.
The bridge is a set of learnable queries: a short list of new vectors that you attach to the end of the MLLM’s input. The MLLM reads them like extra words. What it computes at those positions is handed to the diffusion model as its instructions. That is the whole idea, and the rest of this lesson builds it step by step.
The frozen MLLM is the key. Frozen means its weights are never updated during training. Its understanding stays at the level of its original release, because nothing about it changes. And training needs only paired image and caption data and the ordinary loss of a diffusion model. The paper uses 25 million public image-caption pairs.
Here is the problem in its smallest form. Step through each riddle below, and count how many of its words name the thing to paint.
Choose a prompt from the paper’s Figure 9, then press Reveal the next step. The first box is all a plain text encoder ever sees. The last two boxes are the thinking that a language model can add.
The three prompts are the paper’s Figure 9. The middle and last boxes are what a person would answer, written by us to show the missing step; the paper reports that its model, MetaQuery-L, paints the right thing for these cases.
In every case the answer never appears in the prompt. The riddle holds a pointer to the answer, not the answer. To paint the right thing, something has to follow the pointer. That is a job for a model that has read the web, not for a painter that has only looked at pictures with captions.
By the end you can rebuild the method yourself: where the queries go, what shape they have, how the bridge turns them into something a painter can read, how one loss trains the whole thing while the language model stays still, and how to read each table in the paper. The numbers in the hero come from Table 9, which Chapter 7 reads in full.
Chapter 1
Learn how a language model reads in one direction and a diffusion model paints from noise, and find the one slot where they can meet
Imagine two colleagues in an office. One reads documents all day and writes short answers. The other has never read a document, but can turn a one-line brief into a painting. To get a painting from the reader’s knowledge, you need a way to pass a brief from the first desk to the second. This chapter looks at how each colleague works, so that the brief has the right shape.
A language model does not see letters. It sees tokens: small pieces of text, often a whole word. “The national flag of the country” is about six tokens. A multimodal model also turns an image into tokens, using a vision encoder (a network that cuts a picture into patches and describes each patch as a vector). Image tokens and text tokens then sit in one list.
Each token becomes a vector: a list of D numbers. In the paper’s smallest model, D is 896. The model passes the whole list of vectors through many layers. Every layer rewrites each vector using what the other tokens say. After the last layer, each position holds a hidden state: a vector of 896 numbers that summarizes what the model knows at that position.
So a prompt of T tokens becomes a table of hidden states with shape T by D. Keep that shape in mind. The whole method is about which rows of such a table you hand to the painter.
One rule decides what each row can know. The model is causal: when it builds the state for position i, it may read position i and the positions before it, never the ones after. This rule lets the model learn to predict the next token without peeking at it. The rule is enforced by a mask, a grid that blocks every look into the future.
Slide the reader below along a prompt. See how many tokens each position can read. Then add three empty slots at the end, and see who can read what.
Each row is a position that is reading. Filled cells are tokens it may read. Crossed cells are in its future. Drag the position, and switch on the extra slots.
Toy tokens: a real tokenizer cuts the prompt differently. The rule is the real one: in a causal transformer each position reads itself and the positions before it, and nothing after.
Read the grid from the bottom up. The first token knows only itself. The last token has read everything. Whatever you want a position to know about the whole prompt, put that position after the prompt. Chapter 2 does exactly that.
A diffusion model learns a simple trick. Take a real image and add a little noise. Add more, until only static is left. Then train a network to undo one small amount of noise at a time. To paint, start from static and apply the network again and again. A clean picture appears.
Left alone, the network paints something random. To steer it, you give it a condition: a short table of vectors that says what to paint. Inside the network, cross-attention lets the picture patches look at the condition. Each patch asks a question, the condition rows answer, and the patch updates. The condition has shape N by d, with N rows and d numbers per row.
In ordinary text-to-image models, a text encoder fills this slot. Stable Diffusion v1.5 uses CLIP, a text encoder that always gives 77 rows. Newer models such as Lumina-Next and Sana accept any number of rows, and Sana uses a language model for the job. That flexibility matters, because it lets us choose N freely in Chapter 6.
The condition slot is the one place where the two colleagues can meet. The obvious plan is to feed the painter the reader’s last layer embeddings: the final hidden states, one row for each prompt token. Several models do this already, among them Lumina-Next, Sana and Kosmos-G.
It works for plain prompts. But look at the mask again. Each row at a prompt position has read only the words before it. No position was asked to think about the whole prompt and answer it. The rows describe the question. They do not hold the answer. The paper adds a second complaint: this scheme does not fit tasks such as in-context learning or interleaved output, which a unified model should handle.
Chapter 2
Append a few trained vectors to the prompt, and read the model’s states at those positions as the brief for the painter
You hand a colleague a hard riddle. You also hand over a blank form with a few boxes and say: “When you have thought it through, fill in the form.” Then you pass only the filled form to the painter. The form is short and fixed. What goes in the boxes depends on the riddle.
That is the interface in this paper. The blank form is a table of learnable queries. They are called queries because each one asks the model a question, and the model’s state at that position is the reply. They are called learnable because nobody writes them: training finds them.
Take the input of the MLLM: the embeddings of the prompt tokens and of any image tokens. Call their number T. Now attach N new vectors at the end. The new vectors are the table Q, with shape N by D, where D is the width of the MLLM’s hidden states (896 in the smallest model). They start as random numbers and training adjusts them.
The MLLM then runs as usual, over T + N positions. It stays frozen, and it keeps its causal mask over the whole sequence. The paper says it chose causal masking even for the queries, for simplicity and compatibility, and did not switch on full attention for them. We take the hidden states of the last layer at the N final positions. That table is the condition C.
Why the end? Chapter 1 showed the reason. A position placed after the prompt has read every prompt token. Query number j reads all T prompt tokens and the j − 1 queries before it. The last query reads the entire sequence. Each query can therefore gather a different summary of the same prompt.
This point is easy to miss. Q is one fixed table, learned once. It does not change with the prompt. What changes is the model’s reading of the prompt, and so the states at the query positions. The riddle about Yellowstone and the riddle about the cat use identical queries and receive different conditions C.
The paper links this to a known idea: the frozen LLM acts as a feature resampler, a module that turns a long, uneven input into a short, fixed-size set of features. Flamingo used a small trained resampler for that job. Here the job is done by the big frozen model itself, and only the query table is new.
Try the sizes below. Move the prompt length and the number of queries, and watch how the sequence and the condition change.
The bar is the MLLM’s input: the prompt in blue, the queries in warm. Below it is the condition table C that the painter receives. Change T and N.
D = 896 is the hidden width of the paper’s Base model (Table 3 lists 896 for the connector, matching the 0.5B backbone). The N stops are the ones the paper tries in Figure 2: 1, 4, 16, 32, 64, 128, 256, 512, 1024. T is your choice, and the counts are exact arithmetic.
With 12 prompt tokens and 64 queries, the MLLM reads 76 positions. The condition C holds 64 × 896 = 57,344 numbers. The query table holds the same 57,344 numbers, and those are the only parameters touched in this step. Set N to 256, the value the paper uses for its final models, and the table has 229,376 numbers: a few hundred thousand values beside the hundreds of millions in the backbone.
pythonclass MetaQueries(nn.Module): def __init__(self, mllm, N=256, D=896): super().__init__(); self.mllm, self.N = mllm.eval().requires_grad_(False), N # frozen self.Q = nn.Parameter(torch.randn(N, D) * 0.02) # learnable [N, D] def forward(self, x_embeds): # [B, T, D] q = self.Q.expand(x_embeds.size(0), -1, -1) # [B, N, D] h = self.mllm(inputs_embeds=torch.cat([x_embeds, q], 1)).last_hidden_state # [B, T+N, D], causal return h[:, -self.N:, :] # C: [B, N, D]
The initialization scale 0.02 is our choice for the sketch. The paper says only that the queries are randomly initialized. Gradients still flow through the frozen model to reach Q, which Chapter 4 explains.
Here is a failure mode, with its symptom and its measurement. If you freeze the queries at their random starting values, the pictures still look good, because the painter makes good-looking pictures anyway. But they ignore the prompt. Table 1 measures this: with 64 random queries, the prompt-following score GenEval is 0.35, against 0.56 for 64 learned queries, while the visual-quality score FID barely moves (8.59 against 7.43). The fix is simply to train them. Chapter 5 returns to this table.
Chapter 3
Turn the query states into the exact kind of condition a diffusion model expects, and see why the order of two steps matters
Suppose you carry a letter from a French office to a German office. The writer meant every word, and the reader is smart. Still, the letter is in the wrong language. You need a translator. The frozen MLLM and the diffusion model have the same problem. Their numbers mean different things, and they have different widths.
The connector is that translator. It takes the table C from Chapter 2 and turns it into the condition that the painter’s cross-attention expects. It is the second, and last, new part in the paper’s architecture, after the queries.
The connector is a transformer encoder: a stack of layers, each of which lets every row of the table look at every other row and then rewrites it. The paper builds it with the same design as the Qwen2.5 language model, with one change. Its attention is bi-directional, so every row may read every row, with no causal mask.
That change repairs something. In the MLLM, query 1 could not see query 64, because of the causal mask. After the connector, every row has seen all the others. The N rows can agree on one coherent brief. A causal reader produced them, and a bi-directional reader tidies them up.
The connector also has to change the width. The MLLM’s states are D numbers wide (896 in the small model). The painter’s condition slot wants wider rows: Table 3 lists 2,304 for this setup. A linear projection (one matrix multiplication) does the width change. The open question is when to apply it.
The paper tests two designs. Proj-Enc (projection before encoder) first widens C to the painter’s width, then runs the transformer at that width. Enc-Proj (projection after encoder) runs the transformer at the MLLM’s own width, then widens at the end.
Cost explains why the order matters. The weights of a transformer layer grow with the square of its width. A layer at width 2,304 is about (2,304 ÷ 896)2 = 2.5712 ≈ 6.6 times as large as a layer at width 896. So a 24-layer encoder at width 896 should be about 6.6 times smaller than one at width 2,304.
316M × 6.61 ≈ 2,089M (Table 3 reports 2,046M for 24 layers at width 2,304; the small gap is the projection and rounding)
The measured parameter counts agree with the estimate. Now see what the money buys. The device below holds all four rows of Table 3.
Pick a design to see the order of its two steps. Pick a measure to compare all four designs. The chosen one is outlined.
Table 3 of the paper: LLaVA-OneVision 0.5B frozen, Sana-0.6B at 512 resolution, 64 queries, 25M image-caption pairs for 4 epochs. FID is on MJHQ-30K. Bars start at zero for parameters and GenEval, and at 6 for FID and 60 for DPG-Bench, as the label on each bar chart says, so small gaps stay visible.
Compare the two 24-layer designs. Enc-Proj has 316M parameters and a GenEval of 0.56. Proj-Enc has 2,046M parameters and a GenEval of 0.51. It costs 6.5 times as much and scores lower on prompt following. Its FID is the same to within 0.02 (7.41 against 7.43). So the paper keeps Enc-Proj with 24 layers.
Depth also helps, which you can see by comparing 6 and 24 layers within Enc-Proj. GenEval rises from 0.49 to 0.56, and DPG-Bench from 71.39 to 75.35. Relative wall time, measured against the smallest design as 1x, rises only to 1.06x.
Follow the shapes of the winning design with N = 256. C has shape 256 by 896. The 24 encoder layers keep that shape, because attention mixes the rows without changing their count or width. The final linear map takes each 896-number row to 2,304 numbers, which gives 256 by 2,304 = 589,824 numbers. That map has only 896 × 2,304 = 2,064,384 weights, tiny beside the encoder. The painter then builds the keys and values of its cross-attention from this table.
After the connector, you have a table with N rows at the painter’s width. You use it exactly where the text encoder’s output used to go. Nothing else inside the painter changes. The paper says the diffusion model can be arbitrary, as long as it has a conditional input interface. The authors use Stable Diffusion v1.5 and Sana.
One more choice remains: is the painter itself frozen or trained? Both are possible. The connector and the queries are always trained. The diffusion model may stay frozen or be tuned together with them. Chapter 5 compares the two choices.
Chapter 4
Follow one training step from caption to loss and back, and see how the queries learn through a model whose weights never move
Picture a classroom with one student, one translator and one painter. A teacher shows a photo and its caption, then hides the photo and asks the painter to redraw it. The painter gets the caption only through the translator. The reader, a tall professor in the corner, quietly rereads the caption each time. The professor never changes his mind. The others learn to use him better.
That is the training of MetaQueries. This chapter follows one step with every shape, and then the full recipe.
The form shown is the common one, where the painter predicts the noise. The paper says it keeps the objective of each diffusion model unchanged, so some painters predict a related target, such as a velocity. The point is that nothing new is invented. This is the loss you would use to fine-tune the painter alone.
Learning needs a gradient: for every number we may change, a signal that says which way to move it to shrink the loss. The loss sits at the end of the chain. The gradient flows backward, one stage at a time, in the reverse order of the forward pass.
Here is the subtle part. The MLLM is frozen, but it sits in the middle of the chain. The gradient must pass through it to reach the queries, which sit at its input. Passing a gradient through a layer does not change that layer’s weights. The MLLM acts as a fixed function, and only the numbers that feed it get nudged. The weights of the MLLM get no update, so its understanding stays as it was.
Step through one full round below. Switch the painter between trained and frozen, and read what changes.
Press Next stage nine times: four stages forward, the loss, then four stages backward. The warm outline marks stages that update. A blue lock marks the one that does not.
Parameter counts are approximate and use the paper’s ablation setup: queries 256 × 896 = 229,376 numbers; connector 316M (Table 3); painter Sana-0.6B (its nominal size); MLLM LLaVA-OneVision 0.5B (nominal). The ordering of the stages and which stage updates follow the paper’s description of its training.
Training has two stages, and both keep the MLLM frozen while training the queries, the connector and the diffusion model. Pre-training uses the 25 million image-caption pairs for 8 epochs, with learning rate 1e-4 and a global batch size of 4,096. The learning rate follows a cosine curve. It warms up for 4,000 steps, then decays to 1e-5. The second stage, instruction tuning, is Chapter 8.
A quick count shows the scale. One epoch is 25,000,000 ÷ 4,096 ≈ 6,104 steps. Eight epochs are about 48,800 steps, and the 4,000-step warm-up is about 8% of them. The ablations in Chapters 3, 5 and 6 are lighter: the same data for 4 epochs, with the Base backbone.
The paper trains three sizes, always with N = 256 queries and the 24-layer Enc-Proj connector. Base uses LLaVA-OneVision 0.5B. Large uses Qwen2.5-VL 3B. X-Large uses Qwen2.5-VL 7B. Painters are Stable Diffusion v1.5 and Sana-1.6B.
When you generate, the frozen MLLM and the queries run once for the prompt. You get C′ and keep it. The painter then runs its usual loop of denoising steps, and it reads the same C′ at every step. So the language model adds one forward pass to a loop that was already expensive.
Chapter 5
Read Tables 1 and 2 as two questions with two answers, and learn which differences are large enough to trust
A good design makes two promises. Here they are: you can use learned queries instead of the model’s own output states, and you lose little by leaving the MLLM frozen. The authors test each promise in a small table. We read both, and mark where the evidence is strong and where it is thin.
All the ablations use one setup. The MLLM is LLaVA-OneVision 0.5B, frozen. The painter is Sana-0.6B at 512 pixels. The queries number 64, the connector has 24 layers, and the data is the same 25 million pairs for 4 epochs. Three scores come out, and each one measures something different.
Table 1 compares four conditions. The first reads the MLLM’s last layer embeddings of the input tokens, as in Chapter 1. The second uses 64 queries that stay random. The third uses 64 learned queries. The fourth uses 512 learned queries, because the first option naturally gives a longer sequence.
Two readings come out. First, learning the queries is essential. Random queries give an acceptable FID of 8.59, because the painter makes decent pictures regardless. But GenEval falls to 0.35 and DPG-Bench to 54.81. The painter has stopped listening to the prompt.
Second, 64 learned queries match the last layer embeddings on two of three scores. FID is 7.43 against 7.49, and GenEval is 0.56 against 0.55. They lose on DPG-Bench, with 75.35 against 78.41. With 512 queries, learned queries win or tie on all three (7.34, 0.56, 78.43), by a hair on DPG. The paper’s word is “comparable”, and that is the fair word.
The comparison hides the real reason to prefer queries. Last layer embeddings only describe the prompt, as Chapter 1 showed. Queries can also ask the model to think. Chapter 7 measures that gain.
Table 2 crosses two choices: train the MLLM or not, and train the painter or not. Four rows result.
Start with the two rows that train the MLLM. Fully tuning it (with the painter frozen) gives FID 7.75, GenEval 0.58, DPG 78.97. Training the whole system end to end, MLLM and painter together, gives 6.28, 0.61 and 79.39. These are the “merged” recipes from Chapter 0.
Now the frozen rows. A frozen MLLM with a frozen painter gives 7.43, 0.56 and 75.35. Compare it with the matching row that tunes the MLLM. Its visual quality is a bit better (FID 7.43 against 7.75), and its prompt alignment is a bit lower (GenEval 0.56 against 0.58, DPG 75.35 against 78.97). The paper’s summary is exactly this: slightly lower alignment, slightly better quality.
Finally, add painter tuning to the frozen MLLM. The scores become FID 6.06, GenEval 0.61, DPG 76.66. GenEval ties the end-to-end row. FID is better than it (6.06 against 6.28). Only DPG lags, by 2.73 points. This row is the recipe the paper later scales up, and it keeps the MLLM exactly as released.
Choose a table and a score. The best row is green. Look for the row each table’s argument depends on.
Every number is from the paper’s Tables 1 and 2 (LLaVA-OneVision 0.5B, Sana-0.6B, 64 queries unless a row says 512). Bars start at 5 for FID, 0 for GenEval and 50 for DPG-Bench, so small gaps show. The paper reports one run per row with no error bars.
Not all differences are equal. Random queries against learned queries is a huge gap (0.35 against 0.56). A gap of 0.01 in GenEval, such as 0.55 against 0.56, is a gap you should not lean on. The paper shows one number for each setting, with no repeats and no error bars. We read the large gaps as evidence and the 0.01 gaps as ties.
Here are the gaps worked out. Table 1: learned against random queries is 0.56 − 0.35 = 0.21 in GenEval. Table 2: the tuned MLLM leads the frozen one by 78.97 − 75.35 = 3.62 DPG points, while the frozen one leads in FID by 7.75 − 7.43 = 0.32. Adding painter tuning to the frozen MLLM gains 7.43 − 6.06 = 1.37 FID points and 0.61 − 0.56 = 0.05 GenEval. The 0.21 and the 1.37 are big. The 0.32 and the 0.05 are modest, and one run each cannot say whether they would hold on a new seed.
Chapter 6
Learn why the length of the brief sets how much detail can pass, and how reconstruction becomes a second way to train the same bridge
Describe a photo in three words: “boat, sea, sky.” Now describe it in a paragraph. The paragraph can carry the colour of the hull, the angle of the light and the shape of the clouds. A longer brief holds more. In MetaQueries, the length of the brief is N, the number of queries. It is a free choice.
That was not always so. Earlier systems set N to whatever the image decoder demanded. Stable Diffusion v1.5 reads exactly 77 text rows from its CLIP encoder, so N had to be 77. Others picked 64 with no study. Newer painters such as Lumina-Next and Sana accept any number of rows, so the paper treats N as a knob and turns it from 1 to 1,024.
Figure 2a trains a text-to-image model at each N. The plot has four curves, and they do not all behave the same way. We read approximate values off the plot, so treat them as shapes, not exact numbers.
Figure 2b makes the capacity point directly, with a different task. The model must reconstruct an image it was shown. As N grows, reconstruction FID (rFID) falls from about 12 at N = 1 to about 3 by N = 64. PSNR and SSIM, which reward pixel-level fidelity, rise. LPIPS, which measures perceptual difference, falls. Figure 3 shows the effect in pictures: at N = 1 only a rough impression survives, and by N = 1,024 the output is close to the real image.
The paper settles on N = 256 for all its final models. It gives a good balance of quality and cost. More queries make the MLLM read more positions, and make the painter’s cross-attention look at more rows, so each extra query has a price.
You can feel the capacity effect in a toy. The device below cuts a small picture into one cell per query and rebuilds it from the cell averages. It is a stand-in for the real mechanism, not the paper’s model.
Left: the original toy scene. Right: the scene rebuilt from N cell averages. Below: fidelity in decibels for every N. Drag the number of queries.
Toy, not the paper’s model: a made-up 128 by 64 scene, averaged into a grid of N cells (1 by 1, 2 by 2, 4 by 4, 8 by 4, 8 by 8, 16 by 8, 16 by 16, 32 by 16, 32 by 32 for the nine N values). PSNR is computed live as 10 log10(255² / MSE). The paper’s Figure 2b shows the same direction on real reconstructions: PSNR and SSIM rise and LPIPS falls as N grows.
The curve in the device rises fast at first and then slowly. Doubling the cells from 1 to 4 helps a lot. Doubling from 512 to 1,024 helps a little. The paper’s curves show the same flattening for visual quality, with alignment still creeping up.
So far the data flow is caption in, picture out. Now change the input. Show the MLLM an image. Let the queries read it. Ask the painter to draw that same image. This is an image reconstruction objective, an autoencoder whose narrow middle is the N-row condition. The same architecture and the same frozen MLLM are used, and only the training data and target change.
Why care? It proves the queries can pass detail, not just topic. A caption says “a boat at sea.” Reconstruction asks for this boat, in this light. The paper reports that its fine-tuned Base model matches the best open model, Emu2, on reconstruction, and looks comparable to GPT-4o. It is the base for editing in Chapter 8.
Table 10 compares three ways to train the Base model.
| Training objective | Relative wall time | FID, lower is better | GenEval | DPG-Bench |
|---|---|---|---|---|
| Text-to-image | 1.0x | 7.43 | 0.56 | 75.35 |
| Image reconstruction | 2.79x | 27.42 | 0.32 | 68.36 |
| Mix of both | 2.61x | 8.27 | 0.54 | 76.53 |
Reading it takes three steps. First, reconstruction alone is a poor way to learn to generate: FID 27.42 and GenEval 0.32. The model learned to copy, not to follow words. Second, the mix keeps generation almost intact (8.27, 0.54, 76.53) and adds reconstruction. Third, the wall time is 2.6 to 2.8 times larger than text-to-image alone. The cost is real, and the paper pays it only when it needs reconstruction.
The cost side is simple arithmetic. With a 12-token prompt, 64 queries make the MLLM read 76 positions, and 256 queries make it read 268. That is 268 ÷ 76 ≈ 3.5 times as many positions in the language model. The painter’s cross-attention also sees 256 condition rows instead of 64, four times as many keys and values. Quality that flattens at 64 while cost keeps climbing is why a middle value such as 256 is a sensible stopping point, not 1,024.
Chapter 7
Check three claims against the tables: understanding is kept, generation is strong, and the model’s knowledge reaches the picture
A method is only as good as its evidence. The paper makes three claims, and each deserves its own test. First: freezing the MLLM keeps its understanding intact. Second: the pictures are good. Third, the one that matters most: what the MLLM knows ends up in the picture. We take them in order.
This claim needs almost no experiment. The MLLM’s weights are never touched, so its answers on reading tests cannot change. The understanding columns of Table 4 are simply the backbone’s own scores. The three sizes use LLaVA-OneVision 0.5B, Qwen2.5-VL 3B and Qwen2.5-VL 7B. Their scores on five benchmarks are in the device below.
The five benchmarks test different skills. MME-P tests perception, MMBench general questions about images, SEED-Bench comprehension, MMMU college-level reasoning across subjects and MM-Vet integrated skills. You do not need to learn them. What matters is that the table says “these systems read images at the level of a state-of-the-art MLLM”, and the bridge cost nothing.
The generation columns come from the same three models. MJHQ-30K FID improves with size, from 6.28 (Base) to 6.35 (Large) to 6.02 (X-Large), which is not monotone, so we do not over-read it. GenEval rises 0.74, 0.78, 0.80 and DPG-Bench rises 80.04, 81.10, 82.05. On COCO FID, measured with Stable Diffusion v1.5 as the painter, the X-Large model reaches 8.69. The paper says this beats other unified models built on that painter, MetaMorph (11.8) and Emu (11.66).
Pick a model size. The top five bars are reading scores of the frozen backbone. The lower four are generation scores of the bridge on top of it.
All values are from the paper’s Table 4. Bars are scaled to each score’s own range: MME-P to 2,000, other reading scores to 100, FIDs to 12 (lower is better), GenEval to 1 and DPG-Bench to 100. COCO FID uses Stable Diffusion v1.5 as the painter; the other generation scores use Sana. GenEval entries marked with a dagger in the paper use rewritten prompts.
There is one honest gap. Janus-Pro, which writes image tokens one by one, scores higher than MetaQuery on DPG-Bench (84.19 against 82.05 for the X-Large model) and, with the same prompts, on GenEval too (MetaQuery’s 0.80 uses rewritten prompts). The paper argues that these two tests may not capture the artifacts that autoregressive generators show, and says MetaQuery is clearly better than Janus-Pro on visual quality (FID) and on world knowledge. That is an argument, not a measurement of the gap, so we hold it lightly.
The cleanest test in the paper is Table 9. It holds everything fixed (the Large model, the same data, the same painter) and changes only the interface. One run reads last layer embeddings. The other uses MetaQueries.
On the scores that measure plain picture-making, the two runs tie: FID 6.41 against 6.35, GenEval 0.78 against 0.78, DPG-Bench 81.23 against 81.10. On the scores that need knowledge, queries pull ahead: WiScore on WISE 0.55 against 0.48, and CommonsenseT2I 57.67 against 52.83. That is the hero’s result, and it is what the design promised: a model that reasons about the prompt first.
The reason, in the paper’s words, is in-context learning. A text encoder maps words to vectors. The queries sit after the question, so the MLLM reads the question, works toward its answer, and puts the result in the query states. With the last layer embeddings, no position was asked to answer.
Table 8 adds a useful control. It swaps the Large backbone among Qwen2.5-3B, its instruction-tuned version and Qwen2.5-VL-3B-Instruct. All three give about the same generation scores (GenEval 0.78 to 0.79, FID 6.20 to 6.36), and CommonsenseT2I stays between 54.33 and 57.67. The paper concludes that generation ability is mostly independent of how much better a backbone understands.
Choose a benchmark. MetaQuery rows are warm. Other models are grey. Find the best MetaQuery row and the best row from any other system.
From the paper’s Tables 6 and 7 (a selection of rows). GPT-4o on WISE was evaluated by other authors on a random 200 of the 1,000 test items, so it is not strictly comparable. CommonsenseT2I is scored with CLIP as the evaluator, as in its original code. DALL-E 3 has no result with a negative prompt.
Read the ranking with care. MetaQuery-L and X-Large reach 0.55 on WISE, above the best text-to-image model listed (FLUX.1-dev, 0.50) and far above every other unified model. GPT-4o is at 0.89 on a smaller subset, so the gap to the strongest closed system is large. The paper’s own explanation is that closing it may need more data. Going from Large to X-Large adds nothing on WISE (0.55 to 0.55), so size alone is not the lever. On CommonsenseT2I without a negative prompt, DALL-E 3 with prompt rewriting (40.17) beats MetaQuery-L (28.83), so the lead there holds only with a negative prompt.
Put numbers on the Table 9 gain. WiScore rises by 0.55 − 0.48 = 0.07, which is 0.07 ÷ 0.48 ≈ 15% of the starting score. CommonsenseT2I rises by 57.67 − 52.83 = 4.84 points, about 9% of its start. Meanwhile FID moves by 0.06 and GenEval by 0.00. A change that appears only where knowledge is needed is the signature of a real effect, not of general noise, although the paper again reports a single run.
Chapter 8
Mine training examples from the web, teach the bridge to follow edit instructions, and read what that unlocks for editing and subject-driven pictures
You want to say: “Put this toy on the cat’s head.” To follow that, a model must see the toy, see the cat, read your sentence and paint a new picture that mixes them. Training for this needs examples of the form source images, an instruction, a target image. Where do you find millions of them?
Most earlier work builds these examples with expert models. An editing model, a style model or an object-replacement model turns each source image into a target. The paper names two problems. It is hard to scale, because each expert covers only a narrow set of changes. It may also bring in biases, because the targets carry the expert’s habits. A model trained on such data learns the experts’ small set of tricks.
The web is full of pages that mix text and pictures about one subject: the same car in three photos, the same building from two sides. These pictures are linked by meaning, with no expert needed. The paper, inspired by MagicLens, turns this into data in four steps (its Figure 4).
The result is 2.4 million training examples. The prompt for the instruction writer (the paper’s Appendix A) asks for two things. One is a general, unspecific similarity with the sources, such as “same jersey top”. The other is every difference that only the target has. It forbids details that would let you draw the target without looking at the sources. That rule matters: it forces the model to use the source images.
One loose end: the main text says the instruction writer is Qwen2.5-VL 3B, while Appendix A names Qwen2-VL-7B-Instruct. The paper does not explain the difference. We report both.
Try the grouping step on a toy. The dots are pictures placed by how similar their captions are. Change the threshold and watch which pictures group together and which one is chosen as the target.
Each dot is a picture; close dots have similar captions. Same-coloured dots form a group of up to 6. The ringed dot is the target, and lines run from the sources to it. Raise the threshold to demand closer captions.
Toy: 40 made-up dots from a seeded random draw, with similarity defined as one minus the distance divided by 0.6. The paper uses SigLIP caption embeddings with a threshold of 0.5, a cap of 6 images per group, and the lowest average similarity rule for the target. The grouping here is a simple greedy pass, which the paper does not specify.
Move the threshold up. Groups shrink and more dots stay alone, so fewer examples come out. Move it down. Groups grow to the cap of 6, but their members drift apart in meaning, and the instruction writer has less in common to point to. The paper settles on 0.5 on its own scale.
In this stage the sources enter the MLLM as image tokens, next to the instruction text. The queries sit after them, as before. The frozen MLLM reads the sources and the instruction, and the query states carry the result to the painter. The painter has to produce the target. The recipe repeats the one from Chapter 4: only the queries, the connector and the painter train, for 3 epochs, on the Base model, with batch size 2,048. The count is 2.4M ÷ 2,048 ≈ 1,172 steps per epoch, or about 3,500 steps in all.
Editing is cheaper. Start from the Base model that learned reconstruction (Chapter 6). Keep the MLLM frozen and fine-tune for only 1,000 steps on public editing data. The paper shows edits such as adding a chef hat to a dog, removing a sign, changing the style to line art and replacing fries with salad (its Figure 7). The reconstruction skill already carries the detail, so the new training only teaches “apply this change.”
In subject-driven generation you give one or more photos of a subject and ask for it in a new scene. DreamBench scores it with three numbers. DINO and CLIP-I measure how much the output looks like the subject’s reference photos. CLIP-T measures how well the output matches the text. Higher is better for all three.
| Method | DINO | CLIP-I | CLIP-T |
|---|---|---|---|
| Real images (oracle) | 0.774 | 0.885 | none |
| DreamBooth (tunes per subject) | 0.668 | 0.803 | 0.305 |
| Kosmos-G (zero-shot) | 0.694 | 0.847 | 0.287 |
| BLIP-Diffusion (zero-shot) | 0.594 | 0.779 | 0.300 |
| MetaQuery-B-Instruct (zero-shot) | 0.737 | 0.852 | 0.301 |
Read it in three steps. Zero-shot means no tuning per subject at test time. MetaQuery-B-Instruct leads the zero-shot rows on DINO (0.737 against Kosmos-G’s 0.694) and on CLIP-I (0.852 against 0.847). It also beats per-subject DreamBooth on both image-similarity scores. DreamBooth stays a little ahead on text match (0.305 against 0.301). Kosmos-G was trained on tasks built just for this problem, and MetaQuery was not.
The paper also shows pictures that go past copy and paste: from a photo of a Porsche 911 it identifies the model and draws a new front view, and from a photo of Rockefeller Center it imagines the skyline seen from the top. These are shown as examples, not scored, so we count them as promising, not proven.
Chapter 9
Keep the whole paper on one page: the numbers, a checklist, the core code, the limits and where to read next
You can now rebuild MetaQueries from memory. Two finished models stay as they are. A short table of learned queries rides at the end of the MLLM’s input. A small connector translates the states at those positions. A diffusion decoder paints from them. One ordinary denoising loss trains the pieces, and the gradient crosses the frozen MLLM to reach the queries.
MetaQueries connect a frozen MLLM to a diffusion decoder. N learnable queries (256 in the final models) are appended to the MLLM’s input, the last-layer states at those positions form the condition C, and a bi-directional transformer connector (Enc-Proj, 24 layers) maps C to the decoder’s input space. Training uses 25M public image-caption pairs and the decoder’s own denoising objective. The MLLM never changes, so its understanding is kept, and its knowledge reaches the picture: against last layer embeddings, WiScore rises from 0.48 to 0.55 and CommonsenseT2I from 52.83 to 57.67 with picture quality tied. Instruction tuning on 2.4M mined natural image pairs adds subject-driven generation, and a short fine-tune adds editing.
| What | Number | Why it matters |
|---|---|---|
| Pre-training data | 25M public image-caption pairs | No special data, only captions |
| Recipe | 8 epochs, lr 1e-4 to 1e-5, batch 4,096, 4,000 warm-up steps | About 48,800 steps in all (our arithmetic) |
| Final queries and connector | N = 256; Enc-Proj, 24 layers, 316M | Cheaper and better than Proj-Enc at 2,046M |
| Random against learned queries | GenEval 0.35 against 0.56 | Learning the queries is essential |
| Frozen against tuned MLLM (Table 2) | FID 7.43 against 7.75; GenEval 0.56 against 0.58 | Comparable, with a small alignment cost |
| Frozen MLLM, painter tuned | FID 6.06, GenEval 0.61, DPG 76.66 | Ties end to end on GenEval |
| Queries against embeddings (Table 9) | WiScore 0.55 against 0.48; CommonsenseT2I 57.67 against 52.83 | The knowledge gain, at equal quality |
| WISE overall | MetaQuery-L 0.55; FLUX.1-dev 0.50; Janus-Pro-7B 0.35 | Best listed open system |
| COCO FID with SD v1.5 | 8.69 against MetaMorph 11.8, Emu 11.66 | Best unified model on that painter |
| Reconstruction mix (Table 10) | FID 8.27, GenEval 0.54, 2.61x wall time | Adds reconstruction at a real cost |
| Instruction-tuning data | 2.4M pairs; groups of up to 6; threshold 0.5 | Natural pairs, no expert models |
| Editing fine-tune | 1,000 steps | Reconstruction transfers to editing |
| DreamBench zero-shot | DINO 0.737, CLIP-I 0.852, CLIP-T 0.301 | Ahead of Kosmos-G on the two image scores |
pythondef train_step(batch, mq, connector, dit, vae, opt): x_embeds = mq.mllm.embed(batch["caption_ids"]) # [B, T, D] C = mq(x_embeds) # [B, N, D] frozen MLLM + queries Cp = connector(C) # [B, N, d] encoder at width D, then linear to d x0 = vae.encode(batch["image"]) # [B, c, h, w] image latent t, eps = torch.rand(x0.size(0)), torch.randn_like(x0) xt = add_noise(x0, eps, t) loss = F.mse_loss(dit(xt, t, cond=Cp), target(x0, eps, t)) # the decoder's own objective loss.backward() # gradient crosses the frozen MLLM to reach mq.Q opt.step(); opt.zero_grad() # opt holds mq.Q, connector, dit only @torch.no_grad() def generate(prompt_ids, mq, connector, dit, vae, steps=20): Cp = connector(mq(mq.mllm.embed(prompt_ids))) # one MLLM pass for the whole run x = torch.randn(1, *LATENT_SHAPE) for t in schedule(steps): x = dit.denoise_step(x, t, cond=Cp) return vae.decode(x)
This is a sketch. add_noise, target and schedule stand for whatever the chosen decoder uses, and the paper keeps each decoder’s own objective unchanged.
This paper is a reading for Lecture 8, Large Multimodal Models. It takes the lecture’s picture of an MLLM that reads images and asks what must be added so that it can also drive image generation. Lecture 9 on multimodal generation picks up the generation side. The opposite design, one transformer for both jobs, is Transfusion, which trains the language model to denoise as well.
Now press Present or Teach and explain the bridge out loud, from memory: why the queries sit at the end, what the connector translates, how the gradient reaches the queries through a frozen model, and what Table 9 proves. Then go back to the hero and flip between the two ways of giving the painter its brief.
Credit: all results, tables and figures described here are from Pan, Shukla, Singh, Zhao, Mishra, Wang, Xu, Chen, Li, Juefei-Xu, Hou and Xie, “Transfer between Modalities with MetaQueries” (Meta and New York University, 2025), arXiv 2504.06256; project page xichenpan.com/metaquery. The explanations, devices and toy examples are our own, and every toy is labelled.