InternVL 1.5 cuts a big page into 448 pixel tiles, so an open 26 billion parameter model can read small print. Cut the page below and watch the words come back.
A paper reading for MIT MMAI: how an open model closes much of the gap to GPT-4V on documents, charts and Chinese, with three simple changes.
Pick a tile grid for the same 800 × 1300 page. Watch the letters, the token count and the verdict change. Then we build, piece by piece, the tiling, the shuffle, the encoder, the training and the data that make it work.
You need a feel for what a neural network is and what a pixel is. We build the rest from zero: patches, tokens, projectors and benchmarks.
Paper: Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye and colleagues at Shanghai AI Laboratory, SenseTime, Tsinghua University and others, “How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites” (arXiv 2404.16821, 2024). This paper is a reading of MMAI Lecture 5.1, Large Multimodal Models, from the MIT MMAI course (Paul Liang, MIT Multisensory Intelligence group). Every idea is re-taught in our own words and devices.
Toy page: 800 × 1300 pixels with 12 pixel body text, the paper’s own example size (Figure 4). Each tile costs 256 visual tokens and the thumbnail adds one more view when there is more than one tile (Section 3.2). The rule “letters under 8 pixels smear” is illustrative, not from the paper.
Chapter 0
See the three gaps that kept open models behind GPT-4V, and the three fixes the paper makes
You photograph a shop sign on holiday and ask an AI model: what does it say, and when does the shop open? One model answers in a second, line by line, with the opening hours right. Another says the sign “contains some text” and guesses.
That gap is the subject of this paper. A multimodal large language model (MLLM) is a language model that can also take pictures as input. Think of a very well-read friend who has just been given eyes. In early 2024 the best eyes belonged to closed, commercial models such as GPT-4V, Gemini and Claude-3. Models whose weights anyone could download saw much worse, especially on text inside images.
The authors, a team at Shanghai AI Laboratory and partners, ask the question in their title: how far are we to GPT-4V? Their answer is a model called InternVL 1.5. It is open, it has 26 billion parameters (the learned numbers, or dials, that make up a network), and on several reading tests it matches or beats the closed models.
The introduction says the open models fall behind in three ways. The first is parameter scale. The paper says the closed models usually have at least 100 billion parameters. A typical open model pairs a 300 million parameter vision foundation model (VFM: the network that turns pixels into features) with a 7 or 13 billion parameter language model.
The second is image resolution. Closed models, the paper says, use a dynamic resolution that keeps the original shape of the picture. Open models train at one fixed size, such as 336 × 336 or 448 × 448 pixels. Every picture, tall or wide, small print or not, is squashed into that one square.
The third is multilingual ability. Closed models train on data in many languages. Open models mostly train on English and hope the language model carries the skill over to other languages without being taught, a trick called zero-shot transfer. The paper says this leaves them weak at reading Chinese signs and Chinese documents.
Each gap gets one fix, and the whole paper is the story of these three. A stronger vision encoder: keep training the 6 billion parameter InternViT-6B on better data, so that it can be reused with different language models. Dynamic high resolution: cut each picture into 1 to 12 tiles of 448 × 448 pixels during training, and up to 40 tiles at test time, which the paper says reaches 4K resolution. A high-quality bilingual dataset: collect English and Chinese data on scenes, charts and documents, and translate more with an open language model.
Nothing here is a new kind of network. The authors call the changes “three simple improvements”. That is the lesson for a builder: careful engineering of the input, the encoder and the data can close a gap that looks like it needs a bigger model.
Before the machinery, look at the claim. The paper’s Figure 1 compares InternVL 1.5 with four closed models on seven tests. The tests are called benchmarks: fixed sets of questions with known answers, so that scores can be compared. Pick one below.
Each bar is a percentage score from the paper’s Figure 1. The warm bar is InternVL 1.5. Choose a benchmark and see who leads.
All scores are copied from Figure 1 of the paper (they match Table 2). A missing bar means the paper’s table has a dash for that model. Scores from different test sets are not comparable with each other, only across the five bars of one benchmark.
Look at the pattern as you click through. The warm bar leads on documents (DocVQA), charts (ChartQA), scene text (TextVQA) and math pictures (MathVista). It trails on the broad knowledge test MMMU and on science diagrams (AI2D). The paper’s own summary is that it wins four of these seven. The reading tests are where the three fixes should help most, and that is where it wins.
Three of the seven benchmarks above (DocVQA, ChartQA, TextVQA) test OCR, optical character recognition, which means reading the text inside a picture. Almost every choice in this paper is a way to make the model better at OCR without losing its general skills. Keep that in mind and the design choices stop looking arbitrary.
Chapter 1
Follow one image through the vision encoder, the projector and the language model, with shapes
A photo goes in. A sentence comes out. In between, three separate pieces of software do three different jobs. This chapter walks one image through all three, with the shape of the data at every step. A shape is just the size of each dimension of a table of numbers, for example 1024 × d means 1024 rows and d columns.
Think of a person reading a menu aloud to a friend. Their eyes turn light into a set of features. A translator in the brain puts those features into the form that the language area expects. The language area then writes the reply. The paper builds the same chain and calls it ViT-MLP-LLM, after its three parts.
A computer sees a picture as a grid of numbers. A colour picture of 448 × 448 pixels is a table of shape 448 × 448 × 3, one number each for red, green and blue. The first part, a vision transformer (ViT), cuts that grid into small square patches and treats each patch like a word in a sentence.
The paper does not state the patch size, but the numbers fix it. A 448 pixel tile ends up as 256 visual tokens after a step that cuts the count to one quarter, so the encoder must output 1024 patches first. That is 32 × 32, and 448 divided by 32 is 14 pixels per patch.
Each patch becomes a vector of dv numbers (the paper does not give dv in this report, so we keep the symbol). Then layers of self-attention let every patch look at every other patch and update itself. Out comes a table of shape 1024 × dv: one feature vector per patch. The encoder here is InternViT-6B, a vision transformer with 6 billion parameters (Chapter 4).
The language model cannot read a table of image features. It reads vectors of its own width, dl. The MLP projector is a small network that maps each image feature into that width. MLP means multilayer perceptron: a few stacked layers of weighted sums with a nonlinearity between them.
Before the projector, the paper applies pixel shuffle (Chapter 3). It glues each 2 × 2 block of neighbouring patch vectors into one longer vector. The table becomes 256 × 4dv. The projector turns it into 256 × dl. These 256 vectors are the visual tokens of one tile: image content in the same form as word embeddings.
This is the only part that starts from nothing. The paper says the projector is randomly initialized, while the encoder and the language model are both already trained. It must learn from scratch to translate between two spaces that have never met.
The language model here is InternLM2-20B, with 20 billion parameters. The paper builds on the chat version, which is already tuned to follow instructions, not on the bare base model. It receives one long sequence. In the usual layout, and in our sketch, the visual tokens come first and the tokens of the user’s question follow. It writes the answer one token at a time, as any language model does.
Take a real request. The user sends the 800 × 1300 sign from Chapter 0 and asks “When does the shop open?” The image becomes 7 views of 256 tokens, 1,792 visual tokens in all (Chapter 2 shows why 7). The question adds maybe 10 text tokens. The model receives about 1,800 vectors, each of width dl, and returns a probability for every possible next token. An illustrative reply might begin: “It looks like a bar, open from 16:30 to 22:30” (hours like these appear in one of the paper’s own example answers). The paper reports training with a context length of 4,096 tokens, so this request uses less than half of that.
Each row is one stage. Choose a stage to light the path up to it. The right edge of each row is the shape of the data after that stage.
Shapes follow the paper: a 448 × 448 tile gives 256 visual tokens after the 4 to 1 pixel shuffle (Section 3.1). The 1024 patches and the 14 pixel patch are derived from that, not quoted. dv and dl are the widths of the vision encoder and the language model; the report does not list them.
Suppose the three parts work but the picture is a dense document squeezed to one 448 × 448 square. The symptom: the model writes a fluent description (“a price list with five items”) but copies the prices wrong. Nothing crashes. The words just come from a blur.
The measurement is a document benchmark. The paper’s earlier model, InternVL 1.2, which used one fixed 448 × 448 view, scores 57.7 on DocVQA. InternVL 1.5 uses the same three-part chain and scores 90.9. The fix is not a new part. It is a better way to feed the first part: tiles. That is the next chapter.
Chapter 2
Match any picture to a grid of 448 pixel tiles and count the visual tokens it costs
You hold a long till receipt next to a wide slide and a square photo. A model that wants every input to be 448 × 448 must stretch the receipt wide and squash the slide tall. Letters become the wrong shape. Faint print disappears.
The paper’s answer is old and simple: do what a person does with a big page. Cut it into pieces, read each piece, and keep one small picture of the whole page so that you do not lose your place. This is dynamic high resolution. The paper says it was inspired by an earlier system called UReader.
A tile is a 448 × 448 pixel square. The picture will be resized to a grid of tiles with cols columns and rows rows, so that it is cols × 448 pixels wide and rows × 448 pixels tall. Which grid? The one whose shape is closest to the picture’s own shape.
The paper makes a list of allowed grids. With at most 12 tiles in training, it keeps every pair (cols, rows) whose product is 12 or less. There are 35 of them. You can check: 12 grids with one column, 6 with two, 4 with three, 3 with four, 2 each for five and six columns, and 1 each for the remaining six, which sums to 35.
Now the matching. The image has an aspect ratio, its width divided by its height, call it a. Each grid has one too, cols divided by rows. The paper measures the absolute difference |a − cols/rows| for all 35 grids and picks the smallest.
Work the paper’s own example, an 800 × 1300 image, so a = 800 / 1300 = 0.615. The grid 2 × 3 has shape 0.667, a difference of 0.051. The grid 1 × 2 has 0.500, a difference of 0.115. The grid 3 × 4 has 0.750, a difference of 0.135. The grid 1 × 1 has 1.000, a difference of 0.385. The smallest is 2 × 3.
The image is resized to 2 × 448 = 896 pixels wide and 3 × 448 = 1344 tall, as in the paper’s Figure 4. It is cut into 6 tiles. Notice a closer grid exists: 3 × 5 has shape 0.600, a difference of only 0.015. It needs 15 tiles, so it is not on the list. The cap of 12 is a budget decision, and the paper says it was set by limited computing resources.
One rule breaks ties. A 1 × 1 grid and a 2 × 2 grid have the same shape. The paper prefers the one that does not exceed twice the input image’s area, so that a small image is not blown up into many tiles for no reason.
Tiles see detail but lose the big picture. A tile that holds only the corner of a table does not know it is a table. So the paper adds a thumbnail: the whole image shrunk to a single 448 × 448 tile. The model gets the 6 detail tiles and 1 overview, 7 views. At 256 tokens a view, that is 1,792 visual tokens.
The thumbnail is added whenever there is more than one tile. A one tile image is already a whole view. That explains the paper’s range: in training the visual tokens go from 256 (one tile) up to 13 × 256 = 3,328 (12 tiles plus a thumbnail). At test time the number of tiles can rise to 40, which gives 41 × 256 = 10,496 tokens. The paper says this works without any extra training, a zero-shot extension.
Choose an image, then a tile limit. The dashed outline is the image’s true shape; the solid frame is the grid it is stretched into.
The matching rule is the paper’s (Section 3.3). The tie-break and the search order are our reading of its one-sentence description, written as a sketch in the Field Guide. Image sizes are examples we chose; only the 800 × 1300 to 896 × 1344 case is the paper’s.
Try the receipt. The symptom with a fixed square is that tall, thin print is squeezed to a quarter of its width. The measurement is the shape error: the receipt has a = 0.278, a fixed 1 × 1 square has 1.000, an error of 0.722. The matching rule picks 1 × 4 (0.250), an error of 0.028. The fix costs tokens: 4 tiles plus a thumbnail is 1,280 visual tokens against 256. You pay in tokens for shape and detail. Chapter 8 returns to that price.
Chapter 3
Trade four neighbouring patches for one wider token and cut the language model's work sixteen fold
You have a six tile page. Each tile gives the encoder 1,024 patches, so the language model would have to read 7 × 1,024 = 7,168 visual vectors before it even sees your question. That is more than the 4,096 token context the paper trains with. Something must give.
The cheap fixes lose information. You could drop every second patch, or average patches together, like shrinking a photo. Either way, thin strokes of small letters can vanish. The paper instead uses a rearrangement that throws nothing away: pixel shuffle.
Picture a chess board where each square holds a short list of numbers. Now glue each 2 × 2 block of four squares into one bigger square that holds all four lists end to end. The board has a quarter as many squares. Each square has four times as many numbers. The total count of numbers is unchanged.
This is also called space to depth: it moves neighbours from space (the grid) into depth (the length of the vector). The paper says only that “a simple pixel shuffle operation” reduces the number of visual tokens to one quarter, and that a 448 × 448 tile is therefore 256 tokens. Here is the exact bookkeeping.
In code it is a reshape and a permute. Split the height axis into (16, 2) and the width axis into (16, 2). Move both of the size 2 axes next to the channel axis. Merge them into the channel axis. The result has 16 × 16 = 256 rows, and they go to the projector.
Left: an 8 × 8 toy grid of patches. Right: the tokens after shuffling. Choose a factor, then press Next block to follow one block across.
The 8 × 8 grid is a toy so that you can see each cell; the real tile has 32 × 32 patches. Factor 2 is the paper’s. Factors 1 and 4 are illustrative. Pair counts assume causal attention, n(n + 1)/2 pairs per layer for n tokens, as in the language model, for one tile.
The language model compares tokens with each other. In its attention step, each new token looks at every token before it. For n tokens that is about n²/2 pairs per layer. Cut n by four and the pairs fall by about sixteen. For one tile: 1,024 tokens give 1,024 × 1,025 / 2 = 524,800 pairs; 256 tokens give 256 × 257 / 2 = 32,896. That is 16.0 times fewer.
The projector, the feed-forward parts of the language model and the memory for stored keys and values all scale with n, so they shrink by four. The paper says this choice was made “to enhance scalability for high resolution”. With 12 tiles and a thumbnail the model reads 3,328 tokens, not 13,312.
There is a price. Each token is now four patches wide, so the projector’s first layer takes 4dv numbers, and the language model must find fine detail inside a fatter vector. The paper does not run an ablation of the shuffle factor, so it does not report what a factor of 1 or 4 would score. We show those settings only to make the arithmetic visible.
The symptom without it: a run of many tiles crashes with an out-of-memory error, or becomes slow. The paper itself notes that MMMU, where each question holds several images, could run out of memory at high tile counts, so it was tested only up to 18 tiles. The measurement is simple counting: tokens times layers times width. The fix here is the shuffle, which makes each tile one quarter as heavy. Without it, the practical tile count would be a quarter as large.
Chapter 4
Continue training a 6 billion parameter vision encoder and learn why its last layers go
Teach someone to read using only 224 pixel postcards from the internet: holiday snaps, product shots, memes. Then hand them a scanned contract in 8 point print. They will struggle, not because they are slow, but because they never practised on this kind of page.
That is the problem the paper names for the usual vision encoder. It is a contrastively pre-trained ViT: trained on pairs of an image and its caption from the web, to give matching pairs similar feature vectors and non-matching pairs different ones. The paper says these are trained at a fixed low resolution such as 224 × 224, so they degrade on high-resolution pictures and on images that do not come from the web, such as document scans.
The authors start from InternViT-6B, a 6 billion parameter encoder from an earlier InternVL paper. They improve it in two releases. Read the steps as a recipe.
Step 1, the V1.2 release. First, they drop the last three layers. The paper found that the features of the fourth-to-last layer work best for multimodal tasks, so it throws away the weights of the last three layers and the encoder shrinks from 48 layers to 45. The paper reports the finding but gives no reason. The usual explanation is that the last layers of a contrastive model specialise in the caption-matching goal, while slightly earlier layers keep more general detail.
Second, they raise the input from 224 to 448 pixels. Third, they join the encoder to a 34 billion parameter language model, Nous-Hermes-2-Yi-34B, and train the encoder and the projector together, on a mix of captioning and OCR-specific datasets. The updated encoder weights were released as InternViT-6B-448px-V1.2.
Each bar is one layer of InternViT-6B. Switch on the cut and see what is kept.
The 48 layers, the fourth-to-last finding and the 45 layer result are from Section 3.2. The picture draws layers as equal bars; real layers differ in content, not in size.
Step 2, the V1.5 release. Now they continue training from V1.2. The input becomes dynamic 448: tiles of 448 × 448, from 1 to 12 of them (Chapter 2). They also increase the scale, quality and variety of the pre-training data (Chapter 6). The result is InternViT-6B-448px-V1.5, which the paper says is more robust, better at OCR and better with high resolution.
This idea, called continual learning or continued pre-training, means: do not start over; pick up the trained weights and keep training on data closer to the job. It is cheaper than training from scratch, and each release builds on the last.
One result is easy to miss. Between V1.2 and V1.5 the language model changed from Yi-34B to InternLM2-20B. The encoder kept working. The paper concludes that the visual features learned during the MLLM pre-training stage are broadly useful and not tightly bound to one language model. For a builder this is valuable: you can train a strong encoder once and plug it into different language models.
Why a 6 billion parameter encoder, when the usual one has 300 million? The paper’s claim is that a large language model needs a large vision encoder to match it. The ablation for this is in Chapter 8. The totals are in the device below.
Each model has a vision part (purple) and a language part (pink). Bars use a log scale, so each gridline is ten times bigger. Choose a model.
The 0.3B encoder with 7B or 13B language model is the paper’s description of typical open models (Section 1). InternVL 1.2 is 6B + 34B = 40B and InternVL 1.5 is 6B + 20B = 26B (Table 2). The 100B line is the paper’s figure for the commercial models’ total; it does not say how they split it.
Size has a price. A common rule of thumb says one forward pass costs about 2 operations per parameter per token. This rule is ours, not the paper’s. For one tile, the encoder handles 1,024 patches: 2 × 6 × 109 × 1,024 ≈ 1.2 × 1013 operations. A 2 × 3 page with its thumbnail has 7 views, so about 8.6 × 1013 operations before the language model starts. Pixel shuffle saves work only after the encoder. The big eye is a real, per-image cost, and the paper pays it on purpose.
The symptom of a weak encoder: the language model is smart but the answers about small text are wrong, whatever the prompt. The measurement is a document benchmark. InternVL 1.2 scored 57.7 on DocVQA, and InternVL 1.5 scored 90.9. Be careful with that comparison. Between the two, the encoder training, the tiling, the data and the language model all changed. The jump shows the whole recipe works. The paper does not isolate the contribution of the encoder alone.
Chapter 5
Run the stage that teaches the vision side, then the stage that tunes all 26 billion parameters
You are training a new interpreter. You would not hand them a courtroom on day one. First you teach them the vocabulary of a new subject, then you put them on real cases. InternVL 1.5 follows the same plan in two stages, and the order matters.
A network has parameters, dials that start at some value. Training shows the network an example, compares its answer with the right one, and nudges the dials to make that answer a little more likely next time. A trainable part has dials that move. A frozen part has dials that stay put.
For a model that writes text the standard measure of error is the chance it gave to each correct word. The paper does not print its loss, so what follows is the usual recipe for models of this kind, not a quote.
The paper says that in the first stage, pre-training, the focus is on training the vision encoder InternViT-6B and the MLP projector, to optimise visual feature extraction. The projector starts random, so it needs this stage to learn its job. The encoder is trained further because it must now handle tiles and text-heavy images.
The data is a mix of public sources (Table 1a of the paper). The shares are exact: captioning 53.9%, detection and grounding 5.2%, large OCR 32.0% and small OCR 8.9%. These add to 100.0%. Captioning means writing a sentence about the picture. Detection means finding and naming objects with their positions.
Look at the OCR shares: 32.0% + 8.9% = 40.9%, about two in five examples. The large OCR sets were built by the authors with PaddleOCR, an open tool that reads text in images. They ran it on Chinese images from Wukong and English images from LAION-COCO. The large OCR group also holds Common Crawl PDFs. In 1,000 examples that is about 539 captions, 52 detection, 320 large OCR and 89 small OCR (illustrative rounding of the real shares).
In the second stage, fine-tuning, the entire model’s 26 billion parameters are trained: encoder, projector and language model together. The data is the set of conversations in Table 1b: captioning, general question answering, science, charts, mathematics, knowledge, OCR, documents, grounding, conversation, and some text-only data. The text-only sets (such as OpenHermes2.5 and Alpaca-GPT4) are there “to maintain the original linguistic capabilities of the LLM”. The paper does not give the stage two shares.
Both stages use a context length of 4,096 tokens and the same response-format prompts as LLaVA 1.5, a well-known earlier open model. The language model is the chat version of InternLM2-20B, not the base version.
Choose a stage. Bright blocks learn. Dim blocks are not the focus. Below them, the data that stage uses.
Stage 1 shares are Table 1a of the paper. Stage 2 lists the categories of Table 1b without shares, since the paper gives none. The paper says stage 1 focuses on the encoder and projector; it does not say in words that the language model is frozen, so we dim it rather than mark it frozen.
Both stages use a context of 4,096 tokens, so every training example must fit in it. Take a 12-tile image with its thumbnail: 13 × 256 = 3,328 visual tokens. That leaves 4,096 − 3,328 = 768 tokens for the question and the answer. A short caption fits easily. A long chain of reasoning on a full 12-tile document might not. Larger tile counts therefore compete with text for the same window, which is one reason the cap is 12.
The symptom of skipping stage one: the projector, still random, passes noise to the language model, and the fine-tuning data has to teach it everything at once. The measurement the paper gives is indirect: its OCR scores. InternVL 1.5 scores 724 on OCRBench and 72.5 on InfographicVQA, against 569 and 39.5 for InternVL 1.2. The data share (two fifths OCR) is the clearest lever the paper points to, though it does not run a one-at-a-time test of that share.
Chapter 6
Build the data mix and the translation prompt that give the model its Chinese
A shop owner in Zhengzhou photographs her sign and asks, in Chinese, what the sign says. A model trained almost only on English captions may describe the colours and miss the words. The third gap in Chapter 0 was language, and this chapter is the paper’s fix: data.
A model is shaped by what it practises on. If you want it to answer in Chinese about Chinese street scenes, it must see Chinese questions, Chinese answers and Chinese text in pictures. The authors collect such data, and where none exists, they make it by translation.
Table 1b of the paper lists the fine-tuning data by task. Each row is a skill, with example datasets. Captioning: TextCaps and a bilingual ShareGPT4V. General question answering: VQAv2, GQA and VisualDialog. Science: AI2D, ScienceQA and TQA. Charts: ChartQA, PlotQA and others. Mathematics: GeoQA+, TabMWP and MathQA. Knowledge: KVQA and bilingual Wikipedia. OCR: OCRVQA, TextVQA and several Chinese and English text-reading sets. Documents: DocVQA and Common Crawl PDFs. Grounding: RefCOCO and Visual Genome. Conversation: LLaVA-150K and ALLaVA. Text-only: OpenHermes2.5, Alpaca-GPT4 and others.
Each dataset in Table 1 carries a language tag: en, zh, or both. The paper explains the tag. If a dataset was originally English and is marked “zh”, the authors translated it into Chinese. COYO and GRIT, for example, began in English.
The pipeline is plain. Take a row of an English dataset, such as a caption or a question and its answer. Ask a language model to translate the text. The image stays the same. Write the Chinese text back as a new row. The paper says the translator can be a strong open model or GPT-3.5, and the pipeline extends to more languages by changing the language word in the prompt.
The prompt is the clever part. A plain “translate this” tends to give translationese: text that is correct but stiff, as if a machine had followed English grammar word for word. Figure 5 of the paper gives the instruction. It is a system message that says the model is a translator proficient in English and the target language, asks for a natural and fluent result that avoids translationese, and lists five rules.
Then a user message carries the text: “Text for translation: {text}”, and the assistant turn holds the translation. Build one below.
Choose a language and a sample line. The box shows the exact message the pipeline would send.
The prompt wording is Figure 5 of the paper, with {language} and {text} filled in. The three sample lines are ours. The translation itself is written by a language model; we do not fake it here.
Take the sample line about the Eiffel Tower sign. A good Chinese output keeps “Eiffel Tower” and “SenseTime” in English (rule 1). It writes the Chinese words for optical character recognition and puts OCR in parentheses (rule 5). A cheap automatic check, our suggestion and not the paper’s, is to confirm that every protected name appears unchanged in the output. A row that fails the check can be sent back for another try. At the scale of millions of rows, such cheap checks are how you find a bad prompt before you train on it.
The symptom of an English-only model: it reads Chinese characters in a picture poorly, and answers Chinese questions in a flat way. The measurement is the Chinese benchmarks. On MMBench-CN, InternVL 1.5 scores 82.0, GPT-4V 74.4 and LLaVA-NeXT 79.0. On CCBench, a test of Chinese culture, InternVL 1.5 scores 69.8, GPT-4V 46.5 and InternVL 1.2 59.2. The fix is the bilingual data.
The caveat: translation brings its own errors, and the report gives no measure of the translation quality. Nor does it separate the effect of the translated data from the effect of data that was originally Chinese. The Chinese score gains are real; their exact cause is a joint one.
Chapter 7
Read the tables the way the authors meant them, and see where the model wins and where it loses
A new model arrives with a table of numbers. How do you decide whether to believe it? You ask what each column measures, who ran it, who is missing, and where the new model loses. This chapter reads Table 2 and Table 3 of the paper that way.
The paper tests 18 benchmarks and sorts them into four groups. Table 2 holds 16 of them. Table 3 adds the other two, ConvBench and MMT-Bench.
OCR-related tests ask the model to read. DocVQA is questions about document scans. ChartQA is questions about charts. InfographicVQA is questions about infographics. TextVQA is about text in everyday scenes. OCRBench is a broad test of text recognition.
General multimodal tests are mixed. RealWorldQA checks real-world spatial understanding. HallusionBench checks whether the model invents things that are not in the picture; this failure is called hallucination. MMMU asks college-level questions across subjects. AI2D is science diagrams. MMBench-CN and CCBench test Chinese and Chinese culture. The paper also uses MME, MMBench-EN, MMVet and SEED.
Math is MathVista: maths problems that need a picture. Multi-turn conversation is ConvBench, where a chat goes through perception, reasoning and creation.
Choose a benchmark. The warm bar is InternVL 1.5. Gold-outlined bars are the best score of the five. Watch where it wins and where it does not.
Numbers are copied from Table 2 of the paper (OCRBench is on its own scale, 0 to 1000 or so, so bars are scaled per benchmark). A starred TextVQA score for LLaVA-NeXT and InternVL 1.2 used Rosetta OCR tokens. A dash in the paper is shown as “not reported”. Claude-3 Opus is one of the closed models the paper lists; the full table has more, and some beat InternVL 1.5 on single columns (Qwen-VL-Max scores 93.1 on DocVQA).
Click through the OCR group. On ChartQA, InternVL 1.5 scores 83.8 against 81.3 for Gemini Pro 1.5 and 80.8 for Claude-3 Opus, and the paper says it beats every closed model in Table 2 on ChartQA and on OCRBench (724). On DocVQA it scores 90.9, above the four closed models of Figure 1, though not above Qwen-VL-Max (93.1) in the full table. Against the open models the jump is large: LLaVA-NeXT 84.0 and InternVL 1.2 only 57.7. On InfographicVQA, 72.5 is far above the open models’ 51.5 and 39.5.
Beyond reading, the paper says it has the best HallusionBench score (49.3, against 46.5 for GPT-4V), and that it beats GPT-4V by a clear margin on MathVista (53.5 against 49.9). On the Chinese tests it reaches 82.0 on MMBench-CN, above every closed model in the table (GPT-4V has 74.4), and 69.8 on CCBench, above GPT-4V (46.5). The paper claims both as wins over closed models, but the table shows Step-1V at 71.2 on CCBench, so read that claim as “ahead of the big names”, not “ahead of every row”.
Now click MMMU. InternVL 1.5 scores 45.2. GPT-4V scores 56.8 and Claude-3 Opus 59.4. Worse, its own predecessor InternVL 1.2 scored 51.6. The authors say the drop comes from the smaller language model: 20 billion parameters, against 34 billion. MMMU asks for college-level subject knowledge and reasoning, which the language model supplies.
AI2D shows another gap: 80.7 against 88.1 for Claude-3 Opus. And Table 3 is blunt. On ConvBench pairwise grading, GPT-4V scores 39.51 on the overall R1 column and InternVL 1.5 scores 17.65, and its own predecessor InternVL 1.2 scores 21.17 (the paper still calls it leading among open models). The paper says InternVL still trails GPT-4V “by a considerable margin” here. On MMT-Bench overall, InternVL 1.5 gets 59.0 against 62.0 for GPT-4V and 63.4 for InternVL 1.2.
The headline is “state of the art in 8 of 18 benchmarks”. That is a count of wins among the models the authors chose to compare. The comparison set matters: some closed models have dashes on some benchmarks, because their makers did not publish those numbers. Several scores (OCRBench, MMBench, CCBench, HallusionBench) were collected from the OpenCompass leaderboard, not run by the authors. And the 1.2 versus LLaVA-NeXT comparison in Chapter 8 is, as the authors admit, not entirely fair.
None of this makes the result weak. The pattern is consistent and plausible: a model built for reading wins at reading and at Chinese, and loses on broad knowledge and long chat. That is what the three fixes predict.
Compare InternVL 1.5 with Claude-3 Opus by subtraction. ChartQA: 83.8 − 80.8 = +3.0. DocVQA: 90.9 − 89.3 = +1.6. MathVista: 53.5 − 50.5 = +3.0 (and +0.7 over Grok-1.5V at 52.8). MMMU: 45.2 − 59.4 = −14.2. The wins are small and the loss is large. The paper reports one score per benchmark and no error bars, so a lead of a point or less is a hint, not a proof. The 14 point gap on MMMU is not.
The symptom of reading only the headline: you choose InternVL 1.5 for a college-exam assistant and are disappointed. The measurement is the per-benchmark row: MMMU 45.2 against 56.8. The fix is to pick the benchmark nearest your task, not the count of wins.
Chapter 8
Test the two design claims and name what the paper does not prove
An ablation is an experiment that changes one thing and watches what happens, the way you would test whether a recipe needs the salt by cooking it without. A paper’s ablations show which of its claims it has tested. The report has two, and one honest gap.
The claim: if you give a 34 billion parameter language model a tiny eye, the eye limits it. To test it, the authors compare two models that share the same size of language model. LLaVA-NeXT has a 34B language model and a 300 million parameter encoder. InternVL 1.2 has a 34B language model and the 6 billion parameter InternViT.
The result they report: after excluding five OCR datasets, ConvBench and RealWorldQA, InternVL 1.2 beat LLaVA-NeXT on 9 of the remaining 11 benchmarks. They read this as support for the claim.
Now the caveats, which the authors state themselves. Data for LLaVA-NeXT was not available, so they built a similar dataset. And the comparison is not fair in resolution: InternVL 1.2 was trained at a fixed 448 × 448, while LLaVA-NeXT used a higher dynamic resolution of 672 × 672. The two models differ in more than the encoder. The test is suggestive, not clean. The paper drops the five OCR sets, ConvBench and RealWorldQA from the count, and the text does not say why.
The second test is Figure 6. The authors run InternVL 1.5 on many benchmarks with a different number of tiles each time and plot the score against the tile count. The model was trained with 1 to 12 tiles. At test time they push it as far as 40. What they find, in the paper’s words:
First, OCR tasks (DocVQA, InfoVQA, TextVQA, OCRBench) benefit from more resolution. Second, other tasks (AI2D, MMMU, MMBench, HallusionBench) show a slight decline at high resolution. Third, the model scales to 40 tiles zero-shot, although it only saw up to 12 in training. So the model is robust to the tile count, and the user can choose the cost. We describe these findings in the paper’s words and do not quote plotted values.
The reason is easy to feel. Small print needs pixels. A science diagram or a photo of a dog does not, and more tiles only add tokens that dilute attention. This echoes the paper’s intro: it likens the choice to the “low” and “high” modes of GPT-4V.
Drag the number of tiles. The bar is the visual tokens against the 4,096 token context of training. The tick marks are the paper’s limits: 12 tiles in training, 18 for MMMU, 40 at test.
Tokens use the paper’s rule: 256 per tile, plus one 256 token thumbnail when there is more than one tile (256 at 1 tile, 3,328 at 12, 10,496 at 40). Pairs are n(n + 1)/2 for one layer of the language model, and ignore the few tokens of the question. Which tasks gain from more tiles is the paper’s Figure 6 finding, described in words.
Here is what the report does not test. It does not turn the three fixes on and off one at a time, so it cannot say how much of the gain comes from the encoder, how much from tiling and how much from the data. It does not ablate the pixel shuffle, the 12 tile cap or the OCR share. The jump from InternVL 1.2 to 1.5 mixes the three fixes with a change of language model.
The tables show limits too. The model trails GPT-4V on multi-turn conversation by a wide margin, and falls on MMMU. The authors say they will keep refining multi-turn chat. The qualitative figures (7 to 12) are chosen examples. In them, GPT-4V declined a question about identifying a person for privacy reasons while InternVL 1.5 answered. Whether that is better depends on what you want.
Take the MMMU cap. The paper tests MMMU with at most 18 tiles because each question holds several images. Eighteen tiles with a thumbnail make 19 × 256 = 4,864 visual tokens for one image, already above the 4,096 token context used in training, and a question may hold more than one image. Two simple lessons follow. Multi-image prompts consume the budget fast, and the fair test for a long-context claim is a test that actually runs the long contexts.
The symptom of “more tiles is always better”: a science question gets slightly worse and the bill gets bigger. The measurement is Figure 6 for the score and the device above for the bill: at 40 tiles one layer compares about 55 million pairs, about ten times the 12 tile case. The fix: choose the tile limit per task.
Chapter 9
Keep the numbers, the checklist, the code and the credit on one page
InternVL 1.5 is a ViT-MLP-LLM model with three changes. It cuts every picture into a matched grid of 448 pixel tiles plus a thumbnail. It continues to train a 6 billion parameter vision encoder and fuses four neighbouring patches into one token. It learns in two stages on a bilingual mix that is two fifths OCR. The result reads documents, charts and Chinese signs about as well as the closed models, and is weaker at college-level knowledge and long chat.
| What | Value | Where in the paper |
|---|---|---|
| Tile size | 448 × 448 pixels | Section 3.1 |
| Visual tokens per tile | 256 (1,024 patches, shuffled 4 to 1) | Section 3.1 |
| Tiles in training | 1 to 12, so 256 to 3,328 tokens | Sections 3.2, 3.3 |
| Tiles at test time | up to 40, so 10,496 tokens (4K) | Sections 3.1, 3.3 |
| Allowed grids | 35 (cols × rows ≤ 12) | Section 3.3 |
| Encoder | InternViT-6B, 48 layers cut to 45 | Section 3.2 |
| Language model | InternLM2-20B chat; 26B in total | Sections 3.1, 4.1 |
| Context length | 4,096 in both stages | Section 4.1 |
| Stage 1 data mix | 53.9 caption, 32.0 large OCR, 8.9 small OCR, 5.2 detection (%) | Table 1a |
| Document and chart reading | DocVQA 90.9, ChartQA 83.8, TextVQA 80.6, OCRBench 724 | Table 2 |
| Chinese, hallucination, maths | MMBench-CN 82.0, CCBench 69.8, HallusionBench 49.3, MathVista 53.5 | Table 2 |
| Where it trails | MMMU 45.2; ConvBench R1 17.65 against 39.51 for GPT-4V | Tables 2, 3 |
| Headline claim | best on 8 of 18 benchmarks | Abstract |
| Term | Plain meaning |
|---|---|
| MLLM | A language model that also takes images as input. |
| VFM / ViT | The vision network that turns image patches into feature vectors. |
| Tile, thumbnail | A 448 pixel square piece of the resized image; a small copy of the whole image. |
| Pixel shuffle | Join each 2 × 2 block of patch vectors into one vector, 4 times fewer tokens. |
| Projector | A small network that maps vision features into the language model’s width. |
| Zero-shot (here) | Using 40 tiles at test time though training saw at most 12. |
| Ablation | An experiment that changes one thing to see what it does. |
| Hallucination | The model states things that are not in the picture. |
This is our sketch of the data flow, not the authors’ code. The released implementation is in the OpenGVLab InternVL repository.
pythonimport torch, torch.nn.functional as F TILE = 448 def choose_grid(w, h, max_tiles=12): a, best, best_d = w / h, (1, 1), float("inf") grids = sorted({(c, r) for c in range(1, max_tiles + 1) for r in range(1, max_tiles + 1) if c * r <= max_tiles}, key=lambda g: g[0] * g[1]) # 35 grids for c, r in grids: d = abs(a - c / r) # shape error if d < best_d: best, best_d = (c, r), d elif d == best_d and c * r * TILE * TILE <= 2 * w * h: # tie rule best = (c, r) return best def views(img, max_tiles=12): # img: (3, H, W) _, h, w = img.shape c, r = choose_grid(w, h, max_tiles) big = F.interpolate(img[None], size=(r * TILE, c * TILE), mode="bicubic")[0] tiles = [big[:, i*TILE:(i+1)*TILE, j*TILE:(j+1)*TILE] for i in range(r) for j in range(c)] if len(tiles) > 1: # thumbnail only if 2+ tiles tiles.append(F.interpolate(img[None], size=(TILE, TILE), mode="bicubic")[0]) return torch.stack(tiles) # (V, 3, 448, 448) def pixel_shuffle(x, r=2): # (V, 32, 32, d_v) v, h, w, d = x.shape x = x.view(v, h // r, r, w // r, r, d).permute(0, 1, 3, 2, 4, 5) return x.reshape(v, h // r, w // r, r * r * d) # (V, 16, 16, 4 d_v) def forward(img, question_ids): t = views(img) # (V, 3, 448, 448) f = vit(t) # (V, 1024, d_v) f = pixel_shuffle(f.view(len(t), 32, 32, -1)).flatten(1, 2) # (V, 256, 4 d_v) vis = mlp(f).flatten(0, 1) # (V*256, d_l) visual tokens txt = llm.embed(question_ids) # (T, d_l) return llm(torch.cat([vis, txt])) # next-token logits
InternVL 1.5 sits in the family of models that join a pre-trained vision encoder to a language model through a small projector, such as the LLaVA series and MiniGPT-4, which the paper cites. Its tiling follows the other route the paper names, dividing a high-resolution image into many low-resolution tiles, as opposed to a dual-branch encoder. Read it next to MMAI Lecture 5.1 on large multimodal models, which this paper belongs to.
Credit. This lesson re-teaches “How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites” by Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye and colleagues at Shanghai AI Laboratory, SenseTime Research, Tsinghua University, Nanjing University, Fudan University and the Chinese University of Hong Kong. Read the original: arXiv 2404.16821. Code: OpenGVLab/InternVL. Model: InternVL-Chat-V1-5.
Now press Present or Teach and explain one picture's journey out loud: the grid it gets, the tokens it costs, the shuffle, the two stages and the one benchmark where the model loses. Then go back to the page at the top, pick 2 × 3 tiles, and say why the letters come back.