SPHINX-X

You photograph a receipt and ask a model for the total. Can it read the small print? One recipe, SPHINX-X, trains four model sizes, from 1.1B to 8×7B parameters. Pick a size and watch six scores move.

Learn how SPHINX-X cuts a vision-language model down to two eyes, skips blank tiles, trains in one stage, and grows from a phone-sized model to a mixture of experts.

Press a model size below, or wait and watch the demo. Notice how the bars rise with the size of the language model, and how SPHINX-Plus beats the original SPHINX with the same 13B parameters. Then we build, piece by piece, every part of the recipe.

You need a rough idea of what a neural network is (a stack of adjustable number tables) and a little Python. We build the rest from zero.

Paper: SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models, Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao and colleagues at Shanghai AI Laboratory and CUHK MMLab (ICML 2024). Read for MIT MMAI, Multimodal AI, Spring 2026, as a companion to the lecture on large multimodal models. Every idea is re-taught here in our own words and devices.

Scale the model, read the scores

Every score is from the paper: POPE, MMBench, MM-Vet and MathVista from Table 1, TextVQA from Table 4, RefCOCO val (the share of predicted boxes that overlap the true box by at least half, called IoU 0.5) from Table 5. All are on a 0 to 100 scale, so the bars share one unit. The average is a plain mean of these six numbers, our own summary and not a number the paper reports. SPHINX (2023) is the earlier 13B model the paper compares against.

Chapter 0

Why One Model Needs More Data and More Sizes

See the three gaps in open vision-language models that SPHINX-X sets out to close

Picture a student with a phone. She takes three photos and asks one question about each. The first photo shows a dog on a beach: “What is happening here?” The second shows a restaurant receipt: “What is the total?” The third shows a street: “Draw a box around the tallest building.”

A good open model of early 2023 answers the first question well. It stumbles on the second, because reading small printed digits is a different skill from naming a dog. It cannot do the third at all, unless someone built a special tool for boxes. Three photos, three different skills, and one model is expected to have all of them.

This paper, SPHINX-X, is about building that one model. Its authors work at Shanghai AI Laboratory and the Chinese University of Hong Kong. They start from their own earlier model, SPHINX, and ask what stops it from being a general vision assistant.

What we are building: a multi-modal large language model

A modality is a kind of input: text, image, sound. A multi-modal large language model (MLLM) is a language model that also accepts a second modality, here images. You give it a picture and a question in words. It answers in words. The next chapter opens the box. For now, think of an MLLM as a language model that has been given eyes.

Many open MLLMs appeared within a year of GPT-4V and Gemini: the BLIP series, LLaMA-Adapter, LLaVA, MiniGPT-4, mPLUG-Owl and SPHINX itself. They share one idea. Take a ready-made language model, take a ready-made image encoder, and teach a small connector to translate between them.

The three gaps the paper names

The introduction of the paper argues that these models are held back by their training data and by the sizes they come in. The authors add a third problem, which is about their own method. Here are all three, in plain words.

Gap one: the data covers too few tasks. Models like BLIP-2, LLaVA and LLaMA-Adapter learn from raw image and caption pairs of natural photos (the paper names LAION, SBU and Conceptual Captions) and from instruction data written by GPT-4. They understand natural photos well. In the paper’s words they show “limited or degraded results” on OCR, tables, charts and mathematics. OCR (optical character recognition) means reading text out of a picture, the skill the receipt needs.

The opposite choice is a specialist. Shikra points at objects with boxes. mPLUG-DocOwl reads documents. Kosmos-2.5 reads text-heavy images. Each is good at its own job, but the paper says they pay for it with their general ability.

Gap two: the sizes are too few. Most open MLLMs sit on a dense language model of 7B or 13B parameters (a parameter is one learned number; 7B means seven billion of them). That size is too large for a phone, and too small to find out how good an MLLM can get. The authors want a small one, and a very large one built as a mixture of experts, a design where only part of the network runs for each word.

Gap three: the recipe is clumsy. SPHINX used four vision encoders at once and two separate training stages, with hand-picked parts frozen in each. That is slow and fiddly. The more data you add, the worse it gets.

Press each gap below. The left side shows the state of things before. The right side shows what SPHINX-X changes.

Pick a gap, see the fix

Left: the situation before SPHINX-X. Right: what the paper changes. Press each of the three gaps in turn.

A map of the paper’s text (Section 1 and Section 3), not a measurement. In the data view, a filled mark means the paper says the training data covers the skill; an empty mark means the paper names it as a weakness of earlier generalists. The 8×7B size is the model’s name, not its true parameter count.

Notice how the three fixes depend on each other. A leaner recipe makes it cheap to feed in much more data. More data makes it worth trying more model sizes. And the sizes let the authors test one claim: does the score rise with data and with parameters? The paper’s title is that claim.

The claim as a trade

Written as a trade: SPHINX-X gives one open model that handles photos, documents, charts, math diagrams and boxes, in four sizes, where earlier models either stuck to natural photos or became specialists, because it trains one simple recipe on a very wide data mix. The hero at the top of the page lets you check the second half of that sentence with real numbers. The rest of this lesson explains the first half.

The road

  1. From pixels to tokens: what an MLLM is, and the data flow from image to answer.
  2. Mixture of visual experts: why SPHINX-X keeps two image encoders out of four.
  3. Tiles and skip tokens: how big pictures are cut up, and what to do with blank tiles.
  4. One-stage training: one run, one data pool, the optimiser settings.
  5. The data recipe: public sets, the OCR set and the Set-of-Mark set.
  6. Four language models and sparse experts: the family from 1.1B to 8×7B.
  7. What the tables show: the numbers, the ablations, and what they do not prove.
  8. The paper on one page: numbers, checklist, code, and where to go next.
According to the paper, why do open MLLMs such as LLaVA and BLIP-2 do poorly on OCR, charts and math?

Chapter 1

From Pixels to Tokens

Follow one image through an MLLM until a language model can read it like a sentence

Imagine you must describe a painting to a friend who only understands Morse code. You cannot hand over the canvas. You must first turn it into dots and dashes, and the friend must already know what those signals mean.

A language model has the same problem. It reads a list of vectors, one for each word piece. A photo is a grid of colour values. Before the model can answer a question about the photo, something must turn the grid into a list of vectors that look, to the model, like words.

Three ideas you need first

A token is one piece of text, such as a word or part of a word. The model turns each token into an embedding: a vector, which is a list of numbers, of fixed length d. For LLaMA2-13B that length is 5,120. The model never sees letters. It sees a stack of T vectors, each d numbers long, so the stack has shape T × d.

A vision encoder is a network that turns an image into features. One kind, the vision transformer (ViT), cuts the image into small squares called patches and turns each patch into a vector. Another kind, a convolutional network (CNN), slides small filters across the image and builds up features. Either way the output is again a stack of vectors, one per image region.

The encoders matter because of how they were taught. CLIP learned by matching images to their captions. The paper calls this weakly-supervised, because the captions are noisy hints, not exact labels. DINOv2 learned from images alone, by asking whether two views of one picture agree. That is self-supervised learning. We use both below.

The last piece is a projection: a linear layer that multiplies each feature vector by a matrix. Its job is translation. Image features have their own length, dvis. The language model wants length dllm. The matrix maps one to the other, and it is the part that must be learned from scratch.

How earlier models connected the two

The paper lists four ways to connect eyes to a language model. BLIP-2 and MiniGPT-4 use a query transformer (Q-former): a small network with a fixed number of learned questions that summarise the image into a few vectors. Flamingo adds gated cross-attention, where language layers peek at image features through new attention layers. LLaMA-Adapter adds zero-initialised attention so that new layers start with no effect. LLaVA uses the simplest bridge: one linear layer that turns image features into tokens.

SPHINX and SPHINX-X sit on the simple side. The image features pass through learned linear projection layers, and the results join the text as extra tokens. The language model then treats picture tokens and word tokens as one sequence.

The data flow, step by step

Here is one request. A user sends an image and the question “What is the total?” The steps are these.

  1. Prepare views. The image becomes several views: one small whole image and several sub-images (Chapter 3). Each view has shape 224 × 224 × 3.
  2. Encode. Each view goes through the vision encoders (Chapter 2). Each encoder returns a stack of n × dvis feature vectors.
  3. Project. A linear layer maps every feature vector to the language model width: n × dllm.
  4. Concatenate. The picture tokens from all views and the embedded question tokens form one sequence of length N.
  5. Decode. The language model reads the sequence and writes the answer one token at a time.

The paper does not give the token counts, so we use toy numbers in the device below. Slide the numbers and watch two things: how long the sequence gets, and how small the new, learned projection is next to the language model it feeds.

Count the tokens

Choose a language model, a number of views, and how many feature vectors each view gives. The bar shows what the language model must read.

5
256

Toy numbers, not the paper’s: 1,024 for the width of the image features, 20 text tokens for the question. The widths of the language models (2,048, 4,096, 5,120, 4,096) and the 47B total of Mixtral are from the models’ own cards, not from this paper. Five views is SPHINX’s one global image plus four 224-pixel sub-images; ten views would be the 3×3 split of SPHINX-Plus-2K.

With the default numbers the language model reads 1,280 picture vectors plus 20 text vectors: 1,300 in all. The projection holds about 5.2 million numbers, which is 0.04% of the 13B language model. Almost all the model’s knowledge lives in the language model and the encoders, both of which began pretrained. The connector is tiny, and it is the one piece with no head start.

What the model is trained to say

The model learns to make the correct answer likely. Write V for the picture tokens, q for the question tokens and y1 to ym for the answer tokens. The probability of a whole answer is built one token at a time, as in any language model.

P(y | V, q)
The chance that the model writes the answer y, given the picture tokens V and the question q.
∏t=1m
Multiply over every answer token, from the first (t = 1) to the last (t = m).
P(yt | y<t, V, q)
The chance of the next answer token, given the answer so far, the picture and the question. The model returns one number per word in its vocabulary.

Training pushes this product up on thousands of examples, by changing the weights. The picture tokens appear only as input. This is the whole trick: the language model never learns a new kind of loss. It learns to continue a sequence in which some of the early vectors came from an image.

Now the real question. If the recipe is this simple, what did SPHINX-X change? It changed what goes in at steps 1 and 2 (Chapters 2 and 3), how steps 3 to 5 are trained (Chapter 4), and what examples the model sees (Chapter 5).

In the LLaVA-style design used by SPHINX-X, how do image features reach the language model?

Chapter 2

Mixture of Visual Experts: Keep Two Encoders

Learn why SPHINX-X looks at every picture with two different vision encoders, and why four was too many

Two people stand at a shop window. One is a painter. She notices shapes, edges and textures. The other is a reader. She notices that the sign says “SALE”. Ask them both what they see and you learn more than from either one alone.

A vision encoder is a witness. Each one was taught in its own way, so each one notices different things. SPHINX, the model that SPHINX-X builds on, asked four witnesses about every image and combined their answers. This chapter explains why that helps, why it also hurts, and what SPHINX-X keeps.

The four witnesses in SPHINX

The paper names four encoders in the original SPHINX. We met the two families in Chapter 1. Here is where each one sits.

EncoderNetwork typeHow it was taughtKept in SPHINX-X?
CLIP-ViTVision transformerWeakly supervised: matches images with captionsNo
CLIP-ConvNeXtConvolutional networkWeakly supervised: matches images with captionsYes
DINOv2Vision transformerSelf-supervised: learns from images aloneYes
Q-formerQuery transformer on top of image featuresLearns to summarise the features into a few vectorsNo

The paper calls the group of encoders a mixture of visual experts (MoV). Each encoder reads the same view, and their outputs are mixed before they reach the language model. In SPHINX-X the group has two experts: CLIP-ConvNeXt and DINOv2.

The problem: cost multiplies

Why not keep all four? Because of Chapter 3. SPHINX does not look at a picture once. It cuts a high-resolution image into several sub-images and encodes every one, together with a small whole-image view. With the original setup that is 5 views.

Every encoder must run on every view. The count of encoder passes is simple arithmetic.

passes
How many times an encoder network runs on one image. This drives training and inference time.
E
The number of vision encoders in the mixture: 4 for SPHINX, 2 for SPHINX-X.
V
The number of views one image is cut into: 5 for a global view plus 2×2 sub-images.

For SPHINX that is 4 × 5 = 20 passes per image. For SPHINX-X it is 2 × 5 = 10. The saving is exact: half. The paper puts it in one sentence: the mixture of visual experts “inevitably leads to a significant increase in computational costs, especially for a group of sub-images generated from a high-resolution input”. Because SPHINX-Plus-2K uses 3×3 sub-images, the saving grows: 4 × 10 = 40 passes would drop to 20.

The passes are not the whole bill. Each encoder also produces feature vectors, and those pass through the projection and into the language model. The paper does not give a timing, so we count passes and say no more.

The choice: which two to keep

Cutting from four to two is easy. Choosing which two is the design question. The paper gives one reason. DINOv2 and CLIP-ConvNeXt are “pre-trained by distinct learning approaches (self-supervised vs. weakly-supervised) and network architectures (ViT vs. CNN)”. They differ on both axes. Together they “provide the most complementary and refined visual knowledge”.

Think of the two axes as a small map. One axis is how the encoder was taught. The other is what kind of network it is. If you keep two encoders that sit in the same corner, they will make the same mistakes. If you keep two that sit in opposite corners, one often fixes what the other misses. That is the whole argument.

Try it in the device. Switch encoders off and on, and watch two numbers: the passes, and how much of the map is covered.

Choose your witnesses

Press an encoder to keep it or drop it. The grid shows how it was taught and what network it is. The squares on the right are encoder passes.

The cost is a count of encoder runs, passes = encoders × views, with no timing: the paper reports none. The placement on the map follows the paper’s own labels for DINOv2 and CLIP-ConvNeXt; CLIP-ViT is placed from the CLIP and ViT papers it cites. The Q-former sits on top of ViT features, so it is drawn beside the map, not on it.

With all four on and 5 views the device reads 20 passes. Press SPHINX-X: the two it keeps and it reads 10 passes with both axes still covered. Now try keeping only CLIP-ViT and CLIP-ConvNeXt: the passes are 10 again, but both are weakly supervised, so the self-supervised corner is empty. That is the failure the paper avoids.

Symptom, measurement, fix

Symptom: training on high-resolution inputs is slow and the language model receives long sequences. Measurement: passes = E × V, so the count rises with every sub-image and every encoder. Fix: drop the encoders that repeat what the others know, and choose the survivors from opposite corners of the map.

One honest limit: the paper gives the reasoning but no experiment that removes each encoder in turn. The claim that two are enough is supported by the final benchmark scores (Chapter 7), not by a head-to-head test. A reader who wants proof must run that test.

SPHINX-X drops CLIP-ViT and the Q-former and keeps CLIP-ConvNeXt and DINOv2. What is the paper’s reason for keeping these two?

Chapter 3

Tiles and Skip Tokens

Cut a big picture into tiles, then stop paying for the tiles that hold nothing

You photograph a long receipt. The receipt is thin and tall, and the photo shows it in the middle of a mostly empty frame. You ask the model to read the total. The model needs fine detail, because the digits are small. It also gets a lot of blank space, and it still pays to look at all of it.

This chapter is about that waste. It has two parts. First, why SPHINX cuts images into tiles. Second, the paper’s small fix for the tiles that contain nothing at all.

Why tiles

An encoder such as CLIP is trained at a fixed size. A common choice is 224 × 224 pixels. A phone photo has millions of pixels. If you squash the photo down to 224 × 224, small text turns into a smear. The digits on the receipt are gone before the model starts.

SPHINX avoids this with multi-scale mixing. The paper describes it as dividing “the high-resolution input into several sub-images along with a downsampled image for concurrent encoding”. By default every image is scaled to fit and zero-padded (surrounded by black pixels, which are all zeros) to a square of 448 × 448. The square is then divided into four sub-images of 224 × 224. A separate small copy of the whole picture gives the model the big view.

Think of reading a map with a magnifying glass. The global image is the whole map held at arm’s length. Each sub-image is one quarter of the map, held close, with the print sharp. The model sees both. SPHINX-Plus-2K goes further: it raises the canvas to 672 × 672 and cuts it into 3 × 3 sub-images.

The waste: tiles that are all padding

Now take a picture that is twice as wide as it is tall (a ratio of 2 : 1). To fit in the square it is scaled to 448 wide and 224 tall. The remaining 224 rows are padding. Where the padding goes is not stated in the paper. We draw it at the bottom, so the bottom row of tiles holds only zeros.

The paper calls these fully-padded sub-images. It says they “contain noisy and useless visual signals” and “produce spare visual tokens that waste computational resources within both MoV and LLM”. Two costs, then. The encoders run on a picture of nothing. And the language model must read the tokens that come out.

The count is easy. Let r be the ratio of width to height, and g the number of tiles along one side. The image fills the top S/r of the canvas, which covers ceil(g/r) rows of tiles. Every row below it is fully padded.

padded
The number of sub-images made only of zero pixels.
g
Tiles along one side: 2 for the 448 canvas, 3 for the 672 canvas. The first g counts the tiles in each row.
g
The same g again, here counting the rows of tiles.
g
The same g again. g / r is how many rows of tiles the picture fills.
r
The width of the picture divided by its height. A square picture has r = 1.

Check it. With g = 2 and r = 2: ceil(2/2) = 1 filled row, so padded = 2 × (2 − 1) = 2 tiles. With g = 3 and r = 3: ceil(3/3) = 1, so padded = 3 × (3 − 1) = 6 of the 9 tiles. A three-to-one picture wastes two thirds of its sub-images.

The fix: a learnable skip token

The simple fix is to leave the empty tiles out. The paper’s skip token exists because that is not enough. Tiles reach the language model in a fixed reading order, tile after tile. If you delete the empty ones, the remaining list loses its shape. The model cannot tell whether the second remaining tile was top-right or bottom-left.

So SPHINX-X does something in between. It replaces each fully-padded sub-image with a learnable skip token. A skip token is one vector, the same width as any other token, that the model trains like a weight. It means “a blank tile sits here”. It costs one token, not hundreds. And it “provides explicit relative positional information for LLMs to identify the positions between useful sub-images”.

The vision encoders never see the blank tiles. The paper says the mixture of visual experts “can avoid encoding these zero-pixel sub-images”. That is the saving in encoder passes. The shorter sequence is the saving in the language model. Try it below.

Count the blank tiles

The square is the padded canvas. The blue area is your picture. Drag the shape from square to very wide, switch the split, and turn skip tokens on and off.

2.00 : 1

The count of padded tiles follows the paper’s description exactly. The padding is drawn at the bottom for clarity, since the paper does not say which side. The 256 tokens per tile and the 256 for the global view are toy numbers of ours; a skip token is 1 token. Passes assume the two encoders of Chapter 2.

Start at 2 : 1 with skip tokens off. Two of four tiles are red: 512 tokens of nothing in a 1,280-token sequence. Switch skip tokens on. The two red tiles become two pink tokens, the sequence falls to 770, and the encoder passes fall from 10 to 6. A square picture changes nothing. A 4 : 1 strip on the 672 canvas has six blank tiles, so almost everything is saved.

Symptom, measurement, fix

Symptom: wide or tall inputs, such as banners, receipts and screenshots, make training slower and sequences longer for no gain. Measurement: count the tiles whose pixels are all zero. The formula above does it from the aspect ratio alone. Fix: swap each such tile for one learnable token, and never send it to the encoders.

The paper gives no timing for this fix and no ablation, so we cannot say how much faster training became. We can say how many tokens and passes it removes, because that is arithmetic.

Why does SPHINX-X use a learnable skip token for a blank tile instead of just deleting the tile?

Chapter 4

One-Stage All-in-One Training

Replace two hand-tuned training stages with one run that trains everything except the eyes

A baker has a two-oven recipe. In the first oven, a small tray bakes at low heat for an hour. In the second, a different tray bakes at high heat, and a few loaves are blended afterwards. Which tray goes where, and at what heat, is decided by hand. When the recipe grows to forty ingredients, the baker spends more time choosing than baking.

That was SPHINX. This chapter explains what the two stages did, why they got in the way, and what SPHINX-X does in their place.

Three terms before we start

Training means changing a model’s weights so that its answers get better. A part of the model is frozen when its weights are not allowed to change. A part is unfrozen, or trainable, when they are. Freezing saves memory and keeps a pretrained skill safe. It also stops that part from adapting to the new job.

An instruction example is a request and its correct reply, such as “What is the total?” and “$42.10”. Instruction tuning trains the model on many such examples so that it follows requests. The learning rate is the size of each weight update. Too high and training blows up. Too low and nothing is learned.

What SPHINX did

Many MLLMs, including the LLaVA and LLaMA-Adapter families the paper cites, train in two stages. The first aligns the vision encoders with the language model: it teaches the new connector parts to speak the language model’s language, usually on image captions. The second tunes the model on instructions.

SPHINX followed this pattern and added a twist, weight mixing: it fused the first-stage weights of language models that had been tuned on different data domains. The paper’s complaint is practical. The old pipeline “requires to manually assign various tunable parameters and dataset combinations to different training stages, which is a labor-intensive task”.

What SPHINX-X does

SPHINX-X does one thing instead. The paper says it “treats all collected datasets equally and uniformly transforms them into multi-modal multi-turn dialog formats”. The recipe is to “unfreeze all the parameters of SPHINX (i.e., LLM and intermediate projection layers) except for the two visual encoders in MoV”. One run, one data pool.

Why can the authors skip the alignment stage? The paper’s answer: “Due to the large volume of training data and high reasoning capacity of LLMs”. In plain words, with enough varied data the connector learns its job while the language model is also learning, so a separate warm-up is not needed. The paper gives no experiment to test that claim, so read it as a bet that paid off in the final scores.

Two details tell you where each model starts. SPHINX-Plus is initialised from the earlier SPHINX model. The other SPHINX-X models start from the original encoders and the original language model, with randomly initialised linear projection layers, which means their numbers start as small random values.

The settings, with the reason for each

SettingValue in the paperWhat it does
TrainableLLM, projection layers (all except the two visual encoders)The eyes keep their pretrained skill; everything else adapts
Peak learning rate5e-6 for SPHINX-MoE, 2e-5 for the othersA smaller step for the large mixture of experts
Warm-uplinear, over the first 0.01 epochGentle start while random projection layers settle
Decaycosine, down to 0Big steps early, tiny steps at the end
OptimiserAdamW, weight decay 0, betas (0.9, 0.95)Per-weight step sizes from running averages
Batch size256 (effective)256 examples per weight update
ParallelismZeRO2 data parallel, Megatron model parallel size 8 (MoE), 2 (Plus), 1 (others)Splits optimiser state and layers over many GPUs

The learning-rate curve

The schedule has two parts. In warm-up the rate climbs in a straight line from 0 to its peak. After that it follows half a cosine wave down to 0. Write e for the training progress in epochs (one epoch is one pass over the whole data pool), ew = 0.01 for the warm-up, and E for the total number of epochs.

η(e)
The learning rate after e epochs. It scales every weight update.
ηpeak
The highest learning rate: 2e-5, or 5e-6 for the mixture of experts.
e / ew
How far through the warm-up we are, from 0 to 1.
ηpeak
The same peak, now multiplied by a factor that falls from 1 to 0.
(e − ew) / (E − ew)
How far through the decay we are, from 0 to 1. The cosine turns that into a smooth curve that starts flat, drops fastest in the middle and flattens again.

Worked example with a 1-epoch run and a peak of 2e-5. At e = 0.005, half way through warm-up, the rate is 2e-5 × 0.5 = 1e-5. At e = 0.505, half way through the decay, the cosine term is cos(π/2) = 0, so the rate is 2e-5 × ½ × (1 + 0) = 1e-5. At e = 1 it is 0. The paper does not say how many epochs it trains for, so the device lets you pick.

Scrub through one run

Choose a model family and a run length, then drag the slider through training. The boxes show what is trained. The curve shows the learning rate.

0.30 epoch

Peak rates, the 0.01-epoch warm-up and the cosine decay are the paper’s (Section 4.1). The number of epochs is our choice for the demo: the paper does not report it. The warm-up looks tiny on a 3-epoch curve because 0.01 epoch is 0.3% of the run.

Slide to the very start of the run: the rate is almost 0. Slide a hair to the right and it shoots up to the peak. Then it falls in a long, smooth curve. In the one-epoch case the peak is held for a single instant and then slides away. The warm-up matters because the projection layers begin as random numbers, and a large first step on random numbers can wreck the pretrained language model.

The cost of unfreezing

Unfreezing the whole language model is expensive. The paper cites ZeRO and Megatron for this reason. ZeRO (the paper’s reference) counts about 16 bytes of training state for every trainable parameter when training with Adam in mixed precision: the weights, their gradients, and the optimiser’s running averages. For a 13B language model that is roughly 208 GB, far more than one GPU holds. That is why SPHINX-Plus splits layers over 2 GPUs and the MoE over 8, and why the 1.1B model needs no split at all. This estimate is ours, from the ZeRO paper, not a figure the SPHINX-X authors report.

In SPHINX-X’s one-stage training, which parts of the model stay frozen?

Chapter 5

The Data Recipe

Turn very different datasets into one kind of conversation, and add two new sets for reading text and for pointing at marks

Two students prepare for the same exam. One reads only a book of landscape paintings. The other has a full timetable: maths, a language, a lab, a reading class. On exam day a question about a chemistry diagram arrives. Only one of them has seen anything like it.

That is the second gap from Chapter 0. This chapter lists what SPHINX-X studies, how every item is turned into the same shape, and the two datasets the authors built themselves.

Everything becomes a conversation

A dataset is a pile of labelled examples. The labels of different datasets look nothing alike. A detection set stores a box as four pixel numbers. A question-answering set stores a question and a short answer. A PDF set stores words and where they sit on the page. A model that has one input door, a chat, cannot eat these formats raw.

The paper’s answer is to rewrite every example as a multi-turn dialog: a User turn that asks and an assistant turn that answers. For vision data it writes: “We utilize a task-specific prompt as the question, and regard the ground-truth labels as the answer by textualizing them in language space.” Nothing special is added for any task. As the paper puts it, they “treat them all as pure language problems”. A box becomes the numbers [0.47, 0.44, 0.49, 0.50] written as text.

Why is that a good idea? The language model already knows how to write numbers and continue a sequence. A detection answer is one more sequence to continue. One loss, one input door, no new heads for each task. Try the converter.

Rewrite it as a chat

Pick a kind of data. The top panel is the raw annotation. The bottom panel is what the model is trained on.

Detection numbers, the receipt and its words are our own toy examples, made to show the conversion. The Set-of-Marks example and the way boxes appear as [x1, y1, x2, y2] between 0 and 1 follow the paper’s Figures 2 and 7. Sample counts are from the paper’s Table 9.

The public data, category by category

The paper groups its public data into three families, then adds two curated sets. All counts below are from its Table 9.

FamilyWhat is in itSamples
LanguageMulti-turn dialog (UltraChat, Flan-mini, OpenOrca) 1.8M; math (MetaMathQA, MathInstruct) 0.6M; code (WizardCoder) 80k2.48M
Vision tasksDetection (V3Det, OpenImages, LVIS, COCO, Object365) 4.9M; human pose 0.3M; ImageNet-1K classification 1M; grounding (Visual Genome, RefCOCO and kin, Flickr30k) 1M7.2M
Vision-languageQuestion answering (VQAv2, GQA, ChartQA, DocVQA and more) 0.7M; captions (COCO, ShareGPT4V) 0.5M; visual instruction (LLaVA, LVIS-Instruct4V, LLaVAR) 0.4M1.6M
OCR-intensive (new)PaperText pages from arXiv and Common Crawl PDFs 3M; layout and text spotting (DocBank, M6Doc, PubLayNet, DocLayNet, ICDAR, CTW1500) 1M4.0M
Set-of-Marks (new)GPT-4V captions of marked images across five domainsabout 10k

One choice stands out: the language data. Other MLLMs build on an instruction-tuned chat model such as Vicuna. SPHINX-X starts from the plain pretrained model, for example LLaMA2. The authors want to “investigate the training characteristics of multimodal models from LLMs more clearly”. So the language mixture must teach chat, maths and code from scratch, and it does so with 2.48M samples.

New set one: OCR from real PDFs

Older text-reading data was synthetic: clean words on plain backgrounds. The paper calls it “too simple and far from real-world application”. The authors gather PDFs from Common Crawl and arXiv. A tool called PyMuPDF renders each page as an image and saves every piece of text with its bounding box. Quality checks follow: a Unicode check and merging of split text pieces. The result, called PaperText, has about 3 million text-dense pages, rewritten as questions and answers.

Look at what this gives the model for free. The page image is the input. The text and where it sits is the label. No person typed a label, because the PDF already held the answer. That is how a recipe gets millions of examples cheaply.

New set two: Set-of-Marks

Set-of-Marks prompting is a trick for talking about parts of a picture. You draw numbered marks, such as points, boxes and outlines, on the image. Then you can say “describe Mark 1” instead of “the small thing to the left of the thing near the edge”. The paper notes that existing datasets lack this fine match between image regions and words.

To build the set, the authors take images from several domains and use their annotations to place the marks. They write domain-specific instructions and ask GPT-4V to describe the marked images at three scales: the whole image, each region, and the relations between objects. Then comes a detail that matters. During training SPHINX-X does not see the marked images. It sees the raw images, and the marks appear only in words in the conversation, for example “<Mark 1> [0.56, 0.24]”. So every data type shares one format. The set is small: about 10k samples.

Now the sizes. If you pooled every sample and drew uniformly, how much of each batch would each family fill? The paper does not say it samples this way, so the next device is a toy to build your feel for scale.

Draw a batch

The bars are the expected share of each family in the pooled data. Press the button to draw a batch of 256 and see what you actually got.

Toy assumption: samples are drawn uniformly from the pool of all 15.29M, so each family’s share is its count over the total. The paper says it treats all datasets “equally” and does not give a sampling rule. The counts are from Table 9; Set-of-Marks is the sum 5k + 1k + 2k + 1k + 1k.

Vision tasks, mostly detection, are 47% of the pool. OCR is 26%. The language data is 16%. The vision-language pairs that most MLLM recipes lead with are only 10%. Set-of-Marks is 0.07%: one sample in about six batches. Small data can still teach a skill, if no other source teaches it, but you would expect it to need care. The paper reports no ablation of its two curated sets, so their individual effect is not measured.

How does SPHINX-X train on object-detection data without any detection-specific output layer?

Chapter 6

Four Language Models and the Sparse Experts

Meet the family from 1.1B to 8×7B, and open the router that lets a big model run only part of itself

A workshop keeps several tools on the wall. A small screwdriver for a phone. A medium one for a laptop. A big wrench for an engine. The job decides the tool. Nobody argues that one tool is best.

SPHINX-X is a wall of tools. It uses one visual recipe (Chapters 2 to 5) and fits it to four language models. The paper then asks a plain question: when the recipe stays the same, what does a bigger language model buy?

The four models

ModelLanguage modelSizeWhy it is in the family
SPHINX-TinyTinyLlama-1.1B1.1BA light version of LLaMA, for fast use on mobile devices and small machines
SPHINX-Intern2InternLM2-7B7BPretrained on a large Chinese and English corpus, so it supports bilingual use
SPHINX-PlusLLaMA2-13B13BSame size as the original SPHINX, so it isolates the effect of the new data
SPHINX-MoEMixtral-8×7B8×7BA sparse mixture of experts, for strong reasoning

The Plus model is the cleanest experiment in the paper. The original SPHINX and SPHINX-Plus have the same 13B language model. The difference is the one-stage recipe and the much wider data. When Plus beats SPHINX (try the hero), the credit goes to data and recipe, not to size. There is one caveat: Plus starts from the SPHINX weights, so it has seen the old data as well.

There is also a sharper variant, SPHINX-Plus-2K. It raises the canvas from 448 × 448 to 672 × 672 and cuts it into 3 × 3 sub-images, as in Chapter 3. It helps text reading, as we will see in Chapter 7.

What a mixture of experts is

Each layer of a transformer language model has two main parts. Attention lets each token look at the others. The feed-forward network (FFN) is a small two-layer network that processes each token on its own. Most of a model’s parameters live in the FFNs.

A mixture of experts (MoE) keeps attention as it is and replaces the single FFN with eight. Each of the eight is an expert. A small network called the router reads each token and scores the eight experts. The paper says Mixtral “utilizes 8 feed-forward networks at each transformer layer as experts, and relies on a router network to activate two experts each time”.

The picture is a hospital reception. A patient arrives. The receptionist (the router) does not send the patient to all eight doctors. She picks the two who fit best. The two doctors give their opinions, and the receptionist blends them, trusting the better fit more. The six other doctors never see the patient.

Written as a formula, for one token with hidden vector x and the router’s chosen set S:

y
The layer’s output for this token: a vector the same size as x.
∑i∈S
Add up over the chosen experts only. S holds the k experts with the highest router scores. In SPHINX-MoE, k is 2.
gi
The weight that expert i gets. The weights of the chosen experts add up to 1.
Ei(x)
What expert i, a feed-forward network, outputs when it reads x.
gi = exp(zi) / ∑ exp(zj)
A softmax over the router scores z of the chosen experts, which turns scores into weights. This is the Mixtral design; the SPHINX-X paper itself only says that two experts are activated.

A worked example. The router scores one token as z = [0.3, 1.9, −0.5, 0.8, 2.4, −1.1, 0.1, 1.2]. The top two are expert 5 (2.4) and expert 2 (1.9). Their exponentials are e2.4 = 11.02 and e1.9 = 6.69, which sum to 17.71. So the weights are 11.02 / 17.71 = 0.62 and 6.69 / 17.71 = 0.38. The output is 0.62 × E5(x) + 0.38 × E2(x). Six experts did no work.

That is why MoE is attractive. The model holds all eight experts’ parameters, so it knows a great deal. But each token pays for only two feed-forward networks. In the paper’s phrase, the sparse mechanism lets Mixtral outperform with fewer active parameters.

Play with the router

The device below lets you route a token, change how many experts are active, and prune experts away. These are the three experiments the paper’s appendix runs on SPHINX-MoE.

Route a token

Pick a token. The bars are the router’s scores for the eight experts. Lit experts are the ones that run, with their weights. Press an expert to remove it.

2

The scores are made-up numbers of ours, to show the mechanism; real routers score different tokens differently and the paper publishes no per-token scores. The softmax-over-the-chosen-experts rule is Mixtral’s, not stated in the SPHINX-X paper. The paper trained SPHINX-MoE with 2 active experts and found no domain or modality specialists (Appendix A.1).

What the paper found about the experts

More active experts do not always help. SPHINX-MoE was trained with 2 active experts, and that is the default at inference. When the authors vary the count from 1 to 8, most benchmarks (ScienceQA, TextVQA, RefCOCO, MathVista) peak at 2, the training setting. MME peaks at 4. The paper calls this inconsistency “interesting” and offers no explanation.

Experts are not specialists. One might guess that one expert handles art and another handles text. The authors recorded expert use on three MME subtasks, artwork, celebrity and OCR, and on vision, language and mixed tokens. The usage looked alike across domains and across modalities. Their note: “the experts’ usage distribution of the layers at both ends of the model is more flat than that of the middle layers.”

Some experts matter more. The authors randomly removed 8 − n experts from every layer, three times for each n, and measured the scores. The three runs differed a lot. Remove one set and the model loses most of its ability. Remove another set of the same size and it keeps most. So some experts are “important”, and keeping those while pruning the rest could save most of the ability. Press the expert buttons above to see how routing shifts when one disappears.

In SPHINX-MoE, what does the router do for each token?

Chapter 7

What the Tables Show, and What They Do Not

Read the paper’s main tables benchmark by benchmark, then separate what is proved from what is only suggested

A scoreboard is easy to read and easy to misread. A team wins by ten points. Was the game fair? Did both sides play the same opponents? Did the winner change one thing or five? Before you trust a table, you must know what each number measures, and what was held fixed.

This chapter reads the SPHINX-X tables with those questions in mind. It also reports what the paper does not test.

What the six scores in the hero measure

POPE checks object hallucination: the model is asked yes or no whether an object is in the picture, including objects that are not there. A higher score means fewer invented objects. MMBench is a large set of multiple-choice questions across many visual skills. MM-Vet asks open-ended questions that combine skills, and a language model grades the answers. MathVista asks maths questions that need a figure or a chart. TextVQA asks questions whose answer is text written in the image. RefCOCO gives a phrase such as “the man on the left” and asks for the box around it. The score is the share of boxes that overlap the true box by at least half (“Top-1 Accuracy@0.5” in the paper).

Use the explorer below to compare any benchmark across the four SPHINX-X models and the baselines the paper lists. Rows marked with a dagger in the paper, which used ground-truth OCR text at test time, are left out.

Read one benchmark

Choose a benchmark. Warm bars are SPHINX-X models, grey bars are other open generalists, purple bars are specialist detectors. Then choose the SPHINX-X model to follow.

Scores are copied from the paper’s Table 1 (MMBench, MM-Vet, MathVista, POPE), Table 4 (TextVQA, which also lists a 2K-resolution Plus) and Table 5 (RefCOCO val). Only rows with a value for the chosen benchmark are drawn, and rows are a selection of the paper’s, not all of them. Models differ in base language model, data and resolution, so a bar is a comparison of systems, not of one idea.

Claim one: new data and recipe help, at the same size

The cleanest test is SPHINX-Plus against the original SPHINX. Both use LLaMA2-13B. On MMBench, Plus scores 71.0 against 67.1. On MM-Vet it scores 47.9 against 36.6, a gain of 11.3 points. On MathVista, 36.8 against 27.5. On TextVQA, 65.7 against 58.8. On RefCOCO val, 92.44 against 91.05.

But look at the whole row before you cheer. On POPE, Plus is lower: 89.1 against 90.8. On MME perception it is lower too: 1,457.7 against 1,560.2. So the new data did not lift every score. It lifted the skills that the new data teaches, such as maths, text reading and open-ended reasoning. It did not help object hallucination. The paper’s own words are careful: Plus “surpasses the original SPHINX” across the broad set. The honest summary is “most scores up, a few down”.

Claim two: bigger language models score higher

Read the hero from Tiny to MoE. MMBench: 56.6, 57.9, 71.0, 71.3. MathVista: 26.4, 35.5, 36.8, 42.7. The paper concludes that scaling up the parameters “can consistently boost” multi-modal understanding. That is true for the headline pattern, but the table shows exceptions. MoE scores lower than Plus on MM-Vet (40.9 against 47.9) and on RefCOCO val (90.64 against 92.44). It scores higher on MathVista and TextVQA (68.0 against 65.7).

There is also a confound. The four models use four different language models, pretrained by different teams on different text. Size changes, but so does everything else. A clean test would train one family at several sizes. The paper does not claim that test, and its Figure 1 plots one summed score for each model, not a controlled curve.

Beyond the main table

Maths and science. On MathVerse, SPHINX-MoE scores 15.6 overall, level with LLaVA-NeXT’s 15.6, and far below GPT-4V’s 41.0. On SciVerse it scores 37.3. The paper calls it the best open-source model on these two. That is a fair claim for open models and a far gap to GPT-4V.

Seeing without hallucinating. On MMVP, a test of basic visual understanding, SPHINX-MoE scores 49.3 against Gemini-Pro’s 40.7 and GPT-4V’s 38.7. The paper notes it beats GPT-4V on some other tasks too. On MMMU and CMMMU, which need college-level knowledge in many subjects, it does not. The authors blame the lack of multi-discipline data.

Video from an image model. SPHINX-Plus saw no video in training. To test it on Video-Bench the authors sample frames evenly and feed the middle one. It still outscores video-specific models on understanding and knowledge questions. It lags on tasks such as MOT, which depend on how things change over time. The authors give the obvious reason: the model has no video training.

Text reading. Plus-2K, with its 3×3 split of a 672 canvas, lifts TextVQA from 65.7 to 70.6 and DocVQA from 61.2 to 71.6. More pixels give small text a chance. The paper says the OCR data is “a limited portion”, so it calls the results competitive on text tasks, not leading.

What the paper does not test

Every claim of this paper rests on comparing finished systems. There is no ablation for any of the three changes. We do not know the score with four encoders, with skip tokens off, or with two training stages. We do not know the effect of the OCR set or the Set-of-Marks set alone. Training cost and speed are not reported either, so the efficiency story is about counts of passes and tokens, as in Chapters 2 and 3.

Two more cautions. Many test sets sit close to the public training sets, so overlap is a risk the paper does not measure. And part of the data, the Set-of-Marks captions, was written by GPT-4V, a closed model, so the open recipe leans on a closed one. The paper’s impact statement also flags misuse risks and bias in training data.

One small detail shows the authors’ care. In Figure 1 they leave MME perception out of the summed score, because its scale is so large that it would drown the rest.

SPHINX-Plus and the original SPHINX both use LLaMA2-13B. What does the comparison show?

Chapter 8

The Paper on One Page: A Field Guide

Keep the numbers, the terms, a checklist and the core code, and learn where to read next

SPHINX-X takes a working vision-language model and makes it simpler and wider. It keeps two vision encoders (CLIP-ConvNeXt and DINOv2) out of four. It replaces fully-padded sub-images with a learnable skip token. It trains every part except the encoders in one stage, on a pool built from public sets plus 3M PDF pages for reading text and about 10k marked images for pointing. Then it fits that recipe to four language models, from 1.1B to 8×7B, and shows that scores rise with data and with size. The proof is in the final tables. The paper includes no ablations of its parts.

Try the recipe yourself

This last device ties Chapters 2, 3 and 4 together. Turn each change on or off and count the cost for one 2 : 1 picture on the 448 canvas.

Old recipe or new

Each switch is one change from the paper. Press them one by one and read the counts.

Counts only, from Chapters 2 and 3: a 2 : 1 picture on a 2×2 split has 2 fully padded tiles, each tile gives 256 tokens (a toy number), a skip token is 1 token. The paper reports no timings or ablations, so nothing here is a speed-up measurement.

The numbers to remember

WhatValueWhere
Vision encoders kept2 of 4 (CLIP-ConvNeXt, DINOv2), frozenSec. 3.2
Canvas and tiles448×448 into four 224×224 plus a global view; Plus-2K: 672×672 into 3×3Sec. 3.2, 3.4
Trainingone stage; LR 2e-5 (5e-6 for MoE); warm-up 0.01 epoch, cosine to 0; AdamW (0.9, 0.95), weight decay 0; batch 256Sec. 3.2, 4.1
Data poollanguage 2.48M, vision 7.2M, vision-language 1.6M, OCR 4.0M, Set-of-Marks about 10kTable 9
FamilyTinyLlama-1.1B, InternLM2-7B, LLaMA2-13B, Mixtral-8×7B (2 of 8 experts active)Sec. 3.4
MMBench56.6, 57.9, 71.0, 71.3 (Tiny, Intern2, Plus, MoE); SPHINX 67.1Table 1
MM-Vet23.8, 36.5, 47.9, 40.9; SPHINX 36.6Table 1
MathVista26.4, 35.5, 36.8, 42.7; SPHINX 27.5Table 1
TextVQA57.8, 58.1, 65.7, 68.0; Plus-2K 70.6; SPHINX 58.8Table 4
RefCOCO val82.89, 86.08, 92.44, 90.64; SPHINX 91.05Table 5

Terms in one line each

TermPlain meaning
MLLMA language model that also reads images, by turning image features into extra tokens
MoVMixture of visual experts: two vision encoders whose features are mixed
Sub-imageOne tile cut from the padded canvas and encoded on its own
Skip tokenA learnable vector that stands in for a tile of pure padding
One-stage trainingOne run, one data pool, all parameters trained except the encoders
SoMSet-of-Marks: numbered marks on an image so words can point at regions
MoEMixture of experts: a router picks 2 of 8 feed-forward experts for each token
RECReferring expression comprehension: find the box a phrase describes

A checklist for re-implementing it

  1. Encoders. Load CLIP-ConvNeXt and DINOv2. Freeze both. Check you run each once per view.
  2. Tiling. Scale and zero-pad to 448 (or 672). Cut into a grid. Add one downsampled global view.
  3. Skip tokens. Test each tile for all zeros. Send only the others to the encoders. Put one learnable vector in each empty slot.
  4. Projection. Join the two encoders’ features. Map them to the language model width. Start the weights random.
  5. Data. Rewrite every set as User and assistant turns. Boxes become numbers in text. Build the PDF OCR set and the mark captions.
  6. Training. Unfreeze everything but the encoders. Use the peak rate, warm-up and cosine decay above. Shard the optimiser state.
  7. Models. Repeat for the four language models. For Mixtral, keep 2 active experts.
  8. Evaluate. Run the benchmarks, and add the ablations the paper leaves out.

The core as code

This sketch follows the paper’s description. Where the paper is silent, a comment says what we assumed.

pythonimport torch, torch.nn as nn, torch.nn.functional as F

def tile(img, g=2, S=448):                 # img: (3, S, S), scaled to fit and zero-padded
    t = S // g                                  # tile side: 224 when g = 2
    tiles, empty = [], []
    for j in range(g):
        for i in range(g):
            p = img[:, j*t:(j+1)*t, i*t:(i+1)*t]       # (3, 224, 224)
            tiles.append(p)
            empty.append(bool((p == 0).all()))      # fully padded sub-image?
    glob = F.interpolate(img[None], size=(t, t))[0]   # global downsampled view
    return glob, tiles, empty

class SphinxX(nn.Module):
    def __init__(self, llm, clip, dino, d_vis):
        super().__init__()
        self.clip, self.dino, self.llm = clip, dino, llm
        for p in [*clip.parameters(), *dino.parameters()]:
            p.requires_grad = False               # the two visual experts stay frozen
        d = llm.config.hidden_size
        self.proj = nn.Linear(d_vis, d)            # random init, trained
        self.skip = nn.Parameter(torch.randn(1, d) * 0.02)   # learnable skip token

    def visual_tokens(self, img, g=2):
        glob, tiles, empty = tile(img, g)
        views = [glob] + [t for t, e in zip(tiles, empty) if not e]   # blank tiles never reach the encoders
        x = torch.stack(views)                      # (V, 3, 224, 224)
        with torch.no_grad():
            f = torch.cat([self.clip(x), self.dino(x)], -1)   # (V, n, d_vis); assumed: channel concat, as in SPHINX
        z = self.proj(f)                           # (V, n, d)
        out, k = [z[0]], 1
        for e in empty:                             # rebuild the grid in reading order
            if e: out.append(self.skip)              # (1, d), one token per blank tile
            else: out.append(z[k]); k += 1
        return torch.cat(out, 0)                   # (N_pic, d)

def lr_at(e, peak=2e-5, ew=0.01, E=1.0):       # warm-up, then cosine to 0
    import math
    return peak*e/ew if e < ew else peak*0.5*(1+math.cos(math.pi*(e-ew)/(E-ew)))

def train_step(model, opt, pic, ids, labels, e):
    for grp in opt.param_groups: grp["lr"] = lr_at(e)
    emb = model.llm.get_input_embeddings()(ids)      # (T, d)
    seq = torch.cat([model.visual_tokens(pic), emb], 0)[None]   # (1, N, d)
    logits = model.llm(inputs_embeds=seq).logits
    # labels = -100 on picture and question tokens (assumed: standard instruction tuning)
    loss = F.cross_entropy(logits[0, :-1], labels[1:], ignore_index=-100)
    loss.backward(); opt.step(); opt.zero_grad()
    return loss.item()

# AdamW over everything that is not frozen: LLM, projection, skip token
# opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad],
#                         lr=2e-5, betas=(0.9, 0.95), weight_decay=0.0)

def moe_ffn(x, router, experts, k=2):          # x: (T, d), router: Linear(d, 8)
    top, idx = router(x).topk(k, dim=-1)       # (T, k) scores and expert ids
    w = F.softmax(top, dim=-1)                 # weights over the chosen experts only (Mixtral)
    y = torch.zeros_like(x)
    for j in range(k):
        for n, ex in enumerate(experts):
            m = idx[:, j] == n
            if m.any(): y[m] += w[m, j, None] * ex(x[m])
    return y

Where it sits in the field

SPHINX-X is an open, engineering-heavy member of the LLaVA family. It shares the simple linear bridge with LLaVA 1.5, uses the encoders from CLIP and DINOv2, builds on a transformer language model and the vision transformer, and borrows its largest model from mixture of experts. For the design choices that matter at this scale, compare MM1, which studies the same questions with ablations. For other ways to join a vision model to a language model, read LLaMA-Adapter and Kosmos-2. This paper is a reading of the lecture on large multimodal models.

The takeaway. SPHINX-X shows that a strong open multi-modal model needs fewer parts, not more: two encoders, a token that stands for a blank tile, one training stage, and a wide data pool that turns everything into text. Then it gets better as you add data and parameters. What remains open is which of these parts matters most, because the paper never removes them one at a time.

Now press Present or Teach and explain the recipe from memory: why two encoders, what a skip token saves, what is frozen, how a box becomes text, and what a router does. Then go back to the scoreboard at the top, press Plus and SPHINX, and say which gains came from data and which would need a bigger model.

Credit: SPHINX-X is by Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Yu Qiao, Hongsheng Li and Peng Gao (Shanghai AI Laboratory, CUHK MMLab and others), ICML 2024. arXiv 2402.05935. Code and models: LLaMA2-Accessory. This lesson re-teaches the paper in our own words and devices.

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
Read the paper · Code and models · The MIT MMAI course · Back to the lecture