NExT-GPT lets one language model take in and answer with any mix of text, image, video and audio, while training only 1% of its weights. Pick a request and watch what crosses the gap.
Learn how a chat model can end up making music, pictures and video, by handing a few learned vectors to decoders it never retrains.
Choose a request, then press Send the request with each system. See what reaches the decoder: a caption in plain words, or signal tokens that training can reach. Then we build, piece by piece, every part of the paper.
You need a rough idea of what a chat model does (it writes text one piece at a time) and what a vector is (a list of numbers). We build the rest from zero.
Based on NExT-GPT: Any-to-Any Multimodal LLM by Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji and Tat-Seng Chua (NExT++ Research Center, National University of Singapore), ICML 2024. An assigned reading of MIT MMAI Lecture 8, Large Multimodal Models (course schedule). Every idea is re-taught here in our own words and devices.
Illustrative flow, drawn from the paper's Figures 1, 3 and 4. The three requests are the paper's own qualitative examples. Signal-token counts are 4 for image, 8 for audio and 24 for video (Section 6.2 says images and audio need 4 and 8 and video the most; Section 4.2 lists [IMG0] to [IMG4], [AUD0] to [AUD8] and [VID0] to [VID24]). The width 4,096 is Vicuna-7B's hidden size, our addition, since the paper prints no shapes. The captions are made up for the toy.
Chapter 0
See what a chat assistant cannot do today, and why passing text between tools loses what you wanted
You record a short video of a dog riding a skateboard. You send it to an assistant with one line: “What is unusual here? Then add some funny music, and show me similar scenes.” You want a spoken answer, a sound clip and a picture back, all in one reply.
Most assistants of that era could handle the first half. A model such as LLaVA reads an image and writes text about it. It listens, so to speak, in many modalities. It answers in one: words. A modality is a kind of signal: text, an image, a video or a sound clip. The paper calls these models MM-LLMs, multimodal large language models: a language model that has been taught to read modalities besides text.
The second half needs something different. The assistant must also make a sound clip and a picture. The authors of NExT-GPT call a system that can read any mix and write any mix any-to-any. That is the goal of the paper: accept input in any modality, and deliver a response in any modality that fits.
The paper sorts the earlier work into four groups. The first group only understands: Flamingo, BLIP-2, MiniGPT-4, Video-LLaMA, LLaVA, PandaGPT, SpeechGPT. They put the picture, the video or the sound in, and they get words out. Nearly all work at that time was of this kind, so the output side was nearly empty.
The second group adds some writing. Emu, DreamLLM, GILL and SEED can produce an image next to their text. The paper notes that they stay inside interleaved text and images. There is no sound, and there is no video.
The third group is the pipeline. Visual-ChatGPT, HuggingGPT and AudioGPT use a language model as a manager. The manager reads your request and calls ready-made tools: an image generator, a speech model, a captioner. In theory a pipeline reaches any-to-any, since there is a tool for every modality.
The fourth group is a single model, CoDi. It can read and write any mix of modalities at once, using diffusion models. But it has no language model at its centre. So it cannot reason about an implied wish such as “I am in a bad mood.” The paper says it can only deliver simple paired generation.
Rows are what you give the system. Columns are what it can give back. A lit cell means the pair works. Choose a kind of system and count the lit cells, then look at the two checks underneath.
Our summary of the paper's Introduction and Related Work, one modality at a time (text, image, video, audio). It ignores quality. Understand-only systems read more than they write: we lit text, image, video and audio inputs for them, as PandaGPT does, though a given model may read fewer.
The pipeline looks like the answer, so why did the authors build something else? They name two flaws, and both sit in the hand-off between the language model and the tool.
The first flaw is noise. In a pipeline, everything passes as text. To make your sound clip, the manager writes a sentence such as “a dog rides a skateboard, comic mood”, and a sound tool reads only that sentence. A caption is a lossy summary. The tool never sees what the manager saw. And a second hop adds a second summary, so mistakes pile up. The paper calls this a cascading process that propagates errors.
The second flaw is that nothing is trained together. In a pipeline, the manager and the tools are used only for inference, which means running a model without changing it. No loss, the number that training lowers, ever measures the whole chain. So the manager never learns which sentences make the sound tool do a good job. It cannot get better at the implicit requests, where you hint and do not ask.
Keep that sentence in mind. It has three parts, and each is a chapter. “A few learned vectors” are the signal tokens (Chapter 4). “Trains end to end” is the three-stage recipe (Chapters 5 and 6). “Only 1%” is the plan that reuses frozen parts (Chapter 2).
Go back up to the hero and try both systems on the same request. The relay hands over a short caption. NExT-GPT hands over vectors, and a dotted line returns the learning signal. The rest of this lesson builds every box in that picture.
Chapter 1
Learn embeddings, shared spaces, diffusion decoders and projections, the four parts the paper snaps together
Imagine a post office with one rule: every parcel must be addressed with a list of numbers. A letter, a photo and a song all get an address. Parcels with similar content get addresses close together. If you can write the address, you can find the neighbours. That picture carries most of this chapter.
A language model cannot read letters. It first cuts text into tokens, small pieces such as a word or a part of a word. Then it swaps each token for a vector, a list of numbers, usually a few thousand long. That vector is the token’s embedding: its address in meaning space. The width of the list is the model’s hidden size. For the Vicuna-7B model that NExT-GPT uses, we take this to be 4,096 (the paper does not print it; it is a property of that model family). Every layer inside the model reads and writes lists of that width.
This gives a trick that the whole paper depends on. The model does not care where a vector came from. If you can produce a list of 4,096 numbers that sits in a sensible place, the model will treat it like a token. A picture, once turned into such lists, can be “read” by the same model that reads words.
An encoder is a model that turns a raw signal into vectors. A typical image encoder cuts the picture into a grid of small squares called patches, and gives each patch a vector. The paper calls the result a patch-level grid of features. A video encoder does the same over frames. An audio encoder does it over short slices of a spectrogram, a picture of sound.
The paper uses one encoder family, ImageBind, for image, video and audio. ImageBind was trained so that different modalities land in one shared space: a photo of a dog, the word “dog” and the sound of barking get nearby addresses. The authors like it because they are spared from managing many different encoders. Use the device to see why shared space matters.
Nine items, three concepts, three modalities. Each arrow starts at a word and ends at the nearest sound. Drag the alignment from 0 to 1 and watch the arrows turn from wrong to right.
Toy: nine hand-placed points in two dimensions, three concepts by three modalities. Square is a word, circle an image, triangle a sound. Alignment slides every point linearly from its modality cluster to its concept cluster. The nearest sound to each word is computed live. Real embeddings have thousands of dimensions and ImageBind learns them by contrastive training on image-paired data.
At alignment 0, the sounds sit with the sounds and the words sit with the words. The nearest sound to a word is a matter of luck. At alignment 1, items with the same meaning sit together, whatever their modality. That is what a shared space gives you: you can pick a vector from one modality and meet its meaning in another.
A diffusion model makes a picture by removing noise. Think of a sculptor who starts with a rough block and chips for fifty rounds. The model starts with pure random noise. At each round, a network (the U-Net) predicts which part of the picture is noise and subtracts a little. After many rounds a clean picture remains.
A latent diffusion model does this in a compressed space, not on pixels, which is much cheaper. The compressed picture is called the latent, and a separate decoder inflates it back to pixels. The paper uses three of these, all ready-made: Stable Diffusion 1.5 for images, Zeroscope for video and AudioLDM for audio.
Left alone, the model paints a random picture. To ask for a particular one, you give it a condition: a set of vectors the U-Net looks at while it denoises. In these decoders, the condition comes from a text encoder that turns a prompt into vectors. This is the one door into the decoder, and it expects text-shaped vectors. NExT-GPT will have to push its own vectors through that door.
Training such a model uses one simple loss. Add random noise to a clean latent, then ask the network to name the noise.
Worked, in words: noise a latent at t = 600, hand the network the noisy latent and a condition for “a dog”, and it guesses the noise. The squared error of the guess is the loss. A condition that matches the picture makes the guess easier.
Now the plumbing problem. ImageBind’s vectors live in one space. Vicuna’s token vectors live in another. Stable Diffusion’s condition lives in a third. They do not match, and training all three together would cost a fortune.
The cheap fix is a projection layer: a small network that maps vectors from one space to another, like a plug adapter between two countries’ sockets. You freeze the big models on both sides, and train only the adapter, so the adapter learns the translation. NExT-GPT needs two kinds: an input projection from ImageBind to Vicuna, and an output projection from Vicuna to each decoder.
Chapter 2
Follow a request through the encoders, the LLM and the decoders, and count which weights change
Picture a relay race with three runners. The first runner turns your photo, your clip or your voice into a form the second runner can read. The second runner thinks and decides what to say and what to make. The third runner turns that decision into pictures, sound or video. NExT-GPT has exactly three such tiers, and the paper’s Figure 1 draws them as three columns.
Text goes straight to the LLM, since the LLM already reads tokens. Image, audio and video go through ImageBind, the shared encoder from Chapter 1, which has about 1.2 billion weights. It returns a grid of patch vectors. An input projection then turns that grid into a short list of vectors the LLM can read. The projection is a block with about 28 million weights, and it is the subject of Chapter 3.
The output of this tier is a list of vectors, each as wide as a Vicuna token embedding. Chapter 3 shows that the list is short: a few concept tokens, not hundreds of patches. These vectors are placed in the LLM’s input, in the same row as ordinary word tokens.
The core is Vicuna-7B (version 0), an open chat model with about 7 billion weights. It is a plain text model that the authors did not rebuild. It reads the row of word vectors and projected vectors, reasons over them, and writes a reply. The reply contains two kinds of tokens. The first kind is ordinary text. The second kind is new: modality signal tokens, special tokens that mean “now make an image” or “now make a sound”. Chapter 4 is about these.
So the LLM is both the reader and the director. The paper calls it the core agent. It is the only tier that decides whether to generate and what.
For each output modality, the signal tokens pass through an output projection, a small transformer of about 31 million weights. Its job is to turn the LLM’s hidden states into the condition vectors that one specific decoder expects. The decoders are the three ready-made diffusion models: Stable Diffusion 1.5 for images (1.3B), AudioLDM for audio (975M) and Zeroscope for video (1.8B).
The paper’s Table 1 lists all of it. Here it is in one place.
| Part | Weights | Status |
|---|---|---|
| ImageBind encoder | 1.2B | frozen |
| Input projection (grouping) | 28M | trained |
| Vicuna-7B | 7B | frozen, plus LoRA |
| LoRA adapters in the LLM | 33M | trained |
| Output projection, image | 31M | trained |
| Output projection, audio | 31M | trained |
| Output projection, video | 32M | trained |
| Stable Diffusion (image) | 1.3B | frozen |
| AudioLDM (audio) | 975M | frozen |
| Zeroscope (video) | 1.8B | frozen |
Add the trained parts: 28 + 33 + 31 + 31 + 32 = 155 million. Add the frozen parts: 1.2 + 7 + 1.3 + 1.8 + 0.975 = 12.275 billion. The trained share is 155 / (155 + 12,275) = 155 / 12,430, which is about 1.25%. The paper rounds this to “only 1%”.
Use the device to see how small the trained part is, and to try a what-if: switch off an output modality and watch the share change.
Top bar: every weight in the system, to scale. The thin purple slice is what trains. The lower bar zooms into that slice. Switch output modalities on and off.
Sizes are Table 1 of the paper. The what-if is ours: it drops that modality's output projection and decoder and keeps the encoder, the input projection, Vicuna and LoRA. The paper trains the full system, with all three outputs.
The paper does not print tensor shapes, so these are our reading, with the Vicuna width of 4,096 as the one fixed number. Take a clip of audio.
Nothing in this chain is text except the LLM’s own words. That is the difference from the relay in Chapter 0: the hand-off between tiers 2 and 3 is a block of vectors, not a sentence.
The paper gives two reasons. The first is cost: training the encoders, the LLM and three diffusion models from nothing would be very expensive, so the authors load off-the-shelf weights and avoid a cold start. The second is growth: adding a fourth modality means adding one encoder path and one decoder path with their projections. The core stays as it is.
There is a price. A frozen decoder can only do what its condition door allows, so the whole system inherits its decoders’ quality, and its flaws. Chapter 8 returns to this.
Chapter 3
Group hundreds of patch vectors into a few concept tokens, and teach a frozen LLM to read them
Look at the dog on the skateboard. You do not see 256 little tiles. You see a dog, a board, a road and a sky. Your mind groups tiles into things. A word in a sentence works the same way: the word “dog” stands for one whole concept.
An image encoder does not group. It hands over one vector per patch, a long grid. The paper’s argument is that these patch units “might not best coincide with the intricate textual token semantics”. A patch is a tile. A text token is a concept. Feeding tiles to a model that thinks in concepts may lose information.
Earlier systems used two designs. A linear layer, as in LLaVA, maps every patch vector straight into the LLM’s space. The LLM then gets as many tokens as there are patches. A Q-Former, as in BLIP-2, uses a few learned query vectors that look at the patches through attention and report back. NExT-GPT adds a third design, a grouping mechanism: learned concept tokens claim patches by similarity, stage after stage, and the claimed patches are merged into the concept token. The paper borrows the idea from GroupViT (Xu et al., 2022).
Say the encoder gives N patch vectors, X. We start a stage with M learnable concept tokens, C, set at random. They are weights: training will shape them. Four moves follow.
Move 1: think together. Glue the concept tokens and the patches into one row and run a transformer layer over the row. Now each concept token has looked at the patches, and each patch has looked at the concept tokens.
Move 2: score every patch against every concept. Normalise each vector to length 1, so a dot product is a cosine: 1 means same direction, 0 means unrelated. Add random Gumbel noise, divide by a temperature τ, and apply a softmax over the concepts.
Move 3: choose, but keep the gradient. Each patch joins its single best concept. That choice is an argmax, a hard one-hot vector, and an argmax has no useful gradient: nudge a weight a little and the pick either stays or flips. The fix is the straight-through trick. In the forward pass use the hard pick. In the backward pass pretend you used the soft table A.
Worked: a patch scores [0.9, 0.2] against two concepts. At τ = 0.5 and no noise, A = [e1.8, e0.4] / (6.05 + 1.49) = [0.80, 0.20]. The forward value of  is [1, 0]. The backward pass sees [0.80, 0.20].
Move 4: merge. Each concept token collects the patches that chose it and keeps a residual copy of itself: Xl+1 = Ĉl + MLP(Âl, X̂l). The paper writes the merge only in this form, and we read it as GroupViT does: sum the chosen patches, then pass the sum through a small MLP. After L such stages, the model holds ML concept tokens, and those go to the LLM. The paper does not print the counts Ml.
A 6 by 6 grid of patches becomes four concept tokens, A to D. Tap a patch to see its scores. Turn the noise on and press Draw new noise. Then move τ and see what it changes.
Toy: patch directions follow a smooth made-up field, so neighbouring patches agree. Cosine scores, Gumbel noise and the softmax are computed exactly as in the equations, in two dimensions with four fixed concept directions. In the paper the concept tokens are learned and there are several stages.
Try it. With the noise off, the grid splits into four clean bands, because the argmax is a pure function of the scores. With the noise on, patches near a boundary flip between neighbouring concepts each time you draw. Now move τ. The pick does not change, since dividing by a positive number never changes which score is largest. What changes is the soft table A, the one that carries the gradient. So τ controls how sharp the training signal is, not the choice itself.
The block starts with random weights, so it needs a lesson. The authors use captioning: show an image, a video or an audio clip, and ask the frozen LLM to write its caption. The loss is ordinary cross-entropy on the caption words, which measures how surprised the LLM is by the true caption. Only the input projection is trainable. The paper puts it well: this stage trains “a compatible multimodal tokenizer for the frozen LLM”.
The data are pairs: CC3M for images (over 3 million image and caption pairs), WebVid-2M for video (short stock clips with descriptions) and AudioCaps for audio (about 46,000 clips with human captions). After this stage, an image becomes a short row of concept tokens that the LLM can describe. The LLM was never changed.
Chapter 4
Let a text model choose whether to draw, play or film, by writing a few special tokens into its reply
You order dinner at a busy restaurant. The waiter does not cook. He writes a ticket. Some of the ticket is for you: “Of course, one moment.” Some of it is a slip for the kitchen: “grill, medium rare.” A slip that is not written means that station stays quiet.
NExT-GPT’s language model does the same. It writes its reply as a row of tokens. Most are ordinary words for you. Some are modality signal tokens: special tokens that act as slips for the decoders. The paper adopts the idea from GILL (Koh et al., 2023).
The paper adds three families to the LLM’s vocabulary: image tokens [IMG0] and onward, audio tokens [AUD0] and onward, and video tokens [VID0] and onward. The paper writes the families as [IMGi] for i from 0 to 4, [AUDi] for i from 0 to 8 and [VIDi] for i from 0 to 24. Counted from zero, that is five, nine and twenty-five labels, while the later discussion speaks of 4 and 8 tokens. The paper does not reconcile the two, so we use 4, 8 and 24 as round working numbers. The discussion of Figure 6 says images work well with 4 tokens and audio with 8, and that video needs the most.
The rule for turning decoders on is simple and the paper states it directly. If the LLM decides that a modality should be produced, it writes that family’s tokens. If it writes none, that modality is off. A single reply can switch on several decoders at once, such as audio and image.
A signal token is a word with no dictionary meaning. What matters is the vector the LLM computes at that position: its hidden state, a list of 4,096 numbers after the final layer. The LLM reads the whole conversation before it writes a signal token, so its hidden state can hold the whole request: “a funny, light tune for a dog on a skateboard”. Compare this with the relay in Chapter 0, which squeezes the same request into one short caption.
These hidden states are what the output projection receives (Chapter 5). The paper says the tokens “implicitly carry rich and flexible instructions for the downstream diffusion model”. They are instructions in a learned code that no human wrote down.
Pick what the user says. The strip shows the tokens the LLM writes, with signal tokens boxed in yellow. The lamps show which decoders get a slip. Then change how many image tokens the LLM writes.
The replies are made up for the toy; the mechanism (a family of tokens switches its decoder on, no token switches it off) is the paper's, Section 4.2. Token counts are 8 for audio and 24 for video; the image count is your slider (the paper's Figure 6 tests 1, 2, 4 and 8 and finds 4 enough). Widths of 4,096 are Vicuna-7B's, our addition.
Choose the video question first. The LLM writes only words, so no lamp lights. Nothing is made, and nothing should be. Now choose “A bad mood”. The user never asked for anything. Yet in the paper’s Figure 4 example, the model replies with sympathy and then decides to show a puppy video. That choice is the slips: video tokens appear at the end of the reply. The decision to generate is part of the model’s reasoning, not a rule written by hand.
On the system side, routing is a few lines. The program scans the LLM’s output for signal tokens and collects their hidden states, one batch per modality. This is a sketch of the idea, not the authors’ code.
pythonSIGNALS = {"image": "[IMG", "audio": "[AUD", "video": "[VID"} def route(tokens, hidden): # tokens: list[str] the LLM wrote; hidden: [len(tokens), 4096] final-layer states jobs = {} for modality, prefix in SIGNALS.items(): idx = [i for i, t in enumerate(tokens) if t.startswith(prefix)] if idx: # token present: this decoder is ON jobs[modality] = hidden[idx] # [k, 4096] goes to the output projection return jobs # empty dict: a text-only reply
The tokens are new, so the LLM must learn them. It learns them from examples where the right reply already contains them. In the alignment stage, the paper takes captions from CC3M, WebVid and AudioCaps and treats the caption as the text part of a reply, then appends the signal tokens as the thing to predict. The first loss term is the negative log-likelihood of producing those signal tokens: ordinary next-token cross-entropy. In the final instruction-tuning stage, the gold replies in the MosIT dialogues also carry signal tokens in the right places (Chapter 6).
Training therefore teaches two things. It teaches when a reply should include a slip, and it teaches what to put in the slip’s hidden state. The second is the job of the losses in Chapter 5.
The symptom: you ask for a picture and receive only words. Measure it by counting replies that contain no signal token on prompts where a human would expect one. The fix is data. The paper builds the MosIT dialogues, where users hint and the machine chooses to generate, so that the model sees many cases of implied requests with the correct slips.
Chapter 5
Turn signal-token states into the condition a diffusion model expects, using three losses at once
Suppose you hire a painter who only takes orders written in French. You speak English. You do not want to retrain the painter, and you do not want to learn French by heart. So you hire a translator, and you test the translator by asking: does the French you wrote look like the French a native speaker would have written for the same job?
That is the problem on the output side. The diffusion decoder is the painter. Its condition door expects vectors from its own text encoder, which is the French. The LLM’s signal-token states are the English. The output projection is the translator, and it is the only part of this tier that trains.
The paper describes it as a transformer with a hidden size of 512, 4 attention heads, 4 encoder layers and 4 decoder layers, with dropout 0.1. Its Figure 2(b) shows a linear layer, an encoder, a decoder fed with learnable queries, and a linear layer out. We read the flow as follows. The k signal states (each 4,096 wide) go in through a linear layer down to 512. The encoder mixes them. The decoder starts from a fixed number of learnable query vectors, a set of blank slots, and fills them by looking at the encoder output. A last linear layer lifts each slot to the width the decoder wants. The number of slots matches the length of the decoder’s condition. That is how a handful of signal tokens can become a long condition. There is one projection per modality, of 31, 31 and 32 million weights.
Stage 2 trains the projection with a loss made of three parts. The paper lists them in one sentence; here they are one by one.
Loss 1, the signal-token loss. This is the cross-entropy from Chapter 4. Given the caption, the LLM should write the signal tokens. It teaches when to speak.
Loss 2, the caption alignment loss. Take the real caption of the training image and run it through the decoder’s own text encoder. The result is the condition the decoder would receive from a text prompt. We call it the target. Now ask for the projected signal tokens to be close to it, in squared distance. This is the shortcut: the decoder already understands the target, so a projection that matches it works out of the box.
Worked: suppose P(h) = [0.2, 0.9, −0.1] and the target is [0.5, 0.7, 0.0]. The difference is [−0.3, 0.2, −0.1]. Square each: 0.09, 0.04, 0.01. The loss is 0.14. Training pushes it toward 0.
Loss 3, the conditional denoising loss. This is the loss of Chapter 1, with one change. The condition c is now the output of the projection, not the text encoder. Take the real picture, add noise, and ask the frozen U-Net to name the noise while it looks at the projected signal tokens. The U-Net weights do not change, but the error still flows backwards through it into the projection. The picture, and not the caption, is the judge.
Why both loss 2 and loss 3? Loss 2 is fast and clean, but it only asks the projection to copy a text embedding, and a text embedding is only as good as the caption. Loss 3 asks the real question, does the decoder paint what was wanted, but it is noisy and slow to learn from. The paper uses loss 2 “to accelerate alignment learning” and loss 3 to keep the result honest.
Blue rings are the targets the text encoder would give. Purple dots are the projection of each signal state. Press Run and watch the dots slide onto the rings as the caption loss falls.
Toy: six made-up two-number signal states and targets, and a 2 by 2 projection trained by exact gradient descent on the mean of the caption loss. The real projection is a 31-million-weight transformer and also feels the denoising loss through the U-Net, which this toy leaves out.
Set the learning rate very high and press Run. The dots overshoot, and the loss may rise before it falls, or may blow up. Set it very low and the descent is slow. This is the usual rhythm of gradient descent, and it applies to the real projection.
The symptom: the decoder paints something plausible that ignores the detail you asked for, a generic dog, not a dog on a skateboard. Measure it with a score that compares the output to the prompt, such as CLIPSIM for video, or by counting the signal tokens you give. In Figure 6 the paper finds that images and audio are fine with 4 and 8 tokens, but video needs many more. The fix is more tokens, more or cleaner data, or a stronger decoder. The authors note that the best number depends on the data and on the decoder.
Chapter 6
Follow the training recipe stage by stage, and see why the last stage needs dialogues that switch modalities
Learning to drive has three phases too. First you learn to read the road signs. Then you learn the pedals and the wheel. Last you drive in real traffic with a coach beside you. You would not start in traffic. Each phase trains a different skill and relies on the earlier ones.
NExT-GPT is trained the same way, in three stages, and the paper’s appendix spells them out. The device below shows which blocks are trainable in each one.
Purple blocks train. Grey blocks are frozen. Choose a stage and read off what trains, which loss it uses, and which data feed it.
Which parts train, the losses and the datasets are the paper's Section 4, Section 5 and Appendix C.2 to C.3. The percentages divide the trained weights by the whole system of about 12.43 billion, using Table 1. Stage 2 trains the three output projections together (31, 31 and 32 million).
Only the input projection trains. The task is captioning, and the loss is cross-entropy on the caption. We saw it in Chapter 3. This stage uses CC3M, WebVid-2M and AudioCaps. It makes the projection a “multimodal tokenizer” for a frozen LLM. Trained weights: 28 million, which is about 0.23% of the system.
Only the three output projections train, with the three losses of Chapter 5: signal-token loss, caption alignment, and denoising. The same caption datasets feed it. Trained weights: 31 + 31 + 32 = 94 million, about 0.76%.
Now everything that can train does train. The input projection and the output projections train again, and the LLM gets LoRA adapters. The loss has two parts: cross-entropy between the generated and the gold reply, and the generation loss that keeps the output-side alignment (the paper aligns the signal-token states with the gold caption representation, as in Stage 2). Trained weights: 28 + 94 + 33 = 155 million, about 1.25%.
A big weight matrix W has d × d numbers. LoRA (low-rank adaptation) leaves W alone and learns a small correction next to it: the new layer computes W x + B A x. Here A is a thin r × d matrix and B is a thin d × r matrix, so their product BA is a full d × d change that has only r independent directions. For d = 4,096 and an illustrative rank r = 8, one full matrix has 4,096 × 4,096 = 16,777,216 numbers, while A and B together have 2 × 4,096 × 8 = 65,536. That is 0.39% of the size. The paper reports the total LoRA weights it trains in the LLM, 33 million, and does not print the rank or the layers it adapts. B starts at zero, so at the start the model behaves exactly like the frozen one.
Stages 1 and 2 only see caption pairs: a clip and a sentence. They never see a conversation. So the paper builds the instruction data in three kinds.
The paper compares many multimodal instruction sets in its Table 6 and notes that their replies are limited to text. MosIT is built to fix that. The authors write template dialogues between a Human and a Machine, then prompt GPT-4 to produce more on over 100 topics. They require some implicit requests (the user hints, and does not ask), and reasoning, suggestion and planning in the machine’s replies. They then look for the best-matched real content, from retrieval systems such as YouTube and from generators such as Stable Diffusion XL and Midjourney, and people filter out bad dialogues.
A MosIT dialogue looks like this (our illustration, modelled on the paper’s Figure 4): Turn 1, the human gives a video and asks what is unusual, and the machine answers in text. Turn 2, the human asks for music for the scene, and the machine replies with words plus audio signal tokens. Turn 3, the human asks for similar scenes, and the machine replies with words plus image signal tokens. The modality of the input and of the output changes from turn to turn.
The symptom: after Stages 1 and 2 the model captions well and draws when given a caption, but it ignores a hint like “I am in a bad mood”, or it answers a music request with a picture. Measure it with human judges on instruction following. The fix is Stage 3 on MosIT. In the paper’s human test on 100 complex image requests, NExT-GPT beat two pipeline systems and a caption-only variant on instruction following, rationality and quality (Chapter 8).
Chapter 7
Follow the dog video through every tensor, from your upload to the sound and picture that come back
You tap send on the dog video. You get words, a tune and a picture back. Behind that single reply are nine small events. This chapter walks them in order, with the shapes of the data. The paper’s Appendix C.6 and Figure 7 give the same procedure in a diagram.
Nothing in this chapter trains anything. Inference is a forward pass: weights are fixed, numbers flow one way. (Training in Chapters 3 to 6 adjusted the weights in advance.)
Press Next to move one event at a time. Each block is a tensor, one row per vector. Watch the shape change as the data crosses each tier.
Illustrative request modelled on the paper's Figure 4(A). Row counts are drawn to scale only loosely. N, M and T are not printed in the paper: we use M = 32 concept tokens and T = 21 text tokens as example values. The audio count (8) and image count (4) follow the paper's Section 6.2. Width 4,096 is Vicuna-7B's hidden size.
Let us count what crosses each hand-off in our example. Between events 2 and 3, N patch vectors turn into 32 concept tokens. Between events 6 and 7, 8 audio states and 4 image states go out: 12 × 4,096 = 49,152 numbers. A relay would hand the sound tool a sentence such as “funny music for a dog on a skateboard”. That is 8 words, and the tool must turn them back into meaning on its own.
The number of vectors out is set by design, not by the length of the reply: 8 for audio, 4 for image, whatever you asked. This is the opposite of a sentence, whose length changes with how much you say. A fixed budget keeps the decoder’s input shape stable, which a frozen decoder needs.
The nine events are the longest path. A text-only question skips events 2 and 3, since text goes straight to the LLM. A reply with no signal tokens skips events 6 and 7, so the decoders never run, which saves most of the compute. A request for video output swaps the audio and image branches for a longer video branch. The same code path serves all of these, and the choice is made by what the LLM writes.
Count where the video was a video. It was a video only until event 3. After that, the LLM held vectors, never frames. And look at the hand-off between events 6 and 7: a block of vectors, not a sentence. If the first reply went wrong, say the music is sad, there is no sentence to blame. The cause lies in the vectors, which means in the weights that made them.
That gives the failure modes for inference. A wrong read appears as a text answer that misses what happened in the video; check it by comparing the reply to the clip, and fix it with better Stage 1 data. A wrong slip appears as no music when you asked for music; check it by counting signal tokens, and fix it with Stage 3 data. A weak decoder appears as a clean but generic picture; check it with scores such as FID, and fix it by changing the decoder, which the frozen design allows.
One more property is worth noting. The three decoders do not wait for each other. After event 7 the audio and image can be made at the same time. The paper does not report latency, so we make no claim about speed.
Chapter 8
Read the generation scores, the projection ablation and the human test, and learn which claims they support
A paper’s tables are a set of answers. Each one answers a narrow question, and it is easy to hear a bigger answer than the table gives. This chapter lists the questions NExT-GPT’s tables answer, with the numbers, and then lists the questions they leave open.
Tables 2 and 3 compare NExT-GPT with other models on image captioning, image question answering, video question answering and audio captioning. The paper reports strong image comprehension, results comparable to other models on the benchmarks built only for evaluation (MMBench and SEED-Bench), and better scores than CoDi on video and audio, which it credits to the LLM writing the text directly. Some of the table entries are marked as fine-tuned on the training set, so read the star marks before you compare rows. A few numbers from the ablation that follows are the same runs: VQAv2 66.7, VizWiz 48.4, MSVD-QA 64.5, MSRVTT-QA 61.4 and AudioCaps 81.3.
Table 4 scores generation with three standard measures. FID (Fréchet Inception Distance) compares the statistics of generated images with real ones; lower is better. FAD does the same for audio; lower is better. CLIPSIM measures how well a video matches its prompt, using CLIP; higher is better. A dagger marks zero-shot rows, tested without training on that dataset.
Choose a result. Bars show the paper's own values. Turn the axis zoom off to see how little the bars differ when drawn from zero.
Values are copied from the paper: Table 4 for the first three sets, and Table 5 for the ablation (the text calls it Table 4). The ablation bars are our subtraction: each variant's score minus the linear-layer score, on five benchmarks. A dagger marks zero-shot. Different baselines were run by different groups, so rows are not perfectly controlled.
Look at the image set first. NExT-GPT scores FID 10.07, the best of the seven rows. Plain Stable Diffusion 1.5, the very decoder NExT-GPT uses, scores 11.21. Is that because the LLM adds value? Be careful here. The same model scored zero-shot (11.18) is almost the same as Stable Diffusion alone. The 10.07 row was tuned on the data. So the fair reading: NExT-GPT keeps its decoder’s quality and does not lose it, and with tuning it can improve a little.
Audio and video follow the same pattern. NExT-GPT’s FAD of 1.68 beats CoDi (1.80) and AudioLDM-L (1.96), and its CLIPSIM of 31.97 beats CoDi’s 28.90. The paper calls the video result a clear improvement and the zero-shot row (30.96) still leads CoDi. What the table cannot say is whether the LLM or the tuning caused the gain.
The ablation (the paper’s Table 5) swaps the input projection for a single linear layer, or for a Q-Former plus linear layer, and trains each the same way. Select “Projection ablation” above. The Q-Former helps a little (1.0 to 2.6 points over linear). The grouping block helps more (2.9 to 4.3 points). On all five benchmarks the order is the same: linear, then Q-Former, then grouping. This supports the design choice. It does not tell us how the answer changes with larger models or more data.
Figure 6 sweeps the count: 1, 2, 4 and 8 for images, 4 to 32 for video, and 2 to 16 for audio. The text of the paper reads the curves as follows. Images and audio are satisfied with 4 and 8. Video needs the most. The best count depends on the amount of data and on the power of the decoder.
For this question the authors ran a human test. GPT-4 wrote 100 complex instructions that need implicit reasoning to make an image. Four systems answered with text plus an image: HuggingGPT, Visual-ChatGPT, a variant called NExT-GPT-caption (which writes a caption for the decoder, like a relay), and NExT-GPT itself. Five volunteers judged instruction following, rationality of the image, and quality. NExT-GPT was ahead on all three in the paper’s Figure 5. This is the cleanest test of the claim in Chapter 0, since NExT-GPT-caption differs from NExT-GPT mainly in the hand-off. It is small: 100 prompts and five judges.
Chapter 9
Review the numbers, the build checklist and the core code, then see where the paper sits in the field
This chapter is for the day you want to use the paper again: to explain it in five minutes, to check a number, or to start building. It adds no new ideas.
NExT-GPT (Wu, Fei, Qu, Ji and Chua, ICML 2024) builds an any-to-any assistant by joining existing parts. ImageBind encodes image, audio and video. A grouping input projection turns the patch grid into a few concept tokens that Vicuna-7B can read. Vicuna answers in text and, when it chooses, writes modality signal tokens. An output projection per modality turns those tokens into the condition that a frozen diffusion decoder (Stable Diffusion, AudioLDM, Zeroscope) expects. Training has three stages, and only about 155 million of 12.4 billion weights change. A 5,000-dialogue set, MosIT, teaches the model to switch modalities from turn to turn. The result scores close to or above the baselines on perception, and above the listed baselines on text-to-image, text-to-audio and text-to-video scores, and wins a small human test against pipelines.
| What | Number | Why it matters |
|---|---|---|
| Trained weights | 155M of about 12.43B (about 1.25%) | Reuse frozen parts, train the joints |
| Frozen parts | ImageBind 1.2B, Vicuna 7B, SD 1.3B, AudioLDM 975M, Zeroscope 1.8B | Quality comes from the parts |
| Trained parts | input 28M, LoRA 33M, outputs 31M, 31M, 32M | Table 1 of the paper |
| Output projection | hidden 512, 4 heads, 4 encoder and 4 decoder layers | Small transformer with learnable queries |
| Signal tokens | image 4, audio 8, video the most (24 in our reading) | Video needs more slips |
| Stage 1 data | CC3M (3M+), WebVid-2M, AudioCaps (about 46K) | Captions teach reading |
| Instruction data | T2M 15K instances, MosIT 5K dialogues of 3 to 7 turns | Teaches switching of modalities |
| Image generation, FID | 10.07 (zero-shot 11.18) vs SD-1.5 11.21 | Keeps decoder quality |
| Audio generation, FAD | 1.68 vs CoDi 1.80, AudioLDM-L 1.96 | Lower is better |
| Video generation, CLIPSIM | 31.97 vs CoDi 28.90 | Higher is better |
| Projection ablation | VQAv2 66.7 vs 65.1 (Q-Former) vs 63.8 (linear) | Grouping beat both on five benchmarks |
| Human test | 100 GPT-4 prompts, 5 volunteers | End to end vs pipelines, small scale |
This is a sketch of the idea, with our shapes; it is not the authors’ code. enc, llm, inp, outp and dec stand for the parts of Chapter 2.
pythondef respond(text_ids, media): x = [llm.embed(text_ids)] # [T, 4096] for m in media: # image / audio / video patches = enc(m) # [N, d_enc] frozen ImageBind x.append(inp(patches)) # [M, 4096] grouping, trained tokens, hidden = llm.generate(cat(x)) # text + signal tokens, [len, 4096] outputs = {"text": detok(tokens)} for mod, prefix in SIGNALS.items(): idx = [i for i, t in enumerate(tokens) if t.startswith(prefix)] if idx: # token present: decoder ON cond = outp[mod](hidden[idx]) # [m, d_cond] trained outputs[mod] = dec[mod].sample(cond) # frozen diffusion return outputs def stage2_loss(caption, media_gt, mod): h = llm.hidden_at_signals(caption) # [k, 4096], LLM frozen cond = outp[mod](h) l_nll = llm.nll_of_signal_tokens(caption) l_cap = ((cond - dec[mod].text_encoder(caption)) ** 2).sum() l_den = dec[mod].denoise_loss(media_gt, cond) # U-Net frozen, gradient flows through return l_nll + l_cap + l_den
NExT-GPT is the “LLM as brain, diffusion as hands” design. It belongs to the family of lecture-8 models that bolt encoders onto a language model: see Lecture 8, Large Multimodal Models, the generation lecture Lecture 9, Multimodal Generation, and the other assigned readings LLaMA-Adapter, Kosmos-2 and MM1.
Now press Present or Teach and explain the system from memory: why a caption relay loses detail, what a signal token carries, which three losses train the output projection, and what the 1% means. Then go back to the flow at the top, send each request through both systems, and say what crosses the gap.
Credit: NExT-GPT: Any-to-Any Multimodal LLM, Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji and Tat-Seng Chua, NExT++ Research Center, National University of Singapore, ICML 2024 (PMLR 235). arXiv:2309.05519. Project page: next-gpt.github.io. All figures, tables and numbers here are from that paper unless marked illustrative or ours.