NExT-GPT: Any-to-Any Multimodal LLM

NExT-GPT lets one language model take in and answer with any mix of text, image, video and audio, while training only 1% of its weights. Pick a request and watch what crosses the gap.

Learn how a chat model can end up making music, pictures and video, by handing a few learned vectors to decoders it never retrains.

Choose a request, then press Send the request with each system. See what reaches the decoder: a caption in plain words, or signal tokens that training can reach. Then we build, piece by piece, every part of the paper.

You need a rough idea of what a chat model does (it writes text one piece at a time) and what a vector is (a list of numbers). We build the rest from zero.

Based on NExT-GPT: Any-to-Any Multimodal LLM by Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji and Tat-Seng Chua (NExT++ Research Center, National University of Singapore), ICML 2024. An assigned reading of MIT MMAI Lecture 8, Large Multimodal Models (course schedule). Every idea is re-taught here in our own words and devices.

Send a request through two systems

Illustrative flow, drawn from the paper's Figures 1, 3 and 4. The three requests are the paper's own qualitative examples. Signal-token counts are 4 for image, 8 for audio and 24 for video (Section 6.2 says images and audio need 4 and 8 and video the most; Section 4.2 lists [IMG0] to [IMG4], [AUD0] to [AUD8] and [VID0] to [VID24]). The width 4,096 is Vicuna-7B's hidden size, our addition, since the paper prints no shapes. The captions are made up for the toy.

Chapter 0

Why a Relay of Words Falls Short

See what a chat assistant cannot do today, and why passing text between tools loses what you wanted

You record a short video of a dog riding a skateboard. You send it to an assistant with one line: “What is unusual here? Then add some funny music, and show me similar scenes.” You want a spoken answer, a sound clip and a picture back, all in one reply.

Most assistants of that era could handle the first half. A model such as LLaVA reads an image and writes text about it. It listens, so to speak, in many modalities. It answers in one: words. A modality is a kind of signal: text, an image, a video or a sound clip. The paper calls these models MM-LLMs, multimodal large language models: a language model that has been taught to read modalities besides text.

The second half needs something different. The assistant must also make a sound clip and a picture. The authors of NExT-GPT call a system that can read any mix and write any mix any-to-any. That is the goal of the paper: accept input in any modality, and deliver a response in any modality that fits.

Who had tried, and where each one stopped

The paper sorts the earlier work into four groups. The first group only understands: Flamingo, BLIP-2, MiniGPT-4, Video-LLaMA, LLaVA, PandaGPT, SpeechGPT. They put the picture, the video or the sound in, and they get words out. Nearly all work at that time was of this kind, so the output side was nearly empty.

The second group adds some writing. Emu, DreamLLM, GILL and SEED can produce an image next to their text. The paper notes that they stay inside interleaved text and images. There is no sound, and there is no video.

The third group is the pipeline. Visual-ChatGPT, HuggingGPT and AudioGPT use a language model as a manager. The manager reads your request and calls ready-made tools: an image generator, a speech model, a captioner. In theory a pipeline reaches any-to-any, since there is a tool for every modality.

The fourth group is a single model, CoDi. It can read and write any mix of modalities at once, using diffusion models. But it has no language model at its centre. So it cannot reason about an implied wish such as “I am in a bad mood.” The paper says it can only deliver simple paired generation.

Map who covers what

Rows are what you give the system. Columns are what it can give back. A lit cell means the pair works. Choose a kind of system and count the lit cells, then look at the two checks underneath.

Our summary of the paper's Introduction and Related Work, one modality at a time (text, image, video, audio). It ignores quality. Understand-only systems read more than they write: we lit text, image, video and audio inputs for them, as PandaGPT does, though a given model may read fewer.

The two flaws of the relay

The pipeline looks like the answer, so why did the authors build something else? They name two flaws, and both sit in the hand-off between the language model and the tool.

The first flaw is noise. In a pipeline, everything passes as text. To make your sound clip, the manager writes a sentence such as “a dog rides a skateboard, comic mood”, and a sound tool reads only that sentence. A caption is a lossy summary. The tool never sees what the manager saw. And a second hop adds a second summary, so mistakes pile up. The paper calls this a cascading process that propagates errors.

The second flaw is that nothing is trained together. In a pipeline, the manager and the tools are used only for inference, which means running a model without changing it. No loss, the number that training lowers, ever measures the whole chain. So the manager never learns which sentences make the sound tool do a good job. It cannot get better at the implicit requests, where you hint and do not ask.

The claim, as a trade. NExT-GPT answers in any modality, and trains end to end, where a text relay could not, because the language model hands the decoders a few learned vectors, not a caption. Only about 1% of the whole system's weights need training to make that work.

Keep that sentence in mind. It has three parts, and each is a chapter. “A few learned vectors” are the signal tokens (Chapter 4). “Trains end to end” is the three-stage recipe (Chapters 5 and 6). “Only 1%” is the plan that reuses frozen parts (Chapter 2).

The road

Go back up to the hero and try both systems on the same request. The relay hands over a short caption. NExT-GPT hands over vectors, and a dotted line returns the learning signal. The rest of this lesson builds every box in that picture.

  1. Four ideas you need: embeddings, a shared space, diffusion decoders, and why a projection layer is enough.
  2. Three tiers: the encoders, the LLM and the decoders, and the 155 million weights that train.
  3. Reading any input: concept tokens built by a grouping block.
  4. Answering any way: modality signal tokens that switch decoders on.
  5. Teaching the decoder to listen: three losses that align signal tokens with a diffusion model.
  6. Three stages and MosIT: the recipe, LoRA, and the 5,000 dialogues that switch modalities.
  7. One request, start to finish: every tensor on the path of the dog video.
  8. What the tables show: generation scores, the projection ablation, human judges.
  9. Field guide: numbers, a checklist, the core code, and where to go next.
A tool pipeline answers your video request with a sound clip that does not fit the scene. Which flaw does the paper blame?

Chapter 1

Four Ideas You Need First

Learn embeddings, shared spaces, diffusion decoders and projections, the four parts the paper snaps together

Imagine a post office with one rule: every parcel must be addressed with a list of numbers. A letter, a photo and a song all get an address. Parcels with similar content get addresses close together. If you can write the address, you can find the neighbours. That picture carries most of this chapter.

Idea 1: an embedding is an address made of numbers

A language model cannot read letters. It first cuts text into tokens, small pieces such as a word or a part of a word. Then it swaps each token for a vector, a list of numbers, usually a few thousand long. That vector is the token’s embedding: its address in meaning space. The width of the list is the model’s hidden size. For the Vicuna-7B model that NExT-GPT uses, we take this to be 4,096 (the paper does not print it; it is a property of that model family). Every layer inside the model reads and writes lists of that width.

This gives a trick that the whole paper depends on. The model does not care where a vector came from. If you can produce a list of 4,096 numbers that sits in a sensible place, the model will treat it like a token. A picture, once turned into such lists, can be “read” by the same model that reads words.

Idea 2: an encoder turns a signal into vectors, and a shared space makes them comparable

An encoder is a model that turns a raw signal into vectors. A typical image encoder cuts the picture into a grid of small squares called patches, and gives each patch a vector. The paper calls the result a patch-level grid of features. A video encoder does the same over frames. An audio encoder does it over short slices of a spectrogram, a picture of sound.

The paper uses one encoder family, ImageBind, for image, video and audio. ImageBind was trained so that different modalities land in one shared space: a photo of a dog, the word “dog” and the sound of barking get nearby addresses. The authors like it because they are spared from managing many different encoders. Use the device to see why shared space matters.

Align the post office

Nine items, three concepts, three modalities. Each arrow starts at a word and ends at the nearest sound. Drag the alignment from 0 to 1 and watch the arrows turn from wrong to right.

0.00

Toy: nine hand-placed points in two dimensions, three concepts by three modalities. Square is a word, circle an image, triangle a sound. Alignment slides every point linearly from its modality cluster to its concept cluster. The nearest sound to each word is computed live. Real embeddings have thousands of dimensions and ImageBind learns them by contrastive training on image-paired data.

At alignment 0, the sounds sit with the sounds and the words sit with the words. The nearest sound to a word is a matter of luck. At alignment 1, items with the same meaning sit together, whatever their modality. That is what a shared space gives you: you can pick a vector from one modality and meet its meaning in another.

Idea 3: a diffusion decoder paints from noise, steered by a condition

A diffusion model makes a picture by removing noise. Think of a sculptor who starts with a rough block and chips for fifty rounds. The model starts with pure random noise. At each round, a network (the U-Net) predicts which part of the picture is noise and subtracts a little. After many rounds a clean picture remains.

A latent diffusion model does this in a compressed space, not on pixels, which is much cheaper. The compressed picture is called the latent, and a separate decoder inflates it back to pixels. The paper uses three of these, all ready-made: Stable Diffusion 1.5 for images, Zeroscope for video and AudioLDM for audio.

Left alone, the model paints a random picture. To ask for a particular one, you give it a condition: a set of vectors the U-Net looks at while it denoises. In these decoders, the condition comes from a text encoder that turns a prompt into vectors. This is the one door into the decoder, and it expects text-shaped vectors. NExT-GPT will have to push its own vectors through that door.

Training such a model uses one simple loss. Add random noise to a clean latent, then ask the network to name the noise.

Lden
The conditional latent denoising loss. It is small when the network names the noise correctly.
E
An average over many random picks of picture, noise and step number.
ε
The real noise that we added. It is the answer key.
εθ
The U-Net’s guess of that noise, with weights θ.
zt
The latent after noise at level t was added. At large t it is nearly pure noise.
c
The condition vectors from the text encoder. In NExT-GPT, these are replaced by the LLM’s signal tokens, after a projection.

Worked, in words: noise a latent at t = 600, hand the network the noisy latent and a condition for “a dog”, and it guesses the noise. The squared error of the guess is the loss. A condition that matches the picture makes the guess easier.

Idea 4: a projection layer is a small adapter between two spaces

Now the plumbing problem. ImageBind’s vectors live in one space. Vicuna’s token vectors live in another. Stable Diffusion’s condition lives in a third. They do not match, and training all three together would cost a fortune.

The cheap fix is a projection layer: a small network that maps vectors from one space to another, like a plug adapter between two countries’ sockets. You freeze the big models on both sides, and train only the adapter, so the adapter learns the translation. NExT-GPT needs two kinds: an input projection from ImageBind to Vicuna, and an output projection from Vicuna to each decoder.

Why freeze? The big models are already good at their own jobs. Training them again would cost a lot and might damage them. A small adapter changes only the hand-off, and the rest stays as it was built.
Why does NExT-GPT train a small projection layer between a frozen encoder and the frozen LLM?

Chapter 2

Three Tiers and the 1% That Trains

Follow a request through the encoders, the LLM and the decoders, and count which weights change

Picture a relay race with three runners. The first runner turns your photo, your clip or your voice into a form the second runner can read. The second runner thinks and decides what to say and what to make. The third runner turns that decision into pictures, sound or video. NExT-GPT has exactly three such tiers, and the paper’s Figure 1 draws them as three columns.

Tier 1: encoding

Text goes straight to the LLM, since the LLM already reads tokens. Image, audio and video go through ImageBind, the shared encoder from Chapter 1, which has about 1.2 billion weights. It returns a grid of patch vectors. An input projection then turns that grid into a short list of vectors the LLM can read. The projection is a block with about 28 million weights, and it is the subject of Chapter 3.

The output of this tier is a list of vectors, each as wide as a Vicuna token embedding. Chapter 3 shows that the list is short: a few concept tokens, not hundreds of patches. These vectors are placed in the LLM’s input, in the same row as ordinary word tokens.

Tier 2: the LLM

The core is Vicuna-7B (version 0), an open chat model with about 7 billion weights. It is a plain text model that the authors did not rebuild. It reads the row of word vectors and projected vectors, reasons over them, and writes a reply. The reply contains two kinds of tokens. The first kind is ordinary text. The second kind is new: modality signal tokens, special tokens that mean “now make an image” or “now make a sound”. Chapter 4 is about these.

So the LLM is both the reader and the director. The paper calls it the core agent. It is the only tier that decides whether to generate and what.

Tier 3: generation

For each output modality, the signal tokens pass through an output projection, a small transformer of about 31 million weights. Its job is to turn the LLM’s hidden states into the condition vectors that one specific decoder expects. The decoders are the three ready-made diffusion models: Stable Diffusion 1.5 for images (1.3B), AudioLDM for audio (975M) and Zeroscope for video (1.8B).

The paper’s Table 1 lists all of it. Here it is in one place.

PartWeightsStatus
ImageBind encoder1.2Bfrozen
Input projection (grouping)28Mtrained
Vicuna-7B7Bfrozen, plus LoRA
LoRA adapters in the LLM33Mtrained
Output projection, image31Mtrained
Output projection, audio31Mtrained
Output projection, video32Mtrained
Stable Diffusion (image)1.3Bfrozen
AudioLDM (audio)975Mfrozen
Zeroscope (video)1.8Bfrozen

The 1% arithmetic, worked

Add the trained parts: 28 + 33 + 31 + 31 + 32 = 155 million. Add the frozen parts: 1.2 + 7 + 1.3 + 1.8 + 0.975 = 12.275 billion. The trained share is 155 / (155 + 12,275) = 155 / 12,430, which is about 1.25%. The paper rounds this to “only 1%”.

Use the device to see how small the trained part is, and to try a what-if: switch off an output modality and watch the share change.

Count the weights that train

Top bar: every weight in the system, to scale. The thin purple slice is what trains. The lower bar zooms into that slice. Switch output modalities on and off.

Sizes are Table 1 of the paper. The what-if is ours: it drops that modality's output projection and decoder and keeps the encoder, the input projection, Vicuna and LoRA. The paper trains the full system, with all three outputs.

The data flow, with shapes

The paper does not print tensor shapes, so these are our reading, with the Vicuna width of 4,096 as the one fixed number. Take a clip of audio.

  1. Raw clip goes into ImageBind. Out comes a grid of N × denc feature vectors, where N depends on the clip length and denc is the encoder’s width.
  2. Input projection groups the N vectors into M × 4096 concept tokens. M is small compared with N.
  3. Vicuna reads the text tokens and the M concept tokens as one row of (T + M) × 4096 vectors. It writes text tokens, and, if it chooses, a run of signal tokens. Each signal token is again a 4096-wide hidden state.
  4. Output projection maps the k × 4096 signal states to the m × dcond condition a decoder expects. The numbers k, 4 for image, 8 for audio and 24 for video in our reading, come from the paper’s discussion of signal tokens.
  5. Diffusion decoder denoises a latent under that condition and returns the clip, picture or video.

Nothing in this chain is text except the LLM’s own words. That is the difference from the relay in Chapter 0: the hand-off between tiers 2 and 3 is a block of vectors, not a sentence.

Why build it this way

The paper gives two reasons. The first is cost: training the encoders, the LLM and three diffusion models from nothing would be very expensive, so the authors load off-the-shelf weights and avoid a cold start. The second is growth: adding a fourth modality means adding one encoder path and one decoder path with their projections. The core stays as it is.

There is a price. A frozen decoder can only do what its condition door allows, so the whole system inherits its decoders’ quality, and its flaws. Chapter 8 returns to this.

Which weights does NExT-GPT train?

Chapter 3

Reading Any Input with Concept Tokens

Group hundreds of patch vectors into a few concept tokens, and teach a frozen LLM to read them

Look at the dog on the skateboard. You do not see 256 little tiles. You see a dog, a board, a road and a sky. Your mind groups tiles into things. A word in a sentence works the same way: the word “dog” stands for one whole concept.

An image encoder does not group. It hands over one vector per patch, a long grid. The paper’s argument is that these patch units “might not best coincide with the intricate textual token semantics”. A patch is a tile. A text token is a concept. Feeding tiles to a model that thinks in concepts may lose information.

Three ways to cross the gap

Earlier systems used two designs. A linear layer, as in LLaVA, maps every patch vector straight into the LLM’s space. The LLM then gets as many tokens as there are patches. A Q-Former, as in BLIP-2, uses a few learned query vectors that look at the patches through attention and report back. NExT-GPT adds a third design, a grouping mechanism: learned concept tokens claim patches by similarity, stage after stage, and the claimed patches are merged into the concept token. The paper borrows the idea from GroupViT (Xu et al., 2022).

One grouping stage, step by step

Say the encoder gives N patch vectors, X. We start a stage with M learnable concept tokens, C, set at random. They are weights: training will shape them. Four moves follow.

Move 1: think together. Glue the concept tokens and the patches into one row and run a transformer layer over the row. Now each concept token has looked at the patches, and each patch has looked at the concept tokens.

C
The M learnable concept tokens of this stage, each a vector. Shape: M × d.
X
The N patch vectors coming in. Shape: N × d. After stage 1 these are the previous stage’s concept tokens.
Ĉ
The concept tokens after they looked at the patches.
X̂
The patches after they looked at the concept tokens.

Move 2: score every patch against every concept. Normalise each vector to length 1, so a dot product is a cosine: 1 means same direction, 0 means unrelated. Add random Gumbel noise, divide by a temperature τ, and apply a softmax over the concepts.

A
An M × N table. For each patch (a column), a probability for each concept. Soft: all numbers between 0 and 1.
G
Gumbel(0, 1) noise, a fresh random number for every cell. It turns a pick into a random draw.
τ
A learnable temperature, which the paper calls the significance coefficient. Small τ makes A sharp, large τ makes it flat.

Move 3: choose, but keep the gradient. Each patch joins its single best concept. That choice is an argmax, a hard one-hot vector, and an argmax has no useful gradient: nudge a weight a little and the pick either stays or flips. The fix is the straight-through trick. In the forward pass use the hard pick. In the backward pass pretend you used the soft table A.

Â
The assignment the next step uses. Its value equals the hard one-hot pick.
Sg
Stop-gradient: it keeps the value and hides it from the gradient. So A − Sg(A) is zero in value, but carries A’s gradient.

Worked: a patch scores [0.9, 0.2] against two concepts. At τ = 0.5 and no noise, A = [e1.8, e0.4] / (6.05 + 1.49) = [0.80, 0.20]. The forward value of  is [1, 0]. The backward pass sees [0.80, 0.20].

Move 4: merge. Each concept token collects the patches that chose it and keeps a residual copy of itself: Xl+1 = Ĉl + MLP(Âl, X̂l). The paper writes the merge only in this form, and we read it as GroupViT does: sum the chosen patches, then pass the sum through a small MLP. After L such stages, the model holds ML concept tokens, and those go to the LLM. The paper does not print the counts Ml.

Group the patches

A 6 by 6 grid of patches becomes four concept tokens, A to D. Tap a patch to see its scores. Turn the noise on and press Draw new noise. Then move τ and see what it changes.

0.50

Toy: patch directions follow a smooth made-up field, so neighbouring patches agree. Cosine scores, Gumbel noise and the softmax are computed exactly as in the equations, in two dimensions with four fixed concept directions. In the paper the concept tokens are learned and there are several stages.

Try it. With the noise off, the grid splits into four clean bands, because the argmax is a pure function of the scores. With the noise on, patches near a boundary flip between neighbouring concepts each time you draw. Now move τ. The pick does not change, since dividing by a positive number never changes which score is largest. What changes is the soft table A, the one that carries the gradient. So τ controls how sharp the training signal is, not the choice itself.

How Stage 1 trains this block

The block starts with random weights, so it needs a lesson. The authors use captioning: show an image, a video or an audio clip, and ask the frozen LLM to write its caption. The loss is ordinary cross-entropy on the caption words, which measures how surprised the LLM is by the true caption. Only the input projection is trainable. The paper puts it well: this stage trains “a compatible multimodal tokenizer for the frozen LLM”.

The data are pairs: CC3M for images (over 3 million image and caption pairs), WebVid-2M for video (short stock clips with descriptions) and AudioCaps for audio (about 46,000 clips with human captions). After this stage, an image becomes a short row of concept tokens that the LLM can describe. The LLM was never changed.

An argmax pick has no useful gradient. How does the grouping block still learn?

Chapter 4

Answering in Any Modality with Signal Tokens

Let a text model choose whether to draw, play or film, by writing a few special tokens into its reply

You order dinner at a busy restaurant. The waiter does not cook. He writes a ticket. Some of the ticket is for you: “Of course, one moment.” Some of it is a slip for the kitchen: “grill, medium rare.” A slip that is not written means that station stays quiet.

NExT-GPT’s language model does the same. It writes its reply as a row of tokens. Most are ordinary words for you. Some are modality signal tokens: special tokens that act as slips for the decoders. The paper adopts the idea from GILL (Koh et al., 2023).

The three families of slips

The paper adds three families to the LLM’s vocabulary: image tokens [IMG0] and onward, audio tokens [AUD0] and onward, and video tokens [VID0] and onward. The paper writes the families as [IMGi] for i from 0 to 4, [AUDi] for i from 0 to 8 and [VIDi] for i from 0 to 24. Counted from zero, that is five, nine and twenty-five labels, while the later discussion speaks of 4 and 8 tokens. The paper does not reconcile the two, so we use 4, 8 and 24 as round working numbers. The discussion of Figure 6 says images work well with 4 tokens and audio with 8, and that video needs the most.

The rule for turning decoders on is simple and the paper states it directly. If the LLM decides that a modality should be produced, it writes that family’s tokens. If it writes none, that modality is off. A single reply can switch on several decoders at once, such as audio and image.

What a signal token carries

A signal token is a word with no dictionary meaning. What matters is the vector the LLM computes at that position: its hidden state, a list of 4,096 numbers after the final layer. The LLM reads the whole conversation before it writes a signal token, so its hidden state can hold the whole request: “a funny, light tune for a dog on a skateboard”. Compare this with the relay in Chapter 0, which squeezes the same request into one short caption.

These hidden states are what the output projection receives (Chapter 5). The paper says the tokens “implicitly carry rich and flexible instructions for the downstream diffusion model”. They are instructions in a learned code that no human wrote down.

Read the LLM's ticket

Pick what the user says. The strip shows the tokens the LLM writes, with signal tokens boxed in yellow. The lamps show which decoders get a slip. Then change how many image tokens the LLM writes.

4

The replies are made up for the toy; the mechanism (a family of tokens switches its decoder on, no token switches it off) is the paper's, Section 4.2. Token counts are 8 for audio and 24 for video; the image count is your slider (the paper's Figure 6 tests 1, 2, 4 and 8 and finds 4 enough). Widths of 4,096 are Vicuna-7B's, our addition.

Choose the video question first. The LLM writes only words, so no lamp lights. Nothing is made, and nothing should be. Now choose “A bad mood”. The user never asked for anything. Yet in the paper’s Figure 4 example, the model replies with sympathy and then decides to show a puppy video. That choice is the slips: video tokens appear at the end of the reply. The decision to generate is part of the model’s reasoning, not a rule written by hand.

The router in code

On the system side, routing is a few lines. The program scans the LLM’s output for signal tokens and collects their hidden states, one batch per modality. This is a sketch of the idea, not the authors’ code.

pythonSIGNALS = {"image": "[IMG", "audio": "[AUD", "video": "[VID"}

def route(tokens, hidden):
    # tokens: list[str] the LLM wrote; hidden: [len(tokens), 4096] final-layer states
    jobs = {}
    for modality, prefix in SIGNALS.items():
        idx = [i for i, t in enumerate(tokens) if t.startswith(prefix)]
        if idx:                        # token present: this decoder is ON
            jobs[modality] = hidden[idx]   # [k, 4096] goes to the output projection
    return jobs                        # empty dict: a text-only reply

How the LLM learns when to write slips

The tokens are new, so the LLM must learn them. It learns them from examples where the right reply already contains them. In the alignment stage, the paper takes captions from CC3M, WebVid and AudioCaps and treats the caption as the text part of a reply, then appends the signal tokens as the thing to predict. The first loss term is the negative log-likelihood of producing those signal tokens: ordinary next-token cross-entropy. In the final instruction-tuning stage, the gold replies in the MosIT dialogues also carry signal tokens in the right places (Chapter 6).

Training therefore teaches two things. It teaches when a reply should include a slip, and it teaches what to put in the slip’s hidden state. The second is the job of the losses in Chapter 5.

The failure mode

The symptom: you ask for a picture and receive only words. Measure it by counting replies that contain no signal token on prompts where a human would expect one. The fix is data. The paper builds the MosIT dialogues, where users hint and the machine chooses to generate, so that the model sees many cases of implied requests with the correct slips.

In NExT-GPT, what turns the audio decoder on for a given reply?

Chapter 5

Teaching a Frozen Decoder to Listen

Turn signal-token states into the condition a diffusion model expects, using three losses at once

Suppose you hire a painter who only takes orders written in French. You speak English. You do not want to retrain the painter, and you do not want to learn French by heart. So you hire a translator, and you test the translator by asking: does the French you wrote look like the French a native speaker would have written for the same job?

That is the problem on the output side. The diffusion decoder is the painter. Its condition door expects vectors from its own text encoder, which is the French. The LLM’s signal-token states are the English. The output projection is the translator, and it is the only part of this tier that trains.

What the output projection is

The paper describes it as a transformer with a hidden size of 512, 4 attention heads, 4 encoder layers and 4 decoder layers, with dropout 0.1. Its Figure 2(b) shows a linear layer, an encoder, a decoder fed with learnable queries, and a linear layer out. We read the flow as follows. The k signal states (each 4,096 wide) go in through a linear layer down to 512. The encoder mixes them. The decoder starts from a fixed number of learnable query vectors, a set of blank slots, and fills them by looking at the encoder output. A last linear layer lifts each slot to the width the decoder wants. The number of slots matches the length of the decoder’s condition. That is how a handful of signal tokens can become a long condition. There is one projection per modality, of 31, 31 and 32 million weights.

Three losses, one idea each

Stage 2 trains the projection with a loss made of three parts. The paper lists them in one sentence; here they are one by one.

Loss 1, the signal-token loss. This is the cross-entropy from Chapter 4. Given the caption, the LLM should write the signal tokens. It teaches when to speak.

Loss 2, the caption alignment loss. Take the real caption of the training image and run it through the decoder’s own text encoder. The result is the condition the decoder would receive from a text prompt. We call it the target. Now ask for the projected signal tokens to be close to it, in squared distance. This is the shortcut: the decoder already understands the target, so a projection that matches it works out of the box.

Lcap
The caption alignment loss. Zero when the projection exactly copies the text encoder.
P
The output projection, with trainable weights. It maps signal states to the decoder’s condition shape.
hsig
The hidden states of the signal tokens, from the frozen LLM. Shape: k × 4096.
Etxt
The decoder’s own frozen text encoder.
caption
The human-written caption of the training clip or picture.

Worked: suppose P(h) = [0.2, 0.9, −0.1] and the target is [0.5, 0.7, 0.0]. The difference is [−0.3, 0.2, −0.1]. Square each: 0.09, 0.04, 0.01. The loss is 0.14. Training pushes it toward 0.

Loss 3, the conditional denoising loss. This is the loss of Chapter 1, with one change. The condition c is now the output of the projection, not the text encoder. Take the real picture, add noise, and ask the frozen U-Net to name the noise while it looks at the projected signal tokens. The U-Net weights do not change, but the error still flows backwards through it into the projection. The picture, and not the caption, is the judge.

L
The Stage 2 loss. The paper names three components and prints no weights, so we write a plain sum.
LNLL
Negative log-likelihood of writing the signal tokens.
Lden
The conditional latent denoising loss from Chapter 1, using P(hsig) as the condition.

Why both loss 2 and loss 3? Loss 2 is fast and clean, but it only asks the projection to copy a text embedding, and a text embedding is only as good as the caption. Loss 3 asks the real question, does the decoder paint what was wanted, but it is noisy and slow to learn from. The paper uses loss 2 “to accelerate alignment learning” and loss 3 to keep the result honest.

Train a tiny output projection

Blue rings are the targets the text encoder would give. Purple dots are the projection of each signal state. Press Run and watch the dots slide onto the rings as the caption loss falls.

0.50

Toy: six made-up two-number signal states and targets, and a 2 by 2 projection trained by exact gradient descent on the mean of the caption loss. The real projection is a 31-million-weight transformer and also feels the denoising loss through the U-Net, which this toy leaves out.

Set the learning rate very high and press Run. The dots overshoot, and the loss may rise before it falls, or may blow up. Set it very low and the descent is slow. This is the usual rhythm of gradient descent, and it applies to the real projection.

The failure mode

The symptom: the decoder paints something plausible that ignores the detail you asked for, a generic dog, not a dog on a skateboard. Measure it with a score that compares the output to the prompt, such as CLIPSIM for video, or by counting the signal tokens you give. In Figure 6 the paper finds that images and audio are fine with 4 and 8 tokens, but video needs many more. The fix is more tokens, more or cleaner data, or a stronger decoder. The authors note that the best number depends on the data and on the decoder.

What does the caption alignment loss (Lcap) compare?

Chapter 6

Three Stages, LoRA and the MosIT Dialogues

Follow the training recipe stage by stage, and see why the last stage needs dialogues that switch modalities

Learning to drive has three phases too. First you learn to read the road signs. Then you learn the pedals and the wheel. Last you drive in real traffic with a coach beside you. You would not start in traffic. Each phase trains a different skill and relies on the earlier ones.

NExT-GPT is trained the same way, in three stages, and the paper’s appendix spells them out. The device below shows which blocks are trainable in each one.

Walk the three stages

Purple blocks train. Grey blocks are frozen. Choose a stage and read off what trains, which loss it uses, and which data feed it.

Which parts train, the losses and the datasets are the paper's Section 4, Section 5 and Appendix C.2 to C.3. The percentages divide the trained weights by the whole system of about 12.43 billion, using Table 1. Stage 2 trains the three output projections together (31, 31 and 32 million).

Stage 1: teach the LLM to read

Only the input projection trains. The task is captioning, and the loss is cross-entropy on the caption. We saw it in Chapter 3. This stage uses CC3M, WebVid-2M and AudioCaps. It makes the projection a “multimodal tokenizer” for a frozen LLM. Trained weights: 28 million, which is about 0.23% of the system.

Stage 2: teach the decoders to listen

Only the three output projections train, with the three losses of Chapter 5: signal-token loss, caption alignment, and denoising. The same caption datasets feed it. Trained weights: 31 + 31 + 32 = 94 million, about 0.76%.

Stage 3: teach the whole system to follow instructions

Now everything that can train does train. The input projection and the output projections train again, and the LLM gets LoRA adapters. The loss has two parts: cross-entropy between the generated and the gold reply, and the generation loss that keeps the output-side alignment (the paper aligns the signal-token states with the gold caption representation, as in Stage 2). Trained weights: 28 + 94 + 33 = 155 million, about 1.25%.

LoRA, derived

A big weight matrix W has d × d numbers. LoRA (low-rank adaptation) leaves W alone and learns a small correction next to it: the new layer computes W x + B A x. Here A is a thin r × d matrix and B is a thin d × r matrix, so their product BA is a full d × d change that has only r independent directions. For d = 4,096 and an illustrative rank r = 8, one full matrix has 4,096 × 4,096 = 16,777,216 numbers, while A and B together have 2 × 4,096 × 8 = 65,536. That is 0.39% of the size. The paper reports the total LoRA weights it trains in the LLM, 33 million, and does not print the rank or the layers it adapts. B starts at zero, so at the start the model behaves exactly like the frozen one.

Why Stage 3 needs new data

Stages 1 and 2 only see caption pairs: a clip and a sentence. They never see a conversation. So the paper builds the instruction data in three kinds.

  1. Text plus X to text. Existing sets such as LLaVA-150K and VideoChat. The reply is only words. The cleaned Alpaca set keeps pure text skills alive.
  2. Text to text plus X (T2M). The authors take caption pairs and ask GPT-4 to wrap each caption in a varied written instruction, such as “draw me …”. This gives 15,000 instances where the right reply includes signal tokens.
  3. MosIT. The paper’s own dataset: modality-switching instruction tuning. About 5,000 dialogues of 3 to 7 turns, where the human and the machine switch modalities from turn to turn.

The paper compares many multimodal instruction sets in its Table 6 and notes that their replies are limited to text. MosIT is built to fix that. The authors write template dialogues between a Human and a Machine, then prompt GPT-4 to produce more on over 100 topics. They require some implicit requests (the user hints, and does not ask), and reasoning, suggestion and planning in the machine’s replies. They then look for the best-matched real content, from retrieval systems such as YouTube and from generators such as Stable Diffusion XL and Midjourney, and people filter out bad dialogues.

A MosIT dialogue looks like this (our illustration, modelled on the paper’s Figure 4): Turn 1, the human gives a video and asks what is unusual, and the machine answers in text. Turn 2, the human asks for music for the scene, and the machine replies with words plus audio signal tokens. Turn 3, the human asks for similar scenes, and the machine replies with words plus image signal tokens. The modality of the input and of the output changes from turn to turn.

The failure mode

The symptom: after Stages 1 and 2 the model captions well and draws when given a caption, but it ignores a hint like “I am in a bad mood”, or it answers a music request with a picture. Measure it with human judges on instruction following. The fix is Stage 3 on MosIT. In the paper’s human test on 100 complex image requests, NExT-GPT beat two pipeline systems and a caption-only variant on instruction following, rationality and quality (Chapter 8).

Why does the paper add the MosIT dataset instead of relying on caption pairs alone?

Chapter 7

One Request, Start to Finish

Follow the dog video through every tensor, from your upload to the sound and picture that come back

You tap send on the dog video. You get words, a tune and a picture back. Behind that single reply are nine small events. This chapter walks them in order, with the shapes of the data. The paper’s Appendix C.6 and Figure 7 give the same procedure in a diagram.

Nothing in this chapter trains anything. Inference is a forward pass: weights are fixed, numbers flow one way. (Training in Chapters 3 to 6 adjusted the weights in advance.)

Step through the request

Press Next to move one event at a time. Each block is a tensor, one row per vector. Watch the shape change as the data crosses each tier.

Illustrative request modelled on the paper's Figure 4(A). Row counts are drawn to scale only loosely. N, M and T are not printed in the paper: we use M = 32 concept tokens and T = 21 text tokens as example values. The audio count (8) and image count (4) follow the paper's Section 6.2. Width 4,096 is Vicuna-7B's hidden size.

The nine events

  1. You send. Your video and your sentence arrive. The sentence is cut into T text tokens. In our example T = 21.
  2. Encode. ImageBind reads the video and returns a grid of N patch vectors, shape N × denc. Text does not take this path.
  3. Group. The grouping block (Chapter 3) shrinks N vectors to M concept tokens of the LLM’s width: M × 4096. This is the first time the video becomes something Vicuna can read.
  4. Read. Vicuna sees one row: the T text embeddings and the M concept tokens, (T + M) × 4096. It does not know which came from a video, except through what the vectors say.
  5. Write text. Vicuna writes its reply one token at a time. Each new token looks at everything before it, as in any chat model.
  6. Write slips. Because you asked for music and for scenes, the reply continues with audio tokens and image tokens (Chapter 4). Each is a hidden state of width 4,096: 8 × 4096 and 4 × 4096.
  7. Project. The router collects each family and sends it to its output projection (Chapter 5). Out come the conditions the decoders expect.
  8. Denoise. AudioLDM and Stable Diffusion each start from random noise and denoise in many small steps, steered by their condition. Their weights are frozen. This is the slowest event, because it repeats a large network many times.
  9. You receive. The text, the sound clip and the picture arrive together.

A worked count

Let us count what crosses each hand-off in our example. Between events 2 and 3, N patch vectors turn into 32 concept tokens. Between events 6 and 7, 8 audio states and 4 image states go out: 12 × 4,096 = 49,152 numbers. A relay would hand the sound tool a sentence such as “funny music for a dog on a skateboard”. That is 8 words, and the tool must turn them back into meaning on its own.

The number of vectors out is set by design, not by the length of the reply: 8 for audio, 4 for image, whatever you asked. This is the opposite of a sentence, whose length changes with how much you say. A fixed budget keeps the decoder’s input shape stable, which a frozen decoder needs.

Shorter paths

The nine events are the longest path. A text-only question skips events 2 and 3, since text goes straight to the LLM. A reply with no signal tokens skips events 6 and 7, so the decoders never run, which saves most of the compute. A request for video output swaps the audio and image branches for a longer video branch. The same code path serves all of these, and the choice is made by what the LLM writes.

What this trace shows

Count where the video was a video. It was a video only until event 3. After that, the LLM held vectors, never frames. And look at the hand-off between events 6 and 7: a block of vectors, not a sentence. If the first reply went wrong, say the music is sad, there is no sentence to blame. The cause lies in the vectors, which means in the weights that made them.

That gives the failure modes for inference. A wrong read appears as a text answer that misses what happened in the video; check it by comparing the reply to the clip, and fix it with better Stage 1 data. A wrong slip appears as no music when you asked for music; check it by counting signal tokens, and fix it with Stage 3 data. A weak decoder appears as a clean but generic picture; check it with scores such as FID, and fix it by changing the decoder, which the frozen design allows.

One more property is worth noting. The three decoders do not wait for each other. After event 7 the audio and image can be made at the same time. The paper does not report latency, so we make no claim about speed.

At inference time, what goes into the audio decoder?

Chapter 8

What the Tables Show, and What They Do Not

Read the generation scores, the projection ablation and the human test, and learn which claims they support

A paper’s tables are a set of answers. Each one answers a narrow question, and it is easy to hear a bigger answer than the table gives. This chapter lists the questions NExT-GPT’s tables answer, with the numbers, and then lists the questions they leave open.

Question 1: does it understand inputs as well as other MM-LLMs?

Tables 2 and 3 compare NExT-GPT with other models on image captioning, image question answering, video question answering and audio captioning. The paper reports strong image comprehension, results comparable to other models on the benchmarks built only for evaluation (MMBench and SEED-Bench), and better scores than CoDi on video and audio, which it credits to the LLM writing the text directly. Some of the table entries are marked as fine-tuned on the training set, so read the star marks before you compare rows. A few numbers from the ablation that follows are the same runs: VQAv2 66.7, VizWiz 48.4, MSVD-QA 64.5, MSRVTT-QA 61.4 and AudioCaps 81.3.

Question 2: how good are the things it makes?

Table 4 scores generation with three standard measures. FID (Fréchet Inception Distance) compares the statistics of generated images with real ones; lower is better. FAD does the same for audio; lower is better. CLIPSIM measures how well a video matches its prompt, using CLIP; higher is better. A dagger marks zero-shot rows, tested without training on that dataset.

Read the paper's numbers

Choose a result. Bars show the paper's own values. Turn the axis zoom off to see how little the bars differ when drawn from zero.

Values are copied from the paper: Table 4 for the first three sets, and Table 5 for the ablation (the text calls it Table 4). The ablation bars are our subtraction: each variant's score minus the linear-layer score, on five benchmarks. A dagger marks zero-shot. Different baselines were run by different groups, so rows are not perfectly controlled.

Look at the image set first. NExT-GPT scores FID 10.07, the best of the seven rows. Plain Stable Diffusion 1.5, the very decoder NExT-GPT uses, scores 11.21. Is that because the LLM adds value? Be careful here. The same model scored zero-shot (11.18) is almost the same as Stable Diffusion alone. The 10.07 row was tuned on the data. So the fair reading: NExT-GPT keeps its decoder’s quality and does not lose it, and with tuning it can improve a little.

Audio and video follow the same pattern. NExT-GPT’s FAD of 1.68 beats CoDi (1.80) and AudioLDM-L (1.96), and its CLIPSIM of 31.97 beats CoDi’s 28.90. The paper calls the video result a clear improvement and the zero-shot row (30.96) still leads CoDi. What the table cannot say is whether the LLM or the tuning caused the gain.

Question 3: does the grouping block earn its place?

The ablation (the paper’s Table 5) swaps the input projection for a single linear layer, or for a Q-Former plus linear layer, and trains each the same way. Select “Projection ablation” above. The Q-Former helps a little (1.0 to 2.6 points over linear). The grouping block helps more (2.9 to 4.3 points). On all five benchmarks the order is the same: linear, then Q-Former, then grouping. This supports the design choice. It does not tell us how the answer changes with larger models or more data.

Question 4: how many signal tokens are enough?

Figure 6 sweeps the count: 1, 2, 4 and 8 for images, 4 to 32 for video, and 2 to 16 for audio. The text of the paper reads the curves as follows. Images and audio are satisfied with 4 and 8. Video needs the most. The best count depends on the amount of data and on the power of the decoder.

Question 5: is end to end really better than a pipeline?

For this question the authors ran a human test. GPT-4 wrote 100 complex instructions that need implicit reasoning to make an image. Four systems answered with text plus an image: HuggingGPT, Visual-ChatGPT, a variant called NExT-GPT-caption (which writes a caption for the decoder, like a relay), and NExT-GPT itself. Five volunteers judged instruction following, rationality of the image, and quality. NExT-GPT was ahead on all three in the paper’s Figure 5. This is the cleanest test of the claim in Chapter 0, since NExT-GPT-caption differs from NExT-GPT mainly in the hand-off. It is small: 100 prompts and five judges.

What stays open

Which result best supports the claim that the grouping input projection is worth having?

Chapter 9

Field Guide: NExT-GPT on One Page

Review the numbers, the build checklist and the core code, then see where the paper sits in the field

This chapter is for the day you want to use the paper again: to explain it in five minutes, to check a number, or to start building. It adds no new ideas.

The paper in one paragraph

NExT-GPT (Wu, Fei, Qu, Ji and Chua, ICML 2024) builds an any-to-any assistant by joining existing parts. ImageBind encodes image, audio and video. A grouping input projection turns the patch grid into a few concept tokens that Vicuna-7B can read. Vicuna answers in text and, when it chooses, writes modality signal tokens. An output projection per modality turns those tokens into the condition that a frozen diffusion decoder (Stable Diffusion, AudioLDM, Zeroscope) expects. Training has three stages, and only about 155 million of 12.4 billion weights change. A 5,000-dialogue set, MosIT, teaches the model to switch modalities from turn to turn. The result scores close to or above the baselines on perception, and above the listed baselines on text-to-image, text-to-audio and text-to-video scores, and wins a small human test against pipelines.

The numbers that matter

WhatNumberWhy it matters
Trained weights155M of about 12.43B (about 1.25%)Reuse frozen parts, train the joints
Frozen partsImageBind 1.2B, Vicuna 7B, SD 1.3B, AudioLDM 975M, Zeroscope 1.8BQuality comes from the parts
Trained partsinput 28M, LoRA 33M, outputs 31M, 31M, 32MTable 1 of the paper
Output projectionhidden 512, 4 heads, 4 encoder and 4 decoder layersSmall transformer with learnable queries
Signal tokensimage 4, audio 8, video the most (24 in our reading)Video needs more slips
Stage 1 dataCC3M (3M+), WebVid-2M, AudioCaps (about 46K)Captions teach reading
Instruction dataT2M 15K instances, MosIT 5K dialogues of 3 to 7 turnsTeaches switching of modalities
Image generation, FID10.07 (zero-shot 11.18) vs SD-1.5 11.21Keeps decoder quality
Audio generation, FAD1.68 vs CoDi 1.80, AudioLDM-L 1.96Lower is better
Video generation, CLIPSIM31.97 vs CoDi 28.90Higher is better
Projection ablationVQAv2 66.7 vs 65.1 (Q-Former) vs 63.8 (linear)Grouping beat both on five benchmarks
Human test100 GPT-4 prompts, 5 volunteersEnd to end vs pipelines, small scale

Terms in one line each

A build checklist

  1. Load the frozen parts: ImageBind, Vicuna-7B v0, Stable Diffusion v1.5, AudioLDM l-full, Zeroscope v2-576w.
  2. Add the signal tokens to Vicuna’s vocabulary, one family per output modality.
  3. Build the grouping input projection (concept tokens, transformer, Gumbel assignment, straight-through, MLP merge).
  4. Build one output projection per modality (hidden 512, 4 heads, 4 + 4 layers, learnable queries).
  5. Stage 1: caption loss, train only the input projection.
  6. Stage 2: signal-token loss, caption alignment and denoising loss, train only the output projections.
  7. Stage 3: add LoRA to the LLM, train everything light on T2M and MosIT.
  8. At inference: scan the reply for signal tokens, project, run the decoders that are on.

The core as code

This is a sketch of the idea, with our shapes; it is not the authors’ code. enc, llm, inp, outp and dec stand for the parts of Chapter 2.

pythondef respond(text_ids, media):
    x = [llm.embed(text_ids)]                      # [T, 4096]
    for m in media:                                  # image / audio / video
        patches = enc(m)                              # [N, d_enc]   frozen ImageBind
        x.append(inp(patches))                     # [M, 4096]    grouping, trained
    tokens, hidden = llm.generate(cat(x))          # text + signal tokens, [len, 4096]
    outputs = {"text": detok(tokens)}
    for mod, prefix in SIGNALS.items():
        idx = [i for i, t in enumerate(tokens) if t.startswith(prefix)]
        if idx:                                      # token present: decoder ON
            cond = outp[mod](hidden[idx])             # [m, d_cond]  trained
            outputs[mod] = dec[mod].sample(cond)         # frozen diffusion
    return outputs

def stage2_loss(caption, media_gt, mod):
    h = llm.hidden_at_signals(caption)             # [k, 4096], LLM frozen
    cond = outp[mod](h)
    l_nll = llm.nll_of_signal_tokens(caption)
    l_cap = ((cond - dec[mod].text_encoder(caption)) ** 2).sum()
    l_den = dec[mod].denoise_loss(media_gt, cond)    # U-Net frozen, gradient flows through
    return l_nll + l_cap + l_den

Where this sits

NExT-GPT is the “LLM as brain, diffusion as hands” design. It belongs to the family of lecture-8 models that bolt encoders onto a language model: see Lecture 8, Large Multimodal Models, the generation lecture Lecture 9, Multimodal Generation, and the other assigned readings LLaMA-Adapter, Kosmos-2 and MM1.

Keep going

The takeaway. Do not retrain the experts. Keep the encoders, the language model and the decoders, and train only the small joints between them: a grouping block on the way in, signal tokens and a projection on the way out. Because the joint is a block of vectors, not a sentence, a learning signal can cross it.
You want to add a new output modality to NExT-GPT in the paper’s style. What do you add and train?

Now press Present or Teach and explain the system from memory: why a caption relay loses detail, what a signal token carries, which three losses train the output projection, and what the 1% means. Then go back to the flow at the top, send each request through both systems, and say what crosses the gap.

Credit: NExT-GPT: Any-to-Any Multimodal LLM, Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji and Tat-Seng Chua, NExT++ Research Center, National University of Singapore, ICML 2024 (PMLR 235). arXiv:2309.05519. Project page: next-gpt.github.io. All figures, tables and numbers here are from that paper unless marked illustrative or ours.

NExT-GPT: Any-to-Any Multimodal LLM
Read the paper · Project site · MMAI Lecture 8 · Back to Gleams