Spider answers one request with text, an image, audio and a video together, because its language model writes a separate signal prompt for every decoder.
Learn how one language model can steer many generators at once: the template it writes, the two prompts that guide each decoder, the shared projector, the two losses and the data that teach it.
Pick a request, then switch between NExT-GPT (the any-to-any model Spider improves on) and Spider, and press Send the request. Count which parts come back. Then we build, piece by piece, the template, the fusion, the projector, the losses and the training underneath.
You need a rough idea of what a neural network is and what an image generator does (it turns a text description into a picture). We build the rest from zero.
The paper: Spider: Any-to-Many Multimodal LLM (arXiv 2411.09439) by Jinxiang Lai, Jie Zhang, Jun Liu, Jian Li, Xiaocheng Lu and Song Guo, of HKUST and Tencent. Code: github.com/Layjins/Spider. Assigned reading for MIT MMAI, Multimodal AI (Spring 2026). It goes with the lecture Large Multimodal Models. Every idea is re-taught here in our own words and devices.
The four requests and what each model returned are the paper’s qualitative figures (Fig. 7, 11, 12, 13 and 14, Appendix L). The tiles are our drawing of those outcomes, not the paper’s pictures. For “Panda mask” NExT-GPT drew an image of a bear instead of a panda and gave no mask. For “Detect a tiger” the paper says NExT-GPT named the box and the video in its text but produced neither. Whether NExT-GPT also returned text in each case is our reading of its Text + X format.
Chapter 0
See why a model that returns text plus one extra modality forces you to ask again and again, and what Spider changes
Picture a travel app. You type: “Give me a travel guide for Beijing.” Back comes a page of text. It is a good page, but you wanted to see the Great Wall and hear a Peking opera. So you type a second request for a picture. Then a third for a sound. Then a fourth for a short video. Four requests, four separate answers, and nothing ties them together.
That is how most multimodal assistants work today. A modality is a kind of signal: text, an image, audio, a video. A multimodal large language model (MLLM) is a language model that has been given eyes and ears, and sometimes a voice and a paintbrush. The paper behind this lesson, Spider, starts from one complaint about the newest of them.
The models the paper calls any-to-any or X-to-X accept many kinds of input and can make many kinds of output. Here X stands for one modality. The catch is in the output. In one reply, an any-to-any model gives you Text + X: text, plus one extra modality. NExT-GPT, the model Spider is compared against throughout, works this way. The paper calls the habit multi-round dialogue generation.
Spider aims at something else. The authors call it Any-to-Many Modalities Generation, written AMMG or X-to-Xs. The lowercase s means “several”. One reply gives you Text + Xs: text plus any mix of image, audio, video, a box drawn around an object, and a mask that outlines an object. The appendix states the task in one line. The input query Q is text, or text plus one modality. The output Y is a sequence of k parts, with k at least 1, and each part may be any modality. The goal is to satisfy every requested modality in a single response.
Try the cost of the old way. Slide the number of extra modalities you want and count the rounds each approach needs.
Each box is one round: you ask, the model answers. The top row is an any-to-any model that gives Text + one X per reply. The bottom row is Spider. Drag the number of extra modalities you want.
Illustrative rule, taken from the paper’s Fig. 7 example: every reply carries Text plus at most one extra modality, so m extra modalities need m replies. The paper does not count rounds in a table. The five modalities here are image, audio, video, box and mask.
With one extra modality the two rows tie. By five, the old way needs five rounds and Spider still needs one. That is the user-facing story. It also hides a harder problem.
Think of the language model as a film director. For a normal reply the director only speaks. For a many-modal reply the director must also hand each crew member a clear instruction: the painter gets a scene, the sound engineer gets a sound, the camera operator gets a shot. A mix-up is easy. The paper gives the example “Generate an image of a dog, and I also would like to hear the dog’s bark.” The word image is loud. A model may hear only an image job. It may skip the bark. Or it may hand the painter the words “dog’s bark” and get a picture of a dog barking.
This is a one-to-many task. The language model has to read the whole request, split it into parts, and write the right prompt (a short instruction) for each decoder. A decoder is the generator that finally makes the pixels or the sound. In Spider these are existing models: Stable Diffusion for images, AudioLDM for audio, Zeroscope for video, Grounding DINO for boxes and SAM for masks.
Spider answers with three ideas. A template teaches the language model to write one tidy group per modality (Chapter 2). A Decoders-Controller turns what the model writes into signals each decoder understands (Chapters 3 to 5). And a new dataset teaches the format using text-only answers, so nobody has to retrain the decoders (Chapter 6).
The instrument at the top of the page is a view of this chapter’s claim: the same request, two models, and a count of what comes back. Chapters 2 to 5 explain why Spider’s tiles fill in.
Chapter 1
Follow a request through encoders, a language model, a controller and decoders, and see what shape the data has at each hand-off
Imagine a restaurant that serves many cuisines. Customers order in different languages. A waiter listens to each order and rewrites it on a ticket in one common language. A head chef reads the ticket and writes instructions for the kitchen: “grill station, one steak; pastry station, one tart.” Each station is run by a specialist who only reads instructions in its own style. A runner translates the chef’s note into each station’s style.
Spider has the same four roles. The waiter is the encoders. The chef is the language model. The runner is the Decoders-Controller. The stations are the decoders. This chapter walks one request through all four and names the shape of the data at each hand-off, so that you could draw the whole thing on a whiteboard.
A neural network cannot read a photograph or a sentence. It reads lists of numbers. An embedding is a list of numbers that stands for the meaning of something, built so that similar things get similar lists. A picture of a tiger and the words “a tiger” end up near each other. We write an embedding as a vector, a row of numbers, and a batch of them as a table with one row for each item. The shape of that table is its number of rows and its number of columns. We write it N × D: N rows, D numbers in each.
A token is one row of such a table, as the language model sees it. Text gets cut into word pieces, and each piece becomes one row. A picture can also become rows. Keep this in mind. When the paper says it uses “one token” for an image, it means one row of D numbers carries the whole message.
An encoder turns raw input into an embedding. Spider uses a single one for everything, ImageBind. ImageBind accepts six modalities and puts them all into one shared embedding space. That saves Spider from juggling a separate encoder for every kind of input. The ImageBind output is called EX in the paper (X is the modality). It does not yet live in the language model’s own number space. A small Encoder Projector, a plain linear layer (a matrix multiply plus a shift), maps it there. After that, the language model can treat the picture as if it were a few extra words in the prompt.
The LLM is LLaMA2, an open-source language model. It reads the projected input plus your text instruction and then does what language models do: it writes text, one token at a time. Spider asks it to write three things in one string. First, the ordinary text of the answer. Second, a short T-Prompt for each decoder (T for text). Third, a special marker token called the M-Prompt for each decoder (M for modality). Chapter 2 shows the exact string. Chapter 3 explains why there are two kinds of prompt.
Training the whole LLaMA2 model for this would be costly, so Spider adds LoRA adapters. LoRA keeps the original weights fixed and learns a small correction beside them, built from two thin tables of numbers that multiply into a full-size update. Only the correction trains. Our LoRA lesson builds it from zero.
The language model writes in its number space. Each decoder expects a signal in its own number space. The Decoders-Controller bridges the gap. It has two pieces: the Unified Decoder Projector (Chapter 4) and TM-Fusion (Chapter 3). Together they take a T-Prompt and an M-Prompt and produce one controlling embedding, written SX, per decoder. That embedding is what the decoder reads in place of a normal text prompt.
The five decoders are latent-conditioned generators. Conditioned means each one makes its output from a signal you supply. For Stable Diffusion that signal is normally the embedding of a text prompt. Spider swaps in SX, which has the same job and the same shape. The five decoders and their outputs are Stable Diffusion v1.5 (image), AudioLDM (audio), Zeroscope v2 (video), Grounding DINO (bounding box) and SAM, the Segment Anything Model (mask). Our diffusion lesson shows how such a generator turns a condition into pixels.
Step through the pipeline with the request from the paper’s Fig. 12: an image of a tiger plus the instruction “Detect tiger, and generate an image and a video for it.”
The four boxes are the four parts. Press Next step to move the request along. The panel under the picture says what goes in and what comes out, with shapes.
The four parts, the five decoders and the one-token-per-modality count are the paper’s (Sections 2.1 and 4). The paper writes shapes with symbols (N×D, L×D) and gives no widths for them, so the panel does too. The tiger request is the paper’s Fig. 12.
Read the tiger request as a table of shapes. Input: one photo, EI from ImageBind, a vector of D′ numbers. After the Encoder Projector it becomes rows in the language model’s space. The text instruction becomes more rows. The LLM then writes its output tokens, which include the three groups for box, image and video. For each group, the controller builds one SX of NX rows by D columns. Three decoders run, and three pieces of output come back. The user sees one response.
One design choice here matters a great deal. The paper reports that Spider uses one token each for image, audio and video in its M-Prompt, where NExT-GPT uses 4, 8 and 24. Spider can afford this because most of the control comes from the T-Prompt, which is plain text and easy for a language model to write. Chapter 3 shows the effect.
Chapter 2
Read the template that turns the language model’s plain text into a plan of which decoders to call
A film director has a call sheet. It is a plain page of text, but it has a fixed shape. “Scene 4: SOUND, rain on a window. Scene 5: PICTURE, an empty street.” Each line starts with a department name. The crew does not need to understand the director’s intent. They find their department and read the line under it.
Spider gives its language model a call sheet of this kind. A language model can only write text. So the plan for many modalities has to be inside the text, in a form a small program can read back. The paper calls this the Any-to-Many Instruction Template. It has two halves: a format for the question and a format for the answer.
Every question the model sees has four parts, in this order. [INPUT] is a start marker. Then comes a TaskPrompt, a label that says what kind of output is wanted. Then the input modality, if there is one, wrapped in begin and end tags with its embedding inside: <IMAGE> E </IMAGE>. Last, the user’s own words, called the text instruction.
There are eight TaskPrompts, in three modes. Single Modal has six labels: [TEXT], [IMAGE], [AUDIO], [VIDEO], [BOX], [MASK]. They say “produce only this.” Smart Multimodal is the label [SMARTMULTIMODAL]. It tells the model to choose any mix that fits the request. Specific Multimodal is [SPECIFICMULTIMODAL]. Here the user writes the answer’s layout directly, and the model follows it.
One example from the paper’s Fig. 4b: “[INPUT] [SMARTMULTIMODAL] Generate an image of a dog, and I also would like to hear the dog’s bark.”
The answer starts with [OUT] and ends with [END]. Between them sit one or more modality groups. A modality group is one piece of text, a begin tag, a T-Prompt, an M-Prompt and an end tag:
The paper calls this layout Modality-wise Grouping. Every request for one modality gets its own tagged group, placed right beside the text it belongs to. Groups can be chained in any order and any number. That last point is what makes “any-to-many” possible: the format has no limit on how many groups appear.
The full answer to the dog request is: “[OUT] Here is the generated image of a dog: <IMAGE> dog [IMAGE0] </IMAGE>, and the dog’s bark: <AUDIO> dog’s bark [AUDIO0] </AUDIO> [END]”. Notice that “dog’s bark” sits in its own group, under an AUDIO tag. It cannot be swallowed by the image group. That is the fix for the problem of Chapter 0.
The language model finishes its string. A small program then scans it. For every complete group it reads the tag name, takes the words between the tag and the M-Prompt as the T-Prompt, and calls the matching decoder. Everything outside groups is shown to the user as text. Try the scan yourself. Pick an answer, then break one tag and see what the scan loses.
The box shows the string the language model wrote. Pink marks a group the scan accepts. Red marks a broken tag. Press Break a tag to delete the first end tag.
The three strings are the paper’s examples (Fig. 4b, Fig. 5a), and a short guide built from the Fig. 1d groups (“Forbidden City”, “Peking Opera”). The scan is our own regular expression, run live in your browser. The paper does not publish its parser, and does not say what happens to a broken group.
Symptom. In Fig. 12, a user asks NExT-GPT to detect a tiger and generate an image and a video. NExT-GPT mentions the video and the box in its text reply. It produces neither. It generates only an image.
Measurement. The paper builds Spider-base, a model that is like NExT-GPT (multiple projectors, M-Prompt only) but uses the new template. Table 4 scores both on the image-to-many task. The image FID (lower is better) drops from 12.52 for NExT-GPT to 9.24 for Spider-base. The text score B@4 (higher is better) rises from 32.4 to 40.5. The full Spider reaches 5.11 and 40.8. So the template alone moves a large part of the gap, and the controller of Chapters 3 to 5 moves the rest.
Fix. Write the plan as grouped text, and train the model on it (Chapter 6). That is all the template is: a format, plus the data that teaches it. It needs no new network.
There are three good reasons. First, a language model already writes text well, so the format needs no new network part. Second, text is easy to inspect: you can print the plan and read it, which makes failures such as a missing tag easy to find. Third, text is easy to train on. Chapter 6 shows that a dataset of tagged strings can be built from ordinary caption sets, with no generated images in it.
The three modes also give the user a choice of control. With a Single Modal label the user fixes the output kind. With [SMARTMULTIMODAL] the model decides, as in the dog and bark example. With [SPECIFICMULTIMODAL] the user writes the groups they want and the model fills them in. The same answer format serves all three.
Chapter 3
Combine a text prompt and a modality prompt into one controlling embedding, and see why the hint gets a small weight
You hire a painter. The easy way to brief them is with words: “a tiger walking through snow.” Words are cheap to write and the painter knows what to do with them. But suppose you also have a photo of the exact tiger you mean. Words lose detail: the stripe pattern, the angle of the head. So you hand over the photo as a reference, and tell the painter to lean on the words first and glance at the photo second.
That is the idea behind Spider’s two prompts. The decoder is the painter. The T-Prompt is the words. The M-Prompt is the reference photo. This chapter shows how the two are blended into one signal, and why the blend is lopsided on purpose.
A decoder such as Stable Diffusion does not read text directly. It reads an embedding, the list of numbers from Chapter 1, made from text by its own text encoder. The paper calls that encoder the decoder’s “original conditional encoder”. So the simplest way to control a decoder is to take the T-Prompt, run it through that text encoder, and hand over the result. The paper writes this TeX. It works because a language model finds it easy to write the right short words, and the decoder already understands words.
NExT-GPT does the opposite. It controls decoders with the M-Prompt alone, a few hidden tokens learned by the language model. That needs the model to learn a whole new language for each decoder, and it needs many tokens: 4 for an image, 8 for audio and 24 for video, the paper reports. Spider uses just one token for each.
If the T-Prompt does the main work, what does the M-Prompt add? When the LLM writes a token such as [IMAGE0], it also computes a hidden state for that token, a row of numbers inside the model that has already read the input photo. The paper calls this hidden embedding MeX. It still contains what the model saw, including detail that words cannot hold. That detail is the hint.
Chapter 4 explains how MeX is passed through the projector and becomes Q̄X, a table of rows the decoder can use. For now, treat Q̄X as the reference photo, already translated.
TM-Fusion (“T and M fusion”) is the module that mixes the two. It has two short equations.
Read it as a recipe: take the words, add a fifth of the hint. Here is a worked example with made-up four-number vectors, so you can follow every step. Say the words give Te = [0.8, 0.1, −0.3, 0.5], and the resized hint is fL(Q̄) = [0.5, −1.0, 0.9, 0.0]. Multiply the hint by α = 0.2 and you get [0.1, −0.2, 0.18, 0.0]. Add it to the words: S = [0.9, −0.1, −0.12, 0.5]. The hint nudged every number a little, but the vector still points mostly where the words pointed. Its cosine similarity with Te (1 means the same direction, 0 means unrelated) is 0.961.
Why a small weight? The decoder was trained to understand embeddings made by its own text encoder. A vector that stays close to Te is one it already understands. A vector that strays far is noise. Move the hint in the device below and watch how far α lets S stray.
The teal arrow is the words (Te). The purple arrow is the hint. The pink arrow is what the decoder gets: the words plus α times the hint. Drag α and turn the hint.
A two-number toy: the words are the arrow (1, 0) and the hint has length 1.5, an illustrative size. Real embeddings have hundreds of numbers. α = 0.2 is the paper’s value; the direction slider stands for how different the hint is from the words.
Table 5 tests the recipe three ways. With the M-Prompt alone (the NExT-GPT way, with one token), text-to-image FID is 14.07 (lower is better) and image editing CLIP is 22.19 (higher is better). With the T-Prompt alone, FID is 11.11 and CLIP is 29.04. With both fused, FID is 11.13 and CLIP is 30.52. For text-to-image, the words alone are about as good as the blend. For editing a given image, the hint adds a point and a half of CLIP. Audio editing tells the same story: MCD (lower is better) is 0.424, then 0.322, then 0.279. Video editing CLIP-T: 0.2103, 0.2559, 0.2782.
So the words carry most of the control, and the hint supplies what is left over: the detail of the input. That is the “dominant T-Prompt, auxiliary M-Prompt” design.
Chapter 4
Share a small team of expert stacks and a router across five decoders, instead of building one projector for each
Back to the restaurant. The head chef writes a note. Five stations each speak a different dialect. One plan: hire five translators, one per station. That works, but every new station needs a new hire, and each hire must be trained from scratch. A second plan: set up one translation desk with a small team of two specialists and a dispatcher. For each note, the dispatcher decides how much of each specialist’s version to use. New stations share the same desk.
The second plan is the Unified Decoder Projector (UDP). The first is what the paper calls Multiple Projectors (MP), and it is how NExT-GPT is built. This chapter shows how the desk works and why a team of two is enough for five decoders.
Two ideas first. A transformer layer is a block that lets every row of a table look at every other row and mix in what it finds, then reshapes each row through a small network. A stack of such layers repeatedly refines a table of rows. The projection experts in Spider are stacks like this. A mixture of experts is a design where several experts each compute an answer and a router decides how much of each to use. The router outputs one weight for each expert, and the weights are made by a softmax: a function that turns any list of scores into positive weights that sum to 1. Our mixture-of-experts lesson builds the idea in full.
For an output modality X the projector needs three things.
In words: every expert reads the same two inputs and writes its own version of the hint. The router looks at the M-Prompt embedding and decides the mix. The output is the mixed hint. Because the router reads MeX, which differs for each modality, different modalities can lean on different experts. Yet all five share the same K stacks.
Take K = 2 and look at only one number of the hint. Say the router scores are [0.85, 0]. The softmax gives e0.85 = 2.34 and e0 = 1, a total of 3.34. So w = [0.70, 0.30]. Say expert 1 returns the row [2.0, −1.0] and expert 2 returns [0.0, 1.0]. The mix is 0.7 × [2.0, −1.0] + 0.3 × [0.0, 1.0] = [1.4, −0.7] + [0.0, 0.3] = [1.4, −0.4]. Both experts contributed, in the proportion the router chose. All numbers here are made up for the example.
Try it on the five decoders. The router scores in the device are invented so you can see the idea, because the paper does not publish its learned weights.
Pick a decoder. The router weights show how much of each expert goes into its hint. Then change the number of experts K. Watch the count of stacks you have to train.
The router scores are invented for illustration; the paper does not report learned weights. What is the paper’s: K = 2 is enough for the five decoders (Fig. 3, Table 5 group 3), and the multiple-projector design needs one stack for each modality.
One more detail. The router sees only the M-Prompt embedding, never the T-Prompt words. So its decision depends on what the language model understood about the input and the target modality, which is the very information the hint is meant to carry. The paper gives a larger representation capacity as the reason the mixture beats a single projector.
Table 5 compares them. With Multiple Projectors, image editing CLIP is 29.42 and audio MCD is 0.298. With the UDP, CLIP is 30.52 and MCD is 0.279. One expert (K = 1) gives CLIP 29.34, close to MP, which says the gain comes from the mixing and not just from sharing. Two experts give 30.52. Three give 30.49, no better. On the image-to-many task the image FID goes 6.00, 5.11, 5.03 for K = 1, 2, 3. The paper settles on K = 2: the third expert buys almost nothing.
The structural argument is plain. MP needs one stack per decoder, so adding a sixth modality adds a sixth stack. The UDP adds only a small M-Query for the new modality. The paper calls this “more structure-efficient and scalable”.
Chapter 5
Use one cosine loss to keep the decoder happy and a second to stop the hint from fading to nothing
A student has two jobs on a group project. Job one: hand in something the teacher can read, so the teacher accepts it. Job two: put in some real content. There is an easy way to do job one perfectly. Hand in nothing extra: the teacher already understands blank pages. The student passes job one, fails job two, and nobody notices until the final result is poor.
That is the danger inside Chapter 3. TM-Fusion adds a small hint to the words. If training only rewards “the decoder understands what you hand it”, the cheapest way to win is to shrink the hint to nothing. The paper adds a second rule to block this. Together they are the two losses of the Decoders-Controller.
Both losses use cosine similarity. Take two vectors and draw them as arrows from the same point. The cosine similarity is the cosine of the angle between them. It is 1 when they point the same way, 0 when they stand at a right angle, and −1 when they point opposite ways. It ignores how long the arrows are. It looks only at direction. In symbols, for vectors a and b: cos(a, b) = (a · b) / (|a| · |b|), where a · b is the sum of products of matching numbers and |a| is the length of a. Training programs usually turn “make cosine 1” into a loss by minimising 1 − cos, which is 0 at the target.
The decoder knows how to read embeddings made by its text encoder, TeX. The controlling embedding SX must look like one. So the first loss pulls them together:
Look at S = Te + α · fL(Q̄). The first loss is met when the added part is small or points along Te. The cheapest solution is to let Q̄ shrink toward zero. Then S equals Te and the cosine is exactly 1. That is the student who hands in a blank page.
The second loss ties the hint to something real. EX is the encoder’s embedding of the input photo, sound or clip. If Q̄ must point the way EX points, then it keeps a description of the input. The paper gives two purposes: “to retain the input modality information”, and “to prevent Q̄X from collapsing toward zero in the M-Alignment loss”. Fig. 2(c) in the paper draws the intended picture in embedding space: S near Te, and Q̄ near E.
The two losses do not fight, because α is small. Even when Q̄ is exactly E, S = Te + 0.2 E points only slightly away from Te. Both cosines can be high at once. Drag the hint around in the device to see where that happens.
The teal arrow is the words Te. The blue arrow is the input embedding E. Drag the purple dot, the hint, anywhere. The pink arrow is S = Te + 0.2 × hint. Try the buttons.
A two-number toy with the identity for fL. The words point at 0°, the input at 65°, both with length 1, and α = 0.2 as in the paper. The paper states the two losses as cosine targets (Section 2.2); the 1 − cos form is the usual way to train them, not a quote.
Use the device’s toy. The words point along 0° with length 1, so Te = (1, 0). The input points at 65°, so E = (0.42, 0.91). Put the hint exactly on E and take α = 0.2. Then S = (1 + 0.2 × 0.42, 0.2 × 0.91) = (1.08, 0.18). The cosine of S with Te is 1.08 divided by the length of S, 1.10, which is 0.99. Both losses are met: the hint equals E, and S has barely moved. Now point the hint away, at (−1.2, −0.5). Then S = (0.76, −0.10) and the alignment cosine is still 0.99. The alignment loss hardly notices a hint that points the wrong way. Only the second loss does. That is why the second loss is needed to keep the hint useful.
Symptom. Without the second loss, editing tasks lose detail from the input. The paper’s Fig. 6(a) shows the effect: asked for an image similar to the input image, Spider keeps visual details that NExT-GPT loses.
Measurement. Table 5, group 4, compares Spider without the M-Reconstruction loss (MRL) to Spider with it. Image editing CLIP rises from 29.89 to 30.52. Audio MCD (lower is better) falls from 0.289 to 0.279. Video CLIP-T rises from 0.2737 to 0.2782. On the image-to-many task, image FID falls from 5.73 to 5.11. On text-to-image, nothing changes: FID is 11.14 without and 11.13 with. That fits the story. When the T-Prompt does all the work, the hint does not matter.
Fix. Add the second cosine term. One caution from us, not the paper: cosine looks only at direction, so it cannot by itself force the length of Q̄ to stay above zero. The paper reports the benefit through its ablation numbers rather than proving the collapse argument.
Chapter 6
Build a dataset whose answers are only text, then train in three stages, each one unfreezing a different set of parts
You want to teach a new employee to write a call sheet (Chapter 2). The ideal training set would be thousands of finished productions: sheets paired with the pictures, sounds and clips they produced. Nobody has that. Such pairs do not exist at scale, because no earlier model could make them.
Spider’s trick: train on the call sheets only. The paper builds a dataset whose answers are pure text, with the tags inside. Since the model only has to write text, no decoder has to run during training, and none has to be retrained. This chapter shows how that dataset is built and how the three training stages use it.
TMM stands for Text-formatted Many-Modal. Every example is a question and an answer. The question is text, or text plus one input modality, in the Chapter 2 question format. The answer is TXs: text that contains the many-modal tags. The paper builds three families.
The paper gives a three-step recipe for building one example. Step (a): write a pool of instruction templates that name modality combinations, such as “Please generate an image and a video based on the following text: {}”. Step (b): pick one at random and fill the gap with a caption. Step (c): read the template to find the target modalities and write the matching answer.
Worked example with the paper’s own sentence. Start from the caption “A cat is sitting on a couch”. Pick the template above. The question is “[INPUT] [SMARTMULTIMODAL] Please generate an image and a video based on the following text: A cat is sitting on a couch”. The template names two targets, image and video, so the answer is “[OUT] A cat is sitting on a couch. <IMAGE> A cat is sitting on a couch [IMAGE0] </IMAGE>. <VIDEO> A cat is sitting on a couch [VIDEO0] </VIDEO> [END]”. The caption supplies the T-Prompt twice. No picture or clip is made. Each example is built online during training, by pairing a stored sample with one of the instructions.
The paper trains in three stages. Each one changes which parts of Spider learn and which stay frozen. A frozen part keeps its weights fixed. Freezing saves memory and protects what a part already knows.
Stage 1: X-to-X pretraining. The goal is to connect the four parts so ordinary any-to-any work succeeds. Only three things train: the input-side Encoder Projectors, the LoRA adapters in the language model, and the output-side Decoders-Controller. The encoders (ImageBind) and decoders stay frozen. Tasks are captioning for image, audio and video (this aligns inputs with the language model), text-to-image, text-to-audio, text-to-video, image-to-box and image-to-mask (this aligns the language model’s output with the decoders, using the two losses of Chapter 5).
Stage 2: X-to-TXs finetuning. After stage 1 the model can only write a signal prompt for one modality. Now the LoRA adapters train on the T-to-TXs and X-to-TXs datasets, so the model learns to write many groups. The training target is pure text.
Stage 3: instruction finetuning. The goal is to follow what the user actually asks. The LoRA adapters train again. Each iteration draws 70% of its data from the instruction dataset and 30% from the T-to-TXs and X-to-TXs datasets of stage 2. The 30% is a reminder, so the model does not forget the earlier skills.
Table 22 gives the settings. All three stages use the Adam optimiser (the rule that nudges the weights after each batch), learning rate 0.0001, weight decay 0.001, batch size 4 per GPU and a maximum of 1,024 tokens. An iteration is one weight update on one batch, and an epoch here is a block of iterations. Stage 1 runs 10,000 iterations per epoch for 40 epochs. Stages 2 and 3 run 1,000 iterations per epoch for 20 epochs each. Multiplying out: 400,000, 20,000 and 20,000 iterations. The table has a row “Freeze LLM: No” for all three stages. The text says only the LoRA of the LLM trains. We read the table as “the LLM takes part in training through its LoRA”.
Choose a stage. Yellow tiles train. Grey tiles are frozen. The bar under them shows the data that stage draws from.
Stage contents, the 70/30 mix and the settings are from Section 2.5, Appendix J and Table 22. We list the base LLaMA2 weights as frozen and the LoRA adapters as trainable, following the paper’s text.
There is a bonus. Once trained, Spider can run on its own inputs and actually call the decoders. The result is a new dataset of real many-modal answers: text plus images, audio and video together. The paper calls it the pseudo X-to-Xs dataset, “the first-ever” of its kind, and says it can later supervise end-to-end training of the decoders with the language model. “Pseudo” means the outputs came from a model, not from people. The paper reports statistics in its Table 21, and we do not reproduce them here.
Chapter 7
Read the benchmark tables and the any-to-many table, and separate the tiny gains from the large ones
A report card has many subjects. A student who gets 91 where a classmate got 90 has not changed the story. A student who gets 91 where the classmate got 40 has. When you read a paper’s tables, the first skill is to ask: how big is this gap, and where does it come from? Spider’s tables hold both kinds of gap.
A metric is a number that scores quality. Each table row names one. The ones the paper uses are listed here, with the direction in which they improve.
The first tests follow NExT-GPT’s setup, so the comparison is fair. Table 1 asks the model to write captions for an image, a sound and a video. Spider scores 158.8 CIDEr on COCO (NExT-GPT 158.3, OFA 154.9), 0.537 SPIDEr on AudioCaps (0.534, CoDi 0.480) and 59.8 B@4 on MSR-VTT (58.8, mPLUG-2 57.8). Table 2 asks the model to generate. Text-to-image FID is 11.13 (NExT-GPT 11.18, Stable Diffusion 11.21). Text-to-audio FD is 23.02 (NExT-GPT 23.25, but CoDi at 22.90 is better). Text-to-video CLIPSIM is 0.3258 (0.3197). Table 3 asks the model to edit given text: image edit FID 5.33 (NExT-GPT 6.62, PFB-Diff 5.93), audio MCD 0.279 (0.300), video CLIP-T 0.2782 (0.2684, but Pix2Video at 0.2891 is better).
Look at the pattern. Spider is first or tied nearly everywhere, but by thin margins, and it is not first everywhere: CoDi wins text-to-audio FD, and PFB-Diff wins image-edit CLIP (30.81 against Spider’s 30.52). The paper itself says only “superior to NExT-GPT” and “competitive” with the best methods. Why so close? The decoders are the same frozen models that NExT-GPT uses. If Stable Diffusion draws the picture in both cases, the picture can only differ by the signal it was handed. The signal moves a hundredth of an FID point.
The caption results come from a different place. There the language model writes the answer itself, and the paper credits “the inherent expertise of the LLM”. The gains are still small: half a point of CIDEr.
Pick a task. Bars start at zero, so you see how big each gap really is. The green bar is the best score in the table. Lower is better for FID, FD and MCD.
Every number is from the paper’s Tables 1, 2, 3 and 4. Table 4 comes from the authors’ own TMM-based test sets, which the authors built in the same text-formatted way as Spider’s training data.
Now the large gap. Table 4 tests the new task: one question, many outputs. The paper compares NExT-GPT, Spider-base (NExT-GPT’s projector and M-Prompt-only control, but with the new template) and Spider. Each many-modal answer is cut into groups, and each group is scored alone: I is image FID, A is audio FD, V is video FID, B@4 is text. The four test families start from an image, an audio clip, a video, or text only.
On the image-to-many task, image FID is 12.52 for NExT-GPT, 9.24 for Spider-base and 5.11 for Spider. Audio FD is 39.12, 22.75 and 18.44. Video FID is 32.41, 18.95 and 15.77. On the text-to-many task, with the travel guides, image FID is 35.16, 20.73 and 17.48, audio FD 60.53, 38.27 and 33.02, video FID 37.68, 23.11 and 20.02. Text B@4 is 32.4, 40.5 and 40.8. These gaps are large, often a factor of two or more.
The paper explains: Spider and Spider-base beat NExT-GPT “by a large margin” because the template lets them produce many-modal output, “while NExT-GPT can only produce single-modal X”. Spider beats Spider-base thanks to TM-Fusion and the projector, and the margin is largest on tasks where the T-Prompt dominates.
Two honest readings of our own, not the paper’s. First, the paper does not say how a group is scored when a model fails to produce it, so we cannot tell how much of the gap is “a worse image” and how much is “no image”. Second, the test sets come from the same TMM pipeline that trained Spider, so NExT-GPT faces a format it never saw. Neither point undoes the main result, which Fig. 10 to 14 show by eye. But they say how much weight the table can bear.
Chapter 8
Remove one design choice at a time, read what breaks, and name what the paper does not show
A mechanic suspects a rattle comes from one of five parts. The test is simple: remove one part at a time, drive, and listen. A paper’s ablation study works the same way. It takes the finished model, removes or swaps one design choice, and measures what gets worse. The parts that matter show up as a drop. The parts that do not matter show up as silence.
Table 5 is Spider’s ablation. It has five groups, and each tests one claim from Chapters 3 to 5. It reports nine numbers per row. We follow three that tell the story: T2I (text-to-image FID, lower is better), I2I (image-editing CLIP, higher is better) and XI-to-Xs image FID (the image part of the any-to-many task, lower is better).
Group 0, the baseline. NExT-GPT scores 11.18, 29.32 and 12.52.
Group 1, the projector. Swapping Multiple Projectors for the Unified Decoder Projector moves the scores from 11.17, 29.42 and 5.94 to 11.13, 30.52 and 5.11. The text-to-image number hardly moves, as the paper says: “slightly better on Text-to-X”, because the hint matters little there. The gain shows up in editing, where a hint carries input detail.
Notice the third number. Spider(MP) keeps NExT-GPT’s multiple projectors, yet it already scores 5.94 on the any-to-many task, where NExT-GPT scored 12.52. The new template and the prompt design halve the image FID before the shared projector is even used. This matches the Chapter 2 finding from Spider-base.
Group 2, the prompts. M-Prompt only gives 14.07, 22.19 and 8.13: clearly worse than everything else. T-Prompt only gives 11.11, 29.04 and 6.03. Fused (TMF) gives 11.13, 30.52 and 5.11. The words give most of the quality. The hint wins back 1.48 CLIP points on editing and about 0.9 FID points on the any-to-many task. The paper gives two reasons the M-Prompt-only row is weak: it uses 1, 1 and 1 tokens for image, audio and video where NExT-GPT uses 4, 8 and 24, and it trained 4 times fewer iterations than NExT-GPT. We note these are not controlled: the row mixes several changes.
Group 3, the number of experts. K = 1, 2, 3 give 11.18, 11.13, 11.13 on T2I; 29.34, 30.52, 30.49 on I2I; 6.00, 5.11, 5.03 on image FID. Two experts capture the gain. A third adds almost nothing.
Group 4, the reconstruction loss. Without it the scores are 11.14, 29.89 and 5.73. With it: 11.13, 30.52 and 5.11. The effect appears only where the hint matters.
Pick a row of Table 5. Each bar shows how much worse that metric is than for the full Spider, in percent. A bar at zero means no loss.
The numbers are Table 5 of the paper (T2I, I2I and XI-to-Xs image FID columns). “Percent worse” is our own arithmetic: the difference from full Spider divided by full Spider’s score. The paper does not report variation across runs.
Read the limits as a checklist for your own use of the method.
Several rows in Table 5 are the same model. Spider(UDP), Spider(TMF), Spider(K=2) and Spider(w MRL) all show 11.13, 30.52 and 5.11 on our three columns, because each is the full Spider seen from a different group. When you read an ablation table, look for these repeated rows first. They show the reference point, and every other row is a distance from it. That is exactly what the device above draws.
Second, check that the rows move where the theory says. The projector, the hint and the reconstruction loss all matter for editing and for the input-to-many tasks, and not for plain text-to-image. The table agrees on every one of them. That agreement is the best evidence in the paper that each part does the job the authors claim.
Chapter 9
Review the numbers and terms, run the checklist, read the core as code, and find where to go next
You can now rebuild Spider from the page. Its claim is this: one language model can steer many generators in one reply, if it writes a tagged plan (Chapter 2), if each decoder gets a dominant text prompt plus a light hint (Chapter 3), if a small shared projector turns the hint into the decoder’s space (Chapter 4), if two cosine losses keep the hint alive (Chapter 5), and if training uses text-only plans in three stages (Chapter 6). The cost is small: single-modality quality stays level with NExT-GPT, and the gain on many-modal answers is large (Chapters 7 and 8).
| What | Number or meaning | Where |
|---|---|---|
| Spider’s five decoders | Stable Diffusion v1.5, AudioLDM, Zeroscope v2, Grounding DINO, SAM | Section 2.1 |
| Encoder and LLM | ImageBind (six modalities), LLaMA2 with LoRA | Section 2.1 |
| TaskPrompts | 8: six single, [SMARTMULTIMODAL], [SPECIFICMULTIMODAL] | Fig. 4 |
| TM-Fusion weight | α = 0.2 | Eq. 4 |
| Experts in the projector | K = 2 (K = 3 gains almost nothing) | Table 5, group 3 |
| Tokens per modality | Spider 1, 1, 1; NExT-GPT 4, 8, 24 (image, audio, video) | Section 4 |
| Training stages | 400,000, 20,000 and 20,000 iterations; stage 3 mixes 70% / 30% | Appendix J, Table 22 |
| Image-to-many, image FID | NExT-GPT 12.52, Spider-base 9.24, Spider 5.11 | Table 4 |
| Text-to-image FID | Stable Diffusion 11.21, NExT-GPT 11.18, Spider 11.13 | Table 2 |
| Image-edit CLIP, hint removed | T-Prompt only 29.04, fused 30.52 | Table 5, group 2 |
| Reconstruction loss on editing | 29.89 without, 30.52 with | Table 5, group 4 |
This sketch is ours, written from the paper’s equations. The paper’s code is at github.com/Layjins/Spider.
pythondef spider_reply(question, input_x=None): E = imagebind(input_x) # E^X [1 x D'] (Ch 1) ids = proj_in(E), tokenize(question) # rows the LLM can read out = llama2_lora.generate(ids) # "[OUT] ... <IMAGE> dog [IMAGE0] </IMAGE> ... [END]" (Ch 2) results = [] for g in parse_groups(out): # one closed group per modality Me = hidden_state(llama2_lora, g.mprompt) # M_e^X [L x D] (Ch 3) w = softmax(router(Me)) # K weights summing to 1 (Ch 4) Qbar = sum(w[k] * expert[k](Q[g.mod], Me) for k in range(K)) # [N x D] Te = text_encoder[g.mod](g.tprompt) # the decoder's own text encoder S = Te + 0.2 * linear[g.mod](Qbar) # TM-Fusion (Ch 3) results.append(decoder[g.mod](S)) # SD, AudioLDM, Zeroscope, DINO, SAM return strip_tags(out), results def controller_loss(S, Te, Qbar, E): return (1 - cos(S, Te)) + (1 - cos(Qbar, E)) # M-Alignment + M-Reconstruction (Ch 5)
Spider belongs to the family of language models that act as a controller for specialist generators. NExT-GPT is the closest ancestor. Spider changes three things about it: a many-group answer template, a text-led control signal, and a shared projector. The generators stay separate and frozen. Models that generate every modality inside one network, rather than calling separate decoders, follow a different path.
Now press Present or Teach and explain Spider from memory: why one reply was hard, what the template looks like, how the words and the hint combine, what the projector and the two losses do, and how text-only data trains it. Then go back to the instrument at the top, send the Beijing guide to both models, and say which chapter explains each tile that fills in.