ModaVerse

Ask for a picture of the animal that made a sound. ModaVerse trains 40 million parameters in one stage and lets the language model write plain orders for ready-made image, video and audio makers.

Learn how a paper turns one text language model into a model that reads sound, pictures and video, and answers with all three, without training the output side at all.

Pick a way to build a multimodal model, and watch which boxes must be trained. Then press Swap the generator to see who must retrain. Then we build the paper, piece by piece: the parts, the three blocks, the order the model writes, the alignment, the data, the training, the tables and the failures.

You need a rough idea of what a language model does (it reads text and writes the next word). We build the rest from zero.

A reading for MIT Multimodal AI (Spring 2026, taught by Paul Liang), belonging to Lecture 8, Large Multimodal Models. Paper: Xinyu Wang, Bohan Zhuang and Qi Wu, “ModaVerse: Efficiently Transforming Modalities with LLMs” (arXiv 2401.06395, version 2, April 2024). Every idea is re-taught here in our own words and devices.

Pick a way to build

Data and GPU figures are the paper’s Table 1 (stage I data: Emu 82 million, NExT-GPT 13 million or more, ModaVerse 2 million; GPU time is the GPU count times the time, our product). Which boxes are trained follows the paper’s Figure 1 description of each paradigm, not every detail of each model. What a swap would cost the feature-level designs is our reading of “feature-level alignment”; the plug-and-play claim for ModaVerse is the paper’s.

Chapter 0

One Request, Four Ways to Build

See why answering a sound with a picture is hard for a text model, and why the paper picks a fourth road

You play a recording of a cat meowing. You type: “Show me a picture of the animal that made this sound.” A person does this without effort. Hear a sound, think “cat”, picture a cat.

Now ask a large language model (an LLM: a program that reads text and writes the next word, again and again) to do it. It cannot. It reads only text and writes only text. It has no ears and no pencil.

A modality is a kind of data: text, image, video or audio. A multimodal LLM (an MLLM) reads or writes more than one kind. The paper in this lesson, ModaVerse by Xinyu Wang, Bohan Zhuang and Qi Wu, builds one that takes any mix of text, image, video and audio in, and sends any of those out.

Three ways people built MLLMs before

The paper sorts earlier work into three paradigms (three styles of design). Each solves the cat problem in a different place.

Multimodal pre-training adds new parts to the language model, such as a vision encoder and a decoder, then trains the whole thing on a huge pile of paired data. Think of it as raising a bilingual child from birth. It works well, but it is costly. The paper names Emu, which pre-trains on a large corpus and uses hundreds of GPUs.

Adaptor training keeps the big parts as they are and trains only small adaptors: thin layers that translate between the language model and a ready-made encoder or decoder. Think of a travel adaptor for a plug. It is cheaper. But NExT-GPT, the paper’s example, still needs three training stages, one for each adaptor group.

LLM as an agent trains nothing. A chat model gets a long prompt that says: here are tools, pick one. HuggingGPT works this way. It is the cheapest, but inputs reach the model as text from other models, and the answers depend on how the prompt is written.

The fourth road: adaptor on the way in, agent on the way out

ModaVerse takes the cheap half of each. On the input side it trains small adaptors, so the model can really “hear” the cat. On the output side it trains no adaptor. The language model only writes an order in plain words, and a ready-made image maker carries it out.

The paper names the idea Adaptor+Agent. The order is called a meta-response: text that says which maker to call and what to tell it. Everything the paper does follows from that one choice. Press the buttons in the hero at the top of this page to see which boxes each design must train.

Here is a way to feel the trade. The paper judges a design on three yardsticks: how cheap it is to train, how consistent its output is, and how flexible it is about inputs and outputs. Choose which ones you need. The device shows which design passes, using only what the paper says in words.

Pick what you need

A tick means the paper says the design has it. A cross means the paper says it lacks it. A tilde means partly. A question mark means the paper does not say. Switch the needs on and off.

Our reading of the paper’s introduction and its Figure 1 text, not a measurement. Pre-training: “superior consistency and flexibility” but hundreds of GPUs. Adaptors: cheaper, “still require a complex training procedure”. Agent: training free, “limited flexibility”, “inconsistent results”. The paper does not rate consistency for adaptors or for ModaVerse, so those stay as question marks.

With cheap and flexible switched on, one design passes: ModaVerse. Switch on steady output too, and nobody passes on the paper’s stated words. That is honest: the paper claims ModaVerse “aims to combine” the strengths, and then measures it on six benchmarks (Chapter 7) instead of rating consistency.

Why the cost matters

A lab with four GPUs and a week cannot copy Emu. It can copy ModaVerse. That is the real point of the paper, and it is why the tables in Chapter 7 are read as a price list, not a leaderboard.

The word any-to-any names the goal: any mix of text, image, video and audio in, any mix out. Earlier any-to-any models paid for it with heavy training. The paper asks a plain question. Can we reach the same goal by training less, if we let language carry the output?

What this lesson will do

The paper is eight pages. We will teach it so that you could build it. You will learn the parts, the three blocks, the exact shape of the order the model writes, why the training data must teach orders and not captions, how that data was made, what is trained and what is not, what the tables show, and where it breaks.

  1. The parts you need: an encoder, a linear layer, LoRA and a text-to-x maker.
  2. Three blocks: input projection, meta-response, final response, and the paper’s Equation 1.
  3. The order the model writes: the meta-response and the parser that reads it.
  4. I/O alignment: why captions teach the wrong thing.
  5. Making the data: three kinds of instruction and a ChatGPT template.
  6. Training: one stage, about 40 million parameters.
  7. The tables: six benchmarks, read honestly.
  8. Where it breaks: editing, missing instructions and the limits.
  9. Field guide: numbers, checklist, code and credit.
What does ModaVerse train on the output side, to make a picture from the language model’s answer?

Chapter 1

The Parts You Need

Meet the four ready-made parts the paper snaps together, and see how few new numbers a linear layer adds

Picture a kitchen with four appliances already on the counter: a microphone that turns sound into numbers, a translator who speaks to a chef, a chef who only reads recipes, and a set of machines that cook from a written order. ModaVerse does not build any new appliance. It teaches the chef to read the translator and to write orders.

This chapter names the appliances. You need them all before the method makes sense. Each gets a short, plain explanation first and its real name second.

An embedding: a list of numbers that stands for a thing

A computer cannot look at a photo the way you do. It sees a grid of numbers. A vector is just a list of numbers, such as [0.2, −1.4, 0.7]. An embedding is a vector that a trained model makes to stand for a whole thing, such as a photo, a clip or a sentence. Similar things get similar vectors. The embedding of a cat photo lies near the embedding of another cat photo.

The length of the vector is its dimension. A language model works with vectors of one fixed dimension for every word. Call that size d. Every word it reads becomes a vector of length d.

ImageBind: one encoder for many kinds of data

An encoder is a model that turns raw data into an embedding. Most encoders handle one kind of data. ImageBind is one encoder for several: images, video and audio, among others. The paper calls it a “unified encoder”. Feed it a meow, and out comes one vector. Feed it a cat photo, and out comes another vector in the same shared space.

ModaVerse keeps ImageBind frozen. That means its numbers are never changed during training. Nobody has to teach it anything new. The paper uses it as it is.

A linear layer: a translator made of one grid of numbers

The vector from ImageBind lives in ImageBind’s space. The language model lives in a different space, with its own dimension d. The two do not match. Something must translate.

The paper uses the simplest translator there is: a linear layer. It multiplies the input vector x by a grid of numbers W (a matrix), then adds a vector b. In symbols, y = Wx + b. If x has length din and y must have length dout, then W has dout rows and din columns, and b has dout numbers.

The numbers in W and b are the parameters: the knobs that training turns. Count them: din × dout + dout. The paper trains three such layers, one per input kind (image, video, audio). Try different sizes in the device. The sizes are illustrative, because the paper does not print them.

Translate a vector

Left: the vector from the encoder. Middle: the grid of numbers that is the layer, drawn so its area is its parameter count. Right: the vector the language model receives. Choose the language model’s width.

Illustrative. We take the encoder’s vector as 1,024 numbers long and show three language model widths; the paper names Vicuna as its model but does not print either size. The paper’s own count is about 40 million trainable parameters in all, three linear layers plus the LoRA adaptor.

Look at the numbers. At d = 4,096 one layer holds 4,198,400 parameters. Three layers hold 12,595,200. That is only about a third of the paper’s 40 million. The language model itself has billions. So the translator is tiny next to the thing it serves.

How the vector enters the language model

A language model reads a list of word vectors. The translated vector has the right length, so it can sit in that list like one more word. People call this a soft token: a vector that behaves like a word but does not match any real word. The paper says only that the layers map the features into the “text space” of the model. It does not state how many soft tokens each input makes. We teach the idea, not a guess at that detail.

LoRA: a small patch on a big frozen model

The language model, Vicuna (a chat model built from LLaMA), was trained only on text. Reading a new kind of vector may need small changes inside it. Retraining all billions of numbers would be costly.

LoRA (low-rank adaptation) offers a cheaper path. Keep each big matrix W frozen. Learn a small correction made of two thin matrices, B and A, and use W + BA. If W is d × d and the thin side has size r, the patch has 2·d·r numbers, not d². Chapter 6 counts this.

Text-to-x makers: ready-made generators

The last appliance is the set of text-to-x models: programs that take a short text description and make an image, a video or a sound. The paper uses Stable Diffusion for images and AudioLDM for audio, plus a text-to-video model. All of them are frozen. All of them are replaceable.

These makers like a particular style of text: short, packed with attributes, such as “a photo of a cat, 4k, masterpiece”. A chatty paragraph gives worse results. Keep that in mind; Chapter 4 is built on it.

A linear layer maps a 1,024 number vector to a 4,096 number vector. How many parameters does it hold, counting the bias?

Chapter 2

Three Blocks and One Equation

Follow one request from a meow to a picture, and read the paper’s Equation 1 as three small functions

You hand a waiter a note: “the animal that makes this sound”, with a tape of a meow. The waiter listens, writes a short order for the kitchen, and the kitchen cooks. The waiter never cooks. The kitchen never listens. They meet in a written order.

ModaVerse works the same way. Its pipeline has three blocks (the paper’s word for functional parts): input projection (listening), meta-response generation (writing the order) and final response generation (cooking). Step through the real example from the paper, a meow and the instruction “Generate an image of an animal based on the provided vocalization.”

Step the pipeline

Each press moves the request one stage along. The row shows the six stages, the box below shows what the data looks like at that stage. Warm means trained, blue means frozen, pink means plain words.

The request, the meta-response and the prompt “A photo of a cat” come from the paper’s Figure 3 and Section 3.3. The vector sizes (1,024 and 4,096) are illustrative, as in Chapter 1; the paper does not print them.

Block 1: input projection

The audio clip goes to ImageBind, which returns one embedding. A trained linear layer for audio then maps it into the language model’s space. The text instruction is cut into ordinary word pieces. The language model now holds both: the instruction as words and the sound as a vector.

Why a linear layer, and why only that? Earlier work the paper cites found that a single linear layer is enough to align a vision encoder to a language model. A small trained part is a cheap part. And because the encoder’s embedding is passed on whole, details of the input survive.

Block 2: meta-response generation

A plain language model can only write text. So the paper does not ask it to write a picture. It asks it to write a meta-response: a short text that says which maker to call and what to tell it. For our cat, the meta-response says: use the text-to-image model, with the prompt “A photo of a cat”. Chapter 3 gives its exact shape.

This step is the only place the language model is changed, and only by a LoRA patch. It is where the real reasoning happens: hearing a meow, deciding it is a cat, deciding the user wants an image, and choosing words the image maker likes.

Block 3: final response generation

A small program, the parser, reads the meta-response and finds the maker name and its prompt. It then calls that maker. The maker, frozen and ready-made, returns the image. If the meta-response lists several makers, each runs in turn, and the results go back to the user together.

The paper stresses two gains of this layout. First, the language model and the makers are decoupled (not tied together by trained links), so you can change a maker and keep everything else. Second, no output-side projection layer exists, so nobody has to run the makers during training. Training touches only the input layers and the LoRA patch.

A worked count, with shapes

Take the cat request. In our illustration, the instruction is about 14 word pieces (tokens: the small chunks a model reads). The audio adds one projected vector. So the language model reads 15 positions, each a vector of width d. With d = 4,096 (illustrative), the input is a table of 15 rows by 4,096 columns.

The model then writes the order one token at a time. In our illustration, about 20 tokens come out, each chosen from a vocabulary of tens of thousands of pieces. That string is O2. Notice what never happens: the model never produces a 512 by 512 grid of pixels. The expensive part, drawing, is left to the maker, which was built for exactly that.

What if the request has two inputs, say an image and a clip? Each goes through its own encoder pass and its own linear layer, and each adds its own vector to the table. The paper’s input-side instructions train this case on purpose, with prompts such as “Describe the given image, audio, and video”.

The equation, read as three functions

The paper writes the whole pipeline as three lines. Start with a plain text model. Its input set and output set are both just text. The goal is a model whose input set I′ and output set O′ each hold text, image, video and audio. Here are the three steps.

I′
The user’s input: any mix of text, image, video and audio. In the example, a text instruction plus a meow.
ImageBind
The frozen encoder turns each non-text part into an embedding.
P
The trainable projection, three linear layers. It carries an embedding into the language model’s text space.
O1
The input as the language model sees it: vectors in its own space, with the instruction words.
LLM′
The language model with its LoRA patch applied. The prime says it is the adapted copy.
O2
The meta-response: plain text that names a maker and its prompt.
M
The frozen set of text-to-x makers, run from the parsed meta-response.
O′
The final answer: text, image, video or audio, possibly several at once.

Each line is one block of the figure. Only P and the LoRA patch inside LLM′ are trained. ImageBind and M stay exactly as downloaded.

Now count what goes in and out for the cat request. In: one audio file and one instruction string. After ImageBind: one vector. After P: one vector of the language model’s width. After LLM′: a string of a few dozen characters. After M: one image file. The sizes change at each step, and the only part that crosses from numbers back to words is LLM′. That is the design’s core: the interface between thinking and making is a string.

In Equation 1, which pieces are trained, and which are used as they are?

Chapter 3

The Order the Model Writes

Read the meta-response line by line, run the parser on it, and see what breaks when the order is not in the expected shape

A restaurant order slip has a fixed shape: table number, dish, notes. The kitchen does not read a poem about the guest’s evening. If the slip is scribbled or the dish is not on the menu, nothing gets cooked.

The meta-response is ModaVerse’s order slip. It is the text the language model writes in Block 2, and it has a shape the parser can read. The paper shows it twice. Once as a line of text: “{model: ‘text-to-image’, prompts: ‘a photo of a cat’}”. Once in its training-data example, as a list of pairs: [("text-to-image", "A photo of a cat")]. Both say the same thing: a list of calls, and each call is a maker name plus a prompt.

Why a list, and why plain text

One request may need more than one maker. The paper’s Figure 3 shows a request answered with four calls: a text reply “Sure! Here you are!”, then a text-to-image call with “A photo of a cat.”, a text-to-video call with “A cat is playing.” and a text-to-audio call with “A cat is meowing”. A list holds any number of calls in order.

Why plain text and not a vector? Three reasons follow from the paper. A string is readable by any program. A string can be checked by eye while you debug. And a string is the same no matter which maker will read it, so you can change the maker and keep the language model.

The parser: ordinary code, not a model

The parser is a small, deterministic program (the same input always gives the same output). It does three jobs. It finds each call in the text. It looks up the maker name in a table. It passes the prompt to that maker. Here is our sketch of it. The authors’ real code is on their GitHub page and may differ in detail; the idea is the same.

pythonimport re

MAKERS = {"text": lambda p: p,           # a plain text reply passes straight through
          "text-to-image": t2i, "text-to-video": t2v, "text-to-audio": t2a}

CALL = re.compile(r'\(\s*"([^"]*)"\s*,\s*"([^"]*)"\s*\)')

def run_meta_response(meta: str):
    calls = CALL.findall(meta)               # [(maker, prompt), ...]
    if not calls: return []                    # nothing to run
    out = []
    for maker, prompt in calls:
        if maker not in MAKERS: continue      # unknown maker: skip it
        out.append((maker, MAKERS[maker](prompt)))
    return out

Read it once. The table MAKERS is the whole “model zoo”. To swap Stable Diffusion for another image maker, change one entry. The language model is not touched. This is the plug-and-play claim of the paper, made concrete.

Now try it. Edit the order, or load an example, and watch the parser decide what runs.

Run the parser

Load an order, then change it. The four boxes light up for each maker the parser would call. A red box means a call the parser could not match to any maker.

The examples come from the paper’s Section 3.3 and Figure 3; the unknown maker and the broken quotes are ours. The parser here is the same regular expression as the code above, running in your browser. Chapter 8 returns to what happens when the real model writes something malformed.

Swapping a maker, step by step

Suppose a better text-to-image model appears next month. Here is the whole job. First, load the new model. Second, change one entry in the MAKERS table so that the name “text-to-image” points to it. Third, run a few test requests and check that the pictures look right. You did not retrain the language model, and you did not change a single trained number.

This works only if the new model likes the same kind of prompt. The paper bets that most open-source makers do, because they learned from the same public data. It states this bet, but it does not test a swap in its tables. Treat it as a design promise, and check it on your own examples.

Several calls at once

Look at the four-call order again. The calls run one after another, in the order written. Each maker gets only its own prompt. The text reply, the picture, the clip and the sound come back together as one answer. This is why the order is a list and not a single line, and why the same model can answer a question with words and a picture in one turn.

The failure, as symptom, measurement and fix

Symptom: the user asks for a picture and gets nothing, or gets only a sentence. Measurement: log the meta-response and count the calls the parser finds. If the count is zero, the model wrote something other than an order, as in the chatty example. If a call names an unknown maker, the model invented a tool. Try both in the device.

Fix: the paper’s fix is the training, the topic of Chapters 4 and 5. Every training example ends in a correctly shaped order, so the model learns the shape. Our own suggestion for a deployment, not the paper’s: validate each order, and if it fails, ask the model once more.

One more point. Look again at the four-call example. The first call is a plain text reply. A text answer is also just a call, to the maker called “text”. So one pipeline handles the case where the user wants a sentence, a picture, or all of them.

The user asks for a picture. The model replies with a friendly paragraph and no tool call. What does the parser return, and why?

Chapter 4

I/O Alignment at the Level of Language

Learn why teaching the model to caption would fail, and why the training target must be the order itself

You hire a translator to help you order food abroad. You train her by showing her pictures of dishes and asking her to describe each one. She gets very good at describing. Then you ask her to order the noodles. She describes the noodles, in lovely detail, to the waiter. No noodles arrive.

That is the trap the paper’s I/O Alignment avoids. To align two parts means to train them so the output of one is something the other can use. Most earlier work aligns features: hidden vectors. ModaVerse aligns language: the model’s written output must match what the makers want to read.

The two alignments inside Equation 1

Look back at Equation 1. Two steps need alignment. The first is input: from the encoder’s embedding into the language model’s space (the paper writes I′ → O1). The second is output: from what the model understands into the order it writes (O1 → O2). The paper trains both at once, with one kind of data. We will see how.

Two problems stand in the way.

Why can we not simply write down O1? Because O1 is a hidden state: a vector inside the model that no person can label. We can only judge it by what the model writes next. So the paper trains the two alignments together, and lets the quality of the written order tell the input layers whether they did their job.

Problem 1: a caption is the wrong target

There is no ready-made target for O1. Nobody can write down “the exact vector the language model should see for this meow”. So earlier adaptor methods take a shortcut. They show the model an input and a caption, and train it to write the caption. The paper calls this framing the training as a multimodal captioning task. The caption plays the role of O2.

That works if all you want is a description. ModaVerse wants more. Given the instruction “Generate an image of an animal based on the provided vocalization” and a meow, the right output is not “a cat is meowing”. It is an order: use the text-to-image model, with the prompt “a photo of a cat”. A model trained on captions never sees an order, so it never learns to write one.

So the training target must be the meta-response. The paper says the training should teach the model to say which model to use and how to use it.

Problem 2: the model’s language is not the maker’s language

The second problem is a matter of style. Ask a language model to describe a landscape and it writes a flowing, literary paragraph. Ask Stable Diffusion for a landscape and it prefers a short, concrete prompt with attributes such as “4k” and “masterpiece”. The two were trained on different text. If the order contains the paragraph, the maker gets a prompt in a style it handles poorly.

So alignment must also teach the model to write in the makers’ style. The paper calls this language-level alignment. It is reached by showing the model, as training targets, prompts that look like the makers’ own training captions.

Use the device below to see how each choice of training target changes the order the model writes. The outputs are illustrative, but the structure is the paper’s.

Choose the training target

Pick what the model was trained to write, and what the user asks for. Then switch on the maker’s style. The three tiles check the order from the parser’s point of view.

Illustrative outputs written by us to show each target’s effect; the meow request and the prompt “A photo of a cat” are the paper’s. The paper states that makers prefer concise prompts with attributes such as “4k” and “masterpiece” and that LLMs tend to write literary paragraphs; it does not measure the gap in a table.

The instruction-following recipe

The fix for both problems is one recipe: instruction-following tuning. You give the model an instruction plus inputs, and train it to write the correct output for that instruction. It is the same recipe that turns a plain text model into a chat model. Here the correct output is an order.

The paper lists three reasons this works well. First, the model must truly understand the multimodal input to write the right order, so the input layers learn alignment as a side effect. Second, the datasets used to train image, audio and video makers already pair each item with a text description, which gives aligned samples for free. Third, most open-source makers train on the same public datasets, so a description style that suits one tends to suit the others. Swapping makers then needs no retraining.

The last point repays a second look. The model is trained to write prompts that match the datasets, not one particular maker. That is why a new maker trained on similar data can slot in. It is a bet, not a guarantee. The paper claims the switch is seamless but does not run a swap experiment in its tables.

Why does the paper say that training the projection layers on a captioning task would not be enough?

Chapter 5

Making the Instruction Data

Build the training pairs the way the paper does: three kinds of instruction, and a ChatGPT query with three ingredients

Imagine you must teach a new waiter to write order slips. You need hundreds of thousands of examples: a guest says something, and the correct slip follows. Nobody has such a pile. So you write a few by hand, then ask a skilled assistant to write many more in the same style, using real dishes from your menu.

That is the paper’s plan for data. It needs instruction-invocation pairs. The instruction is the input: a sentence plus any images, clips or audio. The invocation is the output the model must learn to write: the list of makers and prompts. The paper shows one pair:

json{"instruction": ["Generate an image of an animal based on the provided vocalization.", "cat_meowing.wav"],
 "invocation":  [("text-to-image", "A photo of a cat")]}

The instruction holds a sentence and an audio file name. The invocation holds one call. That is the whole training signal. The model reads the first, and learns to write the second.

Three kinds of instruction

The paper sorts its instructions into three kinds. Each trains a different skill, and each uses its own source of data.

Input-side alignment instructions teach the model to read any mix of inputs: text plus image, image plus video, or image plus audio plus video. A sample is “Describe the given image, audio and video”. The model must describe the image first, then the audio, then the video, in order. This is how the projection layers learn to carry meaning.

Output-side alignment instructions teach the model to write the order. A sample is “Generate an image based on the provided audio of an animal sound”, answered with a text-to-image call. These are the data that make ModaVerse different, and the paper calls them “crucial for the success of ModaVerse”.

Reasoning boosting instructions keep the model smart. A sample is “Where might this audio clip have been recorded?”. The model must infer from the input, not just describe. Without such samples, a model trained only to route might lose its reasoning.

Where each kind comes from

Existing instruction-tuning sets already hold the first and third kinds. The paper reuses components of LLaVA, VideoChat and InstructBLIP. But no set holds the output-side kind, because that kind is new. So the authors build it with the ChatGPT API, using a template. Each query has three ingredients.

Seed examples are a few hand-written, correct pairs, picked at random from a manually crafted collection. They show ChatGPT the format and the task. Candidate descriptions are text descriptions picked at random from real paired datasets (descriptions of images, audio and video). They stand in for true inputs, so the generated instructions refer to things that really exist. Language references are text descriptions picked from the training sets of the makers. They show ChatGPT the style the makers like, so the invocation prompts come out in the right language.

Here is a worked case. Suppose the candidate description is “a cat is meowing” for an audio clip. A seed example shows how a pair looks. A language reference reads “A photo of a cat on a sofa”. ChatGPT might return: instruction “Generate an image of an animal based on the provided vocalization” with the clip, and invocation ("text-to-image", "A photo of a cat"). The three ingredients have each played a part: the format came from the seed, the content from the candidate, the style from the reference.

Build the ChatGPT query

Each ingredient guards one thing. Switch one off and watch the matching tile fail, and the generated pair change.

The three ingredients and their roles are the paper’s Section 3.3. The sample texts, and what goes wrong when an ingredient is missing, are our illustration: the paper does not run this ablation and does not give the number of generated pairs.

Here is how an input-side pair looks with several inputs. The instruction is “Describe the given image, audio, and video”, and three files come with it. The model must write three descriptions in the same order as the files. If it describes the video first, the sample is wrong, even if each description is good. That order test is what teaches the model to keep the inputs apart.

A pair, end to end, with its shape

To train, each pair becomes one sequence. The input side is the instruction tokens with the projected audio vector in place. The target side is the invocation written as text, one token after another. The loss is counted only on the target side, which Chapter 6 explains. No maker is ever run to make this data. That is why the output side is cheap to train.

All pairs together, across the three kinds, give about 2 million training samples (Table 1, stage I). The paper does not state how the 2 million split among the three kinds.

The failure, as symptom, measurement and fix

Symptom: the trained model writes orders in the wrong shape, or for inputs that do not exist. Measurement: take a sample of generated pairs and count how many parse, how many name a real maker, and how many prompts are long paragraphs. Fix (ours, not the paper’s): add seed examples for the shape that fails, and filter bad pairs before training.

Which ingredient of the ChatGPT query teaches it the style of prompts that the makers handle well?

Chapter 6

One Stage, About 40 Million Parameters

Count what is trained and what is frozen, write the loss for one pair, and compare the cost with four other recipes

A piano tuner does not rebuild the piano. She turns a few pegs until the strings sound right. ModaVerse treats a language model the same way. The model has billions of numbers. Training turns only about 40 million of them, less than one in a hundred.

What is trained, and what is not

The paper gives a short list. The trainable parts are three linear layers (one each for image, video and audio input) and the LoRA adaptor of the language model. Together they hold about 40 million parameters. Everything else is frozen: ImageBind, the base weights of Vicuna, and all the makers. The makers are not even run during training, because no loss ever depends on their output.

The phrase “frozen” is easy to misread. A frozen part does not change, but it still passes signals. During training, the error signal flows backward through the frozen language model all the way to the linear layers. The model’s weights do not update, but they carry the blame back to the parts that do.

The loss: score the order, token by token

The paper does not print its training loss. For instruction tuning, the standard choice is cross-entropy on the next token, counted only on the answer. We write it here as our reading, not as the paper’s equation. The answer is the invocation text, a string of tokens y1 to yT.

L(θ)
The loss: how surprised the model is by the correct order. Lower is better.
∑t=1T
Add up over every token of the invocation, first to last. Tokens of the instruction are not counted.
pθ
The model’s probability. θ stands for the trainable numbers: the three layers and the LoRA patch.
yt
The correct token at position t of the order, for example the piece “text” of “text-to-image”.
y<t, x
What the model has already: the earlier tokens of the order, and the instruction with its projected vectors.

A worked toy. Suppose the model gives the four tokens of a short order the probabilities 0.9, 0.8, 0.95 and 0.7. The natural logarithms are −0.105, −0.223, −0.051 and −0.357. Add them and flip the sign: the loss is 0.736, or 0.184 per token. If the model were sure of every token, the loss would be 0. The probabilities here are illustrative.

Now the shapes. For a batch of B pairs with T tokens each, the language model holds a tensor of B × T × d numbers inside. At the end it outputs B × T × V scores, where V is the size of the vocabulary (32,000 for the LLaMA family that Vicuna descends from). The loss reads only the positions that belong to the invocation. Backward, the gradient reaches the LoRA matrices and then the linear layers. No other weight receives an update, so the optimizer keeps its memory for only about 40 million numbers.

Why does the count matter for memory? A common optimizer called Adam keeps two extra numbers for every number it trains. Training all 7 billion numbers in 32-bit form would need about 28 GB for the weights and about 56 GB more for Adam’s state. Training only 40 million needs about 160 MB for them and about 320 MB for the state. The frozen base still takes its 28 GB to hold, but it adds no optimizer state. These sums are illustrative, and the paper does not report memory.

There is a second reason to patch and not retrain. A model that has learned from trillions of words of text can forget some of it if you retrain every number on a small new set. A small LoRA patch moves the model less, so it keeps more of its text skills. The paper adds reasoning-boosting instructions for the same goal: to keep the model thinking well.

Why one stage is enough

Look at the alternative. NExT-GPT trains in three stages because it has three links to align: an input link, an output link and the language model’s adaptor. Each stage needs its own data and its own run. ModaVerse has no output link at all. The only two trained parts, the layers and the patch, learn together from the same pairs. So the paper trains once.

Use the device to see the cost against four other recipes, using the paper’s Table 1. Pick what to compare.

Compare the cost

Choose what to compare. In the parameters view, drag the LoRA rank to see how a patch adds to the three layers.

26

GPU hours are the GPU count times the time shown in Table 1 for stage I (Emu 128 GPUs for 2 days, BLIP-2 16 for 6 days, LLaVA 8 for 20 hours, ModaVerse 4 for 20 hours); NExT-GPT gives no GPU time. Data sums the stage columns of Table 1; NExT-GPT’s figures are lower bounds (13 million or more, twice, plus 20 thousand or more). The parameters view is illustrative: three layers of 1,024 by 4,096, and LoRA on 4 attention matrices in each of 32 layers of width 4,096; the paper gives only the total of about 40 million.

In GPU hours, stage I of Emu is about 6,144 against ModaVerse’s 80: roughly 77 times as much. In data, ModaVerse’s 2 million is about 1.9% of Emu’s three-stage total, which fits the paper’s “less than 2%”. These are real gaps. But they compare a design with a training recipe, not equal-quality models. Chapter 7 checks quality.

The failure, as symptom, measurement and fix

Symptom: the model reads audio badly yet writes clean orders for text-only requests. Measurement: compare the loss on text-only pairs with the loss on audio pairs. A gap says the projection layer has not aligned yet. Fix (ours): add more input-side pairs for that modality before adding anything else.

Why does ModaVerse need only one training stage, where NExT-GPT needs three?

Chapter 7

What the Six Tables Show

Learn each metric in plain words, then read every benchmark for what it proves and what it does not

A school sends one student to six contests: drawing, painting, a spelling bee, music, a film quiz and a reading test. She finishes second or third in most, and first in none. Is that a good result? It depends on who else entered, and on how much she practised. A student who trained one week and almost matches the ones who trained a year has a good story.

That is how to read ModaVerse’s results. The paper tests its model on six benchmarks, covering both directions for three kinds of data: text to image, image to text, text to audio, audio to text, text to video and video to text. It follows the evaluation setup of an earlier any-to-any model, CoDi.

The metrics, in plain words

For generation (text to something), the paper uses these scores. FID (Fréchet inception distance) measures how far the generated pictures sit from real pictures, in the feature space of a vision network. Lower is better. FD is the same idea for audio. IS (inception score) rewards samples that are clear and varied. Higher is better. CLIPSIM asks the CLIP model how well a generated picture matches the prompt. Higher is better.

For understanding (something to text), the model writes a caption and the score compares it with human captions. B@4 (BLEU-4) counts how many runs of four words match. METEOR also credits synonyms. CIDEr weights words that humans agree on and ignores common filler. SPIDEr averages two caption scores, SPICE and CIDEr. All of these are higher is better.

One warning. These numbers compare to references, not to a person’s judgement. A caption can be right and score low if it uses different words.

Read a table

Choose one of the six benchmarks, then a metric. Bars run best first. ModaVerse is warm, the best is green. A dash in the paper means no number was given.

Every number is copied from the paper’s Tables 2 to 7 (COCO caption, AudioCaps, MSR-VTT). Ranks are computed live over the methods that have a number. Bar length is the raw score, drawn from zero.

The three datasets are worth a line each. COCO caption holds photos, each with several human captions. AudioCaps holds ten-second sound clips from the wild, each with a written description. MSR-VTT holds short web videos, each with several captions. Each set has human references to score against, which is why the same three tests can run in both directions.

Reading each table

Text to image (Table 2). ModaVerse scores FID 11.24. Stable Diffusion scores 11.21, CoDi 11.26 and NExT-GPT 11.28. That puts ModaVerse second of six, and the gap to first is 0.03. A note: the paper’s prose says ModaVerse “achieved an 11.28 FID”, but the table row says 11.24; 11.28 is NExT-GPT’s row. We use the table. The close match to Stable Diffusion is not a surprise: Stable Diffusion draws the picture, so the image quality is largely the maker’s. This is our reading.

Image to text (Table 3). Here the language model itself does the work. ModaVerse scores B@4 43.9, METEOR 31.8 and CIDEr 151.4, third on all three. It beats CoDi on all three and trails NExT-GPT (44.3, 32.9, 156.7) and OFA (44.9, 32.5, 154.9).

Text to audio (Table 4). FD 23.40 and IS 8.22. On FD it beats NExT-GPT (23.58) and is beaten by CoDi (22.90). It is also behind AudioLDM-L (23.31), which the paper’s prose does not mention. On IS, NExT-GPT (8.35) is ahead. So “outperforms NExT-GPT” holds on one of the two metrics.

Audio to text (Table 5). SPIDEr 0.494 and CIDEr 0.792, second on both, behind NExT-GPT (0.521, 0.802) and ahead of CoDi (0.480, 0.789).

Text to video (Table 6). FID 13.35 and CLIPSIM 0.3014, third on both. NExT-GPT (13.04, 0.3085) and MakeVideo (13.17, 0.3049) are ahead. CoDi has no FID in the table.

Video to text (Table 7). B@4 56.5 (third) and METEOR 35.2 (second), behind NExT-GPT (58.4, 38.5) and, on B@4, mPLUG-2 (57.8).

A fair way to use these tables is to ask three questions. Which route do I need? How close is ModaVerse to the best number on that route? And is that gap worth the savings in training? For a team that needs all six routes on a small budget, the answers are favourable. For a team that needs the best image quality alone, a specialist maker is the better choice.

What the pattern means

Across the twelve metrics, ModaVerse is first on none, and it beats NExT-GPT on two (text-to-image FID and text-to-audio FD). The authors say so openly: it “does not outperform all state-of-the-art methods”. Their argument is efficiency. One model covers every route, where some winners (Stable Diffusion, OFA) cover one. And it trains in one stage on far less data (Chapter 6).

Two cautions keep the reading honest. First, the tables do not measure the claims the paper is most proud of: plug-and-play swaps and consistency. Second, the tables give no standard deviations, so gaps of a few hundredths in FID may be noise. We do not know, because the paper does not say.

Which statement about the paper’s results is supported by the tables?

Chapter 8

Where It Works and Where It Breaks

Route all twelve request types of the paper’s Figure 4, then trigger the two failures of Figure 5 and see why they follow from the design

A courier company has one clerk who reads every parcel label and sends it to a depot. The clerk is quick and the depots are good. But the clerk never opens the parcel, and a parcel with no label goes nowhere useful. That is ModaVerse’s shape, with its strengths and its limits together.

Twelve routes, one model

The paper’s Figure 4 shows twelve example requests, each marked by what goes in and what comes out. The inputs are audio, image or video, always with a text instruction. The outputs are text, image, video or audio. Examples include audio plus text to image (a meow becomes a cat picture), image plus text to audio (a picture of waves becomes the sound of waves), and video plus text to video (a clip becomes a related clip). Also shown: question answering from each input kind, and style transfer.

Every route uses the same three blocks. Only two things change: which input layer fires (audio, image or video), and which maker is called (or none, if the answer is text). The device lets you set the route and see the path.

Route a request, then break it

Choose what goes in and what comes out. Then switch off the text instruction, or ask for an edit of the original picture, to trigger the paper’s two failure cases.

The twelve routes are those shown in the paper’s Figure 4; the two failure cases are its Figure 5, (a) editing and (b) no language clue. What happens on routes the figure does not show (for example image in, text out with no instruction) we infer from the design and do not claim as measured.

Try three routes in the device and read the sentence it gives. Audio in and image out uses the audio layer and the image maker. Video in and text out uses the video layer and no maker at all. Image in and image out works for “draw something in this style”, and breaks only when you ask to change the original.

Failure one: it cannot edit, only redraw

Symptom: you give a picture of a red apple and say “Edit the red apple in this image to green.” You get a green apple, but the table, the background and the layout are different. The paper’s Figure 5(a) shows exactly this. The model “generates an entirely new one” rather than changing the original pixels.

Cause: the order is text. The maker receives only the prompt, such as “a green apple”. It never sees the original picture, so it cannot keep its layout. The text interface that makes the design cheap and plug-and-play also throws away pixel detail on the way out.

Fix: the authors suggest adding an editing model to the zoo of makers, and leave it for future work. That would need the meta-response to carry the input image along. It is a change to the interface, not just a new table entry.

Failure two: no words, no plan

Symptom: you give only a sound of a dog barking and no instruction. The model writes something unrelated. Figure 5(b) shows this. Cause: the model was trained by instruction-following, so it uses the instruction to decide what to do. With no instruction it has no task, and it guesses. Fix: always send an instruction, even a plain one. A deployment could add a default instruction such as “Describe this audio”. That is our suggestion, not the paper’s.

Both failures have one root. The language model and the makers share only words. Words are cheap and swappable, but they carry less than pixels or sound. Any request that needs exact pixels, exact timing or exact pitch from the input will hit this limit. Requests that need only the idea of the input, such as “draw the animal that made this sound”, fit the design well.

Limits that follow from the design

Some limits are the paper’s own words. Others are our reading, and we mark them.

The ceiling is the maker’s. ModaVerse’s image FID of 11.24 nearly equals Stable Diffusion’s 11.21 (Chapter 7). A better maker can lift the score. A weaker one lowers it. The language model can only help by writing better prompts. (Our reading.)

Two stages at answer time. A request runs the language model first, then a maker. The paper does not report latency, so we cannot give a number. Anyone who deploys it should measure it.

The format must hold. If the order is not in the shape the parser expects, nothing runs (Chapter 3). The paper does not report how often that happens.

The data is machine-made. The output-side data comes from ChatGPT (Chapter 5). Its mistakes and habits can reach the trained model. The paper reports no audit of the data. (Our reading.)

The evidence is narrow. There are six benchmarks, one set of numbers each, no standard deviations, no human study, and no experiment that swaps a maker or removes a part of the training data. The paper claims plug-and-play and consistency without testing them in a table.

Why does ModaVerse redraw an image instead of editing the original, according to the design in the paper?

Chapter 9

Field Guide: ModaVerse on One Page

Keep the numbers, the terms, a build checklist and the core code, and see where the paper sits in the field

ModaVerse turns a text language model into a multimodal one with a small trick. On the input side, it trains three linear layers and a LoRA patch, about 40 million parameters in one stage on 2 million samples. On the output side, it trains nothing: the model writes a plain text order, the meta-response, and a parser calls ready-made makers. I/O alignment at the level of language is the idea that holds it together. The price is that the makers cannot see the original pixels, so editing fails.

The numbers

ItemValueWhere in the paper
Trainable parametersabout 40 million (3 linear layers + LoRA)Section 3.3
Language model, encoderVicuna (LoRA), ImageBind (frozen)Sections 3.1, 3.3
Makers, all frozenStable Diffusion (image), AudioLDM (audio), a text-to-video modelSection 3.1
ModaVerse training1 stage, 2 M samples, 4 GPUs for 20 hTable 1
Emu training3 stages: 82 M, 24 M, 1.28 M samples; 128 GPUs for 2 days in stage ITable 1
NExT-GPT training3 stages: 13 M+, 13 M+, 20 k+ samples; no GPU time givenTable 1
Text to image (COCO)FID 11.24 (SD 11.21, NExT-GPT 11.28)Table 2
Image to text (COCO)B@4 43.9, METEOR 31.8, CIDEr 151.4Table 3
Text to audio (AudioCaps)FD 23.40, IS 8.22Table 4
Audio to text (AudioCaps)SPIDEr 0.494, CIDEr 0.792Table 5
Text to video (MSR-VTT)FID 13.35, CLIPSIM 0.3014Table 6
Video to text (MSR-VTT)B@4 56.5, METEOR 35.2Table 7
Failure casescannot edit original pixels; irrelevant output without an instructionFigure 5

The terms

TermIn plain wordsChapter
Adaptor+AgentTrain small adaptors on the way in; let the model act as an agent on the way out0
Meta-responseThe text order the model writes: a list of (maker, prompt) calls3
I/O alignmentTrain the model so its written output is what the makers read4
Instruction-invocation pairOne training sample: instruction with inputs in, order out5
Language referencesMakers’ own captions, shown to ChatGPT so prompts match their style5
FrozenUsed as it is: the numbers never change, signals still pass through6

A checklist for building your own

Tick what you have. It is a reading aid, not a test.

Check your build

The core as code

Here is the whole idea in one sketch. It is ours, written from the paper’s description. The authors’ code is on their GitHub page and differs in detail. Sizes are illustrative, and the paper does not state where the projected vectors sit among the word vectors, so we put them last.

pythonimport torch, re
from torch import nn

class ModaVerseSketch(nn.Module):
    def __init__(self, imagebind, llm, makers, d_enc=1024, d_llm=4096):
        super().__init__()
        self.enc = imagebind.requires_grad_(False)           # frozen encoder
        self.proj = nn.ModuleDict({k: nn.Linear(d_enc, d_llm)  # trained: 3 layers
                                   for k in ("image", "video", "audio")})
        self.llm = llm                                         # Vicuna, LoRA matrices trainable only
        self.makers = makers                                   # frozen: {"text-to-image": fn, ...}

    def embed(self, text_ids, media):
        parts = [self.llm.embed_tokens(text_ids)]                 # (T, d_llm)
        for kind, x in media:
            with torch.no_grad(): e = self.enc(kind, x)         # (1, d_enc)
            parts.append(self.proj[kind](e))                    # (1, d_llm)
        return torch.cat(parts, 0)

    def loss(self, text_ids, media, order_ids):         # one instruction-invocation pair
        x = torch.cat([self.embed(text_ids, media), self.llm.embed_tokens(order_ids)])
        logits = self.llm.forward_embeds(x)                     # (T_in + T_out, V)
        n = len(order_ids)
        return nn.functional.cross_entropy(logits[-n-1:-1], order_ids)  # only the order is scored

    @torch.no_grad()
    def answer(self, text_ids, media):
        order = self.llm.generate_from_embeds(self.embed(text_ids, media))  # a string
        calls = re.findall(r'\(\s*"([^"]*)"\s*,\s*"([^"]*)"\s*\)', order)
        return [(m, self.makers[m](p)) for m, p in calls if m in self.makers]

Where it sits in the field

On one side stand the heavy designs, such as Emu, that pre-train new parts together, and the adaptor designs, such as BLIP-2 and NExT-GPT, that align features on both sides. On the other side stand the pure agents, such as HuggingGPT, that train nothing. ModaVerse takes a middle road, and its main bet is that language is a good enough interface between a model and its makers. The lecture this paper belongs to, Large Multimodal Models, gives the wider map.

Keep going

The takeaway. Let the language model write orders in plain words, and let ready-made makers do the making. You train about 40 million numbers on the way in, and nothing on the way out.

Credit. ModaVerse is the work of Xinyu Wang (University of Adelaide), Bohan Zhuang (Monash University) and Qi Wu (University of Adelaide). Read the original: arXiv 2401.06395. Code: github.com/xinke-wang/ModaVerse. All explanations, devices and the code sketch here are our own re-teaching.

Which sentence best states what ModaVerse contributes?

Now press Present or Teach and explain the paper out loud, from memory: why captions are the wrong target, what the meta-response looks like, what is trained and what is frozen, and why editing fails. Then go back to the top, press Swap the generator on each design, and say who must retrain, and why ModaVerse does not.

ModaVerse: Efficiently Transforming Modalities with LLMs
Read the paper · Authors’ code · The course site · Lecture 8 · Back to Gleams