Cobra swaps the transformer inside a vision-language model for Mamba, which keeps a fixed-size memory. It writes 256 tokens in 1.54 s where MobileVLM v2 needs 5.17 s.
Learn how a model that reads in a straight line instead of comparing every pair of tokens can look at a picture, answer a question about it, and finish before its rivals.
Pick a rival, then press Race. Slide the answer length and race again. Then we build, piece by piece, what makes the lane on the right so short: the memory that does not grow, the way a picture becomes tokens, and the recipe that trains it.
You need a feel for what a neural network layer does (numbers in, numbers out) and a little matrix arithmetic. We build the rest from zero.
A reading for MIT MMAI: Multimodal AI (Spring 2026, Paul Liang), paired with Lecture 8, Large Multimodal Models. Paper: Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang and Donglin Wang (Zhejiang University, Westlake University, DAMO Academy), “Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference”, AAAI 2025. arXiv 2403.14520. Every idea is re-taught here in our own words and devices.
Speeds are the paper’s Table 3 (one image, the prompt “Describe the image specifically”, 256 output tokens, one A100 PCIe 80GB GPU, image encoding included in the total). Other answer lengths are illustrative: time is tokens divided by the measured rate. The race plays slower than real time so you can watch it.
Chapter 0
See why a vision-language model that compares every token with every other token falls behind on a small, fast machine
Picture a small robot with a camera on its head. It runs on one modest chip, not a data centre. A person asks, “Is the bicycle parked to the right of the dog?” The robot must look at the picture, understand the question, and answer in a moment, because the next question is already coming.
The program behind this robot is a multimodal large language model (MLLM: a language model that can also take in pictures). You met the idea in Lecture 8. A picture goes in, a question goes in, and a written answer comes out, one word at a time.
The paper we read here, Cobra, starts from a plain complaint. Nearly every MLLM is built on a transformer, and a transformer is slow in a specific way. The authors write that it has “quadratic computational complexity”. This chapter shows what that means, with numbers you can move.
A token is the small piece a model reads: a word, part of a word, or, for a picture, one patch of the image turned into numbers. Chapter 1 shows how a picture becomes tokens. For now, a token is one item in a list.
A transformer writes its answer one token at a time. To choose the next token, it runs attention: the new token looks back at every token that came before it and scores how much each one matters. If 743 tokens came before, that is 743 scores for one new token, in every layer of the model.
The transformer also keeps a record of every earlier token so it need not recompute them. That record is the KV cache (K for keys, V for values, the two numbers kept per earlier token). It grows by one entry per token. So a long context costs more time for each new token and more memory to store.
Now add up a whole answer. The first answer token compares with all the earlier tokens. The second compares with those plus the first answer token. The work for token number n is about n. Summed over the answer, the work grows like the square of the length. That is the “quadratic” in the complaint.
A picture is expensive here, because the picture alone is long. Cobra turns one image into 729 tokens. The question adds a few more. So the model has read more than 700 tokens before it writes its first word. Drag the slider and watch the pink line, the cost of one new token for a transformer, against the warm line, the cost for the model this paper builds.
Pink: a transformer compares the new token with every earlier token. Warm: a state-space model updates one fixed memory. Drag the number of tokens already read and compare the two lines.
Illustrative units: one comparison with an earlier token and one memory update each count as 1. A real memory update costs more than one comparison, so read the shape of the two lines, not their exact heights. The 729 image tokens and the 256 answer tokens are the paper’s setup; the rest is arithmetic you can check by hand.
The pink line climbs without limit. The warm line stays flat. At 750 tokens the transformer does 750 units of work for one new token. At 4,000 tokens it does 4,000. The state-space model does 1 unit of work at both.
Take the robot’s question. The picture gives 729 tokens. Say the question gives 14 more, so the model has read 743 tokens. (The 14 is an illustration; the 729 is the paper’s.) Now the model writes a 256-token answer.
The Mamba-style model does one memory update per answer token, so its total is 256. The gap between 222,848 and 256 is large, but it is not the whole story. A memory update does more arithmetic than one comparison. The honest message is the shape: the transformer’s cost per token rises as the answer grows, and the state-space model’s cost per token stays where it began.
There is a second bill, memory. The transformer stores 743 cache entries, then 744, and so on, in every layer. On a small chip that memory is scarce. The state-space model stores one fixed block, however long the context. The paper states this advantage directly: inference speed scales linearly with sequence length and memory use stays constant.
The authors measured it. In their Figure 1, the transformer-based MoE-LLaVA generates only 20.33 tokens per second. At that rate, a 256-token answer takes 12.59 seconds. A robot that waits 12 seconds to say what it sees has missed the moment.
They name the goals this blocks: real-time interaction, edge deployment, and, in their closing words, “visual-based robotic feedback control”. All of these need many answers per second from a model that runs on a small budget.
Researchers had already noticed the problem. Their usual fix is to shrink the language model. LLaVA-Phi builds on a small model called Phi-2. MobileVLM builds on a small model of its own. A smaller model has fewer numbers to multiply, so each token costs less.
The paper points out the price. These models are faster but worse. In the authors’ Figure 1, MobileVLM v2-3B has the lowest score on every benchmark they compare. Smaller means faster, and it also means less capable.
Cobra tries a different move. It keeps the size and changes the design. It replaces the transformer with Mamba, a state-space model (SSM: a model that reads tokens in a straight line and keeps a small, fixed-size memory of what it has read). Chapters 2 and 3 build it from zero.
As a trade, the claim reads like this. Cobra writes answers about 3.4 times faster than MobileVLM v2, the fastest small transformer-based model in the paper's speed table, and 4.1 times faster than LLaVA-Phi, because each new token costs one memory update instead of a look back at everything. It does this without making the model smaller, and the paper’s results say it holds its accuracy too. We will check each part of that claim in Chapters 7 and 8.
Chapter 1
Turn a picture into 729 tokens and park them in front of the question, so a model that only reads tokens can look
Imagine you phone a friend who is blind and who only understands spoken words. You want them to judge a photo. You cannot send the photo. So you describe it, square by square, in a fixed order: “top left, green grass; next, grass; next, a brown shape.” Your friend builds a picture in their head from the list.
A language model is that friend. It reads one kind of thing only, a list of tokens. To let it look at a photo, someone must turn the photo into a list. That job is the first half of every model like Cobra, and it is the same job in LLaVA, MobileVLM and the rest. Only the reader at the end of the list changes.
Inside a language model, every token is an embedding: a list of D numbers that stand for the token. Think of an embedding as an address on a map of meaning. Words with close meanings sit close together. D is the model’s width. For the Mamba-2.8B model Cobra uses, D is 2,560 (a property of that checkpoint, not stated in the paper).
So a 14-word question is a table with 14 rows and D columns. The model reads the rows in order. Everything it knows about the question lives in that table.
To feed a picture in, we also need rows of D numbers. We get them in two steps. First, cut the image into a grid of small squares called patches. Second, run each patch through a vision encoder: a network, already trained on huge numbers of pictures, that turns a patch into a list of numbers describing what is in it.
The paper gives the count in one line. An image of C channels (3 for red, green and blue), height H and width W is cut into Nv patches of P × P pixels:
The paper lists 729 image tokens at 384 × 384 (its Table 1). That is a 27 × 27 grid, since 27 × 27 = 729. The paper does not print the patch width. The SigLIP encoder it uses cuts 14 pixel patches, and 384 / 14 is 27.4, so the grid keeps 27 whole patches along each side. Try other grids below and watch the count.
The grid is the picture cut into squares. The bar under it is the whole list the model reads: purple image tokens, then blue question tokens. Drag the grid size.
The scene is a toy drawing. The 729 tokens (27 × 27) and the 196 tokens (14 × 14, the count after the paper’s LDPv2 projector) come from the paper’s Tables 1 and 3. The 14-token question is an illustration.
A finer grid gives the model more detail and also a longer list. That trade returns in Chapters 5 and 7.
The vision encoder and the language model were trained apart. Their numbers do not line up. The encoder’s feature for a patch is a list of Dv numbers, and Dv is not D. So we add a small learnable bridge, the projector. It maps each patch feature to a list of D numbers, so that it looks like a word embedding to the language model.
Now both kinds of token are rows of D numbers. We write the picture rows first, then the question rows, and hand the whole table to the language model. The paper says it directly: image embeddings are “directly concatenated to the beginning of the embedding sequence”. The model never learns which rows came from pixels. It only sees rows.
At the start, the language model has never met picture rows. It must learn to use them. This is done with visual instruction tuning: train on many examples of an image, a question about it, and a good answer. After enough examples, the rows from the projector start to carry meaning the model can use. Chapter 6 gives Cobra’s exact recipe.
Here is a symptom the paper shows. The question is, “Is the bicycle parked to the right of the dog?” The transformer-based LLaVA v1.5-7B answers, “Yes, the bicycle is parked on the right side of the dog.” Cobra-3.5B answers that the bicycle is on the left side of the dog (their Table 6).
The measurement is a benchmark called VSR, which asks true-or-false questions about spatial relations. On VSR, Cobra-3.5B scores 58.4 and LLaVA v1.5-7B scores 51.5. A model can write smooth sentences and still get left and right wrong. The fix the paper explores is better visual features, and that is Chapter 5.
Chapter 2
Build the state-space model from a leaking tank, then see why one idea gives two ways to compute
Imagine a note-taker at a long talk. They have one index card. After each sentence they do two things. They rub out a little of what is on the card, because old notes matter less. Then they write a little of the new sentence. The card never grows. At any moment, it holds a fading summary of everything said so far.
That index card is a state, and a model that works this way is a state-space model (SSM). It is the other half of the paper’s title. This chapter builds it from scratch, because Cobra’s speed comes from this one design.
Picture a tank of water with a small leak. Water pours in at a rate x(t) that changes over time. Water leaks out at a rate proportional to the level. The level h(t) rises with the inflow and falls with the leak. If the inflow stops, the level drains toward zero.
The paper writes this tank as an equation. In plain words it says: the change in level equals the leak plus the inflow. The output y(t) is a reading taken from the level.
Tokens arrive one at a time, not as a smooth flow. So we look at the tank only at fixed moments, a step of size Δ apart. Between moments the level obeys the leak. This is called discretization. The rule the paper uses is the zero-order hold: assume the input stays constant between two moments.
Solving the tank equation under that assumption gives two new numbers, written with a bar:
Now the model is a simple loop. For each token k, the new state is the old state, shrunk by Ā, plus the new input, scaled by B̄. The output reads the state. These are the paper’s Equations 3a and 3b:
hk = Ā hk−1 + B̄ xk yk = C hk
Use one number for the state (N = 1) with A = −1, B = 1, C = 1, and Δ = 0.5. Then Ā = exp(−0.5) = 0.6065. And B̄ = (e−0.5 − 1) / (−1) = 0.3935. Feed in the input 1, 0, 0, 0 and apply the loop.
One input of 1 leaves a trace that fades by a factor of 0.6065 each step. That is the memory: recent tokens count most, old ones fade. The state is one number here, and it stays one number however long the input runs.
This loop is a recurrence, so it is an RNN (recurrent neural network: a model that carries a state forward one token at a time). To make the next token, it needs only the last state and the new input. That is why inference is cheap and why memory stays constant.
For training, a loop is awkward. A GPU wants to work on all tokens at once. Because the loop is linear, we can unroll it. The output at step k is a weighted sum of all inputs up to k, with weights C B̄, C Ā B̄, C Ā2 B̄, and so on. Call that list of weights a kernel K. The paper writes this as a convolution (a sliding weighted sum), Equations 4a and 4b: K = (C B̄, C Ā B̄, …) and y = x ∗ K.
Check it with our numbers. K = (0.3935, 0.2387, 0.1448, …). Feed in x = (1, 2). The loop gives h2 = 0.6065 × 0.3935 + 0.3935 × 2 = 1.0257. The convolution gives 0.3935 × 2 + 0.2387 × 1 = 1.0257. They match. The device below runs both and reports the gap.
Blue bars are the input. The warm line is the state. Pick how it is computed, then drag the step size Δ. A bigger Δ forgets faster.
Toy: one state number, A = −1, B = 1, C = 1, so Ā = exp(−Δ) and B̄ = 1 − Ā. The inputs are 1.0 at step 3 and 0.6 at step 12. Both ways of computing are run live in your browser and compared. A real Mamba layer has N state numbers for each of many channels.
With Δ = 0.5, the first pulse has faded to 0.6065 to the power 9, which is 1.1% of its size, by the time the second pulse arrives. Raise Δ to 3 and almost nothing survives a single step. Lower it to 0.05 and the memory lasts the whole run. The paper’s later chapters depend on this one dial.
Chapter 3
Let each token set its own forgetting and writing, so the memory keeps the key word and skips the filler
Go back to the note-taker with one index card. This time the talk has a lot of small talk and one phone number. The note-taker rubs out the card by the same amount after every sentence. After ten sentences of small talk, the phone number is a faint smudge. A fixed fade cannot tell what matters.
A better note-taker listens first. If a sentence is filler, they leave the card alone. If it is the phone number, they wipe a space and write it down hard. The rubbing and the writing now depend on what was said. That single change is what Mamba adds to the state-space model of Chapter 2.
In Chapter 2, Ā and B̄ were the same for every token. Such a model is time-invariant. How much a token still counts later depends only on how far back it was, never on what it was. The kernel K from Chapter 2 is the proof: it is one fixed list of weights.
That is a real limit for language. A useful memory must hold a name and drop a comma. A fixed fade treats both alike. If you set it to remember for a long time, it also clings to filler. If you set it to forget filler, it forgets the name.
The selective state-space model of the Mamba paper (Gu and Dao, arXiv 2312.00752) makes the parameters depend on the input. The Cobra paper states it this way: the SSM “achieves these functions by making the parameters (A, B, C) depend on the input x”. In the Mamba paper (which this lesson draws on from its published description), small learned layers read each token and produce its own step size Δk and its own Bk and Ck. The leak A stays a learned constant, and Δk sets how much it bites for that token.
Look at what Δk does in our one-number model, where Ā = exp(−Δ) and B̄ = 1 − Ā:
So the model learns to turn one dial per token. It does not run a hand-written rule. Training teaches it which kinds of tokens deserve a large Δ.
Let the input be three filler tokens, then one key token, then m more filler tokens. We ask: how much of the final state is owed to the key? The weight of a token in the final state is its own B̄ times the product of every Ā that came after it.
The fixed model uses Δ = 0.5 everywhere, so Ā = 0.6065 and B̄ = 0.3935. With m = 10 filler tokens after the key, the key’s weight is 0.3935 × 0.606510 = 0.3935 × 0.00674 = 0.0027.
The selective model uses Δ = 3 on the key and Δ = 0.02 on filler. The key’s own B̄ is 0.950. Each filler token after it has Ā = 0.980. The key’s weight is 0.950 × 0.98010 = 0.950 × 0.817 = 0.776. The selective memory holds the key about 290 times stronger. Move the slider to see every token’s share.
Each bar is one earlier token’s share of the final state. The blue bar is the key token. Switch between a fixed memory and a selective one, then add filler after the key.
Toy, one state number. Fixed: Δ = 0.5 for every token. Selective: Δ = 3 on the key and Δ = 0.02 on filler, values chosen by hand to show the effect; a trained Mamba learns its own Δ for each token. A token’s share is its B̄ times every Ā after it.
In fixed mode the key bar shrinks fast as filler piles up, and the filler tokens right next to the end dominate. In selective mode the key bar stays tall and the filler bars are slivers. The memory has the same size in both modes. Only the choice of what to write has changed.
Selection has a price. Once Ā and B̄ change with every token, there is no single kernel K. The convolution trick of Chapter 2 no longer applies. A plain loop would work but would be slow to train.
Mamba solves this with a hardware-aware algorithm called the selective scan. A scan is a running total computed in parallel in a tree, and Mamba keeps the working memory in the GPU’s fast on-chip storage. The Cobra paper only names the method and cites the Mamba paper for it. What matters here is the result: training stays fast and inference stays one cheap update per token.
The Cobra paper describes the backbone as a stack of identical blocks. Each block has a short convolution (a small sliding window that mixes a token with its few neighbours), the selective SSM, a residual connection (the block’s input is added back to its output, so each block only needs to learn a correction), and RMSNorm (a step that rescales each token’s numbers to a steady size). The stack takes Lin rows of D numbers in and gives rows of D numbers out.
Chapter 4
Follow one picture and one question through three parts and three equations, with the shape of every tensor
Think of an assembly line with three stations. At the first, two inspectors look at the photo and write down what they see. At the second, a translator rewrites those notes in the language the third station understands. At the third, a writer reads the notes and the customer’s question and composes the answer one word at a time.
Cobra is that line. The paper names the stations: a vision encoder, a projector, and a Mamba language model (their Figure 2). Chapters 1 to 3 gave you the ideas. This chapter puts them in order and tracks the data.
The photo is Xv, a tensor (a block of numbers) of shape 3 × 384 × 384. Two pre-trained encoders read it. DINOv2 is good at the low-level layout of a scene, such as where things are. SigLIP is good at meaning, because it was trained to match pictures with text. The paper joins their outputs side by side, token by token:
The projector φ (the paper’s Equation 6) maps each row of Rv to the language model’s width D:
Hv = φ(Rv)
The paper tests two projectors. One is an MLP (multilayer perceptron: a few matrix multiplications with a bend between them). The other is the lightweight downsample projector, LDPv2, which also pools the rows so that fewer remain. Chapter 5 compares them.
The question is tokenized and embedded to Hq. The picture rows Hv go in front. The whole table H has Lin rows and D columns. The Mamba stack then writes the answer Y one token at a time. Each token depends on the picture, the question and the answer so far (Equation 7):
Step through the line below with a 14-token question. The widths 1,024, 1,152 and 2,560 are the usual sizes of the DINOv2 ViT-L encoder, the SigLIP ViT-SO encoder, and the Mamba-2.8B model. The paper names only the symbols, so treat these three as illustrative.
Press Next stage to light the line one station at a time. The read-out names the shape. Change the question length and see which rows change.
Image size 384 × 384 and 729 image tokens are the paper’s (Table 1). The widths 1,024, 1,152 and 2,560 are the usual sizes of those backbones and are not stated in the paper. The 14-token question is an illustration. The vocabulary of about 50,000 tokens is that of the GPT-NeoX tokenizer the paper names.
The text side starts as words. The tokenizer that GPT-NeoX uses (the paper names it) cuts the question into token numbers, and an embedding layer turns each number into a row of D values. The question becomes the table Hq. The image rows Hv are glued on top. The result has 743 rows and 2,560 columns for our example.
Each Mamba block takes a table of shape (rows, D) and returns a table of the same shape. Shape in equals shape out. That is why blocks stack like floors of a building. The state sits inside each block, so every block keeps its own fixed-size memory. The paper does not list the block count in the text we have, so we do not quote one.
In training, the answer is known. The model reads the picture and question, predicts each answer token, and a loss measures how much probability it gave the true token. This is the standard cross-entropy on the answer tokens, which is Equation 7 turned into a loss. The paper trains the projector and the Mamba language model together. It lists no step that trains the two vision encoders.
In inference, the line runs in two phases. First the model reads all Lin rows in one parallel pass (the prefill) and ends with a state. Then it loops: pick a token, feed it back, update the state, pick the next. Each loop turn touches only the fixed-size state. That is why the cost per answer token does not depend on the 729 picture tokens (Chapter 7 puts a number on this).
Cobra-3.5B uses Mamba-2.8B as the language model, and the paper counts 3.5B parameters in total. The other 0.7B is the two encoders and the projector (our subtraction). Cobra-8B uses a 7B Mamba model and counts 7.8B in total. For comparison, the paper says Cobra-3.5B has about 48% of the parameters of LLaVA v1.5-7B.
A failure worth naming: skip the projector and the line breaks at once. The concatenated features have 2,176 columns and the language model expects 2,560. The shapes do not match. The projector is the part that makes the two worlds fit.
Chapter 5
Fuse a layout expert with a meaning expert, then test whether squeezing the picture into fewer tokens pays off
Two people watch the same street accident. One has a sharp memory for where things stood: the car was two metres from the kerb, the bicycle left of the dog. The other remembers what things were: a delivery van, a red bicycle, a brown dog. Ask either one alone, and you get half the story. Put them together and you get both halves.
Cobra’s vision encoder does exactly this. It does not trust one encoder. It runs two and glues their notes together. This chapter explains why, reads the paper’s test of the idea, and then looks at the second design choice: the projector.
SigLIP is trained on pictures paired with text captions. Its features carry meaning: it knows a picture is about a dog. But a caption rarely says which of two objects is on the left. So SigLIP’s features can lose fine layout detail. The paper says that CLIP-like models “may offer rich semantics” yet “have the potential to lose the detailed information for images themselves”.
DINOv2 is trained on pictures alone, without captions, to give each patch features that still describe its local structure. The paper calls this the “low-level spatial” side. The authors follow two earlier studies (Tong et al. 2024 and the Prismatic VLMs paper) that found a combined encoder helps.
The join is the simplest one possible. Both encoders see the same 384 × 384 image. Each gives 729 rows. The rows are placed side by side, so patch 1 of DINOv2 sits next to patch 1 of SigLIP. Equation 5 from Chapter 4 does it: Rv is 729 rows by DDINOv2 + DSigLIP columns. No attention, no gate, only concatenation.
An ablation removes one piece and measures what changes. The paper removes DINOv2 and keeps SigLIP alone. The numbers come from the ablation table (Table 5 of the arXiv version), for Cobra-3.5B. Every other part of the recipe stays the same.
The pattern fits the story. The benchmarks that need layout (VSR and the three localization tests) lose 5 to 6 points without DINOv2. The benchmarks about meaning or reading text barely move. The paper sums it up: the fusion “significantly improves the accuracy by 5%-6% on VSR and localization benchmarks”.
The symptom: a model that names every object correctly, then says the bicycle is on the right when it is on the left. The measurement: VSR and RefCOCO fall by 5 to 6 points when the layout encoder is removed. The fix: add DINOv2, at the price of a wider feature row and one more encoder to run once per image.
After the glue, the projector maps 2,176 columns (our illustrative width) to D. The paper tests two. The first is an MLP: it keeps all 729 rows and changes only their width. The second is LDPv2, the lightweight downsample projector from MobileVLM v2. It also pools neighbouring patches, so that 729 rows shrink to 196. Fewer image tokens should mean a shorter list for the language model to read.
For a transformer, that is a real saving. Every token is compared with every other. For Cobra, the paper expected a gain and measured none. Table 3 shows 1.54 seconds with the MLP and 1.53 seconds with LDPv2, which is no meaningful change. The paper explains why: for a recurrent model, the number of prompt tokens only affects the first parallel pass. Chapter 7 returns to this.
Worse, the shrink costs accuracy. With LDPv2, TextVQA drops from 58.2 to 54.7 (3.5 points) and VSR from 58.4 to 56.1 (2.3 points). The three localization scores fall by 2.4, 2.7 and 2.0. Only VizWiz goes up, by 0.5. The authors conclude that such a lightweight projector “may be unnecessary” for RNN-based models. Pooling throws away exactly the precise detail that reading and pointing need.
Use the device to look at all nine benchmarks at once. The tick on each bar marks the default Cobra score.
Choose a variant. The purple bar is its score on each benchmark and the black tick marks the full Cobra-3.5B score. The axis starts at 30.
Scores are the paper’s ablation table for Cobra-3.5B (one training run each, as reported). The average is a plain mean over the nine columns, computed here for the reading; the paper reports no such average. Differences of a point or less may be noise from a single run.
Chapter 6
Skip the warm-up stage, train everything together for two epochs, and see what each choice is worth
A new employee joins a firm. The usual plan has two phases. In week one, the employee only learns the office tools, while the rest of the work waits. From week two, the employee starts real jobs. A new idea says: skip week one. Let them learn the tools while they do real jobs, and give them twice the time.
That is the choice in Cobra’s training. This chapter explains the usual recipe, what Cobra changes, the data, the settings, and the experiments that justify the change.
Chapter 1 ended with a promise: the language model must learn to use the picture rows. The common recipe, from LLaVA, has two stages. In the first, pre-alignment, only the projector is trained, on image and caption pairs, with the language model frozen. Its job is to learn a rough translation from vision features to word space. In the second, the language model and the projector are fine-tuned together on instruction data, for one epoch. (An epoch is one full pass through the training data.)
The Cobra authors cite a recent study (Karamcheti et al. 2024, Prismatic VLMs) that suggests the first stage may be unnecessary. They also saw that after the usual one-epoch fine-tune the model “remains underfitted”: it had not finished learning. So they made two moves.
The fine-tuning set mixes three sources, about 1.2 million images in all:
The sources add to 1.275 million conversations (our sum), and the paper counts about 1.2 million images. The text-only ShareGPT part keeps the model good at plain chat while it learns to see.
The paper’s Table 1 gives the setup. Images are 384 × 384, giving 729 image tokens. The global batch size is 128, there are about 19K training steps, and training runs for 2 epochs. The optimizer is AdamW (a standard optimizer that also shrinks weights slowly, here with weight decay 0.1). The learning rate is 2 × 10−5 with a cosine decay (it falls smoothly along a cosine curve) and a warm-up ratio of 0.03 (the rate climbs from zero over the first 3% of steps).
Do the arithmetic. One epoch is about 1.2 million examples. At 128 per step, one epoch takes 1,200,000 / 128 = 9,375 steps. Two epochs take 18,750 steps. The paper says about 19K. The numbers agree.
Training the 2.8B model took about 26.5 hours on 8 A100 80GB GPUs, which is 212 GPU-hours (our multiplication). It used PyTorch FSDP (fully sharded data parallel, which splits the model across GPUs) and mixed precision in FP32 and BF16.
The 2.8B language model is a Mamba model pre-trained on SlimPajama (627 billion tokens), then given chat training (supervised fine-tuning on UltraChat-200k and DPO on UltraFeedback). The 7B model is a base model, pre-trained on RefinedWeb (1.2 trillion tokens) with no chat tuning. The chat model uses the template <|user|> … <|endoftext|> <|assistant|>. The base model uses a plain In: … Out: form.
The device below uses the paper’s ablations on Cobra-3.5B. Pick a recipe and read its average over the nine benchmarks.
Top: what is trained, and for how long. Bottom: the average over nine benchmarks. Pick a recipe.
Scores are the paper’s ablation table for Cobra-3.5B; the averages are computed here. The paper tests “pre-align plus one epoch” against “one epoch”, and “one epoch” against “two epochs”. It does not test pre-alignment with two epochs, so the data cannot say pre-alignment hurts at any length. Block widths are not to scale.
Two findings stand out. First, two epochs beat one on every benchmark. The average rises from 55.7 to 60.2, and RefCOCO alone rises from 42.5 to 52.7. The paper reads this as a sign of underfitting. Second, with one epoch each, adding pre-alignment lowers the average from 55.7 to 53.1, and it lowers every benchmark except TextVQA. The paper’s conclusion is that a pre-aligned projector “actually harms” the result here, which differs from LLaVA-style and MobileVLM-style recipes.
One more test checks the language model itself. A base model with no chat tuning does worse on VizWiz by 2.5 points and on VSR by 4.0, and is within about one point on the rest. VizWiz includes questions that cannot be answered, and the paper does not say why chat tuning helps there.
Chapter 7
Read the speed table line by line, then see why a picture with 729 tokens costs Cobra almost nothing extra
Two couriers deliver parcels across a city. One has a notebook that grows with every stop: before each new stop, the courier rereads the whole notebook. The other keeps one fixed card and updates it at each stop. On a short route, they look alike. On a long route, the first courier slows down.
This chapter tests that picture against the paper’s speed table. First, how the authors measured. Then the numbers. Then the surprise about the image tokens.
The authors ran every model on the same hardware: one NVIDIA A100 PCIe 80GB GPU. Every model got the same example image and the same question, “Describe the image specifically”. Every model had to write 256 tokens. The clock ran from the start of image encoding to the last token. Call that time Ttotal. The speed is then:
Check it on MobileVLM v2: 256 / 5.17 = 49.5 tokens per second. The table lists 49.50, so the two agree. Now the whole table.
Compare Cobra-3.5B with each rival. Against MobileVLM v2: 166.47 / 49.50 = 3.4 times faster. Against LLaVA-Phi: 166.47 / 40.89 = 4.1 times. Against MoE-LLaVA: 166.47 / 20.33 = 8.2 times, which is the paper’s “over 8 times faster”. Cobra took about 30% of MobileVLM v2’s time (1.54 / 5.17 = 0.30). And the 7.8B Cobra-8B, with more than twice the parameters of MobileVLM v2, still wrote faster: 79.92 against 49.50 tokens per second.
Look at the visual token counts. MobileVLM v2 uses only 144, the fewest of any rival. Cobra-3.5B uses 729, five times as many, and is still 3.4 times faster. Then compare the two Cobra variants. Cutting 729 image tokens to 196 changed the time from 1.54 s to 1.53 s. Nothing.
The reason is in the two phases from Chapter 4. First comes the prefill, where the model reads the whole prompt in one parallel pass. Then comes the decode loop, with 256 turns. The 729 picture rows only matter in the prefill, and a parallel pass over 729 rows costs little. In the decode loop, every turn touches only the fixed-size state. The paper says it directly: “the number of prompt tokens only affects the speed of the model’s first parallel forward process”.
For a transformer, the picture rows matter twice: in the prefill, and then again in every one of the 256 decode turns, because each new token looks back at them. So shrinking the picture helps a transformer. It does not help Cobra. That is why the authors write that a lightweight projector “may be unnecessary” for RNN-style multimodal models.
Each dot is one model from the paper’s speed table. Left to right is more visual tokens. Bottom is faster. Pick a model to read its numbers.
All values are the paper’s Table 3: one image, one prompt, 256 output tokens, one A100 PCIe 80GB, image encoding included in the time. The ratio in the read-out is each model’s time divided by Cobra-3.5B’s.
Divide each total time by 256 and you get the cost of one output token, with the image and the prefill folded in. MoE-LLaVA: 12.59 / 256 = 49.2 milliseconds. LLaVA-Phi: 6.26 / 256 = 24.5 ms. MobileVLM v2: 5.17 / 256 = 20.2 ms. Cobra-8B: 3.20 / 256 = 12.5 ms. Cobra-3.5B: 1.54 / 256 = 6.0 ms. These are our divisions of the paper’s totals.
Why is Cobra-8B slower than Cobra-3.5B, when neither depends on the prompt length? A likely reason is plain size: each decode step must read every weight of the language model, and the 7B model has more weights to read than the 2.8B one. The paper does not test this, so treat it as our reading. What the pair does show is that a bigger Cobra still beats the smaller transformer-based models.
The test is one image, one prompt, one GPU and 256 output tokens. It says nothing about much longer answers, many images in one prompt, or batches of requests. Chapter 0’s logic suggests the gap would widen with longer answers, since a transformer pays more per token as the context grows. That is a prediction from the model of cost, not a result in the paper. The time also depends on how well each code base is tuned, which the paper does not discuss.
Chapter 8
Read the benchmark tables, find where Cobra wins and where it trails, and meet the prompt-order surprise
A fast model that is wrong is no use to the robot. Speed only counts if the answers stay good. So the question for this chapter is simple: what does Cobra give up for its speed? The answer, from the paper’s tables, is mixed in a useful way.
The paper uses nine benchmarks (fixed sets of questions with known answers):
LLaVA v1.5-7B is the standard transformer-based baseline. The device shows the score difference on each benchmark, Cobra minus LLaVA. Green bars mean Cobra scores higher, red bars mean lower. Switch between the two Cobra sizes.
Each bar is Cobra’s score minus LLaVA v1.5-7B’s, in points. Right of the centre line, Cobra is ahead.
Scores are the paper’s Tables 2 and 4. LLaVA v1.5-7B: 78.5, 62.0, 50.0, 58.2, 51.5, 85.9 and grounding 55.1, 49.5, 50.9. Cobra-3.5B: 77.8, 62.3, 49.7, 58.2, 58.4, 88.4 and 52.7, 45.6, 46.9. Cobra-8B: 79.2, 63.9, 56.2, 59.5, 62.9, 87.6 and 58.2, 52.5, 54.4. The ablation table gives 48.9 for Cobra-3.5B on RefCOCOg where Table 4 gives 46.9; this device uses Table 4.
Cobra-3.5B has about 48% of LLaVA’s parameters. It ties on TextVQA, is within 0.7 points on VQA-v2, GQA and VizWiz, and wins clearly on the two closed-set tests: VSR by 6.9 points and POPE by 2.5. The authors read the POPE gain as less hallucination and the VSR gain as better spatial judgment. It trails on all three grounding tests, by 2.4, 3.9 and 4.0 points.
Cobra-8B beats LLaVA on all nine, by between 0.7 and 11.4 points. The paper says it is the best among the larger models on every benchmark it tested. Among the small models, Cobra-3.5B is best on all but VQA-v2 and GQA, where MoE-LLaVA is ahead by 2.1 and 0.3 points in Table 2 (the paper says 1 to 2). But MoE-LLaVA is more than 8 times slower (Chapter 7).
The grounding result carries a lesson. Both Cobra models were trained with the same recipe. Yet the 7B language model beat LLaVA while the 2.8B one trailed it. The paper’s reading: the language model’s own strength drives grounding.
The paper adds side-by-side answers. In Table 6, on the bicycle and dog photo, Cobra says the bicycle is on the left of the dog and LLaVA says it is on the right. On a photo of a monkey and a man, Cobra adds a detail LLaVA misses, the man’s white shirt with a floral pattern. In the appendix, on a painting of a lake, the authors mark LLaVA’s answer as hallucinating: it describes a bench and a bird that Cobra does not mention.
Read these with care. The authors chose them, three or four out of thousands of possible questions. They fit the numbers (VSR and POPE up), but they are an illustration, not a test. The tables carry the evidence.
TextVQA gives each image a list of text strings found by an OCR system (optical character recognition: software that reads printed text in a picture). The usual prompt puts the question first and the OCR list after it. The authors tried both orders on Cobra.
Same words, same model, a 15.2 point swing from order alone. The paper suspects “an inductive bias in the RNN model” and offers no proof. One plausible reading: a state-space model writes each token into one fixed memory, so a long tail of OCR strings read last can crowd out the question just before it. Placing the question last keeps it freshest. Treat this as a hypothesis. The practical lesson is firm: with a recurrent backbone, test your prompt order.
Three prompt layouts for the same TextVQA question. Pick a model and compare the bars.
Scores are the paper’s Table 7: Cobra-3.5B 58.2, 43.0 and 47.9; Cobra-8B 59.5, 43.0 and 50.7 for OCR first, OCR last and no OCR. The explanation in the text is the authors’ hypothesis, not a tested result. The axis starts at 30.
Chapter 9
Keep the numbers, the checklist and the core code, and find the next papers to read
You can now rebuild the paper from memory. A model reads a picture as 729 tokens, places them before the question, and lets a Mamba language model answer. The model keeps a fixed-size state instead of a growing cache. So it writes about 3.4 to 4.1 times faster than MobileVLM v2 and LLaVA-Phi, and it keeps comparable accuracy.
Transformer-based vision-language models pay attention to every earlier token, so each new answer token costs more as the context grows. Cobra replaces the transformer with Mamba, a selective state-space model whose cost per token is constant. Two pre-trained encoders, DINOv2 for layout and SigLIP for meaning, are concatenated into 729 patch rows. A projector maps them to the language model’s width, and they are placed before the question. Cobra trains the projector and the Mamba model together, without a pre-alignment stage, for two epochs on about 1.2 million images. Cobra-3.5B writes 166.47 tokens per second, 3.4 times MobileVLM v2. Cobra-8B beats LLaVA v1.5-7B on all nine tests.
| What | Number | Why it matters |
|---|---|---|
| Image tokens | 729 (27 × 27) at 384 × 384 | All of them sit in front of the question |
| Vision features | DINOv2 + SigLIP, concatenated | Layout plus meaning |
| Language models | Mamba-2.8B (627B-token pretraining, chat tuned); Mamba-7B (1.2T tokens, base) | The part that replaces the transformer |
| Total parameters | 3.5B and 7.8B | Cobra-3.5B has about 48% of LLaVA v1.5-7B |
| Training data | 655K + 220K + 400K, about 1.2M images | LLaVA mix, LVIS-Instruct-4V, LRV-Instruct |
| Recipe | no pre-alignment; 2 epochs; batch 128; about 19K steps; AdamW; learning rate 2e-5 | Train the projector and the Mamba model together |
| Compute | 26.5 h on 8 A100 80GB (2.8B model) | About 212 GPU-hours |
| Speed, 256 tokens, one A100 | Cobra-3.5B 166.47 per second; MobileVLM v2 49.50; LLaVA-Phi 40.89; MoE-LLaVA 20.33; Cobra-8B 79.92 | 3.4, 4.1 and 8.2 times faster |
| LDPv2 projector | 196 tokens, 1.53 s against 1.54 s; TextVQA down 3.5 | No speed gain, lower accuracy |
| Without DINOv2 | VSR down 5.2; RefCOCO down 6.0 | The layout encoder earns its place |
| Recipe ablation (our average of nine) | Cobra 60.2; one epoch 55.7; pre-align + one epoch 53.1 | Two epochs beat one; pre-alignment hurt |
| TextVQA prompt order, Cobra-3.5B | OCR first 58.2; OCR last 43.0; none 47.9 | A recurrent backbone cares about order |
| Against LLaVA v1.5-7B, Cobra-3.5B | VSR +6.9; POPE +2.5; grounding 2.4 to 4.0 lower | Mixed, not a sweep |
This sketch follows the paper’s equations. It is a reading aid, not the authors’ code: dino, siglip and mamba stand for the pre-trained models, and the widths 1,024, 1,152 and 2,560 are the usual ones for those backbones.
pythonclass Cobra(nn.Module): def __init__(self, dino, siglip, mamba, d=2560): self.dino, self.siglip = dino, siglip # pre-trained; the recipe does not update them self.proj = MLP(1024 + 1152, d) # Eq 6, random init (no pre-alignment) self.lm = mamba # Mamba-2.8B chat or Mamba-7B def see(self, img): # img: (B, 3, 384, 384) with torch.no_grad(): r = torch.cat([self.dino(img), self.siglip(img)], dim=-1) # Eq 5: (B, 729, 2176) return self.proj(r) # Eq 6: (B, 729, 2560) def loss(self, img, ids, answer_mask): # ids: question + answer tokens h = torch.cat([self.see(img), self.lm.embed(ids)], dim=1) # image rows first: (B, 729+L, 2560) logits = self.lm(inputs_embeds=h)[:, 729:] # next-token scores over the text part return cross_entropy(logits[:, :-1][answer_mask[:, 1:]], ids[:, 1:][answer_mask[:, 1:]]) # Eq 7 as a loss @torch.no_grad() def generate(self, img, ids, n=256): h = torch.cat([self.see(img), self.lm.embed(ids)], dim=1) logits, state = self.lm.prefill(h) # one parallel pass over 729+L rows out = [] for _ in range(n): # decode: one fixed-size state update per token tok = logits[:, -1].argmax(-1) out.append(tok) logits, state = self.lm.step(self.lm.embed(tok), state) return torch.stack(out, dim=1)
The settings to copy are: 384 × 384 images, batch 128, AdamW with learning rate 2e-5, weight decay 0.1, cosine decay, warm-up ratio 0.03, two epochs. Put the OCR text before the question if you use it. Prompt format: the chat template for the 2.8B model and In: … Out: for the base 7B model.
Tick each item you could do on a blank page, without scrolling up.
Nothing is stored or sent. The checklist is for you only.
Cobra is one answer to a question that Lecture 8 raises: what should sit between the vision encoder and the answer? Most large multimodal models use a transformer. Cobra shows that a recurrent state-space backbone can do the job at comparable accuracy and much lower cost per token. It came a few months after the Mamba paper (December 2023).
The paper leaves open what happens to a fixed-size memory when the context holds many images, a long video or a long document, and whether exact recall holds up. The authors point to robotic feedback control as a future use, where fast, repeated looking matters most.
Paper: Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang and Donglin Wang (Zhejiang University, Westlake University, DAMO Academy of Alibaba Group), “Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference”, AAAI 2025, arXiv 2403.14520. Related papers that Cobra cites and builds on: Mamba (2312.00752), LLaVA v1.5 (2310.03744), Prismatic VLMs (2402.07865), DINOv2 (2304.07193), SigLIP (2303.15343) and MobileVLM v2 (2402.03766). Course: MIT MMAI, Spring 2026.
Now press Present or Teach and explain the paper out loud, from memory: why a transformer slows down, how a state keeps what matters, what the three stations do, and which choice the ablations backed. Then go back to the race at the top, pick the slowest rival, and watch the lanes again.