Picture a web page with a jam recipe and two photos. A normal model spends the same weights on every word and every picture piece. MoMa gives each kind of token its own experts, and it reaches the same loss with 3.7 times less compute.
Learn how a model that reads words and pictures as one stream can stop spending the same weights on both, and how to train it.
Press MoMa 4t4i, then Image only. Watch which experts the picture tokens visit, and how much compute the model needs to match the dense baseline. Then we build, piece by piece, the experts, the routers, the causality fix, the depth skips and the training recipe.
You need a rough idea of what a transformer layer does (it mixes tokens, then transforms each one). We build the rest from zero.
A reading for MIT MMAI, Multimodal AI (Spring 2026), Paul Liang. It goes with the lecture on interactions and fusion models. Paper: MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts (arXiv 2407.21770), by Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer and Armen Aghajanyan, Meta FAIR, 2024. Every idea is re-taught here in our own words and devices.
The savings are the paper’s, for the 1.4B compute-matched models trained on 1 trillion tokens (abstract and Section 1): MoE 8x 3.0x overall (text 3.0, image 2.8), MoMa 4t4i 3.7x (text 2.6, image 5.2), MoMa + MoD 4.2x (text 3.4, image 5.3), measured by pre-training loss. The bar is 1 divided by the saving. The twelve tokens and their routes are illustrative, and the boxes show the idea of each design, not its real expert counts per layer.
Chapter 0
See why a model that reads words and pictures in one stream wastes compute, and what the paper changes about it
Open a web page about strawberry jam. It has a paragraph of recipe text, then a photo of the jar, then more text, then a second photo. A model that reads pages like this must handle both kinds of content in one pass. It must also write them: a caption, then a picture, then another caption.
The usual way to build such a model is to bolt parts together. One part reads images and turns them into numbers. Another part reads text. A third part joins the results. The paper describes it as fusing modality-specific encoders or decoders. It works, but the parts meet late, so the model has little chance to mix what it sees with what it reads.
The other way is early fusion: cut the image into small pieces, turn each piece into a token (a single item from a fixed list, like a word piece), and put those tokens in the same line as the text tokens. One transformer reads the whole line. This is the plan of Chameleon, the Meta model that MoMa builds on (Chapter 1). Early fusion can read and write mixed pages with one set of weights.
A transformer stacks many identical layers. Each layer has two parts. Self-attention lets every token look at the other tokens. The feed-forward network (FFN) then transforms each token on its own, using a big block of weights. Most of a model’s weights and most of its arithmetic sit in the FFNs. (Our transformer lesson builds both parts from zero.)
We count the arithmetic in FLOPs, floating-point operations: one multiply or one add of two decimal numbers. FLOPs are the bill for training. A dense model sends every token through every weight of the FFN. Double the FFN, and you double the bill for every token.
The paper attacks this bill with one question. Does a photo token need the same weights as a word token? Think of how much each kind of token carries. In our picture of it, a photo is made of many small pieces, and neighbours often look alike: a patch of red jam sits next to another patch of red jam. A word seems to carry more in a single token. The authors say the two kinds have distinct information densities and redundancy patterns. Yet a dense model treats them the same.
A sparse model keeps many weights but uses only some of them for each token. The first sparse trick is the mixture of experts (MoE, Chapter 2): split the FFN into several smaller FFNs, called experts, and let a small router pick which experts each token visits. The model can hold many more weights, while each token pays for only a few.
MoMa adds one idea to this. The paper calls it modality-aware sparsity. A modality is a kind of data, here text or image. MoMa makes one group of experts for text tokens and one group for image tokens. A text token may only visit text experts. An image token may only visit image experts. Attention stays shared, so the two kinds still talk to each other.
The result, for the paper’s 1.4B-parameter model trained on 1 trillion tokens: it reaches the same pre-training loss as a dense model that spends 3.7 times more FLOPs. For image tokens alone the saving is 5.2 times. A plain mixture of experts with 8 shared experts gets 3 times. You played with these numbers in the hero.
First, feel the problem. How many of the tokens in a page are image tokens? A model sees a fixed window of tokens at a time. The paper trains with a window of 4,096 tokens, and one 512 by 512 image takes 1,024 of them (Chapter 1 shows why). Add images to the window and watch the mix change.
Each block is 64 tokens. Blue blocks are text, purple blocks are image. Drag the number of images. The bar under the grid shows what a dense model spends on each kind of token.
The window of 4,096 tokens (sequence length) and the 1,024 tokens per 512 by 512 image are the paper’s (Sections 2.1 and 4.1). How many images a real page holds is our illustration. Four images would leave no room for text, so the slider stops there.
With one image, a quarter of the window is picture. With three, three quarters. A dense FFN treats all 4,096 tokens the same way, so it spends the same FLOPs on a jam photo token as on a word of the recipe. If the two kinds need different things, one shared FFN must learn both jobs with one set of weights.
That is the trade the paper explores. Give each kind its own experts, and each group can specialise. Keep attention shared, and the kinds still mix. And do it without raising the FLOPs per token, so the comparison with the dense model stays fair.
We follow the paper’s own order, one mechanism per chapter. We add the training recipe and the systems work, then read the results. The last chapter is a Field Guide you can keep.
Chapter 1
Turn an image into 1,024 whole-number tokens so one transformer can read it in the same line as words
Picture a box of 8,192 crayons. Each crayon has a number on it. You cut a photo into small squares and, for each square, you pick the crayon that looks most like it. Now the photo is a list of crayon numbers. Anyone can read that list aloud, one number at a time, like words in a sentence.
That is the whole trick of image tokenization: change a picture into a list of whole numbers from a fixed set, so a language model can treat it as text. The real crayons are not colours. Each one is a small learned pattern of a patch of the image, but the idea is the same.
MoMa starts from Chameleon (Chameleon Team, 2024), a single transformer that models mixed sequences of image tokens and text tokens. The paper (Section 2.1) gives the exact numbers. A learned image tokenizer encodes a 512 by 512 image into 1,024 discrete tokens from a codebook of size 8,192. Text uses a byte-pair-encoding (BPE) tokenizer with a vocabulary of 65,536 tokens. That vocabulary already contains the 8,192 image codes.
So there is one list of 65,536 possible tokens. About one in eight (8,192 of 65,536, which is 12.5%) is an image code, and the rest are text pieces. Any token in a sequence is just a whole number in that list. The model does not need a different input path for images.
Slide the codebook size below. It shows a tiny made-up image cut into 16 by 16 patches. Each patch picks the closest code. Watch how a small codebook makes the picture blocky, and what that costs.
Left: the original, 16 by 16 patches of brightness. Middle: each patch replaced by its nearest code. Right: the error in each patch. Drag the number of codes.
Toy picture and toy codes: each patch is one brightness number, and each code is one brightness level. The real tokenizer is a learned neural network that maps each patch to one of 8,192 codes. The 8,192 codes, the 1,024 tokens per 512 by 512 image and the 65,536-token vocabulary are the paper’s.
With 2 codes the picture is two flat tones, and the average patch is far from its original. With 64 codes it is close. The paper uses 8,192 codes and cuts each image into 1,024 patches. We cannot tell the exact patch grid from the text, so the 16 by 16 grid here is our own simplification.
Now put words and a picture in one line. Take the prompt in the paper’s Figure 1: What does the flower of strawberry look like? followed by an image. The paper shows special markers [BOI] and [EOI]. They mark the beginning and the end of an image. Here is the data flow, with shapes. Say the batch holds B sequences of 4,096 tokens, and the hidden size is d.
Because the loss is the same for both kinds, the paper can split it afterwards. It reports an image loss and a text loss: the average of the same loss over image positions and over text positions. These two numbers drive every plot in the paper.
python # early fusion, the data flow of one training example ids = tok_text("What does the flower of strawberry look like?") ids += [BOI] + img_tokenizer(photo) + [EOI] # 1,024 codes x = embed[ids] # [T, d] for block in blocks: x = block(x) # attention + FFN, shape kept logits = x @ head # [T, 65536] loss = cross_entropy(logits[:-1], ids[1:]) # next-token, both modalities
The paper lists four gains from early fusion (Section 2.1). The model learns a shared representation space for images and text. It handles arbitrary sequences of both. The token-based design scales uniformly: Chameleon trained on about 10 trillion mixed-modal tokens. And the whole model trains end to end.
Here is what the first gain means in practice. Ask the model for a recipe with a picture. It writes the steps as text tokens, then emits [BOI], then 1,024 image codes one after another, then [EOI], then the next step. It never switches networks. The same next-token loop produces the words and the picture, which is why the paper can train and compare both with one number per modality.
The cost is the one from Chapter 0. Every image is 1,024 tokens, and every token goes through the full FFN. A page with a few photos is mostly image tokens. The paper says scaling such models to larger sizes “presents significant computational challenges”, and it turns to sparse architectures for relief.
Chapter 2
Split one big feed-forward block into several, let a router choose, and see why letting the experts choose the tokens keeps the load even
A clinic has one doctor who sees every patient: sprains, rashes, colds, check-ups. The queue grows, so the clinic hires eight doctors. Now a receptionist must decide who sees whom. If the receptionist sends everyone to the doctor at the desk nearest the door, that doctor drowns and seven others sit idle. The hard part is not hiring. It is the sorting.
A mixture of experts (MoE) is that clinic inside a neural network. Our mixture-of-experts lesson builds it in full. Here we need only the parts MoMa changes.
Recall the feed-forward network (FFN). It takes each token’s vector of size d, widens it to a larger size f, applies a non-linear step, and shrinks it back to d. The paper’s FFN uses SwiGLU, a common form with an extra gate, but any FFN works for our story. The shape never changes: a token vector goes in as [d] and comes out as [d].
An expert is one such FFN. Instead of one FFN per layer, an MoE layer holds E of them. The router is a single small matrix Wg of shape [d, E]. Multiply a token’s vector by it and you get E numbers: one affinity score per expert, a measure of how well that expert suits the token.
Table 1 shows the 435M dense model growing to 1.9B parameters when it gets 8 experts per layer. That fits each expert being about as big as the dense FFN it replaces (our reading; the table does not list the expert width). A sparse model then holds far more weights than its dense twin. We pay for them with memory, not with FLOPs per token. Here is why.
In token-choice routing (the original MoE), each token looks at its E scores and goes to the top one or two experts. This is the receptionist who lets patients pick a doctor. Popular doctors overflow. The usual fix is an extra penalty in the loss that pushes toward even use, a load-balancing loss.
In expert-choice routing (Zhou et al., 2022), the choice flips. Each expert looks at all the tokens in the batch and takes the top ke by score. Every expert gets exactly ke tokens, so the load is even by construction. A token may be chosen by several experts, or by none. The paper writes it this way (Section 2.2): ke = b · ce, where b is the number of tokens in the batch and ce = 1/E is the capacity factor.
Work it with the paper’s numbers. A batch has b = 4,096 tokens and E = 8 experts. Then ce = 1/8 and ke = 4,096 × 1/8 = 512. Each expert processes 512 tokens. Add up the work: 8 experts × 512 tokens = 4,096 token visits. That equals one token visit per token, which is exactly what one dense FFN would do. So the FLOPs per token match the dense model, and the load is perfectly even.
Columns are 12 tokens, rows are 4 experts, and a darker cell means a higher score. Outlined cells are the choices. Switch the rule, then press New batch to draw fresh scores.
Toy batch: 12 tokens, 4 experts, random scores drawn in your browser. With expert choice each expert takes its top 12 × 1/4 = 3 tokens, the same rule as ke = b · ce in the paper. Token choice here sends each token to its single best expert.
Try a few batches. Under token choice, the loads wander: one expert often gets five or six tokens while another gets one or none. Under expert choice, every bar is the same height, always three. But look at the top row of counts: some tokens are picked by two experts and some by none. A token picked by no expert skips the FFN, and the residual connection carries it through unchanged.
For a token x, the layer adds up the outputs of the experts that selected it, each scaled by a gate value. Here is the form, with the paper’s sigmoid gate (Chapter 3 gives the full equation).
Two experts pick x. Gates 0.8 and 0.3. Outputs [1, 0] and [0, 2]. So y = 0.8 · [1, 0] + 0.3 · [0, 2] = [0.8, 0.6].
The gate does a second job. Because the output is scaled by g, the router receives a learning signal: if giving an expert more weight lowers the loss, the router’s weights move to raise that gate. This is how a router learns without being told the right answer.
Symptom: training is slow, and some experts barely learn. Measurement: count tokens per expert over a batch. A few experts hold most of them. In a distributed run, the busiest expert sets the speed of the whole step. Fix: token choice adds a balancing loss. Expert choice builds the balance in. The paper picks expert choice and notes that this “eliminates needing a separate load-balancing loss term”, which makes optimisation more stable. It also keeps the computation graph static, with the same shapes every step, which matters for training speed.
Expert choice has a cost, and Chapter 4 is about it: an expert picks its top tokens by looking at every token in the batch, including tokens that come later in the text.
Chapter 3
Send each token first to the group for its modality, then let a learned router pick inside that group
Back to the clinic. The new building has two wings, one for children and one for adults. A guard at the door sends each patient to the right wing by age. This needs no judgement, only a look at the patient. Inside each wing, a receptionist then sends the patient to the right specialist, and that choice is learned from experience.
That is hierarchical routing, and it is the heart of MoMa (mixture of modality-aware experts). The paper (Section 2.2) divides the experts of each MoE layer into expert groups, one per modality. The set of experts is E = {ET1, …, ETm, EI1, …, EIn}: m text experts, written with a T, and n image experts, written with an I.
Stage 1: modality-aware routing. Each token goes to the group that matches its modality. A text token may never reach an image expert, and an image token may never reach a text expert. There is no learning here and no score. The token’s modality decides.
Stage 2: intra-modality routing. Inside its group M, a learned router picks the experts. The router is a matrix WgM of shape [d, |EM|], one for each group. Each expert in the group takes the top ke tokens of that modality by score, using expert choice from Chapter 2.
The capacity changes in one small way. The bucket size is ke = bM · ce, where bM is the number of tokens of modality M in the batch and ce = 1/|EM|. Each expert takes an equal share of its own modality’s tokens, not of the whole batch.
Equation 1 of the paper says it in one line. It has two parts: the output y, and the gate g.
The structure of the sum is the whole idea. A text token sees only m experts. An image token sees only n. The two groups do not compete for the same slots.
Take a batch with N tokens, flattened to [N, d]. Each step below follows Figure 1b of the paper.
Now put numbers on it. Let N = 4,096 tokens, with 3,072 text and 1,024 image tokens (illustrative: one image). Take 4 text and 4 image experts. Then ce = 1/4 in both groups. Each text expert takes 3,072 × 1/4 = 768 tokens. Each image expert takes 1,024 × 1/4 = 256 tokens. The total is 4 × 768 + 4 × 256 = 4,096 token visits. Again one visit per token, so the FLOPs match a dense FFN. Compare a mixed pool of 8 experts: each takes 512, and which modality fills the slots is left to the router.
A batch of 4,096 tokens. Bars show how many tokens each expert processes. Drag the share of image tokens and switch the layout.
The bucket rule ke = bM · ce with ce = 1/|EM| is the paper’s (Section 2.2). The batch size and the image share are our illustration; the paper says only that it matches the data mix to the expert ratio (Chapter 7). In the mixed pool, how many tokens of each kind land in an expert is up to the router, so those bars are shown as one colour.
Drag the share to 50%. In the MoMa layout, a text expert and an image expert now take the same number of tokens, 512 each. At 25% image, image experts take far fewer tokens than text experts. The image experts are not idle: their buckets are sized to the image tokens’ own share. The total stays at 4,096 token visits whatever the share, so the FLOPs per token never change.
The paper names three benefits. Improved efficiency: a token never spends compute on an expert built for another modality. Enhanced specialisation: a group sees only its own kind of data and can learn finer features for it. Maintained cross-modal integration: attention layers and the residual stream are not split, so text tokens still attend to image tokens and the other way round.
The extreme case is 1t1i: one text expert and one image expert, so ce = 1 and every token is processed by its modality’s FFN. There is no learned routing at all. It is simply modality untying: two separate FFNs, picked by a rule. The paper uses it as a baseline and as the seed for upcycling (Chapter 6).
Chapter 4
Find out how picking the top tokens in a batch lets the future leak into the past, and how a small second router closes the leak
You write a story one word at a time. Before you type each word, a helper must decide: should this word get the expensive specialist or not? A fair helper looks only at the words so far. Now imagine a helper who has the finished story on the desk and picks the 30% of words that look most important. The helper’s choice for word 5 now depends on words 6 to 100. When you write live, those words do not exist.
That is the problem with expert choice (Chapter 2) in a language model. A language model predicts the next token from the tokens before it. This rule is called causality: the output at position t may depend only on positions up to t.
Expert choice says: each expert takes the top ke tokens in the batch. To know whether token t is in the top ke, you must compare its score with every other token’s score. That includes tokens after t. So whether token t is processed, and by how much, depends on the future.
The paper says it plainly (Section 2.2): expert-choice routing “compromises causality in auto-regressive language modeling, as each expert selects the top tokens to process in a batch by comparing token scores across all tokens.” In training it is a leak: the model can use future information, and the training loss comes out too good. At inference it is worse. When a model writes one token at a time, there is no batch of finished text to rank.
The paper is open about this in Section 4.1: its training loss calculation compromises causality, because token selection considers the top proportion of a batch, including future tokens. It therefore checks the final models again with a causal method (Section 4.6, Chapter 8 here).
Fix 1: a sigmoid score. The router scores a token with a sigmoid, a curve that squashes any number into the range 0 to 1. With a sigmoid, each token’s affinity for an expert is computed from that token alone. It does not depend on any other token. This does not remove the top-k step, but it means the score itself is clean and can be predicted from the token’s own hidden state.
Fix 2: auxiliary routers. Train a second, small router to predict, from the token alone, whether the expert would have picked it. The paper (Section 2.4) defines it for each MoE layer as a two-layer network:
The auxiliary router uses only x, the token’s own hidden state. It cannot see other tokens, so it cannot leak. For depth routing (Chapter 5) the paper uses the same recipe with output size 1.
The paper trains in two stages. First it trains the main model to convergence, with ordinary batch top-k routing. Then it trains the auxiliary routers alone, with binary cross-entropy (a loss for yes-or-no labels). The labels are the “ground-truth top-k routing assignments computed over an entire batch”: for each token and expert, did the batch rule pick it? The main model stays fixed.
At inference, the main router keeps one job: it produces the gate values g. The auxiliary router decides who is processed: a token goes to an expert when its auxiliary score is above 0.5. The appendix (Table 6) gives the training recipe: peak learning rate 1e-4 decaying linearly to 1e-6, 1,000 or 2,000 warm-up steps, 5,000 to 10,000 steps, and 4,194,304 tokens per batch at length 4,096.
Try it below. A story has nine written tokens and one still to come. The expert takes the top 3 by main-router score.
Ten tokens, left to right. Bars are scores for one expert. Drag the score of token 10, which has not been written yet, and watch whether an earlier token changes its fate.
Toy scores, made up to show the leak; the top-k rule, the sigmoid scores and the 0.5 threshold are the paper’s (Sections 2.2 and 2.4). The auxiliary scores here are drawn so they agree with the batch rule when token 10 is low. A real auxiliary router is only an approximation, and Chapter 8 shows what happens when it is wrong.
In batch mode, push token 10 above 0.66. Token 8, which was chosen, drops out, even though nothing about token 8 changed. The future moved the past. In auxiliary mode you can slide token 10 anywhere and every earlier token keeps its fate. The cost: the number of chosen tokens is no longer exactly 3. Token 10 simply joins when its own score passes 0.5, so the load is balanced only on average.
Symptom: a model that trains well and then answers worse when used as a language model. Measurement: compare training loss, which sees the batch, with held-out perplexity measured through the auxiliary routers. Fix: sigmoid scores plus a second-stage auxiliary router. For mixture of experts this works: Table 3 shows the MoE models keeping their ranking at inference.
Chapter 5
Add a second kind of sparsity along the depth of the network, so only the top-scoring quarter of tokens enter some layers
A commuter train has a local line and an express line. A passenger who needs the next stop gets off early. A passenger going far stays on and passes every station. Not every passenger needs every stop.
Tokens are like that. In the sentence “The cat sat on the mat”, the word “the” is easy to handle and needs little thought. A rare word or a hard part of a picture may need every layer. A dense model gives every token the same depth. Mixture of depths (MoD, Raposo et al., 2024) lets some tokens skip layers.
So far we added sparsity in width: many experts side by side inside one layer. MoMa adds sparsity in depth: some tokens go around a whole layer. The paper (Section 2.3) says they apply MoD “prior to any mixture-of-experts (MoE) routing, ensuring it is applied to the full batch before modality split”. The depth decision comes first and covers both kinds of token. Then the survivors go through the modality split of Chapter 3.
A token that skips a layer is not deleted. The transformer adds each layer’s output to the token’s vector through a residual connection, a path that carries the old vector forward. If the layer does nothing for a token, the old vector just moves on to the next layer unchanged.
The depth router is a single column of weights. For each MoD layer, the paper computes the token-to-layer affinity score with a projection Ŵg of shape [d, 1] and a sigmoid:
Selection works like expert choice. The layer has a fixed capacity: it admits the kd top-scoring tokens of the batch, where kd = b · cd. Here b is the number of tokens in the batch and cd is the capacity factor between 0 and 1, “chosen empirically”. The paper sets cd = 25% and puts MoD in alternating layers, starting with layer 0.
So half the layers are ordinary and half are MoD layers that admit a quarter of the tokens. Because the choice is the top-scoring tokens of a batch, it has the same causality problem as expert choice. The paper adds an auxiliary router for each MoD layer, of the same form as in Chapter 4 with output size 1.
A MoD layer does only a quarter of the work, so a MoD model with the same number of layers is cheaper than the dense one. To keep the comparison fair, the paper adds layers: “we first fixed the MoD layer interval and capacity factor cd, then adjust the total number of transformer layers” so the FLOPs per token match the dense model. The hidden size stays the same.
Check it for the 435M model (Table 1). The dense model has 24 layers. The MoD model has 40. Of those 40, 20 are ordinary and 20 are MoD layers at 25%. Count the layer-work: 20 × 1 + 20 × 0.25 = 25, close to the dense 24. The match is only approximate, because the paper tunes the layer count for the real FLOPs, which include more than the layer count. The same pair gives 14 layers for the 90M model, where the dense one has 8.
Eight tokens (rows) pass six layers (columns). Layers 0, 2 and 4 are depth-routed. A filled cell means the token entered the layer, a hollow one means it took the bypass. Drag the capacity.
Toy tower: 8 tokens, 6 layers, with a fixed random score for each token in each depth-routed layer. The alternating pattern from layer 0 and the top-k rule are the paper’s (Sections 2.3 and 4.1), whose real default is cd = 25%. The FLOPs figure counts a depth-routed layer as cd of an ordinary layer, ignoring everything else.
At 25%, only 2 of 8 tokens enter each depth-routed layer. The stack as a whole costs 3.75 layers of work, against 6 for the dense stack. To spend the same as the dense one, you could add layers: about 9.6, using 12/(1 + cd). The paper’s real jump, 24 to 40 layers, is a ratio of 1.67; our toy formula gives 1.6, close enough to show the idea.
Where does the saving show up? In attention as well as in the FFN. The paper used the “staged” version, which skips both for tokens that do not enter. The other version in the MoD paper always computes full attention, so it saves less. The paper chose the staged version because it “also introduces sparsity in attention”.
Depth routing is the part of the paper that works best in training and worst in use. Adding MoD to MoMa lowers the training loss (Chapter 8: 4.2 times against 3.7 times). But the final model, tested with its auxiliary routers, scores worse than the plain MoMa model. The reason is simple. A token skipped by mistake has no other path in that layer. An MoE token sent to the wrong expert can still be handled by another. Chapter 8 measures how small the router’s error rate must be.
Chapter 6
Train a small seed model first, copy its experts, and let the routers learn on a space that already means something
A city plans four fire stations. The planner must draw four districts, one for each station. If the planner draws them before anyone knows where people live, the lines are guesses. If the planner waits one year, counts the calls, and then draws the districts, the lines fit the city.
A router has the same job. It cuts the space of token vectors into regions, one for each expert. On day one of training, token vectors are close to random, so the regions mean little. The router learns a poor split, and the experts then learn from a poor split. The paper (Section 2.5) says that a router’s representation space “is sub-optimal in the early stages of model training, leading to a sub-optimally trained routing function”.
Upcycling (Komatsuzaki et al., 2023) starts with a model that has no routing and grows it into a mixture of experts. The paper adapts it to MoMa, in four steps.
The scale is small. For the 1.4B models, Table 5 lists 120,000 steps at 8,388,608 tokens per step, about 1 trillion tokens in all. Ten thousand steps is therefore about 84 billion tokens, under a tenth of the budget.
Count it with an example. Say the full run is 120,000 steps and the seed lasts 10,000. Stage 2 then runs the remaining 110,000 steps, with fresh data and a fresh learning-rate schedule. The seed’s 10,000 steps are not free: they use the same hardware as any other steps. That is why the paper shifts every upcycled curve by the seed’s length before it compares them with the scratch run.
All the copies begin identical. The router must find out how to tell them apart. The paper helps it with noise, Equation 5:
Why this noise in particular? There is a neat fact. If L is logistic noise, then the chance that x + L is above zero equals Sigmoid(x). Work it with x = 1. Sigmoid(1) = 0.731. So a noisy score x + L lands above zero in 73.1% of draws. The noisy score behaves like a coin that comes up heads with the probability the router believes in, yet it stays smooth, so gradients can flow through it. The paper describes it as letting the router “differentiably sample experts”. Our reading: with random draws, a token sometimes lands at an expert that was not its first choice, and the copies get a chance to move apart.
Section 4.4 compares a 2.3B model with MoD, 4 text experts and 4 image experts (mod_moe_4t4i). One copy trains from scratch. Two start from a mod_moe_1t1i seed, trained for 10,000 and 20,000 steps. All curves are shifted by the steps the seed used, so the stage-1 cost counts against them. Upcycling wins: 1.2 times in FLOPs after 10,000 seed steps and 1.16 times after 20,000. The gap between upcycled and scratch models “widens throughout training”.
The top bar is the training run. Teal marks the 1t1i seed, warm marks the 4t4i model. Pick how long the seed trains, and read the gain below.
The gains (1.2x at 10k steps, 1.16x at 20k steps, against training from scratch) are the paper’s (Section 4.4 and Figure 6), measured on a 2.3B model with MoD and 4t4i. The learning-rate sketch is schematic: the paper says only that the scheduler is reset. The paper recommends 10k steps and guesses that the best number may change beyond 1 trillion tokens.
Why not seed longer? The paper suggests that a sweet spot likely exists. Undertraining the seed gives the routers little to work with. Overtraining it “impedes future specialization”, because the copies already know too much alike. The 10k seed beat the 20k seed, but both beat scratch.
Symptom: a mixture of experts trained from scratch lags behind where its FLOPs say it should be. Measurement: compare its loss curve with an upcycled run at equal total FLOPs. Fix: a short dense seed, then the split. The cost is a longer recipe with two stages, and a new knob, the seed length.
Chapter 7
See why fewer FLOPs do not always mean faster training, and how matching the data mix to the expert mix keeps many GPUs in step
A kitchen has 256 cooks. Each one gets a tray of orders for the evening rush. Every few minutes, all cooks must pass their plates to the next station together. If one cook has a tray with twice as many orders, 255 cooks stand idle and wait. The kitchen is only as fast as its slowest cook.
Training a large model on many GPUs works the same way. The paper trains with Fully Sharded Data Parallel (FSDP, Zhao et al., 2023). In FSDP each GPU keeps only a slice (a shard) of the weights. Just before a layer runs, every GPU collects the full weights of that layer from the others, which is called prefetching. After the layer runs, the GPUs drop the borrowed weights and share the gradients. The steps are in lockstep, so all GPUs wait for the slowest.
A dense model gives every GPU the same work. MoMa does not. The paper (Section 3.1) explains: “the ratio of text to image tokens can vary significantly across different GPUs and iterations.” An uneven mix creates a straggler, a GPU that finishes late. It delays the prefetch of weights for later layers and the release of gradients from earlier ones. In the paper’s words, it “effectively bounds the training latency by the maximum time required to process text and image experts across all GPUs”.
Our reading of why this is MoMa’s problem: its text and image experts are separate pools, so a GPU with many photos has a heavy image phase and a light text phase.
The paper’s fix is plain. It builds “a balanced data mix that aligns the text-to-image data ratio with the expert ratio on each GPU”. With 4 text and 4 image experts, the mix is set so that each GPU’s batch has the matching share. It keeps the load balanced in expectation, not exactly. The paper names the alternative, moving tokens between GPUs at every FFN layer, and rejects it for now because of the extra communication.
Try it. Each bar below is one GPU’s step. Blue is the text-expert phase, purple the image-expert phase. We model the phases as lockstep: all GPUs finish the text phase together, then the image phase, so the step takes (slowest text phase) + (slowest image phase).
Eight GPUs, each with a batch of the same size. Choose how the batches are mixed, then press New step a few times. The dashed line is where the step ends.
Illustrative model, not the paper’s measurement. Each GPU’s time in a phase is proportional to its tokens of that kind; the target is 25% image; the spread is how far an image share may stray from it; a matched mix keeps a quarter of that spread. The lockstep rule follows the paper’s remark that latency is bounded by the maximum time to process text and image experts across GPUs.
With a wide spread, the dashed line sits well beyond the average GPU. Every bar shorter than the line is idle time. Matching the mix does not make the bars identical, but it pulls them close. The paper admits the limit in Section 6: matching the ratio holds “in expectation”, and a batch can still deviate on a single GPU in a single step.
The paper tried three ways to run the experts (Section 3.2). First, make the experts the same size for both modalities, forbid cross-modal routing, and run everything at once. This needs every expert to get the same number of tokens. Second, use block-sparse kernels (Gale et al., 2023), which allow uneven counts. Third, run the modality groups one after the other. While the text experts compute, the GPUs can already prefetch the weights of the image experts. This eases memory pressure and assumes nothing about balance.
The paper picks the third: “a clean and flexible choice for our experimental environment at the current scale”. Each GPU has so many tokens that running several matrix multiplies in turn still keeps the hardware busy. Smaller batches might need the other approaches.
The paper also lists other speed-ups. It quantises gradient communication, fuses GPU kernels automatically, and compiles with torch.compile. A MoMa-specific trick reuses the indices of each modality’s tokens across layers, so the CPU and GPU synchronise less. The paper notes that this trick does not work with MoD, so it was left out of the final comparison to keep it fair.
The paper measured throughput on 256 A100 GPUs, with sequence length 4,096 and 6 sequences per GPU, for models whose FLOPs match the 435M dense model. WPS is words per second per GPU. The paper does not define it further, so we read it as tokens per second.
| Configuration | WPS per GPU | Change vs dense |
|---|---|---|
| Dense | 31,970 | baseline |
| MoMa 8X (mixed experts) | 28,990 | −9% |
| MoMa 1t1i | 30,196 | −6% |
| MoMa 4t4i | 26,424 | −17% |
| MoD MoMa 1t1i | 25,367 | −21% |
| MoD MoMa 4t4i | 22,114 | −31% |
Read the rows in order. Sparsity is not free. Plain expert-choice MoE loses 9%, probably from routing work and from synchronising the gradients of all experts. MoMa 1t1i loses 6%, mostly from computing the token indices for each modality, which could be done once and reused. MoMa 4t4i loses 17%: about 11 more points from adding experts, similar to the 9% step from dense to MoE. MoD adds another 14 to 15 points, because of its extra router and because depth routing breaks the planned mix: the number of active tokens of each modality changes at every MoD layer. The paper suggests forcing the MoD routers to admit tokens in the planned ratio, and leaves the effect on quality to future work.
So the headline gain has a price tag. MoMa 4t4i gets 3.7 times in FLOPs and, in the paper’s own words, pays a 17% throughput reduction. A rough net, our arithmetic and not a measurement, is 3.7 × 0.83, about 3.1 times in time. It mixes two model sizes (the 3.7 times is at 1.4B, Table 2 at 435M), so treat it as a sanity check only.
Chapter 8
Read the paper’s speed-up factor, its expert-count ablations and its held-out tests, and see where depth routing breaks
Two runners race. One is the dense baseline, the other the sparse model. Both get the same fuel, and we check who is further along. That is how most papers compare models. MoMa asks a sharper question. How much fuel does the dense runner need to reach the same spot? If the answer is 3.7 times more, the sparse runner is 3.7 times as efficient.
The paper defines the pre-training speed-up factor η (Section 4.2). A sparse model with FLOPs per token matched to a dense model reaches the dense model’s final pre-training loss with only 1/η of the FLOPs. So η = 3.7 means a dense model would need 3.7 times the compute. The paper reports η overall, for text tokens alone and for image tokens alone, since the loss splits by modality (Chapter 1).
The paper uses the training loss as its yardstick, since 1 trillion tokens is less than one pass over its data, so training loss behaves like validation loss. We saw the caveat in Chapter 4: with expert choice the training loss peeks at the future. That is why the paper later re-checks with causal inference.
The setup: three dense sizes (90M, 435M and 1.4B parameters) and compute-matched sparse variants, all trained for 1 trillion tokens with sequence length 4,096. The learning rate peaks at 1e-4 after 4,000 warm-up steps and decays linearly to 1% of the peak. Every layer is an MoE layer, and MoD sits in alternating layers at 25% capacity.
At the 1.4B size the paper reports the following (abstract and Section 1).
Three lessons sit in those numbers. First, untying helps image tokens most: the 1t1i model, with only one expert per modality and no learned routing, nearly matches the 8-expert mixed MoE on image loss, while its text loss stays higher. Second, adding experts per group (4t4i) beats the mixed pool at every size, with a large margin on images, but with a slight text regression. The authors read this as a sign that text may benefit from more experts. Third, MoD helps. The mod_moe_1t1i model does as well as or better than moe_4t4i. Stacking MoD on top of 4t4i yields smaller gains, with the text gain larger than the image gain.
Use the device below to flip between the training view and the held-out view, which we reach next.
First view: how many times fewer FLOPs than dense, from training loss (longer is better). Second view: perplexity on held-out text and pictures, using causal inference (shorter is better). Choose a view and a kind of token.
Training view: the 1.4B compute-matched models at 1 trillion tokens (abstract and Section 1); the paper gives no η for 1t1i. Test view: Table 3, held-out OBELICS interleaved data (overall, text and image tokens), lower is better. The MoMa plus MoD model is the 2.3B-parameter one that is compute-matched to the 1.4B dense model.
Flip to the held-out view. For all tokens, the ranking among the earlier models holds: 4t4i is best, 8x next, 1t1i, then dense. Switch to text only and 8x takes the lead. But the MoD model, which led the training view, now sits behind plain 4t4i. We come to that below.
Section 4.3 varies the expert counts at the 435M size. With the same count in each group, training loss improves as experts grow. The two modalities scale differently. Text keeps improving with each doubling, while image shows diminishing returns going from 2 to 4 experts.
With a fixed total of 8 experts, the paper tries 7t1i, 6t2i and 4t4i. More experts help a modality, but more than 4 text experts has diminishing returns. The three overall losses end up close. The 6t2i model has a slightly lower average loss, yet the paper picks 4t4i, because a balanced allocation fits its data mix better for load balancing (Chapter 7). This is a design choice made for the systems, not only the loss.
Section 4.6 trains the auxiliary routers and tests the 1.4B models on held-out data: perplexity on OBELICS (interleaved web documents) and Shutterstock, caption perplexity on COCO and Flickr30k, perplexity of answers on VQA-v2, and eight text commonsense tasks. They report perplexity because their checkpoints are “under-trained”, so generation metrics would not be meaningful.
| Model (1.4B-matched) | OBELICS overall | COCO text to image | COCO image to text | Commonsense avg (%) |
|---|---|---|---|---|
| Dense | 44.6 | 458.3 | 21.7 | 52.4 |
| MoMa 1t1i | 41.5 | 416.7 | 21.9 | 53.3 |
| MoE 8x | 40.4 | 426.5 | 18.9 | 54.8 |
| MoMa 4t4i | 39.0 | 392.5 | 20.9 | 54.2 |
| MoD MoMa 4t4i (2.3B) | 41.4 | 428.1 | 22.9 | not reported |
Perplexity: lower is better. Commonsense: higher is better. From Tables 3 and 4 of the paper, copied exactly. MoMa 4t4i has the best overall interleaved result, and the best score on every text-to-image caption metric. The 8x mixed model is best at describing an image in text (image to text) and on the text-only commonsense average. So a MoMa model gives up a little on pure text to win a lot on images.
Symptom: the 2.3B MoD MoMa 4t4i model has the best training loss but, in Table 3, trails the smaller 1.4B MoMa 4t4i on every metric (OBELICS overall 41.4 against 39.0). Measurement: the authors take a smaller 635M MoD MoMa 4t4i model. With the training router and a batch top-k, it beats the 435M MoMa 4t4i, as it should. Then they add random noise to the router’s selection. Once the noise ratio σ passes 0.5%, the MoD model falls behind. An auxiliary router cannot reach so small an error. Fix: none yet. The paper calls for “a robust approach for practical MoD applications”.
The paper gives a reason why MoE is tougher than MoD. A token sent to the wrong expert can still be processed by another expert. In MoD each layer has a single switch, so a wrong skip “cannot be remedied”. The paper calls this likely, not proven.
Chapter 9
Keep the numbers, the terms, the checklist and the core code, and see where the paper sits in the field
MoMa takes an early-fusion model that reads text and image tokens in one stream and makes its feed-forward layers sparse in a way that respects the two kinds of token. Experts are split into a text group and an image group. A fixed rule picks the group, and a learned expert-choice router picks tokens inside it. A second router keeps inference causal. A depth router can skip whole layers for 75% of tokens. A short seed stage and noisy routers help the experts split. Matching the data mix to the expert mix keeps training balanced. The reward at 1.4B and 1 trillion tokens: 3.7 times fewer FLOPs for the same loss.
| Number | What it is | Where |
|---|---|---|
| 1,024 / 8,192 / 65,536 | Tokens per 512 by 512 image, image codebook size, text vocabulary (which holds the codes) | Section 2.1 |
| 4,096 | Sequence length in training | Section 4.1 |
| 3.7x (2.6 text, 5.2 image) | FLOPs saving of MoMa 4t4i at 1.4B, 1T tokens | Abstract, Section 1 |
| 3.0x (3.0 text, 2.8 image) | Saving of the mixed 8-expert MoE | Abstract, Section 1 |
| 4.2x (3.4 text, 5.3 image) | Saving of MoMa plus MoD, by training loss only | Abstract, Section 1 |
| 1/E | Expert capacity factor ce, so ke = bM/EM tokens per expert | Section 2.2 |
| 25%, alternate layers | MoD capacity cd and layer pattern, from layer 0 | Section 4.1 |
| > 0.5 | Auxiliary-router threshold at inference | Section 2.4 |
| 10k steps: 1.2x, 20k: 1.16x | Upcycling gain over scratch (2.3B MoD 4t4i) | Section 4.4 |
| −9, −6, −17, −21, −31% | Throughput of MoE 8X, 1t1i, 4t4i, MoD 1t1i, MoD 4t4i vs dense | Table 2 |
| σ > 0.5% | Router noise at which the MoD model loses to MoMa without MoD | Section 4.6 |
| Term | In one line |
|---|---|
| Early fusion | Text and image tokens in one line, read by one transformer from the first layer. |
| Expert | One feed-forward network out of several in a layer. |
| Expert choice | Each expert takes its top-k tokens of the batch, so the load is even. |
| Modality-aware sparsity | Separate expert groups for text and for image tokens. |
| 4t4i, 1t1i | 4 text and 4 image experts; 1 text and 1 image expert (untied FFNs, no routing). |
| Auxiliary router | Small network that predicts from one token whether a router would pick it, used at inference. |
| Mixture of depths | A router lets only the top cd of tokens enter a layer; the rest take the residual path. |
| Upcycling | Start with one expert per modality, copy it into many, reset the schedule, keep the data. |
| Straggler | The slowest GPU, which the whole step waits for. |
| η | Speed-up: the sparse model matches dense loss with 1/η of the FLOPs. |
A sketch of one MoMa feed-forward layer, with expert choice. It follows the paper’s Equation 1 and Figure 1b. It is our reading, not the authors’ code.
python def moma_ffn(x, is_img, router, experts): # x: [N, d], is_img: [N] bool y = torch.zeros_like(x) for M in ("text", "image"): idx = (is_img == (M == "image")).nonzero().squeeze(1) # stage 1: fixed rule xm = x[idx] # [b_M, d] g = torch.sigmoid(xm @ router[M]) # [b_M, E_M] stage 2: learned scores k = xm.shape[0] // len(experts[M]) # k_e = b_M * (1 / E_M) for j, ffn in enumerate(experts[M]): top = g[:, j].topk(k).indices # expert choice: the expert picks tokens y[idx[top]] += g[top, j, None] * ffn(xm[top]) # gate * output, put back in place return y # [N, d], same shape as the input # stage 2 of training: the auxiliary router, trained after the main model converges aux = nn.Sequential(nn.Linear(d, d // 2), nn.SiLU(), nn.Linear(d // 2, E_M)) loss = F.binary_cross_entropy_with_logits(aux(h.detach()), picked_by_batch_topk.float()) # inference: the main router gives only the gate; aux decides who is processed chosen = torch.sigmoid(aux(h)) > 0.5
The authors list their own limits. Load balance depends on matching the data mix to the expert mix, and small per-GPU imbalances remain (Section 6). Depth routing hurts in causal use until the auxiliary router gets better (Section 4.6). Their checkpoints are under-trained, so they give perplexities and no generation scores. They test two modalities only, text and image, and say the design extends to more. The best number of upcycling steps may change beyond 1 trillion tokens. The paper also notes that more experts for images gave diminishing returns, and leaves better-suited sparse designs to future work.
MoMa meets four lines of work. Early fusion: Chameleon (arXiv 2405.09818) is the base model. Sparse experts: expert choice (Zhou et al., 2022), and mixture of depths (Raposo et al., 2024, arXiv 2404.02258). Modality experts: VLMo, BEiT-3 and VL-MoE used modality-specific experts for vision-language encoders and masked language models, and MoMa carries the idea into an autoregressive early-fusion language model. Upcycling: Komatsuzaki et al. (2023).
Now press Present or Teach and explain MoMa out loud, from memory: why text and image tokens get separate experts, how expert choice keeps the load even, why it breaks causality and what the auxiliary router does, how MoD saves FLOPs and where it fails, and why a seed stage helps. Then go back to the instrument at the top, switch between the four designs and say which tokens each one sends where.
By Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer and Armen Aghajanyan, Meta FAIR, 2024. This lesson re-teaches the paper in our own words and devices; the numbers are theirs unless marked illustrative.