MoMa: Experts for Each Modality

Picture a web page with a jam recipe and two photos. A normal model spends the same weights on every word and every picture piece. MoMa gives each kind of token its own experts, and it reaches the same loss with 3.7 times less compute.

Learn how a model that reads words and pictures as one stream can stop spending the same weights on both, and how to train it.

Press MoMa 4t4i, then Image only. Watch which experts the picture tokens visit, and how much compute the model needs to match the dense baseline. Then we build, piece by piece, the experts, the routers, the causality fix, the depth skips and the training recipe.

You need a rough idea of what a transformer layer does (it mixes tokens, then transforms each one). We build the rest from zero.

A reading for MIT MMAI, Multimodal AI (Spring 2026), Paul Liang. It goes with the lecture on interactions and fusion models. Paper: MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts (arXiv 2407.21770), by Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer and Armen Aghajanyan, Meta FAIR, 2024. Every idea is re-taught here in our own words and devices.

Route the tokens

The savings are the paper’s, for the 1.4B compute-matched models trained on 1 trillion tokens (abstract and Section 1): MoE 8x 3.0x overall (text 3.0, image 2.8), MoMa 4t4i 3.7x (text 2.6, image 5.2), MoMa + MoD 4.2x (text 3.4, image 5.3), measured by pre-training loss. The bar is 1 divided by the saving. The twelve tokens and their routes are illustrative, and the boxes show the idea of each design, not its real expert counts per layer.

Chapter 0

One Model, Two Kinds of Token

See why a model that reads words and pictures in one stream wastes compute, and what the paper changes about it

Open a web page about strawberry jam. It has a paragraph of recipe text, then a photo of the jar, then more text, then a second photo. A model that reads pages like this must handle both kinds of content in one pass. It must also write them: a caption, then a picture, then another caption.

The usual way to build such a model is to bolt parts together. One part reads images and turns them into numbers. Another part reads text. A third part joins the results. The paper describes it as fusing modality-specific encoders or decoders. It works, but the parts meet late, so the model has little chance to mix what it sees with what it reads.

The other way is early fusion: cut the image into small pieces, turn each piece into a token (a single item from a fixed list, like a word piece), and put those tokens in the same line as the text tokens. One transformer reads the whole line. This is the plan of Chameleon, the Meta model that MoMa builds on (Chapter 1). Early fusion can read and write mixed pages with one set of weights.

Where the cost comes from

A transformer stacks many identical layers. Each layer has two parts. Self-attention lets every token look at the other tokens. The feed-forward network (FFN) then transforms each token on its own, using a big block of weights. Most of a model’s weights and most of its arithmetic sit in the FFNs. (Our transformer lesson builds both parts from zero.)

We count the arithmetic in FLOPs, floating-point operations: one multiply or one add of two decimal numbers. FLOPs are the bill for training. A dense model sends every token through every weight of the FFN. Double the FFN, and you double the bill for every token.

The paper attacks this bill with one question. Does a photo token need the same weights as a word token? Think of how much each kind of token carries. In our picture of it, a photo is made of many small pieces, and neighbours often look alike: a patch of red jam sits next to another patch of red jam. A word seems to carry more in a single token. The authors say the two kinds have distinct information densities and redundancy patterns. Yet a dense model treats them the same.

The sparse answer

A sparse model keeps many weights but uses only some of them for each token. The first sparse trick is the mixture of experts (MoE, Chapter 2): split the FFN into several smaller FFNs, called experts, and let a small router pick which experts each token visits. The model can hold many more weights, while each token pays for only a few.

MoMa adds one idea to this. The paper calls it modality-aware sparsity. A modality is a kind of data, here text or image. MoMa makes one group of experts for text tokens and one group for image tokens. A text token may only visit text experts. An image token may only visit image experts. Attention stays shared, so the two kinds still talk to each other.

The result, for the paper’s 1.4B-parameter model trained on 1 trillion tokens: it reaches the same pre-training loss as a dense model that spends 3.7 times more FLOPs. For image tokens alone the saving is 5.2 times. A plain mixture of experts with 8 shared experts gets 3 times. You played with these numbers in the hero.

First, feel the problem. How many of the tokens in a page are image tokens? A model sees a fixed window of tokens at a time. The paper trains with a window of 4,096 tokens, and one 512 by 512 image takes 1,024 of them (Chapter 1 shows why). Add images to the window and watch the mix change.

Fill the window with pictures

Each block is 64 tokens. Blue blocks are text, purple blocks are image. Drag the number of images. The bar under the grid shows what a dense model spends on each kind of token.

1

The window of 4,096 tokens (sequence length) and the 1,024 tokens per 512 by 512 image are the paper’s (Sections 2.1 and 4.1). How many images a real page holds is our illustration. Four images would leave no room for text, so the slider stops there.

With one image, a quarter of the window is picture. With three, three quarters. A dense FFN treats all 4,096 tokens the same way, so it spends the same FLOPs on a jam photo token as on a word of the recipe. If the two kinds need different things, one shared FFN must learn both jobs with one set of weights.

That is the trade the paper explores. Give each kind its own experts, and each group can specialise. Keep attention shared, and the kinds still mix. And do it without raising the FLOPs per token, so the comparison with the dense model stays fair.

The road

We follow the paper’s own order, one mechanism per chapter. We add the training recipe and the systems work, then read the results. The last chapter is a Field Guide you can keep.

  1. Early fusion: how pictures become tokens in the same line as words.
  2. Mixture of experts: many small FFNs, a router, and the idea of expert choice.
  3. Modality-aware experts: two groups, hierarchical routing, the equation.
  4. Expert choice and causality: why the router peeks at the future, and the auxiliary router that fixes it.
  5. Mixture of depths: letting tokens skip a whole layer.
  6. Upcycling: start with one expert per modality, then split.
  7. Making it run: load balance, stragglers and throughput.
  8. Results and ablations: 3.7 times, the expert counts, and where MoD breaks.
  9. Field Guide: the numbers, a checklist, the core as code, and where to go next.
What is the one change MoMa makes to a mixture-of-experts early-fusion model?

Chapter 1

Early Fusion: Pictures as Tokens

Turn an image into 1,024 whole-number tokens so one transformer can read it in the same line as words

Picture a box of 8,192 crayons. Each crayon has a number on it. You cut a photo into small squares and, for each square, you pick the crayon that looks most like it. Now the photo is a list of crayon numbers. Anyone can read that list aloud, one number at a time, like words in a sentence.

That is the whole trick of image tokenization: change a picture into a list of whole numbers from a fixed set, so a language model can treat it as text. The real crayons are not colours. Each one is a small learned pattern of a patch of the image, but the idea is the same.

What Chameleon does

MoMa starts from Chameleon (Chameleon Team, 2024), a single transformer that models mixed sequences of image tokens and text tokens. The paper (Section 2.1) gives the exact numbers. A learned image tokenizer encodes a 512 by 512 image into 1,024 discrete tokens from a codebook of size 8,192. Text uses a byte-pair-encoding (BPE) tokenizer with a vocabulary of 65,536 tokens. That vocabulary already contains the 8,192 image codes.

So there is one list of 65,536 possible tokens. About one in eight (8,192 of 65,536, which is 12.5%) is an image code, and the rest are text pieces. Any token in a sequence is just a whole number in that list. The model does not need a different input path for images.

Slide the codebook size below. It shows a tiny made-up image cut into 16 by 16 patches. Each patch picks the closest code. Watch how a small codebook makes the picture blocky, and what that costs.

Rebuild a picture from codes

Left: the original, 16 by 16 patches of brightness. Middle: each patch replaced by its nearest code. Right: the error in each patch. Drag the number of codes.

4

Toy picture and toy codes: each patch is one brightness number, and each code is one brightness level. The real tokenizer is a learned neural network that maps each patch to one of 8,192 codes. The 8,192 codes, the 1,024 tokens per 512 by 512 image and the 65,536-token vocabulary are the paper’s.

With 2 codes the picture is two flat tones, and the average patch is far from its original. With 64 codes it is close. The paper uses 8,192 codes and cuts each image into 1,024 patches. We cannot tell the exact patch grid from the text, so the 16 by 16 grid here is our own simplification.

The sequence the model reads

Now put words and a picture in one line. Take the prompt in the paper’s Figure 1: What does the flower of strawberry look like? followed by an image. The paper shows special markers [BOI] and [EOI]. They mark the beginning and the end of an image. Here is the data flow, with shapes. Say the batch holds B sequences of 4,096 tokens, and the hidden size is d.

  1. Token ids. A tensor of whole numbers with shape [B, 4096]. Text pieces first, then [BOI], then 1,024 image codes, then [EOI], then more text.
  2. Embedding. Each id looks up a row in a table of shape [65536, d]. The result has shape [B, 4096, d]. Text and image ids use the same table.
  3. Transformer blocks. Each block keeps the shape [B, 4096, d]. This is where attention mixes the tokens and where the experts will live.
  4. Output head. A final matrix of shape [d, 65536] gives scores for the next token. The result is [B, 4096, 65536].
  5. Loss. The next-token loss (cross-entropy) compares each position’s scores with the id that really comes next. The same loss covers text and image positions.

Because the loss is the same for both kinds, the paper can split it afterwards. It reports an image loss and a text loss: the average of the same loss over image positions and over text positions. These two numbers drive every plot in the paper.

python
# early fusion, the data flow of one training example
ids = tok_text("What does the flower of strawberry look like?")
ids += [BOI] + img_tokenizer(photo) + [EOI]      # 1,024 codes
x = embed[ids]                                   # [T, d]
for block in blocks: x = block(x)             # attention + FFN, shape kept
logits = x @ head                                # [T, 65536]
loss = cross_entropy(logits[:-1], ids[1:])        # next-token, both modalities

Why early fusion, and what it costs

The paper lists four gains from early fusion (Section 2.1). The model learns a shared representation space for images and text. It handles arbitrary sequences of both. The token-based design scales uniformly: Chameleon trained on about 10 trillion mixed-modal tokens. And the whole model trains end to end.

Here is what the first gain means in practice. Ask the model for a recipe with a picture. It writes the steps as text tokens, then emits [BOI], then 1,024 image codes one after another, then [EOI], then the next step. It never switches networks. The same next-token loop produces the words and the picture, which is why the paper can train and compare both with one number per modality.

The cost is the one from Chapter 0. Every image is 1,024 tokens, and every token goes through the full FFN. A page with a few photos is mostly image tokens. The paper says scaling such models to larger sizes “presents significant computational challenges”, and it turns to sparse architectures for relief.

In Chameleon, how does one transformer read both words and a picture?

Chapter 2

Mixture of Experts, From Zero

Split one big feed-forward block into several, let a router choose, and see why letting the experts choose the tokens keeps the load even

A clinic has one doctor who sees every patient: sprains, rashes, colds, check-ups. The queue grows, so the clinic hires eight doctors. Now a receptionist must decide who sees whom. If the receptionist sends everyone to the doctor at the desk nearest the door, that doctor drowns and seven others sit idle. The hard part is not hiring. It is the sorting.

A mixture of experts (MoE) is that clinic inside a neural network. Our mixture-of-experts lesson builds it in full. Here we need only the parts MoMa changes.

The feed-forward block, and its experts

Recall the feed-forward network (FFN). It takes each token’s vector of size d, widens it to a larger size f, applies a non-linear step, and shrinks it back to d. The paper’s FFN uses SwiGLU, a common form with an extra gate, but any FFN works for our story. The shape never changes: a token vector goes in as [d] and comes out as [d].

An expert is one such FFN. Instead of one FFN per layer, an MoE layer holds E of them. The router is a single small matrix Wg of shape [d, E]. Multiply a token’s vector by it and you get E numbers: one affinity score per expert, a measure of how well that expert suits the token.

Table 1 shows the 435M dense model growing to 1.9B parameters when it gets 8 experts per layer. That fits each expert being about as big as the dense FFN it replaces (our reading; the table does not list the expert width). A sparse model then holds far more weights than its dense twin. We pay for them with memory, not with FLOPs per token. Here is why.

Two ways to choose

In token-choice routing (the original MoE), each token looks at its E scores and goes to the top one or two experts. This is the receptionist who lets patients pick a doctor. Popular doctors overflow. The usual fix is an extra penalty in the loss that pushes toward even use, a load-balancing loss.

In expert-choice routing (Zhou et al., 2022), the choice flips. Each expert looks at all the tokens in the batch and takes the top ke by score. Every expert gets exactly ke tokens, so the load is even by construction. A token may be chosen by several experts, or by none. The paper writes it this way (Section 2.2): ke = b · ce, where b is the number of tokens in the batch and ce = 1/E is the capacity factor.

Work it with the paper’s numbers. A batch has b = 4,096 tokens and E = 8 experts. Then ce = 1/8 and ke = 4,096 × 1/8 = 512. Each expert processes 512 tokens. Add up the work: 8 experts × 512 tokens = 4,096 token visits. That equals one token visit per token, which is exactly what one dense FFN would do. So the FLOPs per token match the dense model, and the load is perfectly even.

Who picks whom

Columns are 12 tokens, rows are 4 experts, and a darker cell means a higher score. Outlined cells are the choices. Switch the rule, then press New batch to draw fresh scores.

Toy batch: 12 tokens, 4 experts, random scores drawn in your browser. With expert choice each expert takes its top 12 × 1/4 = 3 tokens, the same rule as ke = b · ce in the paper. Token choice here sends each token to its single best expert.

Try a few batches. Under token choice, the loads wander: one expert often gets five or six tokens while another gets one or none. Under expert choice, every bar is the same height, always three. But look at the top row of counts: some tokens are picked by two experts and some by none. A token picked by no expert skips the FFN, and the residual connection carries it through unchanged.

What an expert layer computes

For a token x, the layer adds up the outputs of the experts that selected it, each scaled by a gate value. Here is the form, with the paper’s sigmoid gate (Chapter 3 gives the full equation).

y
The layer’s output for token x, a vector of size d.
∑j
Add up over the experts j that selected this token. An unselected expert adds nothing.
g(x)j
The gate: how much the router trusts expert j for this token. A number between 0 and 1 in MoMa.
FFNj(x)
Expert j’s own output for x, also a vector of size d.

Two experts pick x. Gates 0.8 and 0.3. Outputs [1, 0] and [0, 2]. So y = 0.8 · [1, 0] + 0.3 · [0, 2] = [0.8, 0.6].

The gate does a second job. Because the output is scaled by g, the router receives a learning signal: if giving an expert more weight lowers the loss, the router’s weights move to raise that gate. This is how a router learns without being told the right answer.

The failure mode, in three parts

Symptom: training is slow, and some experts barely learn. Measurement: count tokens per expert over a batch. A few experts hold most of them. In a distributed run, the busiest expert sets the speed of the whole step. Fix: token choice adds a balancing loss. Expert choice builds the balance in. The paper picks expert choice and notes that this “eliminates needing a separate load-balancing loss term”, which makes optimisation more stable. It also keeps the computation graph static, with the same shapes every step, which matters for training speed.

Expert choice has a cost, and Chapter 4 is about it: an expert picks its top tokens by looking at every token in the batch, including tokens that come later in the text.

A batch has 4,096 tokens and an MoE layer has 8 experts with expert-choice routing and ce = 1/8. How many tokens does each expert take, and why does the layer cost the same FLOPs as a dense FFN?

Chapter 3

Modality-Aware Experts: Route Twice

Send each token first to the group for its modality, then let a learned router pick inside that group

Back to the clinic. The new building has two wings, one for children and one for adults. A guard at the door sends each patient to the right wing by age. This needs no judgement, only a look at the patient. Inside each wing, a receptionist then sends the patient to the right specialist, and that choice is learned from experience.

That is hierarchical routing, and it is the heart of MoMa (mixture of modality-aware experts). The paper (Section 2.2) divides the experts of each MoE layer into expert groups, one per modality. The set of experts is E = {ET1, …, ETm, EI1, …, EIn}: m text experts, written with a T, and n image experts, written with an I.

The two stages

Stage 1: modality-aware routing. Each token goes to the group that matches its modality. A text token may never reach an image expert, and an image token may never reach a text expert. There is no learning here and no score. The token’s modality decides.

Stage 2: intra-modality routing. Inside its group M, a learned router picks the experts. The router is a matrix WgM of shape [d, |EM|], one for each group. Each expert in the group takes the top ke tokens of that modality by score, using expert choice from Chapter 2.

The capacity changes in one small way. The bucket size is ke = bM · ce, where bM is the number of tokens of modality M in the batch and ce = 1/|EM|. Each expert takes an equal share of its own modality’s tokens, not of the whole batch.

The equation, one symbol at a time

Equation 1 of the paper says it in one line. It has two parts: the output y, and the gate g.

y
The layer output for token x, shape [d]. After this the paper adds a residual connection and a normalization in the Swin Transformer style.
∑j=1m
For a text token, add up over the m text experts. Only the experts that selected x contribute.
gT(x)j
The gate of text expert j for token x. It scales that expert’s output.
∑j=1n
For an image token, add up over the n image experts. Text experts are never in this sum.
gM(x)j
The gate in the group M of the token. M is T for text and I for image.
Sigmoid(x · WgM)j
Score x against all |EM| experts of the group, squash each score into the range 0 to 1, take entry j. Chapter 4 explains why sigmoid, not softmax.
selected by Ej
Expert j chose x among its top ke tokens of that modality in the batch. If not, the gate is 0.

The structure of the sum is the whole idea. A text token sees only m experts. An image token sees only n. The two groups do not compete for the same slots.

Where the shapes go

Take a batch with N tokens, flattened to [N, d]. Each step below follows Figure 1b of the paper.

  1. Split. A modality mask divides the rows into text rows [bT, d] and image rows [bI, d], with bT + bI = N.
  2. Score. The text router makes [bT, m] sigmoid scores. The image router makes [bI, n].
  3. Choose. Each expert picks the top ke rows of its own column.
  4. Compute. Each expert runs its FFN on its ke rows, shape [ke, d], and multiplies by the gates.
  5. Restore. The outputs go back to their original rows and are added up. The paper calls this position restoration. The result is [N, d] again, so the next layer sees nothing new.

Now put numbers on it. Let N = 4,096 tokens, with 3,072 text and 1,024 image tokens (illustrative: one image). Take 4 text and 4 image experts. Then ce = 1/4 in both groups. Each text expert takes 3,072 × 1/4 = 768 tokens. Each image expert takes 1,024 × 1/4 = 256 tokens. The total is 4 × 768 + 4 × 256 = 4,096 token visits. Again one visit per token, so the FLOPs match a dense FFN. Compare a mixed pool of 8 experts: each takes 512, and which modality fills the slots is left to the router.

Count the tokens in each expert

A batch of 4,096 tokens. Bars show how many tokens each expert processes. Drag the share of image tokens and switch the layout.

25%

The bucket rule ke = bM · ce with ce = 1/|EM| is the paper’s (Section 2.2). The batch size and the image share are our illustration; the paper says only that it matches the data mix to the expert ratio (Chapter 7). In the mixed pool, how many tokens of each kind land in an expert is up to the router, so those bars are shown as one colour.

Drag the share to 50%. In the MoMa layout, a text expert and an image expert now take the same number of tokens, 512 each. At 25% image, image experts take far fewer tokens than text experts. The image experts are not idle: their buckets are sized to the image tokens’ own share. The total stays at 4,096 token visits whatever the share, so the FLOPs per token never change.

Why the wall helps, and what stays shared

The paper names three benefits. Improved efficiency: a token never spends compute on an expert built for another modality. Enhanced specialisation: a group sees only its own kind of data and can learn finer features for it. Maintained cross-modal integration: attention layers and the residual stream are not split, so text tokens still attend to image tokens and the other way round.

The extreme case is 1t1i: one text expert and one image expert, so ce = 1 and every token is processed by its modality’s FFN. There is no learned routing at all. It is simply modality untying: two separate FFNs, picked by a rule. The paper uses it as a baseline and as the seed for upcycling (Chapter 6).

An image token arrives at a MoMa layer with 4 text experts and 4 image experts. Which description is right?

Chapter 4

Expert Choice Breaks Causality, and the Router That Fixes It

Find out how picking the top tokens in a batch lets the future leak into the past, and how a small second router closes the leak

You write a story one word at a time. Before you type each word, a helper must decide: should this word get the expensive specialist or not? A fair helper looks only at the words so far. Now imagine a helper who has the finished story on the desk and picks the 30% of words that look most important. The helper’s choice for word 5 now depends on words 6 to 100. When you write live, those words do not exist.

That is the problem with expert choice (Chapter 2) in a language model. A language model predicts the next token from the tokens before it. This rule is called causality: the output at position t may depend only on positions up to t.

Where the leak is

Expert choice says: each expert takes the top ke tokens in the batch. To know whether token t is in the top ke, you must compare its score with every other token’s score. That includes tokens after t. So whether token t is processed, and by how much, depends on the future.

The paper says it plainly (Section 2.2): expert-choice routing “compromises causality in auto-regressive language modeling, as each expert selects the top tokens to process in a batch by comparing token scores across all tokens.” In training it is a leak: the model can use future information, and the training loss comes out too good. At inference it is worse. When a model writes one token at a time, there is no batch of finished text to rank.

The paper is open about this in Section 4.1: its training loss calculation compromises causality, because token selection considers the top proportion of a batch, including future tokens. It therefore checks the final models again with a causal method (Section 4.6, Chapter 8 here).

Two fixes

Fix 1: a sigmoid score. The router scores a token with a sigmoid, a curve that squashes any number into the range 0 to 1. With a sigmoid, each token’s affinity for an expert is computed from that token alone. It does not depend on any other token. This does not remove the top-k step, but it means the score itself is clean and can be predicted from the token’s own hidden state.

Fix 2: auxiliary routers. Train a second, small router to predict, from the token alone, whether the expert would have picked it. The paper (Section 2.4) defines it for each MoE layer as a two-layer network:

gMaux(x)
A vector with one probability per expert in the group: how likely each expert is to select token x. Shape [|EM|].
WMa1
First weight matrix, shape [d, d/2]. It shrinks the token vector to half its size.
SiLU
A smooth non-linear step between the two matrices: SiLU(z) = z times sigmoid(z). It lets the network learn more than a straight line.
WMa2
Second weight matrix, shape [d/2, |EM|]. It gives one score for each expert.
Sigmoid
Squashes each score into 0 to 1 so it reads as a probability.

The auxiliary router uses only x, the token’s own hidden state. It cannot see other tokens, so it cannot leak. For depth routing (Chapter 5) the paper uses the same recipe with output size 1.

Training it, and using it

The paper trains in two stages. First it trains the main model to convergence, with ordinary batch top-k routing. Then it trains the auxiliary routers alone, with binary cross-entropy (a loss for yes-or-no labels). The labels are the “ground-truth top-k routing assignments computed over an entire batch”: for each token and expert, did the batch rule pick it? The main model stays fixed.

At inference, the main router keeps one job: it produces the gate values g. The auxiliary router decides who is processed: a token goes to an expert when its auxiliary score is above 0.5. The appendix (Table 6) gives the training recipe: peak learning rate 1e-4 decaying linearly to 1e-6, 1,000 or 2,000 warm-up steps, 5,000 to 10,000 steps, and 4,194,304 tokens per batch at length 4,096.

Try it below. A story has nine written tokens and one still to come. The expert takes the top 3 by main-router score.

Let the future move the past

Ten tokens, left to right. Bars are scores for one expert. Drag the score of token 10, which has not been written yet, and watch whether an earlier token changes its fate.

0.20

Toy scores, made up to show the leak; the top-k rule, the sigmoid scores and the 0.5 threshold are the paper’s (Sections 2.2 and 2.4). The auxiliary scores here are drawn so they agree with the batch rule when token 10 is low. A real auxiliary router is only an approximation, and Chapter 8 shows what happens when it is wrong.

In batch mode, push token 10 above 0.66. Token 8, which was chosen, drops out, even though nothing about token 8 changed. The future moved the past. In auxiliary mode you can slide token 10 anywhere and every earlier token keeps its fate. The cost: the number of chosen tokens is no longer exactly 3. Token 10 simply joins when its own score passes 0.5, so the load is balanced only on average.

Symptom, measurement, fix

Symptom: a model that trains well and then answers worse when used as a language model. Measurement: compare training loss, which sees the batch, with held-out perplexity measured through the auxiliary routers. Fix: sigmoid scores plus a second-stage auxiliary router. For mixture of experts this works: Table 3 shows the MoE models keeping their ranking at inference.

Why does the paper add an auxiliary router for inference?

Chapter 5

Mixture of Depths: Letting Tokens Skip a Layer

Add a second kind of sparsity along the depth of the network, so only the top-scoring quarter of tokens enter some layers

A commuter train has a local line and an express line. A passenger who needs the next stop gets off early. A passenger going far stays on and passes every station. Not every passenger needs every stop.

Tokens are like that. In the sentence “The cat sat on the mat”, the word “the” is easy to handle and needs little thought. A rare word or a hard part of a picture may need every layer. A dense model gives every token the same depth. Mixture of depths (MoD, Raposo et al., 2024) lets some tokens skip layers.

Width and depth

So far we added sparsity in width: many experts side by side inside one layer. MoMa adds sparsity in depth: some tokens go around a whole layer. The paper (Section 2.3) says they apply MoD “prior to any mixture-of-experts (MoE) routing, ensuring it is applied to the full batch before modality split”. The depth decision comes first and covers both kinds of token. Then the survivors go through the modality split of Chapter 3.

A token that skips a layer is not deleted. The transformer adds each layer’s output to the token’s vector through a residual connection, a path that carries the old vector forward. If the layer does nothing for a token, the old vector just moves on to the next layer unchanged.

The depth router

The depth router is a single column of weights. For each MoD layer, the paper computes the token-to-layer affinity score with a projection Ŵg of shape [d, 1] and a sigmoid:

ĝ(x)
One number between 0 and 1 for each token: how much this token wants to enter this layer. This is Equation 2 of the paper.
Sigmoid
Squashes the raw score into the range 0 to 1, and scores each token on its own.
Ŵg
The depth router: one weight column of shape [d, 1]. The vector x has size d, so x times Ŵg is a single number.

Selection works like expert choice. The layer has a fixed capacity: it admits the kd top-scoring tokens of the batch, where kd = b · cd. Here b is the number of tokens in the batch and cd is the capacity factor between 0 and 1, “chosen empirically”. The paper sets cd = 25% and puts MoD in alternating layers, starting with layer 0.

So half the layers are ordinary and half are MoD layers that admit a quarter of the tokens. Because the choice is the top-scoring tokens of a batch, it has the same causality problem as expert choice. The paper adds an auxiliary router for each MoD layer, of the same form as in Chapter 4 with output size 1.

Keeping the bill fair

A MoD layer does only a quarter of the work, so a MoD model with the same number of layers is cheaper than the dense one. To keep the comparison fair, the paper adds layers: “we first fixed the MoD layer interval and capacity factor cd, then adjust the total number of transformer layers” so the FLOPs per token match the dense model. The hidden size stays the same.

Check it for the 435M model (Table 1). The dense model has 24 layers. The MoD model has 40. Of those 40, 20 are ordinary and 20 are MoD layers at 25%. Count the layer-work: 20 × 1 + 20 × 0.25 = 25, close to the dense 24. The match is only approximate, because the paper tunes the layer count for the real FLOPs, which include more than the layer count. The same pair gives 14 layers for the 90M model, where the dense one has 8.

Send tokens down the tower

Eight tokens (rows) pass six layers (columns). Layers 0, 2 and 4 are depth-routed. A filled cell means the token entered the layer, a hollow one means it took the bypass. Drag the capacity.

25%

Toy tower: 8 tokens, 6 layers, with a fixed random score for each token in each depth-routed layer. The alternating pattern from layer 0 and the top-k rule are the paper’s (Sections 2.3 and 4.1), whose real default is cd = 25%. The FLOPs figure counts a depth-routed layer as cd of an ordinary layer, ignoring everything else.

At 25%, only 2 of 8 tokens enter each depth-routed layer. The stack as a whole costs 3.75 layers of work, against 6 for the dense stack. To spend the same as the dense one, you could add layers: about 9.6, using 12/(1 + cd). The paper’s real jump, 24 to 40 layers, is a ratio of 1.67; our toy formula gives 1.6, close enough to show the idea.

Where does the saving show up? In attention as well as in the FFN. The paper used the “staged” version, which skips both for tokens that do not enter. The other version in the MoD paper always computes full attention, so it saves less. The paper chose the staged version because it “also introduces sparsity in attention”.

Why this chapter ends with a warning

Depth routing is the part of the paper that works best in training and worst in use. Adding MoD to MoMa lowers the training loss (Chapter 8: 4.2 times against 3.7 times). But the final model, tested with its auxiliary routers, scores worse than the plain MoMa model. The reason is simple. A token skipped by mistake has no other path in that layer. An MoE token sent to the wrong expert can still be handled by another. Chapter 8 measures how small the router’s error rate must be.

The 435M model has 24 dense layers. With MoD at cd = 25% in alternating layers, the paper uses 40 layers. Why more layers?

Chapter 6

Upcycling: Start With One Expert, Then Split

Train a small seed model first, copy its experts, and let the routers learn on a space that already means something

A city plans four fire stations. The planner must draw four districts, one for each station. If the planner draws them before anyone knows where people live, the lines are guesses. If the planner waits one year, counts the calls, and then draws the districts, the lines fit the city.

A router has the same job. It cuts the space of token vectors into regions, one for each expert. On day one of training, token vectors are close to random, so the regions mean little. The router learns a poor split, and the experts then learn from a poor split. The paper (Section 2.5) says that a router’s representation space “is sub-optimal in the early stages of model training, leading to a sub-optimally trained routing function”.

The recipe

Upcycling (Komatsuzaki et al., 2023) starts with a model that has no routing and grows it into a mixture of experts. The paper adapts it to MoMa, in four steps.

  1. Stage 1: seed. Train a model with one FFN expert per modality, the 1t1i design from Chapter 3. It has no learned router. The paper trains it for 10,000 steps, or 20,000.
  2. Split. Turn each modality’s FFN into an expert-choice MoE module. Every new expert starts as an exact copy of the stage-1 expert. A 4t4i model gets four copies of the text FFN and four copies of the image FFN.
  3. Reset the schedule, keep the data. The learning-rate schedule starts again. The data loader does not go back. It keeps its place from stage 1, so stage 2 reads fresh text and images, not data the seed already saw.
  4. Add noise to the router. The router now picks with a noisy score so that copies can drift apart and specialise. The next section explains it.

The scale is small. For the 1.4B models, Table 5 lists 120,000 steps at 8,388,608 tokens per step, about 1 trillion tokens in all. Ten thousand steps is therefore about 84 billion tokens, under a tenth of the budget.

Count it with an example. Say the full run is 120,000 steps and the seed lasts 10,000. Stage 2 then runs the remaining 110,000 steps, with fresh data and a fresh learning-rate schedule. The seed’s 10,000 steps are not free: they use the same hardware as any other steps. That is why the paper shifts every upcycled curve by the seed’s length before it compares them with the scratch run.

The noise: Gumbel-sigmoid

All the copies begin identical. The router must find out how to tell them apart. The paper helps it with noise, Equation 5:

Gumbel-Sigmoid(x)
The noisy gate score the router uses while training.
x
The router’s raw score for a token and an expert, before the sigmoid.
G′, G″
Two independent random numbers from the Gumbel distribution. Their difference is a bell-shaped noise with fatter tails, called logistic noise.

Why this noise in particular? There is a neat fact. If L is logistic noise, then the chance that x + L is above zero equals Sigmoid(x). Work it with x = 1. Sigmoid(1) = 0.731. So a noisy score x + L lands above zero in 73.1% of draws. The noisy score behaves like a coin that comes up heads with the probability the router believes in, yet it stays smooth, so gradients can flow through it. The paper describes it as letting the router “differentiably sample experts”. Our reading: with random draws, a token sometimes lands at an expert that was not its first choice, and the copies get a chance to move apart.

What the experiment shows

Section 4.4 compares a 2.3B model with MoD, 4 text experts and 4 image experts (mod_moe_4t4i). One copy trains from scratch. Two start from a mod_moe_1t1i seed, trained for 10,000 and 20,000 steps. All curves are shifted by the steps the seed used, so the stage-1 cost counts against them. Upcycling wins: 1.2 times in FLOPs after 10,000 seed steps and 1.16 times after 20,000. The gap between upcycled and scratch models “widens throughout training”.

Choose when to split

The top bar is the training run. Teal marks the 1t1i seed, warm marks the 4t4i model. Pick how long the seed trains, and read the gain below.

The gains (1.2x at 10k steps, 1.16x at 20k steps, against training from scratch) are the paper’s (Section 4.4 and Figure 6), measured on a 2.3B model with MoD and 4t4i. The learning-rate sketch is schematic: the paper says only that the scheduler is reset. The paper recommends 10k steps and guesses that the best number may change beyond 1 trillion tokens.

Why not seed longer? The paper suggests that a sweet spot likely exists. Undertraining the seed gives the routers little to work with. Overtraining it “impedes future specialization”, because the copies already know too much alike. The 10k seed beat the 20k seed, but both beat scratch.

Symptom, measurement, fix

Symptom: a mixture of experts trained from scratch lags behind where its FLOPs say it should be. Measurement: compare its loss curve with an upcycled run at equal total FLOPs. Fix: a short dense seed, then the split. The cost is a longer recipe with two stages, and a new knob, the seed length.

Why does MoMa train a 1-expert-per-modality seed first, then split it?

Chapter 7

Making It Run: Load Balance, Stragglers and Throughput

See why fewer FLOPs do not always mean faster training, and how matching the data mix to the expert mix keeps many GPUs in step

A kitchen has 256 cooks. Each one gets a tray of orders for the evening rush. Every few minutes, all cooks must pass their plates to the next station together. If one cook has a tray with twice as many orders, 255 cooks stand idle and wait. The kitchen is only as fast as its slowest cook.

Training a large model on many GPUs works the same way. The paper trains with Fully Sharded Data Parallel (FSDP, Zhao et al., 2023). In FSDP each GPU keeps only a slice (a shard) of the weights. Just before a layer runs, every GPU collects the full weights of that layer from the others, which is called prefetching. After the layer runs, the GPUs drop the borrowed weights and share the gradients. The steps are in lockstep, so all GPUs wait for the slowest.

Where MoMa creates a straggler

A dense model gives every GPU the same work. MoMa does not. The paper (Section 3.1) explains: “the ratio of text to image tokens can vary significantly across different GPUs and iterations.” An uneven mix creates a straggler, a GPU that finishes late. It delays the prefetch of weights for later layers and the release of gradients from earlier ones. In the paper’s words, it “effectively bounds the training latency by the maximum time required to process text and image experts across all GPUs”.

Our reading of why this is MoMa’s problem: its text and image experts are separate pools, so a GPU with many photos has a heavy image phase and a light text phase.

The fix: match the data mix to the expert mix

The paper’s fix is plain. It builds “a balanced data mix that aligns the text-to-image data ratio with the expert ratio on each GPU”. With 4 text and 4 image experts, the mix is set so that each GPU’s batch has the matching share. It keeps the load balanced in expectation, not exactly. The paper names the alternative, moving tokens between GPUs at every FFN layer, and rejects it for now because of the extra communication.

Try it. Each bar below is one GPU’s step. Blue is the text-expert phase, purple the image-expert phase. We model the phases as lockstep: all GPUs finish the text phase together, then the image phase, so the step takes (slowest text phase) + (slowest image phase).

Wait for the slowest GPU

Eight GPUs, each with a batch of the same size. Choose how the batches are mixed, then press New step a few times. The dashed line is where the step ends.

12 points

Illustrative model, not the paper’s measurement. Each GPU’s time in a phase is proportional to its tokens of that kind; the target is 25% image; the spread is how far an image share may stray from it; a matched mix keeps a quarter of that spread. The lockstep rule follows the paper’s remark that latency is bounded by the maximum time to process text and image experts across GPUs.

With a wide spread, the dashed line sits well beyond the average GPU. Every bar shorter than the line is idle time. Matching the mix does not make the bars identical, but it pulls them close. The paper admits the limit in Section 6: matching the ratio holds “in expectation”, and a batch can still deviate on a single GPU in a single step.

Running experts, one modality after another

The paper tried three ways to run the experts (Section 3.2). First, make the experts the same size for both modalities, forbid cross-modal routing, and run everything at once. This needs every expert to get the same number of tokens. Second, use block-sparse kernels (Gale et al., 2023), which allow uneven counts. Third, run the modality groups one after the other. While the text experts compute, the GPUs can already prefetch the weights of the image experts. This eases memory pressure and assumes nothing about balance.

The paper picks the third: “a clean and flexible choice for our experimental environment at the current scale”. Each GPU has so many tokens that running several matrix multiplies in turn still keeps the hardware busy. Smaller batches might need the other approaches.

The paper also lists other speed-ups. It quantises gradient communication, fuses GPU kernels automatically, and compiles with torch.compile. A MoMa-specific trick reuses the indices of each modality’s tokens across layers, so the CPU and GPU synchronise less. The paper notes that this trick does not work with MoD, so it was left out of the final comparison to keep it fair.

What it cost: Table 2

The paper measured throughput on 256 A100 GPUs, with sequence length 4,096 and 6 sequences per GPU, for models whose FLOPs match the 435M dense model. WPS is words per second per GPU. The paper does not define it further, so we read it as tokens per second.

ConfigurationWPS per GPUChange vs dense
Dense31,970baseline
MoMa 8X (mixed experts)28,990−9%
MoMa 1t1i30,196−6%
MoMa 4t4i26,424−17%
MoD MoMa 1t1i25,367−21%
MoD MoMa 4t4i22,114−31%

Read the rows in order. Sparsity is not free. Plain expert-choice MoE loses 9%, probably from routing work and from synchronising the gradients of all experts. MoMa 1t1i loses 6%, mostly from computing the token indices for each modality, which could be done once and reused. MoMa 4t4i loses 17%: about 11 more points from adding experts, similar to the 9% step from dense to MoE. MoD adds another 14 to 15 points, because of its extra router and because depth routing breaks the planned mix: the number of active tokens of each modality changes at every MoD layer. The paper suggests forcing the MoD routers to admit tokens in the planned ratio, and leaves the effect on quality to future work.

So the headline gain has a price tag. MoMa 4t4i gets 3.7 times in FLOPs and, in the paper’s own words, pays a 17% throughput reduction. A rough net, our arithmetic and not a measurement, is 3.7 × 0.83, about 3.1 times in time. It mixes two model sizes (the 3.7 times is at 1.4B, Table 2 at 435M), so treat it as a sanity check only.

Why does MoMa build a training data mix whose text-to-image ratio matches the ratio of text to image experts?

Chapter 8

Results and Ablations: What 3.7 Times Really Means

Read the paper’s speed-up factor, its expert-count ablations and its held-out tests, and see where depth routing breaks

Two runners race. One is the dense baseline, the other the sparse model. Both get the same fuel, and we check who is further along. That is how most papers compare models. MoMa asks a sharper question. How much fuel does the dense runner need to reach the same spot? If the answer is 3.7 times more, the sparse runner is 3.7 times as efficient.

How the paper measures a saving

The paper defines the pre-training speed-up factor η (Section 4.2). A sparse model with FLOPs per token matched to a dense model reaches the dense model’s final pre-training loss with only 1/η of the FLOPs. So η = 3.7 means a dense model would need 3.7 times the compute. The paper reports η overall, for text tokens alone and for image tokens alone, since the loss splits by modality (Chapter 1).

The paper uses the training loss as its yardstick, since 1 trillion tokens is less than one pass over its data, so training loss behaves like validation loss. We saw the caveat in Chapter 4: with expert choice the training loss peeks at the future. That is why the paper later re-checks with causal inference.

The setup: three dense sizes (90M, 435M and 1.4B parameters) and compute-matched sparse variants, all trained for 1 trillion tokens with sequence length 4,096. The learning rate peaks at 1e-4 after 4,000 warm-up steps and decays linearly to 1% of the peak. Every layer is an MoE layer, and MoD sits in alternating layers at 25% capacity.

The headline: Figure 3 and Figure 4

At the 1.4B size the paper reports the following (abstract and Section 1).

Three lessons sit in those numbers. First, untying helps image tokens most: the 1t1i model, with only one expert per modality and no learned routing, nearly matches the 8-expert mixed MoE on image loss, while its text loss stays higher. Second, adding experts per group (4t4i) beats the mixed pool at every size, with a large margin on images, but with a slight text regression. The authors read this as a sign that text may benefit from more experts. Third, MoD helps. The mod_moe_1t1i model does as well as or better than moe_4t4i. Stacking MoD on top of 4t4i yields smaller gains, with the text gain larger than the image gain.

Use the device below to flip between the training view and the held-out view, which we reach next.

Compare the designs two ways

First view: how many times fewer FLOPs than dense, from training loss (longer is better). Second view: perplexity on held-out text and pictures, using causal inference (shorter is better). Choose a view and a kind of token.

Training view: the 1.4B compute-matched models at 1 trillion tokens (abstract and Section 1); the paper gives no η for 1t1i. Test view: Table 3, held-out OBELICS interleaved data (overall, text and image tokens), lower is better. The MoMa plus MoD model is the 2.3B-parameter one that is compute-matched to the 1.4B dense model.

Flip to the held-out view. For all tokens, the ranking among the earlier models holds: 4t4i is best, 8x next, 1t1i, then dense. Switch to text only and 8x takes the lead. But the MoD model, which led the training view, now sits behind plain 4t4i. We come to that below.

How many experts, and how to split them

Section 4.3 varies the expert counts at the 435M size. With the same count in each group, training loss improves as experts grow. The two modalities scale differently. Text keeps improving with each doubling, while image shows diminishing returns going from 2 to 4 experts.

With a fixed total of 8 experts, the paper tries 7t1i, 6t2i and 4t4i. More experts help a modality, but more than 4 text experts has diminishing returns. The three overall losses end up close. The 6t2i model has a slightly lower average loss, yet the paper picks 4t4i, because a balanced allocation fits its data mix better for load balancing (Chapter 7). This is a design choice made for the systems, not only the loss.

The held-out test

Section 4.6 trains the auxiliary routers and tests the 1.4B models on held-out data: perplexity on OBELICS (interleaved web documents) and Shutterstock, caption perplexity on COCO and Flickr30k, perplexity of answers on VQA-v2, and eight text commonsense tasks. They report perplexity because their checkpoints are “under-trained”, so generation metrics would not be meaningful.

Model (1.4B-matched)OBELICS overallCOCO text to imageCOCO image to textCommonsense avg (%)
Dense44.6458.321.752.4
MoMa 1t1i41.5416.721.953.3
MoE 8x40.4426.518.954.8
MoMa 4t4i39.0392.520.954.2
MoD MoMa 4t4i (2.3B)41.4428.122.9not reported

Perplexity: lower is better. Commonsense: higher is better. From Tables 3 and 4 of the paper, copied exactly. MoMa 4t4i has the best overall interleaved result, and the best score on every text-to-image caption metric. The 8x mixed model is best at describing an image in text (image to text) and on the text-only commonsense average. So a MoMa model gives up a little on pure text to win a lot on images.

Where depth routing breaks

Symptom: the 2.3B MoD MoMa 4t4i model has the best training loss but, in Table 3, trails the smaller 1.4B MoMa 4t4i on every metric (OBELICS overall 41.4 against 39.0). Measurement: the authors take a smaller 635M MoD MoMa 4t4i model. With the training router and a batch top-k, it beats the 435M MoMa 4t4i, as it should. Then they add random noise to the router’s selection. Once the noise ratio σ passes 0.5%, the MoD model falls behind. An auxiliary router cannot reach so small an error. Fix: none yet. The paper calls for “a robust approach for practical MoD applications”.

The paper gives a reason why MoE is tougher than MoD. A token sent to the wrong expert can still be processed by another expert. In MoD each layer has a single switch, so a wrong skip “cannot be remedied”. The paper calls this likely, not proven.

The MoD plus MoMa model has the best training loss but trails plain MoMa 4t4i on held-out tests. What does the paper identify as the main cause?

Chapter 9

Field Guide: MoMa on One Page

Keep the numbers, the terms, the checklist and the core code, and see where the paper sits in the field

MoMa takes an early-fusion model that reads text and image tokens in one stream and makes its feed-forward layers sparse in a way that respects the two kinds of token. Experts are split into a text group and an image group. A fixed rule picks the group, and a learned expert-choice router picks tokens inside it. A second router keeps inference causal. A depth router can skip whole layers for 75% of tokens. A short seed stage and noisy routers help the experts split. Matching the data mix to the expert mix keeps training balanced. The reward at 1.4B and 1 trillion tokens: 3.7 times fewer FLOPs for the same loss.

The numbers

NumberWhat it isWhere
1,024 / 8,192 / 65,536Tokens per 512 by 512 image, image codebook size, text vocabulary (which holds the codes)Section 2.1
4,096Sequence length in trainingSection 4.1
3.7x (2.6 text, 5.2 image)FLOPs saving of MoMa 4t4i at 1.4B, 1T tokensAbstract, Section 1
3.0x (3.0 text, 2.8 image)Saving of the mixed 8-expert MoEAbstract, Section 1
4.2x (3.4 text, 5.3 image)Saving of MoMa plus MoD, by training loss onlyAbstract, Section 1
1/EExpert capacity factor ce, so ke = bM/EM tokens per expertSection 2.2
25%, alternate layersMoD capacity cd and layer pattern, from layer 0Section 4.1
> 0.5Auxiliary-router threshold at inferenceSection 2.4
10k steps: 1.2x, 20k: 1.16xUpcycling gain over scratch (2.3B MoD 4t4i)Section 4.4
−9, −6, −17, −21, −31%Throughput of MoE 8X, 1t1i, 4t4i, MoD 1t1i, MoD 4t4i vs denseTable 2
σ > 0.5%Router noise at which the MoD model loses to MoMa without MoDSection 4.6

The terms

TermIn one line
Early fusionText and image tokens in one line, read by one transformer from the first layer.
ExpertOne feed-forward network out of several in a layer.
Expert choiceEach expert takes its top-k tokens of the batch, so the load is even.
Modality-aware sparsitySeparate expert groups for text and for image tokens.
4t4i, 1t1i4 text and 4 image experts; 1 text and 1 image expert (untied FFNs, no routing).
Auxiliary routerSmall network that predicts from one token whether a router would pick it, used at inference.
Mixture of depthsA router lets only the top cd of tokens enter a layer; the rest take the residual path.
UpcyclingStart with one expert per modality, copy it into many, reset the schedule, keep the data.
StragglerThe slowest GPU, which the whole step waits for.
ηSpeed-up: the sparse model matches dense loss with 1/η of the FLOPs.

The core as code

A sketch of one MoMa feed-forward layer, with expert choice. It follows the paper’s Equation 1 and Figure 1b. It is our reading, not the authors’ code.

python
def moma_ffn(x, is_img, router, experts):        # x: [N, d], is_img: [N] bool
    y = torch.zeros_like(x)
    for M in ("text", "image"):
        idx = (is_img == (M == "image")).nonzero().squeeze(1)   # stage 1: fixed rule
        xm = x[idx]                                    # [b_M, d]
        g = torch.sigmoid(xm @ router[M])              # [b_M, E_M]   stage 2: learned scores
        k = xm.shape[0] // len(experts[M])             # k_e = b_M * (1 / E_M)
        for j, ffn in enumerate(experts[M]):
            top = g[:, j].topk(k).indices              # expert choice: the expert picks tokens
            y[idx[top]] += g[top, j, None] * ffn(xm[top])   # gate * output, put back in place
    return y                                       # [N, d], same shape as the input

# stage 2 of training: the auxiliary router, trained after the main model converges
aux = nn.Sequential(nn.Linear(d, d // 2), nn.SiLU(), nn.Linear(d // 2, E_M))
loss = F.binary_cross_entropy_with_logits(aux(h.detach()), picked_by_batch_topk.float())
# inference: the main router gives only the gate; aux decides who is processed
chosen = torch.sigmoid(aux(h)) > 0.5

A checklist for using the idea

What the paper leaves open

The authors list their own limits. Load balance depends on matching the data mix to the expert mix, and small per-GPU imbalances remain (Section 6). Depth routing hurts in causal use until the auxiliary router gets better (Section 4.6). Their checkpoints are under-trained, so they give perplexities and no generation scores. They test two modalities only, text and image, and say the design extends to more. The best number of upcycling steps may change beyond 1 trillion tokens. The paper also notes that more experts for images gave diminishing returns, and leaves better-suited sparse designs to future work.

Where it sits, and where to go next

MoMa meets four lines of work. Early fusion: Chameleon (arXiv 2405.09818) is the base model. Sparse experts: expert choice (Zhou et al., 2022), and mixture of depths (Raposo et al., 2024, arXiv 2404.02258). Modality experts: VLMo, BEiT-3 and VL-MoE used modality-specific experts for vision-language encoders and masked language models, and MoMa carries the idea into an autoregressive early-fusion language model. Upcycling: Komatsuzaki et al. (2023).

The takeaway. Not every token deserves the same weights. Give each modality its own experts, let tokens choose, or be chosen, in a way the model can still run causally, and spend a short seed stage to teach the routers. The saving is real in training. What you can keep at inference depends on how well a small router can imitate a batch-wide choice.

Now press Present or Teach and explain MoMa out loud, from memory: why text and image tokens get separate experts, how expert choice keeps the load even, why it breaks causality and what the auxiliary router does, how MoD saves FLOPs and where it fails, and why a seed stage helps. Then go back to the instrument at the top, switch between the four designs and say which tokens each one sends where.

MMAI Reading: MoMa
Read the paper (arXiv 2407.21770) · The MMAI course schedule · The lecture · Back to Gleams

By Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer and Armen Aghajanyan, Meta FAIR, 2024. This lesson re-teaches the paper in our own words and devices; the numbers are theirs unless marked illustrative.