Rebalancing Multimodal Training by Masking Subnetworks

A model that hears and sees often learns only to hear. AMSS mutes part of the loud sense’s learning, and lifts Kinetics-Sound accuracy from 64.55% to 72.25%.

Learn why the strong sense of a multimodal model starves the weak one during training, and how AMSS decides which weights of each sense may learn on each step.

Pick a training method, then press Replay. Watch the audio branch race ahead and the video branch stall under plain training. Then we build, piece by piece, the whole method: the mask, how many weights, which weights, and the fix that keeps the shortcut honest.

You need a rough idea of a neural network (a stack of adjustable numbers that turns an input into a guess) and what a gradient is (the direction that makes the error smaller). We build the rest from zero.

A reading for MIT MMAI, Multimodal AI (Spring 2026, Paul Liang), part of the lecture on large multimodal models. Paper: Yang Yang, Hongpeng Pan, Qing-Yuan Jiang, Yi Xu and Jinghui Tang, “Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks”, arXiv:2404.08347 (2024). Every idea is re-taught here in our own words and devices; the numbers are the paper’s.

Train two senses together

The curves are a toy of our own: two branches share one loss, audio learns four times faster than video, and each step is scaled by the branch’s update ratio. The ratios are the paper’s averages over the first 10 epochs (Table 8: audio 0.3305 and video 0.6695 at tau = 1; audio 0.2479 and video 0.7521 at tau = 0.25). The accuracies are the paper’s Kinetics-Sound results (Table 2: Concat 64.55, AMSS 68.96, AMSS+ 72.25); the middle one is the closest named method to a mild mask.

Chapter 0

The Sense That Stopped Learning

See why a model that hears and sees can score no better than a model that only hears, and meet the one idea that fixes it

Close your eyes and listen to a guitar. You know it is a guitar. Now mute the sound and watch a video of someone strumming. You know it is a guitar again. Either sense alone is enough, and using both should make you surer.

Now ask a computer to do the same. You give a neural network two inputs for each clip: the sound and the picture. You tell it to learn from both. You expect it to beat a network that only hears. Often it does not.

A few words first

A modality is one kind of input: sound, pictures, words, depth maps. Multimodal learning trains one model on several modalities at once. The hope is simple: each modality carries facts the others lack, so together they should win.

The paper opens with a puzzle that breaks this hope. Researchers measured it, and the numbers are in the paper’s Table 2. Look at the device below. It shows four datasets. For each one it shows what each modality scores alone, what plain fusion scores when the model learns from both, and what the paper’s method scores.

Compare one sense, two senses, and the fix

Pick a dataset. Each bar is a test accuracy from the paper’s Table 2. Fusion should beat both single senses. Check whether it does.

All numbers are test accuracy from Table 2 of the paper (Concat is plain feature concatenation fusion). The bar axis starts at 40%, not 0, so that small gaps stay visible. In Kinetics-Sound the paper calls audio the dominant modality, although video scores slightly higher alone.

Read the four panels. On CREMA-D (a dataset of acted emotions), the best single modality scores 63.17% and plain fusion scores 63.31%. Adding the whole second modality bought 0.14 points. On Twitter-15, fusion scores less than the text alone: 70.11% against 73.67%.

The name of the puzzle: modality imbalance

The paper calls this modality imbalance. During joint training one modality, the dominant one, learns fast and takes over the work. The other, the non-dominant one, is starved. It learns slowly, and the final model barely uses it. The paper quotes earlier work for the cause: a model is “greedy” and takes the easiest route to a low error. Chapter 1 shows exactly how that happens.

Think of two students on a group project. One student is quick and finishes the work. The other student has nothing left to do, so never learns the material. On the exam, the group score is the quick student’s score. Imbalance is that group, inside one network.

The claim, as a trade

Earlier fixes slow the quick student down. They shrink the whole branch’s learning signal by one number, for every weight. The paper says this is too blunt, because not every weight matters equally. Its method, AMSS (Adaptively Mask Subnetworks considering modal Significance), picks which weights of each branch may learn on each step. It keeps more weights learning in the quiet branch and fewer in the loud one. The paper’s claim is a trade: AMSS+ lifts Kinetics-Sound accuracy from 64.55% to 72.25% where plain fusion could not, because it lets the weak modality keep learning.

Three questions make the method. What is a mask on learning? How many weights should each modality update? And which ones? The next chapters answer them in that order, then add the correction that makes the idea sound in theory.

The road

  1. Why one sense wins: how a shared loss starves the weak branch. This is the hero’s curves.
  2. The old fix: scale each branch’s learning by one number, and why that is blunt.
  3. Update only some weights: the mask, and the paper’s first experiment.
  4. How many weights: modal significance and the update ratio. This is the tau knob.
  5. Which weights: Fisher information and sampling.
  6. Keeping the shortcut honest: bias, the unbiased AMSS+, and the convergence result.
  7. What the tables show: Tables 2 to 6 and the imbalance curves.
  8. Ablations and limits: what each part buys, and where the method stops helping.
  9. Field guide: numbers, a checklist, the core as code, and where to go next.
On Twitter-15, plain fusion of text and image scores 70.11% and the text alone scores 73.67%. What does the paper say is happening?

Chapter 1

Why One Sense Wins

Follow one shared loss through two branches and watch the loud branch use up the learning signal before the quiet branch gets any

Two students share one grade. The quick student studies first and the group grade jumps to an A. The slow student looks at the grade, sees an A, and stops worrying. Nobody pushes the slow student any more. That small story is the whole mechanism of this chapter. We now build it with real parts.

The parts of a multimodal classifier

A classifier reads an input and picks one label from a fixed list. Our running example is the paper’s Kinetics-Sound task: a 10-second clip goes in, and one of 31 human actions comes out. Each clip has two modalities, a sound track and a few video frames.

The model has three kinds of parts. An encoder is a neural network that turns one modality into a list of numbers, a feature vector. There is one encoder per modality: here two ResNet18 networks (the paper’s choice). Fusion joins the feature vectors; the paper’s default is concatenation, which means gluing them end to end. A classifier head turns the glued vector into 31 scores, called logits.

Here is the data flow with shapes. B is the batch size, the number of clips processed together. The audio spectrogram is B × 1 × 257 × 1004 (the paper’s size for Kinetics-Sound). The audio encoder returns B × 512, the usual output width of a ResNet18. The video encoder returns another B × 512. Concatenation gives B × 1024. The head gives B × 31 logits. The paper does not list the widths, so treat 512 as the standard ResNet18 value.

One loss, shared by both branches

A softmax turns the 31 logits into 31 probabilities that add to 1. The cross-entropy loss is the negative log of the probability given to the true class. If the model gives the right action 0.9, the loss is small. If it gives 0.1, the loss is large. The paper writes it as L = minus the average over clips of yiT log q(xi), where yi marks the true class and q is the softmax output.

Training uses backpropagation: it computes the gradient, the direction in which each weight should move to cut the loss. Stochastic gradient descent (SGD) then moves each weight a small step against its gradient. The step size is the learning rate, written η.

Now the key fact. There is one loss. Both encoders receive their gradients from it. When the loss is large, both get large gradients. When the loss is small, both get small gradients.

How the loud branch starves the quiet one

Suppose audio is easy to learn and video is hard. Early in training, audio learns first. Its features quickly separate the 31 classes well enough that the loss falls. A small loss means a small gradient for everyone. The video branch had not finished learning, but the signal that would teach it is already gone.

We built a toy to show this. The margin z is the sum of what audio has learned (wa) and what video has learned (wv). The loss is log(1 + e−z), and the shared gradient strength is g = 1 / (1 + ez). Each step, audio moves by 0.6 × 1 × g and video by 0.6 × 0.25 × g. Video is four times slower to learn.

step 0: z = 0, g = 0.5, audio +0.300, video +0.075  →  step 1: z = 0.375, g = 0.407, audio +0.244, video +0.061

Drag the slider in the device below. Watch g fall and watch the video’s step shrink with it.

Watch the learning signal drain away

The curve is g, the strength of the shared gradient, over 300 training steps. The two bars are how far each branch moves on the step you pick. Drag the step.

0

Toy model of our own, not the paper’s network: margin z = wa + wv, gradient strength g = 1/(1 + ez), learning rate 0.6, audio rate 1, video rate 0.25, plain training with no mask. The toy shows the starving mechanism, nothing about accuracy.

By step 300 the toy’s g is about 0.004. The video moves 125 times less than it did at step 0. Audio carries almost all of the final margin (the hero showed this as 80% against 20%).

The failure mode, as symptom, measurement and fix

Symptom. The two-modality model scores barely above its best single-modality model, or below it (Chapter 0). Measurement. Train a video-only model and compare it with the video branch taken out of the joint model. The paper does this in its Figure 5: in joint training the audio or text branch nearly matches its stand-alone score, while the video or image branch “exhibits minimal effective learning”. Fix. Give the quiet branch more of the learning, or give the loud branch less. The next chapters show the old way, then the paper’s way.

Why does the video branch learn less once audio has learned well, in the toy and in the paper’s account?

Chapter 2

The Old Fix: Turn One Dial

Slow the loud branch with a single number, see why that works a little, and see what the paper tried instead

A teacher notices the quick student always finishes first. The teacher’s fix: hand the quick student fewer problems. The slow student now gets a turn. That fix is real, and earlier papers use it. It is also blunt, because the teacher cuts every problem by the same amount, easy or hard.

Gradient modulation: scale a branch’s learning

Recall the SGD step: each weight w moves by −η times its gradient. The earlier methods add one number, v, the gradient modulation coefficient. The update becomes

w(t+1)
The weights of one modality’s encoder after step t. A weight is one adjustable number.
w(t)
The same weights before the step, as one long vector.
v(t)
The modulation coefficient. If the modality is dominant, v(t) is below 1. Otherwise v(t) = 1 and this is ordinary SGD.
η
The learning rate, a positive number.
∇L(w(t))
The gradient of the loss with respect to those weights: one number per weight, saying which way to move it.

Earlier methods differ in how they choose v. The paper cites OGM-GE, which lowers the dominant modality’s coefficient on the fly; AGM, an adaptive version; gradient blending (ORG-GB), which weights branches by how much each overfits; and PMR, which adds a prototype loss to lift the weak branch. The paper’s complaint is about what they share: they all multiply the whole modality’s gradient by one number. The paper calls this global-wise updating.

A worked example. An encoder has four weights, and their gradients are [0.80, −0.10, 0.40, 0.02]. Take v = 0.5. The new gradients are [0.40, −0.05, 0.20, 0.01]. Every weight keeps its direction. Every weight learns at half speed. The weight with gradient 0.80, probably doing the most work, is slowed exactly as much as the weight at 0.02, which hardly matters.

Now compare that dial with a mask at the same setting. Keep two of the four weights at random, so each weight is kept half the time. There are six possible pairs. Keeping weights 1 and 3 leaves a total gradient size of 0.80 + 0.40 = 1.20. Keeping weights 2 and 4 leaves 0.10 + 0.02 = 0.12. The six pairs give 0.90, 1.20, 0.82, 0.50, 0.12 and 0.42. Their average is 0.66, and the dial v = 0.5 also leaves 0.5 × 1.32 = 0.66. This is our own arithmetic, not a table from the paper. It shows the two schemes remove the same amount of gradient on average. What differs is the shape: the dial slows every weight a little, while the mask lets some weights take a full step and some take none.

The failure mode is a dial turned on the wrong branch. Look ahead to the paper’s grid. Setting 0.2 on the video branch, the quiet one, with the audio branch at 1.0, scores 59.91% with the dial and 59.72% with the mask. Both sit between 4.6 and 4.8 points below the 64.55% you get with no change. Slowing the quiet branch only starves it further. The gain comes from the other side of the grid.

The paper’s preliminary experiment

The authors test whether something finer is better. They use Kinetics-Sound with concatenation fusion, and a grid of settings for the audio branch (rows) and the video branch (columns). In the global-wise grid, the settings are v values from 1.0 down to 0.2. In the element-wise grid, they are update ratios ρ: the share of weights allowed to learn on a step (Chapter 3 defines this). The weights are chosen by uniform random sampling, with no cleverness. Table 1 holds the results. The best earlier method scores 67.10.

Compare one dial with a random mask

Each cell is a test accuracy from the paper’s Table 1. Pick how much of the audio branch and the video branch may learn. A green outline beats the best earlier method (67.10).

0.2
1.0

Numbers are the paper’s Table 1 (Kinetics-Sound, concatenation fusion, audio dominant). Left grid: global-wise coefficient v. Right grid: share rho of weights updated, chosen uniformly at random. Settings run 1.0, 0.8, 0.6, 0.4, 0.2.

Two things stand out. First, the global-wise grid never beats 67.10; its best cell is 66.13 (audio 0.2, video 1.0). Second, the element-wise grid beats it in four cells, and its best cell scores 68.96 in the same position. Both grids agree on the direction: shrink the loud audio branch, leave the video branch alone. Masking just does it better, even with weights picked at random.

That result is the paper’s springboard. If a random half-mask beats a dial, a mask chosen with care should do even better. The paper’s Table 1 also hints at the rule: a higher update share for the quiet branch than for the loud one tends to help.

What do the paper’s earlier gradient-modulation methods have in common, and what does the paper say is wrong with it?

Chapter 3

Update Only Some Weights

Run the whole network forward, then let only a chosen subset of weights learn on the way back

Imagine a choir of twelve singers rehearsing a song. After each run-through, the director gives feedback. A dial would tell every singer to change their voice a little. A mask picks four singers and tells only them to change. All twelve still sing in every run-through. Only four adjust.

The mask, in one line

The paper writes the idea as a mask vector m(t): a list of 0s and 1s, one per weight, with the same length as the weight vector w. A weight wj gets mj = 1 if it belongs to the chosen subnetwork S(t) on step t, and mj = 0 otherwise. The update becomes

w(t+1)
The weights after the step.
w(t)
The weights before the step.
η
The learning rate.
∇L(w(t))
The full gradient: one number per weight.
⊙
The Hadamard product: multiply two lists of the same length, entry by entry. A gradient times 0 becomes 0. A gradient times 1 stays the same.
m(t)
The 0/1 mask. Its j-th entry is 1 if weight j is in the chosen subnetwork S(t) on step t.

Compare this with Chapter 2. There, one number v multiplied the whole gradient. Here, a whole list of numbers, one per weight, multiplies it. That is why the paper calls the new scheme element-wise: each element of the gradient gets its own factor. Figure 1 of the paper draws the two side by side.

A worked example, with four weights. Let w = [1.0, 2.0, −1.0, 0.5], the gradient = [0.8, −0.1, 0.4, 0.02], η = 0.1, and the mask m = [1, 0, 1, 0]. The masked gradient is [0.8, 0, 0.4, 0]. Then w(t+1) = [1.0 − 0.08, 2.0, −1.0 − 0.04, 0.5] = [0.92, 2.0, −1.04, 0.5]. Two weights moved. Two stayed exactly where they were.

Forward whole, backward partial

Here is the point that is easy to miss. The mask acts only after the gradient is computed. In the forward pass, every weight is used to make the prediction and the loss. In the backward pass, all gradients are computed too. Only then does the mask zero some of them before the update. The paper stresses this to separate its idea from pruning (deleting weights) and from forward-pass methods such as DropConnect. It sits with a second family of methods that keep the whole network in the forward pass and mask gradient updates (the paper cites Mixout and two subnetwork fine-tuning methods).

The mask changes on every step. A weight that is masked now may be chosen on the next step. Over many steps every weight gets updates, but a weight in a heavily masked branch gets them less often.

Mask a gradient and see the step

Top row: the full gradient of twelve weights. Pink bars are kept by the mask. Bottom row: how far each weight moves. Choose how many weights may learn, then draw a new random mask.

4 of 12

Toy numbers of our own: twelve made-up gradients, a uniform random mask like the paper’s preliminary experiment (Table 1), learning rate 1 so the bars equal the gradients. The paper’s own masks are chosen by importance (Chapter 5).

Look at the read-out as you draw masks. A random mask sometimes keeps the big gradients and sometimes drops them. The size of the step swings from draw to draw. A careful mask would keep the weights that matter. That is the job of Chapter 5.

Where the ratio comes in

The update ratio ρ is the share of a modality’s weights that stay in the subnetwork: |S| = ρ × (number of weights). In the device, 4 of 12 means ρ = 0.33. In Table 1 the audio branch at ρ = 0.2 and the video branch at ρ = 1.0 scored 68.96%.

The paper has two questions left to answer for each modality: how many weights should be in S (what is ρ?), and which ones? It says so itself, and the next two chapters take them in that order. Notice, too, the cost of this idea. Masking removes some of the learning on every step, so training moves more slowly. The paper accepts that: its Figure 4 shows slower loss curves but better test accuracy.

In AMSS, what does the mask change, and what does it leave alone?

Chapter 4

How Many Weights: Modal Significance

Measure how much each sense tells the model about the answer, then turn that measurement into a share of weights allowed to learn

Before a quiz, you want to know which of two study helpers actually taught you something. You ask each helper to give you the answer, and you check how much less unsure you feel afterwards. The helper that removes more doubt gets more credit. AMSS does this for each modality, on every batch of data. That credit is the modal significance.

Uncertainty, in nats

Entropy, written H, measures how unsure you are. For a label with 31 equally likely classes, H(Y) = ln 31 = 3.43 nats (the unit when you use the natural logarithm). If you knew the class for sure, the entropy would be 0.

Mutual information, written I(X; Y), measures how much knowing the input X cuts your uncertainty about the label Y. High means the input is very informative. It cannot be computed directly for sounds and pictures, because they have far too many dimensions. The paper uses a standard trick, a lower bound from Barber and Agakov, which it calls Equation 2:

I(X(k); Y)
The mutual information between the k-th modality and the label. Our target, which we cannot compute.
H(Y)
The entropy of the labels in the batch: how unsure you are before looking at any input. The paper computes it from the batch’s label shares p(yc).
E [ log q(y | x(k)) ]
The average, over the batch, of the log of the probability that a classifier using only modality k gives to the true label. It is negative, and closer to 0 when the classifier is surer and right.

The reading is simple. H(Y) is the doubt before. The minus of the second term is the doubt left after. The gap is how much doubt modality k removed. A worked example: the audio classifier gives the true class a probability of 0.9 on average. The doubt left is −ln 0.9 = 0.105 nats. The audio removed at least 3.43 − 0.105 = 3.33 nats. A video classifier that gives 0.30 leaves 1.20 nats and removes at least 2.23 nats.

The bound is tight when q equals the true conditional probability, so the paper needs a classifier that predicts q(y | x(k)). Where does it come from? In late fusion, each modality already has its own classifier, so q = softmax(f(k)(φ(k)(x))) at the true class. In early and hybrid fusion, the paper replaces the other modalities’ feature vectors by zeros and runs the joint classifier. That means one extra read of the head per modality, and no extra network.

From significance to update ratio

The paper then divides by the entropy of the modality’s own input, giving the mutual information rate û(k) = I(X(k); Y) / H(X(k)). It also smooths the number with momentum, a running average over batches so a single odd batch does not swing the result: u(k) = λ u(k) + (1 − λ) û(k), where λ is the decay factor. Last, a softmax over the modalities with a temperature τ gives the update ratio:

ρ(k)
The update ratio of modality k: the share of its weights that may learn on this step.
exp(u(k) / τ)
A positive score that grows with the modality’s significance.
∑ exp(u(n) / τ)
The sum of those scores over all K modalities, so the shares add to 1.

The softmax share is large for a significant modality. Subtracting it from 1 flips it, so the loud modality gets the small update ratio. With two modalities the ratios add to 1. The knob τ sets how sharply the gap shows. The paper says τ below 1 amplifies the differences and τ above 1 shrinks them. It uses τ = 0.25 on the audio-video datasets and 0.5 on text-image and NVGesture.

Turn significance into update ratios

Set how sure each single-modality classifier is about the true class, and the temperature tau. The bars show how large a share of each branch’s weights may learn.

0.90
0.30
0.25

Illustrative: 31 equally likely classes (H(Y) = ln 31 = 3.43 nats), and an input entropy H(X) of 4 nats for both modalities, chosen by us so the numbers sit near the paper’s. The paper does not state how it estimates H(X). The paper’s own averages (Table 8, first 10 epochs): audio 0.2479 and video 0.7521 at tau = 0.25; audio 0.4404 and video 0.5596 at tau = 4.

With the defaults, audio gets ρ of about 0.25 and video about 0.75, which is the paper’s figure for τ = 0.25. Move τ up to 4. The two ratios drift toward 0.5, the same trend as Table 8. Drag the audio confidence down and the ratios swap: a modality that is less significant earns a larger share of learning.

Why this makes sense, and where it can fail

This is a feedback loop. If audio gets very significant, its ratio falls and fewer audio weights learn. Video then catches up and its significance rises. The ratios move back toward each other. The paper adds a warning: do not make the gap too wide. At τ = 0.1, accuracy slips a little, probably because the loud branch loses its own momentum (Chapter 8).

Audio is much more significant than video in a batch. What does AMSS do with the update ratios?

Chapter 5

Which Weights: Fisher Information

Score every weight by how much the prediction cares about it, then sample the subnetwork by score instead of at random

A teacher has time to coach four of twelve students today. Picking names from a hat is fair, but it may skip the students who need the most help. Picking the same four best-scoring students every day is efficient, but eight students never get coached. The paper’s answer sits between the two: pick by score, with some luck.

A score for each weight: Fisher information

Chapter 4 fixed how many weights may learn. Now: which? The paper scores each weight by its Fisher information. In plain words, this measures how much the model’s prediction changes when you nudge that one weight. A weight with a high score is one the answer is sensitive to. The paper cites earlier work showing such weights are the ones that matter for the task.

In practice the paper uses the empirical Fisher, estimated on the current batch. For weight j of modality k, it is the average over the B clips in the batch of the squared slope of the log-probability of the true class:

Fj
The Fisher score of weight j. It is never negative, because it is a square.
(1/B) ∑i=1B
The average over the B clips in the batch.
∂ log p(ŷi | xi; w) / ∂ wj
How fast the log-probability of the true class ŷi changes as weight j changes. This is a gradient, one number for each weight and clip. The paper defines ŷi as the class index of the label.

Then each score becomes a probability: pj = Fj / ∑ F. The pj add up to 1. The more a weight matters, the higher its pj.

Why sample, not just take the top

The paper uses non-uniform adaptive sampling without replacement. It draws one weight with chance pj, removes it from the pool, draws again, and stops when it has ρ × (all weights) of them. Important weights are chosen often, and unimportant ones are chosen sometimes. The paper says this reaches “a more extensive scope” of weights than always taking the highest scores. It also ties to importance sampling, the idea of drawing more often from where it matters.

Mask units: the practical shortcut

A ResNet has millions of weights. Sampling weight by weight would be too slow. So the paper defines a mask unit: all the weights of one convolution kernel (a small filter that slides over the input) move together. A unit’s importance is the sum of its weights’ Fisher scores. In a Transformer, a unit is one attention head. The paper also applies the ratio ρ inside every layer, so no layer is left with nearly all of its kernels frozen.

A worked example for one layer with six kernels. Their normalized scores are [0.40, 0.25, 0.15, 0.10, 0.06, 0.04], and ρ = 0.5 means three kernels. Always taking the top three gives the same kernels 1, 2 and 3, every step, and kernels 4 to 6 never learn. Sampling draws kernel 1 first with chance 0.40. If it draws kernel 5 on some step, with chance 0.06 at the first draw, then that kernel gets a step of learning that top-k would never give it.

Choose a subnetwork three ways

Top row: each kernel’s importance score. Bottom row: how often it was chosen in 20 training steps. Choose a rule, choose how many kernels may learn, and run again.

6 of 16

Toy layer of our own: 16 kernels whose scores fall off by a factor of 0.78 from one kernel to the next, normalized to sum to 1. The paper’s sampling is without replacement and proportional to these probabilities; we draw the same way. The scores of a real layer come from batch gradients.

Try the three rules at 6 of 16. Top scores keep touching the same 6 kernels. Uniform random touches nearly all of them but ignores what matters. Sampling by score picks the high scorers in most steps (about three in four, in our runs) and still reaches the low scorers. The paper’s ablation (Chapter 8, Table 6) finds that this beats uniform sampling on all four datasets it tests.

Why does AMSS sample the subnetwork in proportion to Fisher scores instead of always taking the highest-scoring units?

Chapter 6

Keeping the Shortcut Honest: AMSS+

See why masking shrinks the average gradient, and how scaling each kept weight by the inverse of its chance of being kept puts it back

A pollster calls 100 people out of 10,000. Suppose she calls young people twice as often as old people. If she then reports the plain average, the young are over-counted. Her fix is to give each answer a weight: 1 divided by the chance that person was called. The answers from rarely called groups count for more. Now the average is fair again. AMSS+ applies the same fix to gradients.

The problem: a masked gradient is, on average, too small

Training with SGD has a quiet assumption. The expected value of the gradient on one batch (its average over many batches) should equal the true gradient. A gradient with that property is unbiased. It is the basis for the usual proofs that SGD converges.

Now mask the gradient. Say weight j is kept with probability πj and its true gradient is gj. On average the update uses πj × gj, not gj. The estimate is biased: too small, and by a different factor for each weight. The paper notes a second trouble: the mask depends on the batch (through the Fisher scores), so mask and gradient are not independent.

The paper first analyzes plain AMSS under an extra assumption it calls Mask-Incurred Error: the bias is at most a fraction δ of the true gradient, with δ between 0 and 1. Its Theorem 1 (informal) then says the average squared gradient norm over T steps shrinks like (1 + (1 + ν)2) / (√T (1 + ν)(1 − δ2)). Here ν is a constant that bounds the gradient noise. The factor (1 − δ2) in the bottom makes the bound slightly worse than without masking (δ = 0). So AMSS does converge to a stationary point, a little more slowly.

The fix: divide by the chance of being kept

AMSS+ removes the extra assumption. Its mask holds 1/pj where AMSS held a 1:

m̂j
The AMSS+ mask entry for weight j. It is a scale, not only a 0 or 1.
1 / pj
The inverse of the chance that weight j is picked. The paper’s appendix defines pj as P(wj ∈ S). We write it πj in the device.
wj ∈ S
Weight j (or its unit) is in the sampled subnetwork on this step.

Check it with one number. Let πj = 0.25 and gj = 1. A quarter of the time the weight is kept and the update uses 4 × 1 = 4. Three quarters of the time it uses 0. The average is 0.25 × 4 + 0.75 × 0 = 1. That equals the true gradient. The appendix proves it in general: E[∇ℓ ⊙ m̂] = ∇L. Theorem 2 then gives the same rate as plain SGD, (1 + (1 + ν)2) / (√T (1 + ν)), with no δ term.

The practical catch, and the paper’s patch

With thousands of units, each pj is small, so 1/pj is huge. A kept weight with π = 0.02 would get a 50 times larger step. The paper warns of “gradient explosion”. Its patch is to soften 1/pj into 1/(pj + a), where a is a constant in [0, 1). It then sets a to Ll, the number of mask units in layer l, so the scale that the paper’s final formula prints (page 6, just before Section 4) is 1/(pj + Ll). A layer has tens or hundreds of units, so this scale is usually below 1: a kept unit’s gradient is divided, not boosted. A unit with a larger chance pj still gets a smaller scale, so the inverse link between chance and scale survives.

The softened scale is no longer exactly unbiased: the average gradient becomes π/(π + a) of the truth. For π far below a, that is close to π/a, a smooth, rank-preserving shrink. This is a trade the paper makes on purpose, to keep training stable.

Check the average gradient

One weight with a true gradient of 1. Choose the chance π that it is kept and the softening constant a. The bars show the average gradient it delivers over 2,000 random masks.

0.25
0.25

Monte Carlo of our own with a fixed random seed: 2,000 draws of keep or drop for one weight whose true gradient is 1. The softened rule is 1/(pi + a) as the paper describes it for a in [0, 1); the paper then sets a from the layer’s unit count, which is much larger than 1.

Set π = 0.25 and a = 0. The plain mask delivers about 0.25 of the true gradient. The exact rescale delivers about 1.0. Now raise a. The average falls below 1 again, but the largest single step falls too: with π = 0.02, exact rescaling gives a 50 times boost, while a = 0.25 gives under 4 times.

A weight is kept with chance 0.2 and has true gradient 1. AMSS+ scales it by 1/0.2 when kept. What is its average update over many steps?

Chapter 7

What the Tables Show

Read the paper’s main results one table at a time, and learn which gains are large and which are small

A results table is a claim with evidence. Before you read the numbers, ask three questions. What was compared? On what data? Under what rules? Here the answers are the paper’s setup, and a reader who wants to re-run it needs them.

The setup

There are five datasets. Kinetics-Sound: 31 action classes, about 19,000 ten-second clips (15k train, 1.9k validation, 1.9k test), audio and video. CREMA-D: six emotions from 7,442 acted clips (6,698 train, 744 test), audio and video. Sarcasm Detection: 24,635 text-image pairs, two classes. Twitter-15: 5,338 text-image pairs, three classes. NVGesture: 1,532 hand gestures with three modalities (RGB, depth and optical flow), 25 classes.

The audio-video models use a ResNet18 for each modality, trained with SGD (momentum 0.9, weight decay 1e-4, learning rate 1e-2, cut by 0.1 when the loss flattens). The audio is a spectrogram, 257 × 1004 for Kinetics-Sound. The text-image models use BERT for text and ResNet50 for images, trained with Adam at a learning rate of 1e-5; the text is cut at 128 tokens and the images at 224 × 224. The temperature τ is 0.25 for audio-video and 0.5 for text-image and NVGesture. Everything runs on one RTX 4090. For fairness, all rebalance methods use the same backbones and concatenation fusion.

Table 2: the main comparison

The device below lists ten methods on each dataset’s accuracy. Concat is plain fusion. The rest are earlier rebalancing methods, then AMSS and AMSS+.

Rank the methods on each dataset

Pick a dataset. Pink bars are the paper’s methods. Grey bars are plain fusion and earlier rebalancing methods. The bar axis does not start at zero.

Accuracy values from Table 2 of the paper. Greedy is left out because the paper has no entry for it on the text-image datasets (it reports 66.52 on Kinetics-Sound and 66.64 on CREMA-D). Table 6 lists AMSS+ on CREMA-D as 70.16, while Table 2 lists 70.30; we use Table 2.

On Kinetics-Sound, AMSS+ scores 72.25%. The best earlier method, gradient blending (ORG-GB), scores 67.10%, a gain of 5.15 points. Over Concat it is 7.70 points. On CREMA-D, AMSS+ gets 70.30% against 67.34% for DOMFN, a gain of 2.96, and 6.99 over Concat. These are large gains. On the text-image sets they are small. On Sarcasm Detection AMSS+ leads the best earlier method by 0.12 points (84.35 against 84.23), and on Twitter-15, AMSS alone beats AMSS+ (75.89 against 75.12). With test sets of about 2,400 and 1,000 examples, gaps of 0.1 to 0.2 points are within what one more or fewer correct answers can move. The paper gives no error bars or repeated runs in the text we read, so do not read much into the small gaps.

Three or more modalities, and Transformers

Table 3 (NVGesture, three modalities, trained from scratch) shows plain sum fusion at 78.63% accuracy and AMSS+ at 84.64%. With pre-training the numbers are 82.57% and 83.20%. The best single modality, depth, scores 80.50 from scratch. So here fusion first falls below the best single modality, and AMSS+ lifts it above. Older methods such as OGM-GE and Greedy handle only two modalities; AMSS handles any K, because the softmax in Chapter 4 runs over all modalities.

Table 4 swaps the CNNs for a Transformer (MBT, which fuses audio and video through bottleneck tokens). The gains shrink. From scratch, MBT scores 58.52% on Kinetics-Sound and AMSS+ 60.03%. On CREMA-D the figures are 54.17% and 56.18%. With pre-training: 79.03% and 80.09% on Kinetics-Sound, and 78.63% and 79.57% on CREMA-D. Several earlier methods (OGM-GE and PMR for instance) score below plain MBT in some cells. AMSS+ never does. It is not best everywhere either: on Kinetics-Sound from scratch, AGM scores 60.34 and AMSS+ 60.03.

Table 5 adds AMSS or AMSS+ to five other fusion methods. Every one improves. For example, Affine fusion rises from 64.24 to 65.02 with AMSS and to 69.08 with AMSS+ on Kinetics-Sound. So the masking acts as a plug-in, not as a new architecture.

Does the weak branch really recover?

The paper’s Figure 5 answers this. It defines an imbalance degree from the modal significance of Chapter 4 and plots it over training, with and without AMSS+. The curve is lower with AMSS+. It also compares each branch taken out of the joint model. Without the method, the video branch is far below a video-only model. With AMSS+, the paper says, its Kinetics-Sound performance “close[s] to single-modal training”. The paper gives these results as plots, not tables, so we cannot quote exact values.

Which statement is best supported by Table 2?

Chapter 8

Ablations and Limits

Switch off one part of the method at a time to see which parts earn their place, then list what the paper does not show

A mechanic who adds five new parts to an engine and sees it run faster does not yet know which part helped. The honest test is to remove the parts one at a time. Papers call this an ablation. The paper runs four of them, and each one maps to a chapter of this lesson.

Does importance sampling matter? (Table 6)

Table 6 compares four rules on Kinetics-Sound: Baseline (no subnetwork training), Random (uniform sampling, but with the same modality-specific ratios as AMSS+), AMSS and AMSS+. Accuracy is 64.55, 66.45, 68.96 and 72.25. Two lessons follow. The ratios alone, with random weights, already beat the baseline by 1.9 points. Choosing weights by Fisher score adds 2.5 points on top (AMSS), and the unbiased scaling adds another 3.3. The same order holds on CREMA-D, Sarcasm Detection and Twitter-15, with smaller steps.

Which part of the network? (Table 7)

In late fusion, each modality has an encoder (the “backbone”) and a classifier. Table 7 masks only the classifier, only the backbone, or both. On Kinetics-Sound the scores are 65.37 (classifier only), 66.68 (backbone only) and 68.96 (both) for AMSS; and 66.80, 69.15 and 72.25 for AMSS+. Both parts help and the backbone helps more. A side result from the paper’s text: with ρ set to 1 for both modalities (no mask at all) but the AMSS+ scaling kept, Kinetics-Sound accuracy is 69.39. That is above AMSS (68.96) but below full AMSS+ (72.25), so masking is still needed.

How sharp should the gap be? (Table 8)

The temperature τ sets the gap between the two update ratios. The paper sweeps it from 0.1 to 4 on Kinetics-Sound. Accuracy falls from 72.52 at τ = 0.2 to 70.20 at τ = 1 and 69.69 at τ = 4. The audio ratio rises from 0.2436 to 0.3305 to 0.4404 over the same range. Even τ = 1, which means no sharpening at all, scores 70.20, above the best earlier method (67.10). Going the other way, τ = 0.1 gives 72.17, slightly below 72.52. The paper reads this as: a gap is good, a huge gap starts to hold the loud branch back.

Switch off a part and read the cost

Choose an ablation. Each bar is Kinetics-Sound test accuracy from the paper. Press a bar’s name in your head against the baseline: what did removing the part cost?

All values are Kinetics-Sound accuracy from Tables 6, 7 and 8 of the paper. In Table 8 the number after each tau is the average audio update ratio over the first 10 epochs. The axis starts below the lowest bar, not at zero.

Optimizers, speed and the limits

The theory assumes SGD, so the paper also tests AdaGrad and Adam (its Figure 3). The method beats the baseline with all three, with the largest gain under SGD. Training is slower: Figure 4 shows AMSS and AMSS+ lower the training loss more slowly than the baseline, yet reach better test accuracy, which the paper takes as better generalization.

Honest limits, from the paper and from reading it closely. First, every test is a classification task on a small dataset; there are no large pre-trained multimodal language models here (the lecture’s later topic). Second, the paper does not report how much extra time the Fisher scores and sampling cost per step. Third, gains drop on the Transformer backbone and on text-image data. Fourth, the unbiased AMSS+ is softened in practice, so the theorem and the code are not identical. Fifth, we found no error bars in the text. Sixth, the decay factor λ and the entropy estimate for H(X) are not specified in the parts we read, so a reproduction must choose them.

Imbalance is also not the only reason fusion fails. The paper’s own Table 4 shows that on a Transformer with cross-modal bottlenecks, simple rebalancing helps less. Where modalities interact deeply, other causes matter.

In the paper’s Table 6, what does the “Random” row show?

Chapter 9

Field Guide

Keep the numbers, the terms, a checklist and the core as code, then see where the idea sits in the field

The whole paper in one paragraph. In joint training, one shared loss lets the easy modality learn first and starve the others (Chapter 1). Earlier fixes scale a whole branch by one number (Chapter 2). AMSS masks the backward pass instead: it measures each modality’s significance (Chapter 4), turns it into an update ratio ρ, scores the units of each branch by Fisher information, and samples that many units to learn (Chapter 5). AMSS+ divides each kept unit by its chance of being picked, so the average gradient stays honest (Chapter 6). It beats earlier methods most clearly on audio and video, and still helps on text, image and three-modality data (Chapter 7).

The numbers

ResultValueWhere
Kinetics-Sound accuracyConcat 64.55, AMSS 68.96, AMSS+ 72.25Table 2
Best earlier method, Kinetics-Sound67.10 (ORG-GB)Table 2
CREMA-D accuracyConcat 63.31, AMSS 67.61, AMSS+ 70.30Table 2
Sarcasm Detection accuracyConcat 82.86, AMSS 84.14, AMSS+ 84.35Table 2
Twitter-15 accuracyConcat 70.11, AMSS 75.89, AMSS+ 75.12Table 2
NVGesture, from scratchSum 78.63, AMSS 82.57, AMSS+ 84.64Table 3
Random mask against global scale, best cells68.96 against 66.13Table 1
Rescale only, no mask (ρ = 1)69.39Text after Table 8
Temperature τ0.25 audio-video, 0.5 text-image and NVGestureImplementation details
Average audio update ratio at τ = 0.250.2479 (video 0.7521)Table 8

The terms

TermPlain meaningChapter
Modality imbalanceOne modality learns fast and starves the other during joint training.1
Global-wise updateOne scale factor v for all weights of a modality.2
Element-wise updateA 0/1 mask, one entry per weight, on the gradient.3
Modal significance uA running average of how much doubt a modality removes, per unit of its input entropy.4
Update ratio ρThe share of a modality’s weights that may learn: 1 minus a softmax of u over τ.4
Fisher scoreThe batch average of the squared slope of the true-class log-probability for a weight.5
Mask unitA convolution kernel or an attention head, masked as one piece.5
AMSS+ scaleEach kept unit is multiplied by about 1 over its chance of being kept, softened by a.6

A checklist for using it

The core as code

python (our sketch of the method, not the authors' code)
import torch, torch.nn.functional as F

def amss_plus_step(encoders, fuse, convs, x, y, st, tau=0.25, lam=0.9, a=0.25, lr=1e-2):
    # x: list of K inputs [B, ...]; y: [B] class ids; convs[k]: conv layers of encoder k
    feats = [enc(xk) for enc, xk in zip(encoders, x)]      # K tensors [B, 512]
    loss  = F.cross_entropy(fuse(feats), y)                    # forward uses EVERY weight

    # 1. modal significance: zero-pad the other modalities (Eq 2, Eq 3)
    H_y = st.label_entropy(y)                                  # entropy of batch label shares
    for k in range(len(feats)):
        pad  = [f if j == k else torch.zeros_like(f) for j, f in enumerate(feats)]
        logq = F.log_softmax(fuse(pad).detach(), -1).gather(1, y[:, None]).mean()
        st.u[k] = lam * st.u[k] + (1 - lam) * (H_y + logq) / st.H_x[k]
    rho = 1 - F.softmax(st.u / tau, dim=0)                    # Eq 4: [K] update ratios

    # 2. backward, then mask the gradients unit by unit
    loss.backward()
    for k, layers in enumerate(convs):
        for conv in layers:
            fisher = st.unit_fisher(conv, feats, y)            # [out] sum of squared log-prob slopes per kernel
            p = fisher / fisher.sum()                          # Fisher scores as probabilities
            n = max(1, int(round(rho[k].item() * p.numel())))
            keep = torch.multinomial(p, n, replacement=False)   # sample without replacement
            m = torch.zeros_like(p); m[keep] = 1 / (p[keep] + a)           # AMSS+ scale; use 1.0 for AMSS
            conv.weight.grad *= m.view(-1, 1, 1, 1)
    with torch.no_grad():
        for p_ in st.params: p_ -= lr * p_.grad             # plain SGD step

The sketch leaves out the batchwise Fisher helper (it needs per-example gradients of the true-class log-probability, for example with torch.func), the classifier head, and how H(X) is estimated. The paper does not give these details in the parts we read.

Where it sits in the field

The paper belongs to the family of methods that fight modality competition: OGM-GE, AGM, PMR and gradient blending all appear in its tables. Its new angle is the unit of control: not a branch, but a sampled subnetwork. That connects it to dropout-style regularization and to sparse fine-tuning methods such as Mixout. It is a training-time fix. It adds no parameters and changes nothing at inference.

Keep going

The takeaway. A multimodal model is only as multimodal as its gradients let it be. AMSS keeps the loud sense from using up the learning signal: it limits how many of its weights learn, picks the important ones, and rescales so the average stays true.

Credit: the method, the experiments and the figures belong to Yang Yang, Hongpeng Pan, Qing-Yuan Jiang, Yi Xu and Jinghui Tang (arXiv:2404.08347). All toy numbers on this page are ours and are labelled so. Now test yourself: open Teach and explain the mask on a whiteboard, or Present it to a friend. Then go back to the two senses at the top and replay all three methods.

Rebalancing Multimodal Training by Masking Subnetworks
Read the paper · Back to the lecture on large multimodal models