Multimodal Transformer (MulT)

A reviewer frowns two seconds before the word that explains the frown. Crossmodal attention lets each word go and find that frown, so nobody lines up the video and the words by hand.

Learn how one attention step, with its questions asked by one stream and its answers read from another, fuses words, faces and voice that run at different speeds.

Slide the frown earlier, then switch between Forced alignment and Crossmodal attention. Watch which one still finds it. Then we build, piece by piece, everything underneath: attention itself, the crossmodal block, the stack, and the results.

You need a feel for vectors and dot products (a dot product multiplies two lists of numbers pairwise and adds them up). We build attention and the rest from zero.

Based on Multimodal Transformer for Unaligned Multimodal Language Sequences (arXiv 1906.00295, June 2019) by Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency and Ruslan Salakhutdinov, Carnegie Mellon University. Code: yaohungt/Multimodal-Transformer. An assigned reading of MIT MMAI Lecture 8.

Find the frown

the ending of this movie is too disappointing8 words, 32 video frames at 15 a second

0 frames

Illustrative toy: a 2.1 second clip, a hand-made frown curve and hand-set attention scores (7 times the frown level, so the word “disappointing” looks for frowns). In MulT the scores are learned. Figure 1 of the paper draws the same contrast on a real movie review.

Chapter 0

Why Words and Faces Drift Apart

See why a spoken sentence, a face and a voice cannot be laid side by side, and what the old fix costs

A movie reviewer sits in front of a camera. She says: “but unfortunately the ending of this movie is too horrible and too disappointing.” Her face does not wait politely for each word. She frowns hard on “but unfortunately”, then waves a DVD case at the lens (this is the clip in the paper’s Figure 6).

Your job is to build a program that reads this clip and says how positive or negative she feels. The paper does exactly that on thousands of clips. To do it well, the program must use the words, the face and the voice together.

Three streams, three clocks

Each kind of signal is a modality: a channel that carries information in its own form. Here there are three. Language is the words. Vision is the face, measured as numbers for each video frame. Audio is the voice, measured as numbers for each short slice of sound. Think of three musicians who each keep their own tempo.

Each modality is a time series: a list of feature vectors, one per time step, in order. A feature vector is a list of numbers that describes one moment. The paper writes a sequence as a matrix with T rows (time steps) and d columns (numbers per step).

The three streams do not tick together. In the paper’s CMU-MOSI data, the face features arrive at 15 steps per second and the voice features at 12.5. The words arrive once per word, at no fixed rate. In the unaligned data, the audio and video of one clip can run past 1,000 steps.

The paper names two hard problems that follow. First, the data is non-aligned: step 7 of the words has no natural partner in the video, because the sampling rates differ. Second, there are long-range dependencies across modalities: a frown can come long before the word it goes with, or long after. Her frown may belong to a word she spoke a second ago.

The old fix: force them into line

Earlier work solved this by hand. First, run a tool (the paper names P2FA) that finds the start and end time of every spoken word. Then, for each word, average all the video features inside that word’s time range. Do the same for audio. Now every stream has exactly one step per word. The paper calls this word-level alignment, and in its aligned data every sequence has length 50.

This works, and it was the standard. But it has three costs. It needs extra information about the data, namely the exact time range of each word. It needs engineering by someone who knows the domain. And it averages away anything that happens outside the word’s own time window. A frown that arrives a second early is simply not in the average.

The hero at the top of this page is that cost made visible. Forced alignment hands the last word the mean of its own four frames. If the frown is not in those four frames, the word never learns about it.

Move the frown, count what survives

Each curve shows the frown level that word 8 receives (vertical) when the frown arrives 0 to 24 frames early (horizontal). Drag the marker and compare the two methods.

8 frames

Same toy as the hero: 32 frames at 15 a second, a hand-made frown curve, attention scores set by hand to 7 times the frown level. The shape of the gap is the point. The numbers are illustrative.

Past about five frames (a third of a second) the alignment curve falls to the floor. The attention curve stays near the top until the frown leaves the clip entirely. That gap is what the whole paper is about.

A worked cost: what the average hides

Say the word “disappointing” takes about 0.4 seconds to speak. At 15 frames a second that is 6 frames. The aligned feature for the word is the mean of those 6 frame vectors. Take one number from the face, a frown level between 0 and 1 (illustrative values).

If the frown peaks inside the word, one frame reads 0.9 and the other five read 0.05. The mean is (0.9 + 5 × 0.05) / 6 = 0.19. The peak is already diluted by a factor of almost five. If the frown came 10 frames earlier, all six frames read 0.05 and the mean is 0.05. The frown has vanished.

Treat this as a small case of symptom, measurement, fix. The symptom is that the model misses an expression that clearly belongs to a word. The measurement is the frown level the word receives: 0.19 in sync, 0.05 when early. The fix is to stop averaging inside a window, and let the word read the whole clip.

The idea in one sentence

Do not line the streams up. Instead, let each time step of one stream ask every time step of another stream, “which of you matters to me?”, and take a weighted mix of the answers. The tool for this question-and-answer is attention, and the paper’s name for the cross-stream version is crossmodal attention. The authors build a whole network of it and call it the Multimodal Transformer, or MulT.

Because every step can look at every other step, the model needs no word timings, and it can find a frown that came early or late. The paper shows that this beats the aligned recipes on aligned data, and by more on the harder unaligned data.

The road

  1. The data: what the three streams look like as tensors, and how long they get.
  2. Attention from zero: queries, keys, values, and the softmax, with a calculator.
  3. Crossmodal attention: the paper’s Equation 1, derived with a worked example.
  4. Preparing the streams: the temporal convolution and the position signal.
  5. The crossmodal transformer: one block stacked D times, and why it always reads the raw source.
  6. The whole model: six transformers, the concatenation and the head, with shapes.
  7. The results: the three benchmarks, the baselines and the gaps.
  8. Ablations and what attention learned: which part earns its keep.
  9. Field guide: numbers, a checklist, the core code, and where to go next.
A frown arrives half a second before the word it explains. Why does word-level alignment miss it?

Chapter 1

What the Three Streams Look Like

Open one clip and count what a computer actually receives: three matrices of different heights

Take one clip from the paper’s data: a person speaks one sentence to a camera for about ten seconds. Before any model sees it, software turns the clip into three tables of numbers. This chapter opens those tables, because every later equation works on them.

Think of each table as a spreadsheet. The rows are moments in time. The columns are measurements taken at each moment. The three spreadsheets have different numbers of rows because each sensor takes readings at its own pace.

Language: one row per word

Each spoken word becomes a word embedding: a list of numbers that places the word in a space where similar words sit close together. The paper uses GloVe, a set of embeddings trained on a large body of text in advance. Each embedding has 300 numbers.

So a sentence of 20 words becomes a table with 20 rows and 300 columns. The paper writes this as XL ∈ RTL × 300. Here L stands for language, TL is the number of words, and 300 is dL, the numbers per word.

Vision: one row per video frame

For the face, the paper uses a tool called Facet. For every frame it reports 35 facial action units: numbers that track the movement of individual facial muscles, such as raising the brows or pulling the mouth corners. The frames come at 15 per second.

Ten seconds of video gives 150 rows and 35 columns: XV ∈ RTV × 35.

Audio: one row per slice of sound

For the voice, the paper uses COVAREP. It cuts the sound into short slices and reports 74 low-level numbers for each. They include 12 Mel-frequency cepstral coefficients (MFCCs, a compact summary of the sound’s spectrum), pitch, a flag for whether the voice is on or off, and other measures of how the vocal cords vibrate.

The slice rate differs by dataset. For CMU-MOSI it is 12.5 per second. For CMU-MOSEI it is 20. Ten seconds at 20 per second gives 200 rows and 74 columns: XA ∈ RTA × 74.

Count the time steps

Bars show how many rows each stream has for a clip of the length you pick. The short bar is the word-aligned version, where all three streams are squeezed to 50 steps. Pick a dataset, then drag the clip length.

30 s

The sampling rates (vision 15 Hz; audio 12.5 Hz on MOSI and 20 Hz on MOSEI) and the length of 50 for aligned data are the paper’s (Section 4.1). The speaking rate of 2.5 words a second is illustrative. The paper says unaligned audio and vision sequences can pass 1,000 steps.

Two versions of every dataset

The paper tests on three public sets. CMU-MOSI has 2,199 short monologue clips, each one sentence long. CMU-MOSEI has 23,454 movie-review clips from YouTube, about ten times more. Each clip carries a sentiment score from −3 (strongly negative) to +3 (strongly positive). IEMOCAP has about 10,000 videos labelled with four emotions: happy, sad, angry and neutral. A person can be sad and angry at once, so each emotion is its own yes-or-no question.

Each set comes in two versions. In the word-aligned version, the tool P2FA finds each word’s time range, and the video and audio features inside that range are averaged. All three sequences then have length 50. In the unaligned version, the authors keep the features exactly as the tools produced them. Now the three lengths differ a lot.

A worked alignment, with numbers

Here is what the word-aligned recipe does to one word (illustrative numbers). The tool P2FA says the word “really” runs from 3.00 s to 3.40 s. At 15 frames a second, that covers frames 46 to 51, six frames of 35 numbers each. The recipe averages those six rows into one row of 35. At 20 audio slices a second, the same word covers eight audio rows of 74 numbers. They become one row of 74.

Do this for every word and you get three tables with one row per word. That is easy for a recurrent network to read. But look at what was lost. Six rows became one, and the order and spacing inside the word is gone. Anything outside 3.00 to 3.40 s was never seen by that word.

Failure mode: streams that cannot be stacked

The symptom is a shape error. An early-fusion model glues the three feature vectors of step t into one longer vector. With 75 words, 450 frames and 600 audio slices there is no step t that exists in all three, so there is nothing to glue. The measurement is the three row counts. The fix, for the baselines, is CTC. The fix for MulT is to never glue by step at all.

Why the old models cannot read the raw version

Many earlier models glue the three streams together at each time step. That needs the same number of steps in each stream. For the unaligned data, the authors give those baselines a helper from speech recognition, connectionist temporal classification (CTC).

Here is the paper’s own toy case. Six audio slices must line up with the five words “I am really really happy”. An alignment predictor, a small recurrent network, reads the audio and gives each slice a probability of being each word or a blank (an empty slot). The CTC loss then rewards any assignment that reads out the right words in order. A valid one is [I, am, x, really, really, happy], where x is a blank. An invalid one is [I, am, I, really, really, happy], which breaks the order.

After training, the authors drop the blank probabilities and multiply the remaining ones into the audio. Six slices become five “pseudo-aligned” ones, one per word. The baselines (EF-LSTM, RAVEN, MCTN) then train on those. MulT skips this step entirely.

What long sequences cost

MulT will compare every step of one stream with every step of another. With TL words and TV frames, that is TL × TV pairs. A 30-second MOSEI clip with 75 words and 450 frames gives 33,750 pairs. That is cheap on one small graphics card, and the paper ran all experiments on one GTX 1080 Ti. It would hurt for hour-long inputs. Chapter 8 returns to this.

Why did the authors add a CTC module to EF-LSTM, RAVEN and MCTN in the unaligned experiments?

Chapter 2

Attention From Zero

Build the one operation MulT is made of: ask a question, score every answer, and read a weighted mix

You walk into a room with four experts and ask one question. Each wears a name tag that says what they know. Each also holds a page of notes. You glance at the tags, decide who matches your question best, and listen mostly to that person. You also half-listen to the next best. In the end you carry away a blend of the notes, weighted by how well each tag matched.

That is attention. Everything in MulT is this room, run many times. We build it in five small steps.

Step 1: query, key, value

Each item in a sequence is a vector. Attention gives every item up to three roles. The query is the question: what am I looking for? The key is the name tag: what do I offer? The value is the page of notes: what do I hand over if you pick me?

The three come from the same item by three learned matrices. If X is a table with one row per time step, then Q = XWQ, K = XWK and V = XWV. The weight matrices W are the learned part. Training adjusts them so that useful questions match useful tags.

Step 2: score each key with a dot product

To measure a match, take the dot product of the query with a key: multiply the two vectors entry by entry and add. Two vectors that point the same way give a big positive number. Perpendicular vectors give zero. Opposite vectors give a negative number.

For example, a query (2, 0) and a key (1, 0) score 2 × 1 + 0 × 0 = 2. The same query and the key (0, 1) score 0.

Step 3: scale the scores

Dot products grow with the vector length. If each vector has dk entries that are roughly independent with variance 1, their dot product has variance dk, so a typical size of √dk. Big scores make the next step too spiky. So we divide every score by √dk to keep them near size 1.

Step 4: softmax turns scores into weights

We want weights that are positive and add up to 1, like shares of your attention. The softmax does this: raise e to the power of each score, then divide each result by the total. Higher scores get bigger shares, and a gap of one point multiplies the share by about 2.7.

For scores (2, 0, 1) the powers are 7.39, 1.00 and 2.72. They total 11.11, so the weights are 0.67, 0.09 and 0.24.

Step 5: mix the values

The output for one query is the weighted sum of the values. If the three values are 10, 20 and 30, the output is 0.67 × 10 + 0.09 × 20 + 0.24 × 30 = 15.7. It sits closest to the value of the best-matching key.

Attention(Q, K, V)
The output table: one mixed vector for every query.
QKT
Every query dotted with every key. With Tq queries and Tk keys it has Tq rows and Tk columns.
√dk
The scale. dk is the length of each query and key vector.
softmax
Applied to each row on its own, so each row of weights adds to 1.
V
The values. Multiplying by V mixes the value rows using each row of weights.

Try it on a toy room. Four keys point up, down, left and right. Each carries one value. Turn the query and watch the weights move.

Ask the room

Left: the query arrow (teal) and four keys A to D. Right: the share of attention each key gets. Turn the query, then raise its length.

30°
2.0

Toy: dk = 2, so scores are divided by √2. The keys are (1, 0), (0, 1), (−1, 0), (0, −1), and the values are 10, 20, 30 and 40. All numbers are illustrative.

Two things to notice. A longer query makes the weights sharper: one key takes nearly everything. A short query spreads attention almost evenly. That is why the scale matters. Without it, long vectors would always give sharp, hard-to-train weights.

Failure mode: scores that are too large

Why the scale is not decoration. Suppose dk = 64. A typical dot product is then about √64 = 8 times bigger than with a single number. Take the three scores (2, 0, 1) from Step 4 and multiply them by 8: they become (16, 0, 8). The softmax weights are then about 0.9997, 0.0000 and 0.0003. One key takes everything.

The symptom is a model whose attention is always almost one-hot. The measurement is the gap between the top two scores: here 8 points, which multiplies the share by e8, about 2,981. The fix is the division by √dk, which brings the scores back to (2, 0, 1) and the weights back to 0.67, 0.09 and 0.24. In training, a saturated softmax also passes almost no learning signal back to the losing keys, so they cannot improve.

Self-attention, and what attention forgets

When Q, K and V all come from the same sequence, we call it self-attention: each word asks about all the other words in its own sentence. The next chapter changes where K and V come from. That one change is the paper.

One more fact from the paper’s Appendix A. Attention is order-invariant: shuffle the rows of the input and each output row is the same mix of the same items. It cannot tell “dog bites man” from “man bites dog”. The fix is to add position information to the input first. Chapter 4 shows how.

In attention, what does a larger query vector do to the weights, if nothing else changes?

Chapter 3

Crossmodal Attention, Equation 1

Change where the keys and values come from, and one stream can read another at any time offset

Back to the reviewer. The word “disappointing” is step 75 of the language stream. It wants to know what her face was doing. The face stream has 450 frames. So the word sends a question to all 450 frames and reads back a mix of whatever answers best.

The question comes from language. The tags and the notes come from vision. This is the whole idea of the paper’s Section 3.1, and it needs one change to the attention you built in Chapter 2.

Two modalities, two roles

Pick two modalities and give them roles. The paper calls them α and β. The target α is the stream that asks. The source β is the stream that answers. In the example, α is language and β is vision. The paper writes this direction as β→α, here V→L.

The inputs are two matrices that may have different lengths and different widths:

Xα ∈ RTα × dα,   Xβ ∈ RTβ × dβ

The three attention ingredients are then built as follows. Queries come from the target. Keys and values come from the source.

Qα = XαWQα,   Kβ = XβWKβ,   Vβ = XβWVβ

Each modality has its own weight matrix because the widths differ. In the raw data, language has 300 numbers per step and vision has 35. The matrices WQα ∈ Rdα × dk, WKβ ∈ Rdβ × dk and WVβ ∈ Rdβ × dv bring both streams into a shared question space of size dk, so the dot product is defined.

The equation

Yα
The output, in RTα × dv. It has one row per target step, so it has the target’s length.
Qα
The target’s questions, one per target step. Shape Tα × dk.
KβT
The source’s tags, flipped so each column is one source step. Shape dk × Tβ.
softmax
Row by row. Row i holds the weights target step i gives to every source step, and they add to 1.
Vβ
The source’s notes. Shape Tβ × dv. Each output row is a weighted mix of these rows.

Read the shapes as a chain. The product QαKβT is (Tα × dk) times (dk × Tβ), which gives a score matrix of Tα × Tβ. Entry (i, j) is how much target step i attends to source step j. Multiplying by Vβ (Tβ × dv) leaves Tα × dv.

So the output has the target’s length but lives in the source’s feature space. Each word-step comes back carrying a summary of the video, chosen by that word. Nothing in the equation needs Tα to equal Tβ. That is how the method avoids alignment.

A full worked example

Use Tα = 2 target steps and Tβ = 3 source steps, with dk = 2 and dv = 2. Suppose the projections already gave these tables (illustrative numbers):

Q = [[1, 0],      K = [[2, 0],      V = [[1, 0],
     [0, 2]]          [0, 1],          [0, 1],
                      [1, 1]]          [3, 3]]

Step 1, scores. QKT = [[2, 0, 1], [0, 2, 2]]. For example, row 1 column 3 is 1×1 + 0×1 = 1.

Step 2, scale. Divide by √2 = 1.414: [[1.414, 0, 0.707], [0, 1.414, 1.414]].

Step 3, softmax per row. Row 1 gives [0.576, 0.140, 0.284]. Row 2 gives [0.108, 0.446, 0.446]. Each row adds to 1.

Step 4, mix the values. Output row 1 is 0.576×[1, 0] + 0.140×[0, 1] + 0.284×[3, 3] = [1.428, 0.992]. Output row 2 is [1.446, 1.783].

The two target steps asked different questions, so they read different mixes from the same three source steps. The output has 2 rows, the target’s length, though the source had 3.

Pick a direction, see the grid

The grid is the score matrix: one row per target step, one column per source step. Choose a direction and a clip length and read off the shapes.

30 s

Shapes follow the paper’s Equation 1, using MOSEI rates (2.5 words, 15 frames and 20 audio slices a second; the words rate is illustrative). The glow pattern is a made-up picture: a band like step-diagonal alignment plus one off-diagonal link, which only attention can make.

Alignment is a special case

The paper makes a sharp point in Section 3.3. Classical alignment can be written as an attention matrix too, but a rigid one: a step-diagonal band where each target step looks only at its own few source steps. Crossmodal attention allows that band, and it also allows every cell off the band. The hot spot in the device above is one such cell.

In CM from audio to language (A→L), where do the queries, keys and values come from, and how long is the output?

Chapter 4

Preparing the Streams: Convolution and Position

Give every step its neighbours, put all three streams in one width, and tell attention what comes first

Look at one video frame in isolation. Her eyebrow measure reads 0.4. Is she starting to frown, or finishing? One number cannot say. You need the frames on each side. Also, the three streams are not even the same width: 300 numbers per word, 35 per frame, 74 per audio slice.

Before any crossmodal attention, MulT fixes both problems with two small steps from Section 3.2. They are cheap, and without them the main step could not run.

Step A: the temporal convolution (Equation 2)

A 1D temporal convolution slides a small window along time. At each step it looks at k neighbouring steps, multiplies them by learned weights, adds the results, and writes one new vector. The window size k is the kernel size. Then the window moves one step and repeats.

X̂{L,V,A} = Conv1D(X{L,V,A}, k{L,V,A}) ∈ RT{L,V,A} × d

Each modality gets its own kernel size and its own weights. The output has the same number of steps T as the input. Only the width changes, to a common d. In the paper’s settings d = 40 for all three.

The paper gives two reasons. First, each step now holds some information about its neighbours. This matters because the streams come at different sampling rates, so a single raw step is a poor unit. Second, the convolution projects every modality to the same width d. Without a shared width, the dot product between a query and a key is not defined.

Count the weights for the vision stream. A kernel of size 3 turns 35 input numbers into 40 output numbers, and it needs 3 × 35 × 40 = 4,200 weights (plus 40 bias terms). The window is tiny, but it is learned: the network decides how to blend neighbouring frames.

Slide the window

Blue bars are an input signal with one sharp spike (an eyebrow flick). The warm line is the output of a window that averages k neighbours. Pick a kernel size.

Toy: the real kernel weights are learned, not equal; an average is the simplest stand-in. The signal is made up. The weight count is exact for 35 input channels, d = 40, and k = 1, 3, 5 or 9.

With k = 1 each step sees only itself, so the convolution only changes the width. With larger k each step carries context. The paper uses k = 3 for vision on most tasks and 3 or 1 for language, and 5 for audio on IEMOCAP (its Table 5).

Here is a tiny worked convolution, with a kernel of size 3 and weights (0.25, 0.5, 0.25). The input has one spike: [0, 0, 1, 0, 0]. At each position, multiply the three neighbours by the weights and add. Position 2 gives 0.25 × 0 + 0.5 × 0 + 0.25 × 1 = 0.25. Position 3 gives 0.5. Position 4 gives 0.25. The output is [0, 0.25, 0.5, 0.25, 0]. The spike has widened to three steps, and the total is still 1.

The choice of kernel size is a trade. A bigger window gives more context and costs more weights, but it also blurs fast events. A face flick that lasts two frames is clearer with k = 3 than with k = 9, as the device shows. The paper reports the sizes it used in its Table 5.

Step B: the position signal (Equation 3 and Appendix A)

Chapter 2 ended with a gap: attention cannot tell order. Shuffle the steps and each output row stays the same mix. But order matters here: “not good” and “good, not” differ, and so do a frown that comes before a word and one that comes after.

So the paper adds a positional embedding: a fixed vector for each position, added to the step’s own vector. It copies the original Transformer’s recipe with sines and cosines:

PE[i, 2j] = sin( i / 100002j/d ),   PE[i, 2j+1] = cos( i / 100002j/d )

Here i is the position, j picks a pair of columns, and d is the width. Think of a set of clock hands. The first pair turns fast, once every few steps. The later pairs turn slower and slower. A position is read from where all the hands point at once, like reading hours, minutes and seconds together.

With d = 4, position i = 1 gives [sin 1, cos 1, sin 0.01, cos 0.01] = [0.841, 0.540, 0.010, 1.000]. Position 2 gives [0.909, −0.416, 0.020, 1.000]. Position 3 gives [0.141, −0.990, 0.030, 1.000]. The fast pair changes a lot each step, and the slow pair barely moves. Together they give each position its own signature, and nothing is learned.

Why such strange frequencies? The pairs run at geometrically spaced speeds, from one turn every 6.28 steps down to one turn every 62,800 steps. That range lets the model tell position 3 from 4 using the fast pair and position 3 from 3,000 using the slow pair. A useful side effect: the code for position i + k is a fixed rotation of the code for position i, so a shift of k steps looks the same everywhere. Attention can learn to use such offsets.

Each stream gets its own table, sized to its own length: PE(T{L,V,A}, d). The sum is the starting point of everything that follows:

Z[0]{L,V,A} = X̂{L,V,A} + PE(T{L,V,A}, d)

The paper calls Z[0] the low-level, position-aware features. The superscript [0] means layer zero: no attention has touched them. Chapter 5 shows why that name matters.

One subtle point. A word at position 30 and a frame at position 30 get the same position code, even though they are at very different times. That is fine. The position codes only tell each stream its own order. The model learns from the data how word steps relate to frame steps.

Why does MulT run a Conv1D on each modality before any attention?

Chapter 5

The Crossmodal Transformer

Repeat the crossmodal block D times, and let every layer re-read the untouched source

You read a hard sentence. One pass is not enough, so you read it again, now knowing what to look for. A second look is better because your understanding changed between looks, while the page stayed the same.

The crossmodal transformer works like that. One crossmodal attention (Chapter 3) is a single glance. The paper stacks D of them, so the words can look at the video, update, and look again. The words change from layer to layer. The video they look at does not.

One block, ingredient by ingredient

A crossmodal attention block is the crossmodal attention plus the standard supporting parts of a Transformer. Four parts need a plain name.

Equation 4, line by line

Take V→L: vision is the source, language is the target. For layer i = 1, …, D:

Z[0]V→L = Z[0]L
The starting point. The stream being updated begins as the language features from Chapter 4: TL × d.
CM[i],mulV→L(·, ·)
Multi-head crossmodal attention at layer i. The first argument supplies the queries. The second supplies keys and values.
Ẑ[i]
The halfway result: attention output plus the residual copy of the input. Shape TL × d.
fθ
The feed-forward network with its own weights θ. A second residual follows.
Z[i]V→L
The output of layer i. It has the same shape as the input, so layers can be stacked as many times as you like.

Read the second line closely. The first argument of CM is Z[i−1]V→L: the language stream as updated so far. The second argument is Z[0]V: the vision stream as it was at the very start, with the superscript zero in every layer. The language side evolves. The vision side is the raw, position-aware feature table (Chapter 4).

Multi-head, with numbers

With d = 40 and 8 heads, each head works in 5 dimensions. For each head, WQ is 40 × 5, as are WK and WV. The head returns TL × 5. Eight heads joined side by side give TL × 40 again, and a final matrix WO (40 × 40) mixes them. This way the width never changes, so the residual connection can add the input back without any reshaping. The number of heads must divide d, which is why the paper notes that d should be divisible by the number of heads.

The feed-forward network is the same idea in a different place. It takes one step’s 40 numbers, widens them, applies a nonlinearity (a bend that lets the network represent more than straight lines), and narrows back to 40. It never looks at other steps.

Why always read the raw source

The paper argues that adapting from low-level features lets each crossmodal block keep the low-level information of the source. If layer 3 read a vision stream already churned by layers 1 and 2, the video would arrive pre-digested. Section 3.1 states this and Section 4.3 tests it. The variant that mixes intermediate-level features scores 80.5% binary accuracy on unaligned CMU-MOSEI. MulT as designed scores 81.6%.

The paper also notes that this is unlike the machine-translation Transformer, which has an encoder and a decoder joined by one attention. In MulT there is no encoder-decoder pair and no self-attention inside the crossmodal transformer. Each block attends straight to low-level features of another modality.

Wire the layers

Top row: the target stream (language) after each layer. Bottom row: the source (vision). Arrows show what each layer reads for keys and values. Choose the wiring and the number of layers.

4

The scores are the paper’s Table 4 (binary accuracy on unaligned CMU-MOSEI). The paper uses D = 4 blocks in each crossmodal transformer (its Table 5). The diagram is schematic.

What one crossmodal transformer returns

After D layers the V→L transformer returns Z[D]V→L, a table of TL × d. It has one row per word, and each row has absorbed what the video says about that word. The next chapter builds six such tables and joins them.

In layer 3 of the V→L crossmodal transformer, which tensor supplies the keys and values?

Chapter 6

The Whole Model, With Shapes

Follow one clip from raw tables to a prediction, and count six crossmodal transformers on the way

Three people sit at a table: Words, Face and Voice. Each wants to hear from the other two before giving an opinion. Words listens to Face, then to Voice. Face listens to Words, then to Voice. Voice does the same. That makes six listening acts, and then each person writes a summary.

This is MulT (the paper’s Figure 2). We now put the earlier chapters in order and track the shape of every table.

Six directions, grouped by target

With three modalities there are six ordered pairs: V→L, A→L, L→V, A→V, L→A and V→A. Each pair gets its own crossmodal transformer (Chapter 5), with its own weights. So the paper’s count is six crossmodal transformers.

The outputs are grouped by target. Language is the target of two transformers. Their outputs, each TL × d, are joined side by side:

ZL = [ Z[D]V→L ; Z[D]A→L ] ∈ RTL × 2d

The same is done for vision and for audio. Each target step now carries what it learned from both other modalities. With d = 40 each row has 80 numbers.

A sequence model finishes each stream

The crossmodal transformers move information between streams. They do not read each stream’s own history. So each joined table Z{L,V,A} goes through an ordinary self-attention transformer (Chapter 2, with Q, K and V from the same table). It lets the steps of one stream exchange information along time.

Then the model takes only the last element of each sequence: one vector of 80 numbers per stream. The last step has attended to all earlier steps, so it acts as a summary. Per Figure 2, the three summaries are joined and sent through fully-connected layers (ordinary dense layers) to give the prediction ŷ.

What the prediction is

For CMU-MOSI and CMU-MOSEI, the label is a sentiment score from −3 to 3, and the paper scores the output as a 7-class label, a positive-or-negative call, and a numeric score. For IEMOCAP the task is multi-label: a separate yes-or-no answer for happy, sad, angry and neutral. The paper’s text does not spell out the training loss, so the released code is the place to check it.

Follow one clip through MulT

Press a stage to see the tables at that point. Bar length is proportional to the number of time steps. Drag the clip length to change T.

20 s

Shapes follow the paper’s Section 3.2, with d = 40 (its Table 5), MOSEI rates and an illustrative 2.5 words a second. In stage 6 the width of 240 assumes the three 80-number summaries are joined, as Figure 2 draws.

One word, end to end

Follow the word “disappointing”, row 75 of the language table. After Conv1D and position it is a vector of 40 numbers. In the V→L transformer it asks all 450 frame rows and gets back a 40-number summary of the face, then refines it over four layers. In the A→L transformer it does the same with the voice. The two summaries sit side by side as 80 numbers. The self-attention step then lets this row talk to the earlier words, so the row now holds the word, the face and the voice together.

Only the last row of each stream goes on. The last row has attended to everything before it, so it works as a summary of the clip. This is a design choice, not a law: the paper simply extracts the last element of each sequence.

Counting the work

For the 30 second MOSEI clip of earlier chapters (75 words, 450 frames, 600 audio slices), the six score grids hold these cells per head and per layer: V→L 33,750; A→L 45,000; L→V 33,750; A→V 270,000; L→A 45,000; V→A 270,000. The total is 697,500. The two grids between vision and audio dominate, because those are the longest streams. Over four layers, that is about 2.8 million cells per head. The numbers are illustrative, but the shape of the cost is real: it grows with the product of two stream lengths.

Size and training

The paper keeps its models near 200 thousand parameters when it compares with older models, so a win is not just a bigger network. Across the three tasks (its Table 5), the common settings are the Adam optimizer, d = 40, four blocks per crossmodal transformer, and 8 to 10 heads. The learning rate is divided by 10 when validation performance stops improving. All runs fit on one GTX 1080 Ti.

Three design choices, with reasons

After the six crossmodal transformers, why does each target’s output table have 2d columns instead of d?

Chapter 7

The Results, Table by Table

Read the three benchmark tables, compute the real gaps, and see how much each method loses without alignment

A new method is only as good as the fair fight it wins. Here the fight has three parts: three datasets, two data versions (aligned and unaligned), and a line of earlier models as opponents. This chapter reads what the paper’s Tables 1, 2 and 3 say, and checks the paper’s own summary against the numbers.

The opponents

The baselines are the strong models of the time. EF-LSTM (early fusion) joins the three streams at every time step, then runs one LSTM (a recurrent network that reads a sequence step by step). LF-LSTM (late fusion) runs one LSTM per stream and joins the three summaries at the end. RAVEN adjusts word representations with the accompanying face and voice signals. MCTN learns shared features by translating one modality into another and back. Other aligned-only models appear in the aligned rows.

EF-LSTM, RAVEN and MCTN need equal-length streams. For the unaligned rows, the authors attach CTC (Chapter 1) to them. LF-LSTM needs no alignment, so it runs as is. To keep the fight fair, the authors hold all models near the same size (about 200 thousand parameters in the aligned case).

What a fair fight means here

Three things make this comparison fairer than it looks. The authors match the parameter counts of the models. They run a basic grid search over dropout rate and number of heads (Appendix C). And they give the older models the best available help for unaligned data, CTC. The aligned tables, meanwhile, are the old models’ home turf, which is why a win there counts.

The metrics

On MOSI and MOSEI: Acc7 is accuracy on the seven sentiment classes from −3 to 3. Acc2 is accuracy on positive versus negative. F1 balances precision and recall. MAE is the mean absolute error of the predicted score, where lower is better. Corr is the correlation between prediction and the human score. On IEMOCAP: binary accuracy and F1 per emotion.

Compare the methods

Pick a dataset, a setting and a metric. The warm bar is MulT. Bars of other models are the baselines in the paper’s tables.

Every value is copied from Tables 1 and 2 of the paper. Bars start at a stated axis minimum, not at zero, so small gaps are visible; read the numbers on the bars.

What the sentiment tables say

On word-aligned data, MulT is best on every column of MOSI and MOSEI. On MOSI it reaches 83.0% Acc2 against 79.3% for MCTN, the best baseline. On MOSEI it reaches 82.5% against 80.6% for LF-LSTM.

On unaligned data the gaps are as follows (computed from the tables, in points).

UnalignedMulTBest baselineGap
MOSI Acc281.177.6 (LF-LSTM)+3.5
MOSI Acc739.133.7 (LF-LSTM)+5.4
MOSEI Acc281.679.3 (CTC+MCTN)+2.3
MOSEI Acc750.748.8 (LF-LSTM)+1.9

The abstract says MulT beats the state of the art “by a margin of 5%-15% on most of the metrics”, and the body says 10% to 15% on most attributes against prior methods, some of them with CTC. The paper does not say how it computes these percentages. They may be relative improvements over a range of metrics, including error measures. In points of accuracy, as above, the gaps are smaller, from 1.9 to 5.4 here.

Emotions on IEMOCAP

IEMOCAP has four emotions. The unaligned binary accuracies (Table 3) show the same story, with the gain varying a lot by emotion.

Unaligned accuracyMulTBest baselineGap
Happy84.880.5 (CTC+MCTN)+4.3
Sad77.772.9 (LF-LSTM)+4.8
Angry73.972.7 (CTC+EF-LSTM)+1.2
Neutral62.562.0 (CTC+RAVEN)+0.5

On aligned IEMOCAP, MulT leads on most columns. On Sad, MFM scores 88.4% accuracy to MulT’s 86.7%, and the paper names this exception. On Angry accuracy, MFM’s 87.5% is also 0.1 above MulT’s 87.4%.

A caution about small gaps

The paper reports one number per cell and no spread across repeated runs. A gap of 0.5 points, like Neutral on unaligned IEMOCAP, may be within the noise of a single training run. So the reliable story is the pattern: MulT is ahead in nearly every cell, sometimes by a few points, and the gap grows when the data is unaligned. Single cells with tiny gaps should not carry weight.

How to read the metrics with a number. If a clip’s true score is 2.0 and the model says 1.0, the absolute error is 1.0. MAE averages these. MulT’s 0.591 on unaligned MOSEI means that the average prediction is about 0.6 points of the −3 to 3 scale away from the human score. Acc7 asks for the exact class out of seven, so it is a much stricter test than Acc2, which is why its numbers sit near 50% while Acc2 sits near 80%.

How much does losing the alignment hurt?

The paper says every model drops when it moves from aligned to unaligned, and that MulT drops less. Check on Acc2: MulT loses 1.9 points on MOSI (83.0 to 81.1) and 0.9 on MOSEI (82.5 to 81.6). RAVEN loses 5.3 on MOSI and 3.7 on MOSEI. MCTN loses 3.4 on MOSI, but only 0.5 on MOSEI, less than MulT. LF-LSTM, which never needed alignment, even gains 0.8 on MOSI. So the claim holds against most rivals, not against all of them on every dataset.

Figure 5 of the paper adds one more fact: on unaligned MOSEI, MulT’s validation error falls faster over 20 epochs than that of the baselines, and ends lower.

From the unaligned MOSEI numbers, what is MulT’s Acc2 gain over the best baseline?

Chapter 8

Ablations, Attention Maps and Limits

Remove or swap one part at a time to see which part earns its keep, then look at what the attention found

A recipe gives a good dish. Which ingredient made it good? To find out, you cook it again with one ingredient left out. If the dish gets worse, that ingredient mattered. An ablation study does this to a model: it removes or swaps a part and compares the score.

The paper runs its ablations on unaligned CMU-MOSEI (its Table 4). We read them as a ladder, from the weakest setup up to the full model.

Rung 1: one modality alone

Take the model apart so that only one stream is used, each with a plain self-attention transformer. Binary accuracy (Acc2) is 77.4% for language, 66.4% for vision and 65.6% for audio. Language is far ahead of the other two. The paper notes that earlier work (Pham et al.) saw the same: a good language network can already do well at inference time.

This matters for reading the later numbers. Most of the final score is already there from words alone. The fusion work is fighting for the last few points.

Rung 2: fuse without crossmodal attention

Two simple fusions test the idea that any fusion helps. LF-Transformer (late fusion) runs a separate self-attention transformer on each stream and joins the last elements of the three. EF-Transformer (early fusion) glues the three sequences end to end in time, one long sequence of length TL + TV + TA, and runs a single self-attention transformer on it. They score 78.6% and 78.9%. Both beat every single modality, though only by about one to one and a half points over language alone.

Rung 3: one crossmodal transformer

Now use crossmodal attention, but only for one target. “Only [V, A → L]” keeps the two transformers that feed language, V→L and A→L. This gets 80.1%. The same idea with vision as target, [L, A → V], gets 79.7%. With audio as target, [L, V → A], it gets 79.2%. All three beat both plain fusions on binary accuracy. Language as target works best, as the paper observes.

Rung 4: the full model, and a variant

Using all six directions gives MulT’s 81.6%. The variant that reads intermediate-level instead of raw features from the source (Chapter 5) gets 80.5%.

Climb the ladder

Drag to add one setup at a time, from the weakest to the full model. The newest bar is lit. Read what the step added.

10

All ten values are binary accuracy (Acc2) from the paper’s Table 4, unaligned CMU-MOSEI. The axis starts at 60%, not zero.

The jumps are not large in absolute terms. Going from the better plain fusion (78.9%) to MulT (81.6%) is 2.7 points of Acc2 and 2.9 of Acc7 (47.8% to 50.7%). The MAE falls from 0.648 to 0.591. The pattern is consistent across metrics in Table 4, which is the stronger evidence.

Each ablation asks a question

SetupQuestion it asksAnswer (Acc2)
Single modalitiesHow much does each stream know alone?Language 77.4; vision 66.4; audio 65.6
LF and EF transformersDoes any fusion help?Yes: 78.6 and 78.9
One target at a timeDoes crossmodal attention beat plain fusion?Yes: 79.2 to 80.1
All six directionsDo the targets add up?Yes: 81.6
Intermediate sourceIs the raw source needed?It helps: 80.5 against 81.6

What the attention looked at

Figure 6 of the paper shows part of the layer-3 attention matrix of the V→L transformer on a real MOSEI clip, about a six-second window. The clip is the reviewer who says “but unfortunately the ending of this movie is too horrible and too disappointing”. The strongest attention sits where emotion-laden words (“movie”, “disappointing”) meet strong facial changes at the start and end of the window. The authors stress that this includes cells off the diagonal, which forced alignment could not give.

Figure 5 shows another effect: on the MOSEI validation set, MulT’s error curve drops faster than the baselines’, and settles lower, over 20 epochs.

Read the map the way the figure draws it. Rows are words, columns are video blocks, and each block summarises a few frames through the convolution. A bright cell means a word attends strongly to that block of video. Bright cells away from the diagonal mean a word looked at video that was not spoken at the same time. That is the thing forced alignment could not show.

Limits, honestly

What does the ablation of the three single-target crossmodal transformers show?

Chapter 9

Field Guide

Keep the numbers, the checklist and the core code, then see where this paper sits and where to read next

You can now rebuild the Multimodal Transformer from memory. Words, faces and voice arrive as three tables of different heights. A convolution and a position signal prepare them. Six directional crossmodal transformers let each stream read the others without any alignment. Self-attention, a last step and dense layers make the prediction.

The one-paragraph summary

MulT replaces forced word-level alignment with directional pairwise crossmodal attention: queries come from the target modality, keys and values from the source, so a target step can read any source step. Each of the six directions is a stack of D = 4 blocks that always read the raw, position-aware source features. The outputs are joined by target, refined by a self-attention transformer, and the last step feeds the prediction. It wins on aligned and unaligned CMU-MOSI, CMU-MOSEI and IEMOCAP, with a smaller loss when alignment is removed than most rivals.

The numbers that matter

WhatNumberWhere
Raw feature widths300 (GloVe), 35 (Facet), 74 (COVAREP)Appendix D
Sampling ratesVision 15 Hz; audio 12.5 Hz (MOSI, IEMOCAP), 20 Hz (MOSEI)Section 4.1
Aligned sequence length50 for every modalitySection 4.1
Unaligned lengthaudio and vision can pass 1,000 stepsSection 4.1
DatasetsMOSI 2,199 clips; MOSEI 23,454 clips; IEMOCAP about 10K videosSection 4.1
Common width d40Table 5
Crossmodal transformers6, each D = 4 blocks, 8 to 10 headsSection 3.2, Table 5
Model sizeabout 200K parameters (aligned comparison)Section 4, quantitative analysis
Unaligned MOSEI Acc2MulT 81.6; best baseline 79.3Table 2
Unaligned MOSI Acc2MulT 81.1; best baseline 77.6Table 1
Ablation ladder (Acc2)language only 77.4; plain fusion 78.9; single target 80.1; MulT 81.6Table 4
Raw vs intermediate source81.6 vs 80.5Table 4

The re-implementation checklist

  1. Load the features. One table per modality: (TL, 300), (TV, 35), (TA, 74). Do not resample.
  2. Conv1D to d = 40, with kernel sizes of 1 to 5 as in Table 5. Keep T.
  3. Add the sine and cosine position table, sized to each stream’s own T.
  4. Build six crossmodal transformers, one per ordered pair. Queries from the target stream as it evolves; keys and values from the source Z[0] in every layer.
  5. Per layer: LayerNorm, multi-head crossmodal attention, residual, LayerNorm, feed-forward, residual.
  6. Join by target to get (T, 2d) tables.
  7. Run a self-attention transformer on each, take the last step, join the three, apply dense layers.
  8. Train with Adam, lowering the learning rate by 10 times when validation stops improving. Check the released code for the loss and the exact settings.
  9. Test both versions: word-aligned and unaligned. A good sign is a small gap between them.

The core as code

Here is the whole model as plain numpy, forward pass only. It follows Equations 1 to 4. It uses random weights for shape checks; with d = 40, a 20-word clip with 120 frames and 160 audio slices returns one prediction vector.

pythonimport numpy as np

def softmax(x):
    e = np.exp(x - x.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)

def layer_norm(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)

def pos_enc(T, d):
    """Appendix A: sine and cosine clock hands, positions 1..T."""
    i = np.arange(1, T + 1)[:, None]
    j = np.arange(d // 2)[None, :]
    ang = i / 10000 ** (2 * j / d)
    pe = np.zeros((T, d))
    pe[:, 0::2], pe[:, 1::2] = np.sin(ang), np.cos(ang)
    return pe

def conv1d(X, W, b):
    """Equation 2. X is (T, d_in); W is (k, d_in, d); output is (T, d)."""
    k = W.shape[0]
    Xp = np.pad(X, ((k // 2, k // 2), (0, 0)))
    return np.stack([np.tensordot(Xp[t:t + k], W, axes=([0, 1], [0, 1]))
                     for t in range(X.shape[0])]) + b

def cross_attn(Xa, Xb, Wq, Wk, Wv):
    """Equation 1. Queries from the target Xa, keys and values from the source Xb."""
    Q, K, V = Xa @ Wq, Xb @ Wk, Xb @ Wv
    A = softmax(Q @ K.T / np.sqrt(Q.shape[-1]))      # (T_a, T_b)
    return A @ V                                      # (T_a, d_v)

def crossmodal_layer(Zt, Zs0, p):
    """One layer of Equation 4. Zt is the evolving target; Zs0 is the raw source."""
    Zt, src = layer_norm(Zt), layer_norm(Zs0)
    heads = [cross_attn(Zt, src, *h) for h in p["heads"]]
    Zh = np.concatenate(heads, -1) @ p["Wo"] + Zt     # attention + residual
    h = layer_norm(Zh)
    return np.maximum(0, h @ p["W1"]) @ p["W2"] + h   # feed-forward + residual

def crossmodal_transformer(Zt0, Zs0, layers):
    Z = Zt0
    for p in layers:
        Z = crossmodal_layer(Z, Zs0, p)               # source stays Zs0 at every layer
    return Z

def mult_forward(X, P):
    """X maps 'L','V','A' to (T, d_in) tables. Returns the prediction vector."""
    Z0 = {m: conv1d(X[m], *P["conv"][m]) for m in "LVA"}
    Z0 = {m: Z0[m] + pos_enc(*Z0[m].shape) for m in "LVA"}
    last = []
    for tgt in "LVA":
        outs = [crossmodal_transformer(Z0[tgt], Z0[s], P["cm"][s + tgt])
                for s in "LVA" if s != tgt]
        Z = np.concatenate(outs, -1)                  # (T_tgt, 2d)
        for p in P["sa"][tgt]:
            Z = crossmodal_layer(Z, Z, p)             # self-attention: source is itself
        last.append(Z[-1])                            # last step as the summary
    return np.concatenate(last) @ P["Wout"] + P["bout"]

Where it sits in the field

The paper’s closing note states its hope: that crossmodal attention might be an equally competitive alternative on tasks where alignment used to be considered necessary. It also names visual question answering, where static and time-evolving signals mix, as a place to try it.

Read it alongside the lecture’s chapters on fusion, heterogeneity and alignment, and see how large multimodal models treat the same question.

Keep going

Credit

This lesson teaches Multimodal Transformer for Unaligned Multimodal Language Sequences by Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency and Ruslan Salakhutdinov (Carnegie Mellon University). The authors released their code at github.com/yaohungt/Multimodal-Transformer. All table values here are from the paper; toy numbers are labelled as illustrative.

The takeaway. Do not force three clocks to tick together. Let each step of one stream ask every step of another stream a question, and let the answers decide what matters.

Now press Teach and explain, on the whiteboard and from memory, how a word step reads a video it was never aligned with: what comes from where, what shape each table has, and why the source stays raw. Then go back to the frown at the top, slide it 14 frames early, and say why only one of the two methods finds it.

Multimodal Transformer for Unaligned Multimodal Language Sequences
Read the paper · Code · MMAI Lecture 8 · The course site · Back to Gleams