ViViT: A Video Vision Transformer

Chapter 1: Tubelet Embedding

To run a Transformer you first need tokens: a sequence of vectors. ViViT proposes two ways to extract them from a video volume of shape T × H × W × C (frames, height, width, channels). The naive one is uniform frame sampling: independently embed 2D patches from each frame exactly like ViT, then concatenate all the per-frame tokens into one long sequence. It works, but each token is still purely spatial — it has seen no motion at all. Any sense of time has to be reconstructed later by the attention layers.

The better idea — the one ViViT contributes — is tubelet embedding. Instead of cutting 2D squares out of single frames, you cut out small 3D bricks that span several frames at once. A tubelet of size t × h × w grabs h×w pixels of spatial extent across t consecutive frames. Each such brick is flattened and linearly projected to a token of dimension d. Because a single token now contains pixels from multiple time-steps, motion is fused at the input layer — the projection itself can learn to respond to a moving edge, before any attention has happened.

This is just a 3D generalization of ViT's patchify. Where ViT divides a 2D image into a grid of non-overlapping squares, tubelet embedding divides a 3D volume into a grid of non-overlapping cuboids. Mechanically it is a single 3D convolution with kernel size t×h×w and stride t×h×w (kernel = stride = no overlap), output channels = d. The paper calls this projection E; each output position is one token z = E·x + b.

A worked example, every number

Take a clip T=32, H=W=224, C=3 and a tubelet t=2, h=w=16, embedding dimension d=768 (ViViT's default, inherited from ViT-Base).

So the token grid is 16 × 14 × 14, giving N = 16 × 14 × 14 = 3,136 tokens. Each token is the flattening of t·h·w·C = 2·16·16·3 = 1,536 raw pixel values, linearly mapped to a 768-dimensional vector. After embedding, the sequence tensor has shape [N, d] = [3,136, 768] (plus a class token, giving 3,137).

Notice the temporal halving: with t=2 we already cut the token count in half compared to per-frame embedding (where the time axis would contribute 32, not 16). Tubelets are thus a tokenizer and a first line of defense against the token explosion of Chapter 0 — bigger t means fewer, motion-richer tokens.

Adding position

Like ViT, the raw tubelet tokens carry no notion of where they sit in space or time — self-attention is permutation-invariant. ViViT adds a learned positional embedding p of shape [N, d] to the token sequence: z = [zcls; E·x1; ...; E·xN] + p. The class token zcls, also borrowed from ViT/BERT, is a learned vector prepended to the sequence; its final-layer output is the clip-level representation fed to the classifier. (Some ViViT variants pool over all tokens instead; we will see why in Chapter 3.)

python
# Tubelet embedding = a single strided 3D convolution
import torch
import torch.nn as nn

class TubeletEmbed(nn.Module):
    def __init__(self, d=768, t=2, h=16, w=16, c=3):
        super().__init__()
        # kernel == stride  ->  non-overlapping cuboids
        self.proj = nn.Conv3d(c, d, kernel_size=(t, h, w), stride=(t, h, w))

    def forward(self, x):
        # x: [B, C, T, H, W]  e.g. [B, 3, 32, 224, 224]
        x = self.proj(x)            # [B, d, T/t, H/h, W/w] = [B, 768, 16, 14, 14]
        B, d = x.shape[0], x.shape[1]
        x = x.flatten(2).transpose(1, 2)  # [B, N, d] = [B, 3136, 768]
        return x

clip = torch.randn(1, 3, 32, 224, 224)
tok = TubeletEmbed()(clip)
# tok.shape -> [1, 3136, 768]   (N = 16 * 14 * 14)

Chapter 2: The Token Explosion & Model 1

Now we have a clean sequence of N tubelet tokens. The simplest possible video Transformer — ViViT's Model 1, the "Spatio-temporal attention" model — does the obvious thing: prepend a class token, add positions, and run a plain ViT encoder over all N tokens jointly. Every token attends to every other token, regardless of whether they are separated in space, in time, or both. This is the most expressive option: it can model any spatio-temporal interaction in a single layer. It is also the most expensive.

Why expensive? Self-attention forms an attention matrix A of shape [N, N]. Its compute and memory cost scale as O(N²·d) — quadratic in the number of tokens. For images, N≈200 is fine. For video, N is in the thousands, and N² is in the millions.

where nt = number of temporal tokens (T/t) and ns = number of spatial tokens per frame (H/h · W/w). The product is squared. This is the crux. The whole reason ViViT has four models is that Model 1's N² is unaffordable past short clips, so Models 2-4 each find a different way to factorize the attention so the cost drops from quadratic-in-the-product to roughly quadratic-in-each-factor-separately.

Worked cost comparison

Use our running numbers: nt = 16 temporal tokens, ns = 14×14 = 196 spatial tokens, so N = 16·196 = 3,136.

Read that last row against Model 1: factorizing the attention into a spatial pass plus a temporal pass costs about 0.66 M vs 9.83 M — roughly a 15× reduction in pairwise-comparison count, and the gap widens as the clip gets longer (longer clips raise nt, and joint attention pays for nt inside the square while factorized pays for it linearly). This single arithmetic fact is the engine behind Models 2, 3, and 4.

Model 1 in code, and its data flow

python
# Model 1: joint space-time attention (plain ViT over all N tokens)
import torch, torch.nn as nn

class JointSpaceTime(nn.Module):
    def __init__(self, d=768, heads=12, depth=12):
        super().__init__()
        layer = nn.TransformerEncoderLayer(d, heads, batch_first=True)
        self.enc = nn.TransformerEncoder(layer, depth)
        self.cls = nn.Parameter(torch.randn(1, 1, d))

    def forward(self, tok):       # tok: [B, N, d] = [B, 3136, 768]
        B = tok.shape[0]
        cls = self.cls.expand(B, -1, -1)
        z = torch.cat([cls, tok], dim=1)   # [B, 3137, 768]
        z = self.enc(z)               # attention is [3137 x 3137] per layer/head!
        return z[:, 0]               # class token -> [B, 768]

Chapter 3: Model 2 — Factorized Encoder

The first and most influential factorization is the Factorized Encoder. The idea is a clean two-stage pipeline that mirrors how a person might describe a clip: "first, what is in each frame; then, how do those per-frame summaries change over time." It uses two separate Transformers stacked in series.

Concretely: the spatial encoder treats the video as nt independent images. For each frame it produces a representation hi (the frame-level class-token output, dimension d). You now have nt such vectors — a short sequence of frame summaries. The temporal encoder is a second Transformer that attends over just these nt tokens to fuse them into a clip-level vector. This is architecturally close to the classic "CNN per frame + temporal model on top" recipe of older video work — except both stages are now pure Transformers.

The data flow, with shapes

Start from tubelet tokens of shape [B, n_t, n_s, d] = [B, 16, 196, 768] (reshaped so the frame axis is explicit).

The cost. The spatial encoder pays nt · ns² (a separate 196×196 attention for each of 16 frames). The temporal encoder pays ns · nt² (one 16×16 attention per location). With our numbers: 16·196² + 16² ≈ 614k + 256 ≈ 0.61 M pairwise comparisons, versus Model 1's 9.83 M. The temporal encoder is almost free because nt is small. The FLOPs are dominated entirely by the spatial encoder, which is just a per-frame image ViT — so the Factorized Encoder costs scarcely more than running an image model on each frame. This is why, in the paper, Model 2 is the most FLOP-efficient of the four and a very strong accuracy/cost trade-off.

python
# Model 2: Factorized Encoder — spatial ViT then temporal Transformer
class FactorizedEncoder(nn.Module):
    def __init__(self, d=768, heads=12, Ls=12, Lt=4):
        super().__init__()
        sl = nn.TransformerEncoderLayer(d, heads, batch_first=True)
        tl = nn.TransformerEncoderLayer(d, heads, batch_first=True)
        self.spatial  = nn.TransformerEncoder(sl, Ls)
        self.temporal = nn.TransformerEncoder(tl, Lt)

    def forward(self, tok):       # tok: [B, n_t, n_s, d] = [B, 16, 196, 768]
        B, nt, ns, d = tok.shape
        # Stage 1: fold frames into batch, attend WITHIN each frame
        x = tok.reshape(B * nt, ns, d)   # [B*16, 196, 768]
        x = self.spatial(x)               # cheap: 196x196 per frame
        x = x.mean(dim=1)               # frame summary -> [B*16, 768]
        # Stage 2: attend ACROSS the 16 frame summaries
        x = x.reshape(B, nt, d)        # [B, 16, 768]
        x = self.temporal(x)              # cheap: 16x16
        return x.mean(dim=1)            # clip vector -> [B, 768]

Chapter 4: Model 3 — Factorized Self-Attention

Model 2 factorized at the encoder level: two whole Transformers in series. Model 3 factorizes inside each Transformer block instead. It keeps a single stack of layers operating on the full set of N tokens, but within every block it splits the one self-attention operation into two sequential attentions: first attend only over tokens from the same temporal index (spatial), then attend only over tokens at the same spatial index (temporal).

The trick is purely a matter of reshaping. The token tensor is logically [B, n_t, n_s, d]. To do spatial attention, you reshape so the spatial axis is the sequence and the temporal axis is folded into the batch: [B·n_t, n_s, d] — now standard self-attention only mixes tokens within a frame. To do temporal attention, you reshape the other way: [B·n_s, n_t, d] — now self-attention only mixes tokens across time at a fixed spatial location. No token-to-token interaction is lost permanently; it just happens in two steps within the block, and is recombined as the block repeats over depth.

The reshape, made concrete

With n_t=16, n_s=196, d=768 and batch B:

The combined cost per block is again nt ns² + ns nt² ≈ 0.66 M — the same dramatic reduction from Model 1's N². The number of parameters is identical to Model 1 (it has the same total set of attention projections), but the compute is factorized. The paper notes Model 3 has the same number of layers as Model 1 yet, because the two attentions are split, the per-block FLOPs drop.

python
# Model 3: factorized self-attention inside a single block
def factorized_block(z, spatial_attn, temporal_attn, B, nt, ns, d):
    # z: [B, nt*ns, d]  (the N tokens of one clip)
    # --- spatial: fold time into batch, attend within each frame ---
    zs = z.reshape(B, nt, ns, d).reshape(B * nt, ns, d)
    zs = spatial_attn(zs)                 # 196x196 attention
    z  = zs.reshape(B, nt, ns, d)
    # --- temporal: fold space into batch, attend across time ---
    zt = z.permute(0, 2, 1, 3).reshape(B * ns, nt, d)
    zt = temporal_attn(zt)                # 16x16 attention
    z  = zt.reshape(B, ns, nt, d).permute(0, 2, 1, 3)
    return z.reshape(B, nt * ns, d)

Chapter 5: Model 4 — Factorized Dot-Product

Models 2 and 3 factorize across operations (two encoders, or two attentions in series). Model 4 factorizes at the most granular level possible: inside a single multi-head attention, across the heads. Half the attention heads attend spatially; the other half attend temporally. They run in parallel, and their outputs are concatenated — exactly the standard multi-head concat. There is only one attention operation per block, same depth as Model 1, but each head sees a different neighborhood.

Recall scaled dot-product attention: Attention(Q,K,V) = softmax(QKT/√dk)V. The only thing that determines which tokens a head can attend to is the set of keys K (and values V) it is given. Model 4's move: for the spatial heads, restrict K and V to tokens sharing the same temporal index as the query; for the temporal heads, restrict K and V to tokens sharing the same spatial index. The queries Q are unchanged. So a spatial head computes softmax(Q·KsT/√dk)Vs and a temporal head computes softmax(Q·KtT/√dk)Vt, where the subscript denotes the restricted key/value set.

Why this is even cheaper in the right sense

Crucially, Model 4 has the same number of parameters and the same number of layers as Model 1, but each head's attention is over a smaller key set (n_s or n_t instead of N). Because the spatial and temporal heads run in parallel within the one attention op — not in series as in Model 3 — Model 4 fuses space and time in a single step per block, without the extra serial reshape-attend of Model 3. It is, in a sense, the most "native" multi-head factorization: it changes nothing about the Transformer except which keys each head looks at.

A 3-token worked dot-product split

Suppose a query token sits at (time=1, space=2). Its spatial key set is all tokens with time=1; its temporal key set is all tokens with space=2. If the field has n_t=3 times and n_s=4 spaces, a spatial head compares the query against 4 keys, a temporal head against 3 keys — never against the full 12. Two small softmaxes replace one large one.

python
# Model 4: factorized dot-product — half the heads spatial, half temporal
def factorized_dot_product(Q, K, V, idx_t, idx_s, n_heads=12):
    # Q,K,V: [B, n_heads, N, d_k];  idx_t/idx_s give each token's (time, space) id
    half = n_heads // 2
    Ys = attn_restrict(Q[:, :half], K[:, :half], V[:, :half],   # spatial heads:
                       same_index=idx_t)   # keys must share the query's TIME
    Yt = attn_restrict(Q[:, half:], K[:, half:], V[:, half:],   # temporal heads:
                       same_index=idx_s)   # keys must share the query's SPACE
    return torch.cat([Ys, Yt], dim=1)   # concat heads, then W_O (standard MHA)

Chapter 6: Regularizing on Small Data

There is a reason ViTs were not used for video earlier: Transformers have no built-in inductive bias. A CNN bakes in translation-equivariance and locality — it "knows" nearby pixels are related before it sees any data. A Transformer assumes nothing; it must learn even basic locality from examples. That is why ViT needed enormous datasets (JFT-300M) to beat CNNs. Video datasets are far smaller — Kinetics-400 has ~250k clips, Epic-Kitchens and Something-Something fewer still. Train a fresh ViViT on that and it overfits catastrophically. The paper's third contribution is the recipe that makes it work anyway.

Lever 1 — Initialize from an image model

The single most important trick: do not train from scratch — initialize from a ViT pre-trained on images (ImageNet-21k or JFT). The spatial parts of ViViT are architecturally identical to image ViT, so their weights transfer directly. But two pieces have no image counterpart and need careful initialization:

Filter inflation, in numbers. Suppose the image patch filter is a 2D weight E2D mapping a 16×16×3 patch to a 768-vector. For a tubelet with t=2, the 3D filter E3D maps a 2×16×16×3 brick. Inflation sets E3D[τ] = E2D/t for each of the t=2 temporal slices. Then at step 0, the tubelet token equals the average of the 2D embeddings of its t frames — a sensible warm start that exactly reproduces the image model when the frames are identical.

Levers 2-4 — Strong regularization

Even with a good init, mid-sized video data overfits a big Transformer. ViViT stacks the standard heavy-regularization toolkit, and the ablations show each one matters:

Chapter 7: Experiments & Ablations

ViViT was evaluated on the standard video classification benchmarks: Kinetics-400 and Kinetics-600 (large, appearance-heavy action recognition), Epic-Kitchens-100 (egocentric, fine-grained verbs and nouns), Something-Something v2 (motion-centric — actions defined by how things move, where appearance alone is useless), and Moments in Time. The headline: ViViT, particularly the large model initialized from JFT, set state-of-the-art accuracy on Kinetics-400/600 at the time, surpassing the strong 3D-CNN baselines (SlowFast and friends).

What the four models cost and deliver

The paper's model comparison is the practical takeaway. Reported qualitatively:

Tubelet ablation. Tubelet embedding (t>1, fuse motion at input) consistently beats uniform frame sampling. And a bigger temporal tubelet means fewer tokens, so accuracy trades against compute: larger t is cheaper but coarser; the paper's sweet spot is a small t (2). Central-frame initialization of the 3D filter slightly edges out plain inflation in some settings.

Test-time views. Like 3D-CNN evaluation protocols, ViViT samples multiple temporal "views" (and spatial crops) from each test video and averages their predictions. More views = higher accuracy at higher inference cost; the paper reports the standard multi-view evaluation. This is why a "model accuracy" number is always paired with a view count.

The motion test: Something-Something v2

Kinetics is notoriously appearance-biased — many actions are guessable from a single frame ("playing guitar" has a guitar in it). Something-Something is the opposite: classes like "moving X up" vs "moving X down" are identical in appearance and differ only in motion direction. This is the dataset that punishes a model with weak temporal modeling. Here the more expressive models (joint, and factorized variants with genuine cross-time attention) and tubelet embedding matter most — appearance shortcuts do not save you.

A worked multi-view inference count. Say inference uses 4 temporal views × 3 spatial crops = 12 forward passes per clip, each averaged. If a single forward pass of Model 1 costs C, the joint model pays 12C at test time. Model 2's per-pass cost is roughly 15× smaller (Chapter 2), so it pays ~12C/15 ≈ 0.8C — a full multi-view evaluation of the factorized encoder is cheaper than a single pass of the joint model. That is the deployment argument for Model 2.

Chapter 8: The Variant Explorer

This is the payoff chapter. Pick a clip configuration (temporal tubelet size, clip length) and a model variant, and watch the whole pipeline come alive: tokens get carved, attention gets factorized, and the cost meter responds. Everything here is computed live from the same arithmetic we derived — N = (T/t)·n_s, and each model's attention-comparison count. Use it to build a gut feel for how the four design choices trade compute against cross-time expressivity.

Drive it a moment. Set the model to M1 Joint and drag clip frames up — the comparison count rockets, because n_t lives inside the square. Now switch to M2 Fact-Enc with the same long clip — the count barely moves, because the temporal stage is cheap and the spatial cost is per-frame. Then raise the tubelet t: every model gets cheaper at once, because fewer tokens means a smaller N for all of them — the tokenizer and the attention factorization are independent levers you can pull together.

Chapter 9: Limits & Connections

ViViT proved that a pure-attention model can lead video understanding — but it also exposed the tensions the next wave of papers would resolve. Knowing the limits is as important as knowing the method.

Limitations

What came next

Video Swin Transformer brought hierarchical, windowed attention (local spatio-temporal windows that shift between layers) — recovering the CNN's locality bias inside a Transformer and improving accuracy/efficiency. MViT (Multiscale Vision Transformers) built a feature pyramid by pooling keys/values, giving the model multiple resolutions like a CNN backbone. TimeSformer (concurrent with ViViT) explored the same factorization space — "divided space-time attention" is essentially Model 3. And the masked-autoencoding line (VideoMAE) attacked the data-hunger directly: self-supervised pretraining on video itself, reducing the dependence on image pretraining.

Cheat sheet

Continue learning

• Attention Is All You Need — the Transformer and scaled dot-product attention ViViT is built on. Read this first if attention is still fuzzy.
• Vision Transformer (ViT) — the image patch-embedding model ViViT generalizes to 3D. (If this path 404s, see the Architecture Atlas ViT overview.)
• Self-Attention from scratch — the Q/K/V mechanic underlying every variant here.
• Positional Encoding — why permutation-invariant attention needs position, and how ViViT repeats image positions across time.
• ← Back to all Veanors — browse more paper walkthroughs.