Is Space-Time Attention All You Need for Video?

Chapter 1: From Clip to Space-Time Tokens

Everything in TimeSformer starts the same way ViT does, just extended along time. Let's nail down the exact tensor shapes, because the entire cost analysis later rides on them.

The input clip is a tensor of shape [F, H, W, 3] — F frames, each H×W pixels, 3 color channels. In the paper, the standard setting is F = 8 frames sampled from the video, each H = W = 224 pixels.

Step 1: patchify each frame

Split every frame into non-overlapping patches of size P×P, with P = 16. A 224×224 frame yields (224/16)×(224/16) = 14×14 = N = 196 patches per frame. Each patch is a little square of 16×16×3 = 768 pixel values, flattened into a vector x of length 768.

So a clip becomes N×F = 196×8 = 1568 patches total. Each patch has two coordinates: a spatial index p (which of the 196 grid locations) and a temporal index t (which of the 8 frames). Hold onto that pair (p, t) — divided attention is entirely about which coordinate you hold fixed when you attend.

Step 2: linear embedding

A single learned matrix E of shape [D, 3P²] projects each flattened patch into a D-dimensional token. In the paper D = 768 (the ViT-Base width). The embedded token for patch (p, t) is:

where e^pos is a learned positional embedding that tells the otherwise order-blind attention where (which spatial cell) and when (which frame) this patch came from. TimeSformer adds a separate learned positional embedding for space and for time, so the model can distinguish "patch 5 in frame 2" from "patch 5 in frame 7." A learned classification token z_(0,0) is prepended, exactly as in ViT; its final-layer representation is fed to a linear classifier to predict the action label.

Worked shape trace

Let's carry one concrete clip through, with the paper's numbers:

Notice the token count exploding: 4 frames of a 7×7 grid is already 196 tokens; the real model's 8 frames of a 14×14 grid is 1568. Because attention is quadratic in sequence length, that 1568 is the number whose square we are trying to avoid computing.

Chapter 2: Joint Space-Time Attention

The most obvious way to apply attention to the 1568-token sequence is to just... apply attention. Let every token attend to every other token, with no regard for whether the other token is in the same frame or a different one. The paper calls this joint space-time attention (denoted "ST"). It is the direct video analog of ViT.

Recall scaled dot-product attention. For a query token at position (p, t), its query vector is compared against the key of every token (p', t') in the whole clip:

The softmax runs over all N×F other tokens. The updated representation is the attention-weighted sum of all value vectors:

where d_h = D/h is the per-head dimension (with h heads). This is genuinely powerful: the query for the cup-patch in frame 1 can directly attend to the cup-patch in frame 8, the hand-patch in frame 4, the background in frame 6 — anything. Full space-time reach, layer 1.

The cost, made painfully concrete

The attention score matrix has one row and one column per token, so it is (NF)×(NF). Each entry is a dot product over d_h dimensions. The number of softmax comparisons each query must do is the row length, NF:

Plug in the paper's numbers, N = 196, F = 8:

That 2.46 million entries is per attention head, per layer. With h = 12 heads and 12 layers, the attention maps alone are enormous, and the activation memory to backprop through them is what actually OOMs your GPU. The paper notes that joint space-time attention is impractical to train at the resolutions and clip lengths they want — and worse, as we'll see in the ablations, it doesn't even pay for that cost with accuracy.

Click around: every query attends to all 27 tokens. Now imagine that grid at 196 patches and 8 frames. The visual makes the quadratic blow-up viscerally obvious — every patch is wired to every other patch, in every frame.

Chapter 3: Divided Space-Time Attention

Here is the central idea of the paper. Instead of one attention operation over all N×F tokens, apply two attention operations in sequence within each block: first attend over time, then attend over space. The authors denote this "T+S" and call it divided space-time attention.

Step 1 — temporal attention (over time, same location)

For a query token at spatial location p, frame t, attend only to the tokens at the same spatial location p across all frames t' = 1..F. That is, the cup-patch in frame 4 attends to the cup-patch in frames 1, 2, ..., 8 — the same grid cell, walking through time.

The softmax now runs over only F keys (8), not NF (1568). This produces an intermediate, temporally-mixed representation z^time_(p,t) for each token. Intuitively: "how did the content at my spot in the frame evolve over the clip?"

Step 2 — spatial attention (over space, same frame)

Feed z^time into a second attention. Now each query at (p, t) attends only to the tokens in its own frame t — all N spatial patches p' = 1..N, at fixed time t.

This softmax runs over N keys (196), not NF. The result z^space is the block's output. Intuitively: "given my time-mixed content, what else is in my frame right now?"

Crucially, the temporal and spatial attentions use separate learned weights — distinct W_Q, W_K, W_V projections (and separate LayerNorm and residual connections) for the time step and the space step. So divided attention isn't merely cheaper; it adds capacity exactly where the two kinds of reasoning differ. This is the second reason it beats joint attention, not just the FLOP count.

Worked example: where does a query's attention go?

Take the cup-patch at spatial cell p = 50, frame t = 4, on the 8-frame, 196-patch model.

204 comparisons per token instead of 1568 — a 7.7× reduction per token, and the gap widens fast as F grows because the time cost is only F, not NF. Chapter 4 turns this into the exact asymptotic formula.

Chapter 4: The Cost Math, Derived

Let's derive the exact complexity of each scheme so the "divided is cheaper" claim is not hand-waving. We count the dominant term: the number of query-key comparisons summed over all queries (the size of all attention matrices). Let N = patches per frame, F = frames.

Joint space-time

There are NF queries. Each compares against all NF keys. Total comparisons:

Quadratic in N and quadratic in F. Doubling the clip length (F → 2F) quadruples the cost.

Divided space-time

Two attentions per block. Time step: NF queries, each compares against only F keys (same spatial cell, all frames):

Space step: NF queries, each compares against only N keys (same frame, all cells):

Add them:

The ratio

Divide joint by divided to see the speedup factor:

This is the harmonic-mean-flavored quantity NF/(N+F). When N ≫ F (the usual video case — hundreds of patches, a handful of frames), N + F ≈ N, so the ratio ≈ F. With N = 196, F = 8:

A 7.7× reduction in attention comparisons. And it scales better: push to F = 32 frames (TimeSformer-L) and joint cost grows as F² = 16× from the 8-frame baseline, while divided grows only modestly because its dominant term N²F is linear in F. The longer the clip, the more divided wins.

Chapter 5: Five Attention Schemes

TimeSformer doesn't just compare two options — it studies a small zoo of five attention factorizations, each trading reach against cost differently. Understanding all five clarifies why divided (T+S) is the sweet spot.

How to read the zoo

Space-only (S) is the floor: it's just ViT applied independently to each frame, with the frame results pooled at the end. It has no mechanism to relate a patch in frame 1 to frame 8 inside the attention, so it cannot model motion within a block — it can only learn appearance. Useful as a baseline to measure how much temporal attention actually buys.

Joint (ST) is the ceiling on reach but also on cost — the full quadratic.

Divided (T+S) factorizes the joint operation into a time marginal and a space marginal. It keeps global reach (composed across two steps) at near-linear cost. This is the winner.

Axial (T+W+H) takes factorization further — it splits the spatial step itself into a width pass and a height pass, so each attention is even narrower. Cheaper still, but the paper finds it slightly less accurate than T+S: over-factorizing makes the per-step job too narrow and the composition loses something. Local-global (L+G) mixes a local window with a sparse global step; it lands between T+S and ST in both cost and accuracy.

The footprints make the tradeoff legible: S touches one frame, ST touches everything, T+S touches a column plus a frame (a plus-sign over space-time), axial touches a column plus a width-line plus a height-line (a sparse cross). Divided's plus-sign is the minimal footprint that still reaches both axes.

Chapter 6: Experiments & Results

The architecture is a clean stack: 12 Transformer blocks (ViT-Base backbone), each containing a divided space-time attention (temporal attention + spatial attention, each with its own residual + LayerNorm) followed by an MLP. The cls-token output feeds a linear classifier over the action vocabulary. Training is standard cross-entropy on action labels, initialized from ImageNet pretraining (the spatial weights inherit ViT's ImageNet features; the temporal attention weights are added fresh and initialized to zero so the model starts as pure per-frame ViT and learns to use time).

Benchmarks

TimeSformer was evaluated on the major action-recognition datasets, each stressing a different skill:

The Something-Something-V2 result is the most telling. Its labels are deliberately designed so a single frame is uninformative — "moving something up" and "moving something down" share frames; only the order distinguishes them. A model that ignores time scores near chance on the distinction. TimeSformer's divided temporal attention is what lets it succeed where space-only collapses.

Efficiency and scaling

Two practical findings stand out. First, the divided model trains and runs far cheaper than joint, consistent with the Chapter 4 math — making it feasible to use more frames and higher resolution. The paper exploits this with TimeSformer-L, a longer/larger-input variant (more frames, higher resolution) that pushes accuracy further precisely because divided attention keeps the cost manageable. Second, accuracy improves as you feed the model more frames — a Transformer with global temporal reach genuinely benefits from longer clips, where a local 3D conv's gains saturate sooner.

Chapter 7: The Ablations

The ablation study is where the paper earns its title. Three findings carry the argument.

Finding 1 — divided beats joint, not just ties it

Holding the backbone and training fixed, the paper compares the five schemes. Divided (T+S) is the most accurate — above joint (ST), above space-only (S), above axial (T+W+H) and local-global (L+G). This is the surprising result: the cheaper factorization is also the more accurate one. Two reasons, both established earlier:

• Capacity placement. Divided uses separate learned projections for temporal and spatial attention — extra parameters spent exactly where time-reasoning and space-reasoning differ. Joint forces one shared projection to do both jobs.
• Optimization. The narrower softmaxes (over F or N keys, not NF) are easier to learn and less prone to the diffuse, hard-to-train attention distributions that very long joint sequences produce.

Finding 2 — temporal attention matters

Compare divided (T+S) against space-only (S). The gap is large on temporally-demanding datasets (Something-Something) and small on appearance-driven ones (Kinetics). This isolates the contribution of the temporal step: it is precisely the motion-reasoning component, and you can measure its value by how the S→T+S gap varies with how temporal the dataset is.

Finding 3 — pretraining and clip length

Because the model has weak inductive bias (no conv locality), it is data-hungry. ImageNet pretraining is important — it gives the spatial attention a strong appearance prior so the model doesn't have to learn vision from scratch on limited video. And accuracy rises with the number of frames and the spatial resolution, which the efficient divided attention makes affordable.

The narrative the ablations build: the title's question — "is space-time attention all you need?" — gets a yes, but with a precise qualifier. You don't need the joint form; you need the divided form, with separate temporal and spatial attention, warm-started from image pretraining. That specific recipe is the contribution.

Chapter 8: Attention Explorer

This is the payoff: a fully interactive model of the paper's core mechanism. Pick any token in a small clip as the query, choose an attention scheme, and watch exactly which keys it attends to, with the live comparison count. Use it to feel the difference between joint and divided attention, and to confirm the Chapter 4 cost math by direct observation.

Things to try. (1) Set the query to a center patch, switch joint ↔ divided, and watch the total comparison count: divided should be dramatically lower, and the ratio should track NF/(N+F). (2) Crank F to 6 and N to 25 (5×5): joint's total rockets while divided crawls — that's the F² vs near-linear scaling, seen live. (3) Switch to Space (S): the query's footprint collapses to a single frame, and you can see it has no path to other frames inside this layer — the structural reason S can't model motion.

Chapter 9: Limitations & Connections

TimeSformer settled a real question — pure self-attention, properly factorized, is a strong and scalable video backbone — but it has clear limits and a rich set of descendants.

Limitations

• Still quadratic in patches. Divided attention removes the F² term but the spatial step is still O(N²) per frame. High-resolution video (large N) remains expensive; this motivated windowed/hierarchical successors (Video Swin).
• Data-hungry. Weak inductive bias means it leans heavily on ImageNet pretraining. Without large-scale pretraining it underperforms conv models on small video datasets.
• 2D patch embedding only. All temporal fusion is in attention; the embedding sees no motion. ViViT explored tubelet (3D) embeddings as an alternative way to inject early temporal structure.
• Fixed clip sampling. Frames are sampled uniformly; the model has no built-in mechanism to attend to the most informative frames or to handle very long videos beyond the sampled clip.

Where it leads

Related lessons on this site

Cheat sheet