Video Swin Transformer

Chapter 1: The Locality Inductive Bias

An inductive bias is an assumption you build into a model's architecture that constrains what it can express, so it can learn the right answer from less data. A convolution's inductive bias is locality and translation equivariance: a feature detector slides over the image and only looks at a small neighborhood. That assumption is why CNNs are so data-efficient on images — you don't waste capacity learning that distant, unrelated pixels matter.

Global self-attention deliberately throws that bias away. Every token can attend to every other token, so the model has to learn from scratch which connections matter. On enormous datasets (JFT-300M) ViT learns this fine. On the smaller video datasets available in 2021 it overfits, because the hypothesis space is too big.

Concretely, Video Swin keeps the 2D Swin recipe and extends the locality prior to the time axis. The two ingredients of Swin are:

To see why locality + shifting recovers global reach, picture a token's receptive field. In layer 1 it sees its own window. After the shift in layer 2, the window has moved, so it now overlaps tokens from a different layer-1 window — it can pull in information that crossed the old boundary. Each window+shift pair roughly doubles the reach. This is exactly the convolutional intuition that stacking 3×3 convolutions builds a large receptive field, applied to attention windows.

Worked example: how many layers to go global?

Suppose a spatial axis has 56 tokens and the window is M = 7 wide. A window covers 7 tokens. After the first window layer the query reaches 7 tokens. The shift of M/2 ≈ 3 lets the next layer's window overlap a neighbor, extending reach by about one window (7) every two layers. To cover 56 tokens you need roughly 56/7 ≈ 8 windows of growth, so about 2 × 8 = 16 layers — comfortably inside the depth of stage 3 (which has many transformer blocks). The point: global reach is reached by depth, not by paying O(N²) in any single layer.

Chapter 2: 3D Patch Partition — tokenizing a clip

Before any attention happens, we have to turn a video tensor into tokens. The input clip is a tensor of shape [T, H, W, 3] — T frames, height H, width W, 3 colour channels. In the paper's default setup T = 32, H = W = 224.

Video Swin's 3D patch partition cuts the clip into non-overlapping 3D patches of size 2×4×4×3 — 2 frames deep, 4×4 in space, all 3 channels. Each such cube of 2·4·4·3 = 96 raw values becomes one token. A linear embedding then projects those 96 numbers to the model dimension C (96 for the Tiny model).

Let's trace the shapes for the default clip. T/2 = 16 temporal tokens, H/4 = 56 spatial rows, W/4 = 56 spatial columns. So after patch partition we have a token grid of 16 × 56 × 56, each token a C-dimensional vector. That is 16·56·56 = 50,176 tokens — a lot, which is exactly why we will never run global attention over all of them.

Why 2 frames deep, not 1?

The temporal patch size of 2 is a deliberate choice. A patch size of 1 would make each token belong to exactly one frame, deferring all temporal mixing to attention. Bundling 2 adjacent frames into each token means motion between those two frames is captured at the very first layer, by the linear embedding itself. It also halves the temporal token count (32 frames → 16 temporal tokens), cutting cost. The paper uses a small temporal patch so that fine motion is not blurred away.

Why 4×4 in space, smaller than ViT's 16×16?

Swin is hierarchical: it starts with a fine token grid and progressively merges patches to build a feature pyramid, like a CNN. A small initial patch (4×4) gives a high-resolution stage 1 (56×56), which downsampling later coarsens to 28×28, 14×14, 7×7. ViT, by contrast, uses one big 16×16 patch and a single resolution throughout. The hierarchy is what makes Swin usable as a general backbone for detection and segmentation, and Video Swin inherits it.

python
import torch
import torch.nn as nn

# 3D patch partition = strided Conv3d (kernel == stride)
patch_embed = nn.Conv3d(3, 96, kernel_size=(2,4,4), stride=(2,4,4))

clip = torch.randn(1, 3, 32, 224, 224)  # [B, 3, T, H, W]
tokens = patch_embed(clip)                  # [1, 96, 16, 56, 56]
tokens = tokens.permute(0,2,3,4,1)        # [1, 16, 56, 56, 96]  (B, T', H', W', C)
# 16*56*56 = 50176 tokens — far too many for global attention

Chapter 3: 3D Window Multi-Head Self-Attention

Now the central mechanism. We have a token grid of shape [T', H', W', C]. 3D window-based multi-head self-attention (3D W-MSA) partitions this grid into non-overlapping 3D windows of size P×M×M — default 2×7×7 — and runs ordinary multi-head self-attention independently inside each window.

Each window therefore contains P·M·M = 2·7·7 = 98 tokens. Self-attention inside a window builds a 98×98 attention matrix — tiny and fixed, regardless of how big the clip is. The number of windows scales with the clip size, but the cost per window is constant, so the total cost is linear in the number of tokens.

The cost, derived

Let h = T', and the spatial grid be h×w tokens (using w loosely for both spatial dims). Global MSA over N = T'·H'·W' tokens costs about 4NC² + 2N²C — the second term, 2N²C, is the quadratic attention that kills us. Window MSA over windows of P·M·M tokens costs about 4NC² + 2(P·M·M)NC. The attention term is now 2(P·M·M)NC: linear in N, because the window size PMM is a small constant.

For N = 50,176 and window 2·7·7 = 98, the attention term shrinks from 2N²C to 2·98·NC — a reduction factor of N/98 ≈ 512×. That is the whole ballgame.

Relative position bias (3D)

Inside a window, plain self-attention is permutation-invariant — it does not know where within the window each token sits. Video Swin adds a learned 3D relative position bias B to the attention logits, one per pair of relative offsets along (time, height, width):

Because a P×M×M window has relative offsets ranging over (2P−1)×(2M−1)×(2M−1) possibilities, B is read from a small learned table of that size. For 2×7×7 the table is 3×13×13 = 507 entries per head — cheap, and it encodes the within-window geometry the attention scores otherwise ignore.

python
def window_partition_3d(x, P, M):
    """x: [B, T, H, W, C] -> windows: [num_windows*B, P*M*M, C]"""
    B, T, H, W, C = x.shape
    # group each axis into (#windows_along_axis, window_extent)
    x = x.view(B, T//P, P, H//M, M, W//M, M, C)
    # bring the three window-extent axes together, then flatten
    windows = x.permute(0,1,3,5,2,4,6,7).reshape(-1, P*M*M, C)
    return windows   # attention runs over the P*M*M axis, per window

# default: P=2, M=7  ->  each window has 2*7*7 = 98 tokens
# a 16x56x56 grid yields (16/2)*(56/7)*(56/7) = 8*8*8 = 512 windows

Chapter 4: The 3D Shift — connecting windows

Window attention has a problem you can probably already see: tokens on opposite sides of a window boundary never attend to each other. The action you care about — a hand crossing from one window into the next — would be split by a wall the model cannot see across. If windows never moved, the receptive field would be permanently capped at one window, no matter how deep the network.

Swin's fix, lifted to 3D: alternate between two layer types. Even layers use 3D W-MSA (regular windows). Odd layers use 3D SW-MSA — shifted window MSA — where the window grid is displaced by half a window along each axis before partitioning. The default shift is (P/2, M/2, M/2) = (1, 3, 3) for the 2×7×7 window.

Because the shifted grid is offset by half a window, the new windows straddle the boundaries of the previous (regular) windows. A token that was on the left edge of one regular window is now in the middle of a shifted window, sitting next to tokens that used to be in the neighboring regular window. So in the shifted layer, those previously-separated tokens finally attend to each other. Cross-window information flows.

The cost of the shift: ragged windows

Shifting creates a headache. After offsetting the grid by half a window, the windows at the edges of the volume are no longer full P×M×M cubes — they are partial. A naive implementation would now have to handle many differently-sized windows, which is slow and ugly. For the 2×7×7 / shift (1,3,3) case, a straightforward shift turns each axis's window count from k into k+1, with the extra windows being fragments.

Worked example: where do tokens land after a shift?

Take a 1D axis of 8 tokens [0..7], window M = 4, shift = M/2 = 2. Regular windows: {0,1,2,3} and {4,5,6,7}. The boundary sits between token 3 and token 4 — they never meet. After a cyclic roll by −2 (move everything 2 to the left, wrapping), the sequence becomes [2,3,4,5,6,7,0,1]. Now window 1 = {2,3,4,5}: tokens 3 and 4, formerly split, are together. Window 2 = {6,7,0,1}: this one mixes the true tail (6,7) with the wrapped head (0,1), which are not neighbors — that is precisely where the mask will block attention.

Chapter 5: Cyclic Shift & the Attention Mask

This is the most intricate engineering in the architecture, and it is the second Code Lab's target. The shifted-window layer is implemented in three steps: cyclic shift → window partition → masked attention → reverse cyclic shift.

Step 1: cyclic shift

We roll the token volume by the negative shift along each axis. In code this is torch.roll(x, shifts=(−P/2, −M/2, −M/2), dims=(1,2,3)). Tokens that fall off one edge wrap around to the other. After this roll, partitioning with the regular (unshifted) grid is equivalent to having used shifted windows on the original volume — and crucially, every window is full-sized. No padding, no ragged edges, no change in window count.

Step 2: the problem the roll creates

The roll glues together tokens from opposite ends of the volume. Inside a window that sits on the wrap-around seam, you now have two groups of tokens that are not spatial-temporal neighbors — they only ended up together because of the roll. If we let them attend freely, the model would hallucinate a connection between, say, the bottom of the clip and the top. We must forbid attention across the seam while still allowing attention within each genuine group.

Step 3: the mask

Each token in a seam window gets a group id — which original region it came from. Two tokens may attend only if they share a group id. We build an attention bias matrix where allowed pairs get 0 and forbidden pairs get a large negative number (−100, effectively −∞). Adding this to the attention logits before softmax drives the forbidden weights to zero:

The group ids come from slicing each axis into three regions: the bulk (size M−shift), the to-be-wrapped slice (size shift), and the boundary, then labeling cells by which (time, height, width) region they belong to. Because the mask depends only on the geometry, it is precomputed once and reused for every clip.

Step 4: reverse the shift

After masked attention, we undo the roll with the opposite shift so tokens return to their true positions before the next layer. The whole shifted-window mechanism is thus transparent to the rest of the network — it takes a token grid in, gives a token grid out, same shape, with cross-window information mixed in.

python
def shifted_window_attention(x, P, M, shift, attn, mask):
    """x: [B, T, H, W, C].  shift = (P//2, M//2, M//2)."""
    # 1) cyclic shift: roll so shifted windows become regular full windows
    x = torch.roll(x, shifts=(-shift[0], -shift[1], -shift[2]), dims=(1,2,3))
    # 2) partition into P*M*M windows (now all full-sized)
    wins = window_partition_3d(x, P, M)          # [nW*B, P*M*M, C]
    # 3) attention WITH the precomputed mask (-inf across the seam)
    out = attn(wins, mask=mask)                 # forbidden pairs -> 0 weight
    # 4) un-partition and REVERSE the cyclic shift
    x = window_reverse_3d(out, P, M, x.shape)
    x = torch.roll(x, shifts=(shift[0], shift[1], shift[2]), dims=(1,2,3))
    return x

Chapter 6: The Full Backbone — stages & patch merging

We now have the block: a regular-window layer followed by a shifted-window layer, each wrapped in LayerNorm, residual connections, and a 2-layer MLP. The full Video Swin backbone is a hierarchy of four stages, each a stack of these blocks, with patch merging for spatial downsampling between stages — exactly mirroring 2D Swin.

Patch merging: building the pyramid

Patch merging downsamples the spatial grid by 2× between stages (it does not touch the temporal axis). It groups each 2×2 neighborhood of spatial tokens, concatenates their features into a 4C vector, then linearly projects to 2C. So height and width halve while channels double — the standard CNN feature-pyramid trade. Temporal resolution is left alone because video clips are usually short in time and you do not want to collapse motion too aggressively.

Note the temporal dimension stays at 16 throughout — only space is pyramided. The final stage outputs a 16×7×7 grid of 768-dim tokens. Global average pooling over all those tokens, then a linear classifier, produces the action label.

Model variants and the inflation trick

Video Swin comes in T/S/B/L sizes that mirror 2D Swin's depths and widths. The masterstroke is initialization by inflation: a Video Swin is initialized from a 2D Swin pre-trained on ImageNet. The 2D weights transfer directly except for two places. The patch-embedding conv gains a temporal dimension (the 2D filter is replicated and divided by the temporal patch size, so its mean output matches), and the relative-position-bias table is replicated across the new temporal offsets. This lets Video Swin inherit strong image features and reach state-of-the-art with far less video data than a from-scratch global model.

Chapter 7: Results & Ablations

The claim Video Swin has to back up: locality beats globality for video, given realistic data. The headline numbers (at publication) on the three standard benchmarks:

The two most important framings of these results:

The ablation that matters most: the shift

The single most load-bearing ablation removes the shifted-window mechanism — use only regular 3D windows, never shifted. Accuracy drops, because without the shift, information can never cross window boundaries and the receptive field stays capped at one window. This directly confirms the Chapter 4 story: windowing alone is local; the shift is what makes the network global.

Window size and temporal patch ablations

Two more design knobs were studied. First, the window size: 2×7×7 was a sweet spot — too small a window starves each layer of context; too large a window erodes the efficiency advantage and the locality prior. Second, the temporal dimension of the window matters: a window that spans multiple frames (P = 2) lets joint space-time attention happen inside the window, which is exactly what distinguishes Video Swin from a per-frame 2D Swin. The 3D joint window outperformed treating time separately, validating the joint (non-factorized) design.

Chapter 8: Window Explorer — the mechanism, live

This is the payoff. Everything from Chapters 3–5 in one interactive view of a token grid. Pick a query token; choose regular or shifted windows; set the window size and shift. The simulation partitions the grid into 3D windows, shows which tokens the query can attend to, applies the cyclic shift and seam mask when shifted, and reports the cost. Break it: shrink the window and watch the receptive field starve; turn off the shift and watch the query get boxed in by boundaries.

Things to try, each of which maps to a claim earlier in the lesson:

• Toggle the shift with the query near a window boundary. In Regular mode it is trapped inside its window. In Shifted mode the window straddles the boundary, and the query suddenly reaches tokens that were on the other side — the cross-window flow from Chapter 4.
• Shrink M to 2. The attended set collapses; cost per window plummets but so does each layer's context. This is why too-small windows starve the model (Chapter 7's window-size ablation).
• Grow M to 4. Each window now covers more of the grid; the cost readout rises. Push it to the grid size and you have recovered global attention — and its quadratic cost.
• Read the cost line. It shows (P·M·M)·N for window attention vs N² for global. The ratio is the speedup, made concrete.

Chapter 9: Limits & Connections

Video Swin is a beautifully economical idea, but it is not the last word. Its assumptions and costs are worth naming clearly.

Limitations

Where it sits in the lineage

Cheat sheet

Related lessons

Build out the surrounding map of ideas:

• Attention Is All You Need — the scaled dot-product attention and multi-head machinery that Video Swin runs inside each window.
• Vision Transformer (Gleam) — patch tokenization and global attention on images, the design Swin reins in with locality.
• Transformer (Gleam) — the foundational block: residual + LayerNorm + MLP that wraps every (S)W-MSA layer.
• DiT (Gleam) — another patch-token transformer backbone, showing how universal the tokenize-then-attend recipe is.
• Scaling Vision Transformers — the data-hungry global-attention regime that the locality bias is designed to escape.

"What I cannot create, I do not understand." — and you have now built the 3D window partition and the seam mask by hand, which is the whole engine.