VideoMamba for Efficient Video Understanding

Chapter 1: The Key Insight

State space models process a sequence of L tokens in O(L) time and memory. Not O(L²). Not O(L·log L). Just O(L). If you double the video length, cost doubles — not quadruples.

The insight behind VideoMamba is disarmingly simple: take the Mamba SSM, which was designed for 1D language sequences, and apply it to the flattened spatiotemporal token sequence of a video.

There is one important subtlety. The original Mamba is causal — token t can only see tokens 1 through t−1. This makes sense for language (you predict the next word from previous words). But for video classification, every frame matters equally. You want frame 50 to know about both frame 10 and frame 90. So VideoMamba uses a bidirectional variant: it runs Mamba forward and backward, then sums the outputs. We will examine this in Chapter 4.

Chapter 2: Mamba Recap

Before we see how VideoMamba applies Mamba to video, let's make sure the Mamba mechanism itself is clear. State Space Models (SSMs) are continuous systems that map an input signal x(t) to an output y(t) through a hidden state h(t):

Here A is the evolution matrix (how the hidden state evolves on its own), B controls how the input gets injected, and C controls how we read out the output. Think of it as a differential equation: the hidden state is a dynamical system that gets "pushed" by the input and "read" by the output.

Discretization: From Continuous to Tokens

We don't have a continuous signal — we have discrete tokens x₁, x₂, ..., x_L. So we discretize the continuous ODE using a time-step parameter Δ:

Then the discrete recurrence is:

This is just a linear recurrence. Processing L tokens takes L steps — O(L) time. No pairwise comparisons needed.

The "Selective" in Selective State Space

Traditional SSMs use fixed A, B, C — the same parameters for every input. Mamba's key innovation is making B, C, and Δ input-dependent. For each token x_t, a linear projection produces B_t, C_t, and Δ_t specific to that token. This is the Selective Scan Mechanism (S6).

The Mamba Block

A single Mamba block wraps the SSM in a gated architecture:

1. Input x goes through a linear projection, expanding the channel dimension by 2×
2. One branch passes through a 1D convolution (local context), then the SSM
3. The other branch passes through SiLU activation (gating)
4. The two branches are multiplied element-wise, then projected back down

The 1D convolution before the SSM provides local context (like a small attention window), while the SSM captures long-range dependencies. The multiplicative gating lets the network control information flow.

Chapter 3: Video Tokenization

To feed a video into Mamba, we first need to convert it into a sequence of tokens. VideoMamba uses tubelet embedding — a 3D convolution that carves the video into small spatiotemporal cubes.

The Input

A video clip is a 4D tensor: X_v ∈ R^{3 × T × H × W}, where 3 is RGB channels, T is the number of frames, and H × W is the spatial resolution. For example, a 16-frame clip at 224×224: shape [3, 16, 224, 224].

Tubelet Embedding

A single 3D convolution with kernel size 1×16×16 and stride 1×16×16 converts the video into non-overlapping patches:

where L = t × h × w, with t = T (temporal kernel is 1, so every frame is preserved), h = H/16, and w = W/16. For our 16-frame, 224×224 example: L = 16 × 14 × 14 = 3,136 tokens, each a C-dimensional vector.

Position Embeddings

SSMs are position-sensitive — the recurrence naturally encodes ordering. But VideoMamba still adds explicit position embeddings to help the model distinguish spatial locations and temporal positions:

• Spatial position embedding p_s ∈ R^{(hw+1) × C}: shared across all frames. Tells the model "this token is at row 3, column 7 of the frame."
• Temporal position embedding p_t ∈ R^{t × C}: shared across all spatial positions within a frame. Tells the model "this token is from frame 5."

A learnable CLS token X_cls is prepended, just like in ViT. After all layers, the CLS token's representation is used for classification.

Chapter 4: Bidirectional Mamba for Video

This is the core technical contribution. The original Mamba is causal: it scans left-to-right, and each token can only attend to previous tokens. This is perfect for autoregressive language modeling (predict the next word), but video classification needs global context — every token should know about every other token, regardless of position.

Bidirectional Mamba (B-Mamba)

VideoMamba uses the bidirectional Mamba block from Vision Mamba. The idea is simple:

1. Forward scan: run the SSM from token 1 to token L. Token t sees context from tokens 1..t.
2. Backward scan: run a separate SSM from token L to token 1. Token t sees context from tokens t..L.
3. Sum: add the forward and backward outputs element-wise.

Now every token has context from the entire sequence, in both directions. And both scans are O(L), so the total cost is still O(L).

Scan Order: How to Flatten 3D into 1D

A video has three dimensions: width, height, and time. To feed it into a 1D SSM, we must choose a scan order. The authors tested four strategies:

Why Not Just Use Divided Attention?

TimeSformer and ViViT use divided attention: spatial attention within each frame, then temporal attention across frames. This reduces O(T²·N²) to O(T·N² + T²·N), but it is still quadratic in both T and N separately. VideoMamba's B-Mamba is O(T·N) — linear in the total token count. And it processes spatial and temporal information jointly, not in separate passes.

Chapter 5: The Architecture

VideoMamba deliberately follows the vanilla ViT design as closely as possible. No downsampling layers, no hierarchical features, no window attention. Just a stack of identical blocks. This is called an isotropic architecture.

The Full Pipeline

1. 3D Patch Embedding: Conv3D (1×16×16) maps the video to L tokens of dimension C
2. CLS Token + Position Embeddings: prepend CLS, add spatial + temporal embeddings
3. L × B-Mamba Blocks: each block is normalization → bidirectional Mamba → residual
4. Classification Head: layer norm on CLS token → linear projection to class logits

Model Variants

The SSM uses default Mamba hyperparameters: state dimension N = 16, expansion ratio 2. VideoMamba-Ti has only 7 million parameters — dramatically smaller than TimeSformer-L's 121M or ViViT-L's 311M.

Self-Distillation for Scaling

A problem emerged during experiments: larger VideoMamba models overfit. VideoMamba-Base (98M parameters) performed worse than VideoMamba-S (26M). The same issue was observed in VMamba. The solution is self-distillation:

1. Train a smaller model (VideoMamba-S) to convergence — it generalizes well
2. Use it as a "teacher" to train the larger model (VideoMamba-M)
3. Align the student's final feature map to the teacher's via L2 loss

Chapter 6: Training

VideoMamba uses a two-stage training pipeline: pretrain on images, then fine-tune on video.

Stage 1: ImageNet Pretraining

All models are first trained on ImageNet-1K (1.28M images, 1000 classes) for image classification. This provides a strong spatial feature extractor before any video data is seen.

• Optimizer: AdamW with weight decay 0.05
• Schedule: Cosine learning rate decay over 300 epochs, LR = 1e-3
• Warmup: 5 epochs of linear warmup
• Precision: BFloat16 for stability (no EMA needed)
• Stochastic depth: 0 / 0.15 / 0.5 for Ti / S / M
• Self-distillation: for VideoMamba-M, a pretrained VideoMamba-S serves as teacher (L2 loss on final features)

Stage 2: Video Fine-tuning

The ImageNet-pretrained model is fine-tuned on video datasets. The key trick: the 2D spatial position embeddings transfer directly because VideoMamba's 3D patch embedding with temporal kernel 1 produces the same spatial tokens as 2D ViT. Only the temporal position embeddings are new.

• Kinetics-400: 10-second clips, scene-related actions. LR scaled as 2e-4 · batch/256
• Something-Something V2: 4-second clips, temporal-sensitive actions (opening, closing). LR scaled as 4e-4 · batch/256
• Stochastic depth: 0.8 for VideoMamba-M
• Epochs: 50 (K400) or 30 (SSv2)

Stage 2b: Masked Pretraining (Optional)

For even better performance, VideoMamba can be pretrained with masked alignment inspired by UMT. This masks 80% of video tokens and aligns the unmasked tokens with CLIP-ViT features.

A key finding: the masking strategy matters for Mamba. Because the B-Mamba block includes a 1D convolution before the SSM, it prefers contiguous unmasked tokens. Random masking (which works well for transformers) disrupts the 1D conv's local receptive field.

Chapter 7: Results

VideoMamba is evaluated against 3D CNNs, video transformers, and hybrid models across multiple benchmarks. The results show a consistent pattern: competitive or better accuracy at dramatically lower compute.

Kinetics-400 (Scene-Related Actions)

* With masked pretraining using CLIP-400M teacher.

Something-Something V2 (Temporal-Sensitive Actions)

SSv2 requires understanding fine-grained temporal differences (e.g., "opening" vs "closing"). VideoMamba-M outperforms TimeSformer by +5.9% and ViViT-L by +3.0%. With masked pretraining, it surpasses even MViTv2-B.

Chapter 8: The Long Video Advantage

This is where VideoMamba truly shines. Long videos — cooking demonstrations, movies, procedural tasks — produce tens of thousands of tokens. Transformers either cannot process them at all or must rely on expensive pre-extracted features. VideoMamba handles them end-to-end.

Breakfast (Cooking Activities)

77 hours of cooking videos, 1,712 clips, 10 activity categories. Videos average several minutes long.

COIN (Procedural Tasks)

11,827 videos across 180 procedural tasks, averaging 2.36 minutes.

LVU (Long-Form Video Understanding)

30K movie clips, 1–3 minutes each, with 9 diverse tasks: relationship prediction, scene classification, director identification, and more. Even tiny VideoMamba-Ti beats all previous methods on most tasks.

Why Does Linear Complexity Matter Here?

At 64 frames and 224×224, the total token count is 12,544. The attention matrix for a transformer would be 12,544 × 12,544 ≈ 157 million entries — per layer, per head. At 64 frames and 384×384, it is 36,864 tokens and 1.36 billion entries. TimeSformer literally runs out of memory.

VideoMamba's state vector is fixed-size (dimension N = 16) regardless of sequence length. Whether processing 1,000 or 100,000 tokens, the memory footprint per layer is constant. Only the sequential scan cost grows — linearly.

Chapter 9: Connections

VideoMamba sits at the intersection of state space models and video understanding. Let's map where it fits.

Relation to Mamba (S6)

VideoMamba is a direct application of the Mamba selective scan mechanism to video. The core SSM operator is unchanged — the contribution is showing it works for 3D spatiotemporal sequences, not just 1D text. The bidirectional extension comes from Vision Mamba (Vim).

Relation to Vision Mamba (Vim)

Vision Mamba applied bidirectional Mamba to 2D images. VideoMamba extends this to 3D video by: (a) using 3D patch embedding (tubelets), (b) adding temporal position embeddings, and (c) analyzing scan orders for spatiotemporal tokens. It also simplifies Vim by removing the middle CLS token and RoPE, gaining +0.8% on ImageNet.

Relation to TimeSformer / ViViT

TimeSformer and ViViT are the attention-based video backbones that VideoMamba aims to replace. Both use divided spatiotemporal attention to reduce quadratic cost. VideoMamba achieves better accuracy with linear cost and dramatically fewer parameters (7M vs 121M–311M).

Relation to VideoMAE

VideoMAE is a self-supervised pretraining method for video transformers using masked autoencoding. VideoMamba's masked alignment is inspired by VideoMAE but requires different masking strategies (row/attention masking vs random/tube) due to Mamba's 1D conv sensitivity.

Cheat Sheet