ActionFormer: Localizing Moments of Actions

Chapter 1: The Key Insight

ActionFormer's core insight: treat every moment as an action candidate and let the model decide. Instead of generating proposals (two-stage) or using pre-defined anchor windows (single-stage), just classify every time step and regress its boundaries directly.

This is the temporal equivalent of what FCOS did for object detection: instead of anchor boxes, every spatial location predicts a class and distances to the box edges. ActionFormer does the same thing in 1D: every temporal location predicts a class and distances to the action's onset and offset.

The Representation

For every time step t in the video, ActionFormer outputs:

• p(a_t): C probability values, one per action category (via sigmoid — multi-label, not softmax). If all are below threshold, t is background.
• d_s^t: Distance from t to the action's onset (start). Always positive.
• d_e^t: Distance from t to the action's offset (end). Always positive.

From these, decoding an action is trivial: start = t − d_s, end = t + d_e, label = argmax p(a_t).

Chapter 2: The Architecture

ActionFormer follows a clean encoder-decoder design. The encoder is a multiscale Transformer that builds a temporal feature pyramid. The decoder is a lightweight convolutional network with shared heads.

Projection Layer

The input features X = {x₁, ..., x_T} (each x_t ∈ R^D_in, e.g., D_in = 2048 for I3D) are projected to D = 512 dimensions using a shallow 1D convolutional network with ReLU activation:

Adding convolutions before the Transformer was found to help incorporate local context and stabilize training.

Transformer Encoder (L = 7 blocks)

Each block applies local multi-headed self-attention (MSA) followed by an MLP, with LayerNorm before each and residual connections after:

where α^l and ᾱ^l are per-channel learnable scaling factors, and ↓ is optional 2x downsampling via strided depthwise 1D convolution. The first 2 blocks operate at full resolution T; the remaining 5 blocks downsample by 2x each, creating pyramid levels at T, T/2, T/4, T/8, T/16, T/32.

Decoder

Two separate but architecturally identical convolutional heads are attached to every pyramid level:

• Classification head: 3 layers of 1D conv (kernel=3, LayerNorm on first 2, ReLU) → sigmoid → C action probabilities per time step.
• Regression head: Same architecture → ReLU at output (distances are positive) → (d_s, d_e) per time step.

Both heads share weights across all pyramid levels. A coarser level naturally has larger regression targets (longer actions live at coarser resolutions), so the regression range is normalized by the feature stride of each level. This normalization is crucial: without it, the regression head would need to output values spanning from 2 time steps (short action at level 1) to 500+ time steps (long action at level 7) — an impossible dynamic range for a shared network. With stride normalization, all targets are in the same manageable range.

Chapter 3: The Temporal Feature Pyramid

The feature pyramid is ActionFormer's secret weapon. It elegantly solves the problem of variable-duration actions by distributing different temporal scales across different levels.

Why a Pyramid?

A "long jump" lasts 3 seconds. A "cooking" activity lasts 3 minutes. If you only look at the finest temporal resolution, detecting a 3-minute activity requires a receptive field of 180 time steps (at 1 fps). That's impractical with local attention. But at 32x downsampled resolution, those 180 steps become just 6 — easily captured by a window of 19.

The Structure

With L = 7 Transformer blocks and 2x downsampling on the last 5:

Each level specializes in a different duration range. Short actions (a few seconds) are detected at levels Z¹-Z². Long actions (several minutes) are detected at Z⁶-Z⁷. The regression range per level is normalized by the stride, roughly doubling with each level.

Chapter 4: Local Self-Attention

Standard (global) self-attention computes similarity between every pair of time steps. For a video with T = 2048 time steps, that's T² = 4 million pairs. This is O(T²D) in both time and memory — prohibitively expensive for long videos.

ActionFormer's solution: local self-attention. Each time step only attends to its W nearest neighbors (window size W = 19). This reduces complexity from O(T²D) to O(W²TD) — and since W is a small constant (19 << T), this is effectively O(TD).

The Mechanism

For each time step t, self-attention is computed only within the window [t - W/2, t + W/2]. The queries, keys, and values are computed as usual:

The key observation: temporal context beyond a certain range is less helpful for action localization. Whether something happened 5 minutes ago rarely matters for detecting the current action. But what happened 10 seconds ago matters a lot. Local attention captures exactly this inductive bias.

The ablation confirms this: replacing local attention (W=19) with global attention actually decreases average mAP by 0.9% on THUMOS14 (from 66.8% to 65.9%). Global attention dilutes the model's focus with irrelevant distant context and increases computation. Local attention is both cheaper and better.

No Positional Encoding

Surprisingly, ActionFormer works better without positional encoding. The ablation shows that adding sinusoidal or learned positional encodings slightly hurts performance. The reason: the projection convolutions and strided depthwise convolutions in the Transformer blocks already leak positional information (convolutions are inherently position-aware through their structure). Adding explicit positional encoding is redundant and slightly harmful.

This also means ActionFormer can process any length video at inference time — there's no positional encoding that was trained for a specific sequence length.

Chapter 5: The Decoder Heads

The decoder is deliberately simple — the heavy lifting is done by the Transformer encoder. Two lightweight 1D convolutional heads are applied to every time step at every pyramid level.

Classification Head

Three layers of 1D convolutions (kernel size 3, 512 channels), with LayerNorm on the first two layers and ReLU activation. The final layer outputs C channels (one per action category) followed by sigmoid. This is a multi-label formulation: each category is an independent binary decision. A time step can theoretically be labeled with multiple actions simultaneously (though this is rare in practice).

Regression Head

Same architecture as classification. Output: 2 channels (d_s, d_e) followed by ReLU to ensure positive distances. The distances are normalized by the stride of the current pyramid level, so the network always predicts in "level-relative" units regardless of the actual temporal scale.

The Training Loss

The total loss per video has two terms:

Where:

• L_cls = Focal Loss for C-way binary classification. Focal loss down-weights easy negatives (background moments that the model is already confident about), focusing training on hard cases near action boundaries. Parameters: γ = 2.0, α = 0.75.
• L_reg = DIoU Loss for distance regression, only applied at positive time steps (1_ct = 1 if t is within an action). DIoU combines intersection-over-union with a distance penalty, giving gradients even when predicted and target segments don't overlap.
• T₊ = total number of positive samples across the video. Normalizing by T₊ prevents batches with many actions from dominating.
• λ_reg = 1.0 by default (equal weight for classification and regression).

Why DIoU Loss for Regression?

Standard L1 or L2 regression on (d_s, d_e) has a problem: when the predicted segment and the ground truth segment don't overlap at all, L1/L2 still gives a gradient, but it doesn't reflect how close they are in terms of temporal overlap. DIoU (Distance IoU) loss combines two signals:

• IoU component: Measures the temporal overlap between the predicted and ground truth segments. Ranges from 0 (no overlap) to 1 (perfect match).
• Distance penalty: Penalizes the normalized distance between the centers of the two segments. This gives useful gradients even when IoU = 0 (no overlap), guiding the prediction toward the target.

The regression loss is only applied to positive samples (time steps within an action). Background moments have no meaningful (d_s, d_e) target.

Chapter 6: Training Details

ActionFormer is trained end-to-end with Adam optimizer on the combined classification + regression loss. Several training tricks are critical for performance.

Center Sampling

Not every time step within an action is labeled as positive. Only time steps near the center of the action (within α = 1.5 strides of the center) are considered positive for training. This has two benefits:

• It focuses the model on predicting from the most informative position (the center, where both boundaries are equally far away).
• It reduces confusion at action boundaries, where the model might be uncertain about which action a time step belongs to.

Center sampling adds +1.4% average mAP on THUMOS14. It doesn't affect inference — at test time, every time step still makes a prediction.

Training Configuration

Variable-Length Handling

Videos vary wildly in duration: THUMOS14 videos can be 2-6 minutes, ActivityNet videos can be 5-30+ minutes. During training, sequences are padded or cropped to a fixed maximum length, with proper attention masking to prevent padded positions from influencing the output. This is equivalent to training with sliding windows.

An important finding: varying the maximum input sequence length during training has little impact on performance. The model generalizes to different lengths at inference because (a) there's no positional encoding, and (b) local self-attention doesn't depend on absolute position — only relative context within the window matters.

Chapter 7: Inference & Decoding

At inference, the full video sequence is fed through ActionFormer in a single forward pass. Since there's no positional encoding, the model handles any video length.

Step 1: Forward Pass

The pre-extracted features X = {x₁, ..., x_T} go through the encoder, producing the pyramid Z = {Z¹, ..., Z^L}. The shared heads produce (p(a_t), d_s^t, d_e^t) for every time step t at every level l.

Step 2: Decode Candidates

Each time step t at level l produces a candidate action:

With L = 7 levels and a video of T = 2048 steps, this produces roughly T + T/2 + T/4 + ... = ~4000 candidates. Most will be background (low p(a_t)) and are filtered by a confidence threshold.

Step 3: Soft-NMS

Multiple candidates may overlap in time (especially from adjacent time steps and adjacent pyramid levels). Soft-NMS suppresses overlapping detections by decaying their confidence scores rather than hard-deleting them. This is gentler than standard NMS and preserves detections of overlapping actions (e.g., simultaneous "running" and "dribbling").

Multi-Level Decoding

Each pyramid level independently produces detections. A short "kick" action detected at level Z¹ might also produce a (weaker) detection at level Z³. These cross-level duplicates are handled naturally by Soft-NMS: the higher-confidence detection survives, and the duplicate's score gets decayed. This is why a single round of Soft-NMS suffices — it handles both within-level and cross-level duplicates simultaneously.

The regression targets at each level are clipped to a predefined range. Level Z¹ (stride 1) only predicts actions with d_s + d_e in [0, 4) time steps. Level Z⁷ (stride 32) predicts actions with d_s + d_e in [128, ∞). This prevents a fine-resolution level from trying to predict a 5-minute action (which would require huge regression values) and vice versa.

Computational Cost

At inference, the dominant cost is the Transformer encoder's self-attention. With local attention (window W = 19) and L = 7 blocks, the cost per block is O(W · T^l · D) where T^l is the sequence length at level l. Since T^l halves at each level (after the first two), the total cost is roughly O(W · T · D · L) — linear in video length. A 30-minute video processed in a single forward pass takes well under a second on a modern GPU, making ActionFormer practical for real-world applications.

Chapter 8: Results

ActionFormer establishes new state of the art on all three major TAL benchmarks, surpassing both two-stage and single-stage methods by large margins.

THUMOS14 (20 action categories, 413 videos)

ActionFormer achieves 71.0% mAP at tIoU=0.5 — +14.1 absolute percentage points over the best prior single-stage method (AFSD at 55.5%) and +14.1 over the best two-stage method (MUSES at 56.9%).

EPIC-Kitchens 100 (Egocentric, 100 hours)

On the challenging egocentric dataset, ActionFormer outperforms BMN/G-TAD by +13.5 average mAP.

Key Ablations (THUMOS14)

Chapter 9: Connections

ActionFormer bridges ideas from object detection, NLP sequence modeling, and video understanding into a clean, unified design.

Relation to FCOS (Object Detection)

ActionFormer is essentially FCOS adapted to 1D. FCOS classifies every spatial location and regresses distances to the bounding box edges. ActionFormer classifies every temporal location and regresses distances to action boundaries. The feature pyramid, center sampling, and anchor-free design all come directly from the FCOS lineage (FPN → RetinaNet → FCOS → ActionFormer).

The mapping is almost one-to-one: FPN's 2D feature maps become 1D temporal sequences. FCOS's 2D center sampling becomes 1D center sampling. RetinaNet's focal loss is adopted without changes. Even the regression normalization by stride follows FCOS exactly. This direct inheritance is a strength — decades of object detection research transfers cleanly to the temporal domain.

Relation to DETR / TadTR

TadTR (a concurrent work) uses a DETR-style set prediction approach for TAL: learned object queries, Hungarian matching, no feature pyramid. ActionFormer takes the opposite approach: no learned queries, no bipartite matching, just dense per-moment prediction with NMS. ActionFormer significantly outperforms TadTR (66.8% vs 46.6% on THUMOS14), suggesting that the dense prediction paradigm is more natural for TAL than set prediction.

Relation to Swin Transformer

ActionFormer's local self-attention within a hierarchical pyramid directly parallels Swin Transformer's window attention with shifted windows in 2D. Both use local attention for efficiency and rely on the hierarchical structure for global context. ActionFormer skips the "shifted window" trick (unnecessary in 1D with the pyramid providing cross-level communication).

Influence on Downstream Work

• TemporalMaxer (2023): Showed that replacing ActionFormer's local attention with simple max-pooling still achieves competitive results, questioning how much of the gain comes from attention vs. the pyramid. This sparked an important debate: is the Transformer truly necessary, or is the multiscale architecture the real innovation?
• ActionFormer+ / TriDet: Extensions with improved feature aggregation and boundary refinement that build directly on ActionFormer's codebase.
• TAL community standard: ActionFormer's codebase became the de facto baseline for TAL research, with most subsequent papers comparing against it. The clean, modular design made it easy to swap components in and out for ablation studies.
• Egocentric video understanding: ActionFormer's strong results on EPIC-Kitchens 100 (the largest egocentric action dataset) opened a new direction: applying TAL models to first-person video from AR headsets and wearable cameras.

Cheat Sheet