CogVideoX: Expert Transformer for Video

Chapter 1: The Key Insight

CogVideoX solves all three barriers with a unified design philosophy: treat video as a first-class 3D signal, not a stack of 2D images.

The architecture has three interlocking components:

1. 3D Causal VAE — compresses video jointly across space (4× each axis) and time (4×), giving a total 16× compression in token count. Unlike 2D VAEs that encode each frame independently (causing flicker), this 3D VAE shares information across frames during encoding, producing temporally smooth latents.
2. Expert Transformer with Expert Adaptive LayerNorm — text and video tokens are concatenated into a single sequence, then processed through full 3D attention. But each modality gets its own LayerNorm parameters (the "expert" part), so the network can handle the radically different feature distributions without interference.
3. Progressive Training — instead of training at full resolution from the start, CogVideoX trains at 256px first, then 512px, then 768px. At each stage it also increases video duration. This curriculum saves compute and builds robust representations bottom-up.

Chapter 2: The 3D VAE

A raw 10-second video at 16 fps and 768×1360 resolution has 160 frames. If you encode each frame independently with a standard 2D VAE (like SDXL's, which does 8× spatial compression per axis), you get 160 × 96 × 170 × 4 ≈ 10 million latent values. That's still enormous for a transformer to chew on.

CogVideoX's 3D Causal VAE compresses along all three dimensions: 8× on each spatial axis (same as SDXL) and 4× on the temporal axis. So those 160 frames become 40 temporal steps, and the total latent count drops to 40 × 96 × 170 × 16 ≈ 10 million — wait, that's the same? Not quite. The key difference is the quality: because temporal neighbors share information during 3D convolution, the latents encode smooth motion rather than independent snapshots. This eliminates the frame-to-frame flickering that plagues 2D VAE approaches.

Architecture

The encoder and decoder are symmetric: stages of ResNet blocks interleaved with downsampling (encoder) or upsampling (decoder) layers. Some stages perform 3D downsampling (spatial + temporal), while others do only 2D (spatial). The temporal compression uses causal convolution — all padding is placed at the beginning of the time axis, so the encoding at time t can only depend on frames at time ≤ t. This lets the VAE handle streaming and ensures the first frame can also encode images (single-frame "videos").

Ablation: How much compression?

The authors tested multiple compression settings:

Moving from 2D to 3D (Baseline → B) cuts flickering from 93.2 to 86.3 while improving PSNR. But pushing compression too far (variant E, 16×16×8) makes convergence extremely difficult — the model can't learn to reconstruct from such a compressed bottleneck.

Chapter 3: The Expert Transformer

Once the 3D VAE compresses the video into latent tokens z_vision, and T5 encodes the text prompt into z_text, we need a transformer that can fuse both modalities deeply. CogVideoX's solution is elegantly simple: concatenate the two token sequences and run them through a single transformer stack with full 3D attention — but give each modality its own normalization parameters.

Why not just concatenate and go?

Text embeddings from T5 and video latents from the 3D VAE live in radically different feature spaces. Text embeddings might have values clustered around [-2, 2], while video latents might range [-0.5, 0.5]. If you apply a single LayerNorm to the concatenated sequence, the normalization statistics are dominated by whichever modality has higher variance, distorting the other.

Expert Adaptive LayerNorm

The solution: split the LayerNorm into two "experts." Following DiT's design, the diffusion timestep t drives an adaptive modulation (scale γ and shift β) via a small MLP. But instead of one set of (γ, β), there are two:

• Vision Expert AdaLN — applies modulation to video hidden states only
• Text Expert AdaLN — applies modulation to text hidden states only

After normalization, the tokens are recombined and fed through shared attention and FFN layers. This means the heavy computation (attention, feedforward) is shared, while the lightweight normalization is modality-specific. The parameter overhead is minimal — just two extra sets of (γ, β) per layer.

3D Full Attention

With the concatenated [text ; video] sequence, CogVideoX applies full 3D attention: every token attends to every other token, regardless of modality, spatial position, or temporal position. This means a video patch at frame 50, pixel (100, 200) can directly attend to the text token "lightning" and to the video patch at frame 1, pixel (100, 200).

This is computationally expensive — the sequence length is (text tokens) + (T/4 × H/8 × W/8) — but FlashAttention makes it tractable. The payoff is enormous: objects maintain direct line-of-sight to themselves across time, eliminating the information-leaking problem of separated 2D+1D attention.

3D Rotary Position Embedding (3D-RoPE)

Each video latent lives at a 3D coordinate (x, y, t). CogVideoX extends standard RoPE to three dimensions by independently applying 1D-RoPE to each axis, allocating 3/8 of the hidden channels to x, 3/8 to y, and 2/8 to t. The results are concatenated along the channel dimension. This captures both spatial locality and temporal ordering with a single relative positional encoding.

Chapter 4: Text-Video Alignment

Getting a video model to faithfully follow a text prompt is harder than it sounds. The prompt "a cat wearing a tiny hat jumps over a sleeping dog" requires the model to bind attributes (tiny hat → cat, sleeping → dog), maintain object identity across frames, and orchestrate a temporal narrative (jumping is not standing). CogVideoX's alignment strategy has two parts: the text encoder and the fusion mechanism.

T5 Text Encoder

CogVideoX uses T5-XXL (an encoder-decoder language model with 4.7B parameters) to encode text prompts. Unlike CLIP text encoders used in Stable Diffusion (which are trained on short image captions), T5 was pretrained on diverse language tasks and handles complex, multi-sentence prompts much better. The text is encoded into a sequence of 226 tokens (padded/truncated to a fixed length) with 4096-dimensional embeddings.

Why T5 over CLIP? CLIP maxes out at 77 tokens and was trained with contrastive loss on (image, caption) pairs — it captures visual concepts well but struggles with compositional language ("a red ball on top of a blue cube"). T5 handles long, structured prompts because it was trained as a general-purpose language model.

Deep Fusion via Concatenation

The text embeddings are not injected via cross-attention (as in Stable Diffusion) or via a conditioning mechanism (as in DALL-E). Instead, they're concatenated directly with the video latent tokens along the sequence dimension. This means text tokens participate in every self-attention layer alongside video tokens — there's no information bottleneck.

In cross-attention designs, text information can only influence video through the K/V projections of dedicated cross-attention layers. In CogVideoX's full-attention design, text tokens and video tokens are peers — each text token can attend to every video token and vice versa, at every layer.

Video Captioning Pipeline

The quality of text-video alignment depends critically on the quality of training captions. Most web videos have short, vague descriptions ("funny cat video"). CogVideoX built a dense captioning pipeline:

1. Generate short captions using the Panda70M model
2. Extract key frames and generate dense image captions with CogVLM
3. Use GPT-4 to summarize all frame-level captions into a coherent video description
4. Fine-tune a LLaMA-2 on GPT-4 summaries for scalable captioning
5. Finally, fine-tune an end-to-end model (CogVLM2-Caption) for fast inference

This pipeline produces detailed, temporally-aware captions that describe what happens, not just what's visible in a single frame.

Chapter 5: Progressive Training

Training a 5B-parameter video model at 768×1360 resolution from scratch would require astronomical compute. CogVideoX uses a progressive training curriculum that starts cheap and scales up gradually.

Resolution Progression

The model trains through multiple stages, each at increasing resolution:

At each resolution, the aspect ratio is preserved (the short side is resized to the target, the long side scales proportionally). This means the model sees diverse aspect ratios throughout training — it doesn't just learn square crops.

Multi-Resolution Frame Pack

Training videos come in all lengths: 2 seconds, 5 seconds, 10 seconds. Fixed-duration training wastes data — you'd have to truncate long videos and discard short ones. CogVideoX's Frame Pack strategy (inspired by Patch'n Pack from NaViT) packs videos of different durations and resolutions into the same batch. Each video is patchified and positioned using 3D-RoPE, so the model naturally handles mixed shapes within a single forward pass.

Duration Progression

The 3D VAE is also trained progressively: first on 17-frame videos, then fine-tuned on 161-frame videos using context parallelism. The transformer follows a similar schedule — shorter clips first, then longer ones. This prevents the model from being overwhelmed by the quadratic attention cost of long sequences before it has learned basic visual representations.

Chapter 6: Explicit Uniform Sampling

Diffusion models train by sampling a random timestep t for each data point, adding noise at that level, and learning to denoise. The standard training objective is:

where t is uniformly sampled from [1, T]. In practice, each GPU rank independently samples a random t. With, say, 64 GPUs and T = 1000 timesteps, each batch only covers 64 random timesteps — and due to randomness, some ranges get oversampled while others get missed entirely.

The problem with random sampling

The loss magnitude varies dramatically across timesteps: near t = 0 (almost no noise), the loss is small. Near t = T (almost pure noise), the loss is large. When the sampled timesteps cluster unevenly, the aggregate loss fluctuates wildly between batches — not because the model is unstable, but because the timestep distribution is noisy. These fluctuations slow convergence and make learning rate tuning harder.

The fix: Explicit Uniform Sampling

CogVideoX divides the range [1, T] into n equal intervals (where n is the number of data-parallel ranks). Each rank samples uniformly within its assigned interval. Rank 0 samples from [1, T/n], rank 1 from [T/n + 1, 2T/n], and so on.

Chapter 7: Data Pipeline

A video generation model is only as good as its training data. CogVideoX trains on approximately 35 million filtered video clips (averaging ~6 seconds each) plus 2 billion images from LAION-5B and COYO-700M. But raw web video is messy — full of screen recordings, lecture videos, heavily edited content, and clips with no meaningful motion. The data pipeline is CogVideoX's unsung hero.

Video Filtering

The team defined six categories of "negative" video that hurt training:

• Editing artifacts — Videos with noticeable post-processing, special effects, or re-editing that distort natural dynamics
• Lack of motion connectivity — Videos with hard cuts, spliced segments, or sequences made from static images
• Low quality — Blurry, shaky, or poorly lit footage
• Lecture type — A person talking with minimal motion (low visual diversity)
• Text dominated — Videos with large text overlays or primarily textual content
• Noisy screenshots — Screen recordings with poor quality

They manually labeled 20,000 videos across these categories, then trained 6 specialized classifiers (based on Video-LLaMA) to filter the entire dataset. Additionally, optical flow scores and aesthetic scores are computed for all videos — thresholds are dynamically adjusted during training to ensure clips have sufficient motion and visual quality.

Video Captioning Pipeline

The captioning pipeline described in Chapter 4 deserves emphasis here: going from a noisy web video with a title like "LOL CAT COMPILATION #47" to a dense description like "A grey tabby cat wearing a small red hat leaps over a sleeping golden retriever on a beige carpet, landing softly and turning to look at the camera" is what makes CogVideoX's text-following so strong.

Chapter 8: Results

CogVideoX was released in two sizes: 2B and 5B parameters. Both are text-to-video models with additional image-to-video variants. Let's see how they stack up.

Automated Benchmarks

CogVideoX-5B achieves state-of-the-art in 5 out of 7 VBench metrics, and competitive results in the remaining two. The metrics that matter most for video quality are Dynamic Degree (how much motion is in the video) and Dynamic Quality (motion quality without sacrificing visual fidelity). Many competing models "cheat" by generating near-static videos that score high on visual quality but low on dynamism.

CogVideoX-5B's Dynamic Degree of 70.95 crushes Gen-2 (55.47) and Pika (46.69) — it generates videos with significantly more motion, which is the whole point of video generation.

Human Evaluation

The team compared CogVideoX-5B against Kling (one of the best closed-source models at the time) across four dimensions:

CogVideoX-5B wins on every dimension, with particularly strong gains in instruction following (+35%) and physics simulation (+19%). This validates the Expert AdaLN's deep text-video fusion.

Chapter 9: Connections

CogVideoX sits at the intersection of several major research threads in video generation. Here's how it connects to the broader landscape.

What CogVideoX got right

• 3D-first design — treating video as inherently 3D (not 2D + time afterthought) at every level: VAE, attention, positional encoding
• Minimal modality-specific parameters — Expert AdaLN adds only normalization parameters per modality, not entire transformer streams
• Data engineering as architecture — the captioning pipeline and filtering system are as important as the model design
• Progressive training — a practical necessity that also improves final quality

Open questions

• Can the 3D VAE be pushed further (16×16×8 compression) with better training recipes?
• How does the approach scale to minute-long videos?
• Can the Expert AdaLN framework extend to audio-video or video-action generation?