Meta’s Movie Gen writes a 16 second HD video from one sentence. A plain 30 billion parameter transformer starts from pure noise and walks to a clip in 50 small steps.
Learn how one paper builds a cast of models that make video, sound, personalized people and edits, and how you could build the same pieces.
Drag the steps down to 1, then press Generate. Then try 4 and 50. Watch what a clip looks like when the walk from noise is too coarse. Then we build, piece by piece, the autoencoder, the flow, the transformer, the data, the sampler and the tests.
You need a feel for what a neural network is (a function with adjustable numbers) and a little Python for the code. We build the rest from zero.
A paper Gleam of the MIT MMAI follow-along, a reading of Lecture 9, Multimodal Generation. Paper: Movie Gen: A Cast of Media Foundation Models (arXiv 2410.13720), by the Movie Gen team at Meta. Every idea is re-taught here in our own words and devices.
A toy, not the paper’s model. Each of 240 points stands for one random start; each blue ring is one possible real clip (12 of them, on a circle). The model here is the exact ideal velocity for that data, so the only error left is how coarse the steps are. The paper’s sampler is first-order Euler and uses 50 steps (Section 3.4.2); a real video has millions of numbers to move, so it needs far more steps than this 2-number toy.
Chapter 0
See why a video is far too big to generate directly, and meet the five models that share the work
You type one sentence: “A porcupine wearing a tutu, performing a ballet dance on a stage.” Sixteen seconds later a video plays. The porcupine turns, the tutu swings, the stage lights move. Nobody filmed it. A model made every frame.
This paper, Movie Gen from Meta, shows how to build that model. It is also a model of a good paper: it tells you the shapes, the settings, the data and the failures. This lesson follows it so that you could rebuild each part.
A frame is one still picture. A pixel is one dot in it, stored as 3 numbers (red, green, blue). Movie Gen makes clips of up to 16 seconds at 16 frames per second. That is 256 frames.
Take one frame at 768 × 768 pixels. It holds 768 × 768 × 3 = 1,769,472 numbers. Multiply by 256 frames and one clip holds 452,984,832 numbers. That is about 453 million, for 16 seconds.
Now the problem. A transformer is the network behind chat models. Its core step, attention, compares every piece of the input with every other piece. If every one of the 453 million numbers were a piece, one attention layer would make about 100 quadrillion comparisons (453 million squared, halved). Even with one piece per pixel, the count is far too large. The model must work on something much smaller.
Earlier systems met this limit with a chain. One model made a tiny, short video. A second model added frames in between. A third model made the picture sharper. Each step was a separate model with its own errors. The Movie Gen team wanted one clean path.
The title says a cast. A cast is a group of actors, each with a role. The paper does not build one giant model that does everything. It builds a few models that share ideas and hand work to each other.
A foundation model is a large model trained on huge, broad data, so you can adapt it to many tasks later. Movie Gen has two foundation models. Movie Gen Video has 30 billion parameters (the adjustable numbers inside a network) and makes images and video from text. Movie Gen Audio has 13 billion and makes sound for a video.
Three more models come from post-training: further training of a finished model for a new skill. Personalized Movie Gen Video reads a photo of a face and puts that person in the video. Movie Gen Edit changes a clip from a written instruction. A Spatial Upsampler with 7 billion parameters turns the 768 pixel video into full HD, 1080p.
Pick each member below. Read what goes in, what comes out, and where this lesson explains it.
Choose a model. The left boxes are what you give it, the right box is what you get back.
Sizes and tasks are from the paper’s Section 2 and Sections 3 to 6: 30B, 7B and 13B parameters, 16 seconds at 16 frames per second, 48 kHz audio. The box layout is a simplification.
The paper’s Figure 2 gives the order of work for the video model. First, train on small 256 pixel images only. Next, train on images and videos together at 256 pixels. Then train on images and videos at 768 pixels with up to 16 second clips. Then finetune on a small set of very good videos. Last, add skills: personalization and editing.
The order is a cost trick. Small pictures are cheap, so the model learns what the world looks like first. Long, sharp videos are costly, so the model sees them last. Chapter 4 puts numbers on this.
The paper states its big finding in one sentence: a simple transformer trained with flow matching (Chapter 2) gives high quality video or audio when you scale the data, the compute and the model size together. The word simple matters. Each choice below picks the plain option unless the plain option fails.
Chapter 1
Shrink a video eight times along each of three axes before the big model sees it, and fix the black spots that shrinking creates
Think of a ZIP file for pictures. You do not send the raw bytes. You send a small code, and the receiver rebuilds the picture. A variational autoencoder (VAE) is a learned version of this. An encoder network turns a picture into a small block of numbers. A decoder network turns the block back into a picture. The small block is the latent, and the space of all such blocks is the latent space.
The big generator never sees pixels. It makes latents, and the decoder turns them into pixels at the end. That is why this shrinking step comes first.
Movie Gen calls its autoencoder the TAE, for temporal autoencoder. A raw video V has shape T′ × 3 × H′ × W′: frames, colour channels, height, width. The TAE writes a latent X of shape T × C × H × W. It compresses 8 times along each of time, height and width: T′/T = H′/H = W′/W = 8. The latent has C = 16 channels.
Work the 16 second clip. Raw: 256 × 3 × 768 × 768, which is 452,984,832 numbers. Latent: 32 × 16 × 96 × 96, which is 4,718,592 numbers. That is 96 times fewer numbers: 512 times fewer positions, but 16 channels instead of 3.
Why not compress even more? Each step of compression costs detail. The paper also reports that more channels helps both rebuilding and generating, so it keeps 16.
A bonus: because the latent keeps the original frame rate in a compressed form, the model can make video at its normal speed. The team did not need a frame-interpolation model (a model that invents the frames between two frames), which earlier systems used.
The team started from an existing image autoencoder (the one from Stable Diffusion work) and inflated it: they added time parameters. After each 2D spatial convolution (a small sliding filter over height and width) they added a 1D temporal convolution (a filter over frames). After each spatial attention they added a 1D temporal attention. Spatial weights start from the pretrained image model.
Downsampling in time uses a strided convolution: a filter that moves two frames at a time, so each pass halves the frame count. Three such passes make 8. Upsampling repeats each frame, then convolves.
Strided convolution has a useful side effect. Any length of video can go in, including a single image. The paper treats an image as a one-frame video. The encoder turns T′ frames into ⌈T′/8⌉ latent frames. The decoder gives 8 × ⌈T′/8⌉ frames back, and the extra “spurious” frames are thrown away. Training mixes 1 batch of images to 3 batches of videos.
Move the slider and watch the blocks of 8 frames.
Each small tick is one raw frame. Each block of 8 becomes one latent frame. Drag the slider to change the clip length. Watch the last block when the length is not a multiple of 8.
Exact arithmetic from the paper’s Section 3.1.1 and 3.1.3 for 768 × 768 px at 16 frames per second, C = 16 and 2 × 2 patches (Chapter 3). The frame ticks are a drawing, not data.
The first TAE had a strange fault. Some generated videos showed a dark spot in one corner. The team looked at the latents and found a few positions with very large values, which they call latent dots. They think the network learned a shortcut: store key global information in a few big numbers. When decoded, those numbers show up as spots.
They did not change the network. They added a penalty to the loss, the outlier penalty loss (OPL), which punishes any latent value that sits far from the average.
The max with 0 means a value inside 3 spreads of the average costs nothing. Outside, the cost grows with the distance. Worked toy: if the spread is 1 and one value sits 5 away from the average, that value adds 5 − 3 = 2 to the sum. A value 2 away adds 0. The paper uses loss weight 1e5, a very large number, so the network learns not to hide things in dots. For video, the time axis is folded into the batch so each frame is penalised alone. The fix removes the dots and slightly improves rebuilding: video PSNR (a picture-quality score, higher is better) rises from 31.11 to 31.93 dB on 512 px video (Table 13).
A full 256 frame HD video does not fit in GPU memory. The paper cuts the time axis into tiles of 32 raw frames (4 latent frames). The encoder uses tiles with no overlap. The decoder uses tiles that overlap by 16 raw frames, then blends the overlap: for N shared frames, frame j gets weight wj = j/N from one tile and 1 − wj from the next. The result has no seam at tile edges, at the price of extra compute.
The ablations are small but telling. A full 3D design (3D convolution and attention) beats the 2.5D design (2D plus 1D) only slightly: PSNR 28.80 against 28.51, SSIM 0.852 against 0.845 (Table 12, measured on a smaller test autoencoder with 4 times compression and 8 channels). The team judged that not worth the memory, so it kept 2.5D. Against a frame-by-frame autoencoder, the TAE reaches comparable video quality with 8 times more compression in time (Table 11).
Chapter 2
Draw a straight line from noise to a clip, and train the model to predict the speed along it
Picture a drop of ink in water. It spreads until the glass looks like pure cloud. Now run the film backward: the cloud gathers into a drop. Generating a picture works like that backward film. Start from a cloud of random noise. Move it, bit by bit, until a clear picture appears.
The question is how to teach a network to do the moving. Flow matching gives a very plain answer: draw a straight line from a noise sample to a real sample, and train the network to say how fast to travel along it.
Take the latent of a real clip from the TAE. Call it X1. Draw a noise latent of the same shape, with every number picked from a bell curve of mean 0 and spread 1. Call it X0. Pick a time t between 0 and 1. Then mix the two:
This is a straight line, which the paper calls the optimal transport path (the shortest route between the two points). At t = 0 you hold pure noise. At t = 1 you hold the clip. In between you hold a blend.
Now ask: how fast does Xt move as t grows? Take the derivative (the rate of change) with respect to t. The result does not depend on t at all:
Drag the time below. The dot slides on the line. The arrow stays the same length and direction.
Purple is a noise sample X0. Blue is a real clip X1. The warm dot is the mix Xt. Drag t. Press New pair for another example.
Two numbers stand for a whole latent, so you can see it. The pairs are made up and seeded. The strip below the line is 400 times drawn from the logit-normal rule the paper uses (zero mean, unit spread), computed in your browser; the teal tick is your t.
The network is u. It gets three things: the mix Xt, the text prompt embedding P (Chapter 3), and the time t. It must guess Vt. The loss is the mean squared error, the average of the squared gaps:
A worked number in one dimension. Let X0 = −1.2 and X1 = 0.8, and ignore σmin. At t = 0.25: Xt = 0.25 × 0.8 + 0.75 × (−1.2) = 0.2 − 0.9 = −0.7. The true velocity is 0.8 − (−1.2) = 2.0. If the network guesses 1.4, the loss is (1.4 − 2.0)2 = 0.36. Training lowers that number.
Where does t come from? The paper samples it from a logit-normal distribution: draw z from a bell curve with mean 0 and spread 1, then set t = 1/(1 + e−z). This puts most training at middle times, where the task is hardest, and little at the very ends. The strip in the device shows the shape.
A subtle point explains the hero at the top. At a given Xt, many different (noise, clip) pairs could have made it. The network cannot tell which one. So it learns the average arrow over all clips that fit. Early in the walk, that average points toward the middle of many clips. If you take one huge step, you land on that middle: a smear. The hero showed exactly this with a single step. Many small steps let the average arrow turn toward one clip as the walk goes on.
At inference you start with X0 from pure noise at t = 0. You then solve an ordinary differential equation (ODE): a rule that says how X changes with t, given by the network. The paper uses the simplest solver, Euler: Xt+Δt = Xt + Δt · u(Xt, P, t). In the toy above, from t = 0.25 with the true velocity 2.0 and a step of 0.25, you move from −0.7 to −0.2. Chapter 5 shows how many steps, and where to place them.
Most older video models use diffusion: they add noise in many small steps and learn to reverse them. Diffusion needs a careful noise schedule. If the schedule leaves a trace of the picture at the last step, the model starts generation from a slightly wrong place. The fix is a schedule with zero terminal signal-to-noise ratio (the ratio of picture to noise equals zero at the start of generation). The straight-line mix has this by default: at t = 0 the input is pure noise. The paper reports that flow matching also beat diffusion with the best known fix: in a controlled test, human raters gave flow matching a net win of 16.53 points on overall quality and 7.08 on text alignment (Table 8a).
Chapter 3
Follow one clip through patches, positions, attention, text and time, and see the three changes that turn LLaMa3 into a video model
A chat model reads a row of word pieces and writes the next one. A video model needs a row of pieces too. The trick of this chapter is to cut the compressed video into small blocks and lay the blocks in a row, so that the same machine that reads text can read video.
A token is one piece in that row. In a chat model a token is a word piece. Here a token is a small block of the latent. The row is long. Its length matters for everything that follows.
The latent has shape 32 × 16 × 96 × 96: 32 latent frames, 16 channels, 96 rows, 96 columns. The paper cuts it into patches with a 3D convolution (a filter that slides over time, height and width at once). The filter size is kt × kh × kw = 1 × 2 × 2, and its step equals its size, so patches never overlap. Each patch spans 1 latent frame and a 2 × 2 square, so it holds 16 × 1 × 2 × 2 = 64 numbers. A linear map turns those 64 numbers into a vector of 6,144 numbers.
Count the tokens: T·H·W / (ktkhkw) = 32 × 96 × 96 / 4 = 73,728. That is the “73K” in the abstract. The input to the transformer is a table of shape 73,728 × 6,144.
A transformer treats its input as a bag. It does not know which token is first or which sits above which. So you add position information. The paper uses a learned embedding for each of the three axes, φt, φh and φw, and sums them. Each is a table of vectors. A token at frame 5, row 10, column 20 gets φt(5) + φh(10) + φw(20). Splitting the three axes lets the same model take any size, shape or length. The team adds the position vector in every layer, not only the first. They report that this reduces distortion and morphing in time.
The block copies the one in LLaMa3, Meta’s language model: RMSNorm (a layer that rescales each vector to a steady size) and SwiGLU (a feed-forward layer with a gate). The team made three changes for video:
Table 1 lists the 30B model: 48 layers, model dimension 6,144, feed-forward dimension 16,384, 48 attention heads. Each head has 6,144 / 48 = 128 numbers, so the queries, keys and values have shape 73,728 × 48 × 128.
Step through one block below. Then flip the mask switch to see what bidirectional attention costs.
Choose a stage. The highlighted row shows what goes in and what comes out. Turn the mask on to see the language-model version of attention.
Shapes are the paper’s Section 3.1.3, Table 1 and Figure 8 for a 768 px, 16 second clip; position embeddings in every layer. The pair counts are one head in one layer, computed live. The causal count is what an LLM-style mask would leave: each token sees itself and earlier ones.
Can you get 30 billion from those numbers? One self-attention layer has four square maps (query, key, value, output): 4 × 6,1442 = 150,994,944 weights. Cross-attention has four more, about the same: 150,994,944. SwiGLU has three maps between 6,144 and 16,384: 3 × 6,144 × 16,384 = 301,989,888. One layer adds to 603,979,776. Times 48 layers is 28.99 billion. The rest (time modulation, text projections, patch embedding) brings it to the 30B the paper reports. The check works, so you now know where the weights live.
One text encoder is not enough. The paper uses three, each good at something different, and joins their outputs:
Each encoder gets its own linear layer and LayerNorm to reach 6,144 dimensions. Then the three sequences are glued end to end into one sequence P. The video tokens attend to P in cross-attention. The paper also controls the frame rate through the text: it puts a string such as “FPS-16” in front of the prompt. During training the string holds the clip’s real frame rate. At inference you choose it.
The team states its reason: scale. Language-model recipes are well tested for many sizes, learning rates and batch sizes. The ablation compares the LLaMa3-style block with the DiT block, with all else equal. Raters preferred the LLaMa3-style design by a net 18.63 points on overall quality and 12.60 on text alignment (Table 8c). The two blocks differ in small ways: RMSNorm against LayerNorm, SwiGLU against SiLU, and bias terms (Table 9).
The scaling test is stronger. The team trained models of 5B, 9B, 17B and 30B parameters over a range of compute. For each compute budget they found the best model size. These best points fell very close to the LLaMa3 scaling law. So a rule found for text models predicted the right size for a video model. That is a strong reason to build on an LLM design.
Two LLM habits did not carry over. A causal mask makes attention about twice as fast in a language model, but video needs full attention. And LLaMa3 uses grouped-query attention to shrink memory; the paper leaves that to future work.
Chapter 4
Filter and caption the clips, then train in stages so the costly long-context work comes last
Imagine a warehouse with a hundred million videos, and a librarian who may keep only the best clips. Some are blurry. Some are slideshows. Some show the same cat a thousand times. Whatever the librarian keeps is what the model will learn from. That makes data work half of the paper.
The final pre-training set holds about 100 million video-text pairs and about 1 billion image-text pairs. Each video clip is 4 to 16 seconds long, from a single camera shot, with real motion. Three filter stages and one caption stage make them (Figure 9).
The resampling step uses a simple rule: sample clips by the inverse square root of the cluster size. A worked toy (my reading of the rule; the paper gives no cluster counts): a cluster of dogs holds 10,000 clips and a cluster of rare birds holds 100. Raw, dogs are 99% of the mix. Each dog clip gets weight 1/√10,000 = 0.01, each bird clip 1/√100 = 0.1. The dogs’ total is 10,000 × 0.01 = 100. The birds’ total is 100 × 0.1 = 10. Dogs now make 91% of the mix. The crowd is tamed, not removed.
The model learns to match prompts to videos. A rich caption is a rich lesson. The team tested this. They compared LLaMa3-Video captions with captions built from three still frames (first, middle, last), rewritten into one. Raters preferred the video-based captions 67% of the time, against 15%. Training with them raised prompt alignment by a net 10.8 points, mostly in motion (Table 8b). A caption that says how things move teaches movement.
Training needs equal tensor shapes in a batch. So clips go into five aspect-ratio buckets and five duration buckets (Table 2). For example, an 8 to 12 second clip at 24 down to 16 fps gets 192 frames and 24 latent frames. A 16 second clip at 16 fps gets 256 frames and 32 latent frames. Clips in one bucket have the same latent shape, so they batch with no padding.
Attention cost grows with the square of the token count. A 768 px clip has 73,728 tokens; a 256 px clip of the same length has 8,192. That is 9 times the tokens and 81 times the pairs. So the recipe spends cheap compute first:
Compare the three stages by what each one costs. Switch the measure below.
Each bar is a stage’s share of the total. Switch between counting samples, counting tokens, and counting attention pairs.
Samples seen are from the paper’s Table 3 (1.94 billion, 394.8 million, 73.8 million). Tokens per sample are my own arithmetic: a 1 frame 256 px image is 256 tokens, a 256 frame 256 px clip is 8,192, a 256 frame 768 px clip is 73,728. Treating every clip as 256 frames is a simplification. Pairs are tokens squared, for full attention in one layer.
The last row is the key. The 768 px stage sees only 3% of the samples, but it holds most of the tokens and nearly all of the attention work. The paper says this in words: most training compute goes to long-context 768 px training. A language-model team trains short first and long last. Here the long stage is the main cost.
The team used up to 6,144 H100 GPUs with 80 GB of memory each. One clip alone is 73,728 tokens. No single GPU holds the activations. The team splits the work four ways:
After pre-training, the model is finetuned on a small set of hand-picked clips. The team narrows millions of candidates by four stages: automatic filters, a concept-balanced subset, manual choice of cinematic clips (angled light, vivid colour, no clutter, no shake, no overlay text), and human-edited captions. Finetuning clips are 10.6 to 16 seconds. This stage is small: 64 nodes (512 H100 GPUs) are enough.
Different finetuning runs give different strengths in motion, consistency and camera control. So the team averages the weights of several finetuned models, as LLaMa3 did. The finetuned model beat the pre-trained one by a net 34.65 points on overall quality (Table 7).
Chapter 5
Run the Euler solver with a schedule that spends steps where the video forms, and rewrite the prompt first
A sculptor works a block of stone in two ways. At first, big cuts set the shape of the whole figure. Later, small taps polish the surface. If you took the same number of taps at every stage, you would waste most of them. The sampler of Movie Gen follows the same idea: it spends its steps where the video takes shape.
Follow one request through the whole system. It shows every shape from the earlier chapters in order.
Chapter 2 gave the rule: Xt+Δt = Xt + Δt · u(Xt, P, t). The paper tried better solvers: a midpoint solver and an adaptive one (Dopri5). Plain Euler did better, so Euler stays. Each step costs one pass of the whole 30B model over 73,728 tokens. That is expensive, so the number of steps is the main cost knob.
Video is touchier than images here. The paper reports that clips made with 250, 500 or 1,000 equal steps differ in scene layout and motion quality. Fewer steps are not a free saving. The team looked for a cheaper way that needs no new training.
They measured how much each transformer block changes its input at each step of a 1,000-step walk (Figure 10a). The change is large at the first steps, when t is near 0, and then falls fast. The curve is shaped like an L. Early on, the model sets the layout and the motion. Later, it only refines.
So the team made the linear-quadratic t-schedule. It keeps the first 25 steps of an N-step linear schedule, so the early steps are as small as in the slow run. It then covers the rest of the time range with 25 steps whose sizes grow quadratically. With N = 250, the first 25 steps are 0.004 apart and end at t = 0.1. The other 25 steps share the remaining 0.9, starting small and ending big. The paper reports that 50 such steps give a result close to 250 linear steps, and a 1,000-step walk can be copied with the same 50 steps. That is up to about 20 times faster, with a few lines of code.
Drag the cut-over point below and watch where the 50 steps land on the time axis.
The dashed line is 50 equal steps. The warm line is the linear-quadratic schedule. The ticks under the chart show where its steps fall between t = 0 (noise) and t = 1 (clip). Drag how many of the 50 steps are copied from the slow, fine run.
Illustrative reading of Section 3.4.2: the first m steps are t = i/250, then 50 minus m steps follow a quadratic that starts with the same step size and ends exactly at t = 1. The paper prints no formula, so the exact placement may differ. The paper’s setting is m = 25 (Figure 10b).
With m = 25: step 26 lands at t = 0.1053. The last step jumps from 0.9333 to 1.0, a size of 0.0667. The first quadratic step is 0.0053. So the last step is about 13 times larger than the first of the second half, and 17 times larger than the fine steps. The model has settled by then, so the big last steps cost little quality.
The paper compares to other systems with a classifier-free guidance scale of 7.5. This is a standard trick: run the model twice per step, once with the prompt and once without it, and push the result away from the no-prompt answer. One common form is u = unone + w · (uprompt − unone). With unone = 1.0, uprompt = 1.4 and w = 7.5, the guided velocity is 1.0 + 7.5 × 0.4 = 4.0. The model follows the prompt much harder. It also doubles the passes: 50 steps need 100 model passes. Without the schedule, 250 steps would need 500.
There is a mismatch. Training captions are dense paragraphs of about 100 words. Users type fewer than 10 words. So a language model rewrites the prompt into the training style. The team built it with distillation: a LLaMa3 70B teacher, with detailed instructions and examples, rewrote many prompts. People kept the good rewrites. An 8B model was then finetuned on those pairs, so the rewrite is fast. Two lessons from the team: use plain words, and do not describe motion in too much detail, because over-long motion text makes artifacts.
The base model works at 768 px because tokens are costly. A separate 7B Spatial Upsampler raises it to 1080p. The 768 px video is first stretched to HD by simple bilinear interpolation. A frame-by-frame VAE encodes it. A transformer then makes the HD latent, with the low-resolution latent joined as extra input channels (the new weights start at zero). Because the input is so strong a clue, 20 steps are enough. For long videos it works in sliding windows of 14 latent frames, overlapping by 4. To hide seams, it uses MultiDiffusion: at each step the overlapping latents are averaged, so windows share information.
Chapter 6
Read the win-rate table against four rivals, and the ablations that justify each design choice
Look at two clips of a dog catching a ball. In one, the dog’s legs blur and a second ball appears. In the other, everything holds together. You know at once which is better. Now try to write a formula for it. That is the problem of judging video, and the paper's answer is simple: ask people.
The paper says that automated scores (the kind used for images) do not give reliable results for text alignment, motion quality or realness. So every claim in the paper rests on a pairwise A/B test: an expert sees two videos for the same prompt, side by side, and picks a winner or declares a tie.
The result is a net win rate: the share of wins minus the share of losses, for Movie Gen against a rival. It runs from −100 to +100, and 0 means even. Worked toy: out of 100 comparisons, 55 wins, 20 ties and 25 losses give 55 − 25 = +30.
What does the rater judge? Three groups of axes (Table 4):
The paper takes steps to keep human tests fair. It writes detailed guides with examples. It asks raters for the reason behind each pick. It uses 1,003 prompts (the released Movie Gen Video Bench, more than 3 times larger than earlier prompt sets). It takes the majority of 3 votes for text and quality, and of 6 for the more personal realness and aesthetics. It audits answers. A result counts as significant when it lies beyond 2 standard deviations of annotation noise, moderate between 1 and 2, and on par inside 1.
Table 6 compares Movie Gen Video with four rivals: Runway Gen3, LumaLabs, OpenAI Sora and Kling 1.5. For the first, second and fourth, the team used videos made for its benchmark prompts and did not pick favourites. Sora has no API, so the team used the prompts and clips Sora published, which are probably the best ones. To be fair, the team made five videos per prompt and chose one by hand. Choose a rival below.
Choose a rival. Each bar is Movie Gen Video’s net win rate on one axis. Bars to the right mean Movie Gen won more often; bars to the left mean it lost.
Numbers are Table 6 of the paper (net win rate, win% minus loss%). The paper marks a result as significant beyond 2 standard deviations; those bands are not drawn here, so a small bar can still be a tie. The Sora column uses the published Sora prompts, with one hand-picked Movie Gen video out of five.
Three readings are worth practising. First, Movie Gen Video is ahead of Runway Gen3 and LumaLabs on all or nearly all axes; against Runway the overall quality margin is +35.02, against LumaLabs +60.58. Second, against Sora the margin on overall quality is a moderate +8.23, and realness is a significant +11.62. Third, Kling 1.5 is on par overall (+3.87) and the one loss is instructive: Movie Gen is −10.04 on motion completeness. The paper explains that Kling sometimes makes large motion with distortion. Motion completeness counts how much moves, even if it is wrong. Kling’s weaker frame consistency (Movie Gen +13.5) shows that.
A claim like “flow matching is better” needs a fair test. For its ablations the team trained a smaller 5B model on 352 × 192 px clips of 4 to 8 seconds, changed one thing, and compared on 381 prompts (Movie Gen Video Bench-Mini). Table 8 gives net win rates for overall quality (Q) and text alignment (A):
| Design choice | Quality (Q) | Alignment (A) | What it says |
|---|---|---|---|
| Flow matching vs diffusion | +16.53 | +7.08 | The straight-line objective wins |
| Video captions vs frame-rewrite captions | −0.80 | +10.80 | Motion-aware captions mostly help prompt match |
| LLaMa3-style block vs DiT block | +18.63 | +12.60 | The LLM block is better and scales |
Two more checks. First, the finetuned model beat the pre-trained one by a net 34.65 points on overall quality (Table 7). Second, the validation loss tracks human scores (Figure 15): checkpoints with lower loss scored higher, especially for text alignment and overall quality. This is useful, because a loss is cheap and a human test is not.
The conclusion lists faults. Generated or edited videos still show artifacts around complex shapes, handling of objects, physics and changes of state (a thing that melts or breaks). Human judging itself has bias. The models were also built for research and need safety work before use. The team ends with an argument for fair tests: it releases non-selected samples and the prompt set so others can compare without cherry picking.
Chapter 7
Add a face and an edit instruction to the video model, and train editing with almost no editing data
You have a finished video model. A friend asks for two things. First: “Make a video of me, from this photo, painting a fence.” Second: “Take my holiday clip and change the background to a lake.” The model never saw a pair of (photo, video of that same person) or (clip, edit, edited clip) in the way you need. Both skills have the same problem: the right training data barely exists. This chapter shows the paper’s tricks for making it.
The goal is personalization: the output video must keep the identity of the person in a reference photo and still follow the text. The model, PT2V, starts from the 30B video weights. A vision encoder reads the face image. The paper uses a copy of Long-prompt MetaCLIP's vision tower, trained further, then a projection layer that makes its output the same width as the text features. These vision tokens are joined to the text sequence P, and the video tokens read them in cross-attention. Joining by concatenation means no new kind of layer is needed, which keeps the design simple enough to scale.
The harder question is data. The team picks clips where the same single person appears throughout. It samples a frame every second, finds the face, and keeps the clip when the ArcFace score (a number from 0 to 1 for how alike two faces are) between neighbouring frames is above 0.5. That leaves about a million clips of 4 to 16 seconds.
Paired data takes the reference photo from the same clip: 5 frames per clip, the face cropped and segmented so the model ignores the background. This makes about ten million samples. And it makes a trap.
The fix is cross-paired data: the reference photo comes from a different shot of the same person. The team found about ten thousand real cross-pairs (clips with different camera views of one scene) and made about a million synthetic ones. For the synthetic ones, a personalized image model creates new views of the person from the first frame, with varied pose, expression and light. Any image with ArcFace similarity below 0.7 to the reference is dropped.
Training takes three stages. Stage I, identity injection: short clips only (8 latent frames, or 64 raw frames), vision encoder frozen, paired data. Long clips learn too slowly because attention cost grows with the square of the tokens. Stage II, long videos: return to full length for background and motion consistency. Stage III, naturalness: cross-paired data, with the vision encoder now trainable. A final finetune on a few high-quality clips follows.
The paper measures with 50 unseen people and 330 photo-prompt pairs. Raters judge the identity in the best and worst frame and how stable the face stays. PT2V beats the ID-Animator baseline by a wide margin on all of identity, video quality and text alignment. Two ablations are honest about trade-offs: a trainable vision encoder keeps identity better than a frozen one, and cross-paired training lowers identity similarity a little, but gives more varied poses and more natural faces (Table 15). Against the plain video model, PT2V is on par on overall quality and better on consistency, but worse on motion completeness and prompt match, because its people-focused data covers a narrower set of activities.
Movie Gen Edit takes a video and a written instruction. The input video's latent is joined to the noisy output latent along the channel axis, so the patch layer has extra input channels. A learned task embedding tells the model which kind of edit (add an object, change the background, and so on). All new weights start at zero, so at the start the model behaves exactly like the original and keeps its video skill.
No large set of (video, instruction, edited video) triples exists. The paper builds the skill in three stages. Each stage fixes the flaw of the one before. Walk through them:
Choose a stage. Each example has an input video, an instruction and a target. Blue is real data, warm was made by a model, pink is the instruction. The bar shows how often people preferred this stage over the one before.
Stage contents are from Section 5.1. Win rates are Table 22: Stage II against Stage I overall 89.29%, and the final model against Stage II overall 60.82% (50% means a tie). Stage I has no earlier stage to beat.
Stage I: edit one frame. Image-editing datasets exist. The model treats an image edit as a one-frame video edit. It trains on image edits and on ordinary text-to-video together, alternating batches (image batches five times as often), so it does not forget how to make video. For text-to-video batches the input is a black video, and the caption is rephrased as an instruction. A small trap: if the image uses the first frame's position, video edits come out distorted. Giving the image a random temporal position fixes it.
Stage II: edit many frames. The Stage I model edits stills well but gives blurry video edits. The team's reading: it never saw a multi-frame input. So it makes two synthetic tasks. In animated frame editing, a language model writes an instruction, the Stage I model edits one frame, and random affine moves (shifts, turns, zooms) animate both the original and the edited frame into short sequences. In instruction-guided segmentation, the instruction says “mark the boy in pink”, and two existing vision models, DINO (finds the named object) and SAM 2 (cuts its outline), produce the masked target. Blur goes away, but new objects barely move and colours look over-saturated. The cause: the targets are model-made, with no natural motion.
Stage III: backtranslation. The idea comes from machine translation. The team wants real videos as targets. So it takes a real video, asks a language model for an edit instruction, and lets the Stage II model make an edited video (flawed, but fine as an input). Now it flips the roles. A language model writes the reverse instruction (“remove the basket and put the apples back on the table”). The training example is: flawed edited video as input, reverse instruction, real original video as the target. The target is clean; the flaws sit only in the input, where they do no harm.
The numbers (Table 22) show each stage worked. Stage II beats Stage I by a win rate of 89.29% overall (structure 90.86, quality 91.9). The final model beats Stage II by 60.82%. Backtranslation also beats ordinary finetuning on the same data, 70.23% overall (Table 21). On the TGVE+ benchmark, raters preferred Movie Gen Edit to the best earlier method (EVE) 74% of the time overall.
Chapter 8
Reuse the same flow recipe on sound, and switch one model between five tasks by dropping inputs
Watch a silent clip of a person diving into a pool. It feels wrong. You expect a splash at the instant the body hits the water, then ripples, then a distant echo. A sound for a video must match two things: what the scene is, and exactly when things happen.
Movie Gen Audio does this. It has 13 billion parameters, makes sound effects and music at 48 kHz (48,000 samples a second), and follows both the video and a text caption. The nice surprise is how little is new. The recipe is the one you already know: a latent space, flow matching, a transformer. Only the data shapes change.
One second of 48 kHz audio holds 48,000 numbers. As with video, that is too long for a transformer. The team trains a DAC-VAE: the Descript Audio Codec with its discrete quantiser removed and a VAE loss added (a penalty that keeps latents close to a bell curve). It turns audio into a latent of shape T × C with C = 128 and a rate of 25 frames per second. One latent frame therefore covers 48,000 / 25 = 1,920 samples.
Worked: 10 seconds of audio is 480,000 numbers. As a latent it is 250 frames × 128 = 32,000 numbers, 15 times fewer. A 30 second clip is 750 latent frames, which is the training cap. The rate of 25 Hz is lower than the 75 Hz common in earlier audio diffusion work, and the sample rate of 48 kHz is higher than their 24 kHz. Fewer, richer frames keep the sequence short.
Call the audio latent with noise Xt, of shape N × 128, where N is the number of latent frames. The model gets three kinds of conditioning c:
The network is a Diffusion Transformer with 36 layers and widths of 4,608 (attention) and 18,432 (feed-forward). It predicts a velocity of shape N × 128. The training loss is the same flow-matching loss as in Chapter 2, with the same optimal transport path and logit-normal times. Time enters through a small network shared across layers, which saves parameters.
During training each of the three inputs is dropped on its own, at random. A dropped input is replaced by zeros or left out. So one model learns every combination. At inference you choose which inputs to show it. Try the switches.
Turn the three inputs on or off and set the length of the audio. The task changes, and so do the shapes the model reads.
Shapes are from Section 6.1.3: 25 latent frames per second, 128 channels, 1,024-wide video features, 768-wide T5 features, text up to 512 tokens. The paper names four combinations explicitly (video only, video plus text, and the two extension versions); the others use the same switch, and the abstract also lists text-to-audio. The paper’s tests focus on the first four.
The same switch gives classifier-free guidance a cheap form. In training, all three inputs are dropped together with probability 0.2. At inference the model is run with and without the conditions, with a guidance weight of 3. The sampler is a midpoint solver with 64 steps, and the team creates 8 candidates per sample and re-ranks them by a quality score.
The team tags audio events with a model trained on the AudioSet ontology (527 classes), drops videos where silence dominates, and sorts clips into speech, music and general sound. A score called CAVTP, the cosine similarity between audio and video embeddings, tells how well the sound matches the picture. Thresholds on these tags and scores sort the pre-training clips into groups, and a smaller, higher quality set is kept for finetuning. Pre-training ran 500 thousand updates on 384 GPUs for 14 days, with a batch of 1,536 sequences of up to 30 seconds. Finetuning ran 50 thousand updates on 64 GPUs in one day, and an exponential moving average of the weights (decay 0.999) is used at inference. The model does not make voices; the paper says so.
Training sequences are capped, so a long video needs another method. The paper uses overlapping segments and MultiDiffusion (the same trick as the upsampler): run the model on each segment, average the overlapping parts at each step, and use a triangle-shaped window so the weight of a segment fades at its edges. The result is smoother joins than uniform averaging, and on par with generating the whole piece at once.
Raters compared Movie Gen Audio with several systems, including commercial ones, on sound effects (Table 30). Net win rates were 33.8% to 72.8% on synchronisation, 27.5% to 82.2% on correctness, and 31.3% to 91.0% on overall quality, depending on the rival. The wins were larger for professionalness than for naturalness. Sound effects plus music made together (Table 31) also beat mixing separately made tracks. The paper gives two reasons: separate tracks cannot be balanced against each other (music should drop when a loud effect hits), and a text-only music model is blind to scene changes.
Faults remain. Sound can drift when motion is dense (tap dance), small or hidden (footsteps), or needs fine understanding (which guitar chord). Video and audio are still made by two separate models.
Chapter 9
Keep the numbers, the terms, a checklist and the core code, then see where to read next
You can now follow Movie Gen from a sentence to a video with sound. This page holds what you need to rebuild it from memory. It reads Lecture 9 of the course from the side of one paper; the lecture is Multimodal Generation.
Video is too big to generate in pixels, so a temporal autoencoder shrinks it 8 times along time, height and width into a 16-channel latent. A 30B transformer, built like LLaMa3 with cross-attention for text, adaptive norm for time and full bidirectional attention, reads the latent as 73,728 tokens. It is trained with flow matching: predict the velocity along a straight line from noise to clip. Data is filtered, captioned in detail, and fed in stages from cheap to costly. At inference a rewritten prompt and a linear-quadratic schedule give 50 steps that act like 250, and a 7B upsampler reaches 1080p. Humans judge the results in A/B tests. The same recipe gives a 13B audio model, a personalized model (cross-paired data stops copy-paste) and an editor (backtranslation gives real targets).
| What | Number | Where |
|---|---|---|
| Video model | 30B parameters; 48 layers, width 6,144, FFN 16,384, 48 heads | Table 1 |
| Longest sequence | 73,728 tokens = 32 × 48 × 48, for 16 s, 16 fps, 768 px | Sec 3.1.3, Fig 8 |
| Autoencoder | 8× per axis, 16 channels; 452,984,832 to 4,718,592 numbers | Sec 3.1.1 |
| Outlier loss | r = 3, weight 1e5 | Eq 1 |
| Flow matching | σmin = 1e-5, t from a logit-normal (0, 1) | Sec 3.1.2 |
| Pre-training data | O(100)M videos, O(1)B images; clips 4 to 16 s | Sec 3.2.1 |
| Samples seen | 394.8M at 256 px; 73.8M at 768 px | Table 3 |
| Hardware | up to 6,144 H100 GPUs | Sec 3.1.6 |
| Sampler | Euler, 50 linear-quadratic steps (25 fine) acting like 250; guidance 7.5 | Sec 3.4 |
| Upsampler | 7B, 2×, 20 steps, to 1080p | Sec 3.1.5 |
| Net win, overall quality | Runway +35.02, LumaLabs +60.58, Sora +8.23, Kling +3.87 | Table 6 |
| Ablations (Q, A) | Flow vs diffusion +16.53, +7.08; LLaMa3 vs DiT +18.63, +12.60 | Table 8 |
| Audio model | 13B, 36 layers; 25 Hz × 128 latent; 48 kHz; 500K updates on 384 GPUs | Sec 6.1 |
| Audio sampling | midpoint, 64 steps, guidance 3, 8 candidates re-ranked | Sec 6.1.6 |
| Editing stages | Stage II beats I 89.29%; final beats II 60.82% | Table 22 |
| Term | In plain words |
|---|---|
| TAE | The autoencoder that shrinks video in space and time and rebuilds it. |
| Latent | The small block of numbers the generator works on. |
| Velocity | The direction and speed from noise to clip; the model’s target. |
| Euler solver | Move a little along the predicted velocity, then ask again. |
| Cross-attention | Video tokens read the text (or the face) tokens. |
| Net win rate | Wins minus losses, in percent, from human A/B tests. |
| Cross-paired data | A reference photo from a different shot of the same person. |
| Backtranslation | Train on a flawed input and a real target, with the reverse instruction. |
Two short functions hold the heart of the method: one training step, and the sampler with the 50-step schedule. The model and the autoencoder are assumed.
pythonimport torch SIGMA_MIN = 1e-5 def train_step(model, tae, video, text_emb): x1 = tae.encode(video) # (B, T, 16, H, W) clean latent x0 = torch.randn_like(x1) # noise, same shape t = torch.sigmoid(torch.randn(x1.shape[0])) # logit-normal times, (B,) tb = t.view(-1, 1, 1, 1, 1) xt = tb * x1 + (1 - (1 - SIGMA_MIN) * tb) * x0 # the straight line v = x1 - (1 - SIGMA_MIN) * x0 # target velocity pred = model(xt, text_emb, t) # patchify, 48 blocks, back to (B, T, 16, H, W) return ((pred - v) ** 2).mean() def lq_schedule(steps=50, ref=250, fine=25): a = 1.0 / ref; t0 = fine * a; r = steps - fine b = (1.0 - t0 - a * r) / r**2 # my reading: quadratic ends exactly at t = 1 return [i * a for i in range(fine + 1)] + [t0 + a*j + b*j*j for j in range(1, r + 1)] @torch.no_grad() def sample(model, tae, text_emb, null_emb, shape, w=7.5): x = torch.randn(shape) # X0 ~ N(0, 1) ts = lq_schedule() for ta, tb in zip(ts[:-1], ts[1:]): t = torch.full((shape[0],), ta) u_c, u_n = model(x, text_emb, t), model(x, null_emb, t) u = u_n + w * (u_c - u_n) # one common guidance form x = x + (tb - ta) * u # Euler step return tae.decode(x) # tiled decode, then the upsampler
Movie Gen is a latent diffusion-style model with a transformer backbone, like the DiT family. Its distinct choices are the LLaMa3 block, flow matching at scale, an 8×8×8 autoencoder with an outlier penalty, and a recipe where data and human evaluation get as much space as the model. It stays within one model per modality. The paper ends by naming joint video and audio generation as the next step.
Now press Present or Teach and explain one step of Movie Gen out loud, from memory: why the video is shrunk, what the model predicts, how 50 steps copy 250, and how editing is learned from almost no edit data. Then go back to the walk from noise at the top, set the steps to 2 and to 50, and say why the clips look so different.
Credit: Movie Gen: A Cast of Media Foundation Models, by the Movie Gen team at Meta (Polyak, Zohar, Brown, Tjandra, Sinha, Lee, Vyas, Shi, Ma, Chuang, Yan, and many more core contributors). arXiv 2410.13720. This lesson re-teaches the paper in its own words and toy devices; all paper numbers cite their table or section.