To make a reference character copy someone's motion, almost every system first strips the driving video down to a stick-figure skeleton or a masked-out hole — and throws away the ball in the hand, the depth of two tangled bodies, the texture of the floor. SCAIL-2 deletes that lossy middle-man. It feeds the raw driving video straight into a video-diffusion transformer, alongside the reference, and lets the model read everything: occlusions, objects, environments, even animals with no human skeleton. The catch — paired data for this doesn't exist — is solved with an agentic synthesis loop and a "reverse driving" trick, then polished with a hand-focused preference-optimization stage.
You have a single photo of a character — say an anime girl with twin-tails. You also have a video of a real dancer doing a complicated move: she spins, her ponytail whips around, she catches a ball, all in front of a temple. You want the anime girl to perform that exact motion, in that exact place. This is controlled character animation, and it is the engine behind a huge slice of film and entertainment.
For the last few years the recipe has been remarkably consistent. Take the driving video, run an off-the-shelf pose estimator over it to draw a skeleton — a stick figure of joints — and feed that skeleton into a video generator that "rigs and renders" the reference character onto it. It works for clean, single-person, full-body dancing. The trouble is everything else.
The paper catalogues exactly where skeletons and masks break. Read these slowly — the entire architecture later is a direct answer to each one.
Drag the controls below. As you turn up the interaction complexity and switch what the model is allowed to see, watch how much of the real scene survives the trip through a skeleton or a mask. The shaded "lost" regions are exactly the information the downstream generator never receives.
Here is the entire idea in one sentence: instead of extracting a pose from the driving video and feeding the pose, encode the driving video itself and feed the video. The model already has a powerful visual backbone — let it look at the actual pixels (in latent form) and pull out whatever motion, depth, object, and environment cues it needs, with nothing filtered out in advance.
Concretely, prior pose-driven methods compute a skeleton sequence and inject it. SCAIL-2 instead runs the driving video through the same VAE encoder used for everything else, producing driving latents that are concatenated into the token sequence right next to the reference tokens and the noisy video tokens. The model attends over all of them at once. The paper calls this the In-Context Driving design (inherited from the earlier SCAIL): the condition is placed in the sequence, not stirred into the channels.
The transformer backbone (an image-to-video, or I2V, model) receives the concatenation [zref ; zt ; zdriv] — the reference tokens, the noisy denoising tokens, and the driving tokens. Following SCAIL, the driving tokens get a fixed spatial offset ΔW along the width axis so they sit spatially detached from the video being generated — close enough to attend to, far enough not to be confused for it. (Chapter 6 makes this geometry precise.)
There is no free lunch. End-to-end driving needs a very particular kind of training data: a video of character A doing a motion, paired with a video of a totally different character B doing the same motion in the same (or a different) environment. That pairing essentially never exists in the wild — you cannot film two different people performing identical motion frame-for-frame. The next three chapters are entirely about manufacturing that data and teaching one model to use it for every sub-task at once.
SCAIL-2 is built on a latent video diffusion model. Before we can talk about conditioning, we need the skeleton of the generator itself. This chapter builds it from the ground up so every symbol later has a home.
Raw video is enormous and pixel-space diffusion is wasteful. So a pretrained 3D VAE encoder ℰ squeezes a video x into a compact latent z0 = ℰ(x), shrinking it in time and space. All diffusion happens in this small latent space; a decoder turns the final latent back into pixels. Every token we manipulate — reference, driving, noisy — lives in this same shared latent grid, which is precisely what makes concatenating them sensible.
Diffusion trains a model to reverse a known corruption. The forward process adds a touch of Gaussian noise at each of T steps, following a noise schedule βt:
Read it in words: the new noisy latent is the old one, shrunk slightly by √(1−βt) so it doesn't blow up, plus fresh Gaussian noise with variance βt. Small β early (gentle), larger later (aggressive), until the signal is pure noise.
A denoising network εθ(zt, t, c) looks at a noisy latent, the timestep, and a condition c, and predicts the noise that was added. Train it by minimizing the gap between predicted and actual noise:
At generation time you start from pure noise and repeatedly subtract the predicted noise, conditioned on c, walking back to a clean latent. SCAIL-2's modern backbone (Wan2.1-14B-I2V) uses the flow-matching / v-prediction flavor of this same idea — the network predicts a velocity field rather than raw noise — but the spirit is identical: learn to undo corruption, steered by a condition.
For character animation the condition c bundles a text prompt ctext, a reference image I containing a character set CI = {C1, …, CN} in environment EI, and a motion signal from a driving video y with its own characters and environment Ey. Pose-driven methods set the motion part to cpose = P(y). SCAIL-2 sets it to zdriv = E(y) — same latent space, no estimator.
How does c actually reach the network? There are three classic options, and the choice is the whole ballgame for "end-to-end."
Character animation has historically been a zoo of separate models: one for single-character animation, one for replacement, one for multi-character. SCAIL-2's second big move is to notice they are the same problem wearing different costumes, and to write down a formulation that covers all of them with two knobs.
The driving video has characters Cy = {C1driv, …}. The reference has characters CI = {C1, …}. A binding map π : Cy → CI says which driving character drives which reference character. Driving character Cidriv transfers its motion to its bound target π(Cidriv) — and to no one else. Single-character is just |Cy| = |CI| = 1; multi-character is > 1.
Where does the final background come from? Two choices: E ∈ {EI, Ey}. Keep the reference's environment EI and you have Animation Mode (the character moves within its own world). Take the driving's environment Ey and you have Replacement Mode (swap a new character into the original scene).
Set the knobs yourself below. Watch the binding arrows route each driving character's motion to its target, and watch which environment survives into the output.
Casting every task as "read different dimensions of information from the context and compose them" lets the authors decompose what the model must learn into exactly three objectives. Memorize these three — every later component maps onto one of them.
We hit the wall in Chapter 1: end-to-end driving needs paired videos — character A and character B performing the same motion — and reality refuses to provide them. SCAIL-2's answer is to synthesize the pairs with an agentic loop, and then use them with a counterintuitive twist called reverse driving. This chapter is the data engine.
Start from a real driving video y (a person doing some motion). We want a synthetic video &ytilde; of a different character doing the same motion, via an animation generator 𝒢: &ytilde; = 𝒢(y, I). The hard part is inventing a plausible reference image I that the generator can actually animate well. An agentic editing loop of four cooperating modules builds it:
Run that loop, animate with 𝒢 (the authors use the robust pose-driven SCAIL for most pretraining data, Wan-Animate for close-ups), and you get a candidate training pair. A VLM does a final video check; the authors keep the discard rate under 30%.
Now the subtle, brilliant part. You might think: "great, train the model to produce &ytilde; from y." No. The synthetic &ytilde; is generated — it carries renderer artifacts, fake textures, physically-implausible object interactions. If you supervise toward it, you teach the model to reproduce those defects.
So SCAIL-2 reverses the arrow. The synthetic &ytilde; becomes the driving input (it only needs to convey motion — defects are tolerable there), while the real y becomes the denoising target (authentic, physically consistent — exactly what we want the model to output). A reference frame I sampled from y completes the triplet.
Step through the pipeline below, then flip the reverse-driving switch to see how the training direction inverts.
Two more design decisions complete the data story:
The result is MotionPair-60K — 59,376 end-to-end motion-transfer pairs, animation:replacement roughly 3:1, spanning heterogeneous tasks.
This is the architectural heart of the paper. We have one model that must serve many tasks, told apart only by (π, E). But text instructions and raw video alone are ambiguous: how does the model know whether to keep the reference background or the driving one? How does it know that the left dancer's motion belongs to the left character, not the right? SCAIL-2's answer is a small, elegant set of extra mask channels concatenated to the context — soft guidance on top of the visual information, never replacing it.
One extra channel acts as a binary flag: should the environment come from the reference image or from the driving video? This single bit is what disambiguates Animation Mode from Replacement Mode — exactly objective O2. The environment mask is the complement of the union of the character masks (i.e., "everything that isn't a character").
Then K more channels are binding slots. Each slot is a channel; a character placed in slot k means "this character's motion flows exclusively within slot k." Single-character animation activates one random slot. Multi-character activates several — one per character — so motion can't leak from the left body to the right. And a clever trick: two characters that are far apart can share a slot without confusion, so you can support more than K characters with only K channels.
The masks are extracted by a robust segmenter (SAM3) with rule-based matching, then downsampled spatially to match the latent grid and stacked temporally along the channel dimension. The VAE's temporal compression is 4×, so each logical mask channel becomes 4 latent channels. With 1 environment channel + K binding channels:
This is the simulation to linger on. You have a reference image with characters and a driving video with characters. Assign each character to a binding slot by clicking it (cycling colors = channels), flip the environment switch, and change the mode. Watch three things update together: the binding arrows (O1), the composited output's background (O2), and the live channel stack the model actually receives. If you remove this widget, the relationship between (π, E), the three objectives, and the 28 channels stops being concrete.
Strip away the diffusion machinery and the conditioning is just a concatenation. Here is the essence in PyTorch-flavored pseudocode:
# --- In-Context Mask Conditioning: build the context the DiT sees --- K = 6 # number of binding slots z_ref = vae.encode(reference) # [B, C, T, H, W] reference tokens z_driv = vae.encode(driving) # [B, C, T, H, W] driving tokens (end-to-end!) z_t = add_noise(target, t) # noisy denoising latents # 1 environment channel + K binding channels, each x4 from temporal stacking env_mask = 1.0 - union(char_masks) # Ch0: everything that isn't a character bind_masks = stack([slot_mask[k] for k in range(K)]) # Ch1..ChK M_ref = temporal_stack(env_mask, bind_masks, source=reference) # from ref ONLY M_driv = temporal_stack(env_mask, bind_masks, source=driving) # from driving ONLY M_zero = zeros_like(M_ref) # target latents get an ALL-ZERO mask # concat masks onto their matching latent block along the channel dim -> 4*(K+1) extra ctx_ref = cat([z_ref, M_ref ], dim="C") ctx_t = cat([z_t, M_zero], dim="C") # <-- no ground-truth leakage ctx_driv = cat([z_driv, M_driv], dim="C") # in-context: append along the SEQUENCE, not the channels seq = sequence_concat([ctx_ref, ctx_t, ctx_driv]) # driving carries spatial offset dW (Ch6) v_pred = dit(seq, t, text) # flow-matching v-prediction of the real video
The mask channels tell the model which environment and which binding. But there's a second, geometric difference between the two modes that masks alone don't resolve, and it lives in the model's positional encoding.
Think about the very first frame of the output:
SCAIL-2 encodes this distinction directly into the 3D RoPE coordinates — the (t, h, w) positions each token block is assigned. By giving the reference block a temporal offset in Animation Mode but a spatial offset in Replacement Mode, the model is told, at the level of geometry, what relationship to expect. The driving block always carries a width offset ΔW so it stays spatially detached.
Toggle the mode below to rebuild Table 1 from the paper — watch the reference block's coordinates jump from a temporal shift to a spatial shift.
For a video latent of shape Tv × Hv × Wv, the coordinates are:
Animation Mode z_ref : t=0 h=[0,Hv) w=[0,Wv) # reference is "frame 0" z_t : t=[1,Tv] h=[0,Hv) w=[0,Wv) # video starts at frame 1 z_driv: t=[1,Tv] h=[0,Hv) w=[dW, dW+Wv) # driving offset in width Replacement Mode z_ref : t=0 h=[dH_ref, dH_ref+Hv) w=[0,Wv) # reference offset in HEIGHT z_t : t=[0,Tv-1] h=[0,Hv) w=[0,Wv) # video shares frame 0 z_driv: t=[0,Tv-1] h=[0,Hv) w=[dW, dW+Wv) # driving offset in width
Notice the swap. In Animation Mode, zref sits at t=0 while the video lives at t∈[1,Tv] — a clean temporal separation. In Replacement Mode, the video shares the timeline (t∈[0,Tv−1]) but the reference is pushed away in height by ΔHref — a spatial separation. The driving block keeps its ΔW width offset in both.
The data engine has a built-in flaw. Every training pair was synthesized by pose-driven generators, so the motion accuracy of the end-to-end data is capped by the pose estimators those generators relied on. And the authors notice the damage is most visible in one place: the hands. Finger joints get "incorrectly articulated or simply neglected." Post-training with Bias-Aware DPO surgically repairs this.
Direct Preference Optimization trains a model from pairs: a preferred sample x+ and a dispreferred one x−. Instead of a reward model, DPO directly pushes the policy to assign higher likelihood to the winner than a frozen reference model does, and lower to the loser. Here the "policy" is a trainable flow-matching model vθ compared against a frozen vref.
To do DPO you need a preferred/dispreferred pair that differs only in the fine detail you care about (finger articulation), sharing identity and global pose. The authors construct it by deliberately running the error-prone pipeline an extra time. Given a motion y, a pose estimator P, and a generator G:
r and s share the same pose p but use different reference images. Take s as the driving video and r as the positive. The negative is r pushed through one more round of pose-extract-and-regenerate:
That extra pass inherits one more dose of estimator error, so r− is slightly worse than r in fine details while everything global stays fixed. The gap can be widened by using a strong estimator (SDPose) for the positive and a weaker one (ViTPose) for the negative passes. The final preference tuple is:
If you computed the preference over the whole frame, the tiny finger differences would drown in the noise of everything else. So the DPO loss is masked to hand regions only. With a hand mask M (the union of positive- and negative-sample hand boxes, downsampled to latent space), the per-sample score uses masked velocity-prediction errors:
where ⊙ is element-wise multiply and v = x1 − x0 is the flow-matching target velocity. The DPO term over a pair becomes a log-sigmoid of the margin between the dispreferred and preferred masked errors:
Two stabilizers keep this from going off the rails: an SFT anchor on the positive samples (L = LSFT + λ LMDPO, with λ = 0.01) prevents the preference objective from diverging, and only LoRA adapters (rank 128) are trained while the backbone is frozen. Although the loss touches only the hands, it updates the policy globally — the mask merely up-weights the most salient errors — so in practice mouths and shoulders improve too.
SCAIL-2 is trained on a 14B Wan2.1-I2V backbone: 3,500 pretraining steps (batch 128, lr 1e-5) on 64 H100s for about a week with FSDP-2, then 400 steps of DPO post-training. Does removing the intermediate actually win? The evaluation spans human preference studies, automatic metrics, and a battery of ablations.
Because end-to-end driving targets cross-identity settings (different character than the driver), pixel metrics don't apply — there's no ground truth. So the authors use Good/Same/Bad human studies on Studio-Bench. Use the explorer to flip between tasks and see the win rates against each baseline.
Across single-character animation, SCAIL-2 wins over leading open-source methods on every human metric and stays close to proprietary Kling 3.0. For multi-character — remember, this capability is zero-shot, never explicitly trained — it beats prior methods with a clear margin, especially on Identity Isolation (keeping one character's limbs and clothing from merging into another). For replacement it surpasses inpainting-based Wan-Animate and even MoCha, the very generator used to build its replacement pairs — vindicating reverse driving.
SCAIL-2 also supports old-style pose driving (feed it a skeleton). On Studio-Bench's pose-driven split its SSIM/PSNR are only mediocre with a plain skeleton — it was trained mostly end-to-end, after all. But swap the skeleton for a richer SAM3D-Body mesh and the metrics jump, even though the model has never seen a mesh in training (zero-shot). The authors read this as direct evidence of the end-to-end thesis: a richer driving signal carries more information, and an end-to-end model is built to exploit exactly that.
SCAIL-2 sits at the intersection of three big currents in modern generative modeling. Here's how it links to the rest of the field, and a one-page reference you can rebuild the paper from.
# ===== SCAIL-2 in one screen =====
THE BET: delete the lossy intermediate (skeleton/mask). Feed the WHOLE driving
video, VAE-encoded, concatenated into the token sequence (in-context).
CONDITION: c = ( text , reference I[chars C_I, env E_I] , driving y[chars C_y, env E_y] )
pose-driven: c_pose = P(y) end-to-end: z_driv = E(y)
UNIFY: two knobs cover every task
binding map pi : C_y -> C_I (who drives whom)
env source E in {E_I, E_y} (E_I=Animation, E_y=Replacement)
THREE OBJECTIVES:
O1 Motion Binding -> binding slots (mask Ch1..ChK)
O2 Environment Weaving-> environment switch (mask Ch0) + Mode-Specific RoPE
O3 Universal Transfer -> shared by both modes (pose ⟂ identity)
DATA: MotionPair-60K (59,376 pairs, anim:repl ~ 3:1)
agentic loop: Selector -> Weaver -> edit M(F') -> Quality Checker
REVERSE DRIVING: synthetic y~ = drive, real y = TARGET (beat your generator)
ARCH: seq = [ z_ref ; z_t ; z_driv ] (driving offset dW in width)
mask channels = 4*(K+1) (K=6 -> 28 ch; target latents = all-zero mask)
Mode-Specific RoPE: anim = temporal offset (ref t=0, video t>=1)
repl = spatial offset (ref h += dH_ref)
POST: Bias-Aware DPO (hands). negative r- = G(P(r), R) [one extra error pass]
tuple (s, R1, r, r-); regional mask M; L = L_SFT + 0.01 * L_DPO^M
beta=5000, LoRA rank 128, backbone frozen
WINS: single-char (beats open-source, ~Kling 3.0), multi-char ZERO-SHOT,
replacement > Wan-Animate AND > MoCha (its own pair generator),
pose-driven: skeleton mediocre but SAM3D mesh zero-shot boosts metrics
The paper closes with two framings of its own gains. The first comes from end-to-end training itself: a DiT with strong priors, asked to extract and convert information from visual context, generalizes to far more zero-shot inputs. The second comes from unification under reverse driving: by decoupling concepts and always supervising on real video, the optimization is steered toward plausible composition — so the model surpasses its own data generators rather than inheriting their flaws. The authors' bet is that SCAIL-2 is "positioned to benefit from, rather than be obsoleted by, future advances in data synthesis" — every better generator that comes along just makes its training pairs better.
You've reached the end. Try the Teach Mode button to explain SCAIL-2 back on a whiteboard — the fastest way to find the gaps in your own understanding.