FlashFoley: Fast Interactive
Sketch2Audio Generation

Chapter 1: The Key Insight

FlashFoley is not one idea — it's three ideas stacked in a specific order, and the order is the cleverness. Each one removes exactly one of the three blockers from Chapter 0.

The whole pipeline in one breath

Start with Stable Audio Open Small (a 340M-parameter rectified-flow DiT). Stage 1: finetune it on the velocity loss with sketch controls + inpainting masks injected via Pre-Transformer Projection. Stage 2: ARC post-training replaces the velocity loss with a relativistic adversarial loss plus a contrastive loss, turning the 50-step model into a 1–8 step generator. Inference: for offline use, run the few-step sampler once; for live use, run the block-autoregressive chunker that generates audio in overlapping blocks as the sketch streams in.

We will build each stage from zero. By the end you could re-draw the training diagram on a whiteboard and explain why every arrow is there.

Chapter 2: Rectified Flow — the engine

Before controls or speed, we need the base generative model. FlashFoley is a rectified flow (RF) model operating in a VAE latent space. Let's earn that sentence.

From audio to latent

Raw stereo audio is huge: x₀ ∈ ℝ^2×S·f_s — 2 channels, S seconds at f_s=44,100 samples/sec. About 1 million numbers for an 11 s clip. We never model that directly. A pretrained VAE encoder ℰ compresses it: z₀ = ℰ(x₀) ∈ ℝ^M×N. In the paper's model the compression is 2048× in time, leaving 64 channels × 256 latent frames at ~21.5 Hz. All generation happens on these 256 frames; a VAE decoder turns the result back into audio.

What "flow" means — the straight-line bet

We want to sample new latents from the data distribution p₀. Rectified flow defines a dead-simple bridge between a Gaussian p₁ (pure noise) and data p₀: a straight line. Given a clean latent z₀ and a noise sample ε ∼ 𝒩(0,I), the corrupted point at time t∈[0,1] is just a linear blend:

Symbol by symbol: t=0 is clean data, t=1 is pure noise, and in between you slide linearly. The velocity — the direction you'd move to go from data toward noise — is the derivative of that line:

To generate, you run the line backwards: start at noise z₁ and integrate dz_t = −v\,dt down to t=0. The only thing the network must learn is to predict that velocity at any point and time, given the text prompt:

This is a simulation-free objective: you don't run the ODE during training, you just pick a random t, blend, and regress the velocity. v_θ is the DiT; c_txt is the text embedding; the squared error is the same MSE you've seen a hundred times.

The teal dot is data z₀; the gray dot is a noise sample ε. Drag t to slide along the straight line to the current z_t (orange). The orange arrow is the velocity v=ε−z₀ the network learns to predict. Hit Integrate to walk from noise back to data — that's generation.

Chapter 3: Sketch Controls — turning a hum into numbers

A vocal sketch is a sound. To condition the model on it, we don't feed the raw waveform — we extract three interpretable, time-varying signals that capture the gesture without the timbre. These come from Sketch2Sound, and choosing them well is what lets a hum control a door slam.

Each feature f_i ∈ ℝ^K_i×N is a time series of length N (resampled to match the 256 latent frames). The full control set is F = {f_i}_i=1..l. These are local controls: one value (or vector) per frame, unlike a single global text embedding.

The median filter — the dial between "render" and "sketch"

Here is the most beautiful trick in the control design. If you feed the model the exact extracted features, it tries to reproduce them precisely — great for style transfer, but it overfits to the spectral quirks of your voice. If you blur them, the model treats them as loose suggestions — a true sketch. FlashFoley convolves the controls with a median filter of random width during training, so at inference the filter width becomes a continuous knob:

The faint line is the raw extracted control from a noisy "vocal sketch"; the bold line is after a median filter of width w. Drag w up and watch the jittery sketch smooth into a clean gesture — that's the "render ↔ sketch" dial. Switch the control to see volume (loudness), pitch (a contour through the probability matrix), and brightness (spectral centroid).

Chapter 4: Pre-Transformer Projection — injecting control cheaply

We have three time-varying controls. How do we feed them into a 340M-parameter DiT without retraining everything or adding bulky adapters? The naive options each add 6–50% new parameters. FlashFoley uses Pre-Transformer Projection (PTP), which adds ~0.1%.

Where the controls go in

A DiT first turns the noisy latent z_t into an initial hidden state via an input projector (a small MLP) before any transformer blocks:

where H is the hidden dimension and N=256 frames. PTP learns one tiny linear operator W_i ∈ ℝ^K_i×H per control, projects each control up to hidden dimension, and simply adds it to that initial hidden state:

That's it. Each W_i^⊤ f_i maps a [K_i, N] control to [H, N] and lands as a per-frame additive bias on the hidden state. Then the whole model finetunes on the velocity loss (Eq. of Ch 2). The DiT blocks themselves never change shape.

The subtle point: when concat ≡ PTP, and when it breaks

The authors prove a small, illuminating fact. If ProjIn_θ were a single linear layer, then PTP and channel-wise concatenation are exactly equivalent — concatenating ΣK_i extra channels and learning a wider weight matrix is the same as learning a separate W_i per control. But their ProjIn is a 2-layer residual MLP. With that nonlinearity, concatenation forces implicit weight sharing in the projection matrices, and in short finetuning runs the module can't adapt to each control's distribution separately. PTP sidesteps this by construction.

The latent and the three controls each get projected, then summed into h'_init — which is the only place the controls enter. Everything downstream is the untouched pretrained DiT. Click the nodes to read the shapes.

# Pre-Transformer Projection: ~0.1% new params, no DiT changes
import torch, torch.nn as nn

class PTP(nn.Module):
    def __init__(self, ctrl_dims, H):
        # one independent linear per control: K_i -> H, no bias, no sharing
        self.W = nn.ModuleList([nn.Linear(k, H, bias=False) for k in ctrl_dims])

    def forward(self, h_init, controls):
        # h_init: [B, H, N]   controls[i]: [B, K_i, N]
        for Wi, f in zip(self.W, controls):
            # project K_i -> H along channel dim, add as per-frame bias
            h_init = h_init + Wi(f.transpose(1,2)).transpose(1,2)
        return h_init        # [B, H, N] -> straight into the DiT blocks

Chapter 5: ARC Post-Training (the showcase)

We now have a slow but controllable model: 50 denoising steps, beautiful sketch control. Stage 2 makes it fast. The goal is a generator G_φ that produces good latents in 1–8 steps instead of 50. The method is ARC — Adversarial Relativistic-Contrastive post-training.

Why not just "use fewer steps"?

If you take the 50-step model and run it for 4 steps, the output is blurry garbage — the velocity field assumes small steps. Classic distillation fixes this by training a student to match a strong teacher, but teachers are expensive and limit quality. ARC instead replaces the regression target entirely with an adversarial one: a discriminator D_ψ judges realism at any noise level, so the generator learns to make realistic audio in very few steps with no teacher.

The relativistic loss, decoded

Take a real latent z₀ and conditioning C={c_txt, F}. Noise it to z_t, generate &zcirc;₀=G_φ(z_t,t,C). Re-noise both to a common level s and let D_ψ score them. The min-max objective on the relative scores is:

Read it: D_ψ(&zcirc;_s) − D_ψ(z_s) is "how much higher the fake scores than the real." G_φ wants this difference large (fool the judge); D_ψ wants it small (catch the fake). f is the softplus that turns the score gap into a smooth loss. Everything is relative to the paired real sample — that is the whole anti-collapse mechanism.

Inference: ping-pong sampling

Once trained, G_φ generates by "ping-pong": denoise toward data, re-noise to a slightly lower level, denoise again — a handful of times. The result: 11.88 s of stereo 44.1 kHz audio in 75 ms, ~10× faster than existing controllable TTA, with no quality loss (and, surprisingly, slightly better control accuracy).

The orange sketch control (a loudness/pitch gesture) is fixed. Press Generate: the noisy mel-like field (left) ping-pongs into structured audio (right) in T steps, snapping to the sketch's contour. Drag T to fewer steps and watch quality degrade gracefully — even 1 step stays usable because the sketch supervises the generator. Toggle Relativistic D off to simulate a vanilla GAN collapsing to a flat, generic output.

Chapter 6: The Contrastive Loss — making the judge read the words

The relativistic loss makes audio realistic. But realistic-and-ignoring-the-prompt is a known GAN failure: the generator makes great sound that doesn't match the text. ARC adds a second, auxiliary loss to force the discriminator to care about the conditioning.

The contrastive objective

Same relativistic form, but now both samples are real — they differ only in their conditioning. One gets the correct condition C; the other gets a batch-shuffled, wrong condition P[C] (some random permutation of the batch's conditions):

The discriminator is trained to maximize this — to score a real sample with its correct caption higher than the same real sample with a mismatched caption. To do that, D_ψ must actually read the condition and judge semantic agreement, not just sniff for high-frequency GAN artifacts. The combined objective is:

The design question only sketch2audio raises

Standard ARC (text-only) shuffles the text. But FlashFoley has two kinds of conditioning with very different granularity: a single global text caption, and dense time-varying local controls F. So: which condition should we shuffle in the contrastive loss? The original ARC paper would say "shuffle everything." The authors test this and find it is a trap.

Two simulated spectrograms for the same prompt. Shuffle text only: the discriminator must read the caption, so generations keep rich text-driven timbre (full frequency content). Shuffle all: the discriminator latches onto the sketch and ignores text — watch the high-frequency content collapse and the prompt get ignored, exactly as in Figure 3.

Chapter 7: Block-Autoregressive Streaming

We now have fast, controllable, high-quality offline generation. But it's still non-autoregressive: it needs the whole sketch first. For live jamming, that's a dealbreaker. Stage 3 makes generation streamable without ever training a causal model.

Step one: teach the model to outpaint

During Stage 1 finetuning, the authors add inpainting/outpainting conditions through the same PTP machinery. They feed the model a masked clean latent (plus the mask itself) where the mask always hides some future context. This trains the model to outpaint — to continue audio given a few conditioning frames. These conditions are kept through ARC post-training too, so both G_φ and D_ψ understand "given this much, generate the next."

Step two: the chunking algorithm (Figure 2)

Pick a block size B < N and a stride k. The model generates the next chunk of B frames by inpainting the last B−k frames of the previous chunk. The trick that makes this run on an unchanged non-causal model: it always generates a full N frames, but only fills the first B sketch-control slots (the rest padded with zeros), and uses a "seconds total" embedding to say how much is real. After each pass, the overlap region is forced to equal the previous chunk for seamless transitions; then the new frames stream out.

The timeline is N=256 latent frames. Each Next chunk generates B frames; the overlap (the last B−k frames of the previous chunk, shown striped) is forced to match for a seamless join, and the fresh k frames (solid) stream out. Smaller B → lower latency but the speed–quality tradeoff worsens; k has little effect on quality.

Chapter 8: Experiments & Results

The setup: base model is Stable Audio Open Small (340M DiT, stereo 44.1 kHz, 256 latent frames ≈ 11.88 s). Trained on WavCaps (~400K audio samples), 40K control-finetuning steps + 70K ARC steps, batch 256 across 4 H100s. Evaluated on VimSketch — ~10K real human vocal imitations of 500 audio classes.

What the metrics mean (don't skip this)

The main table, read like a story

Note the expected tradeoff in the first two rows: adding sketch controls raises FD and lowers CLAP (controls are a harder target than text alone) — but raises MOS sharply, because humans find controllable, sketch-following audio far more useful. The whole point is that ARC then makes this controllable model ~10× faster without giving that back.

Chapter 9: Ablations — the surprising tradeoffs

The ablations plot FD against average relative error (control error relative to the baseline SAOS error, averaged over controls). Two knobs are swept; each reveals something non-obvious.

Ablation 1: filter width & sampling steps (non-AR)

As expected from Sketch2Sound, there's a tradeoff: a small median filter (w=1) gives low control error but worse FD; a large filter (w=5) gives better FD but looser control. The surprise is on sampling steps: dropping from 8 steps to 1 degrades quality only slightly.

Ablation 2: block size & stride (BAR)

Block size B is anti-correlated with both FD and error — a clean speed–quality tradeoff (smaller blocks = lower latency but worse quality). Stride k, however, has little relationship to performance. That means FlashFoley needs surprisingly little preceding context to stay coherent — you can use a large stride (stream more fresh audio per chunk) almost for free.

Each dot is a configuration; lower-left is better (low FD, low control error). Hover/tap points to read the config. Filter & steps: see steps barely move FD, while filter width trades control for quality. Block & stride: see block size B drive the frontier while stride k barely matters.

Chapter 10: Connections & Cheat Sheet

FlashFoley is a synthesis paper: it stands on three prior lines and welds them into the first open-source, fast, controllable, streamable sketch2audio system.

The cheat sheet — every key equation

The one-paragraph summary you could give on a whiteboard

FlashFoley starts from a rectified-flow text-to-audio DiT (Stable Audio Open Small). Stage 1: finetune on the velocity loss while injecting three time-varying sketch controls (volume, pitch, brightness) — plus inpainting masks — through Pre-Transformer Projection, a per-control linear added to the hidden state for ~0.1% new params; a random-width median filter becomes the inference dial between "render exactly" and "loose sketch." Stage 2: ARC post-training distills 50 steps down to a few via a relativistic adversarial loss (score the difference between paired real/fake to kill mode collapse) plus a contrastive loss that shuffles text only (shuffling the sketch makes the discriminator ignore text and lose timbre). Result: 11.88 s stereo 44.1 kHz in 75 ms, ~10× faster, no quality loss. Inference: a block-autoregressive chunker generates overlapping B-frame blocks with forced overlap, cutting streaming latency to ~B·S/N so you hear the first chunk while still sketching.