Live Music Diffusion Models

Chapter 0: The Problem

Imagine standing on stage with a synthesizer that doesn't play notes — it plays music. You whisper "drum-n-bass," and a fresh groove streams out of the speakers. You bend a slider, and the texture morphs in real time. You play a guitar lick, and the machine improvises an accompaniment around you, right now, with no perceptible delay.

That is the promise of interactive streaming music generation: a generative model you can play with, like an instrument, not a render farm you wait on.

The core tension: The models good enough to do this (Google's Live Music Models, MusicGen-Large) are discrete autoregressive giants — over 40 GB of VRAM, billions of parameters. You cannot run them on a laptop. The small, controllable, open-source models that could run on a laptop are diffusion models — and diffusion is fundamentally not streamable.

Why isn't diffusion streamable? Because a diffusion (or flow-matching) model generates a whole chunk of audio by iterating over noise levels, not over time. It attends bidirectionally across the entire chunk at every denoising step. There is no natural "left-to-right, one frame at a time" structure to exploit — so the cheap trick that makes language models fast, KV-caching, simply doesn't apply.

A concrete latency wall

Let's make the inefficiency painfully concrete. To generate one block of audio, you need K denoising steps (say K=50). Naive block-wise diffusion runs a full forward pass over all T frames — both the clean context you already have AND the noisy frames you're generating — on every one of those 50 steps.

System	Per-block forward passes	What gets recomputed
Discrete-AR LMM (encoder-decoder)	1 encode + `o` decodes	Context encoded once, cached
Naive block diffusion	`K` full passes over `s+o` frames	Context re-encoded every step

The diffusion model re-encodes its own fixed, unchanging history 50 times per block. That is the bottleneck. The paper's blunt finding: out of the box, block-wise diffusion is strictly worse than the discrete-AR systems it hoped to replace.

The challenge this paper sets itself: Can we keep diffusion's gifts — small size, data efficiency, fine-grained controllability, an open-source ecosystem — while recovering (and then beating) the inference efficiency of the discrete-AR giants? And can we do it with a finetune, not a from-scratch retrain?

Why can't standard audio diffusion models stream like a discrete-AR language model?

They are too small to hold enough context They iterate over noise levels with full bidirectional attention, so the clean context is recomputed every denoising step — no KV cache applies They only support mono audio

Chapter 1: The Key Insight

Here is the whole paper in one sentence: the clean context and the noisy target are two different kinds of thing, so stop mixing them together at the input.

In a naive setup, the model concatenates the clean history frames and the noisy frames-being-generated along the channel dimension and feeds them through one shared input projection with full bidirectional attention. Because the noisy frames change at every denoising step, and because attention lets everything see everything, the representation of the clean history also changes at every step. So you can never cache it.

The insight: Route the clean context through its own projection, and use an attention mask so the context can never attend to the noisy target. Now the context's hidden states are identical for every noise level — a fixed function of the history alone. That means you can encode it once, cache its keys and values, and reuse them across all K denoising steps. KV-caching for diffusion, unlocked.

The three moves

1. ROUTE

A binary routing mask splits each input frame into "clean context" vs "noisy target" and sends each through a separate input projection (matrix A for noisy, B for clean).

↓

2. MASK

A custom attention mask forbids context frames from attending to target frames. The context's encoding is now decoupled — independent of the noise level.

↓

3. CACHE

Encode the context once, cache its K/V at every transformer layer, and run all denoising steps for the target using the cache. No recomputation.

Two extra superpowers diffusion gets for free

Once you've made diffusion stream efficiently, two things fall out that the discrete-AR giants cannot do:

It's a finetune, not a rebuild. The only genuinely new parameters are the clean-context projection (matrix B). Everything else is the pre-trained checkpoint. Total training: under 8 GPU-hours.
The whole rollout is differentiable. Discrete-AR sampling involves drawing from a codebook (non-differentiable), which is why those systems need brittle RL to post-train. Diffusion sampling is a smooth ODE solve — so you can backprop through an entire multi-block rollout. This enables ARC-Forcing (Chapter 6): RL-free, reward-model-free adversarial post-training that kills error accumulation.

What single architectural change unlocks KV-caching for diffusion?

Separately projecting clean context and masking it so it can't attend to noisy frames, making its encoding noise-independent Increasing the number of denoising steps Switching from stereo to mono audio

Chapter 2: Flow Matching Background

Before we can stream it, we need to understand what these models actually do. LMDMs are built on flow matching with an optimal-transport path (also called Rectified Flow). It's diffusion's cleaner cousin, and it's the dominant paradigm in modern audio generation.

The data: audio becomes a latent

You never operate on raw waveforms — they're too long. A stereo clip a ∈ R^2×L·fs (2 channels, length × sample-rate samples) is first compressed by a VAE into a compact latent x ∈ R^C×T: C channels, T latent time frames. Each x_t is one frame of music — think of it as a fraction of a second of sound, encoded as a vector.

Why a latent, not the waveform? A few seconds of 44.1 kHz stereo is hundreds of thousands of samples. The VAE crushes that by ~1000× into a handful of frames, so the diffusion transformer attends over T ≈ tens of frames instead of hundreds of thousands of samples. Streaming a frame at a time is now tractable.

The forward process: straight-line noising

Flow matching defines a dead-simple corruption: linearly interpolate your clean data with Gaussian noise, parameterized by a noise level k ∈ [0,1]:

x(k) = (1 − k) · x + k · ε, ε ~ N(0, I)

Read it like a slider. At k=0 you have pure clean music x. At k=1 you have pure noise ε. In between, a straight-line blend. Generation is just running this slider backwards, from k=1 (noise) to k=0 (music).

The velocity: what direction is "toward music"?

Differentiate the forward path with respect to k and you get the constant velocity that points from data toward noise:

dx(k)/dk = ε − x := v

That's the entire target. We train a network v_θ(x(k), k) to predict this velocity at any noise level, by regressing against the true v:

L = E_{x, k, ε} ∥ v_θ(x(k), k) − (ε − x) ∥²

Where v_θ is the learned velocity field (a DiT — diffusion transformer), x(k) is the noised input, k is the current noise level, and ε−x is the ground-truth direction. At sampling time we just solve the ODE dx/dk = v_θ in reverse using Euler, Heun, or a "ping-pong" sampler, taking K steps from noise to music.

Worked example: one denoising step by hand

Say a single latent value is x = 0.8 and we drew noise ε = −0.2. At noise level k = 0.5:

Noised input: x(0.5) = 0.5·0.8 + 0.5·(−0.2) = 0.40 − 0.10 = 0.30
True velocity: v = ε − x = −0.2 − 0.8 = −1.0
An Euler step toward less noise (e.g. k: 0.5 → 0.4, step Δk = −0.1): x(0.4) = x(0.5) + v·Δk = 0.30 + (−1.0)(−0.1) = 0.40

The estimate moved from 0.30 back toward the clean 0.80. Repeat for all K steps and you arrive at the music. The interactive below lets you watch this slider in both directions.

Flow Matching: drag the noise level, watch the velocity

noise k

k = 0.50 — halfway between music and noise

Why this matters for streaming: Generation iterates over the noise axis k, completely independent of the time axis t. That decoupling is exactly why naive diffusion uses bidirectional time attention — and exactly the property the paper exploits to cache the time-dimension context.

In flow matching, what does the network v_θ learn to predict?

The final clean sample directly The velocity ε−x — the direction along the straight-line path between noise and data The probability of each audio token

Chapter 3: Block-AR Outpainting (the baseline that's too slow)

To stream, you don't generate the whole song at once. You generate it in blocks. Given s frames of past audio context, generate the next o frames, then slide the window forward and repeat — forgetting the oldest history, encoding your fresh output as new context.

model: p_θ(x_s:s+o | x_1:s, c)

Where x_1:s is the s-frame clean history, x_s:s+o is the next o-frame block we're generating, and c is conditioning (text prompt, sketch, another instrument's audio). This is exactly how the discrete-AR LMMs work too — the difference is purely how each block gets generated.

How naive diffusion conditions on history: channel-concat

Standard diffusion outpainting hands the model its clean history by concatenating it as extra channels. You build a fixed conditioning tensor that holds the clean frames where they exist and zeros everywhere else:

x_concat := [x_clean, 0_s:T]_C

i.e. the first s frame-slots carry the clean context; the remaining o = T−s target slots are zero. The model's actual input each step is this stacked alongside the current noisy latent x(k). Training is almost vanilla flow matching: sample a noise level, append x_clean, predict the velocity.

The inference trick — and why it's wasteful

At inference, after every denoising step the model resets the first s frames of its iterate back to the clean context (at the appropriate noise level), so the history never drifts. Algorithm 1 in the paper does exactly this in a double loop: for each block, for each of K noise steps, run a full DiT pass over s+o frames.

The buried cost: The context frames x_clean are fixed for the entire block. Yet because attention is bidirectional and the input is channel-concatenated, their internal representation is recomputed on all K steps. The paper's complexity for naive diffusion is O((E + D)_1:T · K) — encoding and decoding fused into one operation over all frames, run every step. The discrete-AR LMM is O(E_1:s + Σ D_t) — context encoded once.

Pinpointing the leak: the initial hidden state

Here's the precise mechanism. Write the DiT's input projection as W_init = [A, B], where A handles the noisy latent and B handles the concatenated condition. The initial hidden state before any attention block is:

h^init,k = W_init[x(k), x_concat]_C = A·x(k) + B·x_concat

Now split by region. Because x_concat is zero past the first s frames:

h^init,k_s:T = A·x(k)_s:T (target: depends on noise ✓)

h^init,k_1:s = A·x(k)_1:s + B·x_clean (context: ALSO depends on noise ✗)

That second line is the whole problem. The part of the hidden state that's supposed to "encode the past" is contaminated by the noisy latent A·x(k)_1:s — which is different at every noise level k. So the encoder's input changes every step. Chapter 4 fixes exactly this term.

In naive channel-concat diffusion, why does the clean context's hidden state change at every noise level?

Because the VAE is retrained each step Because the context's hidden state mixes in the noisy latent term A·x(k)_1:s, which differs at every k Because the text prompt is re-embedded each step

Chapter 4: Routing + Attention Masks (the SHOWCASE)

This is the heart of the paper, and it's two small surgical edits to the leaky equation from Chapter 3. Watch the noise contamination disappear.

Move 1 — the routing mask

We already know a priori which frames are context and which are target. So define a binary routing mask r := [0_1:s, 1_s:s+o] — zero on context, one on target — and apply it to the noisy latent before projecting. The input becomes:

h^init,k = W_init[r ⊙ x(k), x_concat]_C

Because r zeros out the noisy latent on the first s frames, the leaky A·x(k)_1:s term vanishes:

h^init,k_s:T = A·x(k)_s:T (target, unchanged)

h^init,k_1:s = B·x_clean (context: pure function of history ✓)

The context's initial hidden state is now the same for every noise level. One leak sealed.

Move 2 — the attention mask

But there's a second leak. Even with a clean initial state, bidirectional attention lets the context frames attend forward to the (noisy) target — so after the first attention block, the context encoding starts changing again as a function of the target. The fix is an asymmetric mask:

Context frames (1:s) may attend only to each other — never to the target.
Target frames (s:s+o) may attend to everything — themselves and all preceding context.

Now the context's representation is fully decoupled from the target at every layer. It is an "encoding" of the past that the target "decodes" against — exactly the encoder/decoder split of an LMM, but implemented implicitly inside one DiT via a mask.

The payoff: Pass the clean context through the DiT once, cache the Key/Value states at every transformer layer, then run all K denoising steps for the target attending against the frozen cache — zero recomputation of the context. Complexity drops to O(E_1:s + D_s:T·K): one encode, then iterate the decode. Identical complexity class to the LMM, achieved by a finetune.

SHOWCASE: build the routing pipeline yourself

Below is the paper's Figure 2 made interactive. Toggle the routing mask and the attention mask on/off and watch what happens to the context's hidden state across noise levels. With both ON, the green "context encoding" stays flat as you sweep noise — that flat line is the cacheable representation. With either OFF, it wobbles, and the cache is invalid.

The LMDM routing surgery — flip the masks, watch the cache become valid

Both masks ON — context encoding is noise-independent → KV-cacheable ✓

The core insight in runnable code

# LMDM forward: separate projections + routing mask + asymmetric attention def lmdm_initial_state(x_k, x_clean, s): # x_k: [B, C, T] noisy latent over all T = s+o frames # x_clean: [B, C, s] the clean history T = x_k.shape[-1]; o = T - s r = torch.cat([torch.zeros(s), torch.ones(o)]) # routing mask # Route: noisy term killed on context, clean term injected via B h = A(r[None,None] * x_k) # A * (r ⊙ x_k) h[..., :s] = h[..., :s] + B(x_clean) # + B * x_clean return h # h[:s] depends ONLY on x_clean -> same for every k def attn_mask(s, o): # Enc-Dec variant T = s + o; m = torch.zeros(T, T, dtype=torch.bool) m[:s, :s] = True # context attends only to context m[s:, :] = True # target attends to everything return m # Inference: encode context once, cache K/V, reuse for all K denoise steps KV = vtheta_encode(x_clean, c, k=0) # ONE pass, cached for j in range(K, 0, -1): # target only, no context recompute v = vtheta(x_k, c, k_j, kv_cache=KV) x_k = solver_step(v, x_k, k_j, k_{j-1})

Why are BOTH the routing mask and the attention mask needed for KV-caching?

They are interchangeable — either one alone is sufficient Routing removes noise from the context's input state; the attention mask stops the context from attending to the noisy target. Both are required to fully decouple the context The masks just speed up training; they have no effect on caching

Chapter 5: Enc-Dec vs Block-Causal

The attention mask is a dial, not a switch. The paper offers two settings, trading off how much you can cache against how fast the model adapts to new context.

Encoder-Decoder LMDM: cache over noise

The Chapter-4 mask (context attends bidirectionally within itself; target sees all). This caches K/V across the K denoising steps of a block. But the full s-frame window uses bidirectional self-attention, so when you slide forward and add a new block to the context, the whole window's encoding changes. You must re-encode all s context frames for every new block. Complexity: O(E_1:s + D_s:T·K) — same class as the LMM.

Block-Causal LMDM: cache over noise AND time

Push the mask further: make the context block-causal — each o-sized block of context can attend only to past blocks (or within itself), never forward. Now adding a freshly generated block doesn't disturb the encodings of earlier blocks (nothing attended forward to them). So after a one-time warmup encoding each block, you cache over both noise and time, only ever encoding the single newest block. Complexity: O(E_s−o:s + D_s:T·K) — strictly faster than the LMM, which must re-encode its whole context per block.

Attention masks side-by-side — what each frame can see

Naive: every frame attends to every frame — nothing is cacheable.

The empirical surprise: Block-Causal is theoretically faster, yet Enc-Dec wins on quality. The reason: Enc-Dec re-reads the full context each block, so it can keep re-interpreting the whole musical history as new audio arrives. Block-Causal freezes old encodings. The paper concludes that the ability to adapt the contextual encoding — even at the cost of more compute and slightly faster drift — is worth it, and focuses on Enc-Dec for the rest of the work.

The inference-time speedup, concretely

One more engineering nicety: at inference, the custom attention masks aren't needed at all. The context encode happens in a fully-bidirectional region (the whole context for Enc-Dec, each block for Block-Causal), and the target decode attends against a clean cache — so both can use highly-optimized flash-attention kernels with no custom-mask overhead. Combined with torch.compile, round-trip latency lands at 110–170 ms on a workstation GPU, with KV-caching giving a 20–25% per-pass speedup.

Why does the paper ultimately prefer Enc-Dec over the theoretically-faster Block-Causal variant?

Enc-Dec re-encodes the full context per block, so it adapts its interpretation of musical history as new audio arrives — yielding better quality Block-Causal cannot use KV-caching at all Enc-Dec uses fewer parameters

Chapter 6: ARC-Forcing (RL-free rollout post-training)

We can stream efficiently now. But there's a deeper sickness in any autoregressive generator: error accumulation. Training supervises the model one block at a time, on ground-truth context. At inference, the model conditions on its own outputs — small errors compound, and after a minute the music drifts into mush.

The standard cures in AR music generation are ugly: brittle RL pipelines, or explicitly-trained reward models. ARC-Forcing avoids both, by fusing two recent ideas.

Ingredient 1 — Self-Forcing

Because diffusion sampling is differentiable (no discrete codebook draw), you can generate a full B-block rollout from the model and backprop through the whole thing. The model trains on its own rollouts — exactly matching the inference scenario. To make this affordable: pick a random number of steps k ~ U[2, K_max] per block and only propagate gradients on the final step (the rest run no-grad). KV-caching keeps the rollout cheap.

Ingredient 2 — ARC (Adversarial Relativistic Contrastive)

Instead of a hand-built reward model, learn the reward as a discriminator D_ψ that you co-train. Give it a generated rollout and a real-music clip (same controls, same starting context), noise both to the same level, and have it score which is real via the relativistic loss:

L_R = E[ f( D_ψ(noised generated) − D_ψ(noised real) ) ], f(x)=softplus(x)

The word "relativistic" is the trick: D_ψ never judges a clip in isolation, only relative to a real one. This sidesteps the degenerate collapses that plague ordinary GAN objectives, and — crucially — it provides supervision on the full multi-block rollout, not single blocks. That's what fights long-horizon drift.

Ingredient 3 — a contrastive term for text-following

To stop the discriminator from fixating on high-frequency texture and ignoring the prompt, add an auxiliary contrastive loss on real music with correct-vs-wrong captions:

L_C = E[ f( D_ψ(real, wrong prompt P(c)) − D_ψ(real, right prompt c) ) ]

where P is a random batch permutation pairing each clip with someone else's prompt. The discriminator is trained on L_R + λL_C (with λ=1), so it rewards both realism and prompt adherence.

The non-obvious gotcha they had to fix: Initializing D_ψ directly from the finetuned LMDM gave a weak discriminator that destabilized training — because D_ψ needs a much larger context window (≈30 s) than the generator (≈10 s). The fix: warm-start the discriminator's backbone on longer audio segments (plain flow-matching, a few thousand iters) before adversarial training. A detail you'd only discover by hitting the wall.

The payoff: fewer steps, no CFG, stable for minutes

After ARC-Forcing the model becomes a few-step generator — it can sample in 1 to 8 steps without classifier-free guidance (using a distilled "ping-pong" sampler), dropping total latency to ~30 ms. And error accumulation is crushed: metrics that steadily degraded over 2 minutes without ARC-Forcing stay flat with it.

Error accumulation over time — toggle ARC-Forcing

drift over 2 min of generation

How does ARC-Forcing avoid needing a hand-trained reward model?

It co-trains a relativistic discriminator that learns the reward by comparing generated rollouts against real music — no separate reward model needed It uses reinforcement learning with a fixed reward It copies the rewards from the discrete-AR LMM

Chapter 7: The Live Music Design Space

Because LMDMs model p_θ(x_s:s+o | x_1:s, c) with no restriction on what c is, a single architecture unifies tasks that prior work built bespoke systems for. The paper organizes the space along two axes.

	Global (c ∈ R)	Local (c ∈ R^T)
Instrument-like describes the target output; latency = synth speed	Text prompt + prompt transitions	Sketch controls (top-k CQT pitch, loudness curve)
Accompaniment-like external context on its own clock; must balance reactivity vs latency	—	Live jamming / accompanying stem

The three concrete modalities

Global text-conditioning. c is a single text prompt with no temporal axis. Interaction comes from prompt transitions — cross-fading "driving EDM" into "sad country" in real time. Classified as instrument-like because you actively modulate it.
Instrument-like sketch controls. c ∈ R^T gives the model time-aligned hints about what the future block should contain — e.g. top-k CQT bins (a pitch sketch) plus a loudness envelope. You "draw" the music's contour and the model fills in timbre.
Accompanying stem generation. The condition is a separate live audio stream (a human guitarist, say). The hard part: there are no features describing the target block — the model must infer a musically compatible continuation from the causal history of the other instrument. And it operates with a future visibility t_f < 0: it only sees the partner's stream up to some cutoff before the target block, to leave room for compute latency.

Why future visibility is the whole jamming problem: If t_f ≥ 0, the model can see the partner's audio that corresponds to the very block it's generating — perfectly synced, but impossible to stream (you'd have to wait for audio from the future). If t_f < 0, you reclaim that time for inference, but the model must predict what the partner will play. The paper sweeps t_f from ≈−2 s to +2 s and measures inter-stem alignment with CoCoLA score: tighter visibility hurts alignment, looser visibility kills real-time interaction. Live jamming lives in that trade-off.

In the live-jamming (accompaniment) setting, why is t_f < 0 (negative future visibility) necessary?

To reduce audio quality on purpose Because the model only works in mono To leave time for inference — the model can't see the partner's audio that aligns with the block it's generating, so it must predict ahead

Chapter 8: Experiments

The headline numbers, all from a model with half the parameters and trained on ~100× less data than the discrete-AR LMMs.

Time-to-first-frame (lower = more interactive)

Text-conditioned global metrics (Table 1)

Metrics: FD/KD (lower = better audio quality vs real distribution), CLAP (higher = better text adherence), TTFF (time-to-first-frame, seconds). With audio priming:

Method	D-NFE	Sampler	TTFF↓	FD↓	CLAP↑
Magenta RealTime	800	DPM++	≈4	72.1	0.35
Stable Audio Open	100	—	10.35	96.5	0.41
MusicGen-Large	2.4K	—	10.81	190.5	0.31
LMDM (ED)	50	Euler	0.11	35.4	0.23
LMDM (ED) + ARC-Forcing	8	Ping-Pong	0.03	29.0	0.32

Read the bottom rows: ARC-Forcing takes the Enc-Dec LMDM from 50 denoising steps down to 8, cuts TTFF from 0.11 s to 0.03 s (vs 4–11 s for the baselines), and improves both quality (FD 35→29) and text adherence (CLAP 0.23→0.32). Faster and better — the post-training isn't a trade-off, it's a free lunch.

Per-window: ARC-Forcing kills drift over time

Generating up to 2 minutes and computing metrics on sliding windows: without ARC-Forcing, nearly every metric (FD-OpenL3, KL-PaSST, CLAP) degrades monotonically as the rollout grows. With ARC-Forcing, the curves stay flat. This is the clearest demonstration of error-accumulation control — and it holds for both Enc-Dec and Block-Causal (with one unexplained exception: Block-Causal FD on non-primed generation).

Prompt transitions and accompaniment

Prompt transitions. Naively cross-fading prompts failed — the strong conditioning from past audio dominated, and adding plain CFG over-saturated at low step counts. Two fixes: (1) when one prompt crosses into dominance, drop the first 180 frames of context to weaken the past audio's grip; (2) adapt CFG++ to the distilled ping-pong sampler so you can crank text importance without flying off-manifold. With both, transitions match the LMM.
Accompaniment. CoCoLA alignment score vs future visibility t_f traces the predicted trade-off: more visibility = tighter sync, but you need t_f<0 to actually stream. The model interpolates gracefully between the ground-truth-pair ceiling and the random-pair floor.

The takeaway result: A small, open-source-derived diffusion model, finetuned in <8 GPU-hours and post-trained without any RL, matches a 40 GB+ discrete-AR system on quality and text-following while being two orders of magnitude faster to first sound.

What does the per-window evaluation over 2 minutes primarily demonstrate?

That bigger models are always better That ARC-Forcing keeps quality/text metrics flat over time, whereas without it they degrade monotonically as the rollout grows That diffusion can't generate more than a few seconds

Chapter 9: Real-Time Deployment — the generative delay

The paper doesn't stop at benchmarks. It ships a real instrument: a sketch-conditioned LMDM deployed as a "generative delay" on a consumer gaming laptop, put in front of real musicians, and used in a live performance.

What "generative delay" means

A classic delay pedal repeats your playing after a fixed time. A generative delay takes the musician's improvisation as the conditioning sketch, and instead of echoing it, regenerates it with new timbre and texture — a hallucinated reflection of what you just played. The musician plays, and a transformed version comes back, in time, as a new musical voice to react against.

The deployment stack

Train

PyTorch flow model → LMDM finetune (<8 GPU-hr) → ARC-Forcing (few-step, no CFG)

↓

Export

ONNX export of the few-step generator + VAE; flash-attention paths, no custom masks at inference

↓

Host

C++ / JUCE audio plugin runs the ONNX graph in the real-time audio thread on a consumer laptop GPU

↓

Play

~30 ms round trip → musician improvises, generative delay responds in time

Why each piece is non-negotiable for "playable"

KV-caching recovers the per-pass compute so a small GPU keeps up — without it, diffusion was strictly slower than the giants.
ARC-Forcing collapses 50 steps to ~1–8 and removes CFG (which doubles forward passes), getting latency under the ~30 ms threshold where it feels immediate.
ONNX + JUCE moves the model out of Python and into the audio thread — Python's GIL and scheduling jitter are fatal to real-time audio.
Sketch conditioning gives the musician a real-time control surface (their own playing) rather than a text box you can't perform with.

The arc of the whole paper, in one line: Efficiency (KV-cache) buys you real-time; stability (ARC-Forcing) buys you long-form; controllability (sketch/jam) buys you playability. All three together turn a research checkpoint into something a musician can hold and perform with on hardware they already own.

Honest limitations

Block-Causal is faster on paper but the authors' implementation was slower in wall-clock — "suboptimal implementation," they admit. Fused custom kernels are flagged as future work.
CLAP/text-adherence still trails the largest discrete-AR systems on some settings — the small model gives up a little semantic precision for its enormous speed/size win.
One unexplained result: Block-Causal's FD degrades at high step count on non-primed generation; left for future investigation.

Why is the model exported to ONNX and hosted in a C++/JUCE plugin rather than run in Python?

Real-time audio threads need predictable, low-jitter execution; Python's overhead and scheduling make it unsuitable for the audio thread ONNX makes the model more accurate JUCE is required to train the model

Chapter 10: Connections

LMDMs sit at the intersection of three fast-moving lineages — efficient streaming, autoregressive diffusion, and adversarial post-training. Here's how the pieces relate.

2020

DDPM — iterative denoising diffusion (the noise-axis generator)

↓

2022

Rectified Flow / Flow Matching — straight-line ODE paths, the LMDM backbone

↓

2024

Diffusion Forcing → causal video diffusion — per-frame noise schedules, history conditioning

↓

2025

Self-Forcing (video) + ARC (offline audio) — rollout post-training + relativistic adversarial reward

↓

2026

LMDM — routed-context KV-caching + ARC-Forcing → playable streaming music diffusion

The KV-cache idea, generalized

The deep lesson transcends music: any generative process with a fixed conditioning context and an iterative refinement loop can be sped up by separating the context's representation so it becomes loop-invariant and cacheable. The same "route it separately, mask it, cache it" recipe could apply to image inpainting, video extension, or any conditional diffusion where part of the input never changes during sampling.

Cheat sheet

Symbol / term	Meaning
`x ∈ R^C×T`	VAE latent: C channels, T time frames of compressed audio
`k ∈ [0,1]`	Noise level. k=0 clean music, k=1 pure noise
`v = ε − x`	Flow-matching velocity: direction along the noise↔data path
`v_θ`	Learned DiT velocity network (the generator)
`s / o`	Context length / output (target) block length, in frames
`r = [0_1:s, 1_s:s+o]`	Routing mask: 0 on context, 1 on target
`A, B`	Input-projection sub-matrices for noisy latent / clean context. B is the only new param
`D-NFE`	Diffusion network function evals (denoising steps) per block
`TTFF`	Time-to-first-frame — the latency metric that decides "playable"
`t_f`	Future visibility in jamming. t_f<0 = predict ahead to leave compute time
`L_R / L_C`	ARC relativistic loss / contrastive (right-vs-wrong-prompt) loss

Related lessons: Veanors: Flow Matching • Veanors: SD3 / Rectified Flow • Veanors: Stable Video Diffusion • Veanors: Music Transformer

"Treat the model as an instrument to be played with, not a render farm to wait on."

— the LMDM thesis, paraphrased