Real-Time Musical Co-Performance
with a Look-Ahead Diffusion Model

Chapter 1: The Key Insight

A human ensemble doesn't react instantly either — nerve and muscle have latency too. Yet musicians stay locked together. How? They anticipate: each player continuously predicts what's about to happen and commits to a note slightly ahead of needing it, trusting everyone to land in the shared groove.

The paper steals exactly this. Instead of generating the audio that plays now, the model generates audio for a moment w steps in the future. While that future chunk waits in a buffer, playback consumes the previously generated chunk. By the time playback reaches the future chunk, the model has already finished it. The slow inference happens in the gap the look-ahead created.

The three pillars that make it real

What the whole system is, in one breath

A human plays into MAX/MSP. MAX captures the live audio, sums the non-target instruments into a context mixture, and ships it over OSC/UDP to a Python server. The server encodes context into Music2Latent's compressed latent space, masks out the future region it must invent, and runs a latent diffusion model (or its distilled consistency twin) to inpaint/outpaint the target stem. The decoded audio streams back and gets written into the buffer one step ahead of playback. Repeat forever. We will build every piece — and you'll see exactly which tensor flows where.

Chapter 2: Background — Stems, Context, and the Task

Before the streaming machinery, let's nail the data. A song is a sum of stems — isolated instrument tracks. The paper works with four: bass, drums, guitar, piano, from the Slakh2100 dataset (synthesized multi-track MIDI rendered to audio at 44.1 kHz).

Write the mixture as the sum of S mono stems:

Symbol by symbol, with intuition:

• x_s — one instrument's waveform (e.g. the bass), a 1-D array of samples.
• T — duration in seconds. The model's fixed window is T = 6 s.
• sr — sample rate, 44,100 Hz. So a 6-s clip is T·sr = 264{,}600 samples.
• S — number of stems (4 here).

The task: predict one stem from the rest

Accompaniment generation means: given everything except the target instrument, synthesize the target. The context is the sum of all the other stems:

Read it as: "the predicted bass x̂_s is a function f of the mixture of drums + guitar + piano." During training, one target stem and its one-hot label s are picked at random; the context is the sum of the remaining stems. This is exactly the live scenario: the drummer plays, and the model fills in the bass.

The shapes we'll be tracing

Chapter 3: The Sliding-Window Protocol

The model only sees 6 seconds at a time, but a performance lasts minutes. How do you generate forever from a fixed window? You slide the window forward, generating only the small new slice at each step and reusing what you already made.

Define the step size as T·r, where r ∈ (0, 1] is the fraction of the window advanced each step. Small r = frequent, fine-grained updates; large r = coarse, infrequent ones. At each step the model conditions on the current context plus its own previous prediction shifted forward in time:

Decode it:

• S_rT — the time-shift operator: it slides the previous prediction r·T seconds into the past portion of the new window.
• x̂_s,t−1 — what the model generated last step. It becomes known history this step.
• Only the final r·T slice of the window is unknown; the overlapping (1−r)·T is kept fixed.

So at each step the model inpaints just the new slice and copies the rest. The window advances by T·r and the process repeats — continuous generation, glued together by the overlap.

Worked example with real numbers

Take the paper's defaults: T = 6 s, r = 0.25. Then the step size is T·r = 1.5 s (66,150 samples). Each step, the window slides 1.5 s forward; the model regenerates only the newest 1.5 s while the older 4.5 s carry over as fixed context. In the latent grid (64 frames for 6 s), 1.5 s maps to exactly T_z·r = 64×0.25 = 16 latent frames. That whole-number constraint matters — you cannot mask a fractional frame.

Green = audio already known (observed or generated earlier). Red = the new slice the model inpaints this step. Drag r to change step size and hit Advance to slide the window. Notice only the last r·T is ever red.

Chapter 4: Look-Ahead — Three Regimes (the showcase)

Now the heart of the paper. We have a sliding window — but where in that window does the current playback moment sit? That position, controlled by the integer look-ahead depth w, decides everything: how much context the model has, whether it's filling a gap or extending past the edge, and whether real-time is even possible.

Define w ∈ Z as the number of step-sized intervals T·r between the current playback position (t = 0) and the start of the predicted slice (always the last r·T of the window). Three regimes fall out, by the sign of w:

Where does the current moment sit? (worked)

The paper's key geometric fact: across all regimes, the current time t=0 sits (w+1)·T·r seconds from the right edge of the window. Walk through it with r=0.25, T=6:

At w=1, r=0.25, half the window is unobserved future. That's why quality drops in look-ahead — and why the paper constrains r < 0.5 when w=1 (at r ≥ 0.5 the prediction window would fall entirely outside the context).

The bar is the 6-s receptive field. Green = known audio, striped green = the look-ahead buffer (already generated, waiting to play), red = the slice being generated now. The white line is the playback head t=0. Drag w across −1, 0, 1 to see inpainting become outpainting; hit Run to watch playback consume the buffer while the next slice is generated.

Chapter 5: The Latent Diffusion Backbone

Diffusion on raw 264,600-sample waveforms would be hopelessly expensive. So the model never touches raw audio during diffusion — it works in a compressed latent space produced by a frozen autoencoder, Music2Latent.

The encoder E maps a mono 6-s clip [1, 264600] to a 2-D latent z ∈ R^T_z×F_z = [64, 64]. That's a temporal compression of 264600/64 ≈ 4134× — the audio is squeezed down a time axis (64 frames) and projected into a 64-bin frequency-like axis. The decoder D rebuilds the waveform. Both are frozen during diffusion training.

The score-matching objective

The backbone g_φ is a SongUNet (DDPM++ style). It's trained by denoising score matching (DSM): corrupt a clean latent with noise, ask the network to recover the clean version. Corruption is additive: z_n = z₀ + σ_nε, where ε ∼ N(0,I) and σ_n is the noise level. The loss:

Symbol by symbol:

• z_s — the clean target-stem latent we want back.
• σ_n — noise level, drawn from a log-normal ln(σ)∼N(P_mean, P_std²) (paper: −1.2, 1.2). Big σ = very noisy, easy to spot direction; small σ = nearly clean, fine detail.
• s — the one-hot instrument label (4-dim): tells the network which stem to produce.
• z_context^(s) — the latent of the context mixture, concatenated as an extra input channel so the network sees what to harmonize with.

The network learns the score ∇_z log p(z) — the direction in latent space that points toward cleaner, more plausible audio. Inference follows that field downhill from pure noise to a clean stem via an ODE solver (DPM-2), in just 5 denoising steps (10 forward passes with 2 resamples each).

The diffusion never touches raw audio: it lives entirely on the 64×64 latent grid between encoder E and decoder D (both frozen). Click each stage to see its tensor shapes.

Chapter 6: Masked Context Conditioning

The sliding-window protocol was defined in the audio time domain, but the diffusion model lives in the latent grid. How do you realize "the future part of the window is unknown" inside a 64×64 latent? With a binary mask that zeros out the future region.

The current moment sits (w+1)·T·r from the right edge (Ch 4). In latent frames, that's T_z·r·(w+1) frames of the right side that are unavailable. Define the context mask:

The masked context is z_context ⊙ M_context — multiply elementwise, zeroing the future frames. The constraint T_z·r ∈ N (a whole number of latent frames) is why r can't be arbitrary: you cannot mask half a frame.

At inference, the unmasked target region (M_s=1) is filled with last step's prediction and held fixed; the masked region (M_s=0) starts as Gaussian noise and gets denoised. This asymmetric masking is what preserves continuity across steps while regenerating only the truly new audio.

Why randomize r during training?

At training time the masks for r ∈ {0, 0.125, 0.25} and w ∈ {−1, 0, 1} are applied at random. This exposes the model to every degree of missing context, so a single network handles all three regimes at inference — no separate model per look-ahead setting. It's curriculum-by-randomization: the network learns to be robust to "how much future is missing."

The 64×64 latent grid (time across, freq down). Green columns = kept (mask=1), red columns = zeroed/regenerated (mask=0). Toggle between the context mask (loses w+1 future steps) and the target mask (only the last step is open). Notice how they differ for the same w, r.

import torch

def context_mask(Tz, Fz, r, w):
    """1 where context is known, 0 where future is unavailable."""
    cut = int(Tz - Tz * r * (w + 1))    # whole-frame boundary
    M = torch.zeros(Tz, Fz)
    M[:cut, :] = 1
    return M

def target_mask(Tz, Fz, r):
    """1 = reuse last step (fixed), 0 = the new slice to denoise."""
    cut = int(Tz - Tz * r)
    M = torch.zeros(Tz, Fz); M[:cut, :] = 1
    return M

# r=0.25, w=1, Tz=64 -> context keeps 32 frames, target keeps 48
print(context_mask(64,64,0.25,1).sum(0)[0].item())  # 32
print(target_mask(64,64,0.25).sum(0)[0].item())   # 48

Chapter 7: Consistency Distillation

Recall the trade-off from Ch 4: deeper look-ahead = worse quality. The paper's escape hatch is to attack the other variable — speed. If the model were fast enough, you could use a shallower look-ahead (smaller r), keeping more context and better quality. So: make diffusion faster.

Diffusion is slow because it integrates an ODE over many steps. Consistency distillation (CD) trains a student network g_ω to jump from any point on the diffusion trajectory directly to the clean data estimate — collapsing the iterative walk into one or two steps.

How the student is trained (CTM formulation)

The student is architecturally identical to the teacher. Teacher targets are made by running k ODE solver steps from noise level σ_n down to σ_n−k:

The student is then trained to match that target in one jump, via the CD loss:

The two terms: a target (the EMA student at the lower noise level) and an estimate (the live student at the higher noise level). Matching them forces consistency across noise levels. The target uses a stop-gradient EMA of the student's own weights — a slowly-updated copy that gives a stable goalpost:

Why EMA? If the student chased its own instantaneous output, training would collapse (it could trivially output anything and "match itself"). The slow-moving EMA breaks that feedback loop — it's like a teacher who only slowly revises their answer key. Finally, CD is augmented with the DSM loss for direct data grounding:

The payoff — worked from the timing table

The diffusion model runs 5 denoising steps × 2 resamples = 10 forward passes. The CD model uses just 2 consistency steps. On the Paris RTX A6000 (config 2), sampling drops from 480 ms → 88 ms — a 480/88 ≈ 5.4× speedup. That speedup is the difference between meeting and missing the real-time deadline at small step sizes.

From pure noise (top) to clean data (bottom). The orange path is the teacher walking the ODE one small step at a time. The green path is the distilled student teleporting in 2 jumps. Both reach the same clean target — the student just gets there ~5× faster. Drag teacher steps to compare.

Chapter 8: The MAX/MSP Bridge

A fast, look-ahead model is useless if a musician can't plug into it. The second half of the paper is pure systems engineering: connecting a Python GPU model to a live MAX/MSP instrument with minimal latency. The architecture is client–server: MAX is the client; Python is the server; they speak OSC over UDP.

The multi~track external (C++)

The authors wrote a custom MAX external in C++ called multi~track. Why C++ and not a native MAX patch? Because MAX's control-rate messaging would bottleneck the high-throughput audio streaming — dropping to C++ bypasses that constraint. The external is the sole bridge between MAX and the server, managing the whole lifecycle of audio data:

• Reads context from a shared multichannel buffer~ (the live audio the performer plays in).
• Transmits it to the server over UDP as chunked OSC messages.
• Writes the server's predictions back into the buffer as they arrive.

The windowed read/write — implementing w directly

Each prediction cycle is triggered by a predict <curr> message, where curr is the most recently crossed step boundary (a multiple of r·T samples). The external then:

• Reads context from the interval [curr − r·T,  curr] — the just-played audio.
• Writes the prediction to [curr + w·r·T,  curr + (w+1)·r·T] — w steps in the future.

Chunked OSC with stale-response detection

Audio is sent as fixed-size UDP packets (default 4,410 samples = 0.1 s, giving 15 packets per 1.5-s step). Each OSC message carries a step identifier, a chunk index, and total chunk count. The step ID lets the server discard stale responses — if a new prediction cycle started, results from the old one arriving late are dropped. The chunk index lets the server reassemble samples in order and self-trigger inference once all chunks land.

A fade argument crossfades chunk boundaries: the server returns a few extra samples before the nominal write window, and the external applies a ~20 ms linear fade-in to suppress clicks at the seams between consecutive generated chunks.

The server side — rolling buffers

The Python server holds Music2Latent and the diffusion/CD model on GPU (or Apple MPS), ready between calls. It keeps two rolling buffers — context audio and generated latents — shifted left by r·T_z each cycle to stay aligned with the sliding window without explicit sync with MAX. On each cycle it encodes context, applies M_context, runs the inpainting denoiser, decodes the new r·T slice, and streams it back.

Watch a chunk travel: MAX → (OSC/UDP) → server encode → sample → decode → (OSC/UDP) → MAX. Stage timings are the paper's config-2 (San Diego ↔ Paris) numbers. Toggle CD to watch the sampling stage shrink and the whole cycle drop under the 1.5-s deadline with margin.

Chapter 9: Results — Quality vs. Latency

Two things get measured: generation quality across look-ahead regimes, and end-to-end latency on real hardware. The story is consistent across both: things are great in Retrospective and degrade with look-ahead — and CD trades a little quality for a lot of speed.

Quality — three metrics, three regimes

Evaluated on Slakh2100 against the StreamMusicGen baseline (comparison is indicative, not direct — different codec, sample rate, instrument set). Three metrics:

• COCOLA — harmonic + rhythmic coherence between the generated stem and the mixture (higher = better).
• Beat F1 — agreement of detected beat positions between stem and mixture (higher = tighter rhythm).
• FAD (Fréchet Audio Distance) — perceptual audio quality vs. real distribution (lower = better).

X-axis = net look-ahead T·r·w (seconds), split into Retrospective (green), Immediate (gray), Look-ahead (salmon) zones. Toggle metric. Watch diffusion (orange) and CD (teal) peak in Retrospective and decline as the model is forced to generate further into the unseen future.

Latency — does it actually run in real time?

Each cycle has five stages: MAX→server transfer, encode, sample, decode, server→MAX transfer. At r=0.25 the deadline is T·r = 1500 ms. The dominant term is sampling, and that's exactly what CD shrinks:

At r=0.25 all four deployments meet the deadline. But push to r=0.125 (750 ms budget) and the diffusion model fails (889 ms > 750 ms) while CD still passes (497 ms). That gap — diffusion failing where CD survives — is the entire justification for distillation.

The minimum feasible step (worked)

For local deployment, total cycle time decomposes as d(r) = d_compute + c·r (fixed compute + transfer scaling with chunk size). The real-time constraint d(r) < T·r rearranges to the minimum feasible step ratio:

Plugging the paper's best-case A6000 numbers (d_compute≈596 ms for diffusion, 204 ms for CD, c≈80 ms, T=6 s): r*≈0.100 for diffusion vs. r*≈0.034 for CD. The distilled model can theoretically run at roughly 3× finer step sizes — tighter timing, more context, better quality. (Though such fine steps weren't in training, so quality there is unverified — honest limitation flagged by the authors.)

Chapter 10: Connections & Cheat Sheet

This paper sits at the intersection of three lineages: generative audio modeling, latency-compensation control, and live computer-music systems. Its real novelty is treating inference latency as a first-class musical constraint and engineering around it.

The cheat sheet — every key equation

Why this work matters

• It reframes latency as the design axis. Most music-gen papers optimize quality; this one says "quality is meaningless if it arrives late" and builds the whole method around the deadline.
• The look-ahead/masking paradigm is transferable. Any model with an inpainting/outpainting conditioning scheme can adopt the sliding-window look-ahead parameterization (r, w).
• Distillation buys timing headroom, not just speed. Faster sampling lets you use shallower look-ahead, which restores context, which restores quality — a virtuous loop.
• RTAP closes the tools gap. A model-agnostic C++ MAX external + ready-to-use patch means musicians can actually plug in — the missing piece in prior accompaniment systems.

The whiteboard summary

A latent diffusion model generates one instrument stem conditioned on a mixture of the others, working entirely in Music2Latent's frozen 64×64 latent space. To run live despite ~1-second inference, it generates ahead of playback: a sliding window advances by T·r each step, and a look-ahead depth w places the predicted slice w steps into the future, hiding inference behind a playback buffer. The unknown future is realized by zeroing latent frames with an asymmetric context/target mask, randomized over r, w during training so one model serves all regimes. Consistency distillation collapses 10 denoising passes to 2 (5.4× faster), letting finer steps meet the deadline. A C++ MAX/MSP external streams chunks over OSC/UDP to a Python server and writes predictions w steps ahead in a shared buffer. Quality peaks in Retrospective, degrades gracefully with look-ahead, and beats the StreamMusicGen baseline at practical look-ahead depths — while confirming that high-quality look-ahead generation remains open.