SingSong: Generating Musical
Accompaniments from Singing

Chapter 1: The Key Insight

SingSong's central move is almost cheeky: if paired data doesn't exist, manufacture it.

Take a giant corpus of ordinary recorded music — a million songs, 46,000 hours. Each is a mix of vocals and instruments glued together. Now run a state-of-the-art source separation algorithm (the authors use MDXNet) over every clip. A source separator is a neural network that pulls a mix apart into its constituent stems: it outputs an estimated vocal track. Subtract that estimated vocal from the original mix and you are left with an estimated instrumental.

The second move: don't model waveforms, model codes

For the modeling wall, SingSong borrows the trick that powers WaveNet's successors: don't generate raw audio samples — generate discrete audio codes. A neural codec compresses a 16 kHz waveform into a short sequence of integer tokens (think: "the 47th nearest codebook vector"), and a decoder turns those tokens back into audio. Now the problem becomes a familiar one — autoregressive language modeling over a vocabulary — and you can reuse the entire Transformer toolbox.

Concretely, SingSong adapts AudioLM, a model that generates audio unconditionally by predicting these codes, into a conditional "audio-to-audio" model: feed it codes of the vocals, have it predict codes of the instrumental.

Chapter 2: Audio Codes — the Vocabulary of Sound

Everything in SingSong is built on top of two ways of turning a waveform into integers. We need to understand both precisely, because the entire paper is a story about which codes to feed in and which to leave out.

Why model codes at all? (manufacturing the need)

A 10-second clip at 16 kHz is 160,000 raw samples, each a 16-bit integer (65,536 possible values). An autoregressive model over that is a 160,000-step sequence with a 65,536-way softmax. Intractable. A codec replaces it with a function pair:

Read every symbol: ℝ^f_sT is the raw waveform — f_s samples per second times T seconds. V is a finite vocabulary of codes (like words in a language). f_k is the code rate — codes per second, and crucially f_k \ll f_s. We then model a proxy distribution over codes \hat w = Enc(w) instead of over the waveform, and synthesize by sampling codes and decoding.

Acoustic codes: SoundStream + residual vector quantization

The acoustic codec is a pretrained SoundStream model. It encodes a 16 kHz waveform into 12-dimensional vectors of codes at 50 Hz — but those 12 dimensions are not independent. SoundStream uses residual vector quantization (RVQ): quantize the signal with codebook 1, take the residual (what codebook 1 missed), quantize that with codebook 2, and so on for 12 levels. Each level refines the previous one.

Each level snaps the leftover residual (orange) to its nearest codebook vector (teal dots) and passes the new, smaller residual to the next level. Reconstruction = sum of all chosen vectors. Notice the error collapses fastest in the first few levels — that's why "coarse = first 4" is enough to recognize the sound.

Semantic codes: w2v-BERT + k-means

Acoustic codes are great at fidelity but poor at long-range structure — they describe local timbre, not "this is a chorus in C major." AudioLM's fix is a second, slower stream: semantic codes. Take a self-supervised speech/audio model (w2v-BERT), grab an intermediate layer's continuous embeddings, and quantize them with k-means (k = 1024) into integers at just 25 Hz.

Semantic codes are trained with a self-supervised objective, not an audio-reconstruction loss, so they capture meaning (phonetic/musical content) rather than waveform detail. They are the model's sense of "what is happening musically," at a low enough rate to model long-term structure.

"Flattened" means the 4 coarse code dimensions per 50 Hz frame are unrolled into a single 1-D sequence: 4 × 50 = 200 tokens per second.

Chapter 3: AudioLM — the Model We Adapt

SingSong does not invent a generative model from scratch; it adapts AudioLM. To adapt something you must first understand it, so let's build AudioLM's factorization the way you'd build a language model.

AudioLM generates audio unconditionally by modeling the joint distribution over semantic and acoustic codes. The naive thing would be one giant model over all codes at once. Instead AudioLM factorizes the joint into a cascade of three easier conditional models, each handing off to the next:

Read it right-to-left, the order of generation:

• Stage 1 — P(Sem(w)): generate the slow semantic sketch — "what happens, structurally." A small Transformer, cheap because the rate is only 25 Hz.
• Stage 2 — P(Coarse(w)\mid Sem(w)): flesh the sketch into coarse acoustic codes — recognizable timbre and harmony.
• Stage 3 — P(Fine(w)\mid Coarse(w)): add fine acoustic detail for fidelity.

How conditioning works mechanically

Each stage is an autoregressive, decoder-only Transformer — a "language model" over codes. Conditioning is dead simple: concatenate the condition sequence in front of the target sequence and let causal attention do the rest. Stage 2's input is literally [semantic codes; coarse codes], and the model predicts the coarse part. No cross-attention, no fancy adapters — just prefix concatenation.

To sample, run the three stages in series, then throw the semantic codes away (they were scaffolding) and feed only the acoustic codes to SoundStream's decoder to get a waveform.

Chapter 4: Adapting to a Conditional Seq2Seq Model

Now the adaptation. AudioLM generates instrumentals from nothing. SingSong wants instrumentals given vocals. So we condition every stage on a featurization of the vocals, Feats(x), and write the analogous factorization for the instrumental codes y:

Compare to AudioLM in Chapter 3: identical structure, but Stages 1 and 2 now also condition on Feats(x) — the vocal features. Stage 3 (fine codes) does not see the vocals, by the same independence assumption: Fine(y) \perp Sem(y), Feats(x) \mid Coarse(y). Fine detail is reconstructible from coarse codes alone, vocals or not.

The seq2seq objective, symbol by symbol

Build the target sequence by concatenating the instrumental's semantic and coarse codes:

Here S is the semantic vocabulary (1024), V the acoustic vocabulary; f_k=225 because semantic (25/s) + 4×coarse-acoustic (200/s) = 225 codes per second. The model factorizes the target autoregressively:

Decoding the symbols: &xhat; = Feats(x) is the input vocal feature sequence. E_φ is the T5 encoder (params φ) — it reads the whole vocal once and produces a contextualized memory. D_θ is the T5 decoder (params θ) — it generates instrumental codes left-to-right, attending to its own past outputs &yhat;_<t and the encoder memory via cross-attention. Loss is the usual next-token cross-entropy on the discrete codes — not on waveforms (this is why Figure 1 of the paper has a footnote: loss lives in code space).

This is the SHOWCASE — build the pipeline yourself

The sim below is the whole SingSong training/inference path made interactive. A toy "vocal" waveform is encoded to codes, the T5 maps vocal codes → instrumental codes, and the instrumental is decoded and mixed back with the voice. Toggle which code streams condition the model and watch the data flow update — including the failure you'll learn about in Chapter 5.

The default Noisy / S-SA setting is: input semantic ON, input acoustic OFF, noise ON. You'll see why in Chapters 5–6 — turning input acoustic ON makes the model "cheat" by copying ghosts.

Chapter 5: The Trap — Learning to Cheat

Here is where a clean idea meets a dirty reality. The training vocals are source-separated, not studio-clean. No separator is perfect, so the "vocal" track still contains faint, barely-audible artifacts of the original instrumental — ghostly bleed-through of the drums and bass the separator failed to fully remove.

A powerful code-modeling network notices this immediately. Why bother learning the hard task — compose an instrumental that musically fits the voice — when there's an easier one available: decode the ghost of the real instrumental that's already faintly present in the input? Gradient descent always takes the easy path.

The two-distribution mismatch

This is a textbook train/test distribution shift, and it's worth naming the two regimes precisely because the paper's metrics revolve around them:

A model that secretly relies on the ghost scores great on FAD_s (ghost present) and terribly on FAD_i (ghost gone). The generalization gap is their difference:

A large positive Δ is the fingerprint of cheating: the model is far worse exactly when the crutch is removed. The entire goal of the featurization work in Chapter 6 is to drive Δ toward zero — to make the model equally good (and good for the right reasons) whether or not a ghost is present.

Chapter 6: Featurization — Closing the Gap

How do you stop a model from reading a watermark? Two complementary ideas, both about what you feed the encoder.

Fix 1 — Add noise to hide the ghost

Define a noised vocal Noise(x) = x + z, with z \sim \mathcal{N}(0,σ²) white noise, and featurize that: Feats(x) = Feats_Def.(Noise(x)). The amplitude σ is tuned to sit in a sweet spot: louder than the instrumental artifacts, but quieter than the vocals. The noise floods out the faint ghost (so the model can't copy it) while leaving the actual singing perfectly intelligible.

Fix 2 — Drop the acoustic codes from the input

Recall the default input featurization concatenates semantic + coarse acoustic codes of the vocals (call it SA). But the acoustic codes are exactly the high-fidelity ones that carry the ghost — they describe local waveform texture, artifacts included. The semantic codes are coarse, self-supervised, and capture musical meaning, not waveform bleed.

So: condition on the vocal's semantic codes only (drop acoustic). The naming convention is input–target over {S = semantic, A = acoustic, SA = both}:

The result, quantified

Independently, both fixes shrink the gap. Together — Noisy / S–SA — they essentially eliminate it: FAD_i and FAD_s become nearly identical, and training stabilizes so much that early stopping barely matters. Versus the naive Clean / SA–SA adaptation, the best small model improves FAD_i by 55% relative.

Bars use the paper's Table 1 FAD numbers. Left = FAD_i (clean vocals, real use); right = FAD_s (separated vocals, training-like). The dashed line is the gap Δ. Both toggles ON = the green "Noisy / S–SA" winner.

The core insight as runnable code

# SingSong's two featurization fixes, in ~15 lines.
# Goal: build the vocal CONDITIONING that the T5 encoder reads,
# in a way that cannot smuggle source-separation artifacts.
import torch

def featurize_vocals(vocal_wav, soundstream, w2vbert,
                     add_noise=True, drop_acoustic=True, sigma=0.01):
    # Fix 1: additive white noise drowns the faint instrumental "ghost"
    #         (louder than artifacts, quieter than the voice)
    if add_noise:
        vocal_wav = vocal_wav + sigma * torch.randn_like(vocal_wav)

    sem = w2vbert.semantic_codes(vocal_wav)        # [25*T]  meaning, low-fi
    coarse = soundstream.coarse_codes(vocal_wav)   # [200*T] timbre, carries ghost

    # Fix 2: S-SA -> condition on SEMANTIC codes only.
    #  A 25 Hz k-means stream has no bandwidth to copy a hi-fi artifact.
    if drop_acoustic:
        return sem                                  # x_hat = Feats(x)
    return torch.cat([sem, coarse], dim=-1)        # default SA-SA input

# target instrumental codes ALWAYS keep acoustics -- you need them to synthesize
def target_codes(instr_wav, soundstream, w2vbert):
    return torch.cat([w2vbert.semantic_codes(instr_wav),
                      soundstream.coarse_codes(instr_wav)], dim=-1)

Chapter 7: The Data Pipeline — Following the Tensors

A model is only as good as the data factory feeding it. Let's trace one clip from raw audio to a training pair, watching every shape and decision.

From 1M songs to training pairs

That last row is the punchline: a 10-second instrumental becomes a 2250-token target sequence (max length for both input and target is 2250). The model is, mechanically, a Transformer doing translation — from a vocal token sequence to an instrumental token sequence.

Filtering: bias the model toward always playing

Raw separated data contains junk: clips where the instrumental is basically silent, or where the singer drowns out a whisper-quiet band. If you train on those faithfully, the model learns that "sometimes the right answer is silence" — useless for a user who wants a band every time. So SingSong filters out clips where:

• peak RMS of the instrumental is below −25 dB (essentially silent), or
• the vocals are ≥ 5 dB louder than the instrumental (band too faint to learn from).

Model & sampling config (the knobs that matter)

• Backbone: T5.1.1 (base = 250M params; XL = 3B). Encoder-decoder Transformer.
• Positional encoding: they swap T5's relative position embeddings for vanilla sinusoidal ones — cheaper for the long (2250) sequences and marginally better.
• Vocabulary: 5122 tokens. The 4 coarse acoustic dimensions each get a disjoint 1024-token slice (so each sequence position is tied to one codebook), plus 1024 semantic + 2 sentinel (start/pad). During sampling they mask out invalid sub-vocabularies per position.
• Decoding: temperature sampling at τ=0.85 — slightly below 1.0 to favor coherent, musical output over wild diversity.
• Early stopping: on FAD_i over a dev set (MUSDB18) — never on NLL (Chapter 5).

Chapter 8: Inference & the Mixing Trick

At inference a user gives clean vocals x₀. The generation path:

Why this works at all — the lockstep property

The whole pipeline is built so that x and y are time-aligned by construction. During training, the (vocal, instrumental) pair came from the same 10-second clip of the same song — they already lined up perfectly. The model therefore learns a distribution of instrumentals that share the input's clock. At inference, generated y₀ inherits that alignment, so adding it to x₀ just works.

The cost: sequential decoding

Inheriting AudioLM's autoregressive design means inheriting its bottleneck. The 2250 target codes are generated one at a time, and then Stage 3 generates fine codes sequentially too. This is why SingSong (like WaveNet) is not real-time on commodity hardware — generation is inherently serial. Later non-autoregressive music models (and diffusion-based ones) attack exactly this.

Chapter 9: Experiments & Results

How do you measure "good accompaniment"? You can't compute pixel error on audio. SingSong uses two tools: an automatic metric and a human study.

Fréchet Audio Distance (FAD)

FAD is the audio cousin of FID from image generation. Pass both real and generated audio through a fixed audio classifier (VGGish), fit a multivariate Gaussian to each set of embeddings, and compute the Fréchet distance between the two Gaussians:

Symbols: μ_r,Σ_r are the mean and covariance of reference (real) audio embeddings; μ_g,Σ_g the same for generated audio. Lower = closer to real. The first term compares average sound; the second compares the spread/diversity. They compute it on MUSDB18 against ground-truth mixes, feeding either isolated (FAD_i) or separated (FAD_s) vocals.

The story in one chart: dropping vocal acoustics (S–SA) and the Noisy condition each shrink the gap; together they nearly erase it; scaling to 3B (XL) lowers the absolute FAD further. Configs 3–6 use the Noisy condition.

The human study — the metric that actually counts

FAD is a proxy; the paper's real claim is a listening study. Listeners heard two vocal+instrumental mixes with the same vocals and picked the one they preferred:

Scaling

Take the winning Noisy / S–SA recipe and grow the T5 from base (250M) to XL (3B), everything else fixed. FAD_i drops from 1.36 to 1.28 and the gap stays small (0.19→0.32). Unsurprising but important: the recipe doesn't break when scaled, and bigger is better.

Chapter 10: Connections & Cheat Sheet

SingSong sits at a crossroads of three ideas you may already know — and it points forward to the wave of music-generation systems that followed.

Where it comes from

• VQ-VAE / neural codecs: the "model codes, not waveforms" idea. Neural Audio Codecs, VAE & VQ-VAE.
• AudioLM: the semantic+acoustic cascade SingSong directly adapts.
• T5 / Transformers: the encoder-decoder seq2seq backbone. Transformers, Attention Is All You Need.
• Self-supervised speech: w2v-BERT semantic codes. Self-Supervised Speech.
• Source separation: MDXNet as the data factory. ICA, NMF.

Where it goes

• MusicLM (contemporaneous): same AudioLM lineage, but conditioned on text and melody hints rather than locking to a full vocal.
• Diffusion / non-autoregressive music: attacks SingSong's serial-decoding bottleneck. See Live Music Diffusion, Music Generation.
• WaveNet: the autoregressive-audio ancestor. WaveNet.

Cheat sheet — every symbol in one place