Teaching a neural network to generate musical instrument sounds one audio sample at a time — and discovering a hidden "timbre space" where you can morph a flute into an organ.
You want a computer to play a sound you have never heard before — not a recorded sample, not a copy of a violin, but a new instrument that sounds like the lovechild of a flute and an organ. How do you even represent "the sound of an instrument" so a machine can generate it?
For half a century, the answer was a synthesizer: a human engineer hand-designs oscillators, filters, and envelopes, then exposes knobs labeled "pitch," "cutoff," "resonance." It works, but every new timbre requires a human to invent the recipe. There is no "learn what a trumpet is and give me something nearby."
Images had MNIST, CIFAR, ImageNet — big clean datasets that powered the GAN/VAE revolution. Audio had almost nothing comparable. And audio has a property that makes it nastier than images:
Let's count. One NSynth note:
| Quantity | Value | Why it matters |
|---|---|---|
| Sample rate | 16,000 Hz | 16k numbers describe just one second |
| Note length | 4 seconds | → 64,000 raw samples per note |
| Fundamental pitch range | ~27 Hz to ~4186 Hz | The "wiggle" repeats every ~4 to ~590 samples |
| Loudness envelope | ~1 event over 4 s | A swell spanning all 64,000 samples |
So a model must simultaneously reason about structure at the scale of 4 samples (a high note's period) and 64,000 samples (the whole envelope). That is a dynamic range of 16,000× in timescale. This is the heart of the difficulty.
The paper makes two contributions to crack this: a new WaveNet autoencoder architecture, and NSynth, a dataset of ~306,000 musical notes — an order of magnitude bigger than anything public at the time.
WaveNet (van den Oord et al., 2016) had already shown you could generate raw audio one sample at a time with an autoregressive model — predict sample i from all the samples before it. It sounded astonishing for speech. But left to itself, an unconditioned WaveNet just "babbles": it produces locally plausible audio with no long-term structure, like a person sleep-talking convincing-sounding nonsense.
To get coherent speech, the original WaveNet had to be told what to say — conditioned on externally-provided linguistic features (phonemes, durations) computed by a separate TTS front-end. That external conditioning is the crutch.
The original WaveNet factorizes the joint probability of a waveform x = (x1, ..., xN) as a chain of next-sample predictions:
Read this as: "the probability of the whole sound is the product of, at each step, predicting the next sample given everything heard so far." The autoencoder adds one extra thing to condition on — a learned code f(x):
Here f(x) = Z is the encoder's output: a sequence of hidden codes distributed in time. Every symbol: xi = the i-th audio sample; x1...i-1 = all past samples (the autoregressive context); f(x) = a compressed summary of the entire note that gives the decoder global guidance the local context alone cannot.
Before the autoencoder, you must understand the decoder it wraps. WaveNet is an autoregressive model: to produce a 64,000-sample note it runs 64,000 forward passes, each predicting one sample from the past.
At step i, the network sees a window of prior samples and must output a distribution over the next sample's value. Concretely:
To predict sample i, the model may only use samples 1...i-1. If a convolution kernel reached even one sample into the future, the model would be cheating — it would see the answer. A causal convolution shifts its kernel so the output at time t depends only on inputs at times ≤ t. This is what makes the chain-rule factorization in Ch.1 honest.
This is gorgeous and also painfully slow at generation time. Training is parallel (you have the whole ground-truth waveform, so all 64,000 predictions can be made at once with teacher forcing). But generation is inherently serial: sample i must exist before you can compute sample i+1. 64,000 sequential network evaluations per note. This serial bottleneck is the price of the principled likelihood.
Here is the tension. To capture a low note whose period is hundreds of samples, the model needs a receptive field of hundreds of samples. A normal convolution with a small kernel sees only a handful of neighbors. Stacking many normal layers grows the field linearly — you'd need hundreds of layers. Too expensive.
Dilated convolutions solve this. A dilation of d means the kernel skips d-1 samples between each tap. Stack layers with dilations 1, 2, 4, 8, 16, ... and the receptive field grows exponentially: with kernel size 2, after L layers the field is 2L samples.
Each new layer doubles the dilation. The highlighted output neuron's receptive field (the input samples it can "see") grows exponentially. Causality: it never reaches to the right of itself.
With kernel size k = 2 and dilations doubling each layer, the receptive field R after L layers is:
Plug in numbers. With 10 layers (dilations 1,2,...,512): R = 211 - 1 = 2047 samples. That's ~128 ms at 16 kHz — from just 10 layers. WaveNet stacks several such blocks. The paper's encoder alone is a 30-layer residual stack of dilated convs. The exponential trick is the only reason raw-audio modeling is tractable.
Now we assemble the full architecture from Figure 1b. Trace the data flow end to end — this is the heart of the paper.
x ∈ R64000 (one note).Input: 64,000 samples. Embedding Z at stride 512: 125 × 16 = 2000 numbers. That's a 32× compression. The paper explores the trade-off by varying the stride while holding total embedding size roughly constant: smaller stride → more timesteps but fewer channels each (finer temporal resolution, less per-step expressivity); larger stride → fewer, fatter timesteps. They found stride 512 (32 ms, 16 dims/step) the sweet spot.
The decoder predicts a softmax over 256 values. Why 256, and how do you turn a continuous waveform (a 16-bit float between -1 and 1) into 256 categories without it sounding terrible? The answer is mu-law companding, a trick borrowed from telephony.
If you split [-1, 1] into 256 equal bins (linear quantization), the step size is fixed everywhere. But human hearing is logarithmic — we perceive ratios of loudness, not absolute differences. Quiet sounds, which carry tons of perceptual detail, get crushed into a few coarse bins. Loud peaks get more resolution than they need.
Mu-law applies a logarithmic squashing before quantizing, so quiet signals get fine resolution and loud signals get coarse resolution — matching the ear:
Every symbol: x = the raw sample in [-1, 1]; μ = 255 (gives 256 levels after quantizing); sign(x) preserves the waveform's polarity; the ln compresses the dynamic range. After applying f, you round to one of 256 levels. To play the audio back, you apply the inverse expansion.
With audio quantized to 256 categories, the loss is plain cross-entropy (categorical), summed over every sample:
This is the negative log-likelihood of the chain rule from Ch.1. No phase to estimate, no MSE Gaussian assumption — just "how surprised was the model by the true next bin?" The downside the paper flags honestly: the mu-law quantization itself injects a slight built-in distortion, most audible at low frequencies. Removing this distortion (e.g. with finer or learned quantization) is named as ongoing work.
You cannot study a learned audio manifold without data that spans it cleanly. The authors built NSynth: ~306,043 musical notes, an order of magnitude larger than any comparable public dataset at the time. Its design is deliberate — every axis of variation is controlled so the embedding space has something structured to learn.
Average of 65.4 pitches per instrument × ~5 velocities × 1006 instruments ≈ 300k notes. The controlled grid is the point: by fixing instrument, sweeping pitch and velocity, you can later ask "does the embedding separate which instrument from which pitch?"
| Annotation | Values | What it captures |
|---|---|---|
| Source | acoustic / electronic / synthetic | How the sound was produced |
| Family | brass, keyboard, string, ... (exactly one) | High-level instrument type |
| Qualities | bright, percussive, ... (zero or more) | Sonic character tags |
NSynth shipped as TFRecord files (TensorFlow Example protos), split into train and holdout sets — making it a reusable benchmark, the "ImageNet of musical notes" the field was missing.
A new method means nothing without a strong baseline to beat. The authors built a careful convolutional autoencoder operating on spectrograms — and the story of why it struggles teaches you something deep about audio.
A 10-layer conv encoder and 10-layer conv decoder, 2×2 strides, 4×4 kernels, leaky-ReLU + batch norm, channels growing 128 → 1024, then a fully-connected bottleneck to a single 19,841-dim hidden vector (matched in size to the WaveNet autoencoder's embedding). Classic image-autoencoder design, pointed at audio.
What does the conv net actually see? The authors tried, in order of increasing cleverness:
So they didn't predict phase at all. They predicted only the magnitude spectrum, then used the Griffin-Lim algorithm (1984) to reconstruct a plausible phase iteratively. The recipe: large FFT (1024) relative to hop size (256), run Griffin-Lim for 1000 iterations. As a final heuristic, the MSE loss was frequency-weighted — weight 10 at 0 Hz decreasing linearly to 1 at 4000 Hz and above — to enforce phase coherence on the fundamentals, where frequency errors hurt most.
How do you show that one model sounds better than another in a paper, on paper? The authors invented a visualization called the Rainbowgram — and then proved the point with a quantitative classifier.
It's a constant-Q transform (CQT) spectrogram where:
Why color-code the phase derivative? Because a harmonic that holds a steady frequency produces a constant phase shift each timestep, which shows up as a solid continuous colored line. Phase noise — exactly what Griffin-Lim leaves behind — shows up as speckled, faded rainbow texture. The eye can finally see what the ear hears.
Qualitative plots invite skepticism. So the authors trained a multi-task classifier to predict pitch and quality labels, then ran it on reconstructions from each model. The logic: if a reconstruction preserves the note's identity, a classifier should read it correctly — an audio analog of the Inception Score.
| Audio source | Pitch accuracy | Quality accuracy |
|---|---|---|
| Original audio | 91.6% | 90.1% |
| WaveNet reconstruction | 79.6% | 88.9% |
| Spectral baseline reconstruction | 46.9% | 85.2% |
Read it: the classifier extracts pitch from WaveNet reconstructions ~70% more successfully than from the baseline (79.6 vs 46.9), and quality accuracy is nearly equal to the original audio. The qualitative "WaveNet sounds crisper, baseline buzzes" claim now has a number behind it.
This is the payoff that made the paper famous. Because Z is a learned manifold, points between two real notes are themselves valid sounds. Linearly interpolate the embeddings of a bass and a flute, decode the result — and you get an instrument that has never existed but sounds real.
Top: spectrogram of the morph. Toggle the two modes and compare. Z-space interpolation fuses harmonics into a single coherent new sound; raw-audio interpolation just superimposes two notes playing at once.
The interpolation is dead simple math:
But where you interpolate changes everything:
The authors hoped to hold timbre fixed and freely change pitch by conditioning on a one-hot pitch input. Their initial attempts failed — the model learned to ignore the pitch conditioning. Investigating with a linear pitch classifier trained on the embeddings, they found that when you do add pitch conditioning during training, classification accuracy from Z drops by 13–30% (and up to a 75% relative drop at small embedding sizes). That drop is the evidence of disentangling: the more the conditioning works, the less pitch information leaks into Z. With a small (128-dim) baseline embedding they got it to balance — playing two octaves of a C-major chord from a single timbre embedding.
This paper sits at a pivotal moment in generative audio — bridging the autoregressive waveform era and the latent-space generative era that followed.
Because Z is a local driving function, the model — trained only on isolated 4-second notes — can reconstruct things it never saw: a whole sequence of notes, or a note held longer than 3 seconds. It even glissandos smoothly between pitches (it never saw note transitions, so it slides between them). The honest limitation the authors flag: due to memory constraints, the model can't capture truly global context — Z's receptive field is bounded by the training chunk size.
| Symbol / term | Meaning |
|---|---|
| p(x) = ∏ p(xi | x<i, f(x)) | Autoregressive likelihood, conditioned on learned code Z = f(x) |
| Z = f(x) | Temporal embedding; shape [125, 16] at stride 512 (32× compression) |
| Dilated causal conv | Kernel skips d-1 taps; receptive field grows exponentially; never sees the future |
| R = 2L+1 - 1 | Receptive field after L layers (kernel 2, doubling dilation) |
| mu-law, μ=255 | Log companding → 256 categories; matches logarithmic hearing |
| Softmax over 256 | Categorical (multimodal) output; replaces MSE/Gaussian |
| Zα = (1-α)ZA + αZB | Linear interpolation → timbre fusion (not superposition) |
| Griffin-Lim | Baseline's phase reconstruction (1000 iters); the source of its buzz |
| Rainbowgram | CQT spectrogram: brightness = magnitude, color = instantaneous frequency |
import torch, torch.nn.functional as F def mu_law_encode(x, mu=255): # x in [-1, 1] -> integer category in [0, 255] comp = torch.sign(x) * torch.log1p(mu * x.abs()) / torch.log1p(torch.tensor(float(mu))) return ((comp + 1) / 2 * mu + 0.5).long() # [N] ints in [0,255] def wavenet_ae_loss(encoder, decoder, x): # x: [B, N] raw waveform in [-1, 1], N = 64000 Z = encoder(x) # [B, 125, 16] temporal code (32x compression) Z_up = F.interpolate(Z.transpose(1,2), size=x.shape[-1], mode='nearest').transpose(1,2) # [B, N, 16] upsampled x_in = F.pad(x, (1, 0))[:, :-1] # causal shift: predict x[i] from x[<i] logits = decoder(x_in, Z_up) # [B, N, 256] softmax over mu-law bins, Z biases each layer target = mu_law_encode(x) # [B, N] categories return F.cross_entropy(logits.reshape(-1, 256), target.reshape(-1)) # Timbre morphing at inference: average two notes' codes, decode the result def morph(decoder, z_a, z_b, alpha=0.5): z = (1 - alpha) * z_a + alpha * z_b # fusion, not superposition return decoder.generate(z) # autoregressive, 64000 serial steps