Generate sound one sample at a time — 16,000 predictions per second — using a stack of dilated causal convolutions that reach back thousands of timesteps without a single recurrent connection.
Hold a microphone up to a voice. Sample the air pressure 16,000 times every second. That stream of numbers is the sound. One second of CD-quality speech is 16,000 to 44,100 numbers in a row — and to generate speech, you have to produce every single one of them, in order, and have them line up into something a human ear accepts as a voice.
In 2016, no one generated audio this way. Text-to-speech systems either concatenated pre-recorded snippets of human speech (unit selection) or ran a hand-built vocoder — a signal-processing pipeline that turned predicted spectral features back into a waveform. Both sounded robotic. The waveform itself was never modeled directly; it was assembled or synthesized by formula.
Pixels in an image have a comfortable resolution: a 256×256 image is ~65,000 values, and a pixel two rows up is "close." Audio is different in one terrifying way: temporal resolution. A meaningful chunk of speech — a single phoneme like "ah" — spans roughly 50 milliseconds. At 16 kHz, that is 800 samples. A word spans tens of thousands of samples.
So if you want a model that captures even a single phoneme of context, it must look back hundreds of samples. To capture word-level structure, thousands. This is the central tension of the whole paper: audio needs a huge receptive field — the number of past samples the model can "see" when predicting the next one.
Let's make the numbers concrete. Suppose we want the model to remember 1 second of audio when predicting the next sample:
| Quantity | Value | Why it hurts |
|---|---|---|
| Sample rate | 16,000 Hz | 16,000 predictions per second of output |
| 1 phoneme of context | ~800 samples | Minimum to capture a single speech sound |
| 1 second of context | 16,000 samples | Needed for prosody, word boundaries |
| Naive conv (filter 2) layers for 16k reach | 15,999 layers | Receptive field grows by +1 per layer — hopeless |
That last row is the whole problem in one line. A plain stack of small convolutions grows its receptive field linearly with depth: each layer adds one sample of reach. To see 16,000 samples back you would need ~16,000 layers. That network would never train. WaveNet's job is to get that same reach with ~30 layers.
An LSTM can, in principle, carry information across thousands of timesteps through its hidden state. So why not use one? Two reasons the paper cares about:
So the bet of the paper is: replace recurrence with a clever convolutional architecture that achieves enormous receptive fields cheaply. Everything that follows — causal convolutions, dilation, gating, residuals — is in service of that one bet.
WaveNet factorizes the joint probability of a waveform x = {x1, ..., xT} into a product of conditionals, exactly the way a language model factorizes a sentence into next-token probabilities:
Read this carefully. Every audio sample xt is predicted from all the samples before it. The model is autoregressive: it outputs a probability distribution over the next sample, you draw one, append it, and repeat. Exactly like GPT predicting the next word — except the "words" are 16,000 sound samples per second.
Each word in "dilated causal convolution" solves one specific failure of the naive approach:
A WaveNet is: input waveform → a causal convolution → a tall stack of residual blocks, each containing a dilated convolution with a gated activation → outputs from every block summed via skip connections → two 1×1 convolutions with ReLUs → a 256-way softmax over the next sample's value. We will build every one of those pieces from scratch in the chapters ahead, and you will see exactly what tensor flows where.
Before the architecture, we need to be precise about the data. WaveNet operates on PCM (pulse-code modulation) audio: the raw output of an analog-to-digital converter. Two numbers define it.
Let's trace the shapes, because "what tensor goes in, what comes out" is the whole game:
| Stage | Tensor | Meaning |
|---|---|---|
| Raw clip (1 sec, 16 kHz) | [16000] | 16-bit integers, the ground-truth waveform |
| After µ-law quantize (Ch 5) | [16000] | integers in [0, 255] — one of 256 categories |
| One-hot / embedded input | [16000, 256] or [16000, C] | fed into the causal conv stack |
| Network output per timestep | [16000, 256] | softmax distribution over the next sample's category |
Your first instinct is to predict a real number (the next sample value) and minimize squared error. WaveNet deliberately does not. Instead it outputs a categorical distribution — a softmax over discrete sample values — and is trained with cross-entropy, maximizing the log-likelihood of the true next sample.
But there's a catch: a softmax over 16-bit audio needs 65,536 output units per timestep. That's an enormous, mostly-empty output layer. Chapter 5 is entirely about a transformation that crushes those 65,536 values down to a tractable 256 — without the ear noticing.
This asymmetry is easy to miss and crucial to understand:
The first ingredient. A causal convolution is an ordinary 1-D convolution with one rule enforced: the output at time t may only depend on inputs at times t, t−1, t−2, ... — never t+1 or later. The model is forbidden from peeking at the future.
This is not optional decoration; it's what makes the autoregressive factorization from Chapter 1 true. If output t could see input t+1, then p(xt|x<t) would secretly depend on a future sample, and the product-of-conditionals would be a lie.
For 1-D data the trick is delightfully simple: take a normal convolution and shift the output. Pad the input on the left with (filter_size − 1) zeros and use no padding on the right. Then output position t mechanically only overlaps inputs up to t. (For images, the 2-D equivalent is a masked convolution — you zero out the filter weights that would touch future pixels.)
Click "Add Layer" and watch the highlighted output's receptive field (the input samples it can see) grow. Notice it grows by exactly +1 sample per layer. This is the painfully slow linear growth WaveNet must escape.
For a plain stack of causal convolutions, the receptive field is:
Walk through it. With filter size 2 and one layer, output t sees inputs {t−1, t} → R = (2−1)×1 + 1 = 2. Add a second layer: each of those two becomes a window of two, but they overlap, netting {t−2, t−1, t} → R = (2−1)×2 + 1 = 3. The paper's Figure 2 has 4 layers of filter-2 convs, giving R = 1×4 + 1 = 5. As foreshadowed: to reach 16,000 you'd need ~16,000 layers. Linear growth is the enemy.
A dilated convolution (a.k.a. à trous, "with holes") is a convolution whose filter is applied over a region larger than its length by skipping input values with a fixed step, the dilation factor d. With d=1 it's a normal convolution. With d=2, a filter of size 2 looks at inputs {t, t−2}, skipping t−1. With d=4, at {t, t−4}.
The genius: stack layers with exponentially increasing dilation — 1, 2, 4, 8, ..., 512 — and the receptive field doubles every layer while every input sample is still visited (no gaps in coverage). You get exponential reach at the same per-layer cost as a normal convolution.
Drag Layers to add dilated layers (dilation doubles: 1, 2, 4, 8, ...). Watch the highlighted output's receptive field — the blue lines — fan out across the input. Hit Animate to push a sample through the whole stack and see causality + exponential reach at once.
For one block of dilations 1, 2, 4, ..., 2L−1 with filter size 2, the receptive field is:
Derivation: each layer of dilation d and filter size 2 adds d samples of reach. Summing the dilations of one block:
So a single block of 10 layers (dilations 1→512) gives a receptive field of 210 = 1024 samples. The paper notes this block "can be seen as a more efficient and discriminative counterpart of a 1×1024 convolution." Compare to the plain causal stack of Chapter 3: to reach 1024 you needed 1023 layers; here, 10.
WaveNet repeats the dilation pattern: 1,2,...,512, 1,2,...,512, 1,2,...,512. Why reset to 1 instead of continuing to 1024, 2048? Two reasons: (1) stacking blocks keeps the receptive field growing while also increasing model capacity/non-linearity; (2) resetting to small dilations re-densifies the local coverage. Three blocks of 10 layers each gives roughly 3 × 1024 = ~3072 samples of receptive field — about 190 ms at 16 kHz, enough for the 2–3 phonemes the paper reports.
| Architecture | Layers for R = 1024 | Growth |
|---|---|---|
| Plain causal (filter 2) | ~1023 | linear (+1/layer) |
| Dilated causal (filter 2) | 10 | exponential (×2/layer) |
# A causal dilated 1-D convolution, from scratch import torch, torch.nn.functional as F def causal_dilated_conv1d(x, weight, dilation): """x: [B, C_in, T] weight: [C_out, C_in, K]""" K = weight.shape[-1] # left-pad so output[t] only sees inputs <= t (causality) pad = (K - 1) * dilation x = F.pad(x, (pad, 0)) # pad LEFT only return F.conv1d(x, weight, dilation=dilation) # receptive field of a stack of these def receptive_field(dilations, K=2): return 1 + sum((K - 1) * d for d in dilations) block = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] print(receptive_field(block)) # 1024 print(receptive_field(block * 3)) # 3070 (~190 ms @ 16kHz)
Back to the problem we flagged in Chapter 2: 16-bit audio has 65,536 possible values per sample. A softmax over 65,536 categories is huge and wasteful. We want to collapse it to 256 categories. But naive rounding — just chop the 16 bits to 8 — sounds awful. Why?
Because loudness perception is logarithmic. Your ear is exquisitely sensitive to small changes among quiet sounds and relatively insensitive to small changes among loud sounds. Linear quantization wastes precious levels on loud samples (where you can't hear the difference) and starves quiet samples (where you can). The result is audible quantization noise in the quiet parts — exactly where speech detail lives.
For an input sample xt normalized to −1 < xt < 1, with μ = 255:
Symbol by symbol, with intuition:
After applying f, you uniformly bin the warped value into 256 levels. The paper reports this "produces a significantly better reconstruction than a simple linear quantization scheme" — and for speech, the reconstructed signal "sounded very similar to the original."
Take a quiet sample, x = 0.01:
A linear scheme would map x=0.01 to bin 0.01 (basically silence, indistinguishable from x=0). µ-law lifts it to 0.2285 — pushed well away from zero so it lands in its own distinct quantization bin. Now a loud sample, x = 0.9:
The loud sample barely moves (0.9 → 0.981), so loud sounds get fewer levels — but the ear doesn't care. The quiet sample got 23× the relative resolution. That redistribution is the entire point.
The diagonal gray line is linear quantization (equal-width bins). The orange curve is µ-law. The horizontal ticks show where the 256 levels land on the input axis after the warp: notice how they cluster near zero (quiet) when µ is large. Toggle off to compare against linear.
import numpy as np def mu_law_encode(x, mu=255): """x in [-1, 1] -> integer category in [0, 255]""" f = np.sign(x) * np.log1p(mu * np.abs(x)) / np.log1p(mu) return ((f + 1) / 2 * mu + 0.5).astype(int) # to [0,255] def mu_law_decode(y, mu=255): """inverse: category -> waveform value""" f = 2 * (y / mu) - 1 return np.sign(f) * (1 / mu) * ((1 + mu) ** np.abs(f) - 1) print(mu_law_encode(np.array([0.01, 0.9]))) # [157 252] -> distinct bins
Inside each layer, after the dilated convolution, WaveNet does not use a ReLU. It uses a gated activation unit, borrowed from the gated PixelCNN:
Decode it:
One dilated layer takes the residual stream — say [B, C, T] with C = residual channels (the paper uses on the order of dozens) — and runs it through a single dilated conv that outputs 2C channels. That 2C is then split in half: the first C go to tanh (filter), the second C go to sigmoid (gate). Their elementwise product is [B, C, T] again — same shape in, same shape out, which is exactly what residual connections need.
Drag the gate's pre-activation. The blue curve is the tanh content; the orange is the sigmoid gate (0→1); the green is their product — what actually flows downstream. Push the gate negative and watch the green output get squeezed to zero regardless of the content.
def gated_unit(x, conv_2c): """conv_2c: dilated conv with 2C output channels""" h = conv_2c(x) # [B, 2C, T] f, g = h.chunk(2, dim=1) # each [B, C, T] return torch.tanh(f) * torch.sigmoid(g) # [B, C, T]
We now assemble the pieces into the unit that repeats throughout WaveNet: the residual block. A network 30+ layers deep would never train without two ideas from ResNets and Highway Nets: residual connections and skip connections.
The data flow, step by step:
All the skip contributions are summed, then passed through: ReLU → 1×1 conv → ReLU → 1×1 conv → softmax. That final softmax produces the 256-way categorical distribution over the next sample. Trace the shape: skip sum [B, Cskip, T] → ... → [B, 256, T]. During training, cross-entropy is computed at every one of the T positions in parallel.
class ResidualBlock(nn.Module): def __init__(self, C, C_skip, dilation): self.conv = CausalDilatedConv1d(C, 2*C, kernel=2, dilation=dilation) self.res_1x1 = nn.Conv1d(C, C, 1) # back to residual stream self.skip_1x1 = nn.Conv1d(C, C_skip, 1) # tap to output def forward(self, x): h = self.conv(x) # [B, 2C, T] f, g = h.chunk(2, dim=1) z = torch.tanh(f) * torch.sigmoid(g) # gated [B, C, T] skip = self.skip_1x1(z) # [B, C_skip, T] res = self.res_1x1(z) + x # residual add [B, C, T] return res, skip
So far WaveNet generates unconditional audio — it babbles in a realistic but meaningless way. To make it useful (say a particular speaker, or actual words), we give it an extra input h and model p(x|h):
There are two flavors, distinguished by whether the conditioning signal is constant over time or itself a time series.
A single latent h that influences every timestep — e.g. a speaker-identity embedding. It's injected into the gated activation as a learned linear projection added before the nonlinearities:
Here V∗,kh is a vector that is broadcast over time — the same bias added at every timestep. The paper conditions on a one-hot speaker ID this way, and a single WaveNet then captures all 109 VCTK speakers, switchable by flipping the one-hot.
A second time series ht, often at a lower sample rate than the audio — e.g. linguistic features (which phoneme is being spoken) for TTS, sampled every few milliseconds, not every audio sample. The challenge: align this slow signal to the 16 kHz audio. WaveNet uses a transposed convolution (learned upsampling) to stretch h up to audio resolution, giving y = f(h), then injects it via 1×1 convs:
The paper notes the alternative — just repeating the low-rate values across time instead of learned upsampling — "worked slightly worse." Learned upsampling lets the model smoothly interpolate between, say, phoneme boundaries.
The orange dots are the low-rate conditioning signal (e.g. phoneme features). The teal line is the upsampled version fed into every layer. Toggle "Learned" off to see the crude "repeat across time" alternative — note the staircase the paper found worked worse.
WaveNet was evaluated on four tasks. The headline: in text-to-speech, human listeners rated it more natural than the best existing systems, closing roughly half the gap to real human speech.
Mean Opinion Score: listeners rate naturalness 1 (Bad) to 5 (Excellent). Higher is better.
| System | US English | Mandarin |
|---|---|---|
| LSTM-RNN parametric | 3.67 | 3.79 |
| HMM concatenative | 3.86 | 3.47 |
| WaveNet (L+F) | 4.21 | 4.08 |
| Natural speech (16-bit PCM) | 4.55 | 4.21 |
Read the gap-closing: in US English the best baseline was 3.86 vs natural 4.55 — a gap of 0.69. WaveNet hit 4.21, shrinking the gap to 0.34, a 51% reduction. In Mandarin the gap dropped from 0.42 to 0.13, a 69% reduction. At the time these were "the highest ever reported MOS values" on these datasets.
| Task | Setup | Result |
|---|---|---|
| Multi-speaker generation | VCTK, 109 speakers, conditioned on speaker ID only (no text) | One model captures all 109 voices; babbles realistic non-words; even mimics breathing & room acoustics |
| Text-to-speech | Local conditioning on linguistic features (+ log-F0) | State of the art naturalness, both languages (table above) |
| Music | MagnaTagATune (~200h), YouTube piano (~60h) | Harmonic, "aesthetically pleasing" fragments; larger receptive field = more musical |
| Speech recognition | TIMIT, added mean-pooling + non-causal convs, dual loss | 18.8 PER — best reported (at the time) directly on raw audio |
WaveNet sits at a pivotal junction in generative modeling: it took the autoregressive, pixel-by-pixel idea from PixelRNN/PixelCNN and proved it could conquer the most demanding sequential domain — raw audio at 16,000 samples per second.
| Idea | Equation / Rule | What each symbol means |
|---|---|---|
| Autoregressive factorization | p(x) = ∏ p(xt|x<t) | xt = sample at time t; predicted from all earlier samples |
| Plain causal receptive field | R = (K−1)·L + 1 | K = filter size, L = layers; linear growth |
| Dilated receptive field (K=2) | R = 2L per block | dilations 1,2,4,...,2L−1; exponential growth |
| µ-law companding | f(x) = sign(x)·ln(1+μ|x|)/ln(1+μ) | μ=255; warps amplitude so 256 levels match the ear |
| Gated activation | z = tanh(Wf∗x) ⊙ σ(Wg∗x) | tanh = content, σ = gate (how much passes) |
| Global conditioning | + VfTh inside the gate | h = constant latent (speaker), broadcast over time |
| Local conditioning | + Vf∗y, y = upsample(h) | h = time series (linguistics), learned-upsampled to audio rate |
WaveNet models audio autoregressively: each of 16,000 samples per second is a 256-way softmax conditioned on all previous samples. The conditional is computed by a stack of dilated causal convolutions — causal so it never sees the future, dilated (1,2,4,...,512) so the receptive field grows exponentially and reaches thousands of samples with ~30 layers instead of thousands. Inputs are µ-law companded to 256 categories so the softmax is tractable and the quantization matches human hearing. Each layer uses a tanh⊙sigmoid gated activation; residual connections keep the deep stack trainable and skip connections feed every timescale to the output. Conditioning (speaker ID globally, linguistics locally) turns the babbler into a controllable TTS system that beat every prior method on naturalness.