Put a violin's worth of oscillators and filters inside the neural network — and let gradients flow right through them. High-fidelity audio from 13 minutes of data and a model 10× smaller than the competition.
Here's a deceptively hard problem: get a neural network to generate the sound of a violin. Not classify it, not transcribe it — actually produce the waveform, sample by sample, so it sounds real. By 2020 there were three families of approaches, and the DDSP paper opens by showing that all three fight the physics of sound instead of using it.
To feel why, you need one fact about sound: it oscillates. A violin string vibrates back and forth, periodically, hundreds of times a second. Your ear evolved to be exquisitely sensitive to that phase-coherent oscillation. Any model of audio that ignores this periodic structure is working uphill. Let's see the three uphill battles.
Family one: waveform models. Networks like WaveGAN generate the raw waveform directly, in overlapping frames stamped out by strided convolutions. The trouble: audio oscillates at many frequencies, each with a different period, none of which lines up with the fixed frame hop. So the model must precisely align the wave from one frame to the next, and learn filters covering every possible phase. A brutal bookkeeping job, just to keep the wave from tearing at the seams.
Family two: Fourier models. Networks like GANSynth or Tacotron generate the audio's frequency content instead. But the Short-Time Fourier Transform is itself a representation over windowed wave-packets, so it inherits the same phase-alignment headache — plus spectral leakage: when a true tone's frequency falls between the Fourier bins, a single pure sinusoid smears across many neighboring bins, and the model must learn to reassemble it. More bookkeeping.
Family three: autoregressive models. WaveNet and friends sidestep both by generating one sample at a time — arbitrary waveforms, no frame seams. But they're huge and data-hungry (they get no help from the bias toward oscillation), they suffer exposure bias from teacher-forcing, and — the killer — they're stuck with sample-by-sample losses. Here's the punchline: three waveforms that are identical to the ear (just phase-shifted harmonics) look completely different sample-by-sample, so an autoregressive model is punished for differences nobody can hear.
Switch between the three families and see the specific pain each one fights. Drag the slider to misalign the frames / shift the bin / phase-shift the harmonics — and watch the "cost" the model pays for a sound that hasn't actually changed.
There has always been a fourth family, older than all the neural ones: synthesizers and vocoders. These don't predict waveforms at all. They generate sound the way physics does — with oscillators and filters, controlled by interpretable knobs like pitch and loudness. A synthesizer already knows sound oscillates; that knowledge is built into its bones.
So why hadn't everyone just bolted a neural network onto a synthesizer? Because of one wall: you couldn't train through it. A traditional synthesizer is a fixed lump of signal-processing code. Gradients — the signal a network needs to learn — can't flow backward through it. So the synth's knobs had to be set by hand or by gradient-free search, and small knob errors caused large audio errors that the network never saw. The synthesizer was a black box the learning couldn't reach.
DDSP's one move, the whole paper in a sentence: re-implement the classic DSP building blocks — oscillators, filters, reverb — as differentiable functions, so they live inside the autograd graph and gradients flow straight through them. Now a neural network can output synthesizer controls, the synthesizer renders audio, a loss compares it to the target, and the error flows all the way back — through the oscillators — to update the network. The synth stops being a black box and becomes just another differentiable layer.
The payoff is enormous and counterintuitive. Because the network no longer has to learn that sound oscillates — it gets that for free from the oscillator — it can be small, train on minutes of data, and stay interpretable. The paper achieves high-fidelity synthesis with no autoregression, no adversarial loss, a model up to 10× smaller than competitors, on 13 minutes of violin. The inductive bias of physics does the heavy lifting; the network just steers.
loss.backward() reach the network. The next four chapters build exactly these components: the harmonic oscillator (Ch. 3), the filter and filtered-noise (Ch. 4), the autoencoder that drives them (Ch. 5), and the loss that judges them (Ch. 6).
Before we can build DDSP's synthesizer we need its atom: the sinusoidal oscillator. A pure sine wave is the simplest possible sound — a single frequency, like a tuning fork. Two numbers describe it at any instant: its amplitude (how loud, the height of the wave) and its frequency (how high the pitch, how many cycles per second).
One subtlety that matters for differentiability: the oscillator's phase — where in its cycle it is right now — is the running accumulation of frequency over time. If the frequency is changing (a note bending up), you can't just multiply; you have to add up the frequency at every step from the start. Think of phase as an odometer and frequency as the speedometer: position is the integral of speed. The paper writes phase as exactly that running sum of instantaneous frequency. This is what lets a single oscillator smoothly glide in pitch — essential for expressive instruments.
A single sine, though, sounds thin and electronic. A real violin note is rich because it's many sines at once: the fundamental frequency (the perceived pitch) plus a stack of harmonics — sines at integer multiples of the fundamental: 2×, 3×, 4×, and so on. The fundamental sets the pitch; the relative loudness of the harmonics sets the timbre — what makes a violin sound different from a flute playing the same note.
This is the genius constraint behind a harmonic oscillator: instead of letting every sine roam free, lock them to integer multiples of one fundamental. Now the entire rich tone is parameterized by just the fundamental frequency plus the amplitude of each harmonic. That's a small, structured, musical set of knobs — perfect for a network to control.
Set the fundamental pitch and the loudness of each harmonic. Watch the individual sines (faint) sum into the rich waveform (bold) on the left, and see the harmonic spectrum on the right. A few strong low harmonics → warm; lots of high ones → bright and buzzy.
Now we assemble the harmonic oscillator into DDSP's main sound source: the additive synthesizer. "Additive" because it literally adds up many sinusoids. The output at each instant is the sum, over all harmonics, of each harmonic's amplitude times its sine. Each harmonic's frequency is an integer multiple of the fundamental, and the phase of each is that running accumulation of frequency we met in Chapter 2.
Read it gently. x(n) is the audio sample at time-step n. The sum runs over K harmonics. Ak(n) is the time-varying amplitude of harmonic k — how loud that particular overtone is right now. φk(n) is harmonic k's phase, accumulated from its frequency (which is k times the fundamental). Add them up and you have a rich, evolving tone.
But the paper makes one more move that's pure interpretability gold. Controlling K independent harmonic amplitudes mixes up two things a musician thinks of separately: how loud the note is, and what color it has. So they factorize each harmonic amplitude into two parts:
Here A(n) is a single global amplitude — the overall loudness knob — and c(n) is a normalized harmonic distribution: the amplitudes summing to one, like a probability distribution over harmonics. So c answers "what's the shape of the tone?" and A answers "how loud is it?" — cleanly separated. The network can push loudness up without changing color, or morph timbre without changing volume. (To guarantee both stay positive and bounded, the network's raw outputs pass through a modified sigmoid before becoming A and c.)
One practical wrinkle the paper is careful about: the oscillators need amplitude and frequency values at the full audio rate (tens of thousands per second), but the network only emits controls at a slower frame rate. So DDSP upsamples: frequencies via simple bilinear interpolation, and amplitudes via overlapping smoothed (Hamming) windows centered on each frame, to avoid clicks and artifacts at frame boundaries.
This is the engine. The loudness knob is the global amplitude A; the distribution knobs are the normalized harmonic shape c (they always re-normalize to sum to 1). Move loudness: the whole sound scales, color unchanged. Move the distribution: the timbre morphs, loudness unchanged. Watch the spectrum and the rendered waveform respond — this is exactly what the network learns to control.
Here's the scale that makes this work as a neural target. In the paper's setup, four seconds of 16 kHz audio is described by (1 amplitude + 100 harmonics + 65 noise bands) × 1000 timesteps ≈ 165,000 control numbers — about 2.5× more parameters than the 64,000 raw audio samples. That sounds backwards, but it's the point: these are interpretable, smooth controls a network can comfortably predict, far easier than predicting 64,000 raw samples directly.
# The additive synthesizer, as differentiable code (PyTorch-style) def harmonic_synth(f0, amplitude, harm_dist, n_harmonics, sr=16000): # f0: (batch, time) fundamental frequency per frame # amplitude: (batch, time) global loudness A(n) # harm_dist: (batch, time, K) normalized harmonic shape c(n), sums to 1 harmonics = torch.arange(1, n_harmonics + 1) # [1,2,3,...,K] freqs = f0[..., None] * harmonics # f_k = k * f0 freqs = freqs.masked_fill(freqs > sr / 2, 0.0) # anti-alias: drop above Nyquist amps = amplitude[..., None] * harm_dist # A_k = A * c_k (the factorization) amps = upsample(amps, sr) # frame-rate -> audio-rate (smoothed) freqs = upsample(freqs, sr) phase = torch.cumsum(2 * math.pi * freqs / sr, dim=1) # phase = running sum of frequency audio = (amps * torch.sin(phase)).sum(-1) # sum over harmonics -> x(n) return audio # fully differentiable wrt f0, amplitude, harm_dist
The additive synthesizer makes beautiful harmonic sound — the part that has clear pitch. But real instruments also produce stochastic sound: the breathy hiss of a flute, the scrape of a bow, the air in a voice. None of that is harmonic; it's filtered noise. The classic Harmonic-plus-Noise model captures a sound as exactly these two streams added together, and DDSP follows suit with a subtractive synthesizer for the noisy part.
"Subtractive" is the complement of "additive." Instead of building sound up from pure tones, you start with white noise — a hiss containing all frequencies equally — and carve away the ones you don't want with a filter. Sculpting noise into the right colored hiss is how you get realistic breath and bow texture. The whole trick is the filter, and DDSP needs it differentiable.
Here's how they build a differentiable, controllable filter — the frequency-sampling method. The idea is to describe the filter by what it does in the frequency domain: for each frame, the network outputs a transfer function — literally a curve saying "let this much of each frequency through." A value near 1 at some frequency passes it; near 0 blocks it. To apply the filter you chop the noise into frames, take each frame to the frequency domain (a DFT), multiply by the transfer-function curve (this is the filtering — pointwise scaling of frequencies), then transform back and overlap-add the frames into continuous audio.
Two careful details make it sound clean. First, the network's raw curve is multiplied by a window function (the paper defaults to a Hann window) before use, which controls the time-frequency resolution trade-off and prevents ugly ringing. Second, the impulse response is shifted to zero-phase (symmetric) form before windowing and back to causal form before filtering — this keeps the filter linear-phase, meaning it doesn't smear the timing of different frequencies. Phase coherence, again, is the obsession.
Left: the flat white-noise spectrum (all frequencies present). Drag the filter shape — this is the transfer function the network outputs. Right: the resulting filtered noise. Shape it low for a warm rumble, high for airy breath, narrow for a whistling resonance. The network learns these shapes to add realistic texture on top of the harmonic tone.
We have a differentiable synthesizer with interpretable knobs. Now: what sets those knobs, frame by frame, to reconstruct a real recording? The paper's answer is a deliberately simple autoencoder. An autoencoder squeezes input into a compact latent code (encode), then rebuilds the input from it (decode). DDSP's twist is that the "decoder" isn't a generic network — it's our synthesizer — and the latent code is factorized into meaningful pieces.
The encoder extracts three signals from the input audio, each on a clean perceptual axis:
Then the decoder — a generic network, fully-connected with one recurrent layer — maps the tuple (f, l, z) to the synthesizer's controls: the global amplitude, the harmonic distribution, and the noise filter's transfer function. The synthesizer renders audio; the loss compares it to the original; gradients flow back through the synth into the decoder. The architecture is intentionally plain to prove a point: it's the DSP components, not fancy modeling, that deliver the quality.
One design choice is quietly crucial. The pitch f(t) is fed directly into the additive synthesizer, not just into the decoder network. Why? Because frequency has structural meaning to the oscillator that holds outside any dataset — a 200 Hz fundamental means 200 Hz of oscillation, always. Wiring it straight to the synth is what later lets the model extrapolate to pitches it never saw in training (Chapter 8). The decoder still shapes timbre, but the raw pitch bypasses it on a structural fast-lane.
Click each block to see what flows through it — tensor shapes, what's frozen vs trained, where pitch takes its structural shortcut to the synth. Red = learned network; green = latent code; gold = deterministic DSP.
We keep returning to one fact: comparing audio sample-by-sample is wrong, because two waveforms can sound identical yet differ point-by-point. So how do we measure whether two sounds match? DDSP compares them in the place perception lives: the spectrogram — a picture of how much energy sits at each frequency over time. Phase-shifted-but-identical-sounding waveforms have the same magnitude spectrogram, so a spectrogram loss correctly calls them equal.
The loss takes the original and synthesized audio, computes each one's magnitude spectrogram, and sums two terms: the plain difference of the spectrograms, plus the difference of their logarithms. The log term matters because hearing is roughly logarithmic — it makes the loss care about quiet details (low-energy harmonics, soft breath) that the raw difference would ignore. A small weight balances the two.
where Si is the original's spectrogram and Ŝi the synthesized one (both at FFT size i), the double bars are absolute-difference (L1) distance, and α = 1.0 weights the log term. But here's the "multi-scale" cleverness. A single spectrogram forces a trade-off: a large analysis window sees frequencies precisely but blurs time; a small window sees timing precisely but blurs frequency. You can't have both at once. So DDSP refuses to choose and computes the loss at six FFT sizes at once — 2048, 1024, 512, 256, 128, 64 — and sums them. The big windows police pitch accuracy; the small windows police timing and transients. Together they pin down the sound at every time-frequency resolution.
Slide the FFT window size. Big window (right): sharp frequency, smeared time — great for pitch, blind to a sudden attack. Small window (left): sharp time, smeared frequency — great for transients, fuzzy on pitch. Neither alone is enough; DDSP sums all six. Toggle "phase-shift the target" to confirm the spectral loss stays ~0 while a waveform loss would spike.
# Multi-scale spectral loss — the entire training objective def multiscale_spectral_loss(target, synth, fft_sizes=(2048,1024,512,256,128,64), alpha=1.0): loss = 0.0 for n_fft in fft_sizes: S = torch.stft(target, n_fft, hop_length=n_fft//4, return_complex=True).abs() Sh = torch.stft(synth, n_fft, hop_length=n_fft//4, return_complex=True).abs() loss += (S - Sh).abs().mean() # linear term: overall match loss += alpha * (torch.log(S + 1e-7) - torch.log(Sh + 1e-7)).abs().mean() # log: quiet detail return loss # no adversary, no autoregression — just this
One more component completes the chain, and it showcases DDSP's modularity beautifully: reverb. Every real recording carries the acoustic fingerprint of the room it was made in — the echoes and decay. Most neural synthesizers bake the room into the sound implicitly, with no way to separate them. DDSP instead models the room explicitly, as a final, separate convolution step after the synthesizer.
A room's effect is captured by its impulse response: what you'd hear if you clapped once — the full pattern of decaying echoes. Applying a room to dry audio means convolving the audio with that impulse response. The problem: a realistic room tail can be seconds long — tens of thousands of samples — and doing convolution the direct way scales cubically with length, far too slow. DDSP's fix is the same Fourier trick as the filter: convolution in time equals multiplication in the frequency domain, which scales as n log n. So even a multi-second room is cheap.
Because the room lives in its own module, you can do something remarkable: turn it off. Train the model with reverb on the violin recordings, then bypass the reverb module at synthesis time, and you get dereverberated audio — the violin as if recorded in an anechoic chamber. This solves "blind dereverberation" (removing reverb when you only ever had the reverberant recording), a long-standing acoustics problem, almost for free — purely because the architecture factored the room apart from the source. You can even take the learned room and apply it to different audio, transplanting the violin's hall onto a singing voice.
The dry source (from the synth) convolves with the room's impulse response to give the wet, reverberant output you actually recorded. Toggle the reverb module: ON reconstructs the recording; OFF gives clean dereverbed audio. Drag the room size to hear the tail grow. Because it's a separate factor, the room can be removed or transplanted at will.
Everything we built — the factorized controls, the structural pitch fast-lane, the modular room — pays off as a set of abilities that black-box models simply can't offer. Because the latent code is disentangled into pitch, loudness, and timbre, each one moves its own perceptual axis independently.
Independent control & interpolation. Hold timbre and pitch fixed and interpolate the loudness signal — the output's loudness follows, nothing else changes. Same for pitch, same for timbre (here visible as a smooth slide of the spectral "center of mass"). You can morph one quality at a time, which a tangled latent never allows.
Extrapolation beyond the data. Remember the pitch fast-lane to the synthesizer? Shift the fundamental down an octave — below anything in the training set — and the model still produces coherent audio, now resembling a related lower instrument like a cello. The oscillator handles the new frequency structurally; the decoder, which only knows nearby timbres, rides along. A black-box model asked for an unseen pitch would produce garbage.
Timbre transfer. The headline demo: sing into a model trained only on solo violin, and out comes a violin playing your melody. The recipe traces directly through the architecture: extract f(t) and l(t) from your voice, (optionally shift the pitch into the violin's register and transfer the violin's room onto your voice's loudness contour for a better match), then feed those controls to the violin-trained decoder + synthesizer. It renders your pitch and dynamics in the violin's learned timbre. The voice's performance survives; its sound is replaced.
A source "voice" gives a pitch contour and loudness. Watch them get extracted, optionally octave-shifted into the violin's register, and fed to the violin-trained synth — which redraws the same melody as a violin spectrum. Drag the source pitch and toggle the octave shift; the violin output's pitch tracks the source while its timbre stays violin.
On the numbers: the supervised DDSP autoencoder beats a state-of-the-art WaveRNN baseline on resynthesis accuracy — especially on fundamental-frequency error, since the additive synth uses the conditioning pitch directly — while using up to 10× fewer parameters. Even the harder unsupervised variant, which must learn pitch from scratch, outperforms the supervised WaveRNN. Quality, control, and efficiency, from baking in the physics.
DDSP's lesson is bigger than audio: when you know structure about your domain, bake it into a differentiable module rather than forcing the network to rediscover it. The result is smaller, more data-efficient, more interpretable, and more controllable models. Let's consolidate.
| Module | What it does | Controlled by |
|---|---|---|
| Harmonic oscillator | Sum of sines at integer multiples of f0 | f0, global amp A, harmonic dist c |
| Additive synth | The pitched/harmonic part of the sound | A(n)·c(n), with Σc=1 |
| FIR filter (freq-sampling) | Differentiable, linear-phase filtering | Per-frame transfer function H |
| Filtered noise (subtractive) | The breathy/stochastic part | Filter applied to white noise |
| Reverb | Room acoustics as FFT convolution | Impulse response (learned/fixed) |
| Encoder | Audio → (f, l, z) | CREPE (frozen) · loudness calc · MFCC+GRU |
| Multi-scale spectral loss | Perceptual reconstruction objective | 6 FFT sizes, L1 + α·log-L1 |
# DDSP autoencoder — one forward pass f = crepe(audio).detach() # pitch (frozen pretrained) — also goes STRAIGHT to synth l = loudness(audio) # loudness (deterministic perceptual weighting) z = gru(mfcc(audio)) # residual timbre (16-dim/frame, optional) amp, harm_dist, noise_H = decoder(f, l, z) # network outputs synth controls harmonic = harmonic_synth(f, amp, harm_dist) # additive (pitched) noise = filtered_noise(noise_H) # subtractive (breath/texture) dry = harmonic + noise # Harmonic-plus-Noise audio_hat = reverb(dry) # room (separable / removable) loss = multiscale_spectral_loss(audio, audio_hat) # the entire objective loss.backward() # gradients flow THROUGH the synth into the net
Press Teach Mode and explain the additive synthesizer's A·c factorization out loud — from memory. If you can derive why it separates loudness from timbre, you understand the heart of the paper.