Sinusoidal
+ residual
A flute is not just its pitch. Under the clean tone there's breath; under a bowed string, the faint scrape of horsehair; under a voice, the hiss of consonants. The harmonics carry the melody — but the residual carries the life. This lesson splits a sound into both, and lets you hear each one alone.
Model the tone perfectly. Something is still missing.
In How Synths Make Sound you learned that any periodic tone is a stack of sines — a fundamental plus harmonics. And in The DFT you learned to recover those sines from a recording. Put them together and you can do something powerful: analyse a real flute note, find its handful of strong harmonic peaks, and resynthesize them. This is the sinusoidal model — represent a sound as a few moving sine waves.
It works astonishingly well for the pitched part. Play back those sines and you clearly hear the flute's note, in tune, with the right brightness.
And yet it sounds slightly… dead. Sterile. Like a flute behind glass. What did we throw away?
We threw away everything that wasn't a clean, steady sine: the puff of air across the embouchure, the little key clicks, the turbulent hiss riding on top of the tone. None of that is periodic. None of it sits at a neat harmonic frequency. The DFT sees it as a low, broad smear across the whole spectrum — not spikes, but a floor. Our peak-picker ignored it, and the sound lost its breath.
The exact definition of the residual (what's left after you subtract the modeled sines), the harmonic-plus-residual and harmonic-plus-stochastic decompositions, a by-hand energy split using Parseval's theorem, why the residual can be modeled by just its smooth spectral envelope and regenerated as shaped noise instead of stored, the code in JavaScript and Python, and a live instrument that lets you hear and see the tonal part, the residual, and the full sound separately.
Subtract what you understand. Keep what you don't.
The whole idea fits in one move. You have a real sound x[n]. You model its tonal part as a sum of sinusoids — call that s[n], the part you understand. Then the residual is simply the part you couldn't explain: whatever is left over when you subtract.
Read it left to right. s[n] is the sinusoidal model — a few partials, each with its own slowly-changing amplitude Ap and phase φp. Those are the sharp harmonic spikes you'd see in the spectrum. Subtract that clean reconstruction from the original, sample by sample, and the difference r[n] is the messy remainder: breath, scrape, click, hiss.
Here's the beautiful part. By construction, the residual contains exactly the energy the sinusoidal model failed to grab. If your sines were perfect, r[n] would be silence. In practice it's never silence — and that's not failure, it's the sound's character. We don't discard the residual; we model it separately, with tools suited to noise rather than tones.
So a real instrument note splits cleanly into two tracks that live side by side:
tonal · s[n]
A few sinusoids — the pitch and harmonics. Sharp spikes in the spectrum. This is what the DFT's peak-picking finds.
residual · r[n]
x[n] − s[n]. Breath, bow scrape, consonants. A broad, smooth floor in the spectrum — no spikes. This is the leftover.
The residual is defined by subtraction: it is the original minus the part you could model with sines. Everything that isn't a clean tone ends up there — and that's precisely the part that makes a sound feel alive.
HpR keeps the noise exactly. HpS keeps only its shape.
Once you have r[n], you face a choice — and this choice is the whole reason the model is useful. There are two classic decompositions, named for what they do with the residual.
Harmonic plus Residual (HpR). Keep the residual as a real, sample-accurate signal. The reconstruction is exact: add the sines back to the stored residual and you get the original sound, bit for bit.
HpR is faithful but expensive: you're storing a second full waveform. And it's rigid — if you want to stretch the note in time or shift its pitch, the stored residual fights you, because it's locked to the original samples.
Harmonic plus Stochastic (HpS). Here is the leap. Instead of storing the residual's samples, we store only a description of its character — a smooth spectral envelope E[k] per analysis frame — and at synthesis time we regenerate the noise fresh from that envelope.
The reconstruction x̂[n] is not bit-identical to the original — the exact noise samples are different every time. But to your ear it's indistinguishable, and now the model is wonderfully flexible: stretch time and you just generate more noise from the same envelope; shift pitch and the sines move while the breath stays put. This is the heart of Serra's Spectral Modeling Synthesis (SMS).
| Model | Residual stored as | Reconstruction | Transformable? |
|---|---|---|---|
| HpR | actual samples r[n] | exact | poorly — residual is locked to time |
| HpS | smooth envelope E[k] only | perceptually identical | yes — regenerate freely |
HpS throws away the exact noise samples and keeps only a smooth curve. Why is that allowed? Because the ear, for noise, is deaf to the exact waveform and phase — it hears only the noise's spectral shape. We'll prove the energy bookkeeping by hand (Chapter 3), then build the envelope and see why "smooth" is enough (Chapter 4).
Where did the energy go? Parseval keeps the books.
Here's a worry. We split x[n] into sines s[n] plus residual r[n]. Did energy leak out in the splitting? Can we account for every bit? Yes — and the tool is Parseval's theorem, which says the total energy of a signal is the same whether you measure it in time (sum of squared samples) or in frequency (sum of squared spectrum magnitudes). Energy is conserved across the transform.
Because the modeled sines and the residual are, by design, the two halves of the sound, their energies simply add up to the whole. So the residual energy is just total energy minus sinusoidal energy — and we can get a clean number out of it. Let's grind a real example.
Setup. You analyse one short frame of a bowed cello note. You measure the total energy of the frame, and you measure how much energy your modeled harmonics carry. Suppose:
Ex = Σ x[n]² = 100 (some unit — say, summed squared samples).Es = Σ s[n]² = 91. The harmonics carry most of it — as they should for a pitched note.Er = Ex − Es = 100 − 91 = 9.9/100 = 9% of the frame's energy. In dB: 10·log₁₀(9/100) = 10·log₁₀(0.09) = −10.46 dB below the total — quiet, but the −10 dB of "air" your ear absolutely notices when it's gone.Two things to internalize. First, the subtraction r[n] = x[n] − s[n] is what guarantees the energies add — we didn't have to assume the sines and noise were uncorrelated, the split is exact by construction. Second, this number is how an analyzer decides how loud to make the regenerated noise: the stored envelope is scaled so the synthesized residual carries the same 9 units of energy. Get that scaling wrong and a flute either sounds airless (residual too quiet) or like it's hissing through a vent (too loud).
One subtlety to be honest about: the clean "energies add" picture assumes the residual is the true leftover. If your sinusoidal model is sloppy — wrong frequencies, missed partials — some real harmonic energy spills into r[n], and the residual stops being pure noise. A good analysis is one where r[n] looks and sounds like only noise. That's the quality check.
Mathematician: the residual is an orthogonal-by-construction complement, r = x − s, and Parseval lets you split total energy into Ex = Es + Er. Musician: it's the breath, the bow, the consonant — the articulation that separates a sampled flute from a real one. Artist: in the spectrum it's a soft, drifting haze beneath the bright harmonic spikes — fog under a row of streetlamps. One signal, three readings of the same nine units of energy.
The ear hears the noise's silhouette, not its samples.
Now the central trick of HpS, made concrete. The residual's spectrum is jagged and random — every frame it dances differently. But step back and squint, and a stable silhouette emerges: maybe a gentle hump in the low-mids and a bright shelf up top. That silhouette is the spectral envelope E[k] — a smooth curve that traces the average level of the noise across frequency.
The classic way to capture that silhouette is to approximate the residual's magnitude spectrum with a small number of straight pieces — a line-segment approximation (a piecewise-linear envelope), or equivalently an LPC-style all-pole fit. A handful of breakpoints is enough; that's the entire description of the noise we keep.
Two things get thrown away, and it's worth being precise about why each one is safe.
We throw away the fine spectral detail — the bin-to-bin wiggle — and keep only the smooth trend. Safe, because for noise that wiggle is randomness; it carries no perceptual information. Two noise bursts with the same envelope but different fine structure sound the same.
We throw away the phase entirely. This is the deep one. For a tonal partial, phase matters — it sets where the wave sits in time, and getting it wrong can smear an attack. But for noise, phase is meaningless: random phase is what makes noise sound like noise. Scramble the phase of white noise and it's still the same white noise. So we keep no phase for the residual at all.
And here is the payoff — the answer to "why regenerate instead of store?":
A second of stereo residual is ~88,000 numbers per channel. Its envelope is maybe a dozen breakpoints per frame. Because the ear judges noise only by its spectral shape — not its exact samples, not its phase — we can throw away the 88,000 samples, keep the dozen breakpoints, and at playback generate brand-new white noise and filter it to match the curve. Same perception, a tiny fraction of the data, and now it's a knob you can turn: stretch it, brighten it, pitch the tone without touching the breath. Storage becomes a recipe, exactly as additive synthesis turned a tone into a recipe of harmonics.
How do you generate noise shaped to a curve? Take flat white noise (equal energy at every frequency — a flat spectrum) and push it through a filter whose frequency response is the envelope E[k]. The filter boosts the frequencies where the envelope is high and cuts where it's low. White goes in; the residual's silhouette comes out. The widget at the end builds exactly this: a noise buffer feeding a small bank of broad bandpass filters that paint the smooth envelope.
store (HpR)
~88,000 residual samples / second. Exact, but huge and rigid.
recipe (HpS)
~12 envelope breakpoints / frame. Regenerate noise to match → tiny, and transformable.
Subtract the sines. Fit the leftover's curve. Replay it as noise.
The analysis side is three honest steps: synthesize the modeled sines, subtract to get the residual, then fit a smooth envelope to the residual's spectrum. Here it is end to end — no cleverness, just the definitions from the chapters above.
import numpy as np def synth_sines(freqs, amps, n, fs): # rebuild s[n] from the found partials t = np.arange(n) / fs return sum(a * np.cos(2*np.pi*f*t) for f, a in zip(freqs, amps)) def residual(x, freqs, amps, fs): s = synth_sines(freqs, amps, len(x), fs) r = x - s # r[n] = x[n] - s[n] ← the whole idea return r def residual_envelope(r, n_bands=12): R = np.abs(np.fft.rfft(r)) # |R[k]| — jagged noise spectrum bands = np.array_split(R, n_bands) # group bins into a few segments env = np.array([b.mean() for b in bands]) # smooth: keep only the average per band return env # a dozen numbers — that's all we store # energy check (Parseval / conservation), the Ch.3 hand example in code: # E_x = np.sum(x**2); E_s = np.sum(s**2); E_r = E_x - E_s → 9% for the cello
And the synthesis side — the HpS replay — turns that dozen-number envelope back into sound. We make fresh white noise and shape it to the curve. In the browser, the natural way is a bank of broad bandpass filters fed by a noise buffer, which is exactly what the instrument below does:
// 1) flat white noise — equal energy at every frequency (a fresh recipe each time) const len = ctx.sampleRate * 2; const buf = ctx.createBuffer(1, len, ctx.sampleRate); const d = buf.getChannelData(0); for (let i = 0; i < len; i++) d[i] = Math.random() * 2 - 1; const noise = ctx.createBufferSource(); noise.buffer = buf; noise.loop = true; // 2) paint the smooth ENVELOPE with a few broad bandpass humps (the line segments) const out = ctx.createGain(); [350, 1100, 3000].forEach(fc => { const bp = ctx.createBiquadFilter(); bp.type = 'bandpass'; bp.frequency.value = fc; bp.Q.value = 0.7; // low Q = broad, SMOOTH hump (no sharp resonance) noise.connect(bp); bp.connect(out); // parallel: all humps sum into one envelope }); noise.start(); // white in → the residual's silhouette out // 3) the full HpS sound is just the tonal oscillator + this shaped noise, summed: // tonalGain.connect(master); out.connect(master);
For the tonal part, the recipe was a list of harmonic amplitudes (additive synthesis — sharp spikes). For the residual, the recipe is a list of envelope breakpoints (subtractive / filtered noise — a smooth floor). Two recipes, one sound. The sinusoidal model handles the spikes; the stochastic model handles the floor. That's the entire HpS pipeline.
Hear the tone. Hear the breath. Hear them become one.
Everything above is now one instrument you play. It synthesizes a "tonal + noise" sound from scratch in your browser — a harmonic oscillator for the sines, and white noise through a smooth filter bank for the residual. Use the three buttons to solo each layer, and watch the spectrum:
Tonal gives you sharp, evenly-spaced harmonic spikes — clean pitch, but airless. Residual gives you a smooth, drifting hump with no spikes at all — pure air, no pitch. Full rides the harmonics on top of the noise floor, and the crossfade slider lets you dial how much breath to mix in. The warm line is the residual's smooth envelope — the dozen numbers you'd actually store.
Music (Ableton): you're crossfading the breath of an instrument — the exact knob a sound designer reaches for to make a sample feel real. Math (Stanford): the split is r = x − s; the smooth warm line is the line-segment envelope E[k]; the layers' energies add by Parseval, just like your by-hand 91 + 9 = 100. Art (Nature of Code): the harmonics are bright fixed pillars, the residual a soft generative haze — order and chaos in the same frame. The sound, the equation, and the picture are one object you're holding.
Now that a sound is two independent tracks — sinusoids and residual — you can transform each on its own. Slow the sines for a time-stretch while the breath stays natural; shift the harmonics to pitch-shift while the consonants don't chipmunk; morph one instrument's harmonics onto another's breath. That independence is the entire power of Spectral Modeling Synthesis, and it's the doorway to the transformation lessons — and, downstream, to a Lyria-class model whose generated tone and generated air can be conducted separately across the cinematic canvas.
A flute's pitch is its sines. A flute's soul is what's left over.
Xavier Serra et al., Audio Signal Processing for Music Applications — Module 7, Sinusoidal-plus-residual model (Stanford / UPF); the sms-tools reference implementation. · X. Serra & J. Smith, "Spectral Modeling Synthesis: A Sound Analysis/Synthesis System Based on a Deterministic plus Stochastic Decomposition," Computer Music Journal, 1990. · Julius O. Smith, Spectral Audio Signal Processing (CCRMA) — Parseval, envelopes, LPC. · MDN, BiquadFilterNode & AudioBufferSourceNode — the noise + filter graph behind the instrument.