The sinusoidal
model
A second of sound is 44,100 numbers. But a flute holding a note is really just a few sine waves. Find them, follow them, and you can throw the 44,100 numbers away — and rebuild the sound from a handful. That handful is the model.
A held note hides in a wall of samples.
In The DFT you learned to turn a window of samples into a spectrum — a row of bars, one height per frequency bin. For a held musical note, almost every one of those bars is zero. The energy clumps into a few tall spikes, evenly spaced: the fundamental and its harmonics.
That is a wasteful way to describe a flute. Why keep 1,025 spectral bins when only six of them carry the sound? The sinusoidal model says: don't. Describe the sound as a short list of sinusoids, each with a frequency and an amplitude. A flute note becomes maybe six numbers, not a thousand.
This is the foundation of Spectral Modeling Synthesis (SMS): represent a sound as a small set of time-varying sinusoids. To get there we need three moves, and we'll build each one by hand — find the peaks, refine them, follow them through time — and then resynthesize the sound from nothing but that list.
What a spectral peak is and how to detect it; parabolic interpolation to find a peak's true frequency between the bins — derived and worked on paper; how peaks join into continuous tracks across frames; resynthesis as a sum of sinusoids; the code in JavaScript and Python; and a live instrument that finds the peaks of a tone you play and rebuilds it from them so you can hear how close the model gets.
A peak is just a local maximum.
Look at the magnitude spectrum |X[k]| from the last lesson. A spectral peak is exactly what your eye already calls one: a bin that pokes up above both of its neighbours. Formally, bin k is a peak when its magnitude is at least as large as the bins on either side of it.
That test alone would flag dozens of tiny ripples — the random wiggle in the noise floor will always have local maxima. So we add a threshold: a peak only counts if it stands far enough above the floor. We measure that distance in decibels, relative to the loudest peak in the frame. A typical rule is "ignore anything more than 60 dB below the strongest partial."
Working in dB is not a stylistic choice. Our ears hear loudness as ratios, and a spectrum spans a huge dynamic range — the sixth harmonic of a real note can be a thousand times quieter than the fundamental yet still audible. On a linear scale it would vanish; in dB it's a clear, pickable bump.
Mathematician: a peak is a discrete local maximum of |X[k]| above a threshold — a stationary point you find by comparing neighbours. Musician: each surviving peak is one partial you can actually hear — a sine you could hum. Artist: the spectrum is a mountain range, and you are marking only the true summits, ignoring the foothills of noise.
The true frequency lives between two bins.
Here is the catch that makes the sinusoidal model subtle. The bins are spaced fs/N apart — at CD rate with a 2048-point window, about 21.5 Hz per bin. A real flute note at 261 Hz almost never lands exactly on a bin centre. Its energy falls between two bins, and the tallest bin is only the nearest one — off by up to half a bin, which can be a tenth of a semitone. Audibly wrong.
But the model gives us a gift. Because we analyse with a smooth window (a Hann window), the shape of a single sinusoid's spectral peak is a smooth, symmetric bump — and near its top it looks almost exactly like a parabola. So we don't have to settle for the nearest bin. We fit a parabola through the peak bin and its two neighbours, and read off the vertex — the true sub-bin location.
Let the three points be the peak bin k and its neighbours, with magnitudes in dB:
Put the peak bin at the origin, so the three sample positions are x = −1, 0, +1. Fit a parabola y(x) = ax² + bx + c. Plug in the three points:
x = 0: y(0) = c = β. The constant term is just the peak bin's height.x = −1: a − b + c = α.x = +1: a + b + c = γ.(a+b+c) − (a−b+c) = γ − α ⟹ 2b = γ − α ⟹ b = (γ − α)/2.2a + 2c = α + γ ⟹ a = (α − 2β + γ)/2.The vertex of a parabola sits where the derivative is zero: y′(x) = 2ax + b = 0, so x = −b/(2a). Substitute a and b:
The refined peak location is k + p, and so the true frequency is (k + p)·fs/N. We also recover a better amplitude — the height of the parabola at its vertex, which is more accurate than the raw bin value:
A Hann window's main lobe is close to a parabola on a log (dB) scale, not on a linear one. Fitting the parabola to 20·log₁₀|X[k]| gives a markedly more accurate frequency estimate than fitting to raw magnitude — a small detail that is the difference between an in-tune resynthesis and a sour one.
Three numbers, one true peak.
Formulas only become real once you grind one. Suppose our peak picker flagged bin k = 50, and the three dB magnitudes around it came out as:
The right neighbour (−6.0) is taller than the left (−8.0), so the true peak must sit a little to the right of bin 50. Let's get the exact amount. Plug straight into the vertex formula:
α − 2β + γ = (−8.0) − 2(−4.0) + (−6.0) = −8.0 + 8.0 − 6.0 = −6.0.α − γ = (−8.0) − (−6.0) = −2.0.p = ½ · (−2.0) / (−6.0) = ½ · 0.3333 = +0.1667. Positive — it leans right, exactly as we predicted.k + p = 50 + 0.1667 = 50.1667.fs = 44100, N = 2048, bin spacing ≈ 21.53 Hz): f = 50.1667 × 21.53 ≈ 1080.1 Hz.AdB = β − ¼(α − γ)p = −4.0 − ¼(−2.0)(0.1667) = −4.0 + 0.0833 = −3.92 dB — slightly taller than the raw bin, since the true peak sits above the sample.Without interpolation we'd have called this peak 50 × 21.53 = 1076.6 Hz. The parabola moved it to 1080.1 Hz — a 3.5 Hz correction, about 6 cents of pitch. On a sustained tone that is the difference between a clean resynthesis and a faint, swimming detune. Three numbers, two lines of arithmetic, and the sinusoid is pinned to within a fraction of a bin.
Connect the peaks across frames.
One spectrum is a single instant. A real sound moves — a note swells, vibratos, slides. So we slice the sound into overlapping short windows (the STFT from the spectrogram lesson) and detect peaks in each frame. Now we have a cloud of peaks scattered across time and frequency. The third move of the model is to connect them into continuous tracks.
The idea is plain: a partial doesn't teleport. From one frame to the next its frequency barely moves. So peak continuation matches each peak in the new frame to the closest-in-frequency peak in the previous frame — within a tolerance. Match, and the track continues. No nearby peak within tolerance? The track dies. A new peak with no parent? A track is born.
The output is a set of trajectories: each one is a partial's frequency and amplitude traced through time. A flute with vibrato becomes a few gently undulating lines. The attack of a piano note becomes many short-lived tracks that flare and vanish. This is the "time-varying sinusoids" promised at the start — and it is what makes the model expressive: you can stretch a sound in time by reading the tracks more slowly, or transpose it by scaling every track's frequency, all without artifacts, because you're manipulating meaning, not samples.
A peak with no match within Δtol in the previous frame starts a new track. A track with no continuation in the new frame is closed. Births and deaths are how the model handles attacks, transients, and notes turning on and off.
Continuous tracks make pitch-shift and time-stretch independent. Scale every track's frequency → transpose with no chipmunk speedup. Read the tracks slower → stretch time with no pitch drop. That separation is impossible on raw samples.
Detect, interpolate, resynthesize.
The whole detector is a scan for local maxima above a threshold, with the parabola folded in. Here it is end to end — pick the peaks of a magnitude spectrum and return refined (frequency, amplitude) pairs:
function findPeaks(mag, fs, N, threshDb) { const dB = mag.map(m => 20 * Math.log10(m + 1e-9)); const maxDb = Math.max(...dB), floor = maxDb - threshDb; const peaks = []; for (let k = 1; k < dB.length - 1; k++) { const b = dB[k]; if (b < floor) continue; // below threshold — skip if (b < dB[k-1] || b < dB[k+1]) continue; // not a local max const a = dB[k-1], g = dB[k+1]; const p = 0.5 * (a - g) / (a - 2*b + g); // parabola vertex offset const ampDb = b - 0.25 * (a - g) * p; // interpolated height peaks.push({ freq: (k + p) * fs / N, amp: 10 ** (ampDb / 20) }); } return peaks; }
And resynthesis is the inverse of everything you did in the synth lesson: additive synthesis. Take the list of detected sinusoids and add them back up — one oscillator per peak, at its frequency and amplitude. In NumPy you can do it offline; in the browser you hand each peak to a Web Audio oscillator (exactly what the instrument below does):
import numpy as np def find_peaks(mag, fs, N, thresh_db=60): dB = 20 * np.log10(mag + 1e-9) floor = dB.max() - thresh_db out = [] for k in range(1, len(dB) - 1): b = dB[k] if b < floor or b < dB[k-1] or b < dB[k+1]: continue a, g = dB[k-1], dB[k+1] p = 0.5 * (a - g) / (a - 2*b + g) # sub-bin offset amp = 10 ** ((b - 0.25*(a-g)*p) / 20) out.append(((k + p) * fs / N, amp)) # (freq, amp) return out def resynth(peaks, fs, dur): t = np.arange(int(fs*dur)) / fs return sum(a * np.sin(2*np.pi*f*t) for f, a in peaks) # additive!
You once typed harmonic amplitudes and Web Audio summed them into a tone (additive synthesis). The sinusoidal model goes the other way — it recovers those amplitudes from a recorded sound — and then resynthesis runs the additive summation again. Analysis and synthesis are the same machine, back to back.
Find the peaks. Rebuild the sound.
Time to run the whole pipeline at once. Below is a tone built from a few harmonics — its real spectrum is drawn live. The instrument detects the spectral peaks (the marked dots are the parabola-refined positions, each labelled with its true frequency), and then lets you resynthesize from the peaks only: it throws away the original and rebuilds the sound from nothing but the detected list of sinusoids.
Play the original, then flip to resynth from peaks and listen — for a few clean harmonics they are nearly indistinguishable, because the model captured everything that mattered. Then drag brightness up to pour in more partials, and raise the threshold to see peaks drop out one by one: when you discard the quietest sinusoids, the resynthesis dulls. You are hearing the model's budget — how few sines you can get away with.
Music (Ableton): you hear the original and its reconstruction and decide, by ear, how few sines suffice. Math (Stanford): the markers are local maxima of |X[k]|, each pinned to its true frequency by the parabola you derived; the sound you hear is the additive sum of those numbers. Art (Nature of Code): a thousand-bin spectrum collapses into a sparse constellation of summits — the sound drawn as just its peaks.
Stack the peaks. You get a comb.
Look again at the peaks of a single sustained note. They aren't scattered randomly — they march at even spacing: f₀, 2f₀, 3f₀, 4f₀…. That regular spacing is no accident; it's the signature of a pitched, periodic sound. The peaks line up like the teeth of a comb, and the spacing between teeth is the fundamental frequency f₀ — the pitch.
So far we've treated every peak as an independent sinusoid. The next move is to notice the pattern: if the peaks form a harmonic comb, we can describe the whole sound with one number — f₀ — plus the amplitude of each tooth. That is the harmonic model, and the act of measuring the comb's spacing is fundamental-frequency (F0) estimation. It's how a tuner knows you played A4, and how a vocoder follows a melody.
The free sinusoidal model captures anything — bells, noise, inharmonic clangs. Constrain the peaks to a harmonic comb and you get a tighter, cheaper model for pitched sounds, and a pitch readout for free. That is the harmonic model and F0 estimation — the next lesson in the Signal track, and the step that lets a Lyria-class model's melody be tracked, retuned, and painted across the cinematic canvas.
A second of a held note was 44,100 numbers. You just heard it rebuilt from six.
Serra et al., Audio Signal Processing for Music Applications — Module 5, the sinusoidal model and SMS (Stanford / UPF; tools at sms-tools). · McAulay & Quatieri, "Speech analysis/synthesis based on a sinusoidal representation," IEEE TASSP, 1986 — the original peak-tracking model. · Serra & Smith, "Spectral Modeling Synthesis," CMJ, 1990. · Smith, Spectral Audio Signal Processing (CCRMA) — quadratic peak interpolation, derived.