The harmonic
model & F0
A trumpet's spectrum looks like chaos — a forest of peaks. But the peaks aren't scattered: they stand at perfect integer multiples of one hidden frequency. Find that frequency and the whole timbre collapses into two small numbers. This is how a machine hears pitch.
The peaks aren't scattered. They're a ladder.
In How Synths Make Sound you built timbre by stacking sines at f₀, 2f₀, 3f₀, …. In The DFT you ran the machine that pulls a stack like that back out of raw samples — a spectrum, a row of peaks.
Now look hard at that row of peaks for any pitched, sustained sound — a sung vowel, a bowed cello, a flute. The peaks are not sprinkled at random. They land at evenly spaced rungs: if the lowest is at 220 Hz, the next is at 440, then 660, then 880. Every peak is an integer multiple of the lowest one.
That lowest frequency — the rung the whole ladder is built on — is the fundamental frequency, written F0 (or f₀). It is what you perceive as the pitch. Everything above it is a harmonic (also called a partial), and harmonic h sits at a frequency you can predict exactly:
This is the harmonic model: the claim that a pitched sound's spectrum is, to very good approximation, a comb — a set of peaks at F0, 2·F0, 3·F0, …, each with its own amplitude. The amplitudes are the timbre (the 1/n staircase you heard for a saw). The spacing is the pitch.
And here is the payoff that makes this one of the most important ideas in all of audio. A raw spectrum of a 1024-sample window is 513 numbers. But if the sound is harmonic, you don't need 513 numbers — you need one frequency (F0) plus a short list of harmonic amplitudes. The whole sound compresses to (F0, a₁, a₂, a₃, …). That is the entire premise behind Spectral Modeling Synthesis and a hundred pitch-tracking, auto-tune, and music-transcription systems.
What the harmonic comb is, why estimating F0 is genuinely hard, the two-way mismatch idea that scores candidate fundamentals, a complete score worked by hand on real peaks, the eerie missing-fundamental illusion, the code in JavaScript and Python, and a capstone instrument where you drag a comb until it snaps onto the peaks and reads off the note.
One number generates every peak.
Picture a comb — a row of vertical teeth, all rigidly spaced. The first tooth sits at F0, the second at 2·F0, the third at 3·F0. Move F0 and the whole comb stretches or squeezes at once, because every tooth is pinned to that one number. The teeth can never drift apart unevenly — that rigidity is exactly what makes the model so powerful.
Estimating pitch, then, becomes a beautifully physical question: where do I have to place this comb so its teeth line up with the spectrum's peaks? Slide it too narrow and the teeth fall between peaks. Slide it too wide and they overshoot. There's one spacing where they click into place.
Let's make the comb literal before we score it. The amplitude of each tooth, for a typical saw-like tone, follows the same recipe you already met:
So a harmonic sound is two lists riding together: where the teeth are (controlled by one number, F0) and how tall they are (the timbre). The next chapters are entirely about recovering that one number from the peaks — because once you have it, the amplitudes are just "read the height at each predicted tooth."
Mathematician: the model is a parametric peak set {(h·F0, ah)}h=1…H — a function of a single frequency parameter and an amplitude vector. Musician: it's pitch and timbre, cleanly divorced — slide the comb (change pitch) without touching the tone color, or reshape the heights (change timbre) without changing the note. Artist: it's a rigid luminous rake you drag across a landscape of peaks until the teeth drop into the valleys and the whole thing locks with a snap. Hold all three and "F0 estimation" stops being a formula and becomes a gesture.
Why not just grab the lowest peak?
The naive idea: F0 is the pitch, the pitch is the lowest note, so just take the lowest peak in the spectrum and call it F0. It feels obvious. It is also wrong often enough to break any real system. Here is why.
The lowest peak can be weak — or absent
On a piano's low notes, a phone speaker, or a bowed string, the fundamental is often quieter than its harmonics — sometimes missing entirely. Grab the loudest peak and you'll land on the 2nd or 3rd harmonic and report a pitch an octave (or a fifth) too high.
Spurious peaks lie everywhere
Noise, reverb, two instruments at once, the buzz of a fan — all sprinkle extra peaks that aren't harmonics of anything. The "lowest peak" might be a fluke. You need a method that asks for agreement across many peaks, not one.
The octave trap
If F0 = 220 fits, then F0 = 110 also fits — its comb teeth (110, 220, 330, 440…) cover all the same peaks plus extras in between. And F0 = 440 fits too, using every other peak. A good scorer has to prefer the right octave, not just any comb that touches the peaks.
Peaks are never exact
Real measured peaks land at 218, 442, 658, 880 — not a clean 220, 440, 660, 880. Strings are slightly inharmonic, and the DFT's bin grid quantizes positions. The method must tolerate small misses while still rejecting large ones.
So we need a score that looks at the whole comb against the whole spectrum, forgives small errors, punishes large ones, and resists the octave trap. The classic answer from Stanford's ASPMA toolkit is the Two-Way Mismatch.
Score the comb both directions at once.
The trick in the name is two-way. A one-way score is fooled. Watch how, then watch the fix.
One-way, predicted → measured. Take your candidate comb's teeth — F0, 2·F0, 3·F0, … — and for each one, find the nearest measured peak and record the gap. Sum the gaps. Small sum means "all my predicted teeth landed near real peaks." Good… but a comb with too many teeth (a candidate F0 that's too low) can still score well, because each of its teeth finds some nearby peak. The octave-too-low trap survives.
The other way, measured → predicted. Now flip it: take each measured peak and find the nearest predicted tooth. If the candidate F0 is too low, it has spare teeth, but it also leaves real peaks far from any tooth — no wait, it covers them. The real punishment comes when F0 is too high: then there are peaks the sparse comb can't reach. This direction punishes missed peaks.
Each direction plugs the other's blind spot. Add them and the only candidate that scores well is the one whose teeth and the peaks agree mutually — neither has orphans. That's the two-way mismatch.
Each half is a sum of weighted gaps. A 5 Hz miss on a tooth near 100 Hz is a real problem (5% off); the same 5 Hz miss near 2000 Hz is nothing (0.25% off). So we don't sum raw distances — we sum each distance relative to the frequency it occurs at. Conceptually, for the predicted-to-measured half:
The measured-to-predicted half is the mirror image: loop over the measured peaks, divide each peak's miss by the peak's own frequency, average. (The full ASPMA formula also weights louder peaks more and adds a mild penalty that scales with frequency, so high spurious peaks can't dominate — but the relative-distance core above is the whole intuition.) The candidate F0 that drives this combined E to its minimum is your estimate. It is exactly the spacing where the comb snaps onto the peaks.
Score a candidate pitch by how well its predicted comb explains the measured peaks and how well the measured peaks are explained by the comb. The true F0 is the one where both directions are happy at once.
220 beats 300, on paper.
Formulas hide until you grind one out. A microphone hands us four measured peaks from a sung note:
We'll test two candidate fundamentals, F0 = 220 and F0 = 300, using just the predicted→measured half (each predicted tooth's relative miss to the nearest peak). Whichever has the smaller average wins. Watch the numbers.
Candidate F0 = 220. Predicted comb: 220, 440, 660, 880.
220. Nearest peak 218. Miss |220−218| = 2. Relative: 2/220 = 0.0091.440. Nearest peak 442. Miss 2. Relative: 2/440 = 0.0045.660. Nearest peak 658. Miss 2. Relative: 2/660 = 0.0030.880. Nearest peak 880. Miss 0. Relative: 0/880 = 0.0000.Candidate F0 = 300. Predicted comb: 300, 600, 900, 1200.
300. Nearest peak 218 (or 442 — 218 is closer). Miss |300−218| = 82. Relative: 82/300 = 0.273.600. Nearest peak 658. Miss 58. Relative: 58/600 = 0.097.900. Nearest peak 880. Miss 20. Relative: 20/900 = 0.022.1200. Nearest peak 880. Miss 320. Relative: 320/1200 = 0.267.There it is, in arithmetic you can check on a napkin: E(220) ≈ 0.004 versus E(300) ≈ 0.165. The comb at 220 explains the peaks; the comb at 300 flails between them. A search that tries many candidates and keeps the smallest E will glide straight to 220 — and then the measured-to-predicted half would slam the door on octave-down impostors like F0 = 110. That search is precisely the instrument at the bottom of this page.
| Candidate F0 | Predicted comb | Avg relative mismatch | Verdict |
|---|---|---|---|
| 220 Hz | 220 · 440 · 660 · 880 | 0.0042 | ★ snaps on |
| 300 Hz | 300 · 600 · 900 · 1200 | 0.165 | flails |
You can hear a note that isn't there.
Here is the strangest, most beautiful consequence of the harmonic model — and proof your brain is running it too. Play a sound with peaks at 400, 600, 800, 1000 Hz but with no peak at 200. There is literally no energy at 200 Hz. Yet your ear insists the pitch is 200 Hz, clear as day.
Why? Because 400, 600, 800, 1000 are the 2nd, 3rd, 4th, 5th harmonics of 200. Their spacing is 200 Hz. The comb that explains them has F0 = 200 — and your auditory system, which is itself a harmonic-comb matcher, reports the spacing as the pitch even though the fundamental tooth is missing. This is the missing-fundamental phenomenon (also called the phantom fundamental or residue pitch).
This is also why a tiny phone speaker that can't physically reproduce a bass guitar's 80 Hz fundamental still sounds like it has bass: it reproduces the harmonics (160, 240, 320…), and your brain fills in the missing 80 Hz root from the spacing. Engineers exploit this on purpose — "psychoacoustic bass enhancement" synthesizes harmonics so small speakers feel deep.
And it is the clinching argument for two-way mismatch over "grab the lowest peak": the lowest peak here is 400, which would report the wrong octave. Only a method that scores the whole comb's spacing — and rewards the candidate whose teeth fall on 400/600/800/1000 even though tooth #1 has nothing under it — gets the human answer of 200 Hz. The model and the perception are the same model.
When you drag the comb in the next section and feel it lock, you are doing by hand what your cochlea and auditory cortex do automatically every time you hear a voice. The "snap" is the same computation. That's why pitch perception feels effortless and instant — evolution shipped you a hardware TWM matcher.
F0 estimation is a search.
The math becomes code with no cleverness at all: write the mismatch for one candidate, then try many candidates and keep the smallest. Here is the predicted→measured half — the exact average you computed by hand — and the search around it:
// average RELATIVE miss between a candidate comb and the measured peaks function mismatch(f0, peaks, H = 8) { let err = 0; for (let h = 1; h <= H; h++) { const pred = h * f0; // predicted tooth at h·F0 let best = Infinity; for (const p of peaks) // nearest measured peak best = Math.min(best, Math.abs(pred - p)); err += best / pred; // relative miss (5 Hz hurts more down low) } return err / H; // the number you got by hand } // estimate F0 = the candidate with the lowest mismatch (a search) function estimateF0(peaks, lo = 60, hi = 600) { let bestF = lo, bestE = Infinity; for (let f = lo; f <= hi; f += 0.5) { const e = mismatch(f, peaks); if (e < bestE) { bestE = e; bestF = f; } // keep the valley floor } return bestF; } estimateF0([218, 442, 658, 880]); // → ~220 Hz
import numpy as np def twm(f0, peaks, H=8, rho=0.5): pred = f0 * np.arange(1, H + 1) # comb teeth f0, 2f0, ... # predicted -> measured: each tooth's relative miss to nearest peak e_pm = np.mean(np.min(np.abs(pred[:, None] - peaks), axis=1) / pred) # measured -> predicted: each peak's relative miss to nearest tooth e_mp = np.mean(np.min(np.abs(peaks[:, None] - pred), axis=1) / peaks) return rho * e_pm + (1 - rho) * e_mp # the blended error E(F0) def estimate_f0(peaks, lo=60, hi=600): cands = np.arange(lo, hi, 0.5) scores = np.array([twm(f, peaks) for f in cands]) return cands[np.argmin(scores)] # argmin = the snap point estimate_f0(np.array([218., 442., 658., 880.])) # -> ~220.0
The measured peaks themselves come from peak-picking the DFT magnitude you built in the last lesson — find local maxima, refine each to sub-bin accuracy by fitting a little parabola to the three samples around it. Then this search hands you F0. The whole pipeline — DFT → peaks → TWM → F0 — is what the next section lets you play.
Drag the comb. Feel it lock onto the pitch.
Time to point everything at once. Choose the tone's true fundamental, hit play to hear the harmonic comb, then drag the glowing comb left and right across the spectrum. Watch the two-way mismatch score in the valley below: it stays high while your teeth fall between peaks, then plunges to a deep minimum and the comb turns teal and snaps exactly when its spacing matches the true F0. The instant it locks, it reads off the detected frequency and the nearest note name. Or press auto-find and watch the search slide itself down the valley.
Music (Ableton): you drag the comb by hand and feel it click into the pitch — pitch detection as a tactile gesture. Math (Stanford): the valley is the two-way mismatch functional E(F0) evaluated across hundreds of candidates every frame; the snap is its argmin. Art (Nature of Code): a rigid rake of light dropping into a landscape of peaks, glowing teal the moment it locks. One gesture, one equation, one image — all the same signal.
And it scales straight to the destination. Replace this synthesized tone with the live output of a Lyria-class music model, run this exact F0 tracker on it frame by frame, and you have the melody line extracted in real time — a glowing pitch ribbon a cinematic visual can dance along, conducted by AI-generated music. Nothing in the matcher changes; only the source of the peaks does.
You've now compressed a whole timbre to (F0, harmonic amplitudes). But that's only the part of the sound that is harmonic. A real voice has breath; a real cymbal is almost pure hiss. What's left after you subtract the harmonic comb — the residual — carries the noise, the air, the life. Modeling that leftover is the next lesson.
See the Signal track → (the residual next)A whole timbre, a forest of peaks — and it all collapses to one number the moment the comb snaps on.
Serra et al., Audio Signal Processing for Music Applications — Module 6, Harmonic Model & F0 (Stanford / UPF; tools in sms-tools). · Maher & Beauchamp, "Fundamental frequency estimation of musical signals using a two-way mismatch procedure," JASA 95(4), 1994 — the original TWM. · Smith & Serra, "PARSHL / Spectral Modeling Synthesis," ICMC. · MDN, AnalyserNode — the spectra behind the panels.