STFT & the
spectrogram
The DFT told you what frequencies were in a sound — but it threw away when. Slice the sound into short windows, transform each one, and stack them in time. You get the single most important picture in audio: the spectrogram — the image every audio AI sees.
The DFT can't tell time.
In The DFT you built a machine that takes a wave and hands back its recipe of frequencies. It works perfectly — for a sound that never changes. A held organ chord. A single sustained tone.
But music moves. A melody is a different note every moment. A word is a sweep of changing resonances. If you run one giant DFT over a whole song, you do get the right frequencies — every note that ever played — but smeared into one motionless bar chart. You learn the song's ingredients and lose the recipe's order entirely. You can't tell a rising scale from a falling one. Both have the same set of notes.
Here's the honest problem stated plainly: the DFT trades all time information for perfect frequency information. One number per frequency, valid for the whole window, with no sense of where inside the window it happened.
The fix is almost insultingly simple, and it is the whole of this lesson: don't transform the whole sound at once. Transform little slices of it, one after another, and lay the results side by side. That is the Short-Time Fourier Transform, and the picture it draws is the spectrogram.
Why one DFT loses time and how slicing recovers it · the STFT formula derived from the DFT term by term · the window w[n], hop size H, and overlap · the Gabor–Heisenberg time–frequency tradeoff, derived in numbers · magnitude (in dB) vs phase, and why we plot magnitude · perfect reconstruction and the inverse STFT, briefly · the code in JavaScript and Python · and a flagship instrument where a chirp you synthesize paints its own spectrogram live while a slider trades sharpness in time for sharpness in frequency.
One DFT per slice. Stack them.
Picture the long stream of samples x[0], x[1], x[2], … laid out left to right. Drop a small window over the first chunk — say N samples — and run the DFT you already built. That gives you one column of frequency magnitudes: the spectrum of that instant.
Now slide the window to the right by a small step — the hop size H — and DFT the next chunk. Another column. Slide again. Another. Lay every column side by side, oldest on the left, and you have a 2-D image: time runs across, frequency runs up, and the brightness of each cell is how much of that frequency was present at that moment.
That image is the spectrogram, and its definition is the DFT with one extra index — the slice number m:
x[n + mH] — the signal, but read starting at sample mH: this is what picks out slice number m. Bump m by one and the read-head jumps forward by the hop H.w[n] — the window, a short bump that's nonzero only over the N samples of the slice. It is the only new ingredient versus the plain DFT (Chapter 2 explains why it can't just be a hard cut).e−i·2πkn/N — the exact circling test wave from the DFT lesson. Nothing changed: this is still "does frequency k fit?", asked of one slice.Σ over n — sum the windowed slice against the test wave. The result X[m,k] is one complex number: bin k of slice m. Two indices now — frequency and time.Compare it to the plain DFT, X[k] = Σ x[n]·e−i·2πkn/N. The STFT is literally that same sum, run once per slice, with a window multiplied in. If you only ever took one slice (m = 0) covering the whole signal, the STFT collapses back into the ordinary DFT. The STFT is not a new transform — it is the DFT, used repeatedly and locally.
The spectrogram is the magnitude of the STFT, |X[m,k]|, drawn as a heat-map: x = time (slice m), y = frequency (bin k), color = loudness. Every speech recognizer, every music model, every audio classifier you have ever heard of takes this picture as its input. Learn to read it and you can read what a machine hears.
A hard cut rings. A soft window doesn't.
The obvious way to take a slice is to grab N samples and chop. That is a rectangular window — w[n] = 1 inside, 0 outside. It seems harmless. It is not.
The DFT secretly assumes whatever you hand it repeats forever, wrapping the last sample back to the first. If a slice doesn't contain a whole number of cycles, the wrap-around makes a sudden jump — a discontinuity — at the seam. And a sudden jump is, in Fourier terms, a splash of energy at every frequency. One pure tone, sliced badly, smears into a comb of false neighbours. This artifact is called spectral leakage.
The cure is to taper the slice to zero at both ends before transforming, so there is no seam to jump across. Multiply the slice by a smooth bump w[n] — a window function. The most common is the Hann window, a single raised cosine:
So the windowed slice is just x[n + mH] · w[n] — the signal gently faded in and out — and that is what we feed the DFT. Tapering trades a little: the main peak gets slightly wider, but the false leakage between peaks drops enormously. For looking at music, that is always the right trade. (Web Audio's AnalyserNode applies a Blackman window for you before its internal FFT — same idea, slightly different bump.)
Mathematician: multiplying by w[n] in time is convolving the true spectrum with the window's own spectrum — a tapered window has a narrow, low-sidelobe spectrum, so it blurs less. Musician: the window is a tiny fade-in/fade-out on each grain of sound, exactly like the micro-fades a sampler uses to avoid clicks at loop points. Artist: without it the spectrogram is hazed with vertical smear at every onset; with it, each tone becomes a clean, bright filament. The window is what makes the picture legible.
You can be sharp in time, or sharp in frequency. Never both.
Now the deepest idea in the whole module, and it falls straight out of the formula. The window length N sets two resolutions at once, and they pull in opposite directions.
Time resolution is how finely you can locate when something happened. A slice covers N samples, which is a stretch of real time:
Everything inside one window gets blurred into a single column, so a short window pins down time precisely. Frequency resolution is how finely you can tell two close pitches apart. From the DFT lesson, the bins are spaced by the sample rate over the window length:
There's the vice. Look at what happens when you multiply them together — the sample rate cancels:
The product is fixed at 1 (with this simple definition; tapered windows nudge the constant a bit). You cannot shrink both. Want to pin down when a drum hit landed? Use a short window — but now bass notes blur together because your bins are wide. Want to resolve a 440 Hz note from a 443 Hz note? Use a long window — but now a fast run of notes smears into one fat column because each slice covers a long stretch of time.
This is the Gabor limit, and it is the audio twin of Heisenberg's uncertainty principle: position and momentum, time and frequency — the same mathematics says you cannot localize both at once. It is not a flaw in our method. It is a law about windows.
Sharp in time — onsets crisp, fast melodies readable. Blurry in frequency — close pitches merge, bins are wide. Good for percussion, transients, speech timing.
Sharp in frequency — fine pitch detail, harmonics resolved. Blurry in time — onsets smear, fast notes merge. Good for sustained tones, harmony, tuning.
CD rate, two window sizes, worked out.
Abstract limits become real the moment you plug in CD-quality numbers. Fix the sample rate at fs = 44100 Hz and compare a short window N = 512 against a long one N = 4096. Just substitute into the two formulas.
Δf = fs/N = 44100 / 512 ≈ 86.1 Hz. Each bin is ~86 Hz wide — fat. You could not tell 440 Hz from 480 Hz apart; they land in the same bin.Δt = N/fs = 512 / 44100 ≈ 0.0116 s ≈ 11.6 ms. Each column spans ~12 ms — you can pinpoint a drum hit to a hair.Δf = 44100 / 4096 ≈ 10.8 Hz. Eight times finer — now 440 vs 443 Hz is easy. You can see tuning.Δt = 4096 / 44100 ≈ 0.0929 s ≈ 92.9 ms. Each column now smears ~93 ms of sound — eight times blurrier in time. A fast trill turns to mush.11.6 ms × 86.1 Hz ≈ 1.00 and 92.9 ms × 10.8 Hz ≈ 1.00. Going from 512 to 4096 multiplied frequency sharpness by 8 and divided time sharpness by 8. The product never budged.| Window N | Δf = 44100/N | Δt = N/44100 | Δt · Δf | Best for |
|---|---|---|---|---|
| 256 | 172 Hz | 5.8 ms | ≈ 1 | transients, clicks |
| 512 | 86 Hz | 11.6 ms | ≈ 1 | percussion, timing |
| 2048 | 21.5 Hz | 46 ms | ≈ 1 | general music (default) |
| 4096 | 10.8 Hz | 93 ms | ≈ 1 | harmony, tuning, bass |
This single table is why an audio engineer reaches for N = 512 when chasing a snare and N = 4096 when chasing a slightly flat cello. There is no "best" window — only the right window for the question. In a moment you'll feel this by dragging the size slider on a live spectrogram and watching the picture sharpen one way as it blurs the other.
The STFT is a loop around the DFT.
Because the STFT is just the DFT run per slice, the code is a loop that windows a chunk, transforms it, and pushes one magnitude column. Reuse the very dft you wrote in the last lesson:
// dft(slice) → array of { re, im, mag } (from the DFT lesson) function hann(N) { // the taper from Chapter 2 const w = new Float32Array(N); for (let n = 0; n < N; n++) w[n] = 0.5 * (1 - Math.cos(2 * Math.PI * n / (N - 1))); return w; } function stft(x, N, H) { // N = window len, H = hop size const w = hann(N), frames = []; for (let m = 0; m * H + N <= x.length; m++) { // slide by H each time const slice = new Float32Array(N); for (let n = 0; n < N; n++) slice[n] = x[m * H + n] * w[n]; // window the slice const X = dft(slice); // one DFT → one column frames.push(X.slice(0, N / 2).map(c => c.mag)); // keep magnitudes, lower half } return frames; // frames[m][k] = |X[m,k]| → the spectrogram }
import numpy as np def stft(x, N=1024, H=256): w = np.hanning(N) # the Hann window frames = [] for m in range(0, len(x) - N, H): # slide by hop H slice_ = x[m:m + N] * w # window the slice X = np.fft.rfft(slice_) # one FFT → one column frames.append(np.abs(X)) # magnitude = the spectrogram column S = np.array(frames).T # shape (freq bins, time frames) return 20 * np.log10(S + 1e-6) # to decibels — what we actually plot # every audio library is this loop, optimized: # librosa.stft(x, n_fft=1024, hop_length=256) → identical idea
Each X[m,k] is complex: a magnitude (how much of bin k) and a phase (its alignment). We almost always draw only |X[m,k]|, and in decibels (20·log₁₀|X|), because hearing is logarithmic — dB stretches the quiet detail up where your eye and ear can see it. The phase looks like random noise to the eye and carries the fine timing of each grain; we keep it only when we need to rebuild the sound. For reading a sound — pitch, harmony, onsets — magnitude is everything.
Can you turn the picture back into sound? Yes — if the slices overlap. Inverse-DFT each column back to a little windowed wave, then overlap-add them: drop each reconstructed slice at its hop position and sum where they overlap. With a Hann window and 50% overlap (H = N/2), the overlapping tapers add up to a flat 1 everywhere — the constant-overlap-add (COLA) condition — so the windowing perfectly cancels and you recover x exactly. That round trip — sound → spectrogram → edit → sound — is how every phase-vocoder time-stretch and pitch-shift works.
Watch a sound draw its own spectrogram.
Now make it real. Press play and a sound is synthesized live in your browser — no recording — and its spectrogram scrolls in from the right, one STFT column per frame, exactly the loop you just read. The chirp ramps its pitch low→high, so it paints a clean diagonal line: frequency literally climbing through time. The arpeggio turns notes on and off, stacking bright horizontal segments — a melody you can see.
Then the payoff: drag the window size slider and watch the Gabor tradeoff with your own eyes. Small N — the chirp's diagonal stays crisp left-to-right (sharp in time) but thickens top-to-bottom (blurry in frequency). Large N — the line goes razor-thin vertically (sharp in frequency) but the fast notes smear sideways (blurry in time). The readouts show Δt and Δf updating live; tick freeze & scrub to hover any column and read its dominant frequency.
Music (Ableton): you press play and hear the chirp climb or the arpeggio dance. Math (Stanford): every column on screen is one |X[m,k]| — the STFT sum from Chapter 1, computed live by an AnalyserNode's FFT — and the slider is you choosing N on the Gabor curve. Art (Nature of Code): the heat-map is the generative canvas, a flowing image grown entirely from the signal. The sound, the equation, and the picture are not three things side by side — they are one stream of samples, read three ways.
From the DFT, toward the model.
Back. This whole lesson stood on one earlier idea. The diagonal you watched the chirp paint is nothing but a stack of the DFT columns you built in The DFT — each column is "does this frequency fit?" asked of one slice. And the chirp itself was an oscillator, the very thing from How Synths Make Sound, with its frequency ramped over time. Three lessons, one continuous thread: oscillator → spectrum → spectrum-over-time.
Forward, two ways. First, look at any single column of the spectrogram: it's a row of peaks. If instead of drawing the whole magnitude you just track the location and height of each peak, frame by frame, you get a compact list of "this many sinusoids, at these frequencies, these loudnesses, right now." That is the sinusoidal model — the next lesson — and it's how synthesizers analyze, re-pitch, and resynthesize real instruments.
Second, the destination. The spectrogram is the canvas an audio model speaks on. A Lyria-class music model doesn't emit a waveform directly — it generates a spectrogram (often a perceptually-warped mel-spectrogram), and a vocoder turns that picture back into sound via the inverse you met in Chapter 5. So the very image you just learned to read and paint is the surface where AI-generated music will be rendered cinematically across the Reverbs canvas. You now know how to look at exactly what the machine draws.
A sound is not a waveform or a spectrum — it is a spectrum that changes over time, and the STFT is the lens that lets you see it. Every choice you make is the window length, and every window length is a point on the line Δt·Δf = 1. There is no free lunch; there is only the right window for the question.
You didn't learn the spectrogram. You watched a sound paint one — and chose, with a slider, how it would see itself.
Serra et al., Audio Signal Processing for Music Applications — Module 4, STFT (Stanford / UPF); see the sms-tools reference code. · Smith, Spectral Audio Signal Processing — the STFT, windows, and overlap-add, rigorously. · Gabor, "Theory of Communication" (1946) — the time–frequency limit. · McAulay & Quatieri (1986) — the sinusoidal model the next lesson builds. · MDN, AnalyserNode.getByteFrequencyData — the live FFT this instrument taps.