Sound
transformations
Once you've modelled a sound — pulled it apart into frequencies and frames — you can do something a tape recorder never could: slow it down without dropping its pitch, raise its pitch without slowing it down, and melt one sound into another. The model is the clay.
A sound you've modelled is clay, not stone.
So far the Signal track has been about seeing. In How Synths Make Sound you built a wave from a recipe. In The DFT you did the reverse trip — wave back to recipe. Slice that trip into short windows and you get the Short-Time Fourier Transform: a stack of spectra, one per little frame of time. That stack is a model of the sound.
Here's the payoff. A raw recording is stone — to change its speed you have to physically play the tape faster, and everything moves together. But a model is clay. The pitches live in one place (the frequencies inside each frame) and the timing lives in another (where the frames sit on the clock). Separate parameters. And separate parameters can be changed separately.
That single observation is all of Module 8. Filter the spectrum to sculpt tone; stretch the frame timeline to slow the sound while every note keeps its pitch; scale the frequencies to pitch-shift while the timing holds; interpolate two sounds' recipes to morph between them. Four transformations, one idea: edit the model, then resynthesize.
Why naive resampling fatally couples pitch and speed; how the model decouples them; the phase-vocoder idea — advancing the synthesis hop independently of the analysis hop, with the phase-advance correction — derived and worked by hand for a 1.5× stretch; pitch-shifting two ways; spectral-envelope morphing; the code in JavaScript and Python; and a playable instrument where you hear time and pitch come apart in your hands.
Why "just slow the tape down" ruins the pitch.
The obvious way to make a sound last longer is to play its samples more slowly — the chipmunk-in-reverse trick, called resampling. You take the samples x[n] and read them out at a different rate. To make the sound play α times longer, you read α times slower.
It works — and it wrecks the pitch. Reading slower stretches every cycle of every sine in the sound, so each frequency drops. Reading the samples at rate 1/α turns a tone at frequency f into a tone at f/α. Speed and pitch are welded together:
Why are they welded? Because a raw recording stores only x[n] — air pressure over time. The number "how fast does this wobble?" is not written down anywhere; it is implied by how quickly you walk through the samples. Change the walking speed and you've changed the only thing that encoded pitch. There is no knob for "duration" separate from "pitch" because the storage format never separated them.
To pull them apart you need a representation where pitch and time are different numbers. The STFT model is exactly that: each frame holds the frequencies present right now, and the index of the frame holds when. Now "slow down" can mean "put the frames farther apart" — leaving the frequencies inside each frame untouched.
Resampling is the only tool a raw waveform offers, and it changes pitch and duration together, by the same factor, always. Every honest time-stretch or pitch-shift must first step up into a model where the two live apart.
Time-scaling = move the frames, keep the spectra.
Picture the STFT as a filmstrip. You chopped the sound into overlapping windows and stepped forward by a fixed amount each time — the analysis hop Ha (say, every 256 samples). Each frame got its own little spectrum. To put the sound back together you overlap-add the frames, stepping by a synthesis hop Hs. If Hs = Ha, you reconstruct the original.
Now the trick. To stretch time by a factor α, leave the analysis filmstrip exactly as it is, but lay the frames back down farther apart — advance the synthesis hop by α times as much:
The frequencies inside each frame never changed — so every note keeps its pitch. Only the spacing between frames changed — so the sound lasts α times longer. Pitch and duration, finally on separate knobs. This is the overlap-add family of time-scaling, and at its best it's the phase vocoder.
But there's a subtlety the filmstrip picture hides, and it's the heart of the phase vocoder. Each frame doesn't just have a magnitude per frequency bin — it has a phase, the angle of that sinusoid at the start of the frame. If you re-space the frames but copy the phases verbatim, the sinusoids no longer line up where the frames meet. They fight at the seams: you hear a metallic, smeared "phasiness."
The fix is to advance each bin's phase by exactly the amount it would have rotated over the new, longer hop. A bin at angular frequency ω rotates ω·H radians per hop of H samples. So between the input phase φin and the output phase φout you carry forward:
And ωk isn't just the bin's nominal frequency — it's the instantaneous frequency, recovered from how much the phase actually changed between input frames, divided by the analysis hop. That phase-difference-over-hop estimate is the famous phase-vocoder step:
Read it as a story: measure how fast each frequency is really turning (from the input frames spaced Ha apart), then keep it turning at that same rate as you lay frames down spaced Hs apart. Same pitches, new clock.
Mathematician: time-scaling is a reparametrization of the synthesis grid, Hs = αHa, made phase-coherent by integrating instantaneous frequency, φout += ω·Hs. Musician: it's hitting "half speed" on a loop while every note stays exactly in tune — the producer's holy grail. Artist: it's a spectrogram you grab by the edges and pull wider; the bright ridges hold their height while the canvas stretches sideways.
A 1.5× time-stretch, frame by frame.
Numbers make it real. Take an analysis hop of Ha = 256 samples and ask for a stretch of α = 1.5 (one-and-a-half times longer). The synthesis hop is the single multiplication from Chapter 2:
Now line up the two timelines. The analysis frames are read off the input every 256 samples; the synthesis frames — the very same spectra — are written to the output every 384 samples. Frame index i sits at input sample i·256 and output sample i·384:
256 → written at output 384. Already 128 samples farther along.512 → written at output 768. The same spectrum, just laid down later.768 → written at output 1152. Each step opens a wider gap.2560 → written at output 3840. The output reached 3840 where the input was only at 2560: output length = input × 1.5. Exactly the stretch we asked for.Read the table top to bottom and you can feel the mechanism: identical spectra (so identical pitches), placed on a clock that ticks 1.5× slower. Nothing about the frequencies inside any frame was touched. The only edit was the spacing.
And the phase correction from Chapter 2 rides along here too. Over the synthesis hop of 384 samples, a bin whose true angular frequency is ω must advance its phase by ω·384 radians before that frame is written — not the ω·256 it advanced in the input. Skip that and the seams between frames buzz; include it and the stretched sound is clean.
Raise the pitch, keep the clock.
Pitch-shifting is the mirror image, and there are two honest routes to it — both built from tools you already have.
Route 1 — stretch, then resample. This is the clever bank-shot. Time-stretch the sound by the factor you want to shift by, then resample to undo the duration change. Watch the bookkeeping: to shift up by a ratio β (e.g. β = 2 for an octave up), first time-stretch by α = β — same pitch, β times longer — then resample by reading β times faster. Resampling, from Chapter 1, multiplies both pitch and duration by β: the duration goes back to original (β · 1/β = 1) and the pitch rides up by β. Time-stretch chooses the pitch; resampling cancels the length.
Route 2 — scale the frequencies in the model directly. If your model is harmonic — a sum of sinusoids, like the additive synth you built — pitch-shifting is even more direct. Multiply every sinusoid's frequency by β and leave the frame timeline alone. A semitone shift of s is the ratio you met in the synth lesson:
Octave up is s = +12, so 212/12 = 2 — every frequency doubled. Octave down is s = −12, ratio ½. A perfect fifth up is s = +7, ratio 27/12 ≈ 1.498. The interactive instrument below uses this second route, because it's the most direct demonstration that frequency and timeline are separate dials.
One honest caveat the literature never hides: naively scaling all frequencies also slides the formants — the resonant peaks that make a voice sound like a voice. Shift a voice up an octave this way and it chipmunks, because the formants moved too. Production-grade pitch-shifters keep the spectral envelope fixed while moving the partials underneath it — which is exactly the envelope idea we need for morphing, next.
Melt one sound into another.
The fourth transformation needs two models. Morphing is interpolating between them — and the right thing to interpolate is the spectral envelope: the smooth curve that rides over the top of a spectrum, the shape that gives a sound its character (its formants, its brightness).
Recall additive synthesis: a sound was a list of harmonic amplitudes An. Give sound X amplitudes AnX and sound Y amplitudes AnY. A morph with knob γ from 0 (all X) to 1 (all Y) is just a weighted blend, harmonic by harmonic:
AX = [1.00, 0.20, 0.05] (strong fundamental, weak partials).AY = [1.00, 0.90, 0.70] (rich upper harmonics).γ = 0.5, harmonic 2 = 0.5·0.20 + 0.5·0.90 = 0.55; harmonic 3 = 0.5·0.05 + 0.5·0.70 = 0.375.[1.00, 0.55, 0.375] — measurably between soft and bright. Sweep γ 0→1 and the timbre slides continuously from X to Y.For richer sounds you separate the morph into pieces that should be blended differently: interpolate the spectral envelope (the formant shape) on its own, interpolate the pitch on its own, and optionally cross-fade the noisy residual. Morph the envelopes and you get a true hybrid voice; morph only amplitudes and you get a convincing crossfade. Either way, morphing is arithmetic on models — impossible on raw waveforms, trivial once you've analyzed.
Filter, time-stretch, pitch-shift, morph — every one is edit the model, then resynthesize. Filtering scales the magnitudes per bin. Time-stretch re-spaces the frames. Pitch-shift scales the frequencies. Morph blends two recipes. The analysis (DFT/STFT) earned you a representation where each musical attribute is its own number — and separable numbers are editable numbers.
Stretch and shift, in code.
Here is the granular skeleton the instrument below uses — the honest core of time-stretch and pitch-shift made tiny. A phrase is a list of grains (frame index → frequency). Time-stretch multiplies every onset by α; pitch-shift multiplies every frequency by 2s/12. The two loops touch different fields, so they cannot interfere — that's the decoupling, in syntax:
// the source model: a phrase as grains (onset in "beats", a frequency) const Ha = 0.16; // analysis hop (s per beat) const grains = [220, 261.6, 329.6, 440].map((f, i) => ({ i, f })); function render(alpha, semis) { // alpha = time-stretch, semis = pitch-shift const Hs = alpha * Ha; // SYNTHESIS hop = α·Ha (Ch.2) const ratio = 2 ** (semis / 12); // pitch ratio 2^(s/12) (Ch.4) for (const g of grains) { const onset = g.i * Hs; // ← TIME axis: onsets stretch const freq = g.f * ratio; // ← PITCH axis: frequency scales scheduleGrain(onset, freq); // independent dials, never crossing } }
import numpy as np def time_stretch(x, alpha, n_fft=1024, Ha=256): Hs = int(round(alpha * Ha)) # synthesis hop = α·Ha (Ch.3) win = np.hanning(n_fft) # analysis: one DFT per analysis-hop frame frames = [np.fft.rfft(win * x[i:i+n_fft]) for i in range(0, len(x) - n_fft, Ha)] omega = 2 * np.pi * np.arange(frames[0].size) / n_fft phase = np.angle(frames[0]) # running SYNTHESIS phase out = np.zeros(int(len(x) * alpha) + n_fft) prev = np.angle(frames[0]) for j, F in enumerate(frames): # true bin frequency from wrapped phase diff / Ha (Ch.2) dphi = np.angle(F) - prev - omega * Ha dphi = (dphi + np.pi) % (2*np.pi) - np.pi # wrap to ±π w_true = omega + dphi / Ha phase = phase + w_true * Hs # advance over SYNTHESIS hop grain = np.fft.irfft(np.abs(F) * np.exp(1j * phase)) out[j*Hs : j*Hs + n_fft] += win * grain # OVERLAP-ADD prev = np.angle(F) return out # pitch-shift = time_stretch by β, then resample by β (Ch.4, route 1) def pitch_shift(x, semis): beta = 2 ** (semis / 12) return np.interp(np.arange(0, len(x), beta), np.arange(len(x)), time_stretch(x, beta))
The JavaScript is the granular approximation — clear enough to hear the decoupling in one screen. The Python is the phase vocoder — the same Hs = α·Ha and overlap-add, plus the phase-advance bookkeeping that keeps the cycles continuous. Same idea, two fidelities. The instrument below runs the granular version live, in your browser.
One phrase. Two independent dials.
Here is the whole lesson as something you play. The instrument synthesizes a short arpeggio entirely in your browser — no recording — as a sequence of grains, the model from Chapter 2. Two knobs:
Time-stretch · α
Moves the grains apart or together — Hs = α·Ha. The phrase lasts longer or shorter. Listen: the pitches never move. Watch the spectrogram stretch sideways while its bright rows hold their height.
Pitch-shift · s
Multiplies every grain's frequency by 2s/12 — the timeline is untouched. The melody climbs or dives. Listen: the phrase takes exactly as long as before. The spectrogram rows slide up or down; the columns stay put.
Set the stretch to 2× and the pitch to 0 — a slow-motion melody, dead in tune. Then set the stretch back to 1× and the pitch to +12 — same tempo, an octave up. That impossibility on a tape recorder is one tweak each here, because the model put time and pitch on different axes.
Every earlier lesson is loaded into this one gesture. The sine grains are the pure tones from How Synths Make Sound. The 2s/12 pitch ratio is the equal-temperament step from that same lesson. The per-grain DFT painting the spectrogram is the exact two-loop machine from The DFT, sliced into short frames — the STFT. Module 8 is what you do with the model the rest of the track built.
To transform a sound you first had to analyze it into separable numbers. To search a million sounds, cluster them, or feed them to a model, you need to go one step further and describe each sound as a compact vector of numbers — its brightness, its harmonicity, its loudness contour. Those are audio descriptors / features, the next stop, and the doorway from signal processing into machine listening — and ultimately the Lyria-class generative finale this project is built toward.
A recording is stone. A model is clay — and now you can pull time and pitch apart with your hands.
Serra et al., Audio Signal Processing for Music Applications — Module 8, Sound Transformations (Stanford / UPF); see also the open sms-tools for the analysis/transformation/synthesis pipeline. · Dolson, "The Phase Vocoder: A Tutorial," Computer Music Journal, 1986 — the phase-advance relation, derived. · Laroche & Dolson, "Improved Phase Vocoder Time-Scale Modification of Audio," IEEE TSAP, 1999. · Smith, Spectral Audio Signal Processing (CCRMA) — STFT, overlap-add, and resynthesis. · Roads, Microsound — granular synthesis, the model behind this demo.