Spectral Descriptors

Chapter 1: Spectral Centroid

The spectral centroid is the "center of mass" of the spectrum. It tells you where the average frequency is. High centroid = bright sound. Low centroid = dark sound.

Definition

Since we already normalized s_k to sum to 1, this simplifies to:

where f_k = k · f_s/N is the frequency (in Hz) of bin k. This is literally the expected value of frequency under the spectral distribution.

Worked Example

Suppose N = 8, f_s = 8000 Hz. The frequency bins are at f_k = k · 1000 Hz for k = 0, 1, 2, 3, 4.

Magnitude spectrum (power): |X|² = [0, 4, 1, 0, 0] (energy concentrated at 1000 Hz with a bit at 2000 Hz).

Total power = 5. Normalized: s = [0, 0.8, 0.2, 0, 0].

The centroid is 1200 Hz — pulled slightly above the fundamental by the harmonic at 2000 Hz. If we add more high-frequency harmonics, the centroid rises (brighter sound).

Centroid in Code

python
import numpy as np

def spectral_centroid(magnitude_spectrum, fs, N):
    """Compute spectral centroid in Hz."""
    K = len(magnitude_spectrum)
    freqs = np.arange(K) * fs / N   # frequency of each bin
    power = magnitude_spectrum ** 2
    total = power.sum()
    if total == 0:
        return 0.0
    return (freqs * power).sum() / total

Notice: we use the power spectrum (|X[k]|²) not the magnitude (|X[k]|). Some implementations use magnitude directly — the difference is whether loud frequencies are weighted linearly or quadratically. Power weighting (squared) emphasizes dominant components more heavily.

Centroid Over Time

In practice, we compute the centroid for each short-time frame (10–30 ms of audio). The sequence of centroids over time is a time series that tracks how brightness changes. When a drummer hits a hi-hat, the centroid spikes (sudden high-frequency energy). During a bass note, it drops.

Perceptual Correlation

Psychoacoustic studies show that spectral centroid correlates with perceived "brightness" (r > 0.8 in most experiments). This makes it one of the most reliable audio features for perceptual tasks. In music information retrieval, centroid alone can distinguish:

• Classical piano (centroid ~1000–2000 Hz) from rock guitar (centroid ~3000–5000 Hz)

• Male speech (centroid ~800 Hz) from female speech (centroid ~1200 Hz)

• Muffled recordings (low centroid) from bright/tinny recordings (high centroid)

Chapter 2: Spectral Spread & Bandwidth

The centroid tells you WHERE the spectral energy is centered. The spectral spread tells you how WIDE the distribution is around that center. A narrow spread means a pure, tonal sound. A wide spread means a noisy, broadband sound.

Definition

This is the standard deviation of the spectral distribution. Units are Hz. It measures how far, on average, the spectral energy sits from the centroid.

Worked Example (continued)

From Chapter 1: s = [0, 0.8, 0.2, 0, 0], μ₁ = 1200 Hz, f = [0, 1000, 2000, 3000, 4000] Hz.

The spread is 400 Hz. Compare to a pure tone at 1000 Hz which would have s = [0, 1, 0, 0, 0] and spread = 0. Our signal has some energy at 2000 Hz, giving it a nonzero spread.

Interpretation Table

Bandwidth vs. Spread

Some literature uses bandwidth instead of spread. They're related but not identical. The –3dB bandwidth measures where the spectrum drops by half power. The spectral spread is the standard deviation of the full distribution. For a Gaussian-shaped spectrum, bandwidth ≈ 2.35 · μ₂. For non-Gaussian spectra, they can differ significantly.

Spread Computation in Code

python
def spectral_spread(magnitude_spectrum, fs, N):
    K = len(magnitude_spectrum)
    freqs = np.arange(K) * fs / N
    power = magnitude_spectrum ** 2
    total = power.sum()
    if total == 0:
        return 0.0
    centroid = (freqs * power).sum() / total
    spread_sq = ((freqs - centroid) ** 2 * power).sum() / total
    return np.sqrt(spread_sq)

Why Spread Matters for Classification

Centroid alone can't distinguish all sound types. A narrow-band signal at 2000 Hz (a whistle) and a broad-band signal centered at 2000 Hz (a snare drum) have the same centroid but completely different timbres. Spread captures this difference: the whistle has near-zero spread, the snare has spread > 1000 Hz.

In a feature space with centroid on one axis and spread on the other, different instrument families form distinct clusters. This is why both features together are far more discriminative than either alone.

Chapter 3: Higher-Order Moments

The centroid and spread are the first two moments. But spectra can have complex shapes that two numbers can't capture. The third and fourth moments — skewness and kurtosis — describe asymmetry and peakedness.

Spectral Skewness (μ₃)

This measures the asymmetry of the spectral shape around the centroid:

• μ₃ > 0: More energy above the centroid (right-skewed, "tail towards high frequencies")

• μ₃ < 0: More energy below the centroid (left-skewed, "tail towards low frequencies")

• μ₃ = 0: Symmetric spectrum

A guitar note with strong low harmonics and weak high harmonics has negative skewness. A signal that drops off sharply below the centroid but has a long high-frequency tail has positive skewness.

Spectral Kurtosis (μ₄)

This measures the peakedness (or "tailedness") of the spectral distribution:

• μ₄ > 3: Leptokurtic (sharper peak than Gaussian, heavier tails). A pure tone has very high kurtosis — all energy concentrated at one point.

• μ₄ = 3: Mesokurtic (Gaussian-shaped spectrum).

• μ₄ < 3: Platykurtic (flatter than Gaussian, lighter tails). Uniform/white noise has kurtosis close to 1.8.

Worked Example: Computing All Four Moments

Let's compute moments for a simple 5-bin spectrum: s = [0.1, 0.2, 0.4, 0.2, 0.1] at frequencies f = [1000, 2000, 3000, 4000, 5000] Hz.

Centroid: μ₁ = 1000(0.1) + 2000(0.2) + 3000(0.4) + 4000(0.2) + 5000(0.1) = 100 + 400 + 1200 + 800 + 500 = 3000 Hz

Spread: μ₂² = (1000−3000)²(0.1) + (2000−3000)²(0.2) + 0 + (4000−3000)²(0.2) + (5000−3000)²(0.1) = 400000 + 200000 + 0 + 200000 + 400000 = 1,200,000. μ₂ = 1095 Hz.

Skewness: All terms are symmetric around the centroid (the spectrum is symmetric), so μ₃ = 0.

Kurtosis: μ₄ = ∑((f_k−3000)/1095)⁴·s_k = (1.83)⁴(0.1) + (0.91)⁴(0.2) + 0 + (0.91)⁴(0.2) + (1.83)⁴(0.1) = 11.2(0.1) + 0.69(0.2) + 0 + 0.69(0.2) + 11.2(0.1) = 1.12 + 0.14 + 0.14 + 1.12 = 2.52

Kurtosis = 2.52 < 3 → platykurtic (flatter than Gaussian). This makes sense: our symmetric 5-bin spectrum is more "uniform-like" than a peaked Gaussian.

Summary of Moments

Chapter 4: Entropy, Flatness & Crest

Beyond moments, we can compute information-theoretic and ratio-based descriptors that capture different aspects of spectral "shape."

Spectral Entropy

This is Shannon entropy of the spectral distribution. It measures how "spread out" or "uncertain" the frequency content is:

• Low entropy: Energy concentrated at few bins (tonal, predictable). A pure tone has H ≈ 0 (all mass at one bin).

• High entropy: Energy spread evenly across all bins (noise-like, unpredictable). White noise has H = log₂(N/2) (maximum possible).

Spectral Flatness

Spectral flatness is the ratio of geometric to arithmetic mean of the power spectrum. By the AM-GM inequality, SF ∈ [0, 1]:

• SF = 1: perfectly flat spectrum (white noise)

• SF → 0: highly peaked spectrum (pure tone)

Also called Wiener entropy or tonality coefficient. It distinguishes tonal from noise-like signals without computing pitch.

Spectral Crest

The ratio of peak to average. High crest = one dominant frequency (tonal). Low crest = no single peak dominates (noise-like). This is the spectral analog of the crest factor in the time domain (peak/RMS).

Spectral Entropy in Detail

Shannon entropy H measures the "surprise" or "unpredictability" of a distribution. Applied to the spectral distribution, it answers: "If I pick a random frequency weighted by the spectrum, how surprised am I by the result?"

For a pure tone (all energy at one bin): H = −1·log₂(1) = 0 bits. No surprise — you always know which frequency it will be.

For white noise (K equal bins): H = −K·(1/K)·log₂(1/K) = log₂(K) bits. Maximum surprise — every frequency is equally likely.

For 256 frequency bins: maximum entropy = log₂(256) = 8 bits. A speech frame typically has entropy around 5–6 bits. Background noise: 7+ bits (nearly maximum).

Voice Activity Detection via Entropy

A simple but effective VAD algorithm:

This works because speech concentrates energy in formant regions (3-5 peaks in the spectrum, moderate entropy), while noise distributes energy uniformly (high entropy). The threshold T is calibrated to the specific noise environment.

Worked Example: Entropy of Two Spectra

Spectrum A (tonal): s = [0.9, 0.05, 0.03, 0.02] (4 bins)

H = −(0.9 log₂ 0.9 + 0.05 log₂ 0.05 + 0.03 log₂ 0.03 + 0.02 log₂ 0.02)

H = −(−0.137 − 0.216 − 0.152 − 0.113) = 0.618 bits

Maximum possible = log₂(4) = 2 bits. So normalized entropy = 0.618/2 = 0.31 (very tonal).

Spectrum B (noisy): s = [0.27, 0.23, 0.25, 0.25]

H = −(0.27 log₂ 0.27 + 0.23 log₂ 0.23 + 0.25 log₂ 0.25 + 0.25 log₂ 0.25)

H ≈ 1.99 bits. Normalized = 0.99 (nearly maximum = noise-like).

The AM-GM Inequality and Flatness

Why is spectral flatness always ≤ 1? This follows from the arithmetic-geometric mean inequality: for non-negative numbers, the geometric mean never exceeds the arithmetic mean. Equality holds if and only if all numbers are equal — a flat spectrum.

So SF = GM/AM ≤ 1, with equality iff the spectrum is perfectly flat. In practice, implementations add a small floor (10⁻¹⁰) to all bins to avoid GM = 0 from a single zero bin.

Comparison

Chapter 5: Spectral Flux & Slope

All the descriptors so far describe a single frame. But many audio features are about change over time. Is the spectrum stable (sustained note) or rapidly evolving (speech, drum hits)?

Spectral Flux

Spectral flux measures how much the spectrum changes between consecutive frames. It's the squared Euclidean distance between successive magnitude spectra.

• Low flux: Sustained note, steady noise — spectrum doesn't change.

• High flux: Onset of a note, percussive hit, speech transitions — spectrum changes rapidly.

Half-Wave Rectified Flux

In practice, we often care only about increases in spectral energy (onsets), not decreases (offsets). The half-wave rectified flux is:

This ignores energy that disappears and only responds to energy that appears. Much more robust for onset detection.

Spectral Slope

This is the linear regression slope of the spectrum (magnitude vs. frequency). It describes the overall tilt of the spectrum:

• Negative slope: energy decreases with frequency (typical of most natural sounds — "pink" or "red" spectra)

• Near-zero slope: flat spectrum (white noise)

• Positive slope: energy increases with frequency (unusual, high-frequency dominated)

Most speech and music have negative spectral slope (−3 to −6 dB/octave). The steepness distinguishes vowels (gentle slope) from fricatives (steeper slope).

Spectral Rolloff

A related descriptor: the spectral rolloff is the frequency below which a certain percentage (typically 85% or 95%) of the total spectral energy lies:

High rolloff → energy extends to high frequencies (bright). Low rolloff → energy concentrated at low frequencies (dark). Similar to centroid but less sensitive to outlier high-frequency peaks.

Flux Normalization

Raw flux depends on signal loudness — a louder signal has larger magnitude changes. To compare flux across segments of different volume, normalize:

where ||X_t|| = √(∑|X_t[k]|²). This gives volume-independent flux that responds to spectral shape changes, not loudness changes.

Spectral Slope in Practice

The slope computation is ordinary least-squares regression of log-magnitude vs. log-frequency (for dB/octave measurement) or magnitude vs. frequency (for a simpler linear measure). Pilanci uses the linear form:

python
def spectral_slope(magnitude_spectrum, fs, N):
    K = len(magnitude_spectrum)
    freqs = np.arange(K) * fs / N
    f_mean = freqs.mean()
    s_mean = magnitude_spectrum.mean()
    num = ((freqs - f_mean) * (magnitude_spectrum - s_mean)).sum()
    den = ((freqs - f_mean) ** 2).sum()
    return num / den if den > 0 else 0

A typical speech frame has slope ≈ −30 dB/decade (energy drops 30 dB per factor-10 frequency increase). Music varies more: bass-heavy genres have steeper negative slopes; bright electronic music has flatter slopes.

Pilanci's Hi-Hat Example

In Pilanci's slides, he shows the spectral centroid time series for a drum loop. When the hi-hat hits:

• Centroid jumps from ~1000 Hz (kick/snare region) to ~8000 Hz (hi-hat region)

• Duration of the jump: ~50 ms (the hi-hat's sustain)

• Spread simultaneously increases (hi-hat is broadband)

• Flatness spikes (hi-hat is noise-like, not tonal)

This pattern — simultaneous centroid spike + spread increase + flatness increase — is so characteristic that a simple threshold detector on these three features can locate every hi-hat hit with >95% accuracy, even in a complex mix.

Contrast with a snare hit:

• Centroid jumps to ~3000–4000 Hz (lower than hi-hat)

• Spread is large but less than hi-hat

• Has both tonal (snare wire resonance) and noise (shell vibration) components

• Flatness is moderate (between pure tone and pure noise)

Chapter 6: Feature Extraction Pipeline

Now let's put it all together. In a real audio analysis system, you compute ALL spectral descriptors for each frame, producing a feature vector that characterizes the sound.

The Pipeline

Real-World Pipeline Parameters

Typical values in audio analysis systems:

The choice of window function (Hann, Hamming, Blackman) affects spectral leakage: energy from one true frequency "leaks" into adjacent bins. Longer windows give better frequency resolution but worse time resolution. This is the time-frequency uncertainty principle — you cannot have arbitrarily precise measurements in both time and frequency simultaneously.

Showcase: Draw Your Spectrum

Draw any spectrum shape on the canvas below. All descriptors are computed in real time. Experiment: make it narrow (low spread), shift it left/right (change centroid), add a second peak (increase kurtosis).

Chapter 7: Mastery

Spectral descriptors reduce high-dimensional spectra to interpretable features. They're the bridge between raw Fourier coefficients and machine learning classifiers.

Complete Reference

Connections

• Lecture 6 (DFT): Spectral descriptors require the DFT as input. No DFT, no spectral features.

• MFCCs (coming): Mel-frequency cepstral coefficients are another spectrum summarization, using a perceptual frequency scale and cepstral analysis. They capture formant structure better than raw moments.

• ML connection: Modern audio classifiers (speech recognition, music tagging) often use learned features (spectrograms + CNNs), but spectral descriptors remain useful as hand-crafted baselines and for interpretable systems.

Feature Selection: Which Descriptors for Which Task?

Beyond Hand-Crafted Features

Modern deep learning often bypasses hand-crafted descriptors entirely, feeding raw spectrograms (time-frequency images) into CNNs or transformers. However:

• Spectral descriptors remain useful as baselines — any learned system should beat a simple SVM on these features

• They provide interpretability — a centroid time series is human-readable, a CNN embedding is not

• For low-resource tasks (limited training data), hand-crafted features often outperform learned features

• They're computationally cheap — a microcontroller can compute centroid in real time

• They serve as sanity checks — if a learned model disagrees with simple descriptors, investigate why

The ideal system often combines both: spectral descriptors as an input layer alongside raw spectrogram features, letting the model learn when each is useful.

Delta Features

In practice, temporal derivatives of spectral features are often as informative as the features themselves. The delta (first derivative) and delta-delta (second derivative) features capture rate-of-change:

If you extract 9 base descriptors per frame, adding deltas and delta-deltas gives 27 features per frame. Speech recognition systems (before deep learning) typically used 13 MFCCs + 13 delta + 13 delta-delta = 39 features per frame.

Delta-centroid captures whether the sound is getting brighter (positive) or darker (negative). Delta-flux captures whether onsets are accelerating or decelerating. These temporal dynamics are crucial for distinguishing sounds that have similar static spectra but different temporal envelopes (e.g., a plucked vs. bowed string).

Descriptor Correlations

Some descriptors are highly correlated (measuring similar things differently):

• Entropy and flatness: both measure "noisiness" — correlation r > 0.9

• Centroid and rolloff: both measure "brightness" — correlation r > 0.85

• Crest and 1/flatness: nearly inverse of each other

In practice, using all descriptors together introduces multicollinearity in linear models. Options:

• PCA on the feature matrix (decorrelate, reduce dimensions)

• Select a subset (centroid + spread + flatness + flux covers most variance)

• Use non-linear models (random forests, neural nets) that handle correlation naturally

Pilanci recommends starting with the minimal set {centroid, spread, flatness, flux} and adding descriptors only if classification accuracy demands it. These four capture brightness, bandwidth, tonalness, and temporal change — the four perceptual axes most humans can discriminate.