Acoustic Scene Classification — Hearing the Place, Not the Events

Chapter 1: The Task & Its Famous Dataset

Let us make ASC precise. The input is an audio clip — typically 10 seconds, single channel, sampled at 44.1 kHz (so ~441,000 numbers). The output is a probability distribution over K place categories, and we read off the most probable. A function that eats a soundscape and emits one of K words. Structurally identical to the visual task; only the input medium changed.

The field has a center of gravity: the DCASE Challenge (Detection and Classification of Acoustic Scenes and Events), run by the IEEE audio community, whose "Task 1" has been the ASC benchmark since 2016. Its dataset is TAU Urban Acoustic Scenes — recordings made across a dozen European cities, in ten everyday places:

Look closely at those pairs and you can already feel the difficulty: metro vs metro_station, street_pedestrian vs street_traffic, public_square vs street_pedestrian. These are not random, well-separated places. They are near-neighbors that share most of their sounds — the audio analogue of "bedroom vs hotel_room" from the visual lesson. The taxonomy was designed to be hard.

There is a second, deeper twist baked into the dataset, and it has no clean equivalent in the visual world: the recordings were captured on multiple devices — a high-quality reference recorder plus several consumer phones, plus simulated devices. The same scene sounds different through a cheap phone mic than through a studio recorder, because every microphone has its own frequency response. A model that learns "metro" from the good recorder can fail on a phone. Hold that thought — Chapter 4 makes it the villain of the story.

As before, the model produces K logits (raw scores) and softmax turns them into probabilities. The arithmetic is identical to the visual case, so let us do it once, fast, with the numbers that will matter later.

Click a scene below to hear (in words and a sketched signature) what makes it distinctive — and which other scene it most easily blurs into. The pairs that sit close are the ones that will dominate the confusion matrix in Chapter 8.

Chapter 2: From Waveform to the "Image of Sound"

Here is the first real divergence from the visual task, and it is the most important idea in audio ML. A CNN wants a 2D grid of numbers — an image. But sound is a waveform: a 1D list of air-pressure samples over time, 44,100 of them per second, looking like a jagged squiggle. Feeding that raw squiggle to a classifier works poorly, because the information we care about — which frequencies are present, how loud, when — is hopelessly tangled in the raw samples. We need to untangle it.

The tool is the spectrogram. The idea: chop the waveform into short overlapping frames (say 25 ms each), and for each frame ask "how much energy is at each frequency?" using the Fourier transform. Stack the answers side by side and you get a 2D image: time on the horizontal axis, frequency on the vertical, brightness = energy. A steady train rumble becomes a bright horizontal band low down; a whistle becomes a bright line high up; broadband traffic hiss becomes a vertical smear across all frequencies. The soundscape becomes a picture.

Two refinements turn a raw spectrogram into the representation everyone actually uses. First, human hearing does not treat frequencies equally — we resolve low frequencies finely and high frequencies coarsely. So we warp the frequency axis onto the mel scale, which spaces bins the way the ear does, collapsing thousands of raw frequency bins into ~64–128 perceptually-spaced mel bins. Second, loudness is perceived logarithmically (a whisper to a shout spans a huge energy range), so we take the log of the energies. The result — the log-mel spectrogram — is the canonical input to an acoustic-scene CNN.

Let us see one frame's worth of the math, by hand, so the FFT is not a black box. A discrete Fourier transform asks, for a target frequency, "how much does the signal correlate with a wave at that frequency?" For a tiny 4-sample frame x = [1, 0, −1, 0] and the frequency that completes one cycle over 4 samples, the cosine reference is [1, 0, −1, 0]. The correlation (dot product) is 1·1 + 0·0 + (−1)·(−1) + 0·0 = 2 — a strong response, because our signal is that frequency. For the zero-frequency (DC) reference [1,1,1,1], the correlation is 1+0−1+0 = 0 — no constant offset. The FFT just does this for every frequency at once; the spectrogram stacks the magnitudes.

python
import numpy as np

# waveform: (N,) raw samples at sr Hz. Build a log-mel spectrogram from scratch.
def log_mel(waveform, sr=44100, n_fft=1024, hop=441, n_mels=64):
    frames = [waveform[i:i+n_fft] for i in range(0, len(waveform)-n_fft, hop)]
    spec   = np.array([np.abs(np.fft.rfft(f * np.hanning(n_fft)))**2 for f in frames])  # (T, n_fft/2+1) power
    mel_fb = mel_filterbank(sr, n_fft, n_mels)        # (n_mels, n_fft/2+1) triangular filters
    mel    = spec @ mel_fb.T                           # (T, n_mels) warp to perceptual bins
    return np.log(mel + 1e-6).T                     # (n_mels, T) the "image of sound" (+eps avoids log 0)

python
# The library version — torchaudio gives the whole thing in two lines
import torchaudio.transforms as T
melspec = T.MelSpectrogram(sample_rate=44100, n_fft=1024, hop_length=441, n_mels=64)
logmel  = (melspec(waveform) + 1e-6).log()        # (1, 64, T) — same thing we wrote by hand

The simulation below builds a log-mel spectrogram live. Add a low rumble, a tonal whistle at a frequency you choose, and broadband noise; watch each component paint its signature into the time-frequency image. This is the single most important picture in audio ML: everything downstream treats this as an image.

Chapter 3: Spectrogram as Image — the CNN Just Transfers

Now the payoff of Chapter 2. Once a clip is a (64, T) log-mel image, the entire visual scene-classification pipeline transfers, almost line for line. Run a 2D CNN over the spectrogram, collapse the result to a vector with global pooling, and classify with a linear layer and softmax. The network that recognizes kitchens from photos becomes, with the input swapped, a network that recognizes metros from sound.

What does a convolution "see" on a spectrogram? The same thing it sees on a photo: local 2D patterns. But now the patterns have acoustic meaning. A small vertical edge = a sudden broadband click (a transient). A horizontal bar = a sustained tone. A repeating texture = the rhythmic clatter of a train on rails. A diffuse low-frequency cloud = engine rumble. The CNN learns a library of time-frequency texture detectors, exactly as it learned visual texture detectors — the math does not know or care that one axis is now frequency and the other time.

The aggregation step is, once again, global average pooling (GAP): collapse each feature channel's whole (H×W) time-frequency map to a single number by averaging. For acoustic scenes this is exactly right for the same reason it was right for visual scenes: scene identity is a holistic, smeared-out property of the whole clip, not pinned to one moment or one frequency. A metro is a metro whether the loudest clatter happens at second 2 or second 8. Averaging over time and frequency keeps "how much of each texture is present overall" and discards "exactly when/where" — which is what we want.

python
import torch, torch.nn as nn

# A minimal acoustic-scene CNN: spectrogram (1,64,T) -> 10 scene logits
class ASCNet(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(), nn.MaxPool2d(2),
        )
        self.pool = nn.AdaptiveAvgPool2d(1)   # GAP over time AND frequency
        self.fc   = nn.Linear(64, n_classes)
    def forward(self, x):           # x: (B,1,64,T) log-mel
        z = self.features(x)         # (B,64,H,W) time-frequency feature maps
        z = self.pool(z).flatten(1)  # (B,64)  the scene descriptor
        return self.fc(z)             # (B,10) logits -> softmax -> scene

That AdaptiveAvgPool2d(1) is our GAP; the nn.Linear is the descriptor→logits map; it is the visual lesson's classifier with a spectrogram on the front. Paint patterns into the spectrogram below and watch them flow through GAP to logits to a class probability over three toy scenes — the whole pipeline in one picture.

Chapter 4: Why It's Hard — Ambiguity, and the Device Curse

Acoustic scene classification inherits all the difficulty of the visual task and then adds one of its own. Let us take the inherited forces first, fast, because you have met them, then spend real time on the audio-specific villain.

Inherited Force 1: label ambiguity. A clip recorded while walking through a busy plaza is genuinely both "public_square" and "street_pedestrian"; on a platform with a train arriving it is both "metro_station" and (almost) "metro". One label is chosen; the model picks one; honest near-ties get marked wrong. Force 2: intra-class variance. "Airport" spans a silent gate at dawn and a roaring security line at noon. Force 3: low inter-class margin. The TAU pairs (metro/metro_station, the two streets) sound more like each other than two random airports do. Same squeeze as vision: call dissimilar clips "same", split near-identical clips into "different".

The audio-specific villain: device mismatch. Here is the curse with no clean visual analogue. Every microphone colors sound differently — a cheap phone mic rolls off the bass and boosts a midrange peak; a studio recorder is flat. So the same acoustic scene produces a different spectrogram depending on the device. A model trained on the reference recorder learns "metro = this exact spectral shape", then meets a phone recording where the bass rumble — the key metro cue — has been attenuated by the mic, and it misfires. In DCASE, models are explicitly tested on devices unseen during training, and naive models can lose 10–20 accuracy points crossing that gap.

Why can't you just train on every device? Because you never have them all — new phone models appear constantly, and you must generalize to mics you have never seen. The fix is not "collect more devices" (you can't keep up) but "make the model not care about the device." The dominant tricks: Frequency-MixStyle (randomly mix the per-frequency statistics of different clips during training, so the model stops relying on any one device's spectral coloring) and Device Impulse Response augmentation (convolve clean clips with many simulated microphone responses, manufacturing fake devices to train on). Both teach the same lesson: "the scene is the content, not the coloring."

The widget makes the curse tangible. The top spectrogram is a metro recorded on the reference device. Slide "device change" and watch a different microphone's frequency response reshape the same scene — the low rumble fades, a midrange peak appears — until a model keyed to the original shape would misread it. Then toggle "device-invariant training" to see the model's decision stop following the device.

Chapter 5: The Whole Clip Decides — Temporal Context

Chapter 3 pooled over time almost as an aside. Let us make the principle explicit, because it is the audio echo of the visual lesson's "global context is the whole game." The claim: an acoustic scene is a property of the entire clip's statistics, and the central design question is how to summarize the whole soundscape, not any single instant.

Why is no single moment enough? Because any one 50-millisecond window is ambiguous. A snippet of footsteps could be a park path, a mall, or a platform. A single car-horn instant could be any street. The scene reveals itself only in the aggregate — the persistent rumble that runs under everything, the reverberation tail that says "enclosed vs open", the long-run balance of voices to engines to nature. You must integrate across the full ten seconds to read it.

This is the same question the visual lesson asked — "how do I let the whole input inform the decision?" — transposed to time. And the architectures answer it the same way. GAP is the simplest answer: average every moment equally. Attention is the smarter answer: let the model learn which moments matter. The strongest modern audio backbones — the Audio Spectrogram Transformer (AST) and PaSST — are Vision-Transformer-style models that cut the spectrogram into patches and let every patch attend to every other, so a quiet moment of background rumble can directly inform the verdict alongside a loud transient three seconds later. Global temporal-spectral context, on layer one — exactly the ViT story from the visual lesson, with frequency×time patches instead of image patches.

The widget shows why the whole clip matters. Slide a narrow listening window across the timeline: at most positions the local sound is ambiguous (the per-window guess flickers between scenes), but the running aggregate over everything seen so far converges to the right answer. Watch the local guess flail while the whole-clip verdict stabilizes.

Chapter 6: The Defining Twist — It Must Be Tiny

Here is where acoustic scene classification stops being "vision with a spectrogram" and becomes its own discipline. Visual scene models chase accuracy with billion-parameter giants. ASC, by contrast, lives under a brutal constraint, because of where it runs: always-on, battery-powered devices — hearing aids, earbuds, phones in standby, IoT sensors — that must classify the scene continuously without draining the battery or phoning home to a server.

The DCASE low-complexity track makes the constraint concrete and merciless: a model must fit within about 128 KB of parameters (think INT8 weights) and use no more than about 30 million multiply-accumulate operations (MACs) per inference — budgets modeled on a tiny Cortex-M class microcontroller. For comparison, the AST transformer from Chapter 5 has ~87 million parameters. You must deliver acoustic-scene recognition in roughly one seven-hundredth of that. This is the same world as the MCUNet microcontroller-vision story, transposed to audio.

How do you fit a good classifier in 128 KB? Three moves, each a real engineering decision:

1. Receptive-field-regularized CNNs. The winning small architectures — CP-ResNet, CP-Mobile, BC-ResNet — are not random tiny nets. They carefully control the receptive field so the limited parameters span just enough time-frequency context to read a scene without waste. CP stands for "receptive-field (RF) regularized"; BC-ResNet uses "broadcasting" to share 1D computations cheaply across the 2D map. The insight: with a tiny budget, where you spend the receptive field matters more than depth.

2. Quantization. Store and compute weights in 8-bit integers (INT8) instead of 32-bit floats, with quantization-aware training so accuracy barely drops. This is a 4× size cut almost for free — the difference between fitting in 128 KB and not.

3. Knowledge distillation — the secret weapon. Train a giant transformer teacher (PaSST, ~87M params) to high accuracy, then train the tiny student (CP-Mobile, ~60–130K params) to mimic the teacher's soft probability outputs, not just the hard labels. The teacher's "this is 70% metro, 25% metro_station, 5% tram" carries far more information than "metro" — it teaches the student the similarity structure between scenes. The result: a 128 KB student that punches massively above its weight because it inherited a transformer's judgment. This is the single most important trick in modern low-complexity ASC.

python
import torch.nn.functional as F

# Distill a tiny student from a big (frozen) teacher. T = temperature; a = blend weight.
def distillation_loss(student_logits, teacher_logits, labels, T=2.0, a=0.9):
    soft = F.kl_div(F.log_softmax(student_logits/T, 1),
                    F.softmax(teacher_logits/T, 1),
                    reduction="batchmean") * (T*T)   # mimic the teacher's soft distribution
    hard = F.cross_entropy(student_logits, labels)      # still respect the true label
    return a*soft + (1-a)*hard                       # mostly imitate teacher, anchored by truth

The widget plots accuracy against the parameter budget (log scale), with the 128 KB DCASE wall drawn in. Two curves: a small net trained from scratch, and the same net distilled from a PaSST teacher. Drag the budget and watch distillation lift the tiny-model accuracy — the gap is the teacher's gift. Notice that at the 128 KB wall, distillation is the difference between a usable and an unusable model.

Chapter 7: Showcase — Build & Break an Acoustic-Scene Classifier

Everything converges here. This is a working acoustic-scene classifier you can drive and break. The model holds five candidate scenes — metro, park, airport, street_traffic, shopping_mall — each defined by a learned weight over five interpretable soundscape features. You set the soundscape with the sliders; the model computes a descriptor, multiplies by its weights to get logits, and softmaxes to a probability over the five scenes. The bars are the live distribution; the panel shows top-1 and the full ranking.

The five features are the holistic, whole-clip cues Chapter 5 argued for: low-frequency rumble (engines, trains), crowd murmur (many distant voices), reverberation / echo (enclosed vs open space), traffic hiss (broadband road noise), and PA announcements (the periodic public-address presence of transit hubs). No single feature names a scene; the combination does.

Three experiments, each a chapter made live:

• Make a confident metro (high rumble, some reverb and PA, low traffic) — one bar dominates: the easy, separable case.
• Now build an "airport vs mall" tie (high murmur + reverb + PA, low rumble). Both are big, reverberant, crowded, PA-laden spaces — watch them converge to a near-tie, the ambiguity of Chapter 4 in motion.
• Slide "device mismatch" up. This warps the feature vector the way a different microphone would (rolling off the rumble, boosting midrange). A confident metro can flip to a wrong scene — you are watching the device curse break a model in real time, and you can switch on device-invariant training to defend it.

If you removed this simulation, would you lose understanding? Yes — "device mismatch degrades a confident prediction" is abstract until you slide one control and watch metro flip to bus. The napkin drawing of the whole lesson is exactly this: soundscape in, distribution out, ambiguity and the wrong microphone pulling it apart.

Chapter 8: Evaluation — Log-Loss & Cross-Device Honesty

You understand the models; here is how to judge them without being fooled. ASC evaluation has three pillars: the metric (DCASE leans on log-loss, not just accuracy), the confusion matrix, and — the one that separates lab from reality — cross-device testing.

Why log-loss, not just accuracy? Accuracy only asks "was the top guess right?" It ignores confidence. Log-loss (multiclass cross-entropy) asks "how much probability did you put on the truth?" and punishes confident wrong answers harshly. For a single clip with true class c, log-loss = −log(p_c), the negative log of the probability you assigned to the correct scene. A model that says "0.9 metro" when it is metro pays −log(0.9) = 0.105; one that says "0.1 metro" pays −log(0.1) = 2.303. Same right-or-wrong verdict at the top, wildly different log-loss — because confidence matters when a downstream device trusts the number.

python
import numpy as np

# probs: (N, K) predicted distributions; labels: (N,) true class indices
def log_loss(probs, labels, eps=1e-15):
    p_true = probs[np.arange(len(labels)), labels]   # prob assigned to the correct scene
    return -np.mean(np.log(np.clip(p_true, eps, 1)))  # clip avoids log(0) = -inf

def accuracy(probs, labels):
    return np.mean(probs.argmax(1) == labels)        # top-1 only — ignores confidence

The confusion matrix tells you which errors — and in ASC its shape is diagnostic. A healthy model's off-diagonal mass clusters on the acoustically-similar pairs: metro↔metro_station, the two streets, public_square↔street_pedestrian. That pattern means the model is fighting genuine ambiguity. Mass in absurd cells (park↔metro) means something is broken. Tap a cell below to read the true→predicted story; notice the bright blocks hugging the diagonal.

Cross-device testing is non-negotiable. The single most important number in an ASC report is accuracy on devices unseen during training. A model can score 75% averaged over training devices and 58% on an unseen phone. DCASE explicitly breaks results down per-device and reports the unseen-device gap. If a paper only quotes one average accuracy and never mentions devices, distrust it for deployment — you are almost certainly looking at an optimistic, single-device-flattered number.

One more honest gauge: calibration. Because log-loss rewards well-calibrated confidence, ASC systems are often temperature-scaled (divide logits by a learned constant) so their "0.8" really means 80%. A miscalibrated, over-confident model both scores worse on log-loss and misleads any downstream system — a hearing aid that switches modes on a confident-but-wrong scene call is a worse product than one that knows when it is unsure.

Chapter 9: Connections & Where It's Used

Acoustic scene classification is one node in a family of "read the whole signal, name the place" tasks. Seeing the family is how you turn this lesson into a map rather than a fact.

Its identical twin: visual scene classification. Everything here mirrors the visual scene-classification lesson — events vs scene, holistic context, GAP, CNN→transformer, top-1 ambiguity. The deep point is that "scene classification" is a modality-agnostic pattern: summarize a whole signal into a descriptor and commit to a place. Vision and audio are two instruments playing the same score. RGB-D and remote-sensing scene classification are two more siblings — new sensors, same task.

Where ASC actually ships. Always-on, on-device context: smart hearing aids that adapt to "restaurant vs street", earbuds switching noise-cancellation profiles, phones setting context without GPS, smart-home and surveillance sensors, and automotive cabins sensing the acoustic environment. The 128 KB constraint is not academic — it is why these run on a coin-cell-powered chip and never send your audio to a server.

Where to go next in the corpus. The DSP that builds the spectrogram is taught in depth in EE269: STFT and Cepstrum & MFCC; the front end itself in Audio Representations; the self-supervised cousin in Self-Supervised Speech; the edge/TinyML machinery (quantization, distillation, MCUNet) across TinyML: MCUNet and Knowledge Distillation; and the whole visual mirror in Visual Scene Classification.

Cheat sheet

"We hear a place before we look at it." — the working premise of every always-on device that must understand its world through a microphone alone.