Vision Language Models Are
Few-Shot Audio Spectrogram Classifiers

Chapter 1: The Key Insight

The whole method fits on a sticky note: recast audio classification as image classification, then solve image classification with in-context learning on a VLM. The authors name the new task Visual Spectrogram Classification (VSC) — classifying a sound purely from a picture of its spectrogram.

The pipeline in one breath

Audio clip → STFT → render as a labeled spectrogram image (PNG) → build a prompt that interleaves example images with their class names → append the test image → the VLM emits one class name. Everything downstream of "render as image" is just prompting.

What is actually "learned" here?

Nothing, in the weight sense. This is the part worth slowing down on. In ordinary supervised learning you adjust millions of parameters on the training set. Here the model is frozen. The "learning" happens entirely inside one forward pass: the attention mechanism reads the labeled examples in the context window and uses them to interpret the query. The paper's two engineering levers — how you draw the picture and which examples you show — are the only things you control. We will pull both levers interactively.

Chapter 2: Sound → Picture — the STFT

Before any VLM touches anything, we must turn a waveform into a spectrogram image. This is the one piece of classical signal processing the method rests on, so let's build it from the ground up.

A waveform is a stream of air-pressure numbers — at 22,050 Hz (the paper's resample rate) that's 22,050 numbers per second. A 5-second clip is a vector of 110,250 samples. Useless as a picture: it's a 1-D squiggle. The Short-Time Fourier Transform (STFT) turns that squiggle into a 2-D image of which frequencies are present, when.

The recipe, step by step

The STFT chops the signal into short overlapping windows and asks each window "what frequencies are you made of?" via the Fourier transform. The paper's defaults:

• Window size = 2048 samples. Each frame covers 2048/22050 ≈ 93 ms of audio. Long enough to resolve frequency, short enough that the sound is roughly stationary within it.
• Hop length = 512 samples. We slide the window forward by 512 each step, so frames overlap 75%. The number of time columns is ≈ 110250 / 512 ≈ 215.
• FFT per window → 1025 frequency bins (= 2048/2 + 1). Take the magnitude.

The output is a matrix of shape [1025 freq bins × 215 time frames]. Color each cell by its magnitude and you have an image. That image is the only thing the VLM ever sees.

Drag the window size. A bigger window resolves frequency sharply (thin horizontal bands) but smears time; a small window resolves time but blurs frequency into a vertical haze. This is the time–frequency tradeoff: you cannot have both at once. The 2048 default is the chosen compromise.

Chapter 3: The VSC Task

Now we have an image. The task is to assign it one of 10 labels. The paper studies two regimes that differ only in what you put in the prompt.

Zero-shot

Show the VLM one spectrogram (the test clip) plus a text prompt that lists the 10 class names and instructs "pick the most likely one; respond with the exact class name only." No examples. The model must rely entirely on whatever it absorbed about spectrograms during pre-training. GPT-4o scores 27.5% here on fold 1 — well above the 10% floor, proof that some spectrogram knowledge is latent in the model.

Few-shot (in-context learning)

Now precede the test image with one labeled example spectrogram per class — a 10-shot prompt. The model can now compare the query against ten reference patterns it can see right there in its context. GPT-4o jumps to 70% with random examples, and higher with good ones. Same frozen model; the only change is the contents of the prompt.

The data flow, with shapes

Notice the model never outputs probabilities or logits over classes — it emits free text, which is then matched to one of the 10 class strings. Parsing that text (and handling refusals or off-list answers) is a quiet part of the engineering.

Toggle the two prompt layouts. Zero-shot is just the query image + a text list. Few-shot stacks ten labeled reference images before the query, so the model can pattern-match against same-style examples in one forward pass.

Chapter 4: The Few-Shot Prompt

The prompt is the entire program. There are no weights to set — only words and images. Here is the zero-shot prompt the paper used, lightly paraphrased, so you can see exactly how literal it is:

# Zero-shot: system + user message to the VLM API
system = "You are a helpful assistant with expertise in recognizing
          patterns and identifying classes based on visual
          representations of audio data."

user_text = """Analyze this spectrogram (a visual representation of
  the frequency spectrum of sound over time) and determine the most
  likely sound class. Consider frequency patterns, intensity, and time
  variations. Focus SOLELY on patterns in the spectrogram. Do not let
  assumptions about common sounds influence your decision.
  Classes: [dog, chainsaw, crackling_fire, helicopter, rain,
            crying_baby, clock_tick, sneezing, rooster, sea_waves].
  Your response must contain the EXACT class name only."""

messages = [
  {"role": "system", "content": system},
  {"role": "user", "content": [
      {"type": "text", "text": user_text},
      {"type": "image_url",
       "image_url": {"url": f"data:image/png;base64,{spectrogram_b64}"}}
  ]}
]

The few-shot upgrade

For few-shot, the single user message becomes a sequence: for each class, a small text label ("Example — class: rooster") followed by that class's example image; then finally the query image with "Now classify this:". Ten example images plus one query, all in one context window, all read in one forward pass.

# Few-shot content list (one example image per class, then the query)
content = [{"type": "text", "text": intro}]
for cls, img_b64 in examples.items():        # 10 (label, image) pairs
    content += [
      {"type": "text",  "text": f"This is a {cls} spectrogram:"},
      {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}}
    ]
content += [
   {"type": "text",  "text": "Now classify THIS spectrogram (class name only):"},
   {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{query_b64}"}}
]

That's the realization in full. No training loop, no optimizer, no labels-as-gradient-signal. The labels are words in the prompt, and the analogy happens in attention.

Chapter 5: Picking the Examples (the showcase)

If the few-shot examples are the model's only anchor for "what does a rooster spectrogram look like," then which example you pick per class matters enormously. Pick a weird, atypical rooster clip and you mislead the model. Pick a representative one and you nail it.

The authors' clean idea: don't pick examples randomly. For each class, run k-means clustering on the mel-spectrogram features of all that class's clips, then choose the clip closest to a cluster centroid — i.e. the most prototypical example. This single change moved GPT-4o from 70% (random) to 73.75% (k-means, 1 per class), and to a peak of 76.25% with 2 examples per class.

Why the centroid example, in plain terms

Each clip becomes a point in feature space (flattened mel-spectrogram). k-means finds cluster centers — the "average" rooster sound. The clip nearest a centroid is the least weird, most textbook example of its cluster. Showing the model a textbook rooster gives it the cleanest possible template to match against. Showing it an outlier rooster (a faint, distant one) teaches a misleading template.

Each dot is one clip of a single class in mel-feature space. Run k-means to find centroids (rings). Toggle the selection rule: "centroid" picks the clip nearest a center (prototypical — what the paper does); "random" picks any clip (the baseline). The chosen example becomes the in-context reference. Watch how an outlier pick lands far from the cloud's heart — that's the misleading template.

Chapter 6: Spectrogram Knobs

Here is the most VLM-specific finding in the paper, and the one a signal-processing person would never guess. How you draw the picture changes the score — and the "best for the human ear" choices are not always the best for the model.

The authors swept the spectrogram rendering hyperparameters and measured GPT-4o zero-shot accuracy on ESC-10 fold 1. The defaults (log frequency, log amplitude, viridis colormap, axis labels, no colorbar) gave 27.5%. The single best change was switching to a linear frequency axis, which lifted accuracy to 35%.

The full sweep (GPT-4o zero-shot, ESC-10 fold 1)

Reconstructed from the paper's Appendix A ablation table. The dashed line is the default config. Linear frequency and linear amplitude beat the perceptually-motivated logarithmic defaults. MFCCs — a heavily compressed representation that throws away the visual texture — collapse to 13.75%, near chance: great features for a classical model, terrible pictures for a VLM.

Chapter 7: Results

Now the headline comparisons. Three claims, each with a number.

Claim 1 — Few-shot transforms VLMs

Across all six VLMs, examples helped. GPT-4o went from 35% (tuned zero-shot) to 70% (10-shot). Claude-3.5 Sonnet went 22.5% → 56.25%. The effect is real and large — but uneven: Gemini models barely moved or even dropped, suggesting in-context image reasoning quality varies a lot between model families.

Claim 2 — VLMs beat the commercial audio model

The fair fight: GPT-4o seeing a spectrogram vs. Gemini-1.5 Pro hearing the actual audio (the only commercial model that ingests audio at the time). On the full cross-validated ESC-10, GPT-4o scored 59.00% vs. Gemini-audio's 49.62%. A picture of the sound, read by a model that can't hear, beat a model that can hear it directly.

Claim 3 — GPT-4o matched human experts

Three audio-research experts each labeled the 80 spectrograms of fold 1 with the same 10-shot references. Ensembled, they hit 72.5%. GPT-4o on the identical setup: 73.75%. Inter-annotator agreement (Cohen's κ) was only 0.53 — even the experts disagreed a lot, which tells you VSC is genuinely hard, not that the model is trivially good.

Toggle zero-shot vs few-shot. The dashed line is the human-expert ensemble (72.5%) and the dotted line is random chance (10%). Watch GPT-4o leap past the human line in few-shot while the Gemini models barely move.

Chapter 8: Limits & Failure Modes

The paper is refreshingly honest about where this breaks.

1. It does not beat real audio classifiers

A purpose-built supervised model (e.g. an Audio Spectrogram Transformer) crushes 59% on ESC-10. The VLM approach is impressive given zero audio training, not impressive in absolute terms. Its value is the free language reasoning attached, not raw accuracy.

2. It collapses as classes grow

The sharpest failure. On ESC-10 (10 classes), GPT-4o cross-validated at 59%. On a 50-class ESC-50 subset, 50-shot accuracy fell to 14%. Still above the 2% chance floor, but a brutal drop.

Drag the class count. The solid curve is the model's accuracy (anchored on the paper's two measured points: 59% at 10 classes, 14% at 50); the dashed curve is the 1/N random-chance floor. Notice the model stays well above chance throughout, but the margin over chance shrinks — the structure it relies on is eroding faster than chance does.

3. Why it works at all is partly a mystery

The authors are candid: VLMs were trained on natural images and text, not spectrograms, so the latent spectrogram knowledge is incidental — picked up from incidentally-seen plots online. As VLMs scale and see more such images, this should improve, but it's an emergent side-effect, not a designed capability.

Chapter 9: Connections & Cheat Sheet

This paper sits at a crossroads of three ideas: the spectrogram as the bridge between sound and vision, in-context learning as a substitute for training, and the surprising generality of frontier VLMs.

Two envisioned uses

• Audio caption augmentation. Feed a VLM a vague human caption plus the spectrogram. The picture grounds the caption in the actual sound, letting the model's strong language ability produce accurate, descriptive captions — cheaper than a dedicated audio model and less prone to pure-text hallucination.
• VSC as a VLM benchmark. Visual Spectrogram Classification is a clean, hard test of whether a VLM truly reasons over unfamiliar image types, not just natural photos. A new challenge axis for evaluating multimodal reasoning.

The cheat sheet

Related lessons

• WaveNet — generating raw audio sample-by-sample; the other end of the audio-AI stack.
• In-context learning & few-shot prompting — the mechanism that makes the frozen-VLM trick work.
• STFT / mel-spectrograms — the signal-processing bridge from waveform to image.