AST: Audio Spectrogram Transformer

Chapter 1: Audio as an Image

A raw waveform is a one-dimensional signal — a single number (air pressure) sampled tens of thousands of times per second. AST never touches the waveform directly. It first turns the sound into a picture: a spectrogram, a 2-D map where one axis is time, the other axis is frequency, and the brightness of each cell is how much energy that frequency carried at that moment.

Concretely, AST uses the same front end as a modern speech system. The audio is converted to a log-mel filterbank: a 25 ms analysis window slides forward every 10 ms (a 10 ms hop), and at each step the energy is collected into 128 mel-frequency bins. "Mel" means the frequency bins are spaced the way human hearing perceives pitch — fine resolution low down, coarse resolution up high. The result for a clip is a matrix of shape [frequency = 128, time = T].

How big is T? The hop is 10 ms, so one second of audio produces 100 time frames. AudioSet clips are about 10 seconds, giving roughly 1000 frames. AST's default input is therefore a [128 × 1024] matrix — 128 frequency bins by 1024 time frames. This is, for all the model cares, a 128×1024 single-channel grayscale image. The "pixels" happen to be acoustic energies instead of light intensities, but the data type — a 2-D grid of real numbers — is identical.

The frequency axis is not the time axis

It is tempting to think of the spectrogram as just an image, but the two axes mean very different things, and this asymmetry matters later. The time axis is translation-equivariant in the way images are — a bark at second 2 and the same bark at second 7 look identical, just shifted. The frequency axis is not: shifting a sound up in frequency changes what it is (a low rumble is not a high whistle). So while AST treats the spectrogram as an image to reuse ViT machinery, the model must learn — through the data — that "shift along time = same event, shift along frequency = different event."

A worked size check

Let us nail down the numbers, because every later tensor shape descends from them. Sample rate 16 kHz, hop 10 ms means 160 samples per hop, so a 10.24-second clip is 1024 hops → T = 1024 frames. The mel front end produces F = 128 bins. Input matrix: [128, 1024]. If instead you classify the short ESC-50 clips (5 s) you get T ~ 512; for Speech Commands (1 s) you get T ~ 128. AST simply changes T — the rest of the architecture is identical. That flexibility is the payoff of treating audio as an image: one architecture, many tasks, only the input grid resizes.

Chapter 2: Patch Embedding — the one operation that replaces the CNN

Here is the heart of the paper, and the mechanism you will implement in the Code Labs. A Transformer encoder consumes a sequence of vectors, not a 2-D image. So AST must turn the [128 × 1024] spectrogram into a sequence of token vectors. It does this exactly the way the Vision Transformer turns a photo into tokens: patchify, then linearly embed.

Step 1 — Patchify

Slide a 16×16 window over the spectrogram and cut it into square patches. AST uses a patch size of 16×16 and — crucially — overlapping patches, with a stride of 6 in 6 in time, instead of the non-overlapping 16-stride of vanilla ViT. We will treat the non-overlapping case first because the arithmetic is cleaner, then explain why overlap helps.

With non-overlapping 16×16 patches over a [128 × 1024] spectrogram, the patch grid is (128/16) × (1024/16) = 8 × 64 = 512 patches. Each patch is a 16×16 = 256-number tile. So the image becomes a sequence of N = 512 tiles, each a 256-vector. The general formula for the count of patches along an axis of length L with patch size P and stride s is:

With overlap (stride 6), along time L=1024 gives n_t = ⌊(1024−16)/6⌋+1 = 169, and along frequency L=128 gives n_f = ⌊(128−16)/6⌋+1 = 19 (with the exact mel/time settings the paper reports about 12 frequency × 101 time = 1212 patches for the 10 s case). The exact count shifts with the front-end settings; what matters is the recipe: more overlap → more patches → finer coverage → better accuracy, at higher compute. Setting s = P recovers the non-overlapping ViT count.

Step 2 — Linear embedding (the "patch embedding")

Each patch is flattened from 16×16 to a 256-dimensional vector, then multiplied by a single learned weight matrix E to produce a d-dimensional token. AST uses d = 768 (the ViT-Base / DeiT width). So:

Stack all N patch tokens and you have the sequence the encoder wants: a matrix of shape [N, 768] (e.g. [512, 768] in the non-overlapping case). That is the entire "feature extractor" of AST. No convolution stack, no residual CNN blocks — one linear projection turns each spectrogram tile into a token. Vaswani's encoder does the rest.

Worked example — patchify and embed by hand

Take a tiny 4×4 spectrogram and 2×2 non-overlapping patches (P=2, s=2). The grid is (4/2)×(4/2) = 2×2 = 4 patches. Patch 0 is the top-left 2×2 block; flattened in row-major order it is [a₀₀, a₀₁, a₁₀, a₁₁], a 4-vector. With an embedding matrix E of shape [4, d], token 0 is that 4-vector times E. Repeat for all four patches and you have a [4, d] token matrix. Scale this up — 16×16 patches, 256-vectors, [256, 768] embedding, ~512 patches — and you have exactly AST's input pipeline. Nothing more elaborate happens.

python
# AST patch embedding: spectrogram -> token sequence
import torch
import torch.nn as nn

# Input: 1 clip, 1 channel, 128 mel bins, 1024 time frames
spec = torch.randn(1, 1, 128, 1024)   # [B, C, F, T]

# Patch embedding = ONE strided conv. kernel 16, stride 16 -> non-overlap
# (AST uses stride 6 for OVERLAP; 16 shown for clean arithmetic)
patch_embed = nn.Conv2d(1, 768, kernel_size=16, stride=16)

x = patch_embed(spec)        # [1, 768, 8, 64]  = 8x64 patch grid
x = x.flatten(2)            # [1, 768, 512]    flatten the grid
x = x.transpose(1, 2)       # [1, 512, 768]    N=512 tokens, d=768

# That's it. 512 tokens, each a 768-vector, ready for the encoder.
print(x.shape)              # torch.Size([1, 512, 768])

Chapter 3: The [CLS] Token and Position Embeddings

We now have a sequence of N patch tokens. Two things are missing before the encoder. First, attention is permutation-invariant — it has no idea that patch 5 sits to the right of patch 4 — so we must inject position. Second, the model needs a single fixed slot to read out the final classification, so AST prepends a learnable [CLS] token, exactly as BERT and ViT do.

The [CLS] token

AST prepends one extra learnable vector — call it x_cls, shape [768] — to the front of the patch sequence. It corresponds to no patch; it is a blank slot the model fills in. As it passes through the encoder it attends to every patch token and accumulates a summary of the whole clip. After the final layer, AST reads only the [CLS] output and feeds it to the classifier. So the [CLS] token is a learnable "pooling query": instead of average-pooling the patch outputs, the model learns, through attention, how to aggregate them into one clip-level vector.

The sequence length becomes N+1 (one [CLS] plus N patches). With N=512 that is 513 tokens, each 768-dimensional: a [513, 768] matrix enters the encoder.

Position embeddings — and the transfer trick

AST adds a learnable position embedding E_pos to every token (including [CLS]). Here is the elegant part. AST does not train these from scratch — it inherits them from a ViT/DeiT model pretrained on ImageNet. But ImageNet ViT was trained on a 14×14 = 196-patch grid (one [CLS] + 196 = 197 positions), whereas AST's spectrogram patch grid is a different shape, e.g. 8×64. The position embeddings cannot be copied directly because the grids do not match.

AST's solution is a 2-D interpolation of the position embeddings. Reshape ViT's 196 position vectors back into a 14×14 grid of vectors, then bilinearly resize that grid to AST's target shape (e.g. 8×64), then flatten back to a sequence. Now every AST patch position gets a sensible starting embedding derived from the image-pretrained one, rather than a random vector. This single trick — cut-and-interpolate position embeddings across grid shapes — is what makes weight transfer from a square image model to a wide spectrogram model actually work.

Worked count

ImageNet ViT-Base: 197 position embeddings (1 [CLS] + 14×14 patches), each 768-dim. AST with an 8×64 patch grid: 513 position embeddings (1 [CLS] + 8×64). The [CLS] position is copied straight across; the 196 patch positions are reshaped to 14×14, bilinearly resized to 8×64 = 512, and flattened. Every number in the encoder weights — attention projections, FFN, layer norms — is copied directly; only the position embeddings need this re-gridding, because they are the only weights whose shape depends on the patch-grid geometry.

Chapter 4: The Encoder and the Classification Head

Everything after the patch+position step is a standard Transformer encoder — the very same one from "Attention Is All You Need," with no audio-specific modification. AST uses the ViT-Base / DeiT-Base configuration: 12 encoder layers, 12 attention heads, embedding dimension 768. Each layer is the familiar block: multi-head self-attention, then a position-wise feed-forward network, each wrapped with residual connections and layer normalization.

Trace the shapes. The input z₀ is [513, 768] (1 CLS + 512 patches). Multi-head self-attention forms a [513, 513] attention matrix — every token, [CLS] included, attends to every other token — and the output stays [513, 768]. The FFN expands 768 → 3072 → 768. Twelve such layers, shape preserved, output z₁₂ still [513, 768]. Because attention is global, the [CLS] token at layer 12 has — in principle through stacked layers, and directly through the [513,513] matrix at every layer — gathered information from every time-frequency patch in the clip. That global reach from layer one is the whole reason for dropping the CNN.

The head

Take the [CLS] output vector z₁₂[0], shape [768]. A single linear layer maps it to the label logits. AudioSet has 527 classes, so the head is a [768, 527] matrix. Critically, AudioSet is multi-label — a clip can be "dog" AND "speech" AND "music" at once — so AST does NOT use softmax. It applies an independent sigmoid to each of the 527 logits and trains with binary cross-entropy, treating each label as its own yes/no question:

For single-label tasks (ESC-50, Speech Commands) AST swaps the sigmoid/BCE head for a softmax + cross-entropy head. The encoder is untouched; only the final layer and loss change with the task.

Why mean average precision, not accuracy?

Because AudioSet is multi-label and hugely imbalanced (most clips have a handful of the 527 labels), plain accuracy is meaningless — a model predicting "all negative" would score high. The metric is mAP, mean average precision: for each class, sort all clips by the model's score for that class, compute the area under the precision-recall curve, then average over the 527 classes. AST's headline result, ~0.485 mAP, is this number.

python
import torch
import torch.nn as nn

class AST(nn.Module):
    def __init__(self, n_classes=527, d=768, depth=12, heads=12):
        super().__init__()
        self.patch = nn.Conv2d(1, d, kernel_size=16, stride=16)  # patch embed
        self.cls   = nn.Parameter(torch.zeros(1, 1, d))         # [CLS] token
        self.pos   = nn.Parameter(torch.zeros(1, 513, d))       # 1 CLS + 512 patch
        layer = nn.TransformerEncoderLayer(d, heads, 3072, batch_first=True)
        self.enc  = nn.TransformerEncoder(layer, depth)
        self.norm = nn.LayerNorm(d)
        self.head = nn.Linear(d, n_classes)  # [768, 527]

    def forward(self, spec):                  # spec: [B, 1, 128, 1024]
        x = self.patch(spec)                  # [B, 768, 8, 64]
        x = x.flatten(2).transpose(1, 2)    # [B, 512, 768]
        cls = self.cls.expand(x.size(0), -1, -1)
        x = torch.cat([cls, x], dim=1)        # [B, 513, 768]  prepend CLS
        x = x + self.pos                       # add position embeddings
        x = self.enc(x)                       # [B, 513, 768]  12 layers
        cls_out = self.norm(x[:, 0])           # read ONLY the CLS slot
        return self.head(cls_out)            # [B, 527] logits -> sigmoid

Chapter 5: Cross-Modal Weight Transfer

A pure Transformer has very little built-in structure — no locality, no translation bias, nothing that says "nearby pixels are related." That is a strength (it can learn any relationship) and a weakness (it must learn everything from data, and so it is famously data-hungry). The original ViT needed enormous datasets (JFT-300M) to beat CNNs from scratch. AudioSet is large for audio but tiny next to that. Train AST from random initialization on AudioSet and it underperforms the CNNs it was meant to beat.

The paper's central practical contribution is the answer: do not train from scratch — transfer the weights of a Vision Transformer pretrained on ImageNet. Because AST's architecture is byte-for-byte a ViT (patch embed, [CLS], position embeddings, 12 encoder layers), nearly every parameter can be copied directly from an ImageNet-pretrained DeiT model. AST specifically uses DeiT (a ViT trained efficiently on ImageNet with distillation), which transfers better than vanilla ViT.

The three obstacles to copying, and how AST fixes each

Channel fix in detail. ViT's patch embedding expects 3 channels; its kernel has shape [768, 3, 16, 16]. A spectrogram has 1 channel. AST averages the kernel across the RGB dimension: the new [768, 1, 16, 16] kernel is the mean of the three color kernels. Averaging (rather than picking one channel) preserves the expected output scale, so the downstream activations land in the range the pretrained encoder expects.

Position fix in detail. As shown in Chapter 3: reshape the 196 patch position embeddings into 14×14, bilinearly resize to the audio grid, flatten. The [CLS] position copies directly.

Everything else — all 12 layers of attention projections, the FFN weights, the layer norms, the [CLS] vector itself — is copied verbatim, then the whole model is fine-tuned end to end on the audio task. That is the recipe.

Chapter 6: The Headline Results

AST is evaluated on three benchmarks spanning the breadth of audio classification: AudioSet (the big multi-label event-tagging set), ESC-50 (environmental sounds, single label), and Speech Commands V2 (keyword spotting). The story is the same on all three: the convolution-free, image-pretrained Transformer sets a new state of the art.

AudioSet — the flagship

On the full AudioSet, a single AST model reaches roughly 0.485 mAP, surpassing the previous best (PSLA, a CNN with attention pooling, around 0.474). Ensembling several AST models pushes it higher still (the paper reports roughly 0.485 single / ~0.459 on the balanced set and higher with ensembles). The exact decimals matter less than the qualitative result: a model with zero convolutions beats the carefully engineered CNN-plus-attention systems it replaced.

Notice the transfer chain in the ESC-50 row: best results come from a model pretrained on ImageNet, then on AudioSet, then fine-tuned on ESC-50. Each stage hands down structure to the next — image structure to audio-event structure to environmental-sound structure. The Transformer's lack of built-in bias, which forced it to need pretraining, turns out to make it an excellent vehicle for stacking transfers.

Worked reading of an mAP gain

Why care about 0.474 → 0.485, a gain of ~0.011 mAP? Because on a saturated, fiercely contested benchmark like full AudioSet, a one-point mAP gain typically represents a substantial jump up the leaderboard — and AST achieves it while being simpler (one architecture, no convolutions, no hand-tuned pooling). The result is not "slightly better numbers"; it is "a simpler model that is also better," which is why the architecture was widely adopted.

Chapter 7: The Ablations — what actually matters

A result is only convincing if you know which ingredient produced it. AST's ablation study isolates each design choice. The findings are unusually clean and they tell you exactly how to reproduce the model — and where the easy points are.

1. ImageNet pretraining — the single biggest factor

Remove the cross-modal transfer and train AST from random initialization on AudioSet: the mAP drops sharply, below the CNN baselines. This is by far the largest ablation effect in the paper. The conclusion is blunt: for a convolution-free model on a dataset the size of AudioSet, pretraining is not a nice-to-have, it is the load-bearing wall.

2. Patch overlap (stride) — accuracy for compute

AST uses overlapping patches (stride 6 in time and frequency rather than 16). Increasing overlap raises accuracy because each time-frequency location is covered by several patches, giving the model a finer, more redundant view — at the cost of more patches and thus more compute (the sequence length grows). The ablation shows a smooth accuracy-vs-overlap trade. Setting stride = patch size (no overlap) recovers vanilla ViT tokenization and is cheaper but slightly worse.

3. Position-embedding interpolation — keep it 2-D

How you adapt the pretrained position embeddings matters. Reinitializing them from scratch (discarding the image prior) hurts. Naively reusing them without re-gridding fails because the grids do not match. The 2-D bilinear interpolation that respects the time×frequency layout is what works — confirming that the spatial structure in the pretrained position embeddings is genuinely useful for audio.

4. DeiT vs ViT, and model size

AST transfers best from DeiT (the distillation-trained ViT) rather than the original ViT, and the Base size is the sweet spot for AudioSet — large enough to absorb the pretraining, small enough not to overfit the audio data.

Chapter 8: The AST Explorer

This is the payoff. Below is the full AST pipeline as one interactive instrument. You place sound events on a spectrogram; the explorer patchifies it, runs a (toy) attention pass where the [CLS] token attends to every patch, and shows you the resulting [CLS] attention map and the per-class scores. Move the controls and watch the [CLS] token concentrate its attention on the bright, event-bearing patches — which is exactly how the real AST builds a clip-level prediction.

Things to try. Add a single bark, low on the frequency axis: the [CLS] attention lights up on the low patches and the "Animal" score rises. Add a whistle high up: attention splits between the two regions and a second class activates. Crank the temperature down: attention sharpens to a near one-hot peak on the single brightest patch (the saturation regime). Increase the patch-grid density: more, smaller patches give finer attention localization — the same overlap-vs-resolution trade from the ablations, live.

Chapter 9: Limitations, Legacy, and Connections

AST proved a thesis: the convolution-free Vision Transformer recipe transfers to audio, and cross-modal pretraining makes it win. But the vanilla model has real limitations, and the field moved quickly to address them.

Limitations

• Quadratic cost in sequence length. Self-attention forms an [N, N] matrix. With overlapping patches over a 10-second clip, N can be over a thousand, and longer audio makes the attention matrix grow quadratically. AST is not cheap on long recordings.
• Data hunger / pretraining dependence. The model only shines because of ImageNet transfer. With no large pretrained backbone in your modality, a from-scratch AST is weak.
• Supervised pretraining required. AST leans on a labeled-ImageNet model. The natural next question — can we pretrain on unlabeled audio? — was answered by its successor.
• The image-prior is a crutch and a bias. Inheriting image structure helps, but it also imports assumptions that do not perfectly fit audio (e.g. the symmetric treatment of the two axes that the frequency axis violates).

Legacy and successors

• SSAST (self-supervised AST): pretrain AST on unlabeled audio with a masked-patch objective, removing the dependence on labeled ImageNet.
• PaSST: patchout — randomly drop patches during training to cut the quadratic cost and regularize, with little accuracy loss.
• AST in multimodal systems: the AST encoder became a common audio front end inside audio-language and audio-visual models, precisely because it is a clean Transformer whose tokens compose with text and image tokens.

Cheat sheet

Connections across Engineermaxxing