PaSST: Patchout faSt Spectrogram Transformer

Chapter 1: From Spectrogram to Patch Tokens

Before we can drop patches, we need to be precise about where they come from. PaSST inherits its front end from AST and DeiT, so let's trace the real data flow with shapes.

Step 1 — waveform to spectrogram. A 10-second clip at 32 kHz is 320,000 raw samples. We slide a short window across it and compute a log-mel spectrogram: a 2D array of shape [F, T0] where F is the number of mel frequency bins (128) and T₀ is the number of time frames (~1000 for 10 s). Each entry is the log energy in one frequency band at one instant. Think of it as a grayscale image, 128 pixels tall and ~1000 wide.

Step 2 — patchify. Exactly like ViT, we cut the spectrogram-image into 16×16 patches. PaSST uses a convolutional patch embedding: a Conv2d with kernel 16, stride 10. The stride being smaller than the kernel means patches overlap — this is the AST trick that keeps boundary information. Each patch is linearly projected to a D-dimensional token (D = 768 for the base model).

With 128 mel bins and stride 10 over a ~1000-frame clip, you land on roughly F′ ≈ 12 frequency patches by T′ ≈ 99 time patches, which is about 1188 patch tokens. Add two special tokens — a CLS token (carries the global summary for classification) and a DIST token (a distillation token inherited from DeiT) — and you have ~1190 tokens entering the transformer.

A worked patch count

The patch grid size along a dimension of length L with kernel k and stride s is the standard convolution formula:

Take the frequency axis: L = 128 mel bins, k = 16, s = 10.

Now the time axis with L = 1000, k = 16, s = 10:

So the patch grid is 12×99 = 1188 patches. That single number — 1188 — is the enemy. Every patch we keep costs us its share of the T² attention. Patchout is going to reach into this grid and delete most of it.

python
import torch, torch.nn as nn

# PaSST patch embedding: overlapping 16x16 patches via strided conv
patch_embed = nn.Conv2d(in_channels=1, out_channels=768,
                        kernel_size=16, stride=10)

spec = torch.randn(1, 1, 128, 1000)   # [B, C, F, T0]
x = patch_embed(spec)                  # [1, 768, 12, 99]
Fp, Tp = x.shape[2], x.shape[3]   # 12, 99
x = x.flatten(2).transpose(1, 2)   # [1, 1188, 768] tokens
print(x.shape, Fp * Tp)              # torch.Size([1, 1188, 768]) 1188

Chapter 2: The Patchout Idea

Here is the entire paper in one sentence: during training, randomly delete a subset of the patch tokens before they enter the transformer. That's Patchout. The transformer then runs on a shorter sequence, the attention matrix shrinks quadratically, and the gradients still teach the model everything it needs because the dropped patches were largely redundant anyway.

Let's be careful about where in the pipeline this happens. After patch embedding we have a token sequence of length T = 1188 (plus the CLS and DIST tokens). Patchout picks a random subset of the 1188 patch tokens to keep — say 70% of them — and discards the rest. The CLS and DIST tokens are never dropped; they must always reach the output. The kept tokens, with their positional encodings already added, are concatenated with CLS and DIST and fed to the transformer blocks.

Two jobs in one mechanism

Patchout does double duty, and this is why it works so well:

Job 1 — speed and memory. Fewer tokens means a smaller T² attention matrix. Keep 70% of tokens and the attention cost drops to (0.7)² ≈ 0.49 — roughly half the compute and memory, for free.

Job 2 — regularization. Because a different random subset is dropped every step, the model never sees the same exact input twice. It can't memorize "the bark is in patches 412 and 413"; it must learn redundant, distributed evidence for each class. This is the same logic as dropout, but applied to whole input patches — and it acts like a structural cousin of SpecAugment, which masks spectrogram regions for the same anti-overfitting reason.

Why does dropping input information not just hurt? Two reasons. First, spectrogram patches are extremely redundant — overlap from the strided conv plus the slow-changing nature of audio means adjacent patches carry near-duplicate information. Second, the model is forced to spread its bets: a robust classifier that relies on many weak cues across the spectrogram generalizes better than a brittle one that keys on a single patch. Patchout manufactures exactly that pressure.

python
import torch

def unstructured_patchout(tokens, keep_frac=0.7):
    """tokens: [T, D] patch tokens (CLS/DIST handled separately)."""
    T = tokens.shape[0]
    n_keep = int(round(keep_frac * T))
    # random permutation, keep the first n_keep indices
    perm = torch.randperm(T)
    keep = perm[:n_keep].sort().values   # sort to preserve order
    return tokens[keep], keep            # [n_keep, D]

tok = torch.randn(1188, 768)
kept, idx = unstructured_patchout(tok, 0.7)
print(kept.shape)   # torch.Size([832, 768])  ~30% gone

Chapter 3: The Quadratic Savings, Derived

Patchout's speed argument rests on a single fact about self-attention: its cost is quadratic in sequence length. Let's derive it carefully so the savings are not hand-waved.

Self-attention on a sequence of T tokens of dimension D computes Q, K, V each of shape [T, D], then the score matrix:

That matrix multiply costs T·T·D = T²D multiply-adds, and the matrix itself occupies T² floating-point numbers per head. The softmax and the value aggregation S·V add another T²D. So the attention block scales as:

Now suppose Patchout keeps a fraction ρ of the patch tokens, so the new length is T′ = ρT (ignoring the two constant special tokens). The new attention cost is:

The savings factor is therefore ρ², not ρ. This is the crucial leverage: dropping linearly in token count buys you quadratic savings in attention.

A concrete number

Start with T = 1188 tokens. Suppose structured patchout removes enough time and frequency strips to keep about ρ = 0.55 of the patches, leaving T′ ≈ 653 tokens.

Keeping 55% of the tokens cuts ~70% of the attention work. That is why the paper reports both large speedups and large memory reductions from what looks like a modest amount of dropping — the squaring magnifies everything.

Memory is the other half of the win, and arguably the one that mattered most in practice. The attention matrix and its softmax must be held in GPU memory for the backward pass. Halving T roughly quarters that buffer, which is often the difference between a batch size of 2 and a batch size of 8 — and bigger batches train both faster and more stably.

python
# Relative attention cost scales with rho squared, not rho
T = 1188
for rho in [1.0, 0.7, 0.55, 0.4]:
    Tp = round(rho * T)
    cost = (Tp / T) ** 2
    print(f"keep {rho:.2f} -> {Tp:4d} tokens, attn cost x{cost:.2f}")
# keep 1.00 -> 1188 tokens, attn cost x1.00
# keep 0.70 ->  832 tokens, attn cost x0.49
# keep 0.55 ->  653 tokens, attn cost x0.30
# keep 0.40 ->  475 tokens, attn cost x0.16

Chapter 4: Structured vs Unstructured Patchout

Patchout comes in two flavours, and the difference is geometric. Recall the patch grid is two-dimensional: F′ frequency rows by T′ time columns. Which patches you choose to drop matters.

Unstructured patchout drops individual patches at random, scattered anywhere on the grid. It is the most direct analogue of dropout. It maximizes diversity — every patch is an independent coin flip — but the kept set is a ragged, irregular subset, which is slightly awkward for the disentangled positional scheme we meet in Chapter 5.

Structured patchout drops entire strips of the grid: whole time columns (every frequency at some time instants) or whole frequency rows (every time at some mel bands). This is the flavour PaSST actually leans on, for two reasons:

First, it aligns perfectly with how audio events are organized. Dropping time columns forces the model to recognize an event from a fragment of its duration — robustness to time shifts and truncation. Dropping frequency rows forces recognition from a partial spectral footprint — robustness to band-limited corruption. This is precisely the inductive bias SpecAugment encodes, but realized by removing tokens instead of zeroing them.

Second, dropping along an axis keeps the surviving grid rectangular and indexable, which makes the time/frequency positional encodings (Ch 5) clean to apply — you still know each kept token's row and column.

A worked count for structured patchout

Take the 12×99 grid (1188 patches). Suppose we drop 2 frequency rows out of 12 and 25 time columns out of 99. The kept grid is (12−2)×(99−25) = 10×74:

So two dropped bands plus a quarter of the time axis already removes ~61% of the attention work, while leaving a tidy 10×74 rectangle the positional encoder can address directly.

python
import torch

def structured_patchout(grid, drop_f=2, drop_t=25):
    """grid: [F, T, D] patch tokens laid out on the 2D grid."""
    F, T, D = grid.shape
    keep_f = torch.randperm(F)[: F - drop_f].sort().values
    keep_t = torch.randperm(T)[: T - drop_t].sort().values
    g = grid[keep_f][:, keep_t]        # [F-2, T-25, D], still a grid
    return g.reshape(-1, D), keep_f, keep_t

grid = torch.randn(12, 99, 768)
tok, kf, kt = structured_patchout(grid)
print(tok.shape)   # torch.Size([740, 768])

Chapter 5: Disentangled Time & Frequency Position

A transformer is permutation-invariant: shuffle the tokens and the math doesn't notice. So we must tell it where each patch lives. AST used a single learned positional embedding per patch — one vector for "row 3, column 17". PaSST disentangles this into two separate positional embeddings: one for the time index and one for the frequency index, added together.

Where Efreq is a learned table of F′ vectors (one per frequency row) and Etime is a learned table of T′ vectors (one per time column). A patch at grid cell (f, t) gets the sum of its row vector and its column vector. This is the same idea as factorized positional encodings in vision transformers.

Why disentangle? Three concrete payoffs

1. It makes Patchout clean. When structured patchout drops time column t, you simply don't add Etime[t] for any surviving token — the remaining tokens keep their correct (f, t) coordinates. A single joint table would have a hole shaped like the dropped column; the factorized table just skips an entry.

2. It generalizes across clip lengths. A 20-second clip has more time columns than a 10-second clip. With a joint table you would need a 2D grid sized for the longest clip. With separate axes you extend only Etime — and you can interpolate it — while Efreq stays fixed because the number of mel bands never changes. This is what lets a PaSST trained on 10 s evaluate gracefully on longer audio.

3. Fewer parameters. A joint table needs F′·T′ vectors (12·99 = 1188). The factorized version needs only F′ + T′ = 12 + 99 = 111 vectors — a 10× reduction in positional parameters, with no measured loss.

A worked addition

Say D = 4 for illustration. Suppose Efreq[3] = [0.2, −0.1, 0.5, 0.0] and Etime[17] = [−0.3, 0.4, 0.1, 0.2]. The patch at (f=3, t=17) receives:

This sum is added to the patch's content embedding before the first transformer block. Two patches in the same frequency row share their Efreq term; two in the same time column share their Etime term — a structured prior that mirrors the structure of the grid.

python
import torch, torch.nn as nn

# Disentangled (factorized) positional encoding
Fp, Tp, D = 12, 99, 768
E_freq = nn.Parameter(torch.randn(Fp, D))   # 12 vectors
E_time = nn.Parameter(torch.randn(Tp, D))   # 99 vectors

# broadcast-add to a [Fp, Tp, D] grid of positions
pos = E_freq[:, None, :] + E_time[None, :, :]  # [12, 99, 768]
print(pos.shape, Fp + Tp, "vs joint", Fp * Tp)
# torch.Size([12, 99, 768])  111 vs joint 1188

Chapter 6: Train Short, Test Full

This is the chapter that prevents the most common PaSST mistake. Patchout is applied only during training. At inference, the model receives all the patches — nothing is dropped. The model trains on short, stochastic sequences and then is evaluated on the full-length, deterministic one.

Why is this not a train/test mismatch disaster? Because the transformer's attention and feed-forward layers are length-agnostic: the same weights apply to a sequence of 700 tokens or 1188 tokens. Patchout doesn't change the architecture — it only changes how many tokens flow through each step. So a network trained to classify from any random 700-token subset trivially handles the full 1188-token sequence; if anything, the full set is an easier, information-richer input than the subsets it trained on.

The implicit ensemble view

There's a deeper way to see why this helps. Each training step shows the model a random subset of patches, which is like training an exponential family of sub-models that share weights — one per possible kept-set. This is exactly the ensemble interpretation of dropout (Srivastava et al. 2014). At inference, using all patches is the analogue of dropout's "use all units, no scaling needed here because we removed rather than scaled" — you evaluate the full committee at once. The averaging over training masks is what tightens generalization.

A controlled trade-off

The drop rate is a dial. Drop very aggressively (keep 40%) and each step is cheap but the masks are so degraded that convergence slows and accuracy can dip. Drop nothing (keep 100%) and you're back to vanilla AST — expensive and prone to overfit. The paper tunes the rate to a sweet spot where training is markedly cheaper and validation accuracy is higher than the no-patchout baseline.

python
class PaSST(nn.Module):
    def forward(self, spec):
        x = self.patch_embed(spec)          # [B, T, D] patch tokens
        x = x + self.pos_embed(x)         # add factorized position
        if self.training:                  # <-- the whole trick
            x = self.patchout(x)            # drop tokens, shorten sequence
        x = self.prepend_cls_dist(x)      # CLS + DIST never dropped
        x = self.blocks(x)               # transformer (length-agnostic)
        return self.head(x[:, 0])         # classify from CLS

Chapter 7: Results & Ablations

The headline result: on AudioSet — the standard 2-million-clip, 527-class benchmark — PaSST set a new state of the art in mean Average Precision (mAP), reaching roughly 0.47 mAP with an ensemble, surpassing the prior AST result, while training substantially faster and with much lower memory.

Read that twice, because it's the surprising part. The cheaper method won. Patchout didn't trade accuracy for speed — it bought both, because the speed mechanism (fewer tokens) and the accuracy mechanism (regularization) are the same mechanism.

What the numbers looked like (qualitatively)

Ablations: which knob matters?

The paper sweeps the patchout configuration, and the qualitative findings are consistent:

Some patchout beats none. Turning patchout off (keep 100%) is both the slowest setting and not the most accurate — without the regularization the model overfits. A moderate drop is the best of both worlds.

Too much patchout hurts. Push the drop rate too high and accuracy falls: the masks become so sparse that the model is starved of context. The accuracy-vs-drop curve is the inverted-U we sketched in Chapter 6.

Structured time + frequency patchout is the strong default. Dropping along both axes captures the SpecAugment-style invariances and keeps the grid clean for the factorized positions.

Disentangled position helps and costs nothing. Replacing the joint positional table with the factorized one preserves accuracy while cutting positional parameters ~10× and enabling length generalization.

A back-of-envelope speedup

Suppose patchout keeps ρ = 0.55 of tokens and the model is attention-dominated. Per step the attention work scales by ρ² ≈ 0.30, so you do ~70% less attention compute. Even accounting for the length-independent parts (patch embed, layer norms, the head), realistic end-to-end speedups land in the ~2× range, with the memory win often enabling an additional batch-size speedup on top.

Chapter 8: The Patchout Studio

This is the payoff. Below is a live PaSST front end. You control the spectrogram, the patchout flavour, and the drop level; the studio shows the patch grid, the surviving tokens, the resulting sequence length, the relative attention cost (ρ²), and a memory bar — all updating together. Break it: crank the drop to the extreme, switch between unstructured and structured, and watch the quadratic lever move.

Things to notice as you play:

• The ρ² magnification. Slide the drop from 0.0 to 0.4 (keep 0.6) and the memory bar drops to about 36%, not 60%. The quadratic is doing the work.

• Structured carves stripes. Switch to structured and the dark patches snap into whole rows and columns — the audio-aligned augmentation. Unstructured scatters them.

• CLS and DIST survive. The two special tokens are always counted into the kept total no matter how aggressive you get — they must reach the classifier head.

• Diminishing returns past the cliff. Push drop above ~0.65 and you're in the "starved" regime — the studio flags it. In a real run this is where accuracy would start to fall.

Chapter 9: Limits & Connections

PaSST is one idea executed cleanly, which is also where its limits live. Let's name them honestly.

Limitations

It still inherits the transformer's quadratic floor. Patchout shrinks T by a constant factor; it does not change the O(T²) scaling. For very long audio (minutes, not seconds) you eventually need linear-attention or hierarchical methods — patchout buys headroom, not a new asymptote.

It leans on ImageNet/DeiT pretraining. Like AST, the strong results assume a vision-pretrained backbone fine-tuned on audio. Patchout reduces but does not eliminate that dependence.

The drop rate is a hyperparameter. The inverted-U means there's a real sweet spot to tune; too little wastes the regularizer, too much starves the model. It is robust but not automatic.

Inference is still full-length. The speed and memory wins are training-time. A deployed PaSST processes all patches, so it is no cheaper at test time than AST — patchout is about making training affordable.

Where PaSST went

PaSST became the field's default distillation teacher. Because it trains cheaply to high accuracy, it is the natural large model to distill into small, efficient students — the standard recipe in the DCASE challenges for low-complexity acoustic scene classification. The DIST token it inherited from DeiT is precisely the hook for this: the teacher's predictions supervise a dedicated distillation head, and the resulting student CNNs/transformers carry PaSST's knowledge at a fraction of the cost.

Cheat sheet

Connections — go deeper

PaSST sits at a crossroads of three lineages. Follow them:

And two lateral ideas worth a glance: diffusion shares the "train on corrupted inputs, test on clean" philosophy, and the regularization story rhymes with classic dropout in GPT-style training.