EnCodec: High Fidelity Neural Audio Compression

Chapter 1: What a Codec Actually Is

You now know the budget: 80 bits per 13.3 ms at 6 kbps. Before watching a neural network spend it, watch how fifty years of signal processing spent it — because EnCodec is not a rejection of that tradition, it is a replacement of exactly one stage of it.

The paper compresses the whole tradition into one sentence: "Audio codecs typically employ a carefully engineered pipeline combining an encoder and a decoder to remove redundancies in the audio content and yield a compact bitstream. Traditionally, this is achieved by decomposing the input with a signal processing transform and trading off the quality of the components that are less likely to influence perception."

Unpack that into four stages. Every classical codec — MP3, AAC, Vorbis, Opus — is some arrangement of these:

EnCodec keeps stages 3 and 4 nearly unchanged in spirit. It learns stage 1, and it replaces stage 2 with a trained discriminator. That is the whole conceptual move.

Masking: the idea that made MP3 possible

Stage 2 deserves a paragraph on its own, because it is genuinely beautiful engineering and because understanding it tells you exactly what a learned system has to reinvent.

Auditory masking is the fact that a loud sound makes nearby quieter sounds inaudible. Play a 1 kHz tone at 80 dB and a 1.1 kHz tone at 40 dB simultaneously, and you hear only the first. The second is masked. The masking effect spreads across frequency (asymmetrically — more upward than downward) and across time (a loud transient masks quiet sounds for a few milliseconds before it and tens of milliseconds after it).

So a psychoacoustic model computes, for each short frame, a masking threshold: a curve in dB across frequency, below which anything you add is inaudible. The quantizer then allocates bits so that the quantization noise sits just below that curve. Loud frequency regions get coarse quantization (their noise is hidden anyway); exposed quiet regions get fine quantization.

Masking, with numbers

The sim above is qualitative; do one bit-allocation by hand so the mechanism is not a black box. Consider a single frame with five frequency bands, their measured levels, and a masking threshold computed from a 78 dB masker sitting in band 2:

The rule of thumb every classical codec uses: quantization noise is roughly 6 dB below the signal for each bit you spend, so the bits you need in a band are the headroom divided by 6, rounded up:

Band 0: 12 / 6 = 2 bits. Band 1: 3 / 6 = 0.5, round up to 1 bit. Bands 3 and 4 have negative headroom — they sit under the mask, so anything you transmit there is inaudible next to the masker and gets zero bits. Total for this frame: 2 + 1 + 4 + 0 + 0 = 7 bits instead of the 20 a flat allocation would have spent.

That is the whole classical bargain, and it is a good one. But notice what it depends on: the threshold curve, which came from listening experiments on isolated tones, and the 6-dB-per-bit rule, which assumes the quantization noise is white and uncorrelated with the signal. Both assumptions weaken as the bitrate falls, because at very low rates the error is no longer a small perturbation — it is comparable to the signal itself, and "noise hidden under the mask" becomes "the signal has been replaced by something else."

Here is the same allocation as code, so the comparison with the neural version later is concrete:

python — classical perceptual bit allocation, complete
import math

levels    = [42, 55, 78, 61, 47]     # dB per band, this frame
threshold = [30, 52, 0,  65, 52]     # from the psychoacoustic model

def bits_for(level, thresh):
    headroom = level - thresh
    return max(0, math.ceil(headroom / 6.0))   # ~6 dB of SNR per bit

alloc = [bits_for(l, t) for l, t in zip(levels, threshold)]
print(alloc, sum(alloc))    # [2, 1, 13, 0, 0]  -> masker band capped separately

# The entire perceptual model is the `threshold` list. It is a FIXED prior:
# the same curve shape for speech, for a snare drum, and for birdsong.

Read the last comment twice. Every line of that function is transparent, auditable, and standardised — which is exactly why Opus works identically on every device on earth and will still work in twenty years. The learned alternative buys quality at low bitrate and pays with opacity and a training distribution. That is not a small trade, and it is worth naming before we spend eight chapters admiring the learned side.

The parametric branch, and why it stalled

There is a second, older tradition that the paper cites in its first Audio Codec paragraph: parametric coding. Instead of transmitting a transform of the waveform, transmit the parameters of a model of how the sound was produced, and resynthesise at the far end.

For speech this is linear predictive coding, going back to Atal & Hanauer in 1971. Model the vocal tract as an all-pole filter, transmit the filter coefficients plus a description of the excitation (the buzz from the vocal folds or the hiss of a fricative), and rebuild the waveform. The filter coefficients are cheap. The result can run at extremely low bitrates.

The paper is blunt about the outcome: these methods have long been studied "but their quality has been severely limited. Despite some advances, modeling the excitation signal has remained a challenging task." The filter is easy; the excitation — the actual rich, noisy, non-stationary source signal — is where all the perceptual information lives, and hand-designed excitation models sound buzzy and synthetic.

The two baselines you will see in every table

The neural lineage that leads directly to EnCodec

The paper’s related-work section is a short history you should be able to recite. Each entry fixed one thing and left one thing broken:

Read the last two rows together and EnCodec’s contribution list becomes obvious. SoundStream established the recipe. EnCodec asks: can we make it simpler (one discriminator family instead of two), more stable (the balancer), and smaller on the wire (entropy coding)? The paper’s own contribution list is precisely those three plus the extensive MUSHRA study.

Why a learned transform can beat a designed one

Here is the argument, stated carefully, because "neural networks are better" is not an argument.

The MDCT is a fixed linear transform. It is optimal for signals that are locally stationary and sinusoidal, which describes a lot of music and very little of anything else. When the signal is a transient, a plosive, a room reverberation tail, or three sources mixed together, the MDCT’s energy compaction degrades: the "handful of large coefficients" becomes dozens, and the bit budget explodes.

A learned encoder is a nonlinear, data-adaptive transform. It can devote latent dimensions to whatever structures actually recur in its training distribution — and EnCodec’s training distribution is deliberately enormous and mixed: speech, noisy speech, music, general environmental audio, plus on-the-fly mixtures of two or three sources (Chapter 8 has the exact sampling probabilities). Whatever regularities exist in that distribution become cheap to encode.

The cost is equally real, and the paper names it in the introduction: "the model has to represent a wide range of signals, such as not to overfit the training set or produce artifact laden audio outside its comfort zone." A fixed transform has no comfort zone; a learned one does. The paper’s two answers are a large and diverse training set, and discriminator networks acting as perceptual losses.

Concept check before you move on

Suppose you took EnCodec and swapped its learnt encoder for a plain MDCT, keeping the RVQ quantizer and the neural decoder. Would it still work?

Partly. You would keep the powerful synthesiser and the strong discrete bottleneck, so it would beat a classical codec at the same bitrate on synthesis quality. But you would lose the adaptivity: the MDCT coefficients for a hard signal are high-dimensional and poorly clustered, so the codebooks would have to cover a much messier space and the residual would fall more slowly with each stage (Chapter 4 makes "falls more slowly" a measurable thing). The encoder’s job is not just to transform — it is to produce a latent whose geometry is friendly to quantization. That is a joint-training effect and it is why the whole system is trained end to end.

Chapter 2: The Encoder and the Decoder

Time to open box 1 and box 3. The paper describes them in a single dense paragraph; we will spend a chapter unfolding it, because every constant in it is load-bearing and several of them explain results four chapters later.

Here is the paragraph, quoted, so you can check the unfolding against the source:

Walking the encoder, layer by layer, with shapes

Substitute C = 32, B = 4, strides (2, 4, 5, 8), and one second of 24 kHz mono audio. Track two things at every step: the number of channels, and the number of time steps.

Row 9 is the punchline: 75 latent steps per second at 24 kHz, and 150 at 48 kHz (48000 / 320 = 150). The paper states both figures explicitly and uses the same architecture for both sample rates — the frame rate simply doubles.

Note the channel/time trade in the middle rows. Time steps fall by 320× while channels rise by 16× (32 → 512). The total activation volume shrinks by 20× through the encoder, which is the compression happening before a single bit is spent. The quantizer inherits an already-compact representation.

The decoder is the encoder, run backwards

"The decoder mirrors the encoder, using transposed convolutions instead of strided convolutions, and with the strides in reverse order as in the encoder, outputting the final mono or stereo audio."

So the decoder strides are (8, 5, 4, 2). Starting from [B, D, 75] the transposed convolutions upsample by 8, then 5, then 4, then 2 — back to 24000 — while channels halve at each stage and a final kernel-7 convolution produces C_a output channels. A two-layer LSTM sits on the decoder side too: the paper says the sequential modelling component is applied over the latent representation "both on the encoder and on the decoder side."

Activation is ELU throughout, and normalisation is either layer normalisation or weight normalisation — and which one is not a stylistic choice. It is the hinge on which the streaming variant turns.

The two variants: streamable and non-streamable

This is the subtlest engineering in the paper and it is worth slowing down for, because it is the difference between a codec you can use in a phone call and one you can only use for archiving.

A convolution needs context on both sides of the sample it is producing. In the ordinary (non-causal) setup you pad symmetrically: for a total padding of K − S, you split it equally before the first time step and after the last one, with one extra before if K − S is odd. Output step t then depends on input samples both before and after t. That is fine offline. In a live stream it is fatal — you would have to wait for the future.

The payoff of the causal padding scheme is stated exactly: "the model can output 320 samples (13 ms) as soon as the first 320 samples (13 ms) are received." One frame in, one frame out. Table 5 reports the initial latency of the 24 kHz streaming model as 13.3 ms, and the 48 kHz non-streaming version at 1 second — that whole second is the 1-second chunk needed to compute the normalisation statistics.

What does streaming cost in quality? Table 3, at 6 kbps on an equal mix of speech and music:

Streaming costs 0.79 dB of SI-SNR and 0.04 ViSQOL. The paper’s verdict: "we notice a small degradation switching from non-streamable to streamable but the performance remains strong while this setting enables streaming inference." Meanwhile both EnCodec variants sit around 4.35 ViSQOL where the classical codecs sit at 2.6–2.7 — the streaming penalty is a rounding error next to the method gap.

The latency budget, added up

"13 ms" is quoted so often that it is worth deriving, because latency in a real call is a sum of several things and only one of them is the codec’s algorithmic delay.

Compare the alternatives on that axis alone. Opus at its default runs 20 ms frames plus 2.5 ms look-ahead. MP3 needs a 1152-sample granule — 26 ms at 44.1 kHz — plus the MDCT overlap. EnCodec’s 13.3 ms is competitive with the best low-latency configurations of either, which is remarkable for a neural system and is entirely due to the padding scheme in the previous section.

Ablating the architecture: what each piece is worth

Table A.3 in the appendix takes the base streamable 24 kHz model at 6 kbps and changes one thing at a time. RTF is the real-time factor: the ratio of audio duration to processing time, so RTF > 1 means faster than real time. Profiled on a single thread of a 2019 MacBook Pro CPU.

Three readings, in order of how much they teach:

1. Capacity has brutally diminishing returns. Doubling channels from 32 to 64 buys 0.03 dB of SI-SNR and 0.03 ViSQOL — and costs 7.5× the encoding speed (9.8 → 1.3). Halving to 16 loses 0.27 dB and triples the speed. The paper’s phrasing: "increasing the capacity of the model only marginally affects the scores on objective metrics while it has a high impact on the real-time factor." C = 32 is the knee of the curve.

2. The LSTM is cheap quality. Removing it gains 53% encoding speed and loses 0.27 dB. Replacing it with extra residual units (3 res layers, no LSTM) is worse on both axes: 6.32 SI-SNR and slower than the base model. Recurrence over 75 latent steps per second is a far better use of parameters than more convolution at the waveform rate — because at 75 Hz the LSTM is doing long-range work that a stack of small kernels would need enormous depth to reach.

3. Some normalisation beats none. "norm = None" costs 0.22 dB and 0.06 ViSQOL for essentially no speed gain. The paper notes exactly this: "We notice a small gain over the objective metrics by keeping a form of normalization."

The residual unit in code

Every convolution block is one residual unit followed by one downsample. The residual unit is small enough to write out completely:

python — the residual unit and one encoder block
import torch.nn as nn

class ResidualUnit(nn.Module):
    # "two convolutions with kernel size 3 and a skip-connection"
    def __init__(self, ch):
        super().__init__()
        self.block = nn.Sequential(
            nn.ELU(),
            nn.Conv1d(ch, ch, kernel_size=3, padding=1),   # causal in the streaming variant
            nn.ELU(),
            nn.Conv1d(ch, ch, kernel_size=3, padding=1),
        )
    def forward(self, x):
        return x + self.block(x)          # the skip connection

class EncoderBlock(nn.Module):
    # "a single residual unit followed by a down-sampling layer ...
    #  kernel size K of twice the stride S ... channels doubled"
    def __init__(self, ch_in, stride):
        super().__init__()
        self.res  = ResidualUnit(ch_in)
        self.down = nn.Conv1d(ch_in, 2 * ch_in,
                              kernel_size=2 * stride, stride=stride)
    def forward(self, x):
        return self.down(nn.functional.elu(self.res(x)))

# the whole encoder, exactly as the paper specifies it
strides = (2, 4, 5, 8)                 # product = 320
C, D = 32, 128          # paper fixes C=32; it writes the latent width only as "D"
layers = [nn.Conv1d(1, C, kernel_size=7, padding=3)]
ch = C
for s in strides:
    layers.append(EncoderBlock(ch, s)); ch *= 2   # 32->64->128->256->512
layers += [LSTMWrapper(ch, num_layers=2),
           nn.Conv1d(ch, D, kernel_size=7, padding=3)]
encoder = nn.Sequential(*layers)

The causal variant differs only in where the padding goes: replace each symmetric padding=p with a left-only pad of K − S applied before the convolution. That single change — plus swapping layer norm for weight norm — converts the offline model into the 13 ms streaming model. The weights have the same shapes; the graph has the same layers.

Chapter 3: Vector Quantization, Built From Zero

The encoder hands you a tensor of shape [B, D, 75] full of 32-bit floats. If you transmitted it as-is, one second of mono audio at D = 128 would cost 128 × 75 × 32 = 307,200 bits — 307 kbps, barely better than the raw waveform. The encoder compressed the volume; it did nothing about the precision.

So the real question of this chapter: how do you turn a continuous vector into a small integer, cheaply, and in a way that a gradient can flow back through?

Scalar quantization, and why it is not enough

The obvious answer is to round each of the D numbers to a grid. With b bits per dimension you pay D · b bits per frame. At D = 128 and even a stingy b = 1 you are at 128 bits per frame — 9.6 kbps — and one bit per dimension is a catastrophically coarse grid.

Scalar quantization also throws away every correlation. If two latent dimensions always move together, a per-dimension grid spends bits describing both independently. Real latents are full of such structure; that is what makes them a good representation in the first place.

Vector quantization: name a point, not a coordinate

The alternative is to quantize the whole vector at once. Keep a codebook: a list of N learnt vectors c₀, …, c_N−1, each in R^D. To encode a latent z, find the nearest codebook entry and transmit its index:

The cost is log₂(N) bits regardless of D. That is the entire magic: dimensionality becomes free. A 128-dimensional vector and a 2-dimensional vector both cost 10 bits if the codebook has 1024 entries. What you pay for instead is coverage — the codebook must contain a point near wherever your latents actually live.

Geometrically, a codebook partitions R^D into N Voronoi cells: the set of points closer to c_k than to any other entry. Encoding is "which cell am I in"; decoding is "here is that cell’s representative point". The quantization error is the vector from z to its cell’s centre, and the codebook is good exactly when those errors are small on the distribution you care about.

The wall: why one codebook cannot reach 24 kbps

Now do the arithmetic that forces the rest of the paper. You want 24 kbps at 75 frames per second, so 320 bits per frame. With a single codebook that means:

There are roughly 10⁸⁰ atoms in the observable universe. Even at the modest 6 kbps target you would need 2⁸⁰ ≈ 1.2 × 10²⁴ entries, each a D-dimensional float vector. Three separate walls:

This is the moment the reader should feel the need for the next chapter. A single codebook is a flat code: one index, one point. To get to hundreds of bits per frame you need a code whose effective size grows multiplicatively while its storage grows additively. That is exactly what residual vector quantization is, and it is Chapter 4.

How good can a codebook be? The rate–distortion view

Before fixing VQ’s training problems, it helps to know what a perfect codebook would achieve, so you can tell whether your codebook is bad or the problem is hard.

Suppose your latents fill a D-dimensional region of volume V roughly uniformly. A codebook of N entries partitions that region into N cells, so each cell has volume about V/N. A cell of volume v in D dimensions has a characteristic radius proportional to v^1/D. Therefore:

Substitute N = 2^b for b bits and take logarithms:

Read that exponent carefully, because it is the whole story of this chapter. The error falls with b/D, not b. Halving the error needs D extra bits, not one. At D = 128, buying a factor of two in accuracy costs 128 bits per frame — 9.6 kbps of your budget for one halving.

You can watch this exact effect in the Voronoi sim: cluster the codebook onto the data (k-means) and the mean error drops sharply at the same N, because the entries stop wasting themselves on empty space. Randomly scattered entries behave as if the effective dimension were much higher than it is.

Three problems VQ has, and the three fixes EnCodec uses

Before stacking codebooks, the single-codebook version must actually train. Three things go wrong, and the paper fixes each with a named technique. It follows the same procedure as Dhariwal et al. (Jukebox) and Zeghidour et al. (SoundStream).

Problem 1: argmin has no gradient. The map z → c_k is piecewise constant. Its derivative is zero almost everywhere and undefined on the cell boundaries. Backpropagate through it honestly and the encoder receives nothing at all.

Fix 1: the straight-through estimator. Quoting the paper: "We use a straight-through-estimator to compute the gradient of the encoder, e.g. as if the quantization step was the identity function during the backward phase." Forward, you quantize. Backward, you pretend you did not. In code this is a one-line trick:

python — straight-through in one line
z_q = z + (quantize(z) - z).detach()
# forward:  z + (q - z) == q          -> the decoder sees the quantized vector
# backward: d/dz of the detached term is 0, so grad flows straight to z

It is a biased estimator — the true gradient of a step function is not the identity — but the bias is small when quantization error is small, which is precisely the regime the commitment loss enforces.

Problem 2: the encoder can run away from the codebook. Nothing in the reconstruction loss stops the encoder from drifting its outputs to a region no codebook entry covers. The straight-through gradient does not feel quantization error at all, so the encoder is free to make that error enormous.

Fix 2: the commitment loss. "A commitment loss, consisting of the MSE between the input of the quantizer and its output, with gradient only computed with respect to its input, is added to the overall training loss." So it penalises the encoder for producing latents far from the codebook, while leaving the codebook itself untouched by this term (the codebook has its own update rule — fix 3). The name is apt: the encoder must commit to the code it triggered.

Problem 3: codebook collapse. Entries that start far from the data are never selected, so they are never updated, so they stay far from the data forever. A 1024-entry codebook can silently degenerate into a 40-entry one, and your 10 bits per frame become 5.3.

Fix 3: EMA updates plus dead-entry restarts. "The codebook entry selected for each input is updated using an exponential moving average with a decay of 0.99, and entries that are not used are replaced with a candidate sampled from the current batch." The EMA makes each entry drift toward the running mean of the vectors assigned to it — a soft, online k-means. The restart is the crucial half: any entry that goes unused gets teleported onto a real data point from the current batch, where it will immediately win some assignments.

Worked example: quantizing one vector by hand

Take D = 2 so the arithmetic is visible, and a tiny codebook of four entries:

and the latent z = (0.9, −0.4). Compute all four squared distances, every term shown:

The minimum is 0.17, at index 0. So the transmitted symbol is the integer 0, costing log₂(4) = 2 bits, and the decoder reconstructs c₀ = (1.0, 0.0).

The error vector is z − c₀ = (0.9 − 1.0, −0.4 − 0.0) = (−0.1, −0.4), with length √0.17 = 0.4123. For scale, ‖z‖ = √(0.81 + 0.16) = √0.97 = 0.9849, so we have kept the vector to within 41.9% relative error. That is what 2 bits buys.

The commitment loss contribution from this one vector is exactly the squared distance we already computed: ‖z − q(z)‖²₂ = 0.17.

Hold onto the error vector (−0.1, −0.4). In Chapter 4 we will quantize it, with a second codebook, and watch 41.9% become 10.2%.

The same computation, three ways

python — step by step, exactly the hand arithmetic
import numpy as np

z  = np.array([0.9, -0.4])
cb = np.array([[1.0, 0.0],
               [0.0, 1.0],
               [-1.0, 0.0],
               [0.0, -1.0]])

d2 = []
for c in cb:
    diff = z - c
    d2.append(diff[0]**2 + diff[1]**2)
print(d2)                # [0.17, 2.77, 3.77, 1.17]

k = int(np.argmin(d2))   # 0
q = cb[k]                # [1.0, 0.0]
err = z - q              # [-0.1, -0.4]
print(k, q, err, np.linalg.norm(err))   # 0 [1. 0.] [-0.1 -0.4] 0.41231

python — vectorised, the form you would actually ship
# distances for a whole batch Z of shape [M, D] against codebook [N, D]
d2 = ((Z[:, None, :] - cb[None, :, :])**2).sum(-1)      # [M, N]
idx = d2.argmin(-1)                                       # [M]
q   = cb[idx]                                              # [M, D]

python — the one-liner, using the expansion of the square
# ||z-c||^2 = ||z||^2 - 2 z.c + ||c||^2 ; ||z||^2 is constant per row, so drop it
idx = (cb @ Z.T * 2 - (cb**2).sum(-1, keepdims=True)).argmax(0)

That last form is what real implementations use: one matrix multiply instead of an M × N × D broadcast, which matters when N = 1024 and you are running 75 times a second on one CPU core.

Chapter 4: Showcase — The Residual Vector Quantization Walk

Chapter 3 left you with an error vector: after spending 2 bits on z = (0.9, −0.4) we were still off by (−0.1, −0.4). The obvious question — so obvious that it took the field from 1984 to 2021 to make it work in a neural codec — is: why not quantize the error too?

That is residual vector quantization, in one sentence. The paper’s version: "Vector quantization consists in projecting an input vector onto the closest entry in a codebook of a given size. RVQ refines this process by computing the residual after quantization, and further quantizing it using a second codebook, and so forth."

Each stage has its own codebook, trained on the residuals that reach it. Stage 1’s codebook learns the coarse shape of the latent distribution. Stage 2’s learns the shape of stage 1’s mistakes. Stage 3’s learns the shape of stage 2’s mistakes. It is boosting, applied to quantization.

The multiplicative miracle

Here is why this breaks the wall from Chapter 3. With N_q codebooks of N entries each, the set of reachable reconstruction points is every possible sum of one entry from each codebook:

Multiplicative in the exponent, additive in storage. Put EnCodec’s numbers in: N = 1024, N_q = 32.

Both codes address the same number of points. One requires more storage than there is matter; the other fits in 32,768 × D floats — about 16 MB at D = 128 in float32 — and searches in 32 × 1024 = 32,768 distance evaluations per frame, which a CPU does comfortably 75 times a second.

Hand-worked: two stages, every intermediate number

Pick up exactly where Chapter 3 stopped. z = (0.9, −0.4), and stage 1 uses the same four-entry codebook.

Stage 1. Residual entering the stage is r₁ = z = (0.9, −0.4). Codebook C₁:

Squared distances, recomputed so this chapter stands alone:

So k₁ = 0, and the new residual is:

Stage 2. Its codebook C₂ was trained on residuals, so its entries are small — that is the crucial structural fact, and it is why stage 2 cannot simply be a copy of stage 1:

Squared distances from r₂ = (−0.1, −0.4), every term written out:

So k₂ = 1, and:

The reconstruction. The decoder receives the two integers (0, 1) and sums the corresponding entries — it never sees a residual:

Check against the true z = (0.9, −0.4): the error is (0.0, −0.1), norm 0.1 — which is exactly r₃, as it must be. The final residual is the reconstruction error. That identity is the whole reason the algorithm is written as a loop over residuals.

The scoreboard. Relative error against ‖z‖ = 0.9849:

Two bits took you from 100% error to 41.9%. Two more took you from 41.9% to 10.2%. Each stage cut the residual by roughly a factor of four, using the same number of bits — because each stage’s codebook is scaled to the residuals it actually receives.

And the commitment loss for this frame, from Equation 3 — the sum over residual steps of the squared distance from the residual to its chosen entry, which are the two winning distances we already computed:

THE SHOWCASE — walk the residual yourself

Two behaviours to notice while you play. First, the arrows get shorter every stage — geometric decay, roughly a constant factor per stage, which on the log plot is a straight line. Second, drag the target far outside the data cloud and the decay stalls: the codebooks were trained on residuals from a particular distribution and have nothing useful to offer an outlier. That is the "comfort zone" failure the paper worries about in its introduction, made visible.

Bandwidth is a runtime knob, not a retrain

Now the property that made EnCodec practical. "By selecting a variable number of residual steps at train time, a single model can be used to support multiple bandwidth target."

Because the stages are strictly ordered — stage c only ever sees what stages 1 through c−1 left behind — you can simply stop early. Transmit the first 8 indices instead of all 32 and the decoder reconstructs from 8; quality degrades gracefully rather than catastrophically. The bitstream is nested: the 1.5 kbps stream is a literal prefix of the 24 kbps stream.

The paper trains for this explicitly: "When doing variable bandwidth training, we select randomly a number of codebooks as a multiple of 4, i.e. corresponding to a bandwidth 1.5, 3, 6, 12 or 24 kbps at 24 kHz." Configuration: at most 32 codebooks (16 for the 48 kHz models) with 1024 entries each, "e.g. 10 bits per codebook."

The 48 kHz model supports 3, 6, 12 and 24 kbps. Check the arithmetic there too: 150 frames/s × 10 bits × N_q = 1500 · N_q bits/s, so N_q = 2, 4, 8, 16 — and 16 is stated as the 48 kHz maximum. Everything closes.

RVQ in code, three ways

Encode, exactly the loop from the derivation:

python — RVQ encode, step by step
import numpy as np

def rvq_encode(z, codebooks):
    """z: [D]. codebooks: list of [N, D]. returns list of int indices."""
    residual, indices = z.copy(), []
    for C in codebooks:
        d2 = ((C - residual)**2).sum(axis=1)     # [N]
        k  = int(d2.argmin())
        indices.append(k)
        residual = residual - C[k]              # the ONLY state carried forward
    return indices, residual                    # residual == reconstruction error

def rvq_decode(indices, codebooks):
    return sum(C[k] for k, C in zip(indices, codebooks))

# reproduce the hand-worked example exactly
C1 = np.array([[1.,0.], [0.,1.], [-1.,0.], [0.,-1.]])
C2 = np.array([[.2,.1], [-.1,-.3], [0.,.4], [-.3,.05]])
idx, res = rvq_encode(np.array([0.9, -0.4]), [C1, C2])
print(idx, res, np.linalg.norm(res))
# [0, 1] [ 0.  -0.1] 0.1     <- matches the hand derivation
print(rvq_decode(idx, [C1, C2]))   # [0.9 -0.3]

The training-time forward pass, with straight-through and the commitment loss assembled — note that only two lines differ from inference:

pytorch — RVQ forward with straight-through and commitment
def rvq_forward(z, codebooks, n_q):
    """z: [B, D, T]. Returns quantized z_q, index tensor, commitment loss."""
    residual = z
    z_q      = torch.zeros_like(z)
    commit   = z.new_zeros(())
    idxs     = []
    for c in range(n_q):                       # n_q chosen per batch: 2,4,8,16,32
        C   = codebooks[c]                        # [N, D]
        flat = residual.permute(0,2,1).reshape(-1, C.shape[-1])
        d2  = (flat.pow(2).sum(-1, keepdim=True)
               - 2 * flat @ C.T
               + C.pow(2).sum(-1))
        k   = d2.argmin(-1)
        q   = C[k].view_as(flat).reshape(residual.permute(0,2,1).shape).permute(0,2,1)
        # Eq. 3: MSE between the residual and its quantized value,
        # gradient only w.r.t. the residual -> detach the codebook side
        commit = commit + torch.nn.functional.mse_loss(residual, q.detach())
        z_q      = z_q + q
        residual = residual - q.detach()          # codebooks learn by EMA, not by grad
        idxs.append(k)
    z_q = z + (z_q - z).detach()                  # straight-through to the encoder
    return z_q, torch.stack(idxs, 1), commit

And the library one-liner, because in practice nobody writes the loop:

python — the shipped API
from encodec import EncodecModel
model = EncodecModel.encodec_model_24khz()
model.set_target_bandwidth(6.0)          # 1.5 | 3 | 6 | 12 | 24 -> picks n_q
frames = model.encode(wav)                 # [(codes [B, n_q, T], scale)]
wav_hat = model.decode(frames)

What happens between the quantizer and the decoder

One sentence in the paper is easy to skim past and it matters: "This discrete representation can changed again to a vector by summing the corresponding codebook entries, which is done just before going into the decoder."

So the decoder does not receive N_q separate streams, and it does not receive integers. It receives a single [B, D, T] float tensor that is the sum of the selected entries. The decoder cannot tell whether that tensor came from 2 codebooks or 32 — it only sees a vector that is closer or further from what the encoder produced. This is why a single decoder handles all five bitrates without any conditioning: the bitrate manifests purely as how accurate the input is.

Chapter 5: The Loss Anatomy

You now have an encoder that compresses, a quantizer that discretises, and a decoder that reconstructs. Nothing so far explains why the output sounds good. That is entirely the training objective’s job, and EnCodec’s objective has four terms plus a commitment term, each answering a different failure mode.

Start with the failure modes, so each term is demanded before it appears.

So you need all three, and the paper uses exactly all three plus feature matching. Here is Equation 4, the generator objective, with every symbol defined:

At 48 kHz the paper changes only two: λ_g = 4 and λ_feat = 4.

Term 1: the time-domain loss

Plain L1 between waveforms. L1 rather than L2 because L1’s gradient has constant magnitude, so quiet passages get the same corrective pressure as loud ones — L2 would let the model ignore anything below the noise floor of the loudest section. Its weight is the smallest in the objective (0.1), which tells you the authors regard it as an anchor rather than a driver.

Term 2: the multi-scale mel loss, unpacked symbol by symbol

This is Equation 1, and it looks worse than it is:

Write out the seven scales explicitly, because "multi-scale" stays abstract until you do:

And why mel bins rather than linear frequency bins? Because the mel scale is roughly logarithmic above 1 kHz, matching how human frequency discrimination degrades at high frequencies. Sixty-four mel bins put most of their resolution where hearing is sharpest. This is the one place where classical psychoacoustics survives inside EnCodec — not as a masking model, but as the choice of axis the loss is measured on.

Both L1 and L2 appear on each scale. L1 is robust and treats all bins alike; L2 punishes large individual errors hard. Using both means "get everything roughly right (L1) and never get anything badly wrong (L2)".

Working one mel-loss term by hand

Equation 1 is a double sum, which hides how small each piece is. Reduce it to three mel bins at one scale and compute the whole thing.

Say scale i = 8 (window 256) produces, for one time frame, the log-mel values:

The difference vector is (−2.0 − (−1.6), 1.5 − 1.9, 0.4 − 0.1) = (−0.4, −0.4, 0.3). Now the two norms:

With α_i = 1 the bracket in Equation 1 is 1.1 + 1 × 0.6403 = 1.7403 for this scale.

Now the normaliser. There are |s| = 7 scales and |α| = 1 coefficient set, so the prefactor is 1 / (1 × 7) = 1/7. If all seven scales happened to give the same 1.7403, the total would be (1/7) × 7 × 1.7403 = 1.7403 — unchanged. That is the point of the normaliser: adding scales must not inflate the loss, or λ_f = 1 would silently mean something different for a five-scale model than a seven-scale one.

Notice the two norms disagreeing on what matters. If instead the difference had been (0, 0, 1.1) — the same L1 of 1.1 concentrated in one bin — the L2 would be 1.1 rather than 0.6403, and the bracket would jump to 2.2. Same total error, 26% higher loss, purely because it is concentrated. That is exactly the behaviour you want: a model that gets one band badly wrong is worse than one that is slightly off everywhere, because a single wrong band is an audible artifact while a diffuse error is a slight colouration.

python — Equation 1, complete
import torch, torchaudio

SCALES = range(5, 12)                 # e = 5..11 -> windows 32 .. 2048

mels = {i: torchaudio.transforms.MelSpectrogram(
            sample_rate=24000,
            n_fft=2**i,                    # window size 2^i
            hop_length=(2**i) // 4,         # hop 2^i / 4
            n_mels=64,                     # "a 64-bins mel-spectrogram"
            normalized=True)                # "a normalized STFT"
        for i in SCALES}

def multi_scale_mel_loss(x, x_hat, alphas=(1.0,)):
    total = 0.0
    for a in alphas:                    # the paper uses alpha_i = 1
        for i in SCALES:
            S, S_hat = mels[i](x), mels[i](x_hat)
            total = total + (S - S_hat).abs().mean() \
                          + a * (S - S_hat).pow(2).mean().sqrt()
    return total / (len(alphas) * len(list(SCALES)))    # the 1/(|alpha|·|s|) prefactor

Term 3: the MS-STFT discriminator

Reconstruction losses give you the right average. They do not give you realism, because averaging over the many plausible outputs produces something that is none of them — the classic blurring failure. The fix is a discriminator: a network trained to tell real audio from reconstructed audio, whose confusion becomes the generator’s reward.

EnCodec’s discriminator is the MS-STFT discriminator, and its architecture is specified precisely enough to rebuild:

Note also the asymmetric dilation and stride: dilated in time (1, 2, 4 — growing the temporal receptive field cheaply, so a sub-discriminator can judge rhythm and decay) and strided in frequency (halving the frequency axis each layer, since neighbouring frequency bins are highly redundant). The discriminator is built to look along time and summarise across frequency.

The generator’s adversarial loss is a hinge, averaged over the K discriminators:

and the discriminators minimise the corresponding hinge:

Read the hinge carefully. For real audio the discriminator wants D_k(x) ≥ 1; for fake audio it wants D_k(x̂) ≤ −1. Once a sample is on the correct side by a margin of 1 it contributes zero gradient. That is the hinge’s virtue over cross-entropy: it stops pushing on examples it has already won, which keeps the discriminator from running away.

It runs away anyway, sometimes. The paper: "Given that the discriminator tend to overpower easily the decoder, we update its weight with a probability of 2/3 at 24 kHz, and 0.5 at 48 kHz." A coin flip, biased, deciding whether the discriminator learns this step. Crude, effective, and honest — and note that the 48 kHz model needs the stronger handicap because a longer, higher-resolution STFT gives the discriminator more to work with.

Term 4: relative feature matching

The adversarial signal alone is a single scalar per sub-discriminator: too coarse and too unstable to train a decoder on. Feature matching adds a much richer signal by comparing the discriminator’s internal activations on real versus reconstructed audio. Equation 2:

where D_k^l is the l-th layer of the k-th discriminator, L is the number of layers, and "the mean is computed over all dimensions."

The word doing the work is relative: the denominator. Without it, layers with naturally large activations would dominate the sum, and the effective weighting of the loss would drift as the discriminator’s activation scales change during training. Dividing by the mean magnitude of the real activations makes each layer contribute a fraction of its own scale.

Which discriminator actually matters? The ablation that reverses a metric

Prior work stacked discriminators: the Multi-Scale Discriminator (MSD) on raw waveforms at several resolutions, the Multi-Period Discriminator (MPD) which reshapes the waveform into 2-D by period, and a single STFT discriminator. SoundStream used MSD + Mono-STFT. EnCodec asks whether all that machinery is necessary. Table 2:

Read the MPD row and then the MS-STFT+MPD row, and something should bother you.

MPD has the best SI-SNR in the table (7.35) and the worst MUSHRA (60.7). The waveform-fidelity metric and the human listeners point in opposite directions. Meanwhile MS-STFT has lower SI-SNR (6.67) and humans rate it 17 points higher.

The paper’s conclusion is deliberately modest: "using only a multi-scale STFT-based discriminator such as MS-STFTD, is enough to generate high quality audio. Additionally, it simplifies the model training and reduces training time. Including the MPD discriminator, adds a small gain when considering the MUSHRA score." That gain is 79.0 versus 77.5 — and the confidence intervals (±1.9 and ±1.8) overlap. They chose the simpler model. That is the right call on that evidence, and saying so is a small act of scientific hygiene.

Multi-bandwidth training has a discriminator wrinkle

One more detail, easy to miss and genuinely surprising: "We also noticed that using a dedicated discriminator per-bandwidth is beneficial to the audio quality. Thus, we select a given bandwidth for the entire batch, and evaluate and update only the corresponding discriminator."

So there is not one discriminator stack — there is one per supported bandwidth. This makes sense once you consider what the discriminator has to learn: the artifacts of a 1.5 kbps reconstruction are qualitatively different from those of a 24 kbps one. A single discriminator would have to be an expert on five different artifact distributions at once, and would end up mediocre at all of them. Splitting them costs training memory and buys quality; at inference the discriminators are discarded entirely, so it costs nothing at deployment.

Chapter 6: The Balancer — Making Loss Weights Mean Something

You have five loss terms and five weights. Set λ_t = 0.1, λ_f = 1, λ_g = 3, λ_feat = 3 as the paper does. Now answer a simple question: what fraction of the training signal comes from the adversarial loss?

The honest answer is that you have no idea. Not because the question is ill-posed, but because λ_g = 3 tells you nothing without knowing the natural scale of ℓ_g’s gradient — and that scale changes every batch, because the discriminator is itself being trained.

This is the problem the balancer solves, and it is the one genuinely new mechanism in the paper. The abstract calls it out: "We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss."

Manufacturing the need: a concrete disaster

Work the naive case with real numbers so the failure is not abstract. Suppose at some batch the four gradient norms measured at the model output x̂ are:

Do the division for one row so the arithmetic is fully visible: the mel term’s actual share is 3.000 / 7.802 = 0.3845, i.e. 38.5%. You intended 14.1%. The adversarial term you weighted most heavily (λ_g = 3, tied for the largest) delivers 15.4% of the signal — less than half its intended share — because its gradient happens to be small this batch. And the time-domain loss, which you meant to be a 1.4% anchor, is contributing 0.03%: effectively switched off.

Now make it worse in the way GAN training actually makes it worse. Suppose the discriminator gets sharper and ‖g_g‖ jumps 10× to 4.0. Recompute:

Nothing about your hyperparameters changed. The opponent changed, and your loss mixture silently swung from 15% adversarial to 65% adversarial. Every other term got squeezed toward irrelevance. This is precisely the "varying scale of the gradients coming from the discriminators" the paper names as its motivation, and it is why GAN training is famously twitchy.

The mechanism, derived

The construction has three moves. Follow each one and the final formula becomes inevitable rather than magical.

Move 1: measure the gradient at the model output, not at the parameters. Define, for each loss that depends only on the model output x̂:

Why at x̂ and not at the weights? Because all these losses meet at exactly one place — the output — and everything after that point is a single shared backward pass through the decoder. Rescale the gradients where they meet and you have rescaled their entire downstream contribution, with one backward pass instead of five. It is both the semantically right place and the cheap place.

Move 2: divide out each loss’s natural scale. Let ⟨‖g_i‖₂⟩_β be the exponential moving average of that gradient norm over recent training batches, with β = 0.999. Then g_i / ⟨‖g_i‖₂⟩_β is a vector of typical norm 1, pointing in the direction that loss wants to move the output.

The EMA rather than the instantaneous norm matters: dividing by the current batch’s own norm would make every loss contribute exactly its share every step, destroying the useful information that this particular batch is unusually hard for the mel loss. β = 0.999 means a time constant of about 1000 batches — slow enough to track the trend, far too slow to react to one batch.

Move 3: reassemble with weights that are now pure proportions. Given weights (λ_i) and a reference norm R, define the balanced gradient:

and backpropagate ∑_i g̃_i instead of ∑_i λ_i g_i. The paper sets R = 1 and β = 0.999.

Now check the property that makes it worth doing. Take the norm of one balanced term:

because ‖g_i‖ is, on average, equal to its own moving average. So each term contributes a gradient of norm exactly its weight share, independent of the loss’s units, its scale, or what the discriminator is doing this week. And if the lambdas sum to 1, each λ_i is literally the fraction of the model’s gradient coming from that loss. The paper: "If ∑_i λ_i = 1, then each weight can be interpreted as the fraction of the model gradient that come from the corresponding loss."

The one loss that does not fit

"All the generator losses from Eq. (4) fit into the balancer, except for the commitment loss, as it is not defined with respect to the output of the model."

This exclusion is not arbitrary and it is worth understanding, because it defines the balancer’s applicability. The commitment loss ℓ_w = ∑_c ‖z_c − q_c(z_c)‖²₂ is a function of the encoder’s output z, not the decoder’s output x̂. Its gradient does not pass through x̂ at all — it enters the graph at the quantizer and flows only backwards into the encoder.

So ∂ℓ_w/∂x̂ is undefined, the whole construction has nothing to normalise, and the commitment loss is simply added with a plain scalar weight in the ordinary way. The balancer is a tool for losses that share a common output node. That is a real restriction, and it is exactly why the paper states it explicitly rather than quietly.

The evidence: Table A.4

The appendix trains EnCodec (with the DiffQ quantizer, on the Jamendo music dataset) across many weight settings, with and without the balancer. A selection, with the paired rows adjacent:

Three things this table proves, in ascending order of importance:

1. The balancer helps even at sane settings. All-ones weights: 10.32 versus 6.16 SI-SNR. A 4 dB gap from a normalisation trick, at the setting a careful practitioner would try first.

2. The balancer converts catastrophes into inconveniences. At λ_g = 100 — an absurd setting, deliberately — the unbalanced run reaches −35.83 dB SI-SNR. Negative SI-SNR means the output is further from the target than silence would be: the model has diverged completely. The same setting with the balancer gives 8.41 dB and ViSQOL 4.05, which is a perfectly usable model.

3. The balanced column is nearly flat. Read down the "yes" rows: 10.32, 9.93, 8.41, 9.22. Across weight settings spanning two orders of magnitude, quality varies by under 2 dB. Read down the "no" rows: 6.16, 1.72, −35.83, −16.39. The balancer has turned a knife-edge hyperparameter into a plateau.

The balancer in code

pytorch — the balancer, complete
class Balancer:
    """weights: {name: lambda}.  R = reference norm.  beta = EMA decay."""
    def __init__(self, weights, R=1.0, beta=0.999):
        self.w, self.R, self.beta = weights, R, beta
        self.avg, self.count = {}, 0

    def backward(self, losses, x_hat):
        # 1. gradient of each loss w.r.t. the MODEL OUTPUT (not the params)
        grads = {}
        for name, loss in losses.items():
            g, = torch.autograd.grad(loss, [x_hat], retain_graph=True)
            grads[name] = g

        # 2. exponential moving average of each gradient NORM
        self.count += 1
        norms = {}
        for name, g in grads.items():
            n = g.norm(p=2).item()
            prev = self.avg.get(name, 0.0)
            self.avg[name] = self.beta * prev + (1 - self.beta) * n
            # bias correction, exactly as in Adam: early estimates are shrunk
            norms[name] = self.avg[name] / (1 - self.beta ** self.count)

        # 3. reassemble: each term gets norm R * lambda_i / sum(lambda)
        total_w = sum(self.w[k] for k in grads)
        out = torch.zeros_like(x_hat)
        for name, g in grads.items():
            scale = self.R * self.w[name] / total_w / (norms[name] + 1e-12)
            out = out + scale * g

        # 4. ONE backward pass through the decoder with the balanced gradient
        x_hat.backward(out)

# usage: the commitment loss is NOT passed in -- it does not depend on x_hat
balancer = Balancer({'t': 0.1, 'f': 1.0, 'g': 3.0, 'feat': 3.0})
balancer.backward({'t': l_t, 'f': l_f, 'g': l_g, 'feat': l_feat}, x_hat)
(lambda_w * l_commit).backward()          # added the ordinary way
optimizer.step()

Note the shape of step 4: five losses, but only one backward pass through the decoder and encoder. The per-loss autograd.grad calls stop at x̂, which is a tiny fraction of the graph. The balancer is close to free.

Where else this idea belongs

Any multi-task system where several losses meet at one tensor is a candidate: multi-task detection heads, VAE reconstruction versus KL, physics-informed networks with wildly different residual scales, distillation losses mixed with task losses. The signature symptom is "I retuned the weights and the model got worse in a way I cannot explain." That symptom means your weights are entangled with scales, and the balancer — or the same idea rebuilt for your graph — disentangles them.

Chapter 7: Entropy Coding — Getting Bits Back for Free

Chapter 0 flagged a weakness and promised to fix it here. EnCodec’s bitrate is constant: every frame costs exactly N_q × 10 bits whether it contains a cymbal crash or a held organ note. That is wasteful, because the code indices are not uniformly distributed and they are not independent across time.

The fix is the last stage of the classical pipeline that we have not yet used: lossless entropy coding. And this time the probability model is a small Transformer.

The information-theoretic floor, worked by hand

Shannon’s result is the whole justification, so make it concrete before generalising. A symbol you assign probability p costs log₂(1/p) bits to encode optimally. Rare symbols are expensive; predictable symbols are nearly free. The expected cost per symbol is the entropy:

Worked example. Take a toy codebook with four entries. If you know nothing, the distribution is uniform, p = (0.25, 0.25, 0.25, 0.25), and:

which is just log₂(4) — the fixed-rate cost. Now suppose a language model looks at the previous frame and predicts p = (0.60, 0.20, 0.15, 0.05). Compute every term:

So the same symbol stream now costs 1.533 bits instead of 2. The saving:

Scale that intuition to EnCodec: 1024 entries, uniform cost 10 bits. If a Transformer that has seen the previous frames can concentrate its prediction enough to reach an entropy of about 6.5 bits, you have saved 35%. The paper reports savings of "∼25–40%", so that is roughly the regime it operates in.

The language model, specified exactly

"We additionally train a small Transformer based language model with the objective of keeping faster than real time end-to-end compression/decompression on a single CPU core."

For scale: 5 layers × 200 channels is on the order of a few million parameters. GPT-2 small is 124 million. This is a probability model, not a generator, and the paper is explicit that its smallness limits it — a point that returns at the end of this chapter.

The input and output shapes, traced

The mechanics are unusual enough to walk carefully.

Input. "At train time, we select a bandwidth and the corresponding number of codebooks N_q. For a time step t, the discrete representation obtained at time t − 1 is transformed into a continuous representation using learnt embedding tables, one for each codebook, and which are summed. For t = 0, a special token is used instead."

That last box hides the paper’s most consequential approximation, and it states it in one sentence: "We thus neglect potential mutual information between the codebooks at a single time step."

Unpack it. In RVQ, codebook 2’s index at time t depends heavily on codebook 1’s index at time t — stage 2 is quantizing stage 1’s residual, so knowing stage 1’s choice tells you a lot about stage 2’s. A model that predicted them sequentially could exploit that. This model predicts all N_q in parallel from the same hidden state, conditioning only on the past, so that within-timestep information is thrown away.

Arithmetic coding: turning probabilities into bits

A probability model alone saves nothing — you need a coder that can actually spend a fractional number of bits on a symbol. Huffman codes cannot: they assign whole bits, so a symbol with p = 0.9 (ideal cost 0.152 bits) still costs at least 1. Arithmetic coding can, and EnCodec uses a range-based arithmetic coder (Pasco 1976; Rissanen & Langdon 1981).

The idea in one picture: represent the entire message as a single number in [0, 1). Start with the full interval. For each symbol, subdivide the current interval in proportion to the predicted probabilities and keep the sub-interval belonging to the symbol that actually occurred. After many symbols the interval is tiny, and the number of bits needed to name a point inside it is exactly the sum of the log₂(1/p) costs.

Trace it by hand with the toy distribution p = (0.60, 0.20, 0.15, 0.05) and the message [0, 2]:

Bits required to name a point in an interval of width w is log₂(1/w):

Check it against the per-symbol costs: log₂(1/0.60) + log₂(1/0.15) = 0.73697 + 2.73697 = 3.47394. Identical. Fixed-rate coding would have spent 2 + 2 = 4 bits. The saving is real and it is fractional — 3.474, not 3 or 4.

The floating-point trap, and how the paper defuses it

Here is a detail most papers would omit and this one spends a paragraph on, because it is the difference between a demo and a deployable codec.

An arithmetic decoder must reconstruct the exact same interval subdivisions the encoder used. That means the decoder’s language model must output bit-identical probabilities to the encoder’s. And it does not, in general.

The paper: "evaluation of the same model might lead to different results on different architectures, or with different evaluation procedures due to floating point approximations. This can lead to decoding errors as the encoder and decoder will not use the exact same code. We observe in particular that the difference between batch evaluation (e.g. all time steps at once), and the real-life streaming evaluation that occurs in the decoder can lead to difference larger than 10⁻⁸."

Read that carefully: the same model, the same weights, the same input — but evaluated all-at-once versus one-step-at-a-time, and the answers differ in the eighth decimal place because floating-point addition is not associative and the reduction orders differ. In an arithmetic coder, a discrepancy of 10⁻⁸ in a probability can flip which sub-interval a value falls into, and every subsequent symbol decodes to garbage.

And the honest hedge, in the paper’s own words: "although evaluations in more contexts would be needed for practical deployment." They are telling you this is mitigated, not solved. Cross-platform bit-exactness of neural network inference is an unsolved problem, and any codec whose correctness depends on it inherits that problem.

What it buys, and what it costs

The bandwidth savings, from Table 1 (24 kHz mono) and Table 4 (48 kHz stereo). The "entropy coded" column is the average bandwidth after coding — it is now variable, so only an average is meaningful:

The trend is monotone and the paper explains it: "We observe that for higher bandwidth, the compression ratio is lower, which could be explained by the small size of the Transformer model used, making hard to model all codebooks together." Forty percent at 1.5 kbps (2 codebooks) down to 19% at 24 kbps (32 codebooks, or 16 at 48 kHz). With more codebooks per step, and no within-timestep conditioning, a 5-layer model simply runs out of capacity.

Now the bill, from Table 5. Real-time factor is audio duration over processing time, so greater than 1 means faster than real time. All at 6 kbps, single thread of a 2019 MacBook Pro:

At 24 kHz, entropy coding drops the real-time factor from about 10 to 1.6 — a 6× slowdown — still faster than real time, but with almost no headroom. At 48 kHz it drops below 1: the system is now slower than real time and cannot be used for live streaming at all. The paper is direct: it "could also be used for archiving where real time processing is not required."

And latency: "using entropy coding increases the initial latency, because the stream cannot be 'flushed' with each frame, in order to keep the overhead small. Thus decoding the frame at time t, requires for the frame t + 1 to be partially received, increasing the latency by 13 ms." So the 24 kHz streaming latency roughly doubles, from 13.3 ms to about 26 ms.

Chapter 8: Data, Training, and the Verdict

Everything so far has been mechanism. This chapter is evidence: what the model was trained on, how, and what happened when humans listened.

The training distribution is a design decision

Chapter 1 argued that a learned transform wins inside its distribution and can fail outside it. So the choice of distribution is part of the architecture, and the paper treats it that way. The 24 kHz monophonic model is trained across four domains:

The 48 kHz fullband stereo model, by contrast, is trained on only 48 kHz music. Different product, different distribution.

The mixing strategy is where the paper manufactures the diversity it needs. Four strategies, sampled with these exact probabilities:

They sum to 1.00, as they must. Note the asymmetry in s4: three-way mixes exclude music. Three simultaneous pieces of music are not a signal anyone needs to encode and would only teach the model to handle spectral chaos at the expense of realistic cases.

On top of the mixing, four augmentations:

• Per-file normalisation followed by a random gain between −10 and +6 dB — so the codec cannot assume a fixed input level, which real audio never has.
• Clipping rejection: any sample that has been clipped is discarded, so the model never learns to reproduce clipping artifacts as if they were signal.
• Reverberation with probability 0.2, using room impulse responses from the DNS challenge, with RT60 in the range [0.3, 1.3] seconds — and explicitly not applied to single-source music samples, which already carry their own production reverb.
• Test-set construction into four categories: clean speech from DNS alone; clean speech mixed with an FSD50K sample; a Jamendo sample alone; a proprietary music sample alone. Those are the four columns of Table 1.

Training configuration, complete

Two of these deserve comment. β₁ = 0.5 is the GAN convention, not Adam’s default of 0.9: with an opponent that changes every step, a long momentum window averages over a moving target and slows adaptation. And 1-second examples is short — but recall that the encoder’s temporal receptive field is a few hundred milliseconds and the LSTM operates over only 75 steps in that second. Longer clips would buy little and cost memory that the batch size uses better.

Total audio seen: 600,000 updates × 64 examples × 1 second = 38.4 million seconds ≈ 10,667 hours, or about 445 days of audio. Against the dataset table’s totals — 9,096 h and 2,425 h of speech, 4,989 h and 108 h of general audio, 919 h of music — that is on the order of half an epoch over the full corpus, so the model is closer to single-pass than to heavily repeated exposure. That is why dropout is unnecessary anywhere in the system.

The two objective metrics, defined

Every ablation in this paper is reported in SI-SNR and ViSQOL, and Chapter 5 showed one of them ranking systems backwards. You cannot read that result properly without knowing what each one measures, so define both.

SI-SNR is the scale-invariant signal-to-noise ratio. Given a target x and an estimate x̂, first project the estimate onto the target to remove any overall gain difference:

Work one number to make the scale concrete. Suppose the residual energy is 1% of the target energy. Then the ratio is 100 and SI-SNR = 10 · log₁₀(100) = 20 dB. EnCodec’s 6.67 dB corresponds to a ratio of 10^0.667 = 4.64, i.e. the error carries about 21.5% of the target’s energy.

Twenty-one percent error energy, and listeners rate it 92.9 out of 100 on music. That single juxtaposition is the argument against waveform metrics for generative codecs: the "error" is largely a phase and fine-structure difference that the ear does not encode, and SI-SNR cannot tell that apart from an audible artifact of the same energy.

ViSQOL (Virtual Speech Quality Objective Listener) is the opposite kind of tool: it is a model fitted to predict MUSHRA-like scores. It compares gammatone spectrograms of reference and test using a similarity measure, then maps that similarity through a regression trained on human ratings, producing a Mean Opinion Score on a 1–5 scale. The paper computes it with Google’s open-source implementation using the recommended recipes.

The headline table

Table 1, in full, streamable setting, 24 kHz. Mean MUSHRA with 95% confidence intervals:

Four comparisons worth extracting by hand, each of which the paper makes in one clause:

1. EnCodec at 3 kbps beats Lyra-v2 at 6 kbps and Opus at 12 kbps on average. Check it: 67.0 / 62.5 / 89.6 / 87.8 averages to 76.7. Lyra at 6 kbps: 66.2 / 59.9 / 75.7 / 48.6 averages to 62.6. Opus at 12: 76.5 / 61.9 / 77.8 / 65.4 averages to 70.4. EnCodec wins at a quarter of Opus’s bitrate. This is exactly the paper’s claim: "EnCodec at 3kbps reaches better performance on average than Lyra-v2 using 6kbps and Opus at 12kbps."

2. On music at 6 kbps, EnCodec is within the reference’s confidence interval. Music set 1: reference 93.2 ±2.5, EnCodec 92.9 ±1.8. The intervals overlap heavily. Listeners could barely tell 6 kbps from uncompressed on that material.

3. Noisy speech is the hard case, for everyone. Every model drops in that column. EnCodec at 12 kbps reaches 80.1, exactly matching EVS at 9.6 kbps (80.0). This is where EnCodec’s advantage is smallest, and it makes sense: noise is by construction the least predictable content, so a learned prior has the least to offer.

4. Going from 6 to 12 kbps barely helps on music. Music set 1: 92.9 → 91.8 — within noise, and nominally down. The codec has saturated on that material at 6 kbps; extra codebooks refine a residual listeners cannot hear. Clean speech, by contrast, still gains 7.5 points (83.1 → 90.6). Different content saturates at different rates, which is precisely the argument for a runtime bandwidth knob.

Stereo: the most striking table in the paper

Table 4 evaluates 48 kHz stereo music, where the reference is 1,536 kbps and the compression ratios get absurd:

Line the three 82.x rows up. MP3 at 64 kbps: 82.7. Opus at 24 kbps: 82.9. EnCodec at 6 kbps: 82.9. Statistically indistinguishable quality at one tenth the bits of MP3 and one quarter the bits of Opus. Meanwhile Opus at the same 6 kbps scores 17.7 — a 65-point gap at equal bitrate.

And note the top of the ladder: 12 kbps scores 88.0 while 24 kbps scores 87.5. Within the confidence intervals, they are equal. The paper says it plainly: "EnCodec at 12kpbs achieve comparable performance to EnCodec at 24kbps." Doubling the bitrate buys nothing here, which means the bottleneck at 12 kbps is no longer the bit budget — it is the model.

Against the direct ancestor

Table A.2 is the fairest comparison in the paper, because the authors re-implemented SoundStream themselves (the original is not open sourced) rather than relying on a third-party port. All at 3 kbps except the classical baselines:

Two readings. First, EnCodec-with-DiffQ (72.3) and SoundStream (71.8) are statistically tied — so EnCodec’s architecture and loss changes alone do not beat SoundStream. Second, swapping DiffQ for RVQ adds 4.5 points and pulls clear. The quantizer is doing the work, which retroactively justifies spending two chapters on it.

The full picture, and where it ends

Figure 3 of the paper plots MUSHRA against bitrate for EnCodec (with and without entropy coding), Lyra-v2, EVS, and Opus, on a mix of speech and music. The shape of that plot is the paper’s thesis: EnCodec’s curve sits above and to the left of every baseline at every point tested, and entropy coding shifts its points further left at unchanged height.

But note where the curve stops. At 12 kbps EnCodec reaches 90.6 on clean speech while the reference is 95.5. Five points of quality remain unclaimed, and the 12-versus-24 kbps stereo result says extra bits will not claim them. That gap is the open problem the next generation of codecs inherits, and Chapter 9 follows where they took it.

Chapter 9: Lineage — What EnCodec Became

EnCodec was published as a codec. Within a year it was better known as something else entirely: the standard way to turn audio into tokens.

Chapter 0 planted the observation and it is time to collect on it. The quantizer’s output is a tensor of shape [B, N_q, T] containing integers in [0, 1023] at 75 steps per second. Squint at that and it is a sentence: a discrete sequence over a vocabulary of 1024, with N_q parallel streams. Every technique the language-modelling world has developed — autoregressive generation, conditioning, in-context learning, scaling laws — suddenly applies to raw audio.

The map

Two kinds of audio token, and why both exist

The codec line and the self-supervised-speech line produce different tokens, and understanding the difference is the entry ticket to every audio language model paper written after 2022.

AudioLM’s contribution was to use both: model semantic tokens first to get the content right over long spans, then generate coarse and fine acoustic tokens conditioned on them to get the sound right. Reconstruction quality and long-range coherence are different problems, so they get different token streams. Follow that thread in the AudioLM lesson linked below.

What was built on top

Notice what MusicGen’s problem is. EnCodec hands you N_q parallel streams per timestep. A standard autoregressive Transformer wants one token at a time. Do you flatten fully (sequence length × N_q, slow but exact), predict all N_q in parallel (fast, but ignores within-timestep dependencies — the same approximation the entropy-coding Transformer in Chapter 7 makes), or interleave with a delay pattern? That question exists only because RVQ produces a stack rather than a single index, and it is now a standard design axis in audio generation.

Five misconceptions worth killing

These are the things people reliably get wrong about EnCodec after reading the abstract. Each one is a real confusion with a one-line correction.

The three ideas worth stealing for something else

Most of EnCodec is audio-specific. Three parts are not, and they transfer to any domain where you are compressing or discretising a learned representation.

Honest limitations

The paper is unusually candid, and its limits are worth collecting in one place:

• The 48 kHz model is not streamable. One second of initial latency, from the per-chunk normalisation. Fine for music streaming, unusable for a call.
• Entropy coding costs most of the speed headroom. RTF falls from 9.8 to 1.6 at 24 kHz, and to 0.68 at 48 kHz — slower than real time.
• Cross-platform bit-exactness is mitigated, not solved. "Evaluations in more contexts would be needed for practical deployment."
• Within-timestep dependencies are ignored by the entropy model, which is a known, deliberate loss of compression.
• Quality still trails the reference. 90.6 versus 95.5 on clean speech at 12 kbps, and 12 kbps stereo already matches 24 kbps — the remaining gap is not a bit-budget problem.
• Speed lags the alternative. Lyra v2 encodes at RTF 27.4 and decodes at 67.2 against EnCodec’s 9.8 and 10.4. The paper concedes: "our model is worse than Lyra v2 in term of processing speed."
• Neural codecs have a comfort zone. Stated in the introduction as the first of the two core problems, and addressed with data diversity and discriminators rather than eliminated.

If you were implementing this tomorrow

A build order that follows the dependency graph rather than the paper’s section order. Each step is independently testable, which is the only way a system with five losses and a GAN is ever debuggable.

The cheat sheet

Every symbol, in one table. If you can reconstruct the paper from this, you have the lesson.

Rebuild it from memory: the checklist

If you can answer these without scrolling up, you own the paper.

1. Why is the frame rate 75 Hz, and what would change if the strides were (2, 4, 4, 8)? (Total stride 256 → 93.75 frames/s, a non-integer bit budget per second.)
2. Given 8 codebooks of 1024 entries at 75 Hz, what is the bitrate? Do it without a calculator.
3. Quantize (0.9, −0.4) with the two toy codebooks and report both indices and the final error.
4. Why can one model serve five bitrates, and what does the decoder actually receive?
5. Name the four generator loss terms and say which failure mode each prevents.
6. Why does the discriminator see the complex STFT while the reconstruction loss sees magnitude mel?
7. Write the balancer formula and explain why the commitment loss is excluded.
8. Why is MPD’s best-in-table SI-SNR a warning rather than a recommendation?
9. Why do entropy-coding savings shrink from 40% to 19% as bitrate rises?
10. What is the difference between an acoustic token and a semantic token?

Where to go next