Whose Voice Is This?

Chapter 1: From Frames to One Vector

A microphone gives you a list of numbers: air pressure, sampled sixteen thousand times a second. A verification system needs 192 numbers that mean "Ava". This chapter walks every step between those two, naming every tensor shape, and then stops on the one step that most treatments skip past — because that step is where the speaker actually appears.

Stage 1: the waveform

Sound is a pressure wave. A microphone converts pressure to voltage, an ADC samples that voltage at a fixed rate, and you get an array of floats. For speech, 16 kHz is the near-universal choice.

Why 16,000? The Nyquist–Shannon theorem says a sample rate of R can represent frequencies up to R/2. At 16 kHz that is 8 kHz, and human speech puts almost all of its intelligibility below 8 kHz. (Telephony historically used 8 kHz sampling — a 4 kHz ceiling — which is why "s" and "f" are so easy to confuse on a bad phone line: the energy that distinguishes them lives at 4–8 kHz and the phone threw it away.)

So four seconds of speech is:

Could we feed those 64,000 raw numbers to a network? People have — SincNet learns band-pass filters directly on the waveform, and modern self-supervised models like WavLM start from raw audio. But every system in this lesson starts from a spectrogram instead, and the reason is not laziness.

Stage 2: framing

Speech is not stationary over four seconds — but it is roughly stationary over 25 milliseconds, because the vocal tract cannot reshape itself faster than that. So we cut the signal into short overlapping windows.

The standard settings, used by all three papers: window 25 ms, hop 10 ms. In samples at 16 kHz:

Overlap is 400 − 160 = 240 samples, or 60%. The number of frames from 64,000 samples:

Sanity check that number a second way: with a 10 ms hop you get 100 frames per second, so four seconds is about 400 frames. It is 398 and not 400 because the first window needs 400 samples before it can start and the last one needs 400 samples before it can end — the two-frame shortfall is the edge effect.

Each window is multiplied by a Hamming or Hann taper before the transform. Without the taper, chopping the signal at the window edge creates a discontinuity, and a discontinuity in the time domain is broadband energy in the frequency domain — the artefact is called spectral leakage, and it smears every frame with a false noise floor.

Stage 3: the spectrum, and then the Mel spectrum

Each 400-sample window goes through an FFT (zero-padded to 512 points, the next power of two). A real 512-point FFT gives 512/2 + 1 = 257 independent magnitude bins, each of width 16,000/512 = 31.25 Hz, evenly spaced from 0 to 8,000 Hz.

Evenly spaced is wrong for hearing. The ear's frequency resolution is roughly logarithmic: the difference between 100 Hz and 200 Hz is enormous perceptually, while 7,000 Hz and 7,100 Hz are indistinguishable. The Mel scale encodes that:

Work an example. For a bank of 80 Mel filters spanning 20 Hz to 7,600 Hz:

The span is 2787.0 − 31.75 = 2755.2 mel. With 80 filters we need 82 boundary points, hence 81 equal gaps of 2755.2/81 = 34.02 mel each. Convert the first two gaps back to Hz using the inverse, f(m) = 700(10^m/2595 − 1):

The highest filter is roughly 18 times wider than the lowest. That single fact is the whole point of the Mel warp: it spends resolution where the ear spends it, and where the speaker-discriminating information about vocal-tract length and glottal source happens to live.

Finally, take the logarithm of each filter's energy. Loudness is perceived logarithmically, and — more usefully — a multiplicative channel effect (a microphone that boosts the highs by 6 dB) becomes an additive offset in the log domain, which a network can subtract. That is why the standard feature is log-Mel filterbank energies, or "fbanks".

We went from 64,000 numbers to 80 × 398 = 31,840. A 2× reduction, and every surviving number is perceptually meaningful.

Drag the duration slider from 1 second to 20 and watch every intermediate shape grow twenty-fold while the final vector stays at 192. That invariance is not a convenience. It is the product requirement from Chapter 0, and exactly one operation in the whole chain delivers it.

Stage 4: the frame-level network

The fbank matrix goes into a network that produces one vector per frame. Architecturally this can be a stack of dilated 1-D convolutions (x-vector, ECAPA), a recurrent network (GE2E's LSTM), or a transformer (WavLM-based systems). What they share is the output shape:

For ECAPA-TDNN with C = 1536 after aggregation, that is 1536 × 398 = 611,328 numbers. We have gone up, not down. The frame network's job is not compression; it is to re-describe each 10 ms slice in terms that make the speaker separable — and to give each frame a wide enough view of its neighbours to notice things like formant trajectories, which no single 25 ms window can contain.

Stage 5: pooling — where the speaker actually appears

Now the pivotal step. We have H of shape (C × T), where T depends on the utterance. We need a vector whose size does not. So we collapse the time axis:

Stated that way it sounds like plumbing — a reshape to make the shapes line up. It is not. It is the most consequential line in the architecture, and here is why.

Pooling is permutation-invariant. Averaging over t gives the same answer whichever order the frames arrive in. So the pooled vector cannot represent sequence. It cannot know that "cat" came before "dog", or that a rising pitch contour turned into a falling one. Shuffle the frames of an utterance and the pooled vector is bit-identical.

For a speech recognition system that would be a catastrophe — word order is the entire signal. For a speaker recognition system it is precisely the point.

That reframing pays off immediately. It tells you what to pool: statistics of the frame distribution. The mean is the first one. The x-vector paper's contribution — Chapter 2 — was noticing that the mean alone is not enough, and adding the standard deviation. Attentive pooling — Chapter 4 — is the observation that not all frames deserve an equal vote.

It also tells you the cost. Pooling is why a clip-level speaker embedding cannot tell you when someone spoke, which is why diarization in Chapter 7 has to slice the recording into short windows first and embed each one separately. And it is why an embedding cannot represent two simultaneous speakers as anything but a blend — the overlap problem, also Chapter 7.

Stage 6: the embedding layer, and the head we throw away

The pooled vector goes through one or two fully-connected layers, and the output of the first of them is the embedding. For ECAPA:

Read the "depends on T" column. It flips exactly once, at pooling. Everything above that line is a variable-length representation of a signal; everything below it is a fixed-size description of a person.

The last row is deleted at deployment. The AAM-softmax head has 5,994 × 192 = 1,150,848 parameters — about 7.8% of a C=1024 ECAPA's 14.7M — and every one of them is discarded once training ends. They existed only to apply pressure to the 192 numbers above them.

The shapes, once more, as code

The whole chain in twenty lines, so the shapes are unambiguous. This is PyTorch-flavoured pseudocode — every line corresponds to a row of the table above.

import torch, torchaudio

wav, sr = torchaudio.load("ava.wav")     # wav: (1, 64000), sr = 16000

fbank = torchaudio.compliance.kaldi.fbank(     # log-Mel filterbank energies
    wav, num_mel_bins=80,
    frame_length=25.0, frame_shift=10.0,
    sample_frequency=16000)                    # fbank: (398, 80)  [T, F]

x = fbank.transpose(0, 1).unsqueeze(0)         # x: (1, 80, 398)   [B, F, T]
x = x - x.mean(dim=2, keepdim=True)          # cepstral mean norm — kills the channel

h = frame_network(x)                           # h: (1, 1536, 398)  [B, C, T]

mu    = h.mean(dim=2)                        # mu:  (1, 1536)   — T is gone
sigma = h.std(dim=2)                         # sigma: (1, 1536)
pooled = torch.cat([mu, sigma], dim=1)        # pooled: (1, 3072)

emb = fc(pooled)                               # emb: (1, 192) ← the speaker embedding
emb = torch.nn.functional.normalize(emb, dim=1) # unit length: cosine == dot product

# training only — deleted at deployment:
logits = aam_softmax_head(emb, labels)         # logits: (1, 5994)

Two lines in there are quietly load-bearing. x - x.mean(dim=2) is cepstral mean normalisation: subtracting each Mel bin's average over the utterance removes any constant multiplicative colouring of the channel, because a constant gain in the linear domain is a constant offset in the log domain. It is three characters of code that does more for channel robustness than most architectural choices.

And normalize(emb) makes the embedding unit-length, so the dot product is the cosine. After that line, similarity is one instruction, and the geometry the training objective shaped is the geometry the deployment scorer reads. Chapter 5 shows how much depends on those two being the same.

What each of the three papers puts in each slot

Look at the pooling row. GE2E's LSTM takes the final hidden state, which is not permutation-invariant — the recurrence smuggles order back in. It works, because an LSTM run over four seconds of speech mostly ends up summarising rather than remembering. But it is the one design in this table that fights the Chapter 1 thesis rather than embracing it, and it is not what the field kept.

The uncertainty tradeoff, quantified

Chapter 1 chose a 25 ms window without justifying the number. It is the resolution of a genuine tradeoff, and the arithmetic explains why nobody has moved off it in thirty years.

A window of duration Δt gives a frequency resolution of roughly Δf ≈ 1/Δt. You cannot improve one without ruining the other — this is the time-frequency uncertainty principle, and it is a property of the Fourier transform, not of any implementation.

The bolded row is the compromise everyone lands on: three glottal periods. Fewer and you cannot see the source; more and the filter has changed while you were looking. Twenty-five milliseconds is where those two constraints meet for adult human speech, which is why the number is a constant of the field rather than a hyperparameter anyone tunes.

Why the logarithm, with numbers

Chapter 1 said "take the log" and gave two reasons. Both deserve arithmetic, because the second one is doing real work in every deployed system.

Reason one: dynamic range. Speech energy across a single utterance spans roughly 60 dB from a stressed vowel to a quiet fricative. In linear power that is a ratio of 10⁶. Feed that to a network with any normalisation scheme and the fricatives are numerically invisible — they round to zero against the vowels. In the log domain the same span is 60 units, comfortably handled.

Reason two: channels become additive. Suppose a microphone boosts everything above 2 kHz by 6 dB. In the linear domain, the observed spectrum is the true one multiplied pointwise by a channel response:

A 6 dB boost is a linear factor of 10^6/20 = 1.995 ≈ 2. In natural log that is ln(1.995) = 0.691, added to every affected bin, identically, for the whole recording. So:

Subtracting each bin's own mean over time removes that 0.691 exactly, along with any other constant colouring of the channel. Two recordings of the same person on wildly different microphones become numerically similar after one subtraction. Cepstral mean normalisation is the highest-value three characters in the whole pipeline, and it works only because the log turned a multiplication into an addition.

Two more front-end details that are not decoration

Pre-emphasis. Before framing, most pipelines apply a one-tap high-pass filter, y[n] = x[n] − 0.97·x[n−1]. The glottal source has a spectral tilt of roughly −12 dB per octave and lip radiation adds +6 dB per octave, leaving a net −6 dB per octave slope in the speech spectrum. Pre-emphasis approximately flattens it, so the higher formants — F₃ and F₄, which carry a great deal of speaker-specific information — are not numerically dwarfed by F₁.

The floor. log(0) is negative infinity, and a filterbank energy really can be zero in digital silence. Every implementation adds a floor, either log(x + ε) with ε around 10⁻⁶, or a hard clamp. Get this wrong and a single silent frame produces a NaN that propagates through pooling and destroys the whole utterance's embedding — a failure that is invisible in a research benchmark of curated clips and immediate in production, where silence is everywhere.

What the frame network is actually for

One last idea before pooling, because it is easy to read "frame network" as "a stack of layers" and miss the point.

A single 25 ms window contains a snapshot of the vocal tract. But a great deal of speaker identity lives in how the tract moves — the speed and target of formant transitions, the sharpness of a stop release, the coordination between voicing onset and articulation. None of that is present in any single frame, and all of it is present in a 150 ms neighbourhood.

Read that chain twice. It explains why every architecture in this lesson widens the frame-level receptive field — Chapter 4's ECAPA gets to 245 ms and gives every intermediate scale as well — and it explains what statistics pooling is summarising: not raw spectra, but the distribution of a speaker's articulatory habits.

Chapter 2: x-vectors and Statistics Pooling

In 2018 David Snyder and colleagues at Johns Hopkins published a paper with an unglamorous title — "X-Vectors: Robust DNN Embeddings for Speaker Recognition" — and ended a decade in which the state of the art had been a generative model. The architecture is small enough to hold in your head, and every choice in it is defensible from first principles. We will build it layer by layer, derive its pooling by hand on a five-frame toy, and then explain its scoring backend in words rather than matrix algebra.

What it replaced, in one paragraph

Before x-vectors, the field ran on i-vectors (Dehak et al., 2011). The idea: model the distribution of all speech frames with a giant Gaussian mixture — the "universal background model", typically 2,048 components — then, for a given utterance, note how the mixture's means have to shift to explain that particular recording. Stack all those shifts into one enormous supervector, and assume the shifts live near a low-dimensional linear subspace. The utterance's coordinates in that subspace are the i-vector, usually 400 or 600 dimensions.

It is a beautiful piece of engineering and it has one structural weakness: nothing in the training objective ever mentions speaker identity. The UBM is fit by maximum likelihood on unlabelled audio; the subspace is fit to explain variance. Speaker labels only enter afterwards, at the scoring stage. So the representation is optimised to describe speech, and then you hope description implies discrimination.

The architecture, layer by layer

Here is Table 1 of the paper, expanded so every number is derivable. Input is 24-dimensional filterbank features, mean-normalised over a sliding window of up to 3 seconds.

Two arithmetic points worth checking yourself. The input width of frame1 is 120 because it splices five consecutive 24-dimensional frames: 5 × 24 = 120. The input width of frame2 is 1536 because it splices three of frame1's 512-dimensional outputs: 3 × 512 = 1536.

And the receptive field. Frame1 spans 5 input frames. Frame2 reaches ±2 of frame1's outputs, each of which already spans 5, so its span is 5 + 2×2 = 9. Frame3 reaches ±3 of those, so 9 + 2×3 = 15. Frame4 and frame5 are pointwise, so the span stops at 15. At a 10 ms hop:

165 milliseconds is roughly the length of a syllable. That is a deliberate scale: long enough to contain a formant transition (the glide from a consonant into a vowel, which is heavily shaped by vocal-tract geometry), short enough not to be dominated by which word was said.

Statistics pooling, derived by hand

Now the layer with zero parameters that does the most work. Take a toy: T = 5 frames of a C = 3 channel activation. (Real numbers are 1500 × 398; three and five are enough to see the mechanism.)

Means, channel by channel:

Standard deviations. Take channel 1. Deviations from the mean of 3: (−1, +1, −3, +3, 0). Squares: (1, 1, 9, 9, 0), summing to 20. Divide by T = 5:

Channel 2: deviations (−1, 0, +1, 0, 0), squares (1, 0, 1, 0, 0) summing to 2, so σ₂² = 0.400 and σ₂ = 0.6325.

Channel 3: deviations (−1, +1, +3, −1, −2), squares (1, 1, 9, 1, 4) summing to 16, so σ₃² = 3.200 and σ₃ = 1.7889.

Concatenate means then standard deviations:

Fifteen numbers became six, and — the part that matters — five frames became six numbers, and fifty frames would also have become six numbers. In the real network, 1500 × T becomes 3000, for any T.

Two experiments on that toy that make the design obvious

Experiment 1: reverse time. Feed the frames in the order t = 5, 4, 3, 2, 1. Channel 1 becomes (3, 6, 0, 4, 2). Sum: still 15. Mean: still 3.000. Deviations: (0, +3, −3, +1, −1), squares (0, 9, 9, 1, 1) — sum still 20, so σ is still 2.000. Every number in v is unchanged.

That is Chapter 1's thesis with arithmetic attached. Reversing the utterance destroys every word in it and leaves the speaker vector untouched, because the pool reads the distribution and not the sequence.

Experiment 2: delete the standard deviations. Suppose we kept only the means. Consider two speakers whose channel-1 activations over five frames are:

Identical means. Now the standard deviations. A has zero deviation, so σ_A = 0. B has deviations (−2, +2, 0, −2, +2), squares (4, 4, 0, 4, 4) summing to 16, so σ_B² = 3.200 and σ_B = 1.7889.

Two readings to take away. The same speaker's two sentences have visibly different activation maps — different words, different phonemes — and nearly identical pooled vectors. The different speaker's map may look superficially similar in places and pools somewhere else entirely. Pooling is doing the work that the words were hiding.

The augmentation, which is really the paper's contribution

Here is the part people forget: the x-vector paper's headline claim is not about the architecture. It is that data augmentation helps discriminative embeddings far more than it helps generative ones. The recipe:

The asymmetry the paper measures: augmentation applied to the PLDA backend helps both i-vectors and x-vectors a little; augmentation applied to the embedding training data helps x-vectors substantially and does not apply to i-vectors at all, because the i-vector extractor is not trained with labels and cannot exploit "these two clips are the same speaker in different rooms". A discriminative front end can be told what to ignore. A generative one cannot.

The backend: PLDA in plain words

x-vectors are not scored with a cosine. They are scored with PLDA — probabilistic linear discriminant analysis — and this is worth explaining properly, because the reason it exists tells you something about what the objective did and did not do.

The generative story PLDA tells is one sentence: every embedding is a speaker point plus a channel wobble.

In words: μ is the global average voice; h is a latent identity, drawn once per person; F says how identities spread out; and ε is everything that changes between recordings of the same person — room, mic, mood, cold. Two recordings of Ava share one h and get two independent draws of ε.

Scoring is then a likelihood ratio, and this is the part worth internalising:

Read it as a question a detective asks: which story explains this pair better — one person recorded twice, or two people recorded once each? That is not a distance. Two vectors can be far apart and still score highly, if the direction they differ in is a direction that Σ says varies a lot between sessions. And two vectors can be close and score badly if they are close in a direction that everybody's voice occupies.

The full deployment pipeline, in order, because every step matters and skipping one costs real accuracy:

Every one of those four steps needs data from the target domain to estimate. That is PLDA's practical cost: it is a second training procedure, with a second data requirement, on top of the network. Move to a new domain and your embeddings still work but your backend does not.

What x-vectors got right, and the two things they left open

Those two open items are exactly what the next three chapters close, from two different directions: GE2E attacks the objective by training on comparisons directly, and ECAPA attacks the encoder while adopting an angular-margin objective. Both roads end at cosine scoring with no backend.

How big the i-vector machinery actually was

It is worth pricing the thing x-vectors replaced, because the comparison explains why the field switched so fast once the discriminative version worked at all.

A typical i-vector front end used a universal background model with 2,048 Gaussian components over 60-dimensional features (20 MFCCs plus deltas and double-deltas). The supervector of all component means is:

The total variability matrix T projects that down to a 400-dimensional i-vector, so T alone is:

Forty-nine million parameters — more than three times an ECAPA-TDNN at C = 1024 — estimated by expectation-maximisation, with no speaker label ever entering the objective. And extraction at test time was not a forward pass: it required accumulating Baum-Welch statistics against all 2,048 components and then solving a 400 × 400 linear system per utterance.

The augmentation, priced in decibels

Chapter 2's table said "13–20 dB SNR" without unpacking it. Signal-to-noise ratio in decibels is:

So 20 dB means the speech carries 100 times the power of the noise, 10 dB means 10 times, and 0 dB means they are equal. To mix at a target SNR you measure both powers and scale the noise:

Work one: speech power 0.01, noise power 0.04, target SNR 13 dB. Then 10^1.3 = 19.95, so scale = √(0.01 / (0.04 × 19.95)) = √(0.01/0.798) = √0.01253 = 0.1119. Multiply the noise by 0.1119 and add.

Notice the ranges the paper chose. Babble is mixed at 13–20 dB — comparatively gentle — while general noise goes down to 0 dB, where speech and noise are equally loud. That asymmetry is deliberate: babble is competing speech, so mixing it too loudly produces clips whose "correct" speaker label is genuinely ambiguous, and training on a wrong label is worse than not training at all. Augmentation has to stay on the right side of the line where the label is still true.

The whole encoder, in code

The x-vector is small enough to write out completely. This is the architecture of the table above, with the shapes annotated.

import torch, torch.nn as nn

class TDNNLayer(nn.Module):
    """A dilated 1-D conv. context={-2,0,2} is kernel 3 with dilation 2."""
    def __init__(self, cin, cout, k, dilation):
        super().__init__()
        self.conv = nn.Conv1d(cin, cout, k, dilation=dilation)
        self.act, self.bn = nn.ReLU(), nn.BatchNorm1d(cout)
    def forward(self, x):                 # x: (B, cin, T)
        return self.bn(self.act(self.conv(x)))   # (B, cout, T')

class XVector(nn.Module):
    def __init__(self, feat_dim=24, n_spk=5994):
        super().__init__()
        self.frames = nn.Sequential(
            TDNNLayer(feat_dim, 512, k=5, dilation=1),   # context {-2..2}
            TDNNLayer(512, 512, k=3, dilation=2),        # context {-2,0,2}
            TDNNLayer(512, 512, k=3, dilation=3),        # context {-3,0,3}
            TDNNLayer(512, 512, k=1, dilation=1),
            TDNNLayer(512, 1500, k=1, dilation=1))
        self.seg6 = nn.Linear(3000, 512)      # ← the x-vector comes from here
        self.seg7 = nn.Linear(512, 512)
        self.head = nn.Linear(512, n_spk)      # deleted at deployment

    def forward(self, x, return_emb=False):    # x: (B, 24, T)
        h = self.frames(x)                       # (B, 1500, T-14)
        mu = h.mean(dim=2)                     # (B, 1500)
        sd = h.std(dim=2, unbiased=False)      # (B, 1500)
        stats = torch.cat([mu, sd], dim=1)      # (B, 3000)  ← T is gone
        emb = self.seg6(stats)                   # (B, 512)   ← the x-vector
        if return_emb: return emb
        return self.head(torch.relu(self.seg7(torch.relu(emb))))

Two details in there are the kind that cost a day when you get them wrong. unbiased=False divides by T rather than T−1, matching the derivation above and, more importantly, staying finite when a padded segment has T = 1. And the embedding is taken before the non-linearity of segment 6 — the paper's own choice, and it matters, because the ReLU that follows would clip every negative coordinate and destroy exactly the sign structure a cosine or PLDA score reads.

Why length normalisation is not cosmetic

The third step of the PLDA pipeline — scale every vector to unit length — looks like tidiness. It is a distributional repair, and the reason is worth stating because it recurs in Chapter 5.

PLDA assumes the within-speaker and between-speaker distributions are Gaussian. Real x-vector distributions are not: they are heavy-tailed, because a fraction of utterances are short, noisy, or contain a cough, and those land far from the centre. A heavy tail wrecks a Gaussian model's covariance estimate, and the covariance is the thing PLDA uses to decide which directions are nuisance.

Projecting onto the unit sphere bounds every vector's distance from the origin, which truncates the tail by construction. The empirical result — established well before x-vectors, in the i-vector era — is a large and free accuracy gain. And it hints at the eventual answer: if the natural home of these representations is a sphere, perhaps the training objective should live on the sphere too. That is Chapter 5.

Chapter 3: GE2E — Training on the Comparison

The x-vector is trained to answer a question nobody will ever ask it: which of these 5,000 people is this? At deployment it is asked a completely different question: are these two clips the same person? There is a leap of faith in the middle, and Chapter 2 measured the size of it — a whole second model, PLDA, exists to repair the mismatch.

In late 2017 Li Wan, Quan Wang, Alan Papir and Ignacio Lopez Moreno at Google published "Generalized End-to-End Loss for Speaker Verification" with a simple proposal: stop leaping. Train on the comparison itself.

The batch is the idea

Almost everything distinctive about GE2E is in how a batch is constructed. Instead of sampling utterances independently, you sample a grid:

Every utterance goes through the encoder — a 3-layer LSTM with 768 cells and a 256-unit projection, taking the final frame's output — and is L2-normalised. Call the result e_ji: the embedding of speaker j's utterance i.

Now compute, for each speaker, a centroid:

and build a similarity matrix of every utterance against every centroid:

with w and b learned, initialised to (10, −5). The matrix has N·M rows and N columns — 640 × 64 in the paper's configuration. Each row asks: which of these 64 people does this clip belong with?

The exclusion trick, which is not optional

There is a hole in the definition above and it is fatal. When we score utterance e_ji against its own speaker's centroid c_j, that utterance is one of the M terms inside the centroid. So the model can win by cheating: push every embedding to a fixed point and the centroid follows it there, similarity 1.0, loss 0, and the embedding carries no information at all.

GE2E's fix is to exclude the utterance being scored:

and use that whenever k = j, while using the full centroid for k ≠ j. Now the target is genuinely held out: the utterance must be close to the other recordings of the same person, none of which it contributed to. The paper is explicit that this "makes training stable and helps avoid trivial solutions".

The two losses

Given a row of the similarity matrix, GE2E offers two ways to turn it into a scalar.

Softmax variant — treat the row as logits over N speakers and apply cross-entropy with the true speaker as target:

Contrast variant — push the true similarity toward 1 and the single worst impostor toward 0, through a sigmoid σ:

The total loss is the sum over all N·M rows. These behave differently and the difference matters, so we will now compute both, twice, on a batch small enough to do in your head.

A 2 × 2 batch, worked to a number

Take N = 2 speakers, M = 2 utterances each, and pretend embeddings are two-dimensional unit vectors so we can name them by angle. (In two dimensions, cos(e, c) is just the cosine of the angle between them, which makes every step checkable by hand.)

Case A — a well-trained batch. Speaker A's two utterances sit at 0° and 50°. Speaker B's sit at 130° and 170°.

Because M = 2, the exclusion rule is simple: the own-speaker centroid for utterance i is just the other utterance (one term, so no averaging needed). The other speaker's full centroid is the mean of its two unit vectors, which points at the bisecting angle — 150° for B, 25° for A.

Row 1, utterance A₁ at 0°:

Apply the scale and bias with the paper's initial values w = 10, b = −5:

Softmax loss for this row. Exponentiate: e^1.428 = 4.1706, and e^−13.660 = 0.00000117. Sum = 4.17060.

Essentially zero. Do the same for the other three rows — A₂ at 50° scores cos 50° = 0.6428 against its partner at 0° and cos 100° = −0.1736 against c_B; B₁ at 130° scores cos 40° = 0.7660 against its partner and cos 105° = −0.2588 against c_A; B₂ at 170° scores cos 40° = 0.7660 and cos 145° = −0.8192:

Check row A₂ yourself: e^1.428 = 4.1706, e^−6.736 = 0.0011884, sum 4.17179, log = 1.428285, minus 1.428 = 0.000285. Good.

Case B — a batch the model gets wrong. Now suppose the encoder is undertrained and speaker B's utterances land at 20° and 40°, right between speaker A's two at 0° and 50°. B's centroid is at 30°.

The impostor now scores higher than the true speaker. Softmax loss:

Two point three three four, against 0.0000003 for the good row — seven million times larger. Cross-entropy is not gentle about being confidently wrong.

Now the contrast variant on the same two rows, with σ(z) = 1/(1 + e^−z):

What w and b are actually for

The scale w and bias b look like cosmetic add-ons. They are not, and the sliders above make the argument. A cosine lives in [−1, 1]. Softmax over a row of cosines can never be confident: the best possible logit gap is 2, and with N = 64 speakers the probability of the correct class is capped at

— a loss of −log(0.1050) = 2.254 that no encoder can ever reduce. The model would be permanently pinned in a regime where gradients look like it is failing even when the embedding is perfect. Multiplying by w = 10 gives logits in [−10, 10] and restores the dynamic range. The bias b shifts the operating point so that the sigmoid in the contrast variant is not saturated at initialisation.

The paper constrains w > 0 during training. That constraint is load-bearing: if w were allowed to go negative, the model could satisfy the loss by making genuine pairs dissimilar and flipping the sign, which is a valid optimum of the loss and a useless embedding.

MultiReader — the practical trick worth stealing

GE2E ships a second idea that gets less attention and is arguably more useful day to day. Google had a large dataset of "OK Google" utterances and a small one for a new keyword. Fine-tuning on the small set overfits; pooling the sets lets the large one dominate. MultiReader instead computes the loss on both and combines:

The paper frames α as a regularisation weight: the large dataset acts as a prior that keeps the model honest while the small one steers it. It generalises to K datasets with K weights, and it is the standard answer whenever you have one big generic corpus and one small in-domain one — which, in speaker verification, is always.

What GE2E claims, and what it actually established

The reported headline: more than 10% relative reduction in EER against the previous TE2E system, with training time cut by about 60%. Both come from the same source — a batch of N×M produces N·M×N comparisons rather than N·M/2 pairs, so each gradient step is informed by far more comparisons at essentially the same forward cost.

The honest historical verdict: the field did not keep GE2E's exact loss for text-independent verification — AAM-softmax won that contest. But it kept GE2E's encoder, in an unexpected place. Google's SV2TTS paper took a GE2E-trained speaker encoder and used it to condition a Tacotron 2 synthesiser on a target voice, which is the direct ancestor of every zero-shot voice cloning system in use today. The loss designed to verify voices became the machinery that copies them. Chapter 8 returns to that irony with the seriousness it deserves.

Counting the comparisons, which is where the speedup comes from

GE2E's claim of a 60% training-time reduction sounds like an engineering detail. It is a direct consequence of the batch shape, and the arithmetic makes it obvious.

Triplet loss (the FaceNet approach, and TE2E's ancestor) consumes a batch of B utterances as B/3 triplets. Each triplet yields exactly one comparison-with-a-margin. With B = 640:

GE2E consumes the same 640 forward passes as a 64 × 10 grid and scores every utterance against every centroid:

A factor of 192, for the same encoder cost. The forward passes dominate the compute; the similarity matrix is one (640 × 256)(256 × 64) product, about 21 MFLOPs, which is nothing beside 640 LSTM passes. You get two orders of magnitude more supervision essentially free.

What the gradient actually does

Take the softmax variant and differentiate. For a row with similarities S = (S₁, …, S_N) and true speaker j, write p = softmax(S). Then, exactly as in ordinary cross-entropy:

So the gradient with respect to the true speaker's similarity is p_j − 1, which is negative — push S_j up. For every other speaker it is p_k, which is positive — push S_k down, and in proportion to how much probability mass that impostor currently holds.

Work it on Case B from above, where S = (1.428, 3.660). The softmax is:

Compare Case A, where S = (1.428, −13.660) and p = (0.99999972, 0.00000028):

Then, since S_k = w·cos(e, c_k) + b, the chain rule pushes that scalar through the cosine into the embedding. The gradient on e has a component along c_j (pulling the utterance toward its own speaker's centroid) minus a weighted sum of components along the impostor centroids — and because the cosine's derivative is orthogonal to e itself, the update is purely rotational. Length never changes. The whole optimisation lives on the sphere, which is exactly the geometry Chapter 5 formalises.

The loss in code

import torch, torch.nn.functional as F

def ge2e_loss(emb, w, b):
    """emb: (N, M, D) — L2-normalised. w: positive scalar. b: scalar."""
    N, M, D = emb.shape
    centroids = emb.mean(dim=1)                       # (N, D)  full centroids

    # leave-one-out centroid for the own-speaker column:
    # (sum − self) / (M − 1), computed without a loop
    excl = (emb.sum(dim=1, keepdim=True) - emb) / (M - 1)   # (N, M, D)

    e = F.normalize(emb, dim=2)
    c = F.normalize(centroids, dim=1)                   # (N, D)
    S = torch.einsum('nmd,kd->nmk', e, c)               # (N, M, N) all cosines

    # overwrite the diagonal (own-speaker) entries with the excluded version
    own = F.cosine_similarity(e, F.normalize(excl, dim=2), dim=2)  # (N, M)
    idx = torch.arange(N, device=emb.device)
    S[idx, :, idx] = own

    S = w.clamp(min=1e-6) * S + b                        # w > 0 is enforced here
    labels = idx.unsqueeze(1).expand(N, M).reshape(-1)
    return F.cross_entropy(S.reshape(N * M, N), labels)

The one line worth staring at is excl. Subtracting each embedding from its own speaker's sum and dividing by M−1 computes all N·M leave-one-out centroids in a single vectorised expression — no loop, no masking, no gather. And w.clamp(min=1e-6) is the positivity constraint from the paper, enforced in the one place it cannot be forgotten.

Why the two variants split along the text-dependent line

Return to the table where Case A scored 0.0000003 under softmax and 0.1935 under contrast. That gap is the entire explanation, and it maps cleanly onto Chapter 0's text-dependent axis.

The pattern generalises well past this paper. A relative objective produces a good ranking and a badly calibrated absolute scale; an absolute objective produces compact clusters and can over-fit its single hardest example. Chapter 5's angular margin is, read in this light, a way of getting the softmax's automatic hard-negative weighting and an absolute compactness target, by moving the target rather than changing the loss shape.

Chapter 4: ECAPA-TDNN — Four Upgrades at Once

In 2020 Brecht Desplanques, Jenthe Thienpondt and Kris Demuynck at Ghent published an architecture whose acronym is an inventory of its own contents: Emphasized Channel Attention, Propagation and Aggregation in TDNN. It kept the x-vector skeleton exactly — frame network, pooling, embedding — and improved four things inside it. Six years later it is still the default speaker encoder in SpeechBrain, WeSpeaker, NeMo and most production systems.

This chapter takes the four upgrades one at a time, says what each one buys with actual arithmetic, and then reads the ablation honestly — including the part where the components turn out not to add up.

Upgrade 1: channel attention (squeeze-and-excitation)

Start with the problem. A TDNN layer's frame output at time t is built from a 15-frame window. It has no idea what is happening two seconds away. But whether a given channel's activation is informative often depends on global context: if the whole clip is drowning in babble, the channels that respond to high-frequency detail should be trusted less; if the clip is clean and close-miked, they should be trusted more.

The squeeze-excitation block (Hu et al., 2018, imported from vision) gives a local convolution a global thermostat. Two steps.

Squeeze — average each channel over the entire utterance, producing one number per channel:

Excite — push z through a small bottleneck MLP and a sigmoid to get a gain per channel, then multiply:

The bottleneck is small — ECAPA uses 128 units for C = 512 or 1024. Parameter cost for C = 1024: 1024×128 + 128×1024 = 262,144, about 1.8% of the model. Cheap.

And notice the family resemblance. The SE squeeze is a mean over time. The pooling layer in Chapter 2 is a mean over time. ECAPA computes a global mean at three places in the frame network before computing it once more at pooling. The architecture is, in a real sense, mostly about when and how to average over the time axis.

Upgrade 2: Res2Net — many receptive fields inside one layer

An x-vector's frame network has one receptive field per layer: 5, then 9, then 15 frames. But speaker cues live at wildly different time scales at once. Glottal buzz is a property of a 10 ms slice. Formant transitions take 50–100 ms. Speaking rhythm and prosodic habits take half a second. A single fixed window is a compromise across all three.

Res2Net (Gao et al., 2019) gets several scales out of one layer by splitting the channels and chaining the splits. With C channels and scale s = 8, you cut the input into 8 groups of C/8 channels and process them like this:

Each K_i is a dilated 3-tap convolution. Group 2 passes through one convolution. Group 3 passes through the sum of its own input and group 2's output, so it has effectively been through two convolutions. Group 8 has been through seven.

Work out the receptive fields. A 3-tap convolution with dilation d extends the receptive field by (3−1)·d = 2d frames. For the first ECAPA block, d = 2:

The general formula is RF_i = 1 + (i − 1)·2d, so for d = 2 group 8 gives 1 + 7×4 = 29 frames. One layer, receptive fields from 10 ms to 290 ms, simultaneously. That is the multi-scale claim made concrete.

Now the parameter cost, which is the surprise. Res2Net with C = 512 and s = 8 has 64 channels per group and 7 convolutions of 64 → 64 with 3 taps:

A plain 3-tap convolution over all 512 channels would cost:

ECAPA stacks three of these SE-Res2Blocks with dilations 2, 3 and 4. Total frame-network receptive field, counting the initial 5-tap convolution:

Against the x-vector's 15 frames (165 ms). About 1.5× wider at the top — and, crucially, with every intermediate scale available rather than only the widest.

Upgrade 3: multi-layer feature aggregation

In an x-vector, only the last frame layer feeds the pooling. Everything the earlier layers computed reaches pooling only in whatever form the later layers chose to preserve. ECAPA instead concatenates the outputs of all three SE-Res2Blocks and runs a 1×1 convolution over the stack:

The paper's argument, which the vision literature had already made about feature pyramids: shallow layers hold fine spectral detail that deep layers have abstracted away, and speaker identity lives at both levels. The "propagation and aggregation" in the acronym refers to this plus the residual connections that let each block also add to a running sum.

Concretely, for C = 512, the concatenation is 1536 channels wide and the 1×1 conv maps 1536 → 1536, costing 1536×1536 = 2.36M parameters — the single largest block in the C=512 model. It is not free. Whether it is worth its weight is exactly the kind of question the ablation should answer, and Chapter 4's honest reading is coming in three sections.

Upgrade 4: attentive statistics pooling, per channel

Return to statistics pooling from Chapter 2. Every frame votes equally — including the 40% of a typical recording that is silence, breath, or a door closing. Attentive statistics pooling (Okabe et al., 2018) learns the votes.

A small network scores each frame, softmax over time turns the scores into weights, and the statistics become weighted:

Work it on the Chapter 2 toy. Channel 1 was (2, 4, 0, 6, 3), which pooled to μ = 3.000 and σ = 2.000 with equal weights of 0.2 each. Now suppose the attention network judges frames 1 and 3 to be near-silence and assigns α = (0.05, 0.25, 0.05, 0.50, 0.15) — which sums to 1.00, as it must.

For the standard deviation, use the weighted second moment minus the square of the weighted mean:

Compare: unweighted (3.000, 2.000), attentive (4.550, 1.6875). Down-weighting the two frames the network judged uninformative moved the mean up by more than half a standard deviation and tightened the dispersion. If frames 1 and 3 really were silence, the second pair is a much better description of the speaker.

ECAPA extends this in two ways beyond the 2018 formulation. First, the attention is channel-dependent — a separate α per channel per frame, not one shared weight per frame — because a frame can be informative about a speaker's pitch while being useless about their fricatives. Second, the attention network's input is the frame concatenated with the global context (the mean and standard deviation over the whole utterance, broadcast back to every frame), so a frame is judged relative to the recording it sits in rather than in isolation. That last detail is the same idea as the SE squeeze, applied at pooling.

The full architecture, with shapes

Fourteen million parameters. For context, that is roughly one seven-hundredth of a modern 10B language model, and it runs comfortably in real time on a phone CPU. Speaker embedding is one of the few remaining corners of deep learning where the useful model is small.

The results, and the honest ablation reading

Trained on VoxCeleb2's development set — 5,994 speakers — and evaluated on the three VoxCeleb1 trial lists:

Vox1-O is the original 40-speaker test list (about 37.6k trials). Vox1-E is the extended list over all 1,251 VoxCeleb1 speakers (about 580k trials). Vox1-H is the hard list, where every impostor trial pairs speakers of the same nationality and gender — which is why its error rate is roughly double. Always ask which list a number came from; the same system can look twice as good by quoting Vox1-O.

Now the ablation, and here is where a lesson should be careful. The paper removes one component at a time from the C = 512 model. The full model scores 1.01% EER. Every single-component removal lands in a band roughly between 1.1% and 1.2%.

The second honest note: ECAPA was trained with AAM-softmax and scored with a plain cosine, while the x-vector baseline used plain softmax and a PLDA backend. The comparison therefore bundles an architecture change with an objective change. The objective is the subject of Chapter 5, and it is doing a substantial share of the work in that table.

What ECAPA did not solve

The SE-Res2Block, in code

All four upgrades except the aggregation live inside one repeated block. Writing it out removes any remaining ambiguity about the ordering.

class SERes2Block(nn.Module):
    def __init__(self, C, k=3, dilation=2, scale=8, se_bottleneck=128):
        super().__init__()
        self.scale, self.width = scale, C // scale       # 512 // 8 = 64
        self.pre  = nn.Conv1d(C, C, 1)                    # 1×1 — mixes channels
        self.convs = nn.ModuleList([                     # scale − 1 = 7 convs
            nn.Conv1d(self.width, self.width, k,
                      dilation=dilation, padding=dilation * (k - 1) // 2)
            for _ in range(scale - 1)])
        self.post = nn.Conv1d(C, C, 1)
        self.se   = nn.Sequential(                       # squeeze-excitation
            nn.Linear(C, se_bottleneck), nn.ReLU(),
            nn.Linear(se_bottleneck, C), nn.Sigmoid())

    def forward(self, x):                              # x: (B, C, T)
        res = x
        h = torch.relu(self.pre(x))
        parts = torch.split(h, self.width, dim=1)        # 8 × (B, 64, T)
        out, prev = [parts[0]], None                    # group 1 passes through
        for i, conv in enumerate(self.convs):
            inp = parts[i + 1] if prev is None else parts[i + 1] + prev
            prev = torch.relu(conv(inp))                 # RF grows with i
            out.append(prev)
        h = torch.relu(self.post(torch.cat(out, dim=1)))

        z = h.mean(dim=2)                              # SQUEEZE: (B, C) over ALL T
        h = h * self.se(z).unsqueeze(2)                 # EXCITE: per-channel gain
        return h + res                                  # residual — the "propagation"

Three lines carry the chapter. torch.split plus the accumulating prev is the entire Res2Net mechanism — there is no special layer, only a loop that feeds each group the previous group's output. h.mean(dim=2) is the squeeze, and it is the only place in the block where information crosses the whole utterance. And h + res is what lets the multi-layer aggregation later see a running sum rather than only the final transformation.

Where the compute goes

ECAPA is cheap, and it is worth knowing which parts are cheap. Take C = 1024 and T = 398 frames (four seconds).

The multi-layer aggregation is the single most expensive layer in the network — a 1×1 convolution across 3,072 input channels at every one of 398 frames. That is a real cost for a component whose individual ablation contribution is on the order of a tenth of an EER point, and it is a reasonable first thing to shrink when you need a smaller model.

Notice also that the Res2Net convolutions are only 137 M of each block's 971 M — about 14%. The multi-scale machinery that the paper is named after is nearly free; the expensive part is the 1×1 channel mixing that surrounds it. This is the same accounting as depthwise-separable convolutions in vision: once you factor the spatial and channel operations apart, the spatial one becomes negligible.

The training recipe, which is half the result

Reproducing ECAPA's numbers requires the recipe, not just the architecture. Every item below appears in the paper or its reference implementations, and skipping any of them costs measurable accuracy.

Why the attention needs global context

One detail of the attentive pooling deserves its own argument, because it is easy to dismiss as an implementation flourish.

ECAPA feeds the attention network each frame concatenated with the utterance-level mean and standard deviation. Without that, the attention scores a frame in isolation: it can learn "loud frames are informative" or "frames with high-frequency energy are informative", both of which are rules about absolute levels.

Absolute levels are exactly what changes between recordings. A frame that is loud in a quiet studio recording and a frame that is loud in a noisy car have completely different informativeness, and an attention network that cannot see the recording's overall statistics has no way to tell them apart. Give it the global mean and standard deviation and the rule it can learn becomes relative: this frame is unusually clean for this recording. Relative rules transfer across channels; absolute ones do not.

That is the same argument, in a third place, as the SE block's squeeze and the CMVN of Chapter 1: a local decision made without global context is a decision about a level, and levels are the nuisance. Three components, three global means, one idea.

Chapter 5: The Margin Arrives from the Face World

Chapter 2 left a loose end. x-vectors are trained with plain softmax and then need PLDA, a whole second model, to be scored well. Chapter 3 showed one escape: change the loss so that it trains on comparisons directly. This chapter is the other escape, and it is the one the field actually took.

It did not come from speech. It came from face recognition, where the identical problem — open-set biometric matching with a cosine at test time — had been attacked for five years by a series of papers with escalating names: FaceNet, SphereFace, CosFace, ArcFace. ECAPA-TDNN's loss is ArcFace's loss, unchanged, applied to voices.

Why plain softmax produces a bad metric

Start with the diagnosis. The softmax logit for class j is w_j^Tx + b_j. Write the dot product in polar form:

Three quantities determine the logit: the weight norm, the feature norm, and the angle. Only the third one is about identity. The other two are free parameters the optimiser will happily abuse.

And it does abuse them. Two concrete pathologies, both observed in practice:

Pathology 1 — frequent classes grow longer weight vectors. If speaker 17 has 800 training utterances and speaker 4,022 has 40, the optimiser can lower total loss by simply making ‖w₁₇‖ larger. Nothing about speaker 17's direction improved; the class just got louder. At test time, when you delete the head and compare embeddings by cosine, all of that effort is invisible — you kept only the angles.

Pathology 2 — easy examples inflate their feature norm. A clean, long, close-miked utterance gets a large ‖x‖, which makes its logit large, which makes its loss small — without its angle ever having improved. The loss is satisfiable by being loud rather than by being correctly oriented.

Step one: normalise everything

The fix begins by removing the two nuisance quantities. Set the bias to zero, constrain every class weight to unit length, and normalise the feature to a fixed length s:

Now the only thing the loss can improve is the angle between the embedding and its class direction. Training and testing finally measure the same thing.

But this creates a new problem, and it is severe enough to be worth computing. With logits confined to s·[−1, 1], can the softmax ever be confident? Take s = 1 and the best possible embedding: the target class at cos = 1 and all 5,993 others at cos ≈ 0.

Chance level is log(5994) = 8.699. So a perfect embedding reduces the loss by 1.0 out of 8.7. There is almost no gradient signal distinguishing a great embedding from a mediocre one. Now set s = 30:

Step two: add the margin

Normalising alone gives a loss that is happy as soon as the target class merely wins. But "merely wins" is a terrible standard for an open-set problem: at test time the impostor will be someone who was not in the training set, possibly closer than any training impostor was. We want the target angle to be small with room to spare.

Additive angular margin softmax (AAM-softmax, ArcFace) does this by penalising the target class's angle before the softmax:

Only the target term gets the margin. The model is scored as if its embedding were m radians further from its own class than it actually is, so to satisfy the loss it must get m radians closer than it otherwise would.

ECAPA uses m = 0.2 radians. In degrees: 0.2 × 180/π = 11.459°. Hold on to that number, we are about to use it three times.

The margin, worked in degrees

Set up a concrete situation. An embedding sits 40° from its own speaker's class direction, and the nearest impostor class sits at 55°. Scale s = 30.

Without a margin:

Since only two classes matter here, the probability of the correct class is a logistic of the gap:

The loss is essentially zero. The optimiser has no reason to improve this embedding further, even though 40° is not a comfortable margin against an unseen impostor.

With m = 0.2 rad = 11.459°: the target angle is treated as 40° + 11.459° = 51.459°.

The loss went from 0.0031 to 0.2042 — 66 times larger — for an embedding that did not change at all. That is a gradient the optimiser will act on.

How much closer must it get? Solve for the angle that restores the original loss. We need the gap back at 5.774, so cos(θ_y + m) must reach 0.57358 + 5.774/30 = 0.57358 + 0.19247 = 0.76605, which is cos 40.00°. Therefore:

The decision boundary moves too, and in the same amount. Plain softmax puts the boundary between classes y and j where cosθ_y = cosθ_j, i.e. θ_y = θ_j. AAM puts it where cos(θ_y + m) = cosθ_j, i.e. θ_y = θ_j − m. The boundary is shoved m radians toward the target class, leaving a strip of angle that belongs to neither.

Push the margin slider to 0.6 rad and watch the loss stay enormous even when the embedding is nearly perfect. That is the practical failure mode: too large a margin makes the loss unsatisfiable from a random initialisation, and training either diverges or collapses. The standard remedy is a margin warm-up — start at m = 0 and ramp to the target over the first few epochs — which nearly every production recipe uses and nearly every paper mentions only in a footnote.

The family tree, and what each member changed

The reason ArcFace won, and the reason ECAPA uses it: with CosFace, subtracting a fixed m from the cosine translates into different angular penalties depending on where you are — near θ = 0 the cosine is flat, so a cosine margin of 0.35 costs a lot of degrees; near θ = 90° the cosine is steep, so the same 0.35 costs few. ArcFace's margin is uniform in the quantity that actually gets measured at test time. Geometry that matches the metric.

Why this kills PLDA

Return to Chapter 2's diagnosis. PLDA existed because the x-vector space had identity directions and nuisance directions mixed together, and something had to learn which was which. AAM-softmax trains the encoder so that:

This is the single most practically important consequence in the lesson. It is why a modern speaker verification system is one ONNX file and a dot product, and why the four-step centre-LDA-lengthnorm-PLDA pipeline of 2018 has quietly disappeared from production stacks. The honest caveat: PLDA still helps under severe domain shift, where the training-time cone assumption breaks — and adaptive score normalisation (AS-Norm) remains widely used for exactly that reason.

The loss in code, and the trap inside it

class AAMSoftmax(nn.Module):
    def __init__(self, emb_dim=192, n_spk=5994, s=30.0, m=0.2):
        super().__init__()
        self.W = nn.Parameter(torch.randn(n_spk, emb_dim))
        self.s, self.m = s, m
        self.cos_m, self.sin_m = math.cos(m), math.sin(m)
        self.th  = math.cos(math.pi - m)          # where θ + m crosses π
        self.mm  = math.sin(math.pi - m) * m      # the linear fallback slope

    def forward(self, emb, labels):              # emb: (B, 192)
        x = F.normalize(emb, dim=1)                # ‖x‖ = 1
        W = F.normalize(self.W, dim=1)             # ‖w_j‖ = 1
        cos = x @ W.t()                           # (B, n_spk) — pure cosines
        sin = torch.sqrt((1.0 - cos ** 2).clamp(0, 1))

        # cos(θ + m) = cosθ·cos m − sinθ·sin m — no arccos needed
        cos_m = cos * self.cos_m - sin * self.sin_m

        # θ + m > π: cos(θ+m) turns back UP, so the "penalty" would reward.
        # Fall back to a monotone linear surrogate past that point.
        cos_m = torch.where(cos > self.th, cos_m, cos - self.mm)

        one_hot = F.one_hot(labels, cos.size(1)).float()
        logits = self.s * (one_hot * cos_m + (1 - one_hot) * cos)
        return F.cross_entropy(logits, labels)

The torch.where line is the trap, and it is worth understanding rather than copying. The margin is applied as cos(θ + m), and the cosine is monotonically decreasing only on [0, π]. If an embedding is already almost opposite its class direction — θ close to π, which happens for a few examples early in training — then θ + m exceeds π, the cosine starts increasing again, and the "penalty" becomes a reward. The loss would then actively push those examples further away.

Work the boundary. With m = 0.2, the crossing is at θ = π − 0.2 = 2.9416 rad = 168.5°, where cosθ = −0.9801. So any embedding whose cosine to its own class is below −0.9801 falls into the bad region. The standard fix — ArcFace's, reproduced above — replaces the trigonometric penalty with a linear one, cosθ − m·sin(π−m), which is monotone everywhere and agrees with the correct expression at the boundary.

Also note that the implementation never calls arccos. The angle-addition identity cos(θ+m) = cosθ·cos m − sinθ·sin m gets the same result from the cosine alone, which matters because arccos has infinite derivative at ±1 and would produce NaN gradients for any embedding that lands exactly on a class direction.

How much room is there on the sphere?

A margin only makes sense if the space can hold all the classes at that separation. It is worth a rough count, because the answer explains why 192 dimensions is enough and why the margin cannot simply be raised.

Two class directions separated by at least an angle δ occupy non-overlapping spherical caps of angular radius δ/2. The fraction of a d-sphere's surface covered by one such cap scales, for small caps in high dimension, like sin^d−1(δ/2). So the number of classes that fit is bounded roughly by:

Put d = 192 and a modest required separation of δ = 20°, so δ/2 = 10° and sin(10°) = 0.1736:

Which reframes the margin. It is not fighting for space. It is forcing the encoder to reduce within-speaker spread, by refusing to accept a solution where a speaker's recordings merely land on the right side of a boundary. And that is why a margin that is too large does not "run out of room" — it becomes unsatisfiable because the encoder cannot compress the genuine session-to-session variation of a human voice below some floor, and asking it to do so just breaks training.

Two failure modes worth knowing

Label noise. VoxCeleb is built from YouTube, and a fraction of its clips are mislabelled — a different person in the same video, or a segment where the celebrity is not the one speaking. Under a plain softmax a wrong label is one bad example among millions. Under a margin loss it is much worse: the margin says "get much closer to this class than you would otherwise need to", so the loss on the mislabelled example is enormous and the gradient it produces is correspondingly large. Margin losses amplify label noise by construction.

The published fix is sub-centre ArcFace: give each class K sub-centres (K = 3 typically), take the maximum cosine over them, and let outliers cluster on a sub-centre of their own rather than dragging the main one. The pragmatic fix is data cleaning — the "cleaned" VoxCeleb trial lists exist for exactly this reason.

Margin warm-up. At random initialisation, embeddings are near-orthogonal to every class direction, so θ is around 90° for everything and cos(θ+m) is negative for the true class. The loss is then large and nearly flat across classes, and with a full margin from step zero the model frequently fails to escape. Ramping m from 0 to 0.2 over the first few epochs solves it. Every production recipe does this; most papers mention it in one clause, if at all.

Chapter 6: EER, minDCF, and Where to Stand

Every number quoted so far — 1.01%, 0.87%, 2.12% — is an equal error rate. This chapter derives what that means from zero, computes one by hand on ten trials, then shows that the operating point EER picks is, for a bank, catastrophically wrong — and computes the alternative by hand too.

The trial, and the two ways to be wrong

A verification system does not classify. It scores a trial: a pair consisting of an enrolment and a test utterance, plus a hidden truth about whether they are the same person. Vocabulary, which is worth fixing because the literature uses several sets of words for the same two mistakes:

The system emits a real-valued score. A threshold θ converts it into a decision: accept if score ≥ θ. Both error rates are therefore functions of θ, and they move in opposite directions:

Raise the threshold and you reject more impostors and more customers. Lower it and you admit more of both. There is no θ that reduces both. That trade-off is the system's quality: a better embedding is one where the two score distributions overlap less, so that every threshold is a better deal than it was.

Equal error rate, hand-computed

The EER is the single point where P_miss(θ) = P_fa(θ), and the reported number is that shared value. Let us compute one exactly on a set small enough to enumerate. Ten trials — five target, five non-target — with these cosine scores:

Notice the interleaving: one impostor (0.66) outscores three genuine trials, and one genuine trial (0.31) is beaten by two impostors. That overlap is the system's error. Now sweep θ and count.

Check the bolded row by hand. At θ = 0.55, the target scores at or above it are 0.82, 0.74, 0.61 and 0.55 — four out of five, so one miss, P_miss = 1/5 = 0.20. The non-target scores at or above 0.55: only 0.66 — one out of five, so P_fa = 1/5 = 0.20. The two rates are equal.

(The crossing is an interval rather than a point because scores are discrete. With 37,611 trials, as on VoxCeleb1-O, the curve is fine-grained enough that the crossing is essentially a point, and standard tools interpolate.)

Twenty percent is a terrible system, which is deliberate — the numbers had to be hand-checkable. A real ECAPA at 0.87% would need roughly 115 target trials before it misses one.

Why EER is the wrong place to stand

EER is a fine summary and a bad operating point, because it silently assumes the two errors cost the same. For a bank, a false acceptance is a fraud loss and a regulatory event; a false rejection is a mildly annoyed customer who tries again. The costs might differ by a factor of a thousand. And there is a second asymmetry: almost every trial is an impostor. Fraud attempts are rare, so the population is overwhelmingly genuine, or — in the attack-oriented framing NIST uses — the prior probability of a target trial is low.

The detection cost function puts both asymmetries in one expression:

Read the terms: each error rate is multiplied by how much that error costs and by how often that kind of trial occurs. It is an expected cost per trial. Divide by the cost of the best trivial system so that 1.0 means "no better than always saying no":

minDCF is the minimum of that over all thresholds — the cost you would pay if you had picked the threshold perfectly. The VoxCeleb convention is P_target = 0.01 with both costs at 1; NIST SRE has used 0.01, 0.005 and (with C_miss = 10) 0.001.

minDCF on the same ten trials, by hand

Use P_target = 0.05, C_miss = C_fa = 1. The normaliser is min(0.05, 0.95) = 0.05, so:

A false alarm is now worth nineteen misses. Recompute the sweep:

This is why papers report both numbers and why the two can rank systems differently. EER measures the overall separation of the two distributions. minDCF measures the quality of the tail — how far you can push the threshold up before genuine users start falling out. A system with beautiful average separation and a fat impostor tail wins on EER and loses on minDCF. When you read "0.87% EER / 0.107 minDCF", the second number is the one your security team cares about.

The DET curve, and why it is not a ROC

Plot P_miss against P_fa as θ sweeps and you get a detection error tradeoff curve. It differs from a ROC curve in two ways that both matter.

First, both axes are errors, so lower-left is better and the curve descends. Second — the important one — both axes are on a normal deviate scale: the position of a point at rate p is Φ⁻¹(p), the inverse of the standard normal CDF. Under the (roughly true) assumption that target and non-target scores are Gaussian with equal variance, this warping turns the DET curve into a straight line. Systems become comparable by slope and intercept rather than by squinting at curvature, and the interesting low-error region gets stretched out instead of being crushed into a corner.

The EER is where the DET curve crosses the diagonal P_miss = P_fa. minDCF is where a line of a particular slope (set by the cost ratio) is tangent to the curve. Two different geometric constructions on the same curve — which is exactly why they can pick different systems.

Score normalisation: the fix nobody mentions in the abstract

One practical problem breaks the whole framework: a single global threshold assumes all speakers' scores are on the same scale. They are not. Some voices are generic — they score moderately high against everybody, so their impostor trials cluster near the threshold. Some are distinctive and score near zero against everyone.

Adaptive score normalisation (AS-Norm) fixes this per trial. Keep a cohort of a few thousand impostor utterances. For the enrolment vector e, score it against the cohort, take the top-K most similar (K around 300), and compute that subset's mean μ_e and standard deviation σ_e. Do the same for the test vector t. Then:

In words: instead of asking "how similar are these two?", ask "how unusually similar are these two, compared with how similar this voice usually is to strangers?" It is a z-score against a personalised impostor distribution. In practice AS-Norm is worth 10–20% relative on minDCF, costs a few thousand extra dot products per trial, and appears in essentially every competitive VoxCeleb submission while rarely making it into an architecture diagram.

What to distrust when reading numbers

How many trials do you need to trust an EER?

Here is a question almost nobody asks about a reported error rate, and the arithmetic is elementary enough that there is no excuse.

An EER of 1% measured on N_tar target trials is an estimate of a proportion. The standard error of a proportion p from n independent samples is:

VoxCeleb1-O has roughly 18,800 target trials. For p = 0.01:

So a 95% interval is roughly ±1.96 × 0.073% = ±0.14 percentage points. A system reported at 0.87% and one reported at 0.95% are, on that list, statistically indistinguishable — and papers routinely present such a gap as an improvement.

Now run it the other way. To resolve a difference of 0.05 percentage points at 1% EER with any confidence you would need the standard error below about 0.025%, requiring:

Which is roughly why the extended list Vox1-E, with about 580,000 trials, exists. It is not merely "more data" — it is the resolution needed to distinguish modern systems at all.

Duration: the hidden axis in every table

Every EER in this lesson was measured on VoxCeleb utterances averaging around 8 seconds. Production audio is often much shorter, and the degradation is not gentle.

The mechanism is the pooling estimator again. Statistics pooling estimates a mean from T frames, and the standard error of a mean falls as 1/√T. Going from 8 s (about 800 frames) to 1 s (about 100 frames):

And a second, larger effect stacks on top: with 100 frames, which phonemes occurred stops averaging out. A one-second clip might contain four phonemes; a different one-second clip from the same speaker contains four different phonemes; and the pooled statistics of those two are genuinely different for content reasons, not identity reasons. Short-duration error rates are dominated by content leakage, which is why training on short crops (Chapter 4's recipe) helps far more than any architectural change.

Calibration: turning a score into a decision you can defend

EER and minDCF both assume you can find the right threshold. In deployment you must choose one before seeing the test data, and a raw cosine gives no principled way to do so. Calibration is the missing step.

The goal is to map the raw score to a log-likelihood ratio:

because a log-likelihood ratio composes with a prior by simple addition. Once you have one, the Bayes-optimal threshold is fixed by the costs and the prior alone, with no reference to any dataset:

Work it for the bank of Chapter 6: P_target = 0.01, C_miss = 1, C_fa = 10.

Accept only when the calibrated score exceeds 6.898 nats — that is, when the evidence for "same speaker" is about a thousand times stronger than the evidence against. No threshold sweep, no held-out set at deployment time, and if the fraud team changes C_fa to 50 you recompute the threshold in one line rather than re-running an evaluation.

Getting there in practice is usually one line of logistic regression on a held-out development set: fit s′ = a·s + b to maximise the likelihood of the development labels. Two parameters. The metric that measures whether it worked is C_llr, the cost of log-likelihood-ratio, which is the average information loss of your scores against a perfectly calibrated oracle — and unlike EER, it is sensitive to how wrong a confident wrong answer was, not merely to whether it crossed a line.

Chapter 7: Diarization — Who Spoke When

Verification has an enrolment. Diarization has nothing: forty-five minutes of a meeting, an unknown number of participants, no labelled examples of any of them, and a required output of the form speaker A from 00:03 to 00:19, speaker B from 00:19 to 00:31. It is the purest possible test of an embedding space, because there is no way to compensate with a good backend — if the geometry is wrong, the clustering is wrong.

The classical pipeline, with arithmetic

Put numbers on step 2. A ten-minute meeting with 600 seconds of actual speech, windowed at 1.5 s with a 0.75 s shift:

799 embeddings of 192 dimensions each: 153,408 floats, about 600 kB. The affinity matrix of all pairwise cosines is 799 × 799 = 638,401 entries — computed as one (799×192)(192×799) matrix product, roughly 245 MFLOPs. Nothing here is expensive. The whole difficulty is in choosing the window length and the clustering threshold.

The window-length dilemma

This is the central design tension of the classical pipeline and it has no good resolution.

The mechanism behind the first column is worth naming, because it is the same estimator argument in every field. Statistics pooling estimates a mean and a standard deviation from T frames. The standard error of a mean estimated from T samples falls as 1/√T. Going from a 3 s window (about 300 frames) to a 0.5 s window (about 50 frames) multiplies the noise in the pooled statistics by √(300/50) = √6 = 2.45×. And the extra error is not benign random jitter: with only 50 frames, which phonemes happened to occur becomes a first-order effect, so short-window embeddings drift toward content and away from identity.

Clustering with an unknown number of speakers

Agglomerative hierarchical clustering (AHC) is the simple answer: start with every window as its own cluster, repeatedly merge the two most similar clusters, and stop when the best available similarity falls below a threshold τ. The number of clusters at the stopping point is your speaker count.

The whole system's behaviour hinges on τ, and it fails in both directions. Set it too high and one speaker's windows — recorded across a shift in position, volume, or emotion — split into two clusters, so your two-person interview reports four speakers. Set it too low and two similar-sounding participants merge into one. Because τ is a fixed number applied to a similarity whose scale drifts with recording conditions, a threshold tuned on one corpus rarely transfers to another. AS-Norm from Chapter 6 helps here too, for exactly the same reason.

Spectral clustering is the more robust alternative. Build the affinity matrix A of pairwise similarities, refine it (row-wise thresholding, symmetrisation), form the normalised Laplacian, and take its eigenvalues. The eigengap heuristic picks k as the index of the largest jump between consecutive eigenvalues — the intuition being that a graph with k well-separated components has k near-zero eigenvalues followed by a gap. Then run k-means on the first k eigenvectors.

The eigengap is more stable than a similarity threshold because it reads the structure of the affinity matrix rather than the absolute value of any entry, and structure survives the scale drift that kills τ.

Diarization error rate, computed

DER is the total duration of erroneous audio divided by the total duration of speech. Three error types are summed:

Missed speech is time that was speech and got labelled as silence (or, in overlap, a second speaker who was never emitted). False alarm is silence labelled as speech. Confusion is speech attributed to the wrong speaker after the optimal one-to-one mapping between predicted and reference labels — found with the Hungarian algorithm, because label identities are arbitrary and "speaker 1" in your output may be "speaker 3" in the reference.

Worked example. A ten-minute meeting with 600 s of speech. The system misses 30 s, hallucinates 12 s of speech in silence, and attributes 48 s to the wrong participant:

Now score the overlap. Suppose 8% of the speech time — 48 s — has two people talking at once, and our system, like every clustering-based diarizer, emits exactly one label per instant. Every overlapped second is a missed second for the second speaker:

The same system, the same audio, the same output — and the error rate goes from 15.0% to 23.0% purely by deciding to score the hard part. Overlap rates of 5–15% are typical in real meetings and much higher in casual conversation.

The overlap problem, and why it is structural

Overlap is not an edge case that better engineering will clean up. It is a direct consequence of Chapter 1's design.

Take a 1.5 s window in which Ava and Ben are both talking. The frame network sees a mixture. Pooling averages over all its frames. The resulting embedding is, roughly, a point between Ava's cluster and Ben's cluster — and here is the fatal part: the average of two identities is a perfectly plausible third identity. The embedding space has no way to signal "this is a blend". A point between two clusters looks exactly like a person whose voice sits between them.

Which is why the modern answers all abandon the "embed then cluster" shape:

Note what EEND gives up to gain overlap handling: it is no longer producing a reusable speaker embedding at all. It produces activity curves for the speakers in this recording. You cannot take an EEND output and look someone up in a database. The embedding's weakness on overlap and its strength as a portable identity are two views of the same design decision.

Spectral clustering, worked on a four-window toy

The eigengap heuristic is usually stated and rarely demonstrated. It is small enough to do by hand, and doing it once makes the method stop feeling like magic.

Take four windows: two from speaker A, two from speaker B, with cosine similarities that reflect that structure. Here is the affinity matrix (self-similarities zeroed, as is standard):

The degree of each node is the sum of its row: 0.9 + 0.1 + 0.1 = 1.1, identically for all four. That uniformity makes the normalised Laplacian easy: with D = 1.1·I,

So the eigenvalues of L are 1 − λ_A/1.1, where λ_A are A's eigenvalues. By symmetry, A's eigenvectors are the four sign patterns:

Verify one row. For v₂ = (1, 1, −1, −1), the first component of A·v is 0(1) + 0.9(1) + 0.1(−1) + 0.1(−1) = 0.9 − 0.2 = 0.7, and 0.7 = 0.7 × 1, so λ = 0.7. Good.

Now sort the Laplacian eigenvalues ascending: 0.000, 0.364, 1.818, 1.818. The gaps between consecutive values are 0.364, 1.454, 0.000. The largest gap sits after the second eigenvalue, so the eigengap heuristic reports k = 2. Correct.

And the eigenvector that produced the decision, v₂ = (+1, +1, −1, −1), is the clustering: the sign pattern separates the two speakers. Running k-means on the first k eigenvectors is a way of reading that structure out when the pattern is less clean than this toy.

The pipeline in code

import numpy as np
from scipy.linalg import eigh

def diarize(audio, vad, encoder, win=1.5, shift=0.75, sr=16000):
    segments = vad(audio)                              # [(start, end), ...] speech only
    windows = []
    for (a, b) in segments:
        t = a
        while t + win <= b:
            windows.append((t, t + win)); t += shift
    E = np.stack([encoder(audio[int(a*sr):int(b*sr)]) for a, b in windows])
    E /= np.linalg.norm(E, axis=1, keepdims=True)     # (Nw, 192), unit

    A = E @ E.T                                        # (Nw, Nw) cosine affinity
    A = np.maximum(A, 0); np.fill_diagonal(A, 0)       # standard refinement
    d = A.sum(1) + 1e-9
    L = np.eye(len(A)) - (A / np.sqrt(np.outer(d, d)))  # normalised Laplacian

    vals, vecs = eigh(L)
    gaps = np.diff(vals[:10])
    k = int(np.argmax(gaps)) + 1                        # eigengap → speaker count

    U = vecs[:, :k]
    U /= np.linalg.norm(U, axis=1, keepdims=True) + 1e-9
    labels = kmeans(U, k)                              # (Nw,)
    return list(zip(windows, labels))                     # → RTTM after smoothing

The line that decides everything is k = argmax(gaps) + 1, and note that it searches only the first ten eigenvalues. Without that cap, numerical noise in the tail routinely produces a spurious maximum and the system reports forty speakers in a two-person interview. Capping the search at a plausible maximum speaker count is not a hack; it is a prior, and it should be set from what you know about the recording.

Two more metrics, because DER hides things

The DER-hides-short-speakers problem is worth a concrete case. In a meeting where one person talks for 80% of the time, a system that labels everything as that one speaker achieves a DER of 20% before doing any work at all — better than many real systems on hard audio. JER exists because that number is a lie, and any evaluation of a meeting product should report both.

Online diarization, where the whole design breaks

Everything above assumed the recording is finished. A live captioning product cannot wait: it must label each second as it arrives, and that breaks the pipeline in three specific places.

The practical answers are all forms of buying back information: keep a rolling buffer and permit retroactive relabelling within it, maintain running per-speaker centroids that update as evidence accumulates, and accept a fixed lag of a few seconds as the price of a usable error rate. This is the same tradeoff, in a different costume, as every other streaming inference problem on this site.

Chapter 8: The Lock and the Key Are the Same Vector

Chapter 0 listed four consumers of a speaker embedding. Three of them defend an account. The fourth builds a voice. This chapter is about what happens when the fourth consumer gets good, and it opens with the sentence that the field spent a decade trying not to say out loud.

This is not a hypothetical. Google's SV2TTS paper (2018) took a speaker encoder trained with the GE2E loss from Chapter 3 — the same loss, the same architecture — froze it, and used its 256-dimensional output to condition a Tacotron 2 synthesiser. Give it five seconds of a stranger's speech and it generates arbitrary new sentences in that voice. The verification model became the cloning model's steering wheel, with no modification.

The attack surface, from cheapest to hardest

Read the "effort" column top to bottom. It used to be that cloning a voice required hours of clean studio recordings of the target and a day of fine-tuning. VALL-E (2023) conditions on a three-second acoustic prompt and generates in that voice with no fine-tuning at all. YourTTS reaches zero-shot multilingual cloning from similarly short references. The reference audio requirement collapsed from hours to seconds, which moved the attack from "targeted, expensive" to "anyone who has left a voicemail".

Why the embedding cannot defend itself

Consider what a countermeasure would have to detect. The verification embedding was trained, on purpose, to be invariant to channel, to content, to duration, and to session. A synthetic utterance differs from a real one in exactly those dimensions plus a few subtle artefacts. Every invariance we trained in is an invariance the attacker inherits.

So the defence has to be a separate model asking a different question. Verification asks "is this Ava?". The countermeasure asks "was this produced by a human vocal tract at all?". Those are different classifiers with different features, and stacking them is called a tandem system.

The tandem, with arithmetic

Two systems in series, and a genuine user must pass both. Suppose:

Attack acceptance requires passing both, and the two decisions are roughly independent:

Genuine acceptance also requires passing both:

The countermeasure cut spoof acceptance by a factor of twenty and nearly two and a half times the rate at which real customers get locked out. That trade is the entire subject of the anti-spoofing literature, and it is why the community adopted t-DCF — the tandem detection cost function — which scores the combined system with costs for all three outcomes (miss a genuine user, accept an impostor, accept a spoof) rather than scoring the countermeasure alone.

t-DCF, written out

The tandem cost function generalises Chapter 6's DCF to three trial types instead of two. A trial can now be a genuine target, a zero-effort impostor (a different real person), or a spoof. Each has a prior and each mistake has a cost:

where C₁ and C₂ are constants that already fold in the fixed verifier's behaviour and the three priors and costs. Read what that structure says: the countermeasure is being scored through the verifier it sits in front of. C₂ contains the factor P_fa,spoof^ASV — the rate at which the verifier accepts spoofs it is shown — so a countermeasure's false-accept rate matters exactly in proportion to how vulnerable the verifier already was.

Two consequences fall straight out. If the verifier happened to reject a particular attack anyway (P_fa,spoof^ASV near zero for that attack), the countermeasure's mistakes on it cost almost nothing — so a countermeasure should spend its capacity on the attacks that do get through. And a countermeasure that rejects everything drives P_miss^CM to 1, so C₁ dominates and the cost is maximal, which is precisely the degenerate case a standalone EER would have rewarded with a perfect score.

What a countermeasure actually looks for

The pattern is stark. Every artefact-based signal has a shelf life, because each one names a specific weakness that the next generation of synthesisers will fix — often as a side effect of trying to sound better, not of trying to evade detection. Only the signals grounded in physics (there must be an acoustic path) or in interaction (the response must be produced now, in reply to something unpredictable) resist that erosion.

The irrevocability problem

Passwords have a property biometrics do not: you can change them. If a database of 40 million speaker embeddings leaks, nobody can be issued a new voice.

And embeddings are not opaque. They were built to be geometrically meaningful, which means a leaked template supports:

• Direct replay against any other system that uses the same encoder — and encoder reuse is near-universal, because everyone downloads the same pretrained ECAPA
• Cross-service linkage — the same person's template from two services will be near-identical, so leaked databases can be joined on voice, deanonymising pseudonymous accounts
• Attribute inference — embeddings carry usable information about sex, approximate age, accent and some health conditions, none of which the user consented to disclose
• Reconstruction — a vector that is good enough to condition a TTS decoder is good enough to synthesise the voice it describes, which is Chapter 8's opening sentence read backwards

Mitigations exist and none of them are free. Cancellable biometrics apply a secret, user-specific transform before storage so a leak yields a template that only works with one key — but the transform must preserve enough geometry to keep the comparison accurate, and every bit it preserves is a bit an attacker can exploit. On-device matching never transmits the template at all, which is a real improvement and moves the problem to device security. Fuzzy vaults and homomorphic matching are cryptographically appealing and, at scale, expensive.

Adversarial perturbation: attacking the vector directly

The attacks above try to sound like the target. There is a cheaper attack that does not bother: perturb the audio so the embedding lands where you want, regardless of what it sounds like.

The optimisation is the one every adversarial-example paper writes. Given a starting utterance x (the attacker's own voice) and a target template t, find a perturbation δ that maximises the score while staying imperceptible:

Put a number on the constraint. Audio is commonly represented in [−1, 1], and an ε of 0.002 is roughly 54 dB below full scale — well under the noise floor of any real recording chain. A gradient ascent of a few hundred steps against a white-box model routinely moves the cosine from near zero to well above any deployed threshold, with the result still sounding exactly like the attacker.

The black-box version is slower but real: an attacker who receives a numeric score per attempt can run a gradient-free search. Returning only accept/reject, and rate-limiting attempts per account, removes most of the signal that search needs. Design the interface, not only the model.

Replay, and the one thing physics gives you for free

Replay deserves separate treatment because it is the attack that actually happens at scale, and because it is the one where the defence rests on something an attacker cannot optimise away.

A replayed utterance has been through the recording chain twice: once when the attacker captured it, once when the microphone captured the loudspeaker. Each pass convolves the signal with a channel, and channels compose:

Loudspeakers are the weak link. A phone speaker rolls off sharply below about 300 Hz — it physically cannot move enough air to reproduce a 100 Hz fundamental — and adds measurable harmonic distortion when driven loudly. So a replayed recording carries a signature in the low band and in the harmonic structure that is a property of hardware, not of any generative model.

That is why replay detection generalises better than synthesis detection: the attacker would have to build a physically better loudspeaker, not train a better network. It is also why the ASVspoof challenges keep a separate physical-access track — the two problems have genuinely different structure, even though both are "someone is not who they claim".

A threat model you can actually plan against

Read the last row and then read Chapter 0's enrolment code again. template = normalize(np.mean(embs, axis=0)) writes an unchangeable biometric to a database. That line is thirty characters and it is the largest liability in the entire system.

Where this connects on this site

The generation side of this story is covered in its own lessons, and reading them next to this chapter is the point:

• TTS architectures — how a speaker embedding becomes a conditioning vector inside a synthesiser
• VALL-E — zero-shot cloning from a three-second prompt, with the acoustic prompt replacing the explicit speaker vector entirely
• Neural audio codecs — the discrete token layer that made codec-based synthesis, and its particular artefact signature, possible
• Self-supervised speech models — where modern countermeasures and modern speaker encoders both now get their front ends
• Embedding security — inversion, linkage and leakage risks for embeddings generally, of which the voice case is the sharpest

Chapter 9: Connections and Cheat Sheet

Nine chapters ago we started with three teams in three buildings. Here is what the shared artifact underneath them turned out to be, what the field has done with it since 2020, and where it sits in the rest of this site.

The one-paragraph summary of each paper

The cheat sheet

What changed after 2020

Where this sits in the rest of the site

A practical decision guide

Build it yourself, in the order that will not waste your time

If you want to internalise this lesson by writing code rather than reading, here is the order that keeps every step verifiable against a number you already have.

The ordering is deliberate. Steps 1, 2, 4, 5 and 6 each have a closed-form check in this lesson, so a bug announces itself immediately. The architecture — the part that feels like the real work — is step 7, because a wrong ECAPA block trains to a slightly worse number and tells you nothing, whereas a wrong pooling layer fails a three-line assertion.

Ten numbers worth memorising

The sentence to keep

References

1. Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., Khudanpur, S. "X-Vectors: Robust DNN Embeddings for Speaker Recognition." ICASSP 2018. PDF
2. Wan, L., Wang, Q., Papir, A., Moreno, I. L. "Generalized End-to-End Loss for Speaker Verification." ICASSP 2018. arXiv:1710.10467
3. Desplanques, B., Thienpondt, J., Demuynck, K. "ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification." Interspeech 2020. arXiv:2005.07143
4. Dehak, N., Kenny, P., Dehak, R., Dumouchel, P., Ouellet, P. "Front-End Factor Analysis for Speaker Verification." IEEE TASLP, 2011.
5. Prince, S. J. D., Elder, J. H. "Probabilistic Linear Discriminant Analysis for Inferences About Identity." ICCV 2007.
6. Variani, E., Lei, X., McDermott, E., Moreno, I. L., Gonzalez-Dominguez, J. "Deep Neural Networks for Small Footprint Text-Dependent Speaker Verification." ICASSP 2014.
7. Okabe, K., Koshinaka, T., Shinoda, K. "Attentive Statistics Pooling for Deep Speaker Embedding." Interspeech 2018. arXiv:1803.10963
8. Hu, J., Shen, L., Sun, G. "Squeeze-and-Excitation Networks." CVPR 2018. arXiv:1709.01507
9. Gao, S., Cheng, M.-M., Zhao, K., Zhang, X.-Y., Yang, M.-H., Torr, P. "Res2Net: A New Multi-scale Backbone Architecture." IEEE TPAMI, 2021. arXiv:1904.01169
10. Deng, J., Guo, J., Xue, N., Zafeiriou, S. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition." CVPR 2019. arXiv:1801.07698
11. Schroff, F., Kalenichenko, D., Philbin, J. "FaceNet: A Unified Embedding for Face Recognition and Clustering." CVPR 2015. arXiv:1503.03832
12. Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., Liu, W. "CosFace: Large Margin Cosine Loss for Deep Face Recognition." CVPR 2018. arXiv:1801.09414
13. Nagrani, A., Chung, J. S., Zisserman, A. "VoxCeleb: A Large-Scale Speaker Identification Dataset." Interspeech 2017. arXiv:1706.08612
14. Chung, J. S., Nagrani, A., Zisserman, A. "VoxCeleb2: Deep Speaker Recognition." Interspeech 2018. arXiv:1806.05622
15. Snyder, D., Chen, G., Povey, D. "MUSAN: A Music, Speech, and Noise Corpus." 2015. arXiv:1510.08484
16. Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., Wu, Y. "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis." NeurIPS 2018. arXiv:1806.04558
17. Wang, C., Chen, S., Wu, Y., et al. "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers" (VALL-E). 2023. arXiv:2301.02111
18. Casanova, E., Weber, J., Shulby, C., Junior, A. C., Gölge, E., Ponti, M. A. "YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion." ICML 2022. arXiv:2112.02418
19. Fujita, Y., Kanda, N., Horiguchi, S., Xue, Y., Nagamatsu, K., Watanabe, S. "End-to-End Neural Speaker Diarization with Self-Attention." ASRU 2019. arXiv:1909.06247
20. Medennikov, I., Korenevsky, M., Prisyach, T., et al. "Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario." Interspeech 2020. arXiv:2005.07272
21. Landini, F., Profant, J., Diez, M., Burget, L. "Bayesian HMM Clustering of x-vector Sequences (VBx) in Speaker Diarization." Computer Speech & Language, 2022. arXiv:2012.14952
22. Bredin, H., et al. "pyannote.audio: Neural Building Blocks for Speaker Diarization." ICASSP 2020. arXiv:1911.01255
23. Todisco, M., Wang, X., Vestman, V., et al. "ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection." Interspeech 2019. arXiv:1904.05441
24. Kinnunen, T., Lee, K. A., Delgado, H., et al. "t-DCF: a Detection Cost Function for the Tandem Assessment of Spoofing Countermeasures and Automatic Speaker Verification." Odyssey 2018. arXiv:1804.09618
25. Chen, S., Wang, C., Chen, Z., et al. "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing." IEEE JSTSP, 2022. arXiv:2110.13900
26. Koluguri, N. R., Park, T., Ginsburg, B. "TitaNet: Neural Model for Speaker Representation with 1D Depth-wise Separable Convolutions and Global Context." ICASSP 2022. arXiv:2110.04410
27. Ravanelli, M., Bengio, Y. "Speaker Recognition from Raw Waveform with SincNet." SLT 2018. arXiv:1808.00158