Audio Data Pipelines

A thousand hours of raw podcasts are not training data yet. A pipeline cuts, labels and filters them. Turn up the strictness: fewer hours survive, and their labels get cleaner.

Press Light, then Standard, then Strict. Each green square is a 4-hour recording that passes. Watch the hours fall and the label error fall with them. Then we build the pipeline piece by piece: decoding, loudness, speech detection, speakers, transcripts, alignment, text tagging, quality scores, and leaks.

You need a rough idea of what a sound wave is and a little Python. We build the rest from zero.

Built from ten primary sources, among them Whisper, WhisperX, CTC-Segmentation, Emilia and YODAS. The sources are listed in Chapter 11.

Filter a thousand hours

Illustrative: 250 made-up recordings of 4 hours each, generated the same way on every visit. Each has a noise level, a music flag, an overlap share and a label error. Light drops music and very noisy audio. Standard also drops overlap above 15% and clips where two ASR systems disagree by over 20%. Strict tightens all three.

Chapter 0

A Thousand Hours of Raw Audio

See why a pile of podcast audio is not yet training data, and what a pipeline must do to it

Picture a shared drive with a thousand hours of podcast episodes on it. A teammate says, “Train a speech model on this by Friday.” You press play on episode one. It opens with 90 seconds of theme music. Two hosts talk over each other for a minute. A truck drives past the window. Then a long, quiet pause, and an ad read in a different voice.

Every one of those moments will teach the model something. Most of it is the wrong thing. This chapter shows what is wrong with raw audio, and what a pipeline does about it.

Audio is not data until you say what each second is

A speech model learns from pairs: a short clip of sound, and the words spoken in it. We call the words the label, the way a photo of a cat carries the label “cat”. Raw episodes give you neither part cleanly. The sound has parts that are not speech. The words are missing, or they come as one long show-notes page, or they sit in a caption file that drifts out of step with the voice.

A data pipeline is a chain of small programs. Each one takes the output of the last one and fixes one problem. The word comes from plumbing: the audio goes in at one end, and clean pairs come out of the other. Think of a water treatment plant. No single step makes water safe. Each step removes one kind of dirt.

Why this work matters at scale

Big speech systems are built from huge piles of audio. Whisper trained on 680,000 hours of multilingual and multitask supervision (Radford et al., arXiv 2212.04356). The YODAS dataset holds over 500k hours in more than 100 languages (arXiv 2406.00899). Emilia starts with over 101k hours of speech across six languages (arXiv 2407.05361). The People’s Speech is a 30,000-hour and growing English set (arXiv 2111.09344). Nobody listens to that much audio. A pipeline decides, and a pipeline can be wrong in quiet ways.

The point is not to keep everything. The point is to keep the hours where the sound and the words match, and to know how many hours that leaves. The hero at the top of this page showed the trade: a stricter filter keeps fewer hours and cleaner labels. Now we look at one hour of raw audio to see what the filter is cutting.

One raw hour, minute by minute

Below is one illustrative hour of a podcast, drawn as a 60-minute bar. Four kinds of trouble sit in it. Music is sound that is not speech, such as theme tunes. Silence is a stretch with no voice. Overlap is two people talking at once, so no single transcript fits. Noise is sound that hides the voice, such as a truck. Tap each one to cut it, and watch the usable minutes.

Cut the unusable minutes

The warm bar is one 60-minute episode. Press a button to cut that kind of audio. Cut regions turn hollow.

Illustrative hour: 6 minutes of music (intro and outro), 5 of silence, 4 of overlap, 7 of noise, 38 of clean speech. The minutes are made up to show the idea; real shares differ by show.

Even in this friendly hour, 22 minutes are not clean speech. If every hour looked like this one, a thousand hours would give about 633 usable hours. The other 367 are not lost by accident. A good pipeline finds them on purpose, and it can tell you how many it cut and why.

The pipeline, in the order we build it

Here is the whole chain on one line, so you can see where each chapter fits. First, decode every file into numbers and set one sample rate. Second, set one loudness and one channel layout. Third, find where speech lives and cut it into chunks. Fourth, say who speaks. Fifth, get words. Sixth, pin each word to the clock. Seventh, tag clips with text. Eighth, score the quality. Ninth, remove copies and keep the test set clean. Then check the scale, and learn how real teams fail.

Each step adds a column to a record. By the end, one training example looks like this, in the JSON-lines style that many teams use:

json{"id": "pod042_000317", "audio": "shards/pod042/000317.flac", "sr": 16000, "dur": 11.84,
 "speaker": "SPK_01", "text": "so the first thing we tried was a smaller model",
 "words": [["so", 0.12, 0.31], ["the", 0.31, 0.44], ["first", 0.44, 0.83]],
 "snr_db": 24.1, "quality": 3.7, "lang": "en", "agree_wer": 0.04, "split": "train"}

Every field in that record comes from one chapter. When a model behaves oddly, you read the record, find the column that lied, and fix the step that wrote it.

  1. Decode, resample and count: arrays, sample rates and the size of a thousand hours.
  2. Loudness and channels: one level, one layout.
  3. Speech detection and segments: find the voice and cut it to length.
  4. Diarization: who is talking.
  5. Transcripts: weak labels, pseudo-labels and agreement.
  6. Forced alignment: word timestamps from a trellis.
  7. CLAP: tag clips with text, score captions, find copies.
  8. Quality filters: noise, DNSMOS-style scores, language, music, overlap.
  9. Dedup and leaks: copies, eval decontamination, speaker splits.
  10. Real datasets and scale: Emilia, YODAS, MMS and the storage math.
  11. Field guide: numbers, checklist, code and sources.
A team has 1,000 hours of podcasts and trains on all of them. What is the main problem?

Chapter 1

Decode, Resample and Count the Samples

Turn a file into an array, learn its shape, and size a thousand hours before you buy the disk

You double-click an episode. It is an MP3 file of 58 MB, and a media player turns it into sound. A neural network cannot use the file as it is. It needs a plain list of numbers. This chapter turns a file into that list, shows its shape, and prices a thousand hours of it.

Sound becomes a list of numbers

A microphone has a thin sheet that moves when air pressure changes. The computer reads how far the sheet has moved, again and again, many times each second. Each reading is one sample: a single number. The number of readings each second is the sample rate, measured in hertz (Hz). A rate of 16,000 Hz means 16,000 numbers for every second of sound.

So one mono clip is a one-dimensional array. For a clip of 11.84 seconds at 16,000 Hz, the shape is (189,440,), because 11.84 × 16,000 = 189,440. A stereo clip has two channels, left and right, and its shape is (2, N) in most Python libraries. The type of each number matters too. int16 stores whole numbers from −32,768 to 32,767 in 2 bytes. float32 stores decimals, usually between −1.0 and 1.0, in 4 bytes. Models want float32; disks are happier with int16.

Decoding: from a compressed file to samples

MP3, AAC and Opus files are compressed: a program threw away detail that people rarely hear, so the file is small. Decoding is the reverse step. It rebuilds the full list of samples. Most pipelines call ffmpeg, a free tool that reads almost any format, and ask it to write raw samples at the rate and layout they want. That is the one place where you set the sample rate for the whole pipeline.

pythonimport subprocess, numpy as np

def load_mono_16k(path):
    # ffmpeg decodes, mixes to 1 channel (-ac 1), resamples to 16 kHz (-ar 16000)
    cmd = ["ffmpeg", "-v", "error", "-i", path, "-ac", "1", "-ar", "16000", "-f", "s16le", "-"]
    raw = subprocess.run(cmd, capture_output=True, check=True).stdout
    x = np.frombuffer(raw, dtype=np.int16)           # shape (N,), dtype int16
    return x.astype(np.float32) / 32768.0           # shape (N,), values in [-1, 1)

x = load_mono_16k("pod042.mp3")
print(x.shape, x.dtype, len(x) / 16000)           # (57600000,) float32 3600.0

A worked size calculation

Before you buy disks, do the arithmetic once, by hand. One hour has 3,600 seconds. At 16,000 Hz that is 16,000 × 3,600 = 57,600,000 samples. In int16, each sample takes 2 bytes, so one hour is 115,200,000 bytes, or 115.2 MB. A thousand hours is 115.2 GB.

Now change two things. Use float32 and the same hour takes 230.4 MB, so a thousand hours take 230.4 GB. Keep 44,100 Hz stereo float32 (a common studio format) and one hour is 44,100 × 3,600 × 2 channels × 4 bytes = 1,270,080,000 bytes. A thousand hours then fill 1,270 GB. The format you pick can change the bill by more than ten times.

Why the sample rate cannot be anything

A pure tone repeats its wave many times per second. To see one wave, you need at least two samples inside it: one near the top, one near the bottom. This is the Nyquist limit: a sample rate of fs can only describe tones below fs/2. At 16,000 Hz that limit is 8,000 Hz.

A tone above the limit does not vanish. It folds back and shows up as a lower tone that was never in the room. This is aliasing. The formula for the fake tone is short.

falias
The frequency you hear after sampling. It is the fake tone.
f
The true tone in the room, in Hz.
k
The whole number of sample rates that f is closest to. It tells how many times the tone folded over.
fs
The sample rate in Hz.

f = 6,000, fs = 8,000: k = round(0.75) = 1, so falias = |6,000 − 8,000| = 2,000 Hz

Try it. Slide the tone up and pick a sample rate. The thin line is the true tone; the dots are the samples; the thick line is what the dots seem to say.

Sample a tone, break the rule

Choose a sample rate, then drag the tone upward. When it passes half the rate, the thick line stops following the thin one.

6,000 Hz

Tap to hear a sweep from 300 to 10,000 Hz at the rate you chose. One version keeps every k-th sample. The other low-passes first.

The geometry is exact: the dots are samples of a sine of the chosen tone over 1 millisecond, and the thick line is the sine of the alias frequency from the formula above. Storage uses one thousand hours, mono, int16. Real resamplers low-pass the audio first so no tone above the new limit survives to fold.

Failure: resampling artifacts

Symptom. A model scores well on studio recordings and poorly on phone calls, or you hear a faint whistle in the training clips. Measurement. Plot the spectrum (energy by frequency) of a few dozen clips. Clips that began at 8,000 Hz and were later raised to 16,000 Hz have almost no energy between 4,000 and 8,000 Hz. They carry a sample rate of 16,000 Hz, but only the detail of 8,000 Hz. Clips that were lowered badly show hills of fake energy near the top. Fix. Resample once, with a proper low-pass filter (the soxr or sinc resamplers do this), and store the original rate in the record so you can filter the “fake 16 kHz” clips later.

You take a recording made at 8,000 Hz and save it as a 16,000 Hz file. What did you gain?

Chapter 2

Make Every Clip Sound Like the Same Room

Set one loudness and one channel layout so the model never learns the microphone

Two episodes sit side by side in your folder. One host whispers into a cheap microphone. Another host shouts into a studio mic and the signal hits the ceiling. A model trained on both would learn one more thing than you wanted: how loud the recording was. This chapter removes that accident, and fixes a second one hiding in stereo files.

Loudness is a number you can measure

Samples in a float array run from −1.0 to 1.0. A loud clip has large swings; a quiet clip has small ones. To measure the typical swing, square every sample, average the squares, and take the square root. This is the root mean square, or RMS. Squaring matters: it makes negative swings count the same as positive ones.

People hear loudness on a log scale, so we write it in decibels relative to full scale (dBFS): 20 × log10(RMS). Full scale is 1.0, which is 0 dBFS. Everything quieter is negative. Every 6 dB halves or doubles the swing, roughly.

Take four samples: 0.1, −0.2, 0.2, −0.1. The squares are 0.01, 0.04, 0.04, 0.01. Their mean is 0.025, and the square root is 0.158. That is 20 × log10(0.158) = −16.0 dBFS.

Normalize: one gain per clip

Loudness normalization picks a target level and multiplies the whole clip by one number, the gain, to reach it. In decibels the gain is simple subtraction: target minus measured. If the target is −20 dBFS and the clip sits at −16.0, the gain is −4.0 dB, which is a multiplier of 10(−4.0/20) = 0.631. A whisper clip at −38 dBFS needs +18 dB, a multiplier of 7.94.

Many teams use LUFS, a loudness unit weighted to how ears hear, instead of plain RMS. The recipe is the same: measure, subtract, apply one gain. Plain RMS is enough to understand the idea, and it is what the device below uses.

Gain has a risk. Speech has short, sharp peaks, much higher than its average. When you push the whole clip up, the peaks can pass 1.0. The format cannot store them, so they are cut flat. That is clipping, and it adds harsh crackle the speaker never made. Set the clip’s starting level, pick a target, and see how many samples clip.

Normalize, and watch the ceiling

The top wave is the clip as recorded. The bottom wave is after one gain. Dashed lines are the ceiling at plus and minus 1.0. Samples past the ceiling turn red.

-38 dBFS

Tap to hear the clip as recorded, then after the gain you set. Push the target to -6 and hear the crackle of clipping.

Illustrative: 800 synthetic samples shaped like syllables, the same on every visit. RMS, gain and clip counts are computed from those samples in your browser. Real speech peaks differ by voice and microphone.

Failure: a clip that was already clipped

Symptom. The model produces crackly output, or it fails on loud speakers only. Measurement. Count the samples whose absolute value is 0.999 or higher. In a healthy clip the count is near zero. In a clipped clip it can be thousands. Fix. Reject clips above a small share (for example 0.1%) before you normalize. Gain cannot undo flat tops; it only makes them louder.

Channels: how to make stereo into mono without losing the voice

Most speech models take one channel. A stereo file has two, left and right, and the easy fix is to average them: mono = (left + right) / 2. Usually this works, because both channels hear the same voice.

Sometimes it does not. If a recording chain flips the sign of one channel, the two waves are exact opposites. Averaging them gives zero: silence. This is phase cancellation. Partial offsets give partial loss. For two copies of the same tone shifted by an angle φ, the average has amplitude |cos(φ/2)|. At 0 degrees it is 1. At 90 degrees it is 0.71. At 180 degrees it is 0.

Average the channels

Left and right carry the same tone. Turn the phase offset and look at the mono mix, the average of the two.

0 degrees

Exact: both channels are unit sine waves, and the mono mix is their average, computed point by point. The check in the code below, comparing the RMS of the mono mix with the louder channel, catches this fault in real files.

pythondef to_mono_checked(stereo, target_db=-20.0):      # stereo: shape (2, N), float32
    mono = stereo.mean(axis=0)                          # shape (N,)
    rms = lambda a: np.sqrt(np.mean(a ** 2)) + 1e-9
    if rms(mono) < 0.5 * max(rms(stereo[0]), rms(stereo[1])):
        return None, "phase_cancel"                    # drop or repair: do not train on it
    if (np.abs(mono) >= 0.999).mean() > 0.001:
        return None, "clipped"
    gain_db = target_db - 20 * np.log10(rms(mono))
    out = mono * 10 ** (gain_db / 20)
    return np.clip(out, -1, 1), "ok"
A clip measures −38 dBFS and the target is −20 dBFS. What happens, and what is the risk?

Chapter 3

Find the Speech and Cut It to Size

Detect where people talk, then cut the hours into chunks of a length a model can train on

You now hold a two-hour episode as one array of 115,200,000 numbers at 16,000 Hz. You cannot train on that as one example. A model needs examples of a few seconds each, and each one must start and end in sensible places. This chapter finds where speech lives, then cuts the array into pieces.

Why not cut every 10 seconds?

A fixed cut is easy and wrong. It slices words in half, so the left piece ends with “fi” and the right piece starts with “rst”. Neither half matches any transcript. It also keeps silence and music, which we wanted to drop. We need to look at the sound first.

The tool for that is a voice activity detector, or VAD. It is a small model that listens to short frames, often 10 to 30 milliseconds long, and for each frame outputs the probability that someone is speaking. Think of a light switch controlled by a very fast, very tired listener. The output for a whole episode is a list of numbers between 0 and 1, one for each frame.

From probabilities to segments: three rules

Probabilities are not segments yet. Three rules turn them into start and end times. Each rule has one knob.

  1. Threshold. A frame counts as speech if its probability is above the threshold, say 0.5. Lower it and you catch quiet words but also noise.
  2. Minimum silence to split. Speakers breathe. If the gap between two speech runs is shorter than this, join them. Otherwise one sentence turns into several fragments.
  3. Maximum length. If a joined run is longer than your cap, split it. Split where the probability is lowest, at the quietest moment, not at the exact cap.

Here is a worked case with a frame every 0.5 seconds. The probabilities are 0.1, 0.2, 0.9, 0.95, 0.8, 0.3, 0.85, 0.9, 0.1, 0.05. With threshold 0.5, frames 2, 3, 4, 6 and 7 are speech. Frame 5 is a half-second gap. With minimum silence set to 0.5 seconds, the gap is joined, and we get one segment from frame 2 to frame 7: 3.0 seconds. With minimum silence set to 0.25, the gap stays and we get two segments, of 1.5 and 1.0 seconds. The same audio gives different examples depending on one knob.

Cut 90 seconds into chunks

The warm curve is the VAD output. The pink line is your threshold. Blue bars are the chunks you get. The thin green strip is where people really spoke.

0.50
0.5 s
15 s

Tap to hear only the kept chunks, squeezed in time, with a click at every cut. A hiss among the voices is a false alarm.

Illustrative: a made-up 90-second recording with a known speech mask, a door slam at 55 seconds and short pauses between turns. The VAD curve is the mask plus seeded noise, with 0.25-second frames. Recall and junk are counted against the mask.

Look for the trade. A high threshold gives clean chunks but loses the soft ends of words. A low threshold keeps every word but also keeps the door slam. A long minimum silence gives long, natural chunks; a short one gives many fragments. There is no setting that is right for every show. You pick one by measuring on a small hand-checked sample, which Chapter 8 returns to.

One more design choice matters: the length cap. Short chunks are cheap to batch and to align, and an error in one cannot spread far. Very short chunks lose context, so a model hears less of a sentence. A common compromise is a cap in the range of ten to thirty seconds. Treat the exact number as a setting to measure, not a rule.

Where real pipelines do this

WhisperX (arXiv 2303.00747) puts a VAD step in front of the recognizer. Its abstract describes speech-region detection and a “VAD Cut & Merge” pre-segmentation, which splits long audio into pieces that are then transcribed in batches. The abstract reports a twelve-fold transcription speedup from batching. The lesson for a data builder: cutting is not only for tidy examples. It also lets you run many chunks through a model at once.

pythondef segment(p, hop=0.25, thr=0.5, join=0.5, max_len=15.0, min_len=0.5):
    # p: shape (T,), speech probability per frame, from a VAD model
    on = p > thr
    runs, i = [], 0
    while i < len(on):
        if on[i]:
            j = i
            while j + 1 < len(on) and on[j + 1]: j += 1
            runs.append([i, j + 1]); i = j + 1
        else: i += 1
    merged = []
    for a, b in runs:                                  # join runs separated by a short gap
        if merged and (a - merged[-1][1]) * hop <= join: merged[-1][1] = b
        else: merged.append([a, b])
    out, cap = [], int(max_len / hop)
    for a, b in merged:
        while b - a > cap:                                # too long: cut at the quietest frame
            lo = a + int(0.6 * cap)
            k = lo + int(np.argmin(p[lo:a + cap]))
            out.append((a * hop, k * hop)); a = k
        if (b - a) * hop >= min_len: out.append((a * hop, b * hop))
    return out                                          # list of (start_s, end_s)

Failure: clipped word starts

Symptom. Transcripts miss the first word of many chunks, or the model drops initial consonants. Measurement. Add 0.2 seconds of padding on each side for a sample of 200 chunks, re-run the recognizer, and compare. If the word error rate drops, your cuts were too tight. Fix. Pad each chunk by 0.1 to 0.2 seconds, use a lower threshold with a short minimum-silence join, and never let the padding overlap a neighbour.

A long chunk must be split because it passes the maximum length. Where should the cut go?

Chapter 4

Who Is Talking: Speaker Diarization

Label each stretch of speech with a speaker, and see why conversational models need it

Two hosts chat for an hour. The transcript says, “So the first thing we tried was a smaller model.” Who said it? For a model that writes one long monologue, nobody cares. For a model that answers in a conversation, everything depends on it. This chapter adds the missing column: who spoke, and when.

Why speaker labels matter

A conversational speech model must learn turn-taking: when to talk, when to wait, when to say “mm-hm”. It can only learn that from data where each turn is tied to a speaker. A model that copies a voice needs the opposite guarantee: a clip must hold one voice, not two. Both needs come from the same step, called speaker diarization. The name means “who spoke when”.

Diarization does not know anyone’s name. It gives anonymous labels: SPK_00, SPK_01. Think of a seating chart made by listening. You do not learn who sits where, only that the same person talks from seat 1 each time.

Four steps: window, embed, cluster, label

First, take the speech regions from the VAD and cut them into short windows, about 1 to 2 seconds. Second, run each window through a speaker embedding model. It turns a window into a short list of numbers, a vector, that works like a voiceprint. Windows of the same voice land near each other in space. Windows of different voices land far apart. Real embeddings have a few hundred numbers; we draw two.

Third, cluster the vectors: group the ones that are close. Fourth, give each group a label and write each window’s label back onto the timeline. Neighbouring windows with the same label join into one turn.

How do we say two voiceprints are close? A common measure is the cosine similarity: the angle between the two vectors. Take (3, 4) and (4, 3). The dot product is 3×4 + 4×3 = 24. Each has length 5, so the similarity is 24 / (5 × 5) = 0.96, nearly the same direction. Compare (3, 4) with (−4, 3): the dot product is −12 + 12 = 0, so the similarity is 0, perpendicular. The first pair sounds like one voice; the second like two.

Here is what the output looks like for the first ten seconds of our two-host episode. Each line is one turn, written as start time, end time and label. The labels are anonymous, and the times come from the VAD cuts of Chapter 3.

json[{"start": 3.20, "end": 9.85, "speaker": "SPK_00"},
 {"start": 9.85, "end": 11.10, "speaker": "SPK_01"},
 {"start": 11.40, "end": 17.62, "speaker": "SPK_00"}]

A conversational model reads this as a script: speaker 0 talks for 6.65 seconds, speaker 1 answers for 1.25 seconds, speaker 0 resumes after a gap of 0.3 seconds. That gap is a signal for turn-taking. If the labels are wrong, the gaps are wrong too.

The threshold is the whole game

The clustering rule has one important knob: how close is close enough to join? Set it too tight and one speaker splits into two labels, because her voice drifts when she laughs. Set it too loose and two speakers merge into one. The hardest windows are overlaps, where two people talk at once. Their voiceprint lands between the two speakers and acts as a bridge. Link the bridge and the two clusters become one.

Group voiceprints into speakers

Each dot is a 1.5-second window drawn at its voiceprint. Colour shows the speaker label your threshold gives it. Grey dots are windows left unlabelled. Drag the threshold, and try adding the overlap windows.

0.45

Illustrative: 60 made-up windows from 3 speakers (20 each, seeded), plus 4 overlap windows between speakers A and B when switched on. Clustering is single-link: two windows join when the gap between them is below your threshold. Clusters of fewer than 3 windows stay unlabelled. Real embeddings have hundreds of dimensions.

pythonfrom sklearn.cluster import AgglomerativeClustering

def diarize(windows, embed, thr=0.5):
    # windows: list of (start_s, end_s) inside speech regions; embed: audio -> vector (D,)
    E = np.stack([embed(w) for w in windows])             # shape (n_windows, D)
    E /= np.linalg.norm(E, axis=1, keepdims=True)
    lab = AgglomerativeClustering(n_clusters=None, distance_threshold=thr,
                                  metric="cosine", linkage="average").fit_predict(E)
    return [{"start": s, "end": e, "speaker": f"SPK_{l:02d}"} for (s, e), l in zip(windows, lab)]

Failure: two voices in one “single-speaker” clip

Symptom. A voice model trained on your clips produces a blended, unstable voice, or a dialogue model answers as the wrong person. Measurement. For each clip, embed its windows and check the spread. If they form two groups, the clip has two speakers. Also count how many clips get a speaker label seen in fewer than three windows. Fix. Cluster inside each episode, not across the whole corpus, unless you have a separate step that links the same person across episodes. Mark overlap regions and drop or flag them. Keep the tightest threshold that does not split a speaker, found by checking a small hand-labelled set.

A clustering threshold is set very loose. What most likely happens to an episode with two hosts?

Chapter 5

Where the Words Come From

Get text for every chunk from the web and from a strong ASR model, then keep only what agrees

You have clean chunks of speech, and a speaker label for each. You still have no words. Where do ten thousand hours of transcripts come from? Paying people to type them would cost a fortune. So pipelines use two cheaper sources, and a trick to check them.

Three places a transcript can come from

Human labels are typed by people and checked. They are the best and the rarest. Weak labels are text that already exists near the audio: video captions, subtitles, show notes. They cost nothing, but nobody checked them, and they can be late, summarised or wrong. The Whisper paper (arXiv 2212.04356) puts the idea in its title, “Large-Scale Weak Supervision”, and reports training on 680,000 hours of multilingual and multitask supervision.

Pseudo-labels come from a strong speech recognizer, an ASR model (automatic speech recognition), that you run yourself on your chunks. The word “pseudo” means “pretend”: the labels are a good guess, not a fact. The recognizer is a student who has read a lot. It is often right and sometimes confidently wrong.

How wrong is a transcript? Word error rate

To compare two transcripts we need a number. Word error rate (WER) counts the fewest edits that turn one into the other. An edit is a substitution (one word swapped), a deletion (a word missing) or an insertion (a word added). Divide the edits by the number of words in the reference.

WER = (S + D + I) / N

Worked example. The reference is “we trained it on four thousand hours”: 7 words. The recognizer writes “we trained it on forty thousand hours”. One word differs, so S = 1, D = 0, I = 0. WER = 1 / 7 = 14.3%. The error is small in words and large in meaning: four thousand became forty thousand.

Filter by agreement

Suppose you have two independent guesses for the same chunk: the web caption and your recognizer’s output. If they match, it is unlikely both are wrong in the same way. If they differ, at least one is wrong, and you cannot tell which one. The cheap, safe move is to compare them with WER and keep the chunk only when the WER is below a threshold. This is agreement filtering.

It is like asking two friends the way to the station. If both point left, go left. If one says left and one says right, do not guess. Ask neither, and skip this chunk.

Keep what agrees

Each bar is the word error rate between a chunk’s web caption and the recognizer’s text. Move the threshold. Pick a chunk below to read both texts.

10%

Illustrative: eight short made-up chunks. The WER is computed live by a word-level edit distance. The recognizer text is the label we would train on; whether it is truly right was decided by the author of the toy. Chunk 8 is a harmless spelling difference that the filter drops anyway.

Move the threshold up to 20% and a chunk with “forty thousand” slips through. Move it down to 5% and you lose every chunk where the two sources differ in harmless ways. This is the usual trade: a stricter filter keeps cleaner labels and fewer hours. To lose fewer good chunks, normalise both texts first: lower-case, no punctuation, numbers and spellings written one way.

Failure: hallucinated transcripts

A recognizer that writes fluent text will write fluent text even when there is nothing to transcribe. Over music or a long silence it may produce a plausible sentence, or repeat one phrase again and again. Symptom. Transcripts of non-speech chunks contain polite closing phrases, or the same sentence appears thousands of times. Measurement. Count the most common whole transcripts; a real corpus has few exact repeats. Check words-per-second: a 10-second chunk with 60 words, or with 0.3 words per second, is suspect. Agreement helps here, as chunk 5 shows: the web caption says “[music]” and the recognizer writes a sentence. Fix. Filter by agreement, by speech rate, and by the VAD and music scores of Chapter 8. Agreement cannot catch an error both sources share, so keep a small human-checked set to measure it.

pythonimport jiwer, re

def norm(s):
    s = re.sub(r"\[.*?\]", "", s.lower())                     # drop [music] style tags
    return re.sub(r"[^a-z0-9' ]", "", s).strip()

def keep(web, asr, max_wer=0.10, min_wps=0.8, max_wps=5.0, dur=1.0):
    a, b = norm(web), norm(asr)
    if not a or not b: return False, 1.0
    w = jiwer.wer(a, b)                                       # (S + D + I) / N
    wps = len(b.split()) / dur                              # words per second
    return (w <= max_wer and min_wps <= wps <= max_wps), w
A web caption and an ASR output for the same chunk differ by 40% WER. What should the pipeline do?

Chapter 6

Pin Each Word to the Clock

Use CTC forced alignment to turn a transcript into word timestamps, one trellis cell at a time

A subtitle file says, “the cat sat”, on screen from second 4 to second 7. You want more: the exact second each word starts and ends. With word times you can cut a long chunk at a word edge, check that a caption really matches the audio, and train models that need timing. Getting those times from audio plus known text is called forced alignment. “Forced” means we do not ask what was said. We already know the words, and we force them onto the sound.

What a speech model outputs: a posterior grid

An acoustic model reads the audio in small frames and, for each frame, gives a probability for every token it knows. A token can be a letter, a sound or a word. We call this grid of numbers the posteriors. For a toy with three words and 8 frames, the grid has 4 rows (the three words plus one special token) and 8 columns, and every column adds up to 1.

The special token is the blank, written “−”. It means “nothing new starts in this frame”. A model trained with CTC (connectionist temporal classification) uses it. CTC has one reading rule: collapse repeated tokens, then delete the blanks. So the frame labels −, the, the, −, cat, cat, cat, −, sat, − read as “the cat sat”. A repeat of one word in a row means “still saying it”, and a blank between two copies of the same word means “said it twice”.

The trellis: a map of every allowed path

We know the words, so we know which paths are legal. Write the words with a blank before, between and after: −, the, −, cat, −, sat, −. That is 7 states. A path walks through them one frame at a time. At each step it may stay in the same state, move to the next state, or skip over a blank to the next word. It may never go back. The grid of states against frames, with those moves, is the trellis.

Many paths fit. We want the single best one: the one with the highest total probability. Multiplying many small probabilities gives a number too tiny for a computer to hold, so we take the log of each one and add the logs instead. A Viterbi search fills the trellis frame by frame. For each state, it records the best total score of any path that ends there, and which previous state it came from. At the last frame, pick the better of the two final states, and follow the arrows back.

Worked step. Frame 0, state “−”: probability 0.70, log 0.70 = −0.36. State “the”: 0.20, log = −1.61. States further right cannot be reached yet. Frame 1, state “the”: probability 0.70, log = −0.36. It can come from “−” (−0.36) or stay in “the” (−1.61); the better is −0.36, so the score is −0.36 + −0.36 = −0.72. Fill the rest the same way. The device does it for you, one frame at a time.

Fill the trellis

Rows are the seven states, columns are frames. Each number is the best total log-probability reaching that cell. Drag to fill more frames. At the last frame the best path lights up.

8

Illustrative: the posteriors are made up, and each frame is 0.1 s so three words fit in one picture. Real acoustic models use much shorter frames and letters or sounds as tokens. The dynamic programming is exact: scores are natural-log sums and the path is traced back from the better final state.

Flip to the blurry posteriors. The model is less sure, the scores are lower, and the best path may place word edges one frame earlier or later. The path is still legal; it is just less trustworthy. That total score is what a real aligner uses as a confidence number.

From the path to word times

Read the path left to right. The first frame in a word’s state is its start. The last frame in that state is its end. Multiply the frame number by the frame length to get seconds. In the real systems the tokens are characters or phonemes, and the aligner groups them into words afterward.

Two real examples use this. CTC-Segmentation (arXiv 2007.09127) takes a CTC-trained recognizer and uses it to cut unaligned audio into training utterances. Its abstract reports a German dataset of over 1700 hours built from public corpora, including unlabeled audio, and a model trained on it with a 12.8% WER on the Tuda-DE test set, against a 14.4% hybrid DNN/HMM baseline. WhisperX (arXiv 2303.00747) notes that Whisper gives no word-level timestamps out of the box, and adds forced alignment of phonemes to get them.

pythonimport torch, torchaudio.functional as F

# logp: shape (1, T, V) log-probabilities from a CTC model; tokens: ids of the known words/characters
targets = torch.tensor([tokens], dtype=torch.int32)                # shape (1, L)
align, scores = F.forced_align(logp, targets, blank=0)           # one token id per frame
spans = F.merge_tokens(align[0], scores[0].exp())                  # one span per token
FRAME = 0.02                                                    # seconds per frame: read it off your model
words = [(s.token, s.start * FRAME, s.end * FRAME, s.score) for s in spans]

Failure: misaligned timestamps

Symptom. Chunks start in the middle of a word, or the text lags the voice by a second near the end of long files. Measurement. For 200 chunks, listen at the aligned word edges, or compare each aligned start with the caption’s own start. A drift that grows with time points at a caption that was cut from a different version of the audio. Also keep the alignment score for each chunk. Fix. Align in short windows of a minute or less, so an error cannot pile up. Drop chunks whose alignment score is low, and re-check the lowest-scoring ones by ear.

In a CTC trellis, what does the Viterbi search record at each cell?

Chapter 7

Tag and Filter With Text: CLAP

Use one model that knows sounds and words to tag clips, check captions and spot copies

A teammate drops 40,000 clips from a field-recording site on the drive. Some are music. Some are applause. Some are traffic hum. A few are people talking. Nobody wrote a tag for any of them. You want the speech for the pipeline you just built, and you want the rest gone.

The speech detector from Chapter 3 cannot do this. It answers one question: is there a voice? It cannot say “applause” or “a cello”. You need a model that can match a sound to a sentence you type.

Two encoders that share one space

An encoder is a network that turns an input into a list of numbers. We call that list an embedding. CLAP (Contrastive Language-Audio Pretraining; Elizalde et al., arXiv 2206.04769) has two encoders. The audio encoder reads a clip. The text encoder reads a sentence. Each writes a list of 1,024 numbers.

Training used 128,010 pairs of a clip and its caption. In each batch, the model pulls every clip toward its own caption and pushes it away from the other captions. This is contrastive learning. CLAP is the audio cousin of CLIP, which does the same for photos and captions. After training, a clip of clapping and the words “people clapping” sit close together.

“Close” has a number. It is the cosine similarity: scale each list to length 1, then take the dot product. The result runs from −1 to 1, and higher means more alike. Chapter 4 used the same arithmetic to compare voiceprints.

Here is the data flow for one clip and one prompt, with the sizes the paper reports.

# one clip, one prompt: shapes from Elizalde et al. (2022)
waveform   (220500,)        # 5 s at 44,100 Hz; longer clips are cropped, shorter ones padded
log-mel    (~690, 64)       # hop 320, window 1024, 64 mel bins from 50 to 8,000 Hz
CNN14      (2048,)          # the audio encoder, 80.8 million parameters
audio_emb  (1024,)          # one learned projection matrix

"This is a sound of applause"
BERT [CLS] (768,)           # the text encoder, 110 million parameters
text_emb   (1024,)          # a second learned projection matrix

score = cosine(audio_emb, text_emb)   # one number per clip and prompt

A log-mel spectrogram is a picture of the clip: time across, pitch bands up, loudness as brightness. Chapter 1 gave you the array it starts from. Only the two projection matrices put sound and words into the same 1,024-number space, and that shared space is why a typed sentence can score a clip.

A later model, LAION-CLAP (arXiv 2211.06687), trains on 633,526 pairs called LAION-Audio-630K. Its abstract adds feature fusion, so clips of different lengths fit, and keyword-to-caption augmentation. The data and the model are public.

Job 1: tag clips with words you type

Zero-shot means the model never saw your label list in training. You write one prompt per tag and embed the prompts once. You embed each clip once. Then you take the cosine of every clip with every prompt. The best prompt is the tag, if its score clears a minimum. The paper used the template “This is a sound of [label]”.

Take one clip. Its scores on four prompts are 0.12 for music, 0.31 for applause, 0.08 for traffic noise and 0.15 for speech. With a minimum of 0.20, the tag is applause. If no score reached 0.20, the clip would get no tag and go to an “other” pile.

Now run it over all 40,000 clips. Suppose the tags come out as 12,400 speech, 9,200 music, 3,100 applause and 15,300 noise (illustrative). Route the 12,400 speech clips, 31%, to Chapter 3 and drop the rest. The paper reports 100% zero-shot accuracy on the GTZAN music-versus-speech task, which is the split this job needs.

Job 2: score audio and caption pairs

Web audio often comes with a caption, such as a title or a description. Many captions do not describe the sound. For photos, CLIP-score filtering keeps a pair only when photo and caption sit close. Do the same here. The CLAP score of a pair is the cosine of the clip embedding and the caption embedding. Keep pairs at or above a threshold.

A pair that scores 0.34 passes a threshold of 0.25. A mismatched pair that scores 0.11 does not. The cost shows up in recall, the share of the good pairs you keep. In the toy below (300 invented pairs), a threshold of 0.15 keeps 97% of the good pairs, 0.25 keeps 77%, and 0.30 keeps 60%. Precision rises the other way, from 87% to 99% to 100%.

Job 3: find near-identical clips

One viral clip shows up in many uploads, re-encoded, trimmed, each with a different caption. The text hashes from Chapter 9 miss these copies. Embeddings catch them. Say two clips have unit embeddings (0.8, 0.6) and (0.6, 0.8). Their dot product is 0.48 + 0.48 = 0.96. A threshold of 0.95 calls them copies and drops the second. A threshold of 0.98 keeps both. Real embeddings have 1,024 numbers; the sum is longer and the idea is the same.

Comparing every pair of a million clips takes about 5 × 1011 comparisons. At that size use an approximate nearest-neighbour index, and compare each clip only with its neighbours.

Tag a clip, then filter pairs

Tag mode: pick a synthesized sound, tap Hear it, and slide the minimum score. Filter mode: slide the threshold over 300 caption pairs and watch what you keep and what you lose.

0.20

Illustrative. Each sound is synthesized in your browser: a tone pattern stands in for music, noise bursts for applause, a vowel-like buzz for speech, a low rumble for traffic. The toy embedding is three numbers measured from the waveform (pitch steadiness, burstiness, noisiness). The four prompt vectors are set by hand, and the scores are cosines times 0.4. Real CLAP uses 1,024 learned numbers. The 300 pairs are invented, 240 common and 60 rare, the same on every visit.

pythonimport numpy as np
def unit(x): return x / np.linalg.norm(x, axis=-1, keepdims=True)

# audio_emb: (N, 1024) from the audio tower. text_emb: (K, 1024) from the text tower.
def tag(audio_emb, text_emb, labels, min_score=0.20):
    s = unit(audio_emb) @ unit(text_emb).T               # (N, K) cosine scores
    best, top = s.argmax(1), s.max(1)
    return [labels[i] if t >= min_score else "other" for i, t in zip(best, top)]

def clap_score(audio_emb, caption_emb):             # one score per audio-caption pair
    return (unit(audio_emb) * unit(caption_emb)).sum(-1)  # (N,)

def dedup(audio_emb, sim=0.95):                     # keep the first of each near-copy group
    e, keep = unit(audio_emb), []
    for i in range(len(e)):
        if not keep or (e[keep] @ e[i]).max() < sim: keep.append(i)
    return keep

Failure: the score looks sure and is wrong

Wording swings the score. Symptom. The same clip gets a different tag after you reword the prompt. Measurement. Hand-label 200 clips, tag them with three wordings, and compare accuracy. Fix. Pick the wording that wins on the checked sample. The paper reports that changing the prompt lifted ESC50 accuracy by about 5%. Try the Bare word switch in the device: the traffic hum falls below the minimum.

Speech content and long clips. Symptom. CLAP tags speech but cannot say what was said. The paper reports Emotion Recognition and Keyword Spotting only up to 4 points above random. Use an ASR model for words (Chapter 5). CLAP also trained on 5-second crops. Fix. Embed a long episode in 5-second windows and take the best score per tag, or use a model built for variable lengths, as LAION-CLAP does with feature fusion.

One threshold, rare sounds. Symptom. After filtering, a rare sound class nearly vanishes. Rare sounds tend to have fewer training pairs, so their right captions score lower, and a global cut removes them first. Measurement. Report recall for each class on a hand-checked sample. Fix. Set the threshold per class. Switch the device to Filter mode and compare the two recall curves.

Paper walkthrough: CLAP: Learning Audio Concepts from Natural Language Supervision.

A clip scores 0.31 on “applause”, 0.15 on “speech”, 0.12 on “music” and 0.08 on “traffic noise”. The minimum is 0.20. What happens to it in a speech pipeline?

Chapter 8

Score the Quality, Drop the Rest

Measure noise, quality, language, music and overlap, then set thresholds you can defend

A chunk can pass every test so far and still be a poor example. The words are right and the times are right, but a fridge hums through the whole minute, or the line was a bad phone call, or the speaker switched to Spanish for a joke. This chapter adds a column of scores to each chunk, and then sets thresholds on them.

Measure one: how loud is the voice compared with the noise?

The signal-to-noise ratio (SNR) compares the power of the voice with the power of everything else. It is written in decibels, the same log scale as Chapter 2.

SNR
Signal-to-noise ratio in decibels. Higher is cleaner.
Psignal
The average squared sample while someone speaks.
Pnoise
The average squared sample in the pauses, where only the room is heard.

Psignal = 0.04, Pnoise = 0.0004: ratio 100, so SNR = 10 · log10(100) = 20 dB

In a real recording you never get a clean copy of the voice alone. A cheap estimate takes the quietest 10% of frames as noise and the loudest 50% as signal. It is crude, and it is good enough to rank clips.

Measure two: how good does it sound to a person?

SNR does not capture everything. Muffled speech can have a fine SNR and still sound bad. What we want is the score a listener would give. DNSMOS (arXiv 2010.15258) is built for that. Its abstract explains that classic quality metrics need a clean reference signal, which real recordings do not have, which makes them hard to use on field recordings. DNSMOS is a no-reference model, trained in several stages with self-teaching, to predict human perceptual ratings, and its abstract says it correlates highly with them. It was designed to evaluate noise suppressors; data teams use the same idea as a cheap quality score for each chunk. The scale in our toy runs from 1 (bad) to 5 (excellent), like a listener rating. That scale is the usual convention for such scores, and the toy numbers are illustrative.

Measures three to five: language, music and overlap

Language ID is a classifier that names the language and says how sure it is. Keep a chunk only when the language is the one you want and the confidence is high. The MMS paper (arXiv 2305.13516) shows how far this goes: its language identification model covers 4,017 languages. Music detection flags chunks where music or singing is on top of the voice. Overlap detection flags frames where two people speak at once, so the single transcript cannot be right. Each of these is a small model that gives a score or a yes-no flag per chunk.

Setting thresholds you can defend

Every threshold throws away hours, so each one needs a reason. The rule of thumb: run the cheap scores on everything first, hand-check a sample of a few hundred chunks near each cut, and move the cut until the checked sample is clean enough for your use. Then write the thresholds into the dataset card. Try the trade below: the hidden label error of every clip is known to the toy, so you can see what each cut buys.

Set the thresholds

Each dot is a 5-hour recording, placed by SNR and quality score. Green dots pass. Hollow dots are dropped. Move the sliders and switch on the flags.

10 dB
2.5

Tap to hear a synthetic voice clean, then with white noise added at the Minimum SNR on the slider.

Illustrative: 200 made-up recordings of 5 hours each, seeded. SNR, a DNSMOS-style quality score from 1 to 5, a music flag, a language flag and an overlap share are all invented, as is each recording’s label error. The point is the trade, not the numbers.

pythondef passes(c, snr_min=10, q_min=2.5, lang="en", lang_p=0.9, ov_max=0.10):
    # cheapest scores first, so most chunks leave before the expensive models run
    if c["snr_db"] < snr_min:                       return False, "noisy"
    if c["lang"] != lang or c["lang_p"] < lang_p:      return False, "language"
    if c["music_p"] > 0.5:                         return False, "music"
    if c["overlap"] > ov_max:                       return False, "overlap"
    if c["quality"] < q_min:                       return False, "quality"
    return True, "ok"

Failure: a threshold from the wrong domain

Symptom. After filtering, one accent, one language or one device nearly vanishes from the data, and the model does worse on it. Measurement. Report the share of hours kept for every slice: language, source, microphone type. A slice that keeps far less than the rest was judged by a score that fits it badly. Fix. Set thresholds per slice, from hand-checked samples of that slice, and log the reject reason on every dropped chunk so you can audit later.

Why does the DNSMOS paper argue for a no-reference quality metric?

Chapter 9

Dedup, Decontaminate, Split by Speaker

Remove copies, keep the eval set clean, and stop test speakers from leaking into training

Your model scores 4% word error rate on the test set. A great result. Then a colleague finds that the test episodes were also in the training crawl, and half the test speakers appear in training too. The real number is nowhere near 4%. This chapter closes three doors through which test data leaks into training.

Door one: copies inside the training set

Podcasts repeat. The same ad read runs in 5,000 episodes. A show is uploaded twice, once as MP3 and once as a re-encoded video. If you keep every copy, the model sees that one sentence thousands of times and learns to recite it. Deduplication removes the extra copies.

Two levels work together. At the text level, normalise each transcript (lower-case, no punctuation) and hash it. Equal hashes mean exact copies. For near copies, compare the sets of short word runs. At the audio level, an audio fingerprint keeps the strongest peaks of the sound’s spectrum and hashes pairs of them. A re-encoded or slightly noisy copy keeps most of its peaks, so its fingerprint still matches. Hashing methods for near-duplicates are taught in the data filtering and dedup lesson.

Door two: eval decontamination

The eval set is the set of examples you hold back to measure the model. Decontamination removes anything from the training data that overlaps with it. For speech there are two checks. The transcript check compares word runs. An n-gram is a run of n words in a row. If a training transcript shares an n-gram with an eval transcript, flag it. The audio check compares fingerprints, in case the transcripts differ but the sound is the same.

Choosing n is a trade. Tiny n flags innocent lines that share a common phrase such as “we trained the model on”. Huge n misses a paraphrase that carries the same content. Try it.

Find the leaks

The eval transcript is shown below. Each row is a training transcript; its bar counts shared n-grams. Change n and read the right-hand verdict.

5

Illustrative: one made-up eval sentence and six made-up training lines. Rows 1, 2 and 4 are real leaks (an exact copy, a near copy and a paraphrase of the same content); rows 3, 5 and 6 are innocent. The shared n-grams are counted live after lower-casing.

Notice what the device cannot show. A line that is a faithful paraphrase in different words, and a recording that was re-read by another speaker, share no run at all. Text overlap is a floor, not a guarantee. That is why the audio fingerprint and the speaker split sit beside it.

Door three: the same voice on both sides

Even with every transcript clean, a leak remains: the speaker. If a person talks in both training and test, the model has learned that voice, its accent and its microphone. The test no longer asks, “can you understand a new person?” It asks, “can you understand someone you already know?” The fix is to split by speaker (or at least by episode or show), never by clip. Chapter 4’s speaker labels exist partly for this.

Split the speakers

Each row is a speaker; each square is a clip. Blue squares train. The others are eval: green if the speaker never appears in training, red if the speaker does.

Illustrative: 8 made-up speakers with 5 clips each. The error figures are invented to show the direction of the effect (seen speakers are easier); the count of leaking clips is computed from the split you chose.

pythondef ngrams(text, n=8):
    w = text.lower().split()
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}

eval_grams = set().union(*(ngrams(t) for t in eval_texts))
clean = [c for c in train if not (ngrams(c["text"]) & eval_grams)]    # text overlap
clean = [c for c in clean if c["fp"] not in eval_fingerprints]           # audio overlap

eval_spk = {c["speaker"] for c in eval_set}                      # and keep speakers apart
clean = [c for c in clean if c["speaker"] not in eval_spk]

Failure: leaking eval speakers into training

Symptom. The test score is much better than scores on a fresh set of new voices, and gets worse when you re-split. Measurement. Count test clips whose speaker label, or whose episode, also appears in training. Any number above zero is a leak. Fix. Split on speaker (and on show), compute the split once, store it as a column, and never re-draw it. Run the n-gram and fingerprint checks against every new eval set before each training run. Chapter 4 warned that labels are per episode; to split by speaker across episodes you need a step that links the same person.

The test and training transcripts share no n-gram, yet the test score looks too good. What else could leak?

Chapter 10

Real Datasets and the Scale Math

Read how Emilia, YODAS, MMS and others do it, then price the storage and compute

You can now build every step by hand. The next question is the practical one: how big is the job, and how do the teams that publish datasets describe theirs? This chapter reads five public datasets, shows how to document your own, and prices storage and compute before you commit.

Five public datasets, from their abstracts

Every number below comes from the paper’s abstract. Where the abstract is silent, the table says so.

Dataset or systemScaleWhat the abstract says
Whisper (2212.04356)680,000 hoursMultilingual and multitask supervision, with weak supervision in the title
The People’s Speech (2111.09344)30,000 hours and growingSupervised English speech for commercial use; a model on it reaches 9.98% WER on Librispeech test-clean
Emilia (2407.05361)over 101k hours, six languagesFor speech generation; Emilia-Pipe is the first open-source preprocessing pipeline of its kind
YODAS (2406.00899)over 500k hours, over 100 languagesUnder a Creative Commons license; recognition baselines for the top-15 languages
MMS (2305.13516)1,107 languages (one ASR model), 4,017 (language ID)Raises language coverage 10 to 40 times; pre-trained wav2vec 2.0 models for 1,406 languages; more than halves Whisper’s word error rate on 54 FLEURS languages
CTC-Segmentation (2007.09127)over 1700 hours of GermanAligned from unlabeled audio with a CTC model; 12.8% WER on Tuda-DE

Two things stand out. First, the sizes span more than an order of magnitude, from tens of thousands of hours to hundreds of thousands. Second, the pipeline matters as much as the audio: Emilia’s authors released Emilia-Pipe, their open-source preprocessing pipeline, so others can repeat the processing. Read the papers for the steps; the abstracts give only the scale. Our Whisper lesson and weak supervision paper walkthrough take the training side of this story further.

What a dataset card should say

A dataset card is a short document that travels with the data. A reader should be able to rebuild your choices from it. Write at least these things: the sources and their licenses; hours at each stage (raw, after segmentation, after filtering); languages with counts; every threshold from Chapters 3 to 8 and the tool that set it; the labelling method (human, weak, pseudo) with a measured error rate on a hand-checked sample; the dedup and decontamination rules; and the known failure modes.

yamlname: podcast-en-v1
sources: [{name: public-podcast-feeds, license: "per-feed, see sources.csv"}]
hours: {raw: 1000, after_vad: 790, after_filters: 612, train: 580, dev: 16, test: 16}
audio: {sr: 16000, channels: 1, dtype: int16, loudness_db_rms: -20}
segments: {max_len_s: 15, vad_thr: 0.5, join_gap_s: 0.5, pad_s: 0.15}
labels: {method: pseudo+web, agree_wer_max: 0.10, human_checked: 300, measured_wer: 0.043}
filters: {snr_db_min: 10, quality_min: 2.5, lang: en, lang_p_min: 0.9, overlap_max: 0.10}
splits: {by: speaker, ngram_decontam: 8, audio_fp_decontam: true}
known_issues: ["names and numbers mis-transcribed", "few children's voices"]

All figures in this card are toy numbers that follow the lesson’s running example.

Price it before you start

Storage and compute follow from the arithmetic of Chapter 1. Storage is hours × megabytes per hour. At 16,000 Hz, mono, int16, one hour is 115.2 MB. So 101,000 hours (Emilia’s size) is 101,000 × 115.2 MB = 11.6 TB, and 680,000 hours (Whisper’s) is 78.3 TB. Your raw, compressed downloads add more, and you may keep several copies.

Compute follows from one number: GPU-hours per audio hour. If a recognizer runs 100 times faster than real time on one GPU, each audio hour costs 0.01 GPU-hours. A 101,000-hour set then needs 1,010 GPU-hours for one transcription pass, about 4.2 days on 10 GPUs. Every extra model in the pipeline (VAD, speaker, quality, language) adds its own pass. Batching helps: the WhisperX abstract reports a twelve-fold speedup from batched inference after pre-segmenting audio. Measure your own number on your own hardware.

Price a dataset

Pick a dataset size and an audio format. The bars show storage on a log scale. Then set how many GPU-hours one hour of audio costs, and how many GPUs you have.

0.010
10

Hours are the dataset sizes from the abstracts (YODAS and Emilia are lower bounds, “over”). Storage is exact arithmetic: hours times MB per hour. The GPU-hours per audio hour is yours to set; the default of 0.010 is an invented example, not a measurement of any system.

Failure: skipping the cost line

Symptom. The job runs for a week, then the disk fills, or the second pass would take a month. Measurement. Run the whole pipeline on 10 hours, time each stage, and multiply. Keep a table of seconds of compute per hour of audio for every step. Fix. Put cheap filters first, store intermediate results as shards of FLAC or int16, and write the per-stage hours into the dataset card.

One hour of 16 kHz mono int16 audio is 115.2 MB. About how much is 101,000 hours of it?

Chapter 11

Field Guide

Keep the numbers, the checklist, the core code and the sources on one page

You can now follow a thousand hours of raw podcast audio from a shared drive to a table of clean training pairs. This page keeps the whole pipeline in one place: the numbers, a checklist, the code that ties the steps together, where each failure lives, and the sources.

The one-paragraph summary

A speech data pipeline turns raw audio into records of sound, text, time and scores. It decodes and resamples once, sets one loudness and one channel layout, finds speech with a VAD and cuts chunks at pauses, labels speakers, gets text from the web and from a strong ASR model, keeps chunks where the two agree, uses forced alignment to pin each word to the clock, scores quality (SNR, a DNSMOS-style score, language, music, overlap), removes copies, decontaminates the eval set, and splits by speaker. Each step writes a column, and every threshold goes in the dataset card.

The numbers that matter

WhatNumberWhy it matters
One hour, 16 kHz mono int16115.2 MB (1,000 h = 115.2 GB)The unit for every storage plan
One hour, 44.1 kHz stereo float321,270,080,000 bytesThe format alone can change the bill ten times
Nyquist limit at 16,000 Hz8,000 Hz; a 6,000 Hz tone at 8,000 Hz reads as 2,000 HzLow-pass before you resample down
Loudness exampleRMS 0.158 = −16.0 dBFS; gain to −20 is −4.0 dB (×0.631)One gain per clip, then check for clipping
Mono downmix, phase offset 180°amplitude |cos(φ/2)| = 0Check the mono RMS against the channels
Word error rate example1 substitution in 7 words = 14.3%The agreement metric
SNR examplepower ratio 100 = 20 dBA cheap noise score
Whisper (2212.04356)680,000 hoursWeak supervision at scale
The People’s Speech (2111.09344)30,000 hours; 9.98% WER on Librispeech test-cleanA commercial-use English set
Emilia (2407.05361)over 101k hours, six languagesOpen pipeline for speech generation data
YODAS (2406.00899)over 500k hours, over 100 languagesCreative Commons audio at scale
MMS (2305.13516)1,107 languages ASR; 4,017 language ID; 54 FLEURS languages at under half Whisper’s WERCoverage beyond about one hundred languages
CTC-Segmentation (2007.09127)over 1700 h German; 12.8% vs 14.4% WERAlignment turns unlabeled audio into data
CLAP (2206.04769)128,010 training pairs; 1,024 number embeddings; 100% zero-shot on GTZAN music versus speechText prompts can tag and filter audio
LAION-CLAP (2211.06687)633,526 audio-text pairs, publicVariable-length clips and open data
WhisperX (2303.00747)twelve-fold speedupPre-segmenting enables batching
101,000 hours at 115.2 MB per hour11.6 TBPrice storage first

The checklist

  1. Decode once. One sample rate, one dtype, the original rate stored in the record.
  2. Set the level. One RMS target; reject clipped clips and phase-cancelled stereo first.
  3. Cut at pauses. VAD threshold, join gap, maximum length, and 0.1 to 0.2 s of padding.
  4. Label speakers per episode. Flag overlap; do not trust labels across episodes.
  5. Get text two ways, keep agreement. Normalise first; check words per second.
  6. Align and keep the score. Short windows; drop low scores; spot-check by ear.
  7. Tag with text when the source is mixed. One prompt per tag, a minimum score, a threshold for each class; check by ear.
  8. Score quality and log reasons. SNR, a quality model, language ID, music, overlap; audit by slice.
  9. Dedup, decontaminate, split by speaker. Text hashes, n-grams and audio fingerprints against every eval set.
  10. Write the card. Hours at every stage, thresholds, measured label error, known issues.

The core as code

Every step above is a function that adds columns to a record. This sketch strings them together for one episode. The helpers (vad_probs, embed, asr, web_text, align, scores) stand for whichever models you choose.

pythondef process_episode(path, eval_grams, eval_speakers):
    x = load_mono_16k(path)                                   # Ch 1: shape (N,), float32
    x, why = normalize_checked(x)                              # Ch 2: level, clipping, phase
    if x is None: return []
    chunks = segment(vad_probs(x), max_len=15.0)               # Ch 3: [(start_s, end_s), ...]
    spk = diarize(chunks, embed, thr=0.5)                      # Ch 4: speaker per chunk
    rows = []
    for (s, e), sp in zip(chunks, spk):
        clip = x[int(s * 16000):int(e * 16000)]
        text, w = asr(clip), web_text(path, s, e)             # Ch 5: two sources
        ok, wer = keep(w, text, dur=e - s)
        if not ok: continue
        words, a_score = align(clip, text)                       # Ch 6: word times + score
        tag = clap_tag(clip)                                    # Ch 7: text-prompt tag, optional
        q = scores(clip)                                       # Ch 8: snr, quality, lang, music, overlap
        ok, why = passes(q)
        if not ok or a_score < -1.0: continue
        if ngrams(text) & eval_grams or sp["speaker"] in eval_speakers: continue   # Ch 9
        rows.append({"text": text, "words": words, "speaker": sp["speaker"], "agree_wer": wer, **q})
    return rows

Where each failure lives

Pick a failure. The strip shows the stage where it starts. The panel gives the symptom, the measurement and the fix, and tells you which chapter has the detail.

Trace a failure to its stage

The failures, measurements and fixes are the ones taught in Chapters 1 to 9. The hand-check sizes (for example 200 chunks) are working suggestions, not standards.

Where this sits in the field, and keep going

Audio data work sits between signal processing and language-model data work. The signal side is taught in audio representations. The recognizer that supplies most pseudo-labels is explained in Whisper and the Whisper paper walkthrough. Models that learn from unlabeled audio are in self-supervised speech. For the output side, see streaming speech and voice turn-taking, which need the speaker and timing columns from this pipeline. Text-style dedup and filtering are in data filtering and dedup, and how data size affects speech language models is in speech language model scaling.

The takeaway. A speech model learns whatever the pairs teach. A pipeline decides what the pairs are. Every cut, label and threshold is a choice about what the model will believe, so measure each one on a small hand-checked sample, write it down, and keep the eval set far from the training set.

Sources

Facts and numbers come from the abstract pages of these papers. Everything labelled illustrative is a toy.

  1. Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv 2212.04356
  2. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. arXiv 2303.00747
  3. CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition. arXiv 2007.09127
  4. Scaling Speech Technology to 1,000+ Languages (MMS). arXiv 2305.13516
  5. Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation. arXiv 2407.05361
  6. YODAS: Youtube-Oriented Dataset for Audio and Speech. arXiv 2406.00899
  7. DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors. arXiv 2010.15258
  8. Elizalde et al. CLAP: Learning Audio Concepts from Natural Language Supervision. arXiv 2206.04769
  9. Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (LAION-CLAP). arXiv 2211.06687
  10. The People’s Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage. arXiv 2111.09344
Your test score looks great, but a fresh set of new speakers scores much worse. Which stage should you inspect first?

Now press Present or Teach and walk through one episode from memory: what the array looks like, where the cuts go, how two sources of text agree, how the trellis finds word times, and which three scores decide whether a chunk stays. Then go back to the thousand hours at the top, press Strict, and explain what you gain and what you pay.

Audio Data Pipelines
Emilia · YODAS · Back to Gleams