A thousand hours of raw podcasts are not training data yet. A pipeline cuts, labels and filters them. Turn up the strictness: fewer hours survive, and their labels get cleaner.
Press Light, then Standard, then Strict. Each green square is a 4-hour recording that passes. Watch the hours fall and the label error fall with them. Then we build the pipeline piece by piece: decoding, loudness, speech detection, speakers, transcripts, alignment, text tagging, quality scores, and leaks.
You need a rough idea of what a sound wave is and a little Python. We build the rest from zero.
Built from ten primary sources, among them Whisper, WhisperX, CTC-Segmentation, Emilia and YODAS. The sources are listed in Chapter 11.
Illustrative: 250 made-up recordings of 4 hours each, generated the same way on every visit. Each has a noise level, a music flag, an overlap share and a label error. Light drops music and very noisy audio. Standard also drops overlap above 15% and clips where two ASR systems disagree by over 20%. Strict tightens all three.
Chapter 0
See why a pile of podcast audio is not yet training data, and what a pipeline must do to it
Picture a shared drive with a thousand hours of podcast episodes on it. A teammate says, “Train a speech model on this by Friday.” You press play on episode one. It opens with 90 seconds of theme music. Two hosts talk over each other for a minute. A truck drives past the window. Then a long, quiet pause, and an ad read in a different voice.
Every one of those moments will teach the model something. Most of it is the wrong thing. This chapter shows what is wrong with raw audio, and what a pipeline does about it.
A speech model learns from pairs: a short clip of sound, and the words spoken in it. We call the words the label, the way a photo of a cat carries the label “cat”. Raw episodes give you neither part cleanly. The sound has parts that are not speech. The words are missing, or they come as one long show-notes page, or they sit in a caption file that drifts out of step with the voice.
A data pipeline is a chain of small programs. Each one takes the output of the last one and fixes one problem. The word comes from plumbing: the audio goes in at one end, and clean pairs come out of the other. Think of a water treatment plant. No single step makes water safe. Each step removes one kind of dirt.
Big speech systems are built from huge piles of audio. Whisper trained on 680,000 hours of multilingual and multitask supervision (Radford et al., arXiv 2212.04356). The YODAS dataset holds over 500k hours in more than 100 languages (arXiv 2406.00899). Emilia starts with over 101k hours of speech across six languages (arXiv 2407.05361). The People’s Speech is a 30,000-hour and growing English set (arXiv 2111.09344). Nobody listens to that much audio. A pipeline decides, and a pipeline can be wrong in quiet ways.
The point is not to keep everything. The point is to keep the hours where the sound and the words match, and to know how many hours that leaves. The hero at the top of this page showed the trade: a stricter filter keeps fewer hours and cleaner labels. Now we look at one hour of raw audio to see what the filter is cutting.
Below is one illustrative hour of a podcast, drawn as a 60-minute bar. Four kinds of trouble sit in it. Music is sound that is not speech, such as theme tunes. Silence is a stretch with no voice. Overlap is two people talking at once, so no single transcript fits. Noise is sound that hides the voice, such as a truck. Tap each one to cut it, and watch the usable minutes.
The warm bar is one 60-minute episode. Press a button to cut that kind of audio. Cut regions turn hollow.
Illustrative hour: 6 minutes of music (intro and outro), 5 of silence, 4 of overlap, 7 of noise, 38 of clean speech. The minutes are made up to show the idea; real shares differ by show.
Even in this friendly hour, 22 minutes are not clean speech. If every hour looked like this one, a thousand hours would give about 633 usable hours. The other 367 are not lost by accident. A good pipeline finds them on purpose, and it can tell you how many it cut and why.
Here is the whole chain on one line, so you can see where each chapter fits. First, decode every file into numbers and set one sample rate. Second, set one loudness and one channel layout. Third, find where speech lives and cut it into chunks. Fourth, say who speaks. Fifth, get words. Sixth, pin each word to the clock. Seventh, tag clips with text. Eighth, score the quality. Ninth, remove copies and keep the test set clean. Then check the scale, and learn how real teams fail.
Each step adds a column to a record. By the end, one training example looks like this, in the JSON-lines style that many teams use:
json{"id": "pod042_000317", "audio": "shards/pod042/000317.flac", "sr": 16000, "dur": 11.84,
"speaker": "SPK_01", "text": "so the first thing we tried was a smaller model",
"words": [["so", 0.12, 0.31], ["the", 0.31, 0.44], ["first", 0.44, 0.83]],
"snr_db": 24.1, "quality": 3.7, "lang": "en", "agree_wer": 0.04, "split": "train"}
Every field in that record comes from one chapter. When a model behaves oddly, you read the record, find the column that lied, and fix the step that wrote it.
Chapter 1
Turn a file into an array, learn its shape, and size a thousand hours before you buy the disk
You double-click an episode. It is an MP3 file of 58 MB, and a media player turns it into sound. A neural network cannot use the file as it is. It needs a plain list of numbers. This chapter turns a file into that list, shows its shape, and prices a thousand hours of it.
A microphone has a thin sheet that moves when air pressure changes. The computer reads how far the sheet has moved, again and again, many times each second. Each reading is one sample: a single number. The number of readings each second is the sample rate, measured in hertz (Hz). A rate of 16,000 Hz means 16,000 numbers for every second of sound.
So one mono clip is a one-dimensional array. For a clip of 11.84 seconds at 16,000 Hz, the shape is (189,440,), because 11.84 × 16,000 = 189,440. A stereo clip has two channels, left and right, and its shape is (2, N) in most Python libraries. The type of each number matters too. int16 stores whole numbers from −32,768 to 32,767 in 2 bytes. float32 stores decimals, usually between −1.0 and 1.0, in 4 bytes. Models want float32; disks are happier with int16.
MP3, AAC and Opus files are compressed: a program threw away detail that people rarely hear, so the file is small. Decoding is the reverse step. It rebuilds the full list of samples. Most pipelines call ffmpeg, a free tool that reads almost any format, and ask it to write raw samples at the rate and layout they want. That is the one place where you set the sample rate for the whole pipeline.
pythonimport subprocess, numpy as np def load_mono_16k(path): # ffmpeg decodes, mixes to 1 channel (-ac 1), resamples to 16 kHz (-ar 16000) cmd = ["ffmpeg", "-v", "error", "-i", path, "-ac", "1", "-ar", "16000", "-f", "s16le", "-"] raw = subprocess.run(cmd, capture_output=True, check=True).stdout x = np.frombuffer(raw, dtype=np.int16) # shape (N,), dtype int16 return x.astype(np.float32) / 32768.0 # shape (N,), values in [-1, 1) x = load_mono_16k("pod042.mp3") print(x.shape, x.dtype, len(x) / 16000) # (57600000,) float32 3600.0
Before you buy disks, do the arithmetic once, by hand. One hour has 3,600 seconds. At 16,000 Hz that is 16,000 × 3,600 = 57,600,000 samples. In int16, each sample takes 2 bytes, so one hour is 115,200,000 bytes, or 115.2 MB. A thousand hours is 115.2 GB.
Now change two things. Use float32 and the same hour takes 230.4 MB, so a thousand hours take 230.4 GB. Keep 44,100 Hz stereo float32 (a common studio format) and one hour is 44,100 × 3,600 × 2 channels × 4 bytes = 1,270,080,000 bytes. A thousand hours then fill 1,270 GB. The format you pick can change the bill by more than ten times.
A pure tone repeats its wave many times per second. To see one wave, you need at least two samples inside it: one near the top, one near the bottom. This is the Nyquist limit: a sample rate of fs can only describe tones below fs/2. At 16,000 Hz that limit is 8,000 Hz.
A tone above the limit does not vanish. It folds back and shows up as a lower tone that was never in the room. This is aliasing. The formula for the fake tone is short.
f = 6,000, fs = 8,000: k = round(0.75) = 1, so falias = |6,000 − 8,000| = 2,000 Hz
Try it. Slide the tone up and pick a sample rate. The thin line is the true tone; the dots are the samples; the thick line is what the dots seem to say.
Choose a sample rate, then drag the tone upward. When it passes half the rate, the thick line stops following the thin one.
Tap to hear a sweep from 300 to 10,000 Hz at the rate you chose. One version keeps every k-th sample. The other low-passes first.
The geometry is exact: the dots are samples of a sine of the chosen tone over 1 millisecond, and the thick line is the sine of the alias frequency from the formula above. Storage uses one thousand hours, mono, int16. Real resamplers low-pass the audio first so no tone above the new limit survives to fold.
Symptom. A model scores well on studio recordings and poorly on phone calls, or you hear a faint whistle in the training clips. Measurement. Plot the spectrum (energy by frequency) of a few dozen clips. Clips that began at 8,000 Hz and were later raised to 16,000 Hz have almost no energy between 4,000 and 8,000 Hz. They carry a sample rate of 16,000 Hz, but only the detail of 8,000 Hz. Clips that were lowered badly show hills of fake energy near the top. Fix. Resample once, with a proper low-pass filter (the soxr or sinc resamplers do this), and store the original rate in the record so you can filter the “fake 16 kHz” clips later.
Chapter 2
Set one loudness and one channel layout so the model never learns the microphone
Two episodes sit side by side in your folder. One host whispers into a cheap microphone. Another host shouts into a studio mic and the signal hits the ceiling. A model trained on both would learn one more thing than you wanted: how loud the recording was. This chapter removes that accident, and fixes a second one hiding in stereo files.
Samples in a float array run from −1.0 to 1.0. A loud clip has large swings; a quiet clip has small ones. To measure the typical swing, square every sample, average the squares, and take the square root. This is the root mean square, or RMS. Squaring matters: it makes negative swings count the same as positive ones.
People hear loudness on a log scale, so we write it in decibels relative to full scale (dBFS): 20 × log10(RMS). Full scale is 1.0, which is 0 dBFS. Everything quieter is negative. Every 6 dB halves or doubles the swing, roughly.
Take four samples: 0.1, −0.2, 0.2, −0.1. The squares are 0.01, 0.04, 0.04, 0.01. Their mean is 0.025, and the square root is 0.158. That is 20 × log10(0.158) = −16.0 dBFS.
Loudness normalization picks a target level and multiplies the whole clip by one number, the gain, to reach it. In decibels the gain is simple subtraction: target minus measured. If the target is −20 dBFS and the clip sits at −16.0, the gain is −4.0 dB, which is a multiplier of 10(−4.0/20) = 0.631. A whisper clip at −38 dBFS needs +18 dB, a multiplier of 7.94.
Many teams use LUFS, a loudness unit weighted to how ears hear, instead of plain RMS. The recipe is the same: measure, subtract, apply one gain. Plain RMS is enough to understand the idea, and it is what the device below uses.
Gain has a risk. Speech has short, sharp peaks, much higher than its average. When you push the whole clip up, the peaks can pass 1.0. The format cannot store them, so they are cut flat. That is clipping, and it adds harsh crackle the speaker never made. Set the clip’s starting level, pick a target, and see how many samples clip.
The top wave is the clip as recorded. The bottom wave is after one gain. Dashed lines are the ceiling at plus and minus 1.0. Samples past the ceiling turn red.
Tap to hear the clip as recorded, then after the gain you set. Push the target to -6 and hear the crackle of clipping.
Illustrative: 800 synthetic samples shaped like syllables, the same on every visit. RMS, gain and clip counts are computed from those samples in your browser. Real speech peaks differ by voice and microphone.
Symptom. The model produces crackly output, or it fails on loud speakers only. Measurement. Count the samples whose absolute value is 0.999 or higher. In a healthy clip the count is near zero. In a clipped clip it can be thousands. Fix. Reject clips above a small share (for example 0.1%) before you normalize. Gain cannot undo flat tops; it only makes them louder.
Most speech models take one channel. A stereo file has two, left and right, and the easy fix is to average them: mono = (left + right) / 2. Usually this works, because both channels hear the same voice.
Sometimes it does not. If a recording chain flips the sign of one channel, the two waves are exact opposites. Averaging them gives zero: silence. This is phase cancellation. Partial offsets give partial loss. For two copies of the same tone shifted by an angle φ, the average has amplitude |cos(φ/2)|. At 0 degrees it is 1. At 90 degrees it is 0.71. At 180 degrees it is 0.
Left and right carry the same tone. Turn the phase offset and look at the mono mix, the average of the two.
Exact: both channels are unit sine waves, and the mono mix is their average, computed point by point. The check in the code below, comparing the RMS of the mono mix with the louder channel, catches this fault in real files.
pythondef to_mono_checked(stereo, target_db=-20.0): # stereo: shape (2, N), float32 mono = stereo.mean(axis=0) # shape (N,) rms = lambda a: np.sqrt(np.mean(a ** 2)) + 1e-9 if rms(mono) < 0.5 * max(rms(stereo[0]), rms(stereo[1])): return None, "phase_cancel" # drop or repair: do not train on it if (np.abs(mono) >= 0.999).mean() > 0.001: return None, "clipped" gain_db = target_db - 20 * np.log10(rms(mono)) out = mono * 10 ** (gain_db / 20) return np.clip(out, -1, 1), "ok"
Chapter 3
Detect where people talk, then cut the hours into chunks of a length a model can train on
You now hold a two-hour episode as one array of 115,200,000 numbers at 16,000 Hz. You cannot train on that as one example. A model needs examples of a few seconds each, and each one must start and end in sensible places. This chapter finds where speech lives, then cuts the array into pieces.
A fixed cut is easy and wrong. It slices words in half, so the left piece ends with “fi” and the right piece starts with “rst”. Neither half matches any transcript. It also keeps silence and music, which we wanted to drop. We need to look at the sound first.
The tool for that is a voice activity detector, or VAD. It is a small model that listens to short frames, often 10 to 30 milliseconds long, and for each frame outputs the probability that someone is speaking. Think of a light switch controlled by a very fast, very tired listener. The output for a whole episode is a list of numbers between 0 and 1, one for each frame.
Probabilities are not segments yet. Three rules turn them into start and end times. Each rule has one knob.
Here is a worked case with a frame every 0.5 seconds. The probabilities are 0.1, 0.2, 0.9, 0.95, 0.8, 0.3, 0.85, 0.9, 0.1, 0.05. With threshold 0.5, frames 2, 3, 4, 6 and 7 are speech. Frame 5 is a half-second gap. With minimum silence set to 0.5 seconds, the gap is joined, and we get one segment from frame 2 to frame 7: 3.0 seconds. With minimum silence set to 0.25, the gap stays and we get two segments, of 1.5 and 1.0 seconds. The same audio gives different examples depending on one knob.
The warm curve is the VAD output. The pink line is your threshold. Blue bars are the chunks you get. The thin green strip is where people really spoke.
Tap to hear only the kept chunks, squeezed in time, with a click at every cut. A hiss among the voices is a false alarm.
Illustrative: a made-up 90-second recording with a known speech mask, a door slam at 55 seconds and short pauses between turns. The VAD curve is the mask plus seeded noise, with 0.25-second frames. Recall and junk are counted against the mask.
Look for the trade. A high threshold gives clean chunks but loses the soft ends of words. A low threshold keeps every word but also keeps the door slam. A long minimum silence gives long, natural chunks; a short one gives many fragments. There is no setting that is right for every show. You pick one by measuring on a small hand-checked sample, which Chapter 8 returns to.
One more design choice matters: the length cap. Short chunks are cheap to batch and to align, and an error in one cannot spread far. Very short chunks lose context, so a model hears less of a sentence. A common compromise is a cap in the range of ten to thirty seconds. Treat the exact number as a setting to measure, not a rule.
WhisperX (arXiv 2303.00747) puts a VAD step in front of the recognizer. Its abstract describes speech-region detection and a “VAD Cut & Merge” pre-segmentation, which splits long audio into pieces that are then transcribed in batches. The abstract reports a twelve-fold transcription speedup from batching. The lesson for a data builder: cutting is not only for tidy examples. It also lets you run many chunks through a model at once.
pythondef segment(p, hop=0.25, thr=0.5, join=0.5, max_len=15.0, min_len=0.5): # p: shape (T,), speech probability per frame, from a VAD model on = p > thr runs, i = [], 0 while i < len(on): if on[i]: j = i while j + 1 < len(on) and on[j + 1]: j += 1 runs.append([i, j + 1]); i = j + 1 else: i += 1 merged = [] for a, b in runs: # join runs separated by a short gap if merged and (a - merged[-1][1]) * hop <= join: merged[-1][1] = b else: merged.append([a, b]) out, cap = [], int(max_len / hop) for a, b in merged: while b - a > cap: # too long: cut at the quietest frame lo = a + int(0.6 * cap) k = lo + int(np.argmin(p[lo:a + cap])) out.append((a * hop, k * hop)); a = k if (b - a) * hop >= min_len: out.append((a * hop, b * hop)) return out # list of (start_s, end_s)
Symptom. Transcripts miss the first word of many chunks, or the model drops initial consonants. Measurement. Add 0.2 seconds of padding on each side for a sample of 200 chunks, re-run the recognizer, and compare. If the word error rate drops, your cuts were too tight. Fix. Pad each chunk by 0.1 to 0.2 seconds, use a lower threshold with a short minimum-silence join, and never let the padding overlap a neighbour.
Chapter 4
Label each stretch of speech with a speaker, and see why conversational models need it
Two hosts chat for an hour. The transcript says, “So the first thing we tried was a smaller model.” Who said it? For a model that writes one long monologue, nobody cares. For a model that answers in a conversation, everything depends on it. This chapter adds the missing column: who spoke, and when.
A conversational speech model must learn turn-taking: when to talk, when to wait, when to say “mm-hm”. It can only learn that from data where each turn is tied to a speaker. A model that copies a voice needs the opposite guarantee: a clip must hold one voice, not two. Both needs come from the same step, called speaker diarization. The name means “who spoke when”.
Diarization does not know anyone’s name. It gives anonymous labels: SPK_00, SPK_01. Think of a seating chart made by listening. You do not learn who sits where, only that the same person talks from seat 1 each time.
First, take the speech regions from the VAD and cut them into short windows, about 1 to 2 seconds. Second, run each window through a speaker embedding model. It turns a window into a short list of numbers, a vector, that works like a voiceprint. Windows of the same voice land near each other in space. Windows of different voices land far apart. Real embeddings have a few hundred numbers; we draw two.
Third, cluster the vectors: group the ones that are close. Fourth, give each group a label and write each window’s label back onto the timeline. Neighbouring windows with the same label join into one turn.
How do we say two voiceprints are close? A common measure is the cosine similarity: the angle between the two vectors. Take (3, 4) and (4, 3). The dot product is 3×4 + 4×3 = 24. Each has length 5, so the similarity is 24 / (5 × 5) = 0.96, nearly the same direction. Compare (3, 4) with (−4, 3): the dot product is −12 + 12 = 0, so the similarity is 0, perpendicular. The first pair sounds like one voice; the second like two.
Here is what the output looks like for the first ten seconds of our two-host episode. Each line is one turn, written as start time, end time and label. The labels are anonymous, and the times come from the VAD cuts of Chapter 3.
json[{"start": 3.20, "end": 9.85, "speaker": "SPK_00"},
{"start": 9.85, "end": 11.10, "speaker": "SPK_01"},
{"start": 11.40, "end": 17.62, "speaker": "SPK_00"}]
A conversational model reads this as a script: speaker 0 talks for 6.65 seconds, speaker 1 answers for 1.25 seconds, speaker 0 resumes after a gap of 0.3 seconds. That gap is a signal for turn-taking. If the labels are wrong, the gaps are wrong too.
The clustering rule has one important knob: how close is close enough to join? Set it too tight and one speaker splits into two labels, because her voice drifts when she laughs. Set it too loose and two speakers merge into one. The hardest windows are overlaps, where two people talk at once. Their voiceprint lands between the two speakers and acts as a bridge. Link the bridge and the two clusters become one.
Each dot is a 1.5-second window drawn at its voiceprint. Colour shows the speaker label your threshold gives it. Grey dots are windows left unlabelled. Drag the threshold, and try adding the overlap windows.
Illustrative: 60 made-up windows from 3 speakers (20 each, seeded), plus 4 overlap windows between speakers A and B when switched on. Clustering is single-link: two windows join when the gap between them is below your threshold. Clusters of fewer than 3 windows stay unlabelled. Real embeddings have hundreds of dimensions.
pythonfrom sklearn.cluster import AgglomerativeClustering def diarize(windows, embed, thr=0.5): # windows: list of (start_s, end_s) inside speech regions; embed: audio -> vector (D,) E = np.stack([embed(w) for w in windows]) # shape (n_windows, D) E /= np.linalg.norm(E, axis=1, keepdims=True) lab = AgglomerativeClustering(n_clusters=None, distance_threshold=thr, metric="cosine", linkage="average").fit_predict(E) return [{"start": s, "end": e, "speaker": f"SPK_{l:02d}"} for (s, e), l in zip(windows, lab)]
Symptom. A voice model trained on your clips produces a blended, unstable voice, or a dialogue model answers as the wrong person. Measurement. For each clip, embed its windows and check the spread. If they form two groups, the clip has two speakers. Also count how many clips get a speaker label seen in fewer than three windows. Fix. Cluster inside each episode, not across the whole corpus, unless you have a separate step that links the same person across episodes. Mark overlap regions and drop or flag them. Keep the tightest threshold that does not split a speaker, found by checking a small hand-labelled set.
Chapter 5
Get text for every chunk from the web and from a strong ASR model, then keep only what agrees
You have clean chunks of speech, and a speaker label for each. You still have no words. Where do ten thousand hours of transcripts come from? Paying people to type them would cost a fortune. So pipelines use two cheaper sources, and a trick to check them.
Human labels are typed by people and checked. They are the best and the rarest. Weak labels are text that already exists near the audio: video captions, subtitles, show notes. They cost nothing, but nobody checked them, and they can be late, summarised or wrong. The Whisper paper (arXiv 2212.04356) puts the idea in its title, “Large-Scale Weak Supervision”, and reports training on 680,000 hours of multilingual and multitask supervision.
Pseudo-labels come from a strong speech recognizer, an ASR model (automatic speech recognition), that you run yourself on your chunks. The word “pseudo” means “pretend”: the labels are a good guess, not a fact. The recognizer is a student who has read a lot. It is often right and sometimes confidently wrong.
To compare two transcripts we need a number. Word error rate (WER) counts the fewest edits that turn one into the other. An edit is a substitution (one word swapped), a deletion (a word missing) or an insertion (a word added). Divide the edits by the number of words in the reference.
WER = (S + D + I) / N
Worked example. The reference is “we trained it on four thousand hours”: 7 words. The recognizer writes “we trained it on forty thousand hours”. One word differs, so S = 1, D = 0, I = 0. WER = 1 / 7 = 14.3%. The error is small in words and large in meaning: four thousand became forty thousand.
Suppose you have two independent guesses for the same chunk: the web caption and your recognizer’s output. If they match, it is unlikely both are wrong in the same way. If they differ, at least one is wrong, and you cannot tell which one. The cheap, safe move is to compare them with WER and keep the chunk only when the WER is below a threshold. This is agreement filtering.
It is like asking two friends the way to the station. If both point left, go left. If one says left and one says right, do not guess. Ask neither, and skip this chunk.
Each bar is the word error rate between a chunk’s web caption and the recognizer’s text. Move the threshold. Pick a chunk below to read both texts.
Illustrative: eight short made-up chunks. The WER is computed live by a word-level edit distance. The recognizer text is the label we would train on; whether it is truly right was decided by the author of the toy. Chunk 8 is a harmless spelling difference that the filter drops anyway.
Move the threshold up to 20% and a chunk with “forty thousand” slips through. Move it down to 5% and you lose every chunk where the two sources differ in harmless ways. This is the usual trade: a stricter filter keeps cleaner labels and fewer hours. To lose fewer good chunks, normalise both texts first: lower-case, no punctuation, numbers and spellings written one way.
A recognizer that writes fluent text will write fluent text even when there is nothing to transcribe. Over music or a long silence it may produce a plausible sentence, or repeat one phrase again and again. Symptom. Transcripts of non-speech chunks contain polite closing phrases, or the same sentence appears thousands of times. Measurement. Count the most common whole transcripts; a real corpus has few exact repeats. Check words-per-second: a 10-second chunk with 60 words, or with 0.3 words per second, is suspect. Agreement helps here, as chunk 5 shows: the web caption says “[music]” and the recognizer writes a sentence. Fix. Filter by agreement, by speech rate, and by the VAD and music scores of Chapter 8. Agreement cannot catch an error both sources share, so keep a small human-checked set to measure it.
pythonimport jiwer, re def norm(s): s = re.sub(r"\[.*?\]", "", s.lower()) # drop [music] style tags return re.sub(r"[^a-z0-9' ]", "", s).strip() def keep(web, asr, max_wer=0.10, min_wps=0.8, max_wps=5.0, dur=1.0): a, b = norm(web), norm(asr) if not a or not b: return False, 1.0 w = jiwer.wer(a, b) # (S + D + I) / N wps = len(b.split()) / dur # words per second return (w <= max_wer and min_wps <= wps <= max_wps), w
Chapter 6
Use CTC forced alignment to turn a transcript into word timestamps, one trellis cell at a time
A subtitle file says, “the cat sat”, on screen from second 4 to second 7. You want more: the exact second each word starts and ends. With word times you can cut a long chunk at a word edge, check that a caption really matches the audio, and train models that need timing. Getting those times from audio plus known text is called forced alignment. “Forced” means we do not ask what was said. We already know the words, and we force them onto the sound.
An acoustic model reads the audio in small frames and, for each frame, gives a probability for every token it knows. A token can be a letter, a sound or a word. We call this grid of numbers the posteriors. For a toy with three words and 8 frames, the grid has 4 rows (the three words plus one special token) and 8 columns, and every column adds up to 1.
The special token is the blank, written “−”. It means “nothing new starts in this frame”. A model trained with CTC (connectionist temporal classification) uses it. CTC has one reading rule: collapse repeated tokens, then delete the blanks. So the frame labels −, the, the, −, cat, cat, cat, −, sat, − read as “the cat sat”. A repeat of one word in a row means “still saying it”, and a blank between two copies of the same word means “said it twice”.
We know the words, so we know which paths are legal. Write the words with a blank before, between and after: −, the, −, cat, −, sat, −. That is 7 states. A path walks through them one frame at a time. At each step it may stay in the same state, move to the next state, or skip over a blank to the next word. It may never go back. The grid of states against frames, with those moves, is the trellis.
Many paths fit. We want the single best one: the one with the highest total probability. Multiplying many small probabilities gives a number too tiny for a computer to hold, so we take the log of each one and add the logs instead. A Viterbi search fills the trellis frame by frame. For each state, it records the best total score of any path that ends there, and which previous state it came from. At the last frame, pick the better of the two final states, and follow the arrows back.
Worked step. Frame 0, state “−”: probability 0.70, log 0.70 = −0.36. State “the”: 0.20, log = −1.61. States further right cannot be reached yet. Frame 1, state “the”: probability 0.70, log = −0.36. It can come from “−” (−0.36) or stay in “the” (−1.61); the better is −0.36, so the score is −0.36 + −0.36 = −0.72. Fill the rest the same way. The device does it for you, one frame at a time.
Rows are the seven states, columns are frames. Each number is the best total log-probability reaching that cell. Drag to fill more frames. At the last frame the best path lights up.
Illustrative: the posteriors are made up, and each frame is 0.1 s so three words fit in one picture. Real acoustic models use much shorter frames and letters or sounds as tokens. The dynamic programming is exact: scores are natural-log sums and the path is traced back from the better final state.
Flip to the blurry posteriors. The model is less sure, the scores are lower, and the best path may place word edges one frame earlier or later. The path is still legal; it is just less trustworthy. That total score is what a real aligner uses as a confidence number.
Read the path left to right. The first frame in a word’s state is its start. The last frame in that state is its end. Multiply the frame number by the frame length to get seconds. In the real systems the tokens are characters or phonemes, and the aligner groups them into words afterward.
Two real examples use this. CTC-Segmentation (arXiv 2007.09127) takes a CTC-trained recognizer and uses it to cut unaligned audio into training utterances. Its abstract reports a German dataset of over 1700 hours built from public corpora, including unlabeled audio, and a model trained on it with a 12.8% WER on the Tuda-DE test set, against a 14.4% hybrid DNN/HMM baseline. WhisperX (arXiv 2303.00747) notes that Whisper gives no word-level timestamps out of the box, and adds forced alignment of phonemes to get them.
pythonimport torch, torchaudio.functional as F # logp: shape (1, T, V) log-probabilities from a CTC model; tokens: ids of the known words/characters targets = torch.tensor([tokens], dtype=torch.int32) # shape (1, L) align, scores = F.forced_align(logp, targets, blank=0) # one token id per frame spans = F.merge_tokens(align[0], scores[0].exp()) # one span per token FRAME = 0.02 # seconds per frame: read it off your model words = [(s.token, s.start * FRAME, s.end * FRAME, s.score) for s in spans]
Symptom. Chunks start in the middle of a word, or the text lags the voice by a second near the end of long files. Measurement. For 200 chunks, listen at the aligned word edges, or compare each aligned start with the caption’s own start. A drift that grows with time points at a caption that was cut from a different version of the audio. Also keep the alignment score for each chunk. Fix. Align in short windows of a minute or less, so an error cannot pile up. Drop chunks whose alignment score is low, and re-check the lowest-scoring ones by ear.
Chapter 7
Use one model that knows sounds and words to tag clips, check captions and spot copies
A teammate drops 40,000 clips from a field-recording site on the drive. Some are music. Some are applause. Some are traffic hum. A few are people talking. Nobody wrote a tag for any of them. You want the speech for the pipeline you just built, and you want the rest gone.
The speech detector from Chapter 3 cannot do this. It answers one question: is there a voice? It cannot say “applause” or “a cello”. You need a model that can match a sound to a sentence you type.
An encoder is a network that turns an input into a list of numbers. We call that list an embedding. CLAP (Contrastive Language-Audio Pretraining; Elizalde et al., arXiv 2206.04769) has two encoders. The audio encoder reads a clip. The text encoder reads a sentence. Each writes a list of 1,024 numbers.
Training used 128,010 pairs of a clip and its caption. In each batch, the model pulls every clip toward its own caption and pushes it away from the other captions. This is contrastive learning. CLAP is the audio cousin of CLIP, which does the same for photos and captions. After training, a clip of clapping and the words “people clapping” sit close together.
“Close” has a number. It is the cosine similarity: scale each list to length 1, then take the dot product. The result runs from −1 to 1, and higher means more alike. Chapter 4 used the same arithmetic to compare voiceprints.
Here is the data flow for one clip and one prompt, with the sizes the paper reports.
# one clip, one prompt: shapes from Elizalde et al. (2022) waveform (220500,) # 5 s at 44,100 Hz; longer clips are cropped, shorter ones padded log-mel (~690, 64) # hop 320, window 1024, 64 mel bins from 50 to 8,000 Hz CNN14 (2048,) # the audio encoder, 80.8 million parameters audio_emb (1024,) # one learned projection matrix "This is a sound of applause" BERT [CLS] (768,) # the text encoder, 110 million parameters text_emb (1024,) # a second learned projection matrix score = cosine(audio_emb, text_emb) # one number per clip and prompt
A log-mel spectrogram is a picture of the clip: time across, pitch bands up, loudness as brightness. Chapter 1 gave you the array it starts from. Only the two projection matrices put sound and words into the same 1,024-number space, and that shared space is why a typed sentence can score a clip.
A later model, LAION-CLAP (arXiv 2211.06687), trains on 633,526 pairs called LAION-Audio-630K. Its abstract adds feature fusion, so clips of different lengths fit, and keyword-to-caption augmentation. The data and the model are public.
Zero-shot means the model never saw your label list in training. You write one prompt per tag and embed the prompts once. You embed each clip once. Then you take the cosine of every clip with every prompt. The best prompt is the tag, if its score clears a minimum. The paper used the template “This is a sound of [label]”.
Take one clip. Its scores on four prompts are 0.12 for music, 0.31 for applause, 0.08 for traffic noise and 0.15 for speech. With a minimum of 0.20, the tag is applause. If no score reached 0.20, the clip would get no tag and go to an “other” pile.
Now run it over all 40,000 clips. Suppose the tags come out as 12,400 speech, 9,200 music, 3,100 applause and 15,300 noise (illustrative). Route the 12,400 speech clips, 31%, to Chapter 3 and drop the rest. The paper reports 100% zero-shot accuracy on the GTZAN music-versus-speech task, which is the split this job needs.
Web audio often comes with a caption, such as a title or a description. Many captions do not describe the sound. For photos, CLIP-score filtering keeps a pair only when photo and caption sit close. Do the same here. The CLAP score of a pair is the cosine of the clip embedding and the caption embedding. Keep pairs at or above a threshold.
A pair that scores 0.34 passes a threshold of 0.25. A mismatched pair that scores 0.11 does not. The cost shows up in recall, the share of the good pairs you keep. In the toy below (300 invented pairs), a threshold of 0.15 keeps 97% of the good pairs, 0.25 keeps 77%, and 0.30 keeps 60%. Precision rises the other way, from 87% to 99% to 100%.
One viral clip shows up in many uploads, re-encoded, trimmed, each with a different caption. The text hashes from Chapter 9 miss these copies. Embeddings catch them. Say two clips have unit embeddings (0.8, 0.6) and (0.6, 0.8). Their dot product is 0.48 + 0.48 = 0.96. A threshold of 0.95 calls them copies and drops the second. A threshold of 0.98 keeps both. Real embeddings have 1,024 numbers; the sum is longer and the idea is the same.
Comparing every pair of a million clips takes about 5 × 1011 comparisons. At that size use an approximate nearest-neighbour index, and compare each clip only with its neighbours.
Tag mode: pick a synthesized sound, tap Hear it, and slide the minimum score. Filter mode: slide the threshold over 300 caption pairs and watch what you keep and what you lose.
Illustrative. Each sound is synthesized in your browser: a tone pattern stands in for music, noise bursts for applause, a vowel-like buzz for speech, a low rumble for traffic. The toy embedding is three numbers measured from the waveform (pitch steadiness, burstiness, noisiness). The four prompt vectors are set by hand, and the scores are cosines times 0.4. Real CLAP uses 1,024 learned numbers. The 300 pairs are invented, 240 common and 60 rare, the same on every visit.
pythonimport numpy as np def unit(x): return x / np.linalg.norm(x, axis=-1, keepdims=True) # audio_emb: (N, 1024) from the audio tower. text_emb: (K, 1024) from the text tower. def tag(audio_emb, text_emb, labels, min_score=0.20): s = unit(audio_emb) @ unit(text_emb).T # (N, K) cosine scores best, top = s.argmax(1), s.max(1) return [labels[i] if t >= min_score else "other" for i, t in zip(best, top)] def clap_score(audio_emb, caption_emb): # one score per audio-caption pair return (unit(audio_emb) * unit(caption_emb)).sum(-1) # (N,) def dedup(audio_emb, sim=0.95): # keep the first of each near-copy group e, keep = unit(audio_emb), [] for i in range(len(e)): if not keep or (e[keep] @ e[i]).max() < sim: keep.append(i) return keep
Wording swings the score. Symptom. The same clip gets a different tag after you reword the prompt. Measurement. Hand-label 200 clips, tag them with three wordings, and compare accuracy. Fix. Pick the wording that wins on the checked sample. The paper reports that changing the prompt lifted ESC50 accuracy by about 5%. Try the Bare word switch in the device: the traffic hum falls below the minimum.
Speech content and long clips. Symptom. CLAP tags speech but cannot say what was said. The paper reports Emotion Recognition and Keyword Spotting only up to 4 points above random. Use an ASR model for words (Chapter 5). CLAP also trained on 5-second crops. Fix. Embed a long episode in 5-second windows and take the best score per tag, or use a model built for variable lengths, as LAION-CLAP does with feature fusion.
One threshold, rare sounds. Symptom. After filtering, a rare sound class nearly vanishes. Rare sounds tend to have fewer training pairs, so their right captions score lower, and a global cut removes them first. Measurement. Report recall for each class on a hand-checked sample. Fix. Set the threshold per class. Switch the device to Filter mode and compare the two recall curves.
Paper walkthrough: CLAP: Learning Audio Concepts from Natural Language Supervision.
Chapter 8
Measure noise, quality, language, music and overlap, then set thresholds you can defend
A chunk can pass every test so far and still be a poor example. The words are right and the times are right, but a fridge hums through the whole minute, or the line was a bad phone call, or the speaker switched to Spanish for a joke. This chapter adds a column of scores to each chunk, and then sets thresholds on them.
The signal-to-noise ratio (SNR) compares the power of the voice with the power of everything else. It is written in decibels, the same log scale as Chapter 2.
Psignal = 0.04, Pnoise = 0.0004: ratio 100, so SNR = 10 · log10(100) = 20 dB
In a real recording you never get a clean copy of the voice alone. A cheap estimate takes the quietest 10% of frames as noise and the loudest 50% as signal. It is crude, and it is good enough to rank clips.
SNR does not capture everything. Muffled speech can have a fine SNR and still sound bad. What we want is the score a listener would give. DNSMOS (arXiv 2010.15258) is built for that. Its abstract explains that classic quality metrics need a clean reference signal, which real recordings do not have, which makes them hard to use on field recordings. DNSMOS is a no-reference model, trained in several stages with self-teaching, to predict human perceptual ratings, and its abstract says it correlates highly with them. It was designed to evaluate noise suppressors; data teams use the same idea as a cheap quality score for each chunk. The scale in our toy runs from 1 (bad) to 5 (excellent), like a listener rating. That scale is the usual convention for such scores, and the toy numbers are illustrative.
Language ID is a classifier that names the language and says how sure it is. Keep a chunk only when the language is the one you want and the confidence is high. The MMS paper (arXiv 2305.13516) shows how far this goes: its language identification model covers 4,017 languages. Music detection flags chunks where music or singing is on top of the voice. Overlap detection flags frames where two people speak at once, so the single transcript cannot be right. Each of these is a small model that gives a score or a yes-no flag per chunk.
Every threshold throws away hours, so each one needs a reason. The rule of thumb: run the cheap scores on everything first, hand-check a sample of a few hundred chunks near each cut, and move the cut until the checked sample is clean enough for your use. Then write the thresholds into the dataset card. Try the trade below: the hidden label error of every clip is known to the toy, so you can see what each cut buys.
Each dot is a 5-hour recording, placed by SNR and quality score. Green dots pass. Hollow dots are dropped. Move the sliders and switch on the flags.
Tap to hear a synthetic voice clean, then with white noise added at the Minimum SNR on the slider.
Illustrative: 200 made-up recordings of 5 hours each, seeded. SNR, a DNSMOS-style quality score from 1 to 5, a music flag, a language flag and an overlap share are all invented, as is each recording’s label error. The point is the trade, not the numbers.
pythondef passes(c, snr_min=10, q_min=2.5, lang="en", lang_p=0.9, ov_max=0.10): # cheapest scores first, so most chunks leave before the expensive models run if c["snr_db"] < snr_min: return False, "noisy" if c["lang"] != lang or c["lang_p"] < lang_p: return False, "language" if c["music_p"] > 0.5: return False, "music" if c["overlap"] > ov_max: return False, "overlap" if c["quality"] < q_min: return False, "quality" return True, "ok"
Symptom. After filtering, one accent, one language or one device nearly vanishes from the data, and the model does worse on it. Measurement. Report the share of hours kept for every slice: language, source, microphone type. A slice that keeps far less than the rest was judged by a score that fits it badly. Fix. Set thresholds per slice, from hand-checked samples of that slice, and log the reject reason on every dropped chunk so you can audit later.
Chapter 9
Remove copies, keep the eval set clean, and stop test speakers from leaking into training
Your model scores 4% word error rate on the test set. A great result. Then a colleague finds that the test episodes were also in the training crawl, and half the test speakers appear in training too. The real number is nowhere near 4%. This chapter closes three doors through which test data leaks into training.
Podcasts repeat. The same ad read runs in 5,000 episodes. A show is uploaded twice, once as MP3 and once as a re-encoded video. If you keep every copy, the model sees that one sentence thousands of times and learns to recite it. Deduplication removes the extra copies.
Two levels work together. At the text level, normalise each transcript (lower-case, no punctuation) and hash it. Equal hashes mean exact copies. For near copies, compare the sets of short word runs. At the audio level, an audio fingerprint keeps the strongest peaks of the sound’s spectrum and hashes pairs of them. A re-encoded or slightly noisy copy keeps most of its peaks, so its fingerprint still matches. Hashing methods for near-duplicates are taught in the data filtering and dedup lesson.
The eval set is the set of examples you hold back to measure the model. Decontamination removes anything from the training data that overlaps with it. For speech there are two checks. The transcript check compares word runs. An n-gram is a run of n words in a row. If a training transcript shares an n-gram with an eval transcript, flag it. The audio check compares fingerprints, in case the transcripts differ but the sound is the same.
Choosing n is a trade. Tiny n flags innocent lines that share a common phrase such as “we trained the model on”. Huge n misses a paraphrase that carries the same content. Try it.
The eval transcript is shown below. Each row is a training transcript; its bar counts shared n-grams. Change n and read the right-hand verdict.
Illustrative: one made-up eval sentence and six made-up training lines. Rows 1, 2 and 4 are real leaks (an exact copy, a near copy and a paraphrase of the same content); rows 3, 5 and 6 are innocent. The shared n-grams are counted live after lower-casing.
Notice what the device cannot show. A line that is a faithful paraphrase in different words, and a recording that was re-read by another speaker, share no run at all. Text overlap is a floor, not a guarantee. That is why the audio fingerprint and the speaker split sit beside it.
Even with every transcript clean, a leak remains: the speaker. If a person talks in both training and test, the model has learned that voice, its accent and its microphone. The test no longer asks, “can you understand a new person?” It asks, “can you understand someone you already know?” The fix is to split by speaker (or at least by episode or show), never by clip. Chapter 4’s speaker labels exist partly for this.
Each row is a speaker; each square is a clip. Blue squares train. The others are eval: green if the speaker never appears in training, red if the speaker does.
Illustrative: 8 made-up speakers with 5 clips each. The error figures are invented to show the direction of the effect (seen speakers are easier); the count of leaking clips is computed from the split you chose.
pythondef ngrams(text, n=8): w = text.lower().split() return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)} eval_grams = set().union(*(ngrams(t) for t in eval_texts)) clean = [c for c in train if not (ngrams(c["text"]) & eval_grams)] # text overlap clean = [c for c in clean if c["fp"] not in eval_fingerprints] # audio overlap eval_spk = {c["speaker"] for c in eval_set} # and keep speakers apart clean = [c for c in clean if c["speaker"] not in eval_spk]
Symptom. The test score is much better than scores on a fresh set of new voices, and gets worse when you re-split. Measurement. Count test clips whose speaker label, or whose episode, also appears in training. Any number above zero is a leak. Fix. Split on speaker (and on show), compute the split once, store it as a column, and never re-draw it. Run the n-gram and fingerprint checks against every new eval set before each training run. Chapter 4 warned that labels are per episode; to split by speaker across episodes you need a step that links the same person.
Chapter 10
Read how Emilia, YODAS, MMS and others do it, then price the storage and compute
You can now build every step by hand. The next question is the practical one: how big is the job, and how do the teams that publish datasets describe theirs? This chapter reads five public datasets, shows how to document your own, and prices storage and compute before you commit.
Every number below comes from the paper’s abstract. Where the abstract is silent, the table says so.
| Dataset or system | Scale | What the abstract says |
|---|---|---|
| Whisper (2212.04356) | 680,000 hours | Multilingual and multitask supervision, with weak supervision in the title |
| The People’s Speech (2111.09344) | 30,000 hours and growing | Supervised English speech for commercial use; a model on it reaches 9.98% WER on Librispeech test-clean |
| Emilia (2407.05361) | over 101k hours, six languages | For speech generation; Emilia-Pipe is the first open-source preprocessing pipeline of its kind |
| YODAS (2406.00899) | over 500k hours, over 100 languages | Under a Creative Commons license; recognition baselines for the top-15 languages |
| MMS (2305.13516) | 1,107 languages (one ASR model), 4,017 (language ID) | Raises language coverage 10 to 40 times; pre-trained wav2vec 2.0 models for 1,406 languages; more than halves Whisper’s word error rate on 54 FLEURS languages |
| CTC-Segmentation (2007.09127) | over 1700 hours of German | Aligned from unlabeled audio with a CTC model; 12.8% WER on Tuda-DE |
Two things stand out. First, the sizes span more than an order of magnitude, from tens of thousands of hours to hundreds of thousands. Second, the pipeline matters as much as the audio: Emilia’s authors released Emilia-Pipe, their open-source preprocessing pipeline, so others can repeat the processing. Read the papers for the steps; the abstracts give only the scale. Our Whisper lesson and weak supervision paper walkthrough take the training side of this story further.
A dataset card is a short document that travels with the data. A reader should be able to rebuild your choices from it. Write at least these things: the sources and their licenses; hours at each stage (raw, after segmentation, after filtering); languages with counts; every threshold from Chapters 3 to 8 and the tool that set it; the labelling method (human, weak, pseudo) with a measured error rate on a hand-checked sample; the dedup and decontamination rules; and the known failure modes.
yamlname: podcast-en-v1
sources: [{name: public-podcast-feeds, license: "per-feed, see sources.csv"}]
hours: {raw: 1000, after_vad: 790, after_filters: 612, train: 580, dev: 16, test: 16}
audio: {sr: 16000, channels: 1, dtype: int16, loudness_db_rms: -20}
segments: {max_len_s: 15, vad_thr: 0.5, join_gap_s: 0.5, pad_s: 0.15}
labels: {method: pseudo+web, agree_wer_max: 0.10, human_checked: 300, measured_wer: 0.043}
filters: {snr_db_min: 10, quality_min: 2.5, lang: en, lang_p_min: 0.9, overlap_max: 0.10}
splits: {by: speaker, ngram_decontam: 8, audio_fp_decontam: true}
known_issues: ["names and numbers mis-transcribed", "few children's voices"]
All figures in this card are toy numbers that follow the lesson’s running example.
Storage and compute follow from the arithmetic of Chapter 1. Storage is hours × megabytes per hour. At 16,000 Hz, mono, int16, one hour is 115.2 MB. So 101,000 hours (Emilia’s size) is 101,000 × 115.2 MB = 11.6 TB, and 680,000 hours (Whisper’s) is 78.3 TB. Your raw, compressed downloads add more, and you may keep several copies.
Compute follows from one number: GPU-hours per audio hour. If a recognizer runs 100 times faster than real time on one GPU, each audio hour costs 0.01 GPU-hours. A 101,000-hour set then needs 1,010 GPU-hours for one transcription pass, about 4.2 days on 10 GPUs. Every extra model in the pipeline (VAD, speaker, quality, language) adds its own pass. Batching helps: the WhisperX abstract reports a twelve-fold speedup from batched inference after pre-segmenting audio. Measure your own number on your own hardware.
Pick a dataset size and an audio format. The bars show storage on a log scale. Then set how many GPU-hours one hour of audio costs, and how many GPUs you have.
Hours are the dataset sizes from the abstracts (YODAS and Emilia are lower bounds, “over”). Storage is exact arithmetic: hours times MB per hour. The GPU-hours per audio hour is yours to set; the default of 0.010 is an invented example, not a measurement of any system.
Symptom. The job runs for a week, then the disk fills, or the second pass would take a month. Measurement. Run the whole pipeline on 10 hours, time each stage, and multiply. Keep a table of seconds of compute per hour of audio for every step. Fix. Put cheap filters first, store intermediate results as shards of FLAC or int16, and write the per-stage hours into the dataset card.
Chapter 11
Keep the numbers, the checklist, the core code and the sources on one page
You can now follow a thousand hours of raw podcast audio from a shared drive to a table of clean training pairs. This page keeps the whole pipeline in one place: the numbers, a checklist, the code that ties the steps together, where each failure lives, and the sources.
A speech data pipeline turns raw audio into records of sound, text, time and scores. It decodes and resamples once, sets one loudness and one channel layout, finds speech with a VAD and cuts chunks at pauses, labels speakers, gets text from the web and from a strong ASR model, keeps chunks where the two agree, uses forced alignment to pin each word to the clock, scores quality (SNR, a DNSMOS-style score, language, music, overlap), removes copies, decontaminates the eval set, and splits by speaker. Each step writes a column, and every threshold goes in the dataset card.
| What | Number | Why it matters |
|---|---|---|
| One hour, 16 kHz mono int16 | 115.2 MB (1,000 h = 115.2 GB) | The unit for every storage plan |
| One hour, 44.1 kHz stereo float32 | 1,270,080,000 bytes | The format alone can change the bill ten times |
| Nyquist limit at 16,000 Hz | 8,000 Hz; a 6,000 Hz tone at 8,000 Hz reads as 2,000 Hz | Low-pass before you resample down |
| Loudness example | RMS 0.158 = −16.0 dBFS; gain to −20 is −4.0 dB (×0.631) | One gain per clip, then check for clipping |
| Mono downmix, phase offset 180° | amplitude |cos(φ/2)| = 0 | Check the mono RMS against the channels |
| Word error rate example | 1 substitution in 7 words = 14.3% | The agreement metric |
| SNR example | power ratio 100 = 20 dB | A cheap noise score |
| Whisper (2212.04356) | 680,000 hours | Weak supervision at scale |
| The People’s Speech (2111.09344) | 30,000 hours; 9.98% WER on Librispeech test-clean | A commercial-use English set |
| Emilia (2407.05361) | over 101k hours, six languages | Open pipeline for speech generation data |
| YODAS (2406.00899) | over 500k hours, over 100 languages | Creative Commons audio at scale |
| MMS (2305.13516) | 1,107 languages ASR; 4,017 language ID; 54 FLEURS languages at under half Whisper’s WER | Coverage beyond about one hundred languages |
| CTC-Segmentation (2007.09127) | over 1700 h German; 12.8% vs 14.4% WER | Alignment turns unlabeled audio into data |
| CLAP (2206.04769) | 128,010 training pairs; 1,024 number embeddings; 100% zero-shot on GTZAN music versus speech | Text prompts can tag and filter audio |
| LAION-CLAP (2211.06687) | 633,526 audio-text pairs, public | Variable-length clips and open data |
| WhisperX (2303.00747) | twelve-fold speedup | Pre-segmenting enables batching |
| 101,000 hours at 115.2 MB per hour | 11.6 TB | Price storage first |
Every step above is a function that adds columns to a record. This sketch strings them together for one episode. The helpers (vad_probs, embed, asr, web_text, align, scores) stand for whichever models you choose.
pythondef process_episode(path, eval_grams, eval_speakers): x = load_mono_16k(path) # Ch 1: shape (N,), float32 x, why = normalize_checked(x) # Ch 2: level, clipping, phase if x is None: return [] chunks = segment(vad_probs(x), max_len=15.0) # Ch 3: [(start_s, end_s), ...] spk = diarize(chunks, embed, thr=0.5) # Ch 4: speaker per chunk rows = [] for (s, e), sp in zip(chunks, spk): clip = x[int(s * 16000):int(e * 16000)] text, w = asr(clip), web_text(path, s, e) # Ch 5: two sources ok, wer = keep(w, text, dur=e - s) if not ok: continue words, a_score = align(clip, text) # Ch 6: word times + score tag = clap_tag(clip) # Ch 7: text-prompt tag, optional q = scores(clip) # Ch 8: snr, quality, lang, music, overlap ok, why = passes(q) if not ok or a_score < -1.0: continue if ngrams(text) & eval_grams or sp["speaker"] in eval_speakers: continue # Ch 9 rows.append({"text": text, "words": words, "speaker": sp["speaker"], "agree_wer": wer, **q}) return rows
Pick a failure. The strip shows the stage where it starts. The panel gives the symptom, the measurement and the fix, and tells you which chapter has the detail.
The failures, measurements and fixes are the ones taught in Chapters 1 to 9. The hand-check sizes (for example 200 chunks) are working suggestions, not standards.
Audio data work sits between signal processing and language-model data work. The signal side is taught in audio representations. The recognizer that supplies most pseudo-labels is explained in Whisper and the Whisper paper walkthrough. Models that learn from unlabeled audio are in self-supervised speech. For the output side, see streaming speech and voice turn-taking, which need the speaker and timing columns from this pipeline. Text-style dedup and filtering are in data filtering and dedup, and how data size affects speech language models is in speech language model scaling.
Facts and numbers come from the abstract pages of these papers. Everything labelled illustrative is a toy.
Now press Present or Teach and walk through one episode from memory: what the array looks like, where the cuts go, how two sources of text agree, how the trellis finds word times, and which three scores decide whether a chunk stays. Then go back to the thousand hours at the top, press Strict, and explain what you gain and what you pay.