Datasets and DataLoaders

A fast GPU fed by a slow data path does nothing most of the time. With 64 images per batch, 2 ms to load each one and a 40 ms GPU step, one loading process keeps the GPU busy only 24% of the time. Four workers make it 100%.

Learn what a Dataset and a DataLoader do inside: samplers, batches, collation, padding, workers, prefetching, seeds, and the splits and statistics that keep your test score honest.

Slide the number of workers. Each blue block is one batch being built by a worker; each purple block is one GPU step. Red gaps are time the GPU waits for data.

You need PyTorch tensors (part 1) and the idea of a training loop. We build everything else from zero.

Keep the GPU fed

Ready
0
2.0
64
40

A deterministic simulation of PyTorch's DataLoader rules: each worker builds whole batches, batches go to workers in turn, and at most prefetch_factor × num_workers = 2 × W batches are in flight. With 0 workers the main process loads each batch itself, so loading and computing alternate. Collation and the copy to the GPU are left out. In real training the GPU step time also grows with batch size; here you set it directly. Numbers are illustrative.

Chapter 0

The GPU is waiting

A training run is slow, and the GPU is not the reason. Find where one training step spends its time

You train an image classifier on a new GPU. It should take a day. After an hour the progress bar says three days. You run nvidia-smi and the GPU shows 25% busy. The model is not slow. The GPU is waiting for data.

Look at what one training step needs. Somebody must read 64 image files from disk, decode each JPEG into pixels, crop and flip them, stack them into one tensor, and copy that tensor to the GPU. Only then can the GPU run the forward and backward pass.

Write the plain loop and all of that happens in a line, one thing after another. The GPU computes, then sits idle while the CPU prepares the next batch. Then the CPU sits idle while the GPU computes.

B
The batch size: how many samples go through the model in one step.
ts
The time to load one sample on the CPU: read, decode, augment.
tg
The GPU's time for one step: forward, backward and the optimizer update.
busy
The fraction of wall-clock time the GPU does useful work. This is what nvidia-smi calls utilisation.

Worked: B = 64, ts = 2 ms, tg = 40 ms → Tstep = 128 + 40 = 168 ms, busy = 40 / 168 = 24%. You paid for a GPU and use a quarter of it.

Where one step's time goes

Pick a kind of data and a batch size. The bar is one step of the plain loop, stage by stage. Then switch to "loading in the background" to see the loading leave the GPU's path.

64

Stage costs per sample are illustrative round numbers for a desktop CPU: JPEG 224 × 224 (read 0.3 ms, decode 1.2 ms, augment 0.5 ms), 10 s audio at 16 kHz (read 0.4 ms, decode 0.8 ms, augment 0.8 ms), 512 pre-tokenised ids (read 0.02 ms). GPU step times are illustrative too. Measure your own with the code in Chapter 7.

The fix has two parts. First, describe your data so PyTorch can fetch any sample by itself: that is a Dataset. Second, let background processes build the next batches while the GPU works on this one: that is a DataLoader with workers. The rest of this lesson takes both apart, and then deals with the quieter bugs: padding, imbalance, seeds, and test scores that lie.

The road map of this lesson

  1. The Dataset protocol: two methods, and why loading is lazy.
  2. Inside the DataLoader: sampler, batch sampler, fetch, collate. Build it
  3. Sequences of different lengths: padding, masks and pad_sequence. Build it
  4. Bucketing by length: the maths of padding waste. Why it works
  5. Samplers: shuffle, rare classes, and many GPUs. Build it
  6. Seeds and reproducibility: generators and worker seeds. Did it work?
  7. Workers, prefetching and pinned memory: when the GPU starves. Why it works
  8. Transforms and statistics: augment train only, fit on train only. When not
  9. Splits that do not leak: random versus grouped. Did it work?
  10. Streams and caches: IterableDataset, sharding and stored features. Where else
  11. Connections: the cheat sheet, a symptom guide and the rest of the series.

PyTorch, Zero to Expert

This is part 4 of an eleven-part series. Each lesson has Code Labs that run the PyTorch API in your browser.

  1. PyTorch Tensors: Storage, Strides and Views
  2. Autograd: the Graph Behind backward()
  3. nn.Module Inside Out
  4. Datasets and DataLoaders (this lesson)
  5. The Training Loop, Line by Line
  6. Debugging Training: When the Loss Won't Go Down
  7. Memory and Mixed Precision
  8. Making PyTorch Fast
  9. Distributed PyTorch: DDP and FSDP from the Inside
  10. PyTorch Internals: From a Python Call to a Kernel
  11. Build PyTorch from Scratch
The plain loop loads 32 samples at 3 ms each, then runs a 32 ms GPU step. How busy is the GPU?

Chapter 1

The Dataset protocol

Two methods, __len__ and __getitem__, describe any data. Keep the description small and load each sample only when it is asked for

Think of a library catalogue. It does not hold the books. It holds a count, and a way to fetch book number 7 when you ask. A PyTorch Dataset is that catalogue for your data.

A map-style dataset is any object with two methods. __len__ returns how many samples there are. __getitem__(i) returns sample number i. That is the whole protocol. The class torch.utils.data.Dataset adds almost nothing; it marks your class as one.

import torch
from torch.utils.data import Dataset
from PIL import Image

class ImageFiles(Dataset):
    def __init__(self, paths, labels, transform=None):
        self.paths = paths              # a list of strings: a few bytes each
        self.labels = labels            # a list of ints
        self.transform = transform

    def __len__(self):
        return len(self.paths)          # the DataLoader asks this once per epoch

    def __getitem__(self, i):           # called once per sample, often in a worker
        img = Image.open(self.paths[i]).convert("RGB")   # read + decode NOW
        if self.transform:
            img = self.transform(img)   # e.g. crop, flip, ToTensor: (3, 224, 224)
        return img, self.labels[i]      # one sample: (tensor, int)

Notice where the work happens. __init__ stores only paths. The file is opened inside __getitem__, at the moment sample i is needed. This is lazy loading. The other choice, eager loading, decodes everything in __init__ and keeps it in memory. Eager loading is fine for small data. For large data it fails, and the maths says why.

N
The number of samples in the dataset.
bytes per sample
The decoded size: number of values times bytes per value (4 for float32, 1 for uint8).
RAMeager
The memory an eager dataset needs before training can start.

Worked, ImageNet's 1,281,167 training images at 3 × 224 × 224: 150,528 values × 4 bytes = 602 KB each → 771 GB as float32, or 193 GB as uint8. A lazy dataset holds 1.28 million paths: about 100 MB.

Eager or lazy?

Pick what one sample is, then grow the dataset. The bars compare the memory each approach needs against a 64 GB machine, and the wait before the first training step.

100,000

Image: 3 × 224 × 224. Audio: 160,000 float32 samples. Tokens: 512 int64 ids. Lazy cost: 100 bytes of path per sample, plus 4 workers × 2 batches of 64 decoded samples in flight. Eager start-up assumes 1 ms of decoding per image or clip on 8 cores; token ids need no decoding. Illustrative.

Lazy loading also moves the work to the right place. When the DataLoader has workers, each worker calls __getitem__ in its own process. The decoding then runs on several CPU cores at once, in parallel with the GPU. An eager dataset does all its decoding in one process, before step 1.

There is one more rule. __getitem__ should depend only on i, plus any randomness you meant to add for augmentation. Then any worker can fetch any sample in any order, and the result is the same. Code that reads "the next line of a file" inside __getitem__ breaks this rule, and breaks under shuffling and workers.

Your dataset has 2 million images. Each decodes to 3 × 256 × 256 uint8. How much RAM would an eager dataset need?

Chapter 2 Build it

Inside the DataLoader

A DataLoader is four small parts in a chain: a sampler picks indices, a batch sampler groups them, a fetch calls your Dataset, and a collate function stacks the results

A DataLoader looks like one big machine. It is four small parts, and you can replace each one.

Written out in Python, a DataLoader with no workers is just this loop:

def simple_loader(dataset, batch_size, shuffle, collate_fn, drop_last=False):
    n = len(dataset)
    order = torch.randperm(n).tolist() if shuffle else list(range(n))  # sampler
    for s in range(0, n, batch_size):                                   # batch sampler
        idx = order[s:s + batch_size]
        if drop_last and len(idx) < batch_size:
            break
        samples = [dataset[i] for i in idx]                             # fetch
        yield collate_fn(samples)                                       # collate

The number of batches per epoch follows from that loop:

N
len(dataset), or the sampler's length if you pass a sampler.
B
batch_size. The default is 1.
drop_last
If True, the last batch is dropped when it is short. Use it when a short batch would break something, such as BatchNorm on 1 sample.
batches
What len(loader) returns.

Worked: N = 10, B = 4 → ⌈2.5⌉ = 3 batches of sizes 4, 4 and 2. With drop_last: 2 batches, and samples from the short batch are skipped that epoch.

One epoch, one part at a time

Press Step to move one part forward. Turn on shuffle, change the batch size, or turn on drop_last, then step through again.

4

A toy dataset of 10 samples; each sample is (a tensor of 3 numbers, an int label). The shuffled order is drawn in your browser, not by PyTorch.

What default_collate does with each type

The default collate function, default_collate, looks at the first sample and applies one rule per type. It recurses into containers, so a dict of tensors becomes a dict of batched tensors.

Each sample holdsThe batch holds
A tensor of shape SOne tensor of shape B × S (torch.stack). All shapes must match.
A NumPy arrayA stacked tensor with the array's dtype.
A Python intAn int64 tensor of shape B.
A Python floatA float64 tensor of shape B.
A stringA plain list of B strings.
A dictA dict with the same keys, each value collated.
A tuple or listA list, transposed: entry k collates the k-th item of every sample.

When tensor shapes differ, the stack fails with RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [5] at entry 1. That error means your samples have different lengths, and you need your own collate function. Chapter 3 writes one.

Each sample is (a tensor of shape 3 × 32 × 32, a Python int). What does one batch of 16 hold?

Chapter 3 Build it

Sequences of different lengths

Sentences, audio clips and sensor logs come in many lengths. Pad them to one length, and carry a mask that says which positions are real

A batch must be one rectangular tensor. Sentences are not rectangular: one has 6 tokens, the next has 19. The usual answer is padding: fill every sequence up to the longest one in the batch with a filler value, often 0.

Padding creates a new danger. The model now sees fake positions. Anything that sums or averages over time will mix them in. So a padded batch always travels with a mask: a B × T tensor of booleans that is True at real positions and False at padding.

Lb
The true length of sequence b in the batch.
mb,t
The mask: 1 (True) if position t is real, 0 if it is padding.
xb,t
The feature vector of sequence b at position t, padding included.
zb
The masked mean: the average over real positions only.

Worked: a sequence (2, 4) padded to length 4 is (2, 4, 0, 0). Plain mean = 6 / 4 = 1.5. Masked mean = 6 / 2 = 3.0, the true answer.

Pad, mask, pool

Each row is one sequence. Warm cells are real tokens; red cells are padding. Draw a new batch, switch the padding side (padding_side in pad_sequence), and compare plain and masked mean pooling.

Each token holds one number between 1 and 3, so the true mean of every sequence is about 2. Lengths are drawn between 2 and 16. Illustrative.

PyTorch gives you the padding step. torch.nn.utils.rnn.pad_sequence(sequences, batch_first=False, padding_value=0.0, padding_side='right') takes a list of tensors of shape (Li, …) and returns one tensor. With batch_first=True its shape is B × Tmax × …. The mask you build yourself, with one broadcast comparison:

import torch
from torch.nn.utils.rnn import pad_sequence

def collate_varlen(batch):
    """batch: list of (x_i, y_i); x_i has shape (L_i, D)."""
    xs, ys = zip(*batch)
    lengths = torch.tensor([len(x) for x in xs])        # (B,)
    x = pad_sequence(xs, batch_first=True)              # (B, T_max, D), zeros after each end
    T = x.shape[1]
    mask = torch.arange(T)[None, :] < lengths[:, None]   # (1, T) < (B, 1) -> (B, T) bool
    return x, mask, lengths, torch.tensor(ys)

loader = DataLoader(ds, batch_size=32, shuffle=True, collate_fn=collate_varlen)

Where does the mask go next? Pooling uses it as above. Attention layers take it as a key padding mask, so no query attends to a padded key: in nn.MultiheadAttention the argument key_padding_mask is True where a position should be ignored, the opposite of our mask. Losses use it to skip padded targets, for example with ignore_index=-100 in F.cross_entropy.

A batch has lengths (3, 5, 2) and is padded on the right. What is row 2 of the mask?

Chapter 4 Why it works

Bucketing by length

Every batch pays for its longest member. Put sequences of similar length together, and most of the padding disappears

Padding is not free. The GPU computes on every position, real or not. A batch of 32 sentences padded to 300 tokens costs 32 × 300 = 9,600 token slots, even if most sentences have 40 tokens. The fraction of slots that hold padding is the padding waste.

Li
The length of sample i. The top sum counts every real token in the epoch.
Bk
The number of samples in batch k.
max L in batch k
The longest sample in batch k: every row of the batch is padded to it.
waste
The fraction of computed slots that are padding.

Worked: one batch with lengths (10, 12, 50, 8): real = 80, computed = 4 × 50 = 200 → waste = 1 − 80/200 = 60%. Group them as (8, 10, 12) and (50): computed = 3 × 12 + 50 = 86 → waste 7%.

Random batches are the worst case for a long-tailed length distribution, which real text and audio have. Almost every batch catches one long sample, and every row pays for it. For a transformer it is worse than the slot count says: attention cost grows with the square of the padded length.

The fix is bucketing. Shuffle the dataset, take a large pool of indices, sort the pool by length, and cut it into batches. Then shuffle the order of the batches. Samples in one batch now have similar lengths. The training order stays random at the level of batches.

Where the padding goes

2,000 sequences with a long tail of lengths. Each row is one batch: warm is real tokens, red is padding. Compare the three ways of forming batches, then change the pool size.

32
50

Lengths drawn from a log-normal distribution (median about 37, clipped to 4 to 400), the same shape as the Code Lab below. The first 24 batches of the epoch are drawn; the waste figure covers the whole epoch. Illustrative.

Why not sort the whole dataset? Look at the "fully sorted" option. Its waste is lowest, but every epoch now runs from the shortest sequences to the longest. The model sees a hidden curriculum, and each batch is far from a random sample of the data, which biases the gradient. The pool is the knob between the two: larger pools mean less padding but less mixing.

In PyTorch a bucketing scheme is a batch sampler: an iterable that yields lists of indices. You pass it as DataLoader(ds, batch_sampler=my_sampler, collate_fn=collate_varlen). With a batch sampler you do not pass batch_size, shuffle, sampler or drop_last; it owns all four.

A batch holds sequences of lengths 20, 20, 20 and 100. What fraction of its computed slots is padding?

Chapter 5 Build it

Samplers: order, rare classes and many GPUs

The sampler decides which samples a run sees and how often. Use it to shuffle, to lift rare classes, and to give each GPU its own share

A sampler is any iterable of indices with a length. shuffle=True is a shortcut for sampler=RandomSampler(ds): a fresh permutation every epoch. shuffle=False means SequentialSampler: 0, 1, 2, and so on. Two more samplers solve problems that shuffling cannot.

Rare classes: WeightedRandomSampler

Say you detect a fault that appears in 1 sample out of 21: 2,000 normal and 100 faulty. With plain shuffling, many batches contain no faulty sample at all. The gradient from those batches says "always predict normal".

p
The share of the rare class in the data: 100 / 2,100 = 0.048.
B
The batch size.
P(none)
The chance that a uniformly drawn batch holds no rare sample.
nc(i)
The number of samples in the class of sample i.
wi
Sample i's weight. Each class's weights then sum to 1, so each class is drawn equally often.

Worked: B = 32 → (20/21)32 = 0.21: one batch in five has no rare sample. Weights: normal 1/2000, rare 1/100, so a rare sample is 20× as likely to be drawn as a normal one.

Lift the rare class

Each row is one batch; pink dots are rare samples, yellow dots are common ones. Switch the sampler, then read the bottom line: how many distinct samples the epoch actually used.

32

2,000 common and 100 rare samples; an epoch draws 2,100 indices. The first 14 batches are drawn. Sampling runs in your browser with the same rule as PyTorch: with replacement, probability proportional to weight.

The device shows the price. Weighted sampling draws with replacement, so each rare sample appears about 10 times per epoch, and about 60% of the common samples are not drawn in a given epoch. The rare class may now be overfitted. Watch its validation loss separately, and consider a softer weight such as 1 / √nc, or a weighted loss instead (Imbalance and Calibration compares them).

Many GPUs: DistributedSampler

In data-parallel training, each GPU runs its own copy of the model and its own DataLoader. They must see different samples, or you pay for 4 GPUs and train on one GPU's data. DistributedSampler splits the indices. With R processes ("replicas") and rank r, it takes every R-th index starting at r. If N does not divide by R, it repeats a few indices from the start so every rank gets the same count:

N
The dataset size.
R
The number of processes, usually one per GPU (num_replicas, the world size).
per rank
How many samples each process gets per epoch. Equal counts keep the processes in step.
repeats
Indices seen twice per epoch so that the counts match. With drop_last=True the tail is dropped instead.

Worked: N = 10, R = 3 → 4 per rank, 2 repeats. Without shuffling: rank 0 gets [0, 3, 6, 9], rank 1 gets [1, 4, 7, 0], rank 2 gets [2, 5, 8, 1].

One dataset, R processes

Each row below the order is one rank's share; the r-labels say which rank takes each slot. Red outlines mark repeated indices. Turn on shuffle, then step through epochs with and without set_epoch.

3
0

N = 10. The split rule (pad by repeating from the start, then take every R-th index) is PyTorch's. The shuffled order is drawn in your browser, seeded with seed + epoch as PyTorch does, so the exact order differs from PyTorch's.

from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler

sampler = DistributedSampler(ds, shuffle=True, seed=0)   # reads rank and world size from torch.distributed
loader = DataLoader(ds, batch_size=32, sampler=sampler, num_workers=4, pin_memory=True)
for epoch in range(epochs):
    sampler.set_epoch(epoch)        # shuffle with seed + epoch: forget this and every epoch repeats
    for x, y in loader:
        ...
You train on 4 GPUs with DistributedSampler and shuffle=True, but forget set_epoch. What happens?

Chapter 6 Did it work?

Seeds and reproducibility

A result you cannot repeat is hard to debug. Give the data its own random stream, and know how each worker gets its seed

You change one line of the model and the loss curve changes. Was it your change, or a different data order? If the order moved, you cannot tell. Reproducible data order is the first tool of debugging.

PyTorch has one global random stream, seeded by torch.manual_seed. Weight initialisation draws from it. Dropout draws from it. And a shuffling DataLoader without a generator draws its shuffle seed from it at the start of every epoch. So any change that draws a different count of random numbers, such as a wider layer, shifts the data order too.

The fix is a generator: a separate random stream that belongs to the loader alone.

import random
import numpy as np
import torch

def seed_worker(worker_id):
    s = torch.initial_seed() % 2**32      # this worker's torch seed: base_seed + worker_id
    np.random.seed(s)                     # hand it on to NumPy and random
    random.seed(s)

g = torch.Generator()
g.manual_seed(0)                          # the data order now depends on this seed only
loader = DataLoader(ds, batch_size=64, shuffle=True, num_workers=4,
                    worker_init_fn=seed_worker, generator=g)

Workers bring a second question: what random numbers does each worker see? Augmentation runs inside workers, and four workers that draw the same numbers would apply the same crop to different images in lockstep. PyTorch prevents this. At the start of each epoch it draws one base seed from the loader's generator, and seeds each worker from it:

base_seed
One 64-bit number drawn from generator (or the global stream) when an epoch's iterator starts.
worker_id
0 to num_workers − 1.
seedw
The seed of torch's and Python's random inside worker w. Current PyTorch also seeds NumPy's global generator in each worker from base_seed and worker_id.

Worked: base_seed = 1,000 (illustrative) and 4 workers → seeds 1,000, 1,001, 1,002, 1,003. Next epoch a new base_seed, so new augmentations. A fixed generator makes the whole sequence of base seeds repeat from run to run.

Run it again

Pick a seeding choice. Run 1 and run 2 use the same code; run 3 has a wider model. Chips are the epoch-2 sample order; red chips differ from run 1. Below, four workers draw a random crop each.

A model of PyTorch's rules, run in your browser with its own random stream: init and dropout draw from the global stream; without a generator the shuffle seed is drawn from it too. The NumPy case models a np.random.default_rng(0) object built in __init__ and copied into each worker. Illustrative.

The second switch shows the trap that survives PyTorch's seeding. If your Dataset builds its own random object in __init__, such as self.rng = np.random.default_rng(0), each worker gets a copy of that object, in the same state. PyTorch seeds the global generators, not your object. All workers then draw the same numbers. Draw augmentation randomness from torch inside __getitem__, or rebuild the object per worker in worker_init_fn from torch.initial_seed().

You set torch.manual_seed(0) and shuffle=True with no generator. You add a Dropout layer, and the data order changes. Why?

Chapter 7 Why it works

Workers, prefetching and pinned memory

Workers are producers and the GPU is the consumer. Compute how many producers you need, why a buffer absorbs slow samples, and how the batch reaches the GPU

With num_workers=W greater than 0, the DataLoader starts W worker processes. The main process sends each worker a list of indices; the worker fetches and collates a whole batch, and sends it back through a queue. Batches go to workers in turn: batch 0 to worker 0, batch 1 to worker 1, and so on. The main process hands batches to your loop in their original order.

This is a producer and consumer system. The workers produce batches; the training loop consumes them. One worker needs B · ts to build a batch. The GPU needs tg to use one. The slower side sets the pace:

W
num_workers. Together the workers make W / (B ts) batches per millisecond.
ts
Load time per sample inside a worker.
tg
The GPU's time per step. The GPU can take 1 / tg batches per millisecond.
throughput
Samples per millisecond in steady state; multiply by 1,000 for samples per second.
W*
The fewest workers that keep the GPU busy, if each worker has a CPU core of its own.

Worked: B = 64, ts = 2 ms, tg = 40 ms → one batch takes a worker 128 ms; W* = ⌈3.2⌉ = 4. With W = 2: 64 × 2/128 per ms = 1,000 samples/s and the GPU is 62.5% busy. With W = 4: GPU-bound at 64 / 40 ms = 1,600 samples/s.

That is the steady state with steady times. Real load times jump: a large file, a slow disk read, a heavy augmentation. A buffer of ready batches absorbs those jumps. PyTorch keeps up to prefetch_factor batches per worker in flight; the default is 2 whenever W > 0, so 2W batches in total.

Jitter and the buffer

The workers here make exactly as many batches per second as the GPU uses, on average. Add jitter and watch the GPU starve. Then raise prefetch_factor, add a fifth worker, or pin memory.

4
60%
2

Mean batch build time 160 ms per worker; GPU step 40 ms, so 4 workers match the GPU on average. Build times are log-normal with the chosen spread. The copy of one batch to the GPU takes 8 ms from pageable memory and blocks the GPU's stream; from pinned memory it takes 4 ms and overlaps the previous step. 200 batches, same random draws for every setting. Illustrative.

Pinned memory and the copy to the GPU

A batch is built in ordinary CPU memory. The operating system may move such memory around, so the GPU cannot read it directly. The driver first copies it into a fixed buffer, then sends it. Pinned memory (page-locked memory) is memory the operating system promises not to move. The GPU can copy from it directly, faster, and in the background.

With pin_memory=True, a thread in the main process copies each finished batch into pinned memory. Then x.to('cuda', non_blocking=True) starts the copy and returns at once. The copy runs while the GPU still computes the previous step. Without pinned memory, non_blocking=True cannot overlap the copy.

Run this on a machine with PyTorch and a GPU

import time
import torch
from torch.utils.data import DataLoader

def loader_speed(ds, batch_size=64, n_batches=100, **kw):
    """Samples per second from the loader alone: no model, no GPU work."""
    loader = DataLoader(ds, batch_size=batch_size, shuffle=True, **kw)
    it = iter(loader)
    next(it)                                 # the first batch includes worker start-up
    t0 = time.perf_counter()
    for _ in range(n_batches):
        x, y = next(it)
        x = x.to("cuda", non_blocking=True)  # include the copy
    torch.cuda.synchronize()                 # wait for the last copy to finish
    return n_batches * batch_size / (time.perf_counter() - t0)

for w in [0, 2, 4, 8]:
    print(w, round(loader_speed(ds, num_workers=w, pin_memory=True)), "samples/s")

Compare that number with your training speed in samples per second. If the loader alone is not much faster than training, data is the bottleneck. Then raise W up to the number of CPU cores, make __getitem__ cheaper (smaller files, faster decoding, caching), or move augmentation to the GPU.

Batch size 128, 1.5 ms to load each sample, 60 ms GPU step. How many workers keep the GPU busy?

Chapter 8 When not

Transforms and statistics

Augment the training data only. Fit every statistic, such as a mean and a standard deviation, on the training data only. Break either rule and your test score lies

A transform is a function applied to each sample in __getitem__. Two kinds behave very differently.

x
One raw value, for example a blood measurement or a pixel channel.
μtrain
The mean of that feature over the training rows only.
σtrain
Its standard deviation over the training rows only.
z
The normalised value the model sees, with the same μ and σ for every split.

Worked: μtrain = 50, σtrain = 4, a test value x = 56 → z = 6 / 4 = 1.5. If the test hospital reads 6 units high on average, its z values sit near 1.5, not 0, and the model will tell you how much that hurts.

Why "training rows only"? Your test set stands in for the future. In deployment the data arrives one sample at a time, and you know only what you knew at training time. Any number computed with test rows in it is information from the future. This is leakage. Fitting on everything is the mild form. Normalising the test set with its own statistics is the strong form: it silently removes any shift between the test data and the training data.

The score that lies

A classifier is trained on hospital A and tested on hospital B, whose scanner reads high. Pick where the statistics come from and grow the shift. Compare the test score you would report with the score once deployed, one patient at a time.

1.00

One feature, two classes, 600 training and 600 test patients. The classifier is a threshold fitted on the normalised training data. "Deployed" normalises each new patient with the statistics frozen at training time, as a real service must. The shift is in units of the class spread. Illustrative.

With "train only", the reported score equals the deployed score: an honest number, even when it is bad news. With per-split statistics, the report looks good and deployment does not. The same rule covers every fitted step: vocabularies, PCA, feature selection, imputation values, class weights, and thresholds tuned on validation data.

In code, compute the statistics once from the training split, save them with the model, and pass them into each Dataset. Build the training and test datasets with different transform lists:

import torch
from torch.utils.data import Subset

X_train = X[train_idx]
mean, std = X_train.mean(dim=0), X_train.std(dim=0)    # training rows only
torch.save({"mean": mean, "std": std}, "norm_stats.pt") # ship them with the weights

train_ds = Visits(X[train_idx], mean, std, train=True)  # augmentation on
val_ds = Visits(X[val_idx], mean, std, train=False)     # augmentation off
test_ds = Visits(X[test_idx], mean, std, train=False)   # same mean and std everywhere

# Trap: Subset shares the parent's transform, so this augments the test set too:
# full = ImageFolder(root, transform=train_tf); test = Subset(full, test_idx)
You normalise your validation set with its own mean and standard deviation. What goes wrong?

Chapter 9 Did it work?

Splits that do not leak

Samples from one patient, speaker or scene look alike. Split by that unit, or the test set rewards memory instead of skill

You build a skin-lesion classifier from 3,000 photos of 400 patients. A random 80/20 split gives 94% test accuracy. In the clinic it gets 71%. What happened?

Photos of one patient share skin tone, lighting, camera and even the same lesion from two angles. A random split puts some of a patient's photos in training and others in test. The model can score well by recognising the patient. The test set measured memory of people it had already seen. In the clinic every patient is new.

The fix is a grouped split: assign whole groups, here patients, to one side. The unit of the split must be the unit you will meet new examples of in deployment.

f
The test fraction of a random per-sample split, such as 0.2.
k
The number of samples in one group, such as one patient's photos.
P(one side)
The chance that a random split keeps that whole group on one side. One minus this is the chance the group leaks.

Worked: f = 0.2, k = 10 → 0.810 + 0.210 = 0.107 + 0.0000001 ≈ 0.11. So about 89% of 10-photo patients end up on both sides of a random split.

Random split or grouped split?

Each column is one patient and each dot one sample; filled dots are in the test set. A nearest-neighbour model (pure memory) is scored on the test set and on brand-new patients. Make the patients more alike inside, and compare the splits.

0.80

30 patients with 6 to 14 samples each; 16 features. A sample is a class signal plus a personal offset shared by all of that patient's samples, plus noise. Labels depend on the class signal only. The model is 1-nearest-neighbour. "New patients" are 30 more patients drawn the same way. Illustrative.

The rule has many names. In speech, split by speaker. In vision, split by scene, video or photographer, because frames of one video are near-copies. In recommendation, split by user. In time series, split by time: train on the past, test on the future, with a gap so that windows do not overlap. And always remove exact and near duplicates across splits first. Audio Data Pipelines shows speaker-disjoint splits and deduplication on real audio collections.

PyTorch's random_split splits samples, not groups, so it is the wrong tool here. Split the indices yourself by group, then wrap them with Subset(ds, train_idx) and Subset(ds, test_idx). scikit-learn's GroupShuffleSplit and GroupKFold do the same for index arrays.

You build a speaker-independent keyword spotter from 10,000 clips of 200 speakers. Which split measures what you want?

Chapter 10 Where else

Streams, shards and caches

When the data has no index, stream it with an IterableDataset and give each worker its own shards. When the expensive part never changes, compute it once and cache it

A map-style dataset needs random access: "give me sample 7,203". Some data does not allow that. A web crawl lives in thousands of compressed files read from start to end. A log arrives as a live stream. For these, PyTorch has the IterableDataset: you write __iter__, and it yields samples one after another.

Workers bring a trap. Each worker gets its own copy of the dataset and calls __iter__ on it. Without care, all W workers read the same stream from the start, and every sample arrives W times. The fix is sharding: each worker reads only its own part. Inside a worker, torch.utils.data.get_worker_info() tells the worker its id and the number of workers; in the main process it returns None.

rank, R
This process's number and the number of processes (GPUs). R = 1 on one GPU.
W
Workers per process.
worker_id
From get_worker_info().id.
k
A global reader number from 0 to R·W − 1.
s
A shard file's number. Every shard has exactly one reader.

Worked: 64 shards, 2 GPUs × 4 workers = 8 readers → 8 shards each. With 10 shards and 8 readers, two readers get 2 shards and six get 1: the run waits on the slowest. Use a shard count that is a multiple of R·W.

Shard the stream

8 shard files of 6 samples each; each sample shows its shard number and gets darker with it. Turn sharding off to see the duplicates. Then grow the shuffle buffer and watch the shards mix.

2
1

Workers take turns producing one sample each into the output stream (real loaders hand over whole batches). The shuffle buffer keeps K samples and emits a random one each time a new sample arrives. Illustrative.

import torch
from torch.utils.data import IterableDataset, DataLoader, get_worker_info

class ShardStream(IterableDataset):
    def __init__(self, shard_paths, buffer=1000, seed=0):
        self.shards, self.buffer, self.seed = shard_paths, buffer, seed

    def __iter__(self):
        info = get_worker_info()                       # None in the main process
        wid, nw = (info.id, info.num_workers) if info else (0, 1)
        mine = self.shards[wid::nw]                     # this worker's shards only
        g = torch.Generator().manual_seed(self.seed + wid)
        buf = []
        for path in mine:
            for sample in read_records(path):           # your file reader
                buf.append(sample)
                if len(buf) >= self.buffer:            # emit a random buffered sample
                    yield buf.pop(int(torch.randint(len(buf), (1,), generator=g)))
        while buf:                                      # drain at the end
            yield buf.pop(int(torch.randint(len(buf), (1,), generator=g)))

loader = DataLoader(ShardStream(paths), batch_size=64, num_workers=4)  # no shuffle= here

An IterableDataset cannot use shuffle or a sampler; the DataLoader raises an error if you try. Shuffling happens in two places instead: shuffle the shard order each epoch, and keep a shuffle buffer as above. A buffer of K samples can only move a sample about K places, so a small buffer gives a nearly ordered stream.

Cache what never changes

The cheapest sample is the one you do not compute. If your pipeline runs a frozen pretrained backbone on every sample, its output is the same every epoch. Run it once, save the features, and make your Dataset read the saved features instead. Backbone + Head derives the speed-up: often 20 to 200 times for head training.

import numpy as np
import torch
from torch.utils.data import Dataset

class CachedFeatures(Dataset):
    """Features computed once by a frozen backbone, saved as one (N, D) float16 file."""
    def __init__(self, path, labels):
        self.feats = np.load(path, mmap_mode="r")   # memory-mapped: read pages on demand
        self.labels = labels
    def __len__(self):
        return len(self.labels)
    def __getitem__(self, i):
        x = torch.from_numpy(np.array(self.feats[i], dtype=np.float32))  # (D,)
        return x, self.labels[i]

A cache has two costs. Augmentation dies: the cached sample is the same every epoch, unless you cache several augmented copies. And the cache goes stale: re-create it whenever the backbone, its weights or its preprocessing change. Name the cache file after all three.

An IterableDataset reads 16 shard files with no sharding code. You set num_workers=4. What does one epoch yield?

Chapter 11

Connections

Match a symptom to its cause, keep the one-screen cheat sheet, know the limits, and go on to the training loop

Most data-pipeline bugs show up as one of a few symptoms. Pick one below to see the usual causes in order, the check that tells them apart, and the fix.

From symptom to fix

Pick the symptom you see. The card lists the causes from most to least common, the one check to run first, and the fix.

A field guide from the chapters of this lesson, not an exhaustive list.

The cheat sheet

PartIn one line
Dataset__len__ + __getitem__(i); store paths, load lazily; depend only on i.
DataLoader chainsampler → batch sampler → fetch [ds[i] for i in idx] → collate_fn.
default_collateTensors stack to B × S; ints → int64; floats → float64; strings → list; dicts per key; tuples transposed.
Variable lengthpad_sequence(xs, batch_first=True) + mask arange(T)[None] < lengths[:, None]; mask every mean, attention and loss.
Padding waste1 − ΣL / Σ(B · max L); bucket: shuffle, pool, sort, cut, shuffle batches.
SamplersWeightedRandomSampler with w = 1/nclass; DistributedSampler + set_epoch(epoch).
Seedsgenerator=torch.Generator().manual_seed(s); worker seed = base_seed + worker_id; no RNG objects made in __init__.
ThroughputB · min(W / (B ts), 1 / tg); W* = ⌈B ts / tg⌉; prefetch 2W batches; pin_memory=True + non_blocking=True.
TransformsAugment train only; fit μ, σ and every other statistic on train only; save them with the model.
SplitsSplit by patient, speaker, scene, user or time; dedupe first; Subset with your own indices.
Streams and cachesIterableDataset + get_worker_info() sharding + shuffle buffer; cache frozen-backbone features, and re-cache on change.

Bugs experts have learned to spot

Honest limits

Where this sits, and where to go next

The takeaway. A Dataset is a recipe: count the samples, fetch sample i lazily. A DataLoader is a chain: sampler, batch sampler, fetch, collate, with workers to overlap loading and computing. Get the plumbing right, then guard the honesty of your numbers: augment and fit statistics on training data only, split by the unit you deploy on, and seed the loader with its own generator.
Training is slow and the GPU shows 40% use. The loader alone, with no model, gives fewer samples per second than training needs. What do you try first?

Now press Present or Teach and explain, out loud and from memory, why a GPU waits for data and how many workers it needs. Then go back to the hero, set the load time to 4 ms and find the fewest workers that close every red gap.

Datasets and DataLoaders, part 4 of PyTorch, Zero to Expert, in The Builder's Toolkit. Sources: the PyTorch 2.10 documentation and source for torch.utils.data (torch.utils.data docs; dataloader.py, sampler.py, distributed.py, _utils/collate.py, _utils/worker.py) and torch.nn.utils.rnn.pad_sequence; every default, error message and collate rule quoted here was checked by running PyTorch 2.10. ImageNet's training-set size (1,281,167 images) is from O. Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge" (IJCV 2015, arXiv). Timing numbers on devices are illustrative and labelled so.
Back to Gleams