A fast GPU fed by a slow data path does nothing most of the time. With 64 images per batch, 2 ms to load each one and a 40 ms GPU step, one loading process keeps the GPU busy only 24% of the time. Four workers make it 100%.
Learn what a Dataset and a DataLoader do inside: samplers, batches, collation, padding, workers, prefetching, seeds, and the splits and statistics that keep your test score honest.
Slide the number of workers. Each blue block is one batch being built by a worker; each purple block is one GPU step. Red gaps are time the GPU waits for data.
You need PyTorch tensors (part 1) and the idea of a training loop. We build everything else from zero.
A deterministic simulation of PyTorch's DataLoader rules: each worker builds whole batches, batches go to workers in turn, and at most prefetch_factor × num_workers = 2 × W batches are in flight. With 0 workers the main process loads each batch itself, so loading and computing alternate. Collation and the copy to the GPU are left out. In real training the GPU step time also grows with batch size; here you set it directly. Numbers are illustrative.
Chapter 0
A training run is slow, and the GPU is not the reason. Find where one training step spends its time
You train an image classifier on a new GPU. It should take a day. After an hour the progress bar says three days. You run nvidia-smi and the GPU shows 25% busy. The model is not slow. The GPU is waiting for data.
Look at what one training step needs. Somebody must read 64 image files from disk, decode each JPEG into pixels, crop and flip them, stack them into one tensor, and copy that tensor to the GPU. Only then can the GPU run the forward and backward pass.
Write the plain loop and all of that happens in a line, one thing after another. The GPU computes, then sits idle while the CPU prepares the next batch. Then the CPU sits idle while the GPU computes.
nvidia-smi calls utilisation.Worked: B = 64, ts = 2 ms, tg = 40 ms → Tstep = 128 + 40 = 168 ms, busy = 40 / 168 = 24%. You paid for a GPU and use a quarter of it.
Pick a kind of data and a batch size. The bar is one step of the plain loop, stage by stage. Then switch to "loading in the background" to see the loading leave the GPU's path.
Stage costs per sample are illustrative round numbers for a desktop CPU: JPEG 224 × 224 (read 0.3 ms, decode 1.2 ms, augment 0.5 ms), 10 s audio at 16 kHz (read 0.4 ms, decode 0.8 ms, augment 0.8 ms), 512 pre-tokenised ids (read 0.02 ms). GPU step times are illustrative too. Measure your own with the code in Chapter 7.
The fix has two parts. First, describe your data so PyTorch can fetch any sample by itself: that is a Dataset. Second, let background processes build the next batches while the GPU works on this one: that is a DataLoader with workers. The rest of this lesson takes both apart, and then deals with the quieter bugs: padding, imbalance, seeds, and test scores that lie.
This is part 4 of an eleven-part series. Each lesson has Code Labs that run the PyTorch API in your browser.
Chapter 1
Two methods, __len__ and __getitem__, describe any data. Keep the description small and load each sample only when it is asked for
Think of a library catalogue. It does not hold the books. It holds a count, and a way to fetch book number 7 when you ask. A PyTorch Dataset is that catalogue for your data.
A map-style dataset is any object with two methods. __len__ returns how many samples there are. __getitem__(i) returns sample number i. That is the whole protocol. The class torch.utils.data.Dataset adds almost nothing; it marks your class as one.
import torch
from torch.utils.data import Dataset
from PIL import Image
class ImageFiles(Dataset):
def __init__(self, paths, labels, transform=None):
self.paths = paths # a list of strings: a few bytes each
self.labels = labels # a list of ints
self.transform = transform
def __len__(self):
return len(self.paths) # the DataLoader asks this once per epoch
def __getitem__(self, i): # called once per sample, often in a worker
img = Image.open(self.paths[i]).convert("RGB") # read + decode NOW
if self.transform:
img = self.transform(img) # e.g. crop, flip, ToTensor: (3, 224, 224)
return img, self.labels[i] # one sample: (tensor, int)
Notice where the work happens. __init__ stores only paths. The file is opened inside __getitem__, at the moment sample i is needed. This is lazy loading. The other choice, eager loading, decodes everything in __init__ and keeps it in memory. Eager loading is fine for small data. For large data it fails, and the maths says why.
Worked, ImageNet's 1,281,167 training images at 3 × 224 × 224: 150,528 values × 4 bytes = 602 KB each → 771 GB as float32, or 193 GB as uint8. A lazy dataset holds 1.28 million paths: about 100 MB.
Pick what one sample is, then grow the dataset. The bars compare the memory each approach needs against a 64 GB machine, and the wait before the first training step.
Image: 3 × 224 × 224. Audio: 160,000 float32 samples. Tokens: 512 int64 ids. Lazy cost: 100 bytes of path per sample, plus 4 workers × 2 batches of 64 decoded samples in flight. Eager start-up assumes 1 ms of decoding per image or clip on 8 cores; token ids need no decoding. Illustrative.
Lazy loading also moves the work to the right place. When the DataLoader has workers, each worker calls __getitem__ in its own process. The decoding then runs on several CPU cores at once, in parallel with the GPU. An eager dataset does all its decoding in one process, before step 1.
There is one more rule. __getitem__ should depend only on i, plus any randomness you meant to add for augmentation. Then any worker can fetch any sample in any order, and the result is the same. Code that reads "the next line of a file" inside __getitem__ breaks this rule, and breaks under shuffling and workers.
Chapter 2 Build it
A DataLoader is four small parts in a chain: a sampler picks indices, a batch sampler groups them, a fetch calls your Dataset, and a collate function stacks the results
A DataLoader looks like one big machine. It is four small parts, and you can replace each one.
batch_size, such as [7, 2, 9, 0].dataset[i] for each index in the list. This is the slow part, so it is the part workers run.Written out in Python, a DataLoader with no workers is just this loop:
def simple_loader(dataset, batch_size, shuffle, collate_fn, drop_last=False):
n = len(dataset)
order = torch.randperm(n).tolist() if shuffle else list(range(n)) # sampler
for s in range(0, n, batch_size): # batch sampler
idx = order[s:s + batch_size]
if drop_last and len(idx) < batch_size:
break
samples = [dataset[i] for i in idx] # fetch
yield collate_fn(samples) # collate
The number of batches per epoch follows from that loop:
len(dataset), or the sampler's length if you pass a sampler.batch_size. The default is 1.len(loader) returns.Worked: N = 10, B = 4 → ⌈2.5⌉ = 3 batches of sizes 4, 4 and 2. With drop_last: 2 batches, and samples from the short batch are skipped that epoch.
Press Step to move one part forward. Turn on shuffle, change the batch size, or turn on drop_last, then step through again.
A toy dataset of 10 samples; each sample is (a tensor of 3 numbers, an int label). The shuffled order is drawn in your browser, not by PyTorch.
The default collate function, default_collate, looks at the first sample and applies one rule per type. It recurses into containers, so a dict of tensors becomes a dict of batched tensors.
| Each sample holds | The batch holds |
|---|---|
| A tensor of shape S | One tensor of shape B × S (torch.stack). All shapes must match. |
| A NumPy array | A stacked tensor with the array's dtype. |
| A Python int | An int64 tensor of shape B. |
| A Python float | A float64 tensor of shape B. |
| A string | A plain list of B strings. |
| A dict | A dict with the same keys, each value collated. |
| A tuple or list | A list, transposed: entry k collates the k-th item of every sample. |
When tensor shapes differ, the stack fails with RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [5] at entry 1. That error means your samples have different lengths, and you need your own collate function. Chapter 3 writes one.
Chapter 3 Build it
Sentences, audio clips and sensor logs come in many lengths. Pad them to one length, and carry a mask that says which positions are real
A batch must be one rectangular tensor. Sentences are not rectangular: one has 6 tokens, the next has 19. The usual answer is padding: fill every sequence up to the longest one in the batch with a filler value, often 0.
Padding creates a new danger. The model now sees fake positions. Anything that sums or averages over time will mix them in. So a padded batch always travels with a mask: a B × T tensor of booleans that is True at real positions and False at padding.
Worked: a sequence (2, 4) padded to length 4 is (2, 4, 0, 0). Plain mean = 6 / 4 = 1.5. Masked mean = 6 / 2 = 3.0, the true answer.
Each row is one sequence. Warm cells are real tokens; red cells are padding. Draw a new batch, switch the padding side (padding_side in pad_sequence), and compare plain and masked mean pooling.
Each token holds one number between 1 and 3, so the true mean of every sequence is about 2. Lengths are drawn between 2 and 16. Illustrative.
PyTorch gives you the padding step. torch.nn.utils.rnn.pad_sequence(sequences, batch_first=False, padding_value=0.0, padding_side='right') takes a list of tensors of shape (Li, …) and returns one tensor. With batch_first=True its shape is B × Tmax × …. The mask you build yourself, with one broadcast comparison:
import torch
from torch.nn.utils.rnn import pad_sequence
def collate_varlen(batch):
"""batch: list of (x_i, y_i); x_i has shape (L_i, D)."""
xs, ys = zip(*batch)
lengths = torch.tensor([len(x) for x in xs]) # (B,)
x = pad_sequence(xs, batch_first=True) # (B, T_max, D), zeros after each end
T = x.shape[1]
mask = torch.arange(T)[None, :] < lengths[:, None] # (1, T) < (B, 1) -> (B, T) bool
return x, mask, lengths, torch.tensor(ys)
loader = DataLoader(ds, batch_size=32, shuffle=True, collate_fn=collate_varlen)
Where does the mask go next? Pooling uses it as above. Attention layers take it as a key padding mask, so no query attends to a padded key: in nn.MultiheadAttention the argument key_padding_mask is True where a position should be ignored, the opposite of our mask. Losses use it to skip padded targets, for example with ignore_index=-100 in F.cross_entropy.
Chapter 4 Why it works
Every batch pays for its longest member. Put sequences of similar length together, and most of the padding disappears
Padding is not free. The GPU computes on every position, real or not. A batch of 32 sentences padded to 300 tokens costs 32 × 300 = 9,600 token slots, even if most sentences have 40 tokens. The fraction of slots that hold padding is the padding waste.
Worked: one batch with lengths (10, 12, 50, 8): real = 80, computed = 4 × 50 = 200 → waste = 1 − 80/200 = 60%. Group them as (8, 10, 12) and (50): computed = 3 × 12 + 50 = 86 → waste 7%.
Random batches are the worst case for a long-tailed length distribution, which real text and audio have. Almost every batch catches one long sample, and every row pays for it. For a transformer it is worse than the slot count says: attention cost grows with the square of the padded length.
The fix is bucketing. Shuffle the dataset, take a large pool of indices, sort the pool by length, and cut it into batches. Then shuffle the order of the batches. Samples in one batch now have similar lengths. The training order stays random at the level of batches.
2,000 sequences with a long tail of lengths. Each row is one batch: warm is real tokens, red is padding. Compare the three ways of forming batches, then change the pool size.
Lengths drawn from a log-normal distribution (median about 37, clipped to 4 to 400), the same shape as the Code Lab below. The first 24 batches of the epoch are drawn; the waste figure covers the whole epoch. Illustrative.
Why not sort the whole dataset? Look at the "fully sorted" option. Its waste is lowest, but every epoch now runs from the shortest sequences to the longest. The model sees a hidden curriculum, and each batch is far from a random sample of the data, which biases the gradient. The pool is the knob between the two: larger pools mean less padding but less mixing.
In PyTorch a bucketing scheme is a batch sampler: an iterable that yields lists of indices. You pass it as DataLoader(ds, batch_sampler=my_sampler, collate_fn=collate_varlen). With a batch sampler you do not pass batch_size, shuffle, sampler or drop_last; it owns all four.
Chapter 5 Build it
The sampler decides which samples a run sees and how often. Use it to shuffle, to lift rare classes, and to give each GPU its own share
A sampler is any iterable of indices with a length. shuffle=True is a shortcut for sampler=RandomSampler(ds): a fresh permutation every epoch. shuffle=False means SequentialSampler: 0, 1, 2, and so on. Two more samplers solve problems that shuffling cannot.
Say you detect a fault that appears in 1 sample out of 21: 2,000 normal and 100 faulty. With plain shuffling, many batches contain no faulty sample at all. The gradient from those batches says "always predict normal".
Worked: B = 32 → (20/21)32 = 0.21: one batch in five has no rare sample. Weights: normal 1/2000, rare 1/100, so a rare sample is 20× as likely to be drawn as a normal one.
Each row is one batch; pink dots are rare samples, yellow dots are common ones. Switch the sampler, then read the bottom line: how many distinct samples the epoch actually used.
2,000 common and 100 rare samples; an epoch draws 2,100 indices. The first 14 batches are drawn. Sampling runs in your browser with the same rule as PyTorch: with replacement, probability proportional to weight.
The device shows the price. Weighted sampling draws with replacement, so each rare sample appears about 10 times per epoch, and about 60% of the common samples are not drawn in a given epoch. The rare class may now be overfitted. Watch its validation loss separately, and consider a softer weight such as 1 / √nc, or a weighted loss instead (Imbalance and Calibration compares them).
In data-parallel training, each GPU runs its own copy of the model and its own DataLoader. They must see different samples, or you pay for 4 GPUs and train on one GPU's data. DistributedSampler splits the indices. With R processes ("replicas") and rank r, it takes every R-th index starting at r. If N does not divide by R, it repeats a few indices from the start so every rank gets the same count:
num_replicas, the world size).drop_last=True the tail is dropped instead.Worked: N = 10, R = 3 → 4 per rank, 2 repeats. Without shuffling: rank 0 gets [0, 3, 6, 9], rank 1 gets [1, 4, 7, 0], rank 2 gets [2, 5, 8, 1].
Each row below the order is one rank's share; the r-labels say which rank takes each slot. Red outlines mark repeated indices. Turn on shuffle, then step through epochs with and without set_epoch.
N = 10. The split rule (pad by repeating from the start, then take every R-th index) is PyTorch's. The shuffled order is drawn in your browser, seeded with seed + epoch as PyTorch does, so the exact order differs from PyTorch's.
from torch.utils.data import DataLoader from torch.utils.data.distributed import DistributedSampler sampler = DistributedSampler(ds, shuffle=True, seed=0) # reads rank and world size from torch.distributed loader = DataLoader(ds, batch_size=32, sampler=sampler, num_workers=4, pin_memory=True) for epoch in range(epochs): sampler.set_epoch(epoch) # shuffle with seed + epoch: forget this and every epoch repeats for x, y in loader: ...
Chapter 6 Did it work?
A result you cannot repeat is hard to debug. Give the data its own random stream, and know how each worker gets its seed
You change one line of the model and the loss curve changes. Was it your change, or a different data order? If the order moved, you cannot tell. Reproducible data order is the first tool of debugging.
PyTorch has one global random stream, seeded by torch.manual_seed. Weight initialisation draws from it. Dropout draws from it. And a shuffling DataLoader without a generator draws its shuffle seed from it at the start of every epoch. So any change that draws a different count of random numbers, such as a wider layer, shifts the data order too.
The fix is a generator: a separate random stream that belongs to the loader alone.
import random
import numpy as np
import torch
def seed_worker(worker_id):
s = torch.initial_seed() % 2**32 # this worker's torch seed: base_seed + worker_id
np.random.seed(s) # hand it on to NumPy and random
random.seed(s)
g = torch.Generator()
g.manual_seed(0) # the data order now depends on this seed only
loader = DataLoader(ds, batch_size=64, shuffle=True, num_workers=4,
worker_init_fn=seed_worker, generator=g)
Workers bring a second question: what random numbers does each worker see? Augmentation runs inside workers, and four workers that draw the same numbers would apply the same crop to different images in lockstep. PyTorch prevents this. At the start of each epoch it draws one base seed from the loader's generator, and seeds each worker from it:
generator (or the global stream) when an epoch's iterator starts.random inside worker w. Current PyTorch also seeds NumPy's global generator in each worker from base_seed and worker_id.Worked: base_seed = 1,000 (illustrative) and 4 workers → seeds 1,000, 1,001, 1,002, 1,003. Next epoch a new base_seed, so new augmentations. A fixed generator makes the whole sequence of base seeds repeat from run to run.
Pick a seeding choice. Run 1 and run 2 use the same code; run 3 has a wider model. Chips are the epoch-2 sample order; red chips differ from run 1. Below, four workers draw a random crop each.
A model of PyTorch's rules, run in your browser with its own random stream: init and dropout draw from the global stream; without a generator the shuffle seed is drawn from it too. The NumPy case models a np.random.default_rng(0) object built in __init__ and copied into each worker. Illustrative.
The second switch shows the trap that survives PyTorch's seeding. If your Dataset builds its own random object in __init__, such as self.rng = np.random.default_rng(0), each worker gets a copy of that object, in the same state. PyTorch seeds the global generators, not your object. All workers then draw the same numbers. Draw augmentation randomness from torch inside __getitem__, or rebuild the object per worker in worker_init_fn from torch.initial_seed().
Chapter 7 Why it works
Workers are producers and the GPU is the consumer. Compute how many producers you need, why a buffer absorbs slow samples, and how the batch reaches the GPU
With num_workers=W greater than 0, the DataLoader starts W worker processes. The main process sends each worker a list of indices; the worker fetches and collates a whole batch, and sends it back through a queue. Batches go to workers in turn: batch 0 to worker 0, batch 1 to worker 1, and so on. The main process hands batches to your loop in their original order.
This is a producer and consumer system. The workers produce batches; the training loop consumes them. One worker needs B · ts to build a batch. The GPU needs tg to use one. The slower side sets the pace:
num_workers. Together the workers make W / (B ts) batches per millisecond.Worked: B = 64, ts = 2 ms, tg = 40 ms → one batch takes a worker 128 ms; W* = ⌈3.2⌉ = 4. With W = 2: 64 × 2/128 per ms = 1,000 samples/s and the GPU is 62.5% busy. With W = 4: GPU-bound at 64 / 40 ms = 1,600 samples/s.
That is the steady state with steady times. Real load times jump: a large file, a slow disk read, a heavy augmentation. A buffer of ready batches absorbs those jumps. PyTorch keeps up to prefetch_factor batches per worker in flight; the default is 2 whenever W > 0, so 2W batches in total.
The workers here make exactly as many batches per second as the GPU uses, on average. Add jitter and watch the GPU starve. Then raise prefetch_factor, add a fifth worker, or pin memory.
Mean batch build time 160 ms per worker; GPU step 40 ms, so 4 workers match the GPU on average. Build times are log-normal with the chosen spread. The copy of one batch to the GPU takes 8 ms from pageable memory and blocks the GPU's stream; from pinned memory it takes 4 ms and overlaps the previous step. 200 batches, same random draws for every setting. Illustrative.
A batch is built in ordinary CPU memory. The operating system may move such memory around, so the GPU cannot read it directly. The driver first copies it into a fixed buffer, then sends it. Pinned memory (page-locked memory) is memory the operating system promises not to move. The GPU can copy from it directly, faster, and in the background.
With pin_memory=True, a thread in the main process copies each finished batch into pinned memory. Then x.to('cuda', non_blocking=True) starts the copy and returns at once. The copy runs while the GPU still computes the previous step. Without pinned memory, non_blocking=True cannot overlap the copy.
Run this on a machine with PyTorch and a GPU
import time
import torch
from torch.utils.data import DataLoader
def loader_speed(ds, batch_size=64, n_batches=100, **kw):
"""Samples per second from the loader alone: no model, no GPU work."""
loader = DataLoader(ds, batch_size=batch_size, shuffle=True, **kw)
it = iter(loader)
next(it) # the first batch includes worker start-up
t0 = time.perf_counter()
for _ in range(n_batches):
x, y = next(it)
x = x.to("cuda", non_blocking=True) # include the copy
torch.cuda.synchronize() # wait for the last copy to finish
return n_batches * batch_size / (time.perf_counter() - t0)
for w in [0, 2, 4, 8]:
print(w, round(loader_speed(ds, num_workers=w, pin_memory=True)), "samples/s")
Compare that number with your training speed in samples per second. If the loader alone is not much faster than training, data is the bottleneck. Then raise W up to the number of CPU cores, make __getitem__ cheaper (smaller files, faster decoding, caching), or move augmentation to the GPU.
Chapter 8 When not
Augment the training data only. Fit every statistic, such as a mean and a standard deviation, on the training data only. Break either rule and your test score lies
A transform is a function applied to each sample in __getitem__. Two kinds behave very differently.
Worked: μtrain = 50, σtrain = 4, a test value x = 56 → z = 6 / 4 = 1.5. If the test hospital reads 6 units high on average, its z values sit near 1.5, not 0, and the model will tell you how much that hurts.
Why "training rows only"? Your test set stands in for the future. In deployment the data arrives one sample at a time, and you know only what you knew at training time. Any number computed with test rows in it is information from the future. This is leakage. Fitting on everything is the mild form. Normalising the test set with its own statistics is the strong form: it silently removes any shift between the test data and the training data.
A classifier is trained on hospital A and tested on hospital B, whose scanner reads high. Pick where the statistics come from and grow the shift. Compare the test score you would report with the score once deployed, one patient at a time.
One feature, two classes, 600 training and 600 test patients. The classifier is a threshold fitted on the normalised training data. "Deployed" normalises each new patient with the statistics frozen at training time, as a real service must. The shift is in units of the class spread. Illustrative.
With "train only", the reported score equals the deployed score: an honest number, even when it is bad news. With per-split statistics, the report looks good and deployment does not. The same rule covers every fitted step: vocabularies, PCA, feature selection, imputation values, class weights, and thresholds tuned on validation data.
In code, compute the statistics once from the training split, save them with the model, and pass them into each Dataset. Build the training and test datasets with different transform lists:
import torch from torch.utils.data import Subset X_train = X[train_idx] mean, std = X_train.mean(dim=0), X_train.std(dim=0) # training rows only torch.save({"mean": mean, "std": std}, "norm_stats.pt") # ship them with the weights train_ds = Visits(X[train_idx], mean, std, train=True) # augmentation on val_ds = Visits(X[val_idx], mean, std, train=False) # augmentation off test_ds = Visits(X[test_idx], mean, std, train=False) # same mean and std everywhere # Trap: Subset shares the parent's transform, so this augments the test set too: # full = ImageFolder(root, transform=train_tf); test = Subset(full, test_idx)
Chapter 9 Did it work?
Samples from one patient, speaker or scene look alike. Split by that unit, or the test set rewards memory instead of skill
You build a skin-lesion classifier from 3,000 photos of 400 patients. A random 80/20 split gives 94% test accuracy. In the clinic it gets 71%. What happened?
Photos of one patient share skin tone, lighting, camera and even the same lesion from two angles. A random split puts some of a patient's photos in training and others in test. The model can score well by recognising the patient. The test set measured memory of people it had already seen. In the clinic every patient is new.
The fix is a grouped split: assign whole groups, here patients, to one side. The unit of the split must be the unit you will meet new examples of in deployment.
Worked: f = 0.2, k = 10 → 0.810 + 0.210 = 0.107 + 0.0000001 ≈ 0.11. So about 89% of 10-photo patients end up on both sides of a random split.
Each column is one patient and each dot one sample; filled dots are in the test set. A nearest-neighbour model (pure memory) is scored on the test set and on brand-new patients. Make the patients more alike inside, and compare the splits.
30 patients with 6 to 14 samples each; 16 features. A sample is a class signal plus a personal offset shared by all of that patient's samples, plus noise. Labels depend on the class signal only. The model is 1-nearest-neighbour. "New patients" are 30 more patients drawn the same way. Illustrative.
The rule has many names. In speech, split by speaker. In vision, split by scene, video or photographer, because frames of one video are near-copies. In recommendation, split by user. In time series, split by time: train on the past, test on the future, with a gap so that windows do not overlap. And always remove exact and near duplicates across splits first. Audio Data Pipelines shows speaker-disjoint splits and deduplication on real audio collections.
PyTorch's random_split splits samples, not groups, so it is the wrong tool here. Split the indices yourself by group, then wrap them with Subset(ds, train_idx) and Subset(ds, test_idx). scikit-learn's GroupShuffleSplit and GroupKFold do the same for index arrays.
Chapter 10 Where else
When the data has no index, stream it with an IterableDataset and give each worker its own shards. When the expensive part never changes, compute it once and cache it
A map-style dataset needs random access: "give me sample 7,203". Some data does not allow that. A web crawl lives in thousands of compressed files read from start to end. A log arrives as a live stream. For these, PyTorch has the IterableDataset: you write __iter__, and it yields samples one after another.
Workers bring a trap. Each worker gets its own copy of the dataset and calls __iter__ on it. Without care, all W workers read the same stream from the start, and every sample arrives W times. The fix is sharding: each worker reads only its own part. Inside a worker, torch.utils.data.get_worker_info() tells the worker its id and the number of workers; in the main process it returns None.
get_worker_info().id.Worked: 64 shards, 2 GPUs × 4 workers = 8 readers → 8 shards each. With 10 shards and 8 readers, two readers get 2 shards and six get 1: the run waits on the slowest. Use a shard count that is a multiple of R·W.
8 shard files of 6 samples each; each sample shows its shard number and gets darker with it. Turn sharding off to see the duplicates. Then grow the shuffle buffer and watch the shards mix.
Workers take turns producing one sample each into the output stream (real loaders hand over whole batches). The shuffle buffer keeps K samples and emits a random one each time a new sample arrives. Illustrative.
import torch
from torch.utils.data import IterableDataset, DataLoader, get_worker_info
class ShardStream(IterableDataset):
def __init__(self, shard_paths, buffer=1000, seed=0):
self.shards, self.buffer, self.seed = shard_paths, buffer, seed
def __iter__(self):
info = get_worker_info() # None in the main process
wid, nw = (info.id, info.num_workers) if info else (0, 1)
mine = self.shards[wid::nw] # this worker's shards only
g = torch.Generator().manual_seed(self.seed + wid)
buf = []
for path in mine:
for sample in read_records(path): # your file reader
buf.append(sample)
if len(buf) >= self.buffer: # emit a random buffered sample
yield buf.pop(int(torch.randint(len(buf), (1,), generator=g)))
while buf: # drain at the end
yield buf.pop(int(torch.randint(len(buf), (1,), generator=g)))
loader = DataLoader(ShardStream(paths), batch_size=64, num_workers=4) # no shuffle= here
An IterableDataset cannot use shuffle or a sampler; the DataLoader raises an error if you try. Shuffling happens in two places instead: shuffle the shard order each epoch, and keep a shuffle buffer as above. A buffer of K samples can only move a sample about K places, so a small buffer gives a nearly ordered stream.
The cheapest sample is the one you do not compute. If your pipeline runs a frozen pretrained backbone on every sample, its output is the same every epoch. Run it once, save the features, and make your Dataset read the saved features instead. Backbone + Head derives the speed-up: often 20 to 200 times for head training.
import numpy as np
import torch
from torch.utils.data import Dataset
class CachedFeatures(Dataset):
"""Features computed once by a frozen backbone, saved as one (N, D) float16 file."""
def __init__(self, path, labels):
self.feats = np.load(path, mmap_mode="r") # memory-mapped: read pages on demand
self.labels = labels
def __len__(self):
return len(self.labels)
def __getitem__(self, i):
x = torch.from_numpy(np.array(self.feats[i], dtype=np.float32)) # (D,)
return x, self.labels[i]
A cache has two costs. Augmentation dies: the cached sample is the same every epoch, unless you cache several augmented copies. And the cache goes stale: re-create it whenever the backbone, its weights or its preprocessing change. Name the cache file after all three.
Chapter 11
Match a symptom to its cause, keep the one-screen cheat sheet, know the limits, and go on to the training loop
Most data-pipeline bugs show up as one of a few symptoms. Pick one below to see the usual causes in order, the check that tells them apart, and the fix.
Pick the symptom you see. The card lists the causes from most to least common, the one check to run first, and the fix.
A field guide from the chapters of this lesson, not an exhaustive list.
| Part | In one line |
|---|---|
| Dataset | __len__ + __getitem__(i); store paths, load lazily; depend only on i. |
| DataLoader chain | sampler → batch sampler → fetch [ds[i] for i in idx] → collate_fn. |
| default_collate | Tensors stack to B × S; ints → int64; floats → float64; strings → list; dicts per key; tuples transposed. |
| Variable length | pad_sequence(xs, batch_first=True) + mask arange(T)[None] < lengths[:, None]; mask every mean, attention and loss. |
| Padding waste | 1 − ΣL / Σ(B · max L); bucket: shuffle, pool, sort, cut, shuffle batches. |
| Samplers | WeightedRandomSampler with w = 1/nclass; DistributedSampler + set_epoch(epoch). |
| Seeds | generator=torch.Generator().manual_seed(s); worker seed = base_seed + worker_id; no RNG objects made in __init__. |
| Throughput | B · min(W / (B ts), 1 / tg); W* = ⌈B ts / tg⌉; prefetch 2W batches; pin_memory=True + non_blocking=True. |
| Transforms | Augment train only; fit μ, σ and every other statistic on train only; save them with the model. |
| Splits | Split by patient, speaker, scene, user or time; dedupe first; Subset with your own indices. |
| Streams and caches | IterableDataset + get_worker_info() sharding + shuffle buffer; cache frozen-backbone features, and re-cache on change. |
__getitem__ that collate to float64 and promote the loss.set_epoch missing on a DistributedSampler.__init__, so every worker draws the same augmentations.drop_last=False with BatchNorm and a final batch of 1 sample.num_workers, pin_memory and worker_init_fn are accepted but do nothing. The worker and memory chapters are taught with simulators built on PyTorch's rules.Now press Present or Teach and explain, out loud and from memory, why a GPU waits for data and how many workers it needs. Then go back to the hero, set the load time to 4 ms and find the fewest workers that close every red gap.
torch.utils.data (torch.utils.data docs; dataloader.py, sampler.py, distributed.py, _utils/collate.py, _utils/worker.py) and torch.nn.utils.rnn.pad_sequence; every default, error message and collate rule quoted here was checked by running PyTorch 2.10. ImageNet's training-set size (1,281,167 images) is from O. Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge" (IJCV 2015, arXiv). Timing numbers on devices are illustrative and labelled so.