PyTorch Tensors &
Computation

Chapter 1: Tensor Basics — Shape, Dtype, Device

A tensor is a multi-dimensional array with three essential properties: its shape (how many elements along each dimension), its dtype (how each element is stored in memory), and its device (which hardware holds the data). Every operation in PyTorch flows through these three facts.

Let's create our first tensor and interrogate its properties:

python
import torch

# Create a 2x3 tensor of zeros
x = torch.zeros(2, 3)
print(x.shape)   # torch.Size([2, 3])
print(x.dtype)   # torch.float32
print(x.device)  # cpu

# Memory usage: 2 * 3 * 4 bytes = 24 bytes
print(x.element_size())  # 4 (bytes per float32)
print(x.nelement())      # 6 (total elements)
print(x.element_size() * x.nelement())  # 24 bytes

Let's break this down. A shape of [2, 3] means 2 rows and 3 columns — 6 elements total. Each element is float32 (the default), which takes 4 bytes. So the entire tensor occupies exactly 2 × 3 × 4 = 24 bytes of contiguous memory.

Here are the most common creation functions:

python
# Common creation functions
a = torch.zeros(3, 4)           # all zeros
b = torch.ones(3, 4)            # all ones
c = torch.randn(3, 4)           # normal distribution N(0,1)
d = torch.rand(3, 4)            # uniform [0, 1)
e = torch.arange(0, 12)         # [0, 1, 2, ..., 11]
f = torch.tensor([[1,2],[3,4]]) # from Python list
g = torch.empty(3, 4)           # uninitialized (garbage values!)

# From numpy (SHARES memory by default!)
import numpy as np
arr = np.array([1.0, 2.0, 3.0])
t = torch.from_numpy(arr)  # shared memory: modify one, both change

Worked example: how much memory does a batch of 32 RGB images at 224×224 resolution use?

Chapter 2: Memory Layout — Strides Explain Everything

Your computer's memory is one-dimensional — a long tape of bytes at consecutive addresses. But tensors are multi-dimensional. How does a 2D matrix live in a 1D tape? The answer is strides.

A stride tells you how many elements to skip in memory to move one step along each dimension. For a [3, 4] tensor stored row-by-row (the default, called row-major or C-contiguous order), the strides are (4, 1). That means: to move to the next row, skip 4 elements. To move to the next column, skip 1 element.

python
import torch

x = torch.arange(12).reshape(3, 4)
print(x)
# tensor([[ 0,  1,  2,  3],
#         [ 4,  5,  6,  7],
#         [ 8,  9, 10, 11]])

print(x.stride())  # (4, 1)
# stride[0]=4: jump 4 elements to go from row i to row i+1
# stride[1]=1: jump 1 element to go from col j to col j+1

# Element at [i, j] lives at memory offset: i*stride[0] + j*stride[1]
# x[2, 1] is at offset 2*4 + 1*1 = 9 --> value is 9 ✓

Column-major (Fortran-contiguous) is the opposite: elements in the same column are contiguous. For a [3, 4] tensor stored column-major, strides would be (1, 3) — jump 1 to move down a row, jump 3 to move right a column. PyTorch defaults to row-major. NumPy lets you choose.

Here's the key insight: transposing a tensor just swaps the strides. No data is copied.

python
x = torch.arange(12).reshape(3, 4)
print(x.stride())    # (4, 1) -- row-major

y = x.t()            # transpose
print(y.shape)       # torch.Size([4, 3])
print(y.stride())    # (1, 4) -- strides swapped!
print(y.is_contiguous())  # False

# Same underlying data, different view
print(x.data_ptr() == y.data_ptr())  # True -- same memory!

Chapter 3: Views vs Copies — Shared Memory Traps

When you reshape, transpose, or slice a tensor in PyTorch, does it copy the data? Usually no. Most operations create a view — a new tensor object that points to the SAME underlying memory with different metadata (shape, strides, offset). This is fast (O(1) instead of O(n)) but dangerous: modifying the view modifies the original.

Operations that create views (shared memory):

python
x = torch.arange(12).reshape(3, 4)

# ALL of these share memory with x:
a = x.view(6, 2)       # reshape (must be contiguous)
b = x.reshape(6, 2)    # reshape (may copy if non-contiguous)
c = x.t()              # transpose
d = x[0]               # indexing a row
e = x[:, 1]           # indexing a column
f = x.expand(5,3,4)   # broadcast expand (stride=0!)
g = x.permute(1,0)    # dimension reorder
h = x.unsqueeze(0)    # add dimension

# Proof: modify the view, original changes
a[0, 0] = 999
print(x[0, 0])  # 999 -- x changed too!

Operations that create copies (independent memory):

python
# These allocate NEW memory:
a = x.clone()          # explicit deep copy
b = x.contiguous()     # copy only if non-contiguous
c = x + 0              # any arithmetic creates new tensor
d = x.to(torch.float16)  # dtype cast = new allocation
e = torch.tensor(x)   # explicit new tensor from data

The key question: when is a tensor non-contiguous? After any operation that changes strides without moving data:

python
x = torch.arange(12).reshape(3, 4)
print(x.is_contiguous())  # True -- strides (4,1), data is sequential

y = x.t()  # transpose: shape [4,3], strides (1,4)
print(y.is_contiguous())  # False!

# Why? For y to be contiguous with shape [4,3], strides must be (3,1)
# But y has strides (1,4) -- elements aren't sequential in memory

# Fix it:
z = y.contiguous()  # copies data into new sequential memory
print(z.stride())   # (3, 1) -- now contiguous
print(z.data_ptr() == x.data_ptr())  # False -- different memory

Chapter 4: Broadcasting — Implicit Shape Alignment

You have a [3, 4] matrix and want to add a [4] vector to each row. In raw code, you'd write a loop. But PyTorch does it automatically through broadcasting — the rules for stretching tensors to compatible shapes for element-wise operations.

The broadcasting rules are simple but must be followed exactly:

python
import torch

# Example 1: [3,4] + [4] --> works!
A = torch.ones(3, 4)   # shape [3, 4]
b = torch.arange(4)    # shape [4]
# Step 1: pad b to [1, 4]
# Step 2: dim 0: 3 vs 1 ✓ (1 stretches)  dim 1: 4 vs 4 ✓ (match)
# Result: [3, 4]
result = A + b  # b is added to each row

# Example 2: [3,1] + [1,4] --> [3,4]!
col = torch.tensor([[1],[2],[3]])  # shape [3, 1]
row = torch.tensor([[10,20,30,40]])  # shape [1, 4]
# dim 0: 3 vs 1 ✓   dim 1: 1 vs 4 ✓
result = col + row  # outer sum! shape [3, 4]
# [[11,21,31,41],
#  [12,22,32,42],
#  [13,23,33,43]]

# Example 3: [3,4] + [3] --> ERROR!
# Step 1: pad to [1, 3]
# Step 2: dim 1: 4 vs 3 -- neither is 1! FAIL

Broadcasting is lazy — it never actually copies data. Under the hood, it uses stride=0 for stretched dimensions. A [3, 1] tensor broadcast to [3, 4] has strides (1, 0) — moving along the stretched dimension doesn't advance in memory. Zero memory cost!

python
x = torch.tensor([[1],[2],[3]])  # shape [3,1], strides (1,1)
y = x.expand(3, 4)              # shape [3,4], strides (1,0)!
print(y.stride())  # (1, 0) -- stride 0 means "repeat this element"
print(y)
# tensor([[1, 1, 1, 1],
#         [2, 2, 2, 2],
#         [3, 3, 3, 3]])
# Only 3 elements in memory, but LOOKS like 12!

Chapter 5: Dtypes & Precision — Every Bit Counts

A dtype (data type) determines how each number is encoded in binary. More bits = more precision = more memory. The four dtypes you'll encounter in deep learning are:

The floating-point format is: (-1)^sign × 2^exponent × (1 + mantissa). More exponent bits = bigger range. More mantissa bits = finer granularity between representable numbers.

python
import torch

# Dtype casting
x = torch.randn(3, 3)                # float32 by default
x16 = x.half()                        # float16
xbf = x.bfloat16()                    # bfloat16
x8 = x.to(torch.int8)                 # int8 (truncates!)

# float16 overflow danger:
big = torch.tensor(70000.0)
print(big.half())   # tensor(inf) -- overflow! 70000 > 65504
print(big.bfloat16())  # tensor(69632.) -- representable (lower precision)

# Mixed precision training pattern:
model = model.float()  # master weights in float32
with torch.cuda.amp.autocast():
    # Forward pass in float16 (2x faster on GPU)
    output = model(input.half())
    loss = criterion(output, target)
# Backward in float32 for numerical stability
scaler.scale(loss).backward()

Notice: at amplitude=100, all dtypes look the same. At amplitude=70000, float16 clips to infinity while bfloat16 tracks correctly (with visible stairstepping from low precision). At any amplitude, int8 looks like a staircase because it can only represent 256 distinct values.

Chapter 6: GPU Tensors — The Transfer Bottleneck

Your CPU has RAM (system memory). Your GPU has VRAM (video memory). These are physically separate pools of memory connected by a PCIe bus (or NVLink on high-end systems). Moving data between them is the #1 performance bottleneck in deep learning.

A PCIe 4.0 x16 bus transfers at ~25 GB/s. Sounds fast? An A100 GPU computes matmul at 312 TFLOPS. For a [4096, 4096] float16 matrix (32 MB), transfer takes 1.3 ms but the matmul takes 0.04 ms. The transfer is 30x slower than the computation.

python
import torch

# Move tensor to GPU
x = torch.randn(1000, 1000)
x_gpu = x.to('cuda')        # copies to GPU memory
x_gpu = x.cuda()            # same thing
x_gpu = x.to('cuda:0')     # specific GPU

# Create directly on GPU (no transfer!)
y = torch.randn(1000, 1000, device='cuda')

# Move back to CPU
x_cpu = x_gpu.cpu()
x_cpu = x_gpu.to('cpu')

# DANGER: can't mix devices!
# x + x_gpu  --> RuntimeError: tensors on different devices

Pinned memory is a special CPU memory region that the GPU can access directly via DMA (Direct Memory Access). Normal CPU memory might be swapped to disk, so the driver must first copy it to a pinned staging area. If YOU pin it upfront, the driver skips that step:

python
# Pinned memory for faster transfers
x = torch.randn(1000, 1000).pin_memory()
x_gpu = x.to('cuda', non_blocking=True)  # async transfer!

# DataLoader with pinned memory
loader = DataLoader(dataset, batch_size=32,
                    pin_memory=True,   # allocate batches in pinned RAM
                    num_workers=4)    # parallel data loading

# In training loop:
for batch, labels in loader:
    batch = batch.to('cuda', non_blocking=True)
    labels = labels.to('cuda', non_blocking=True)
    # GPU compute overlaps with next batch's transfer

Chapter 7: Tensor Playground — Build & Inspect

Time to put it all together. This interactive playground lets you create tensors, apply operations, and see exactly what happens to shape, strides, memory, and contiguity at each step. Think of it as a visual debugger for tensor operations.

Experiments to try in the playground:

The playground shows you the exact internal state PyTorch maintains for every tensor: a data pointer, shape tuple, strides tuple, dtype, and device. Every operation is just a transformation of this metadata — or, for copies, allocation of new memory + metadata.

Chapter 8: Sparse & Quantized — Beyond Dense Floats

Dense tensors waste memory when most elements are zero. A [10000, 10000] matrix with only 1% non-zero values stores 100 million floats — but only 1 million matter. Sparse tensors store only the non-zero elements and their coordinates.

PyTorch supports two sparse formats:

python
import torch

# COO format: specify indices and values
indices = torch.tensor([[0, 1, 2],    # row indices
                        [2, 0, 1]])   # col indices
values = torch.tensor([3.0, 4.0, 5.0])
sparse = torch.sparse_coo_tensor(indices, values, (3, 3))

# Dense version would use 9 floats = 36 bytes
# Sparse uses 6 ints (indices) + 3 floats (values) = 36 bytes
# Break-even! Sparse only wins when sparsity > ~67%

# CSR format (better for computation)
crow = torch.tensor([0, 1, 2, 3])   # row pointers
col = torch.tensor([2, 0, 1])      # column indices
vals = torch.tensor([3.0, 4.0, 5.0])
csr = torch.sparse_csr_tensor(crow, col, vals, (3, 3))

Quantized tensors represent floating-point values using integers plus a scale factor. The idea: if your weights range from -1.5 to +1.5, you can map this to int8 (-128 to +127) with scale = 3.0/255:

python
import torch

# Manual quantization
x = torch.randn(4, 4)  # float32 weights
scale = (x.max() - x.min()) / 255
zero_point = int((-x.min() / scale).round())
q = (x / scale + zero_point).round().clamp(0, 255).to(torch.uint8)

# Dequantize
x_approx = (q.float() - zero_point) * scale
print((x - x_approx).abs().max())  # max error ~ scale/2

# PyTorch quantization API
x_q = torch.quantize_per_tensor(x, scale=0.01, zero_point=128,
                                 dtype=torch.quint8)
print(x_q.int_repr())  # the underlying int8 values

Chapter 9: Mastery & Connections

You now understand tensors at the byte level — shape, dtype, device, strides, views, copies, broadcasting, and memory formats. This is the foundation every PyTorch operation is built on. Let's consolidate with reference tables and connections to the broader ML engineering stack.

Cheat Sheet: Creation Functions

Cheat Sheet: Memory Rules

Cheat Sheet: Broadcasting Rules

Where This Goes Next