Vector Calculus

Chapter 1: Derivatives From Scratch

The derivative of f at x measures the instantaneous rate of change. Start from the definition:

This is the slope of the tangent line. Take f(x) = x². Plug in:

So f'(x) = 2x. At x = 3, the slope is 6 — the function is increasing at a rate of 6 units per unit of x. At x = 0, the slope is 0 — a flat spot (the minimum). At x = −2, the slope is −4 — the function is decreasing.

The derivative rules you need for ML:

Two rules compose derivatives of compound expressions:

The chain rule is by far the most important for ML. It says: if y = f(g(x)), then dy/dx = (dy/dg) · (dg/dx). Derivatives of compositions multiply. This innocent-looking rule is the entire foundation of backpropagation.

Chapter 2: Taylor Series

If you know f and all its derivatives at a single point x₀, you can reconstruct the entire function (at least locally). The Taylor series is this reconstruction:

The first two terms give the linear approximation (tangent line). Add the quadratic term and you get the second-order approximation (a parabola that matches the function's curvature). Each term adds finer detail.

Why does ML care about Taylor series? Three reasons:

The factorial denominators (1/n!) are crucial. Without them, each successive term would be far too large and the series would diverge wildly. The factorials tame the growth so the polynomial stays close to the true function near x₀.

Chapter 3: Going Multivariate

So far, f takes one number in and produces one number out. But a neural network loss depends on millions of parameters simultaneously. We need derivatives of functions of many variables.

The partial derivative of f(x₁, x₂, ..., x_n) with respect to x_i is simply: hold all other variables fixed and differentiate with respect to x_i alone.

Example: f(x, y) = x²y + 3y. The partial derivatives are:

At the point (x, y) = (2, 1): ∂f/∂x = 2·2·1 = 4 and ∂f/∂y = 4 + 3 = 7. The function is increasing faster in the y-direction than the x-direction at this point.

The total derivative matters when variables are not independent. If x = x(t) and y = y(t), then by the chain rule:

This is the multivariate chain rule in action: the total rate of change is the sum of each partial derivative times the rate at which that variable changes. This structure — sum of products of local derivatives — is exactly what backpropagation exploits.

Chapter 4: The Gradient

Collect all the partial derivatives into one vector, and you get the gradient:

The gradient has a beautiful geometric meaning: at any point, it points in the direction of steepest ascent. Its magnitude tells you how steep that ascent is. To minimize a function, you walk in the opposite direction of the gradient.

For f(x, y) = x² + 2y², the gradient is ∇f = [2x, 4y]. At the point (3, 2), the gradient is [6, 8]. The steepest ascent direction is (6, 8)/||(6, 8)|| = (0.6, 0.8). To descend, we go the opposite way: (−0.6, −0.8).

The gradient descent update rule is:

where α is the learning rate. Too large and you overshoot. Too small and you crawl. Finding the right α is a central practical challenge in ML optimization.

Chapter 5: The Chain Rule

In a neural network, the loss L depends on the output, which depends on the last layer's weights, which depend on the previous layer's output, and so on. To compute ∂L/∂w for a weight w buried deep in the network, we need to chain together derivatives through every layer in between.

The multivariate chain rule handles this. If f = f(g₁(x), g₂(x), ..., g_m(x)), then:

Worked example: Let f(x) = (x² + 1)³. Define g(x) = x² + 1, so f = g³.

Now a two-layer example closer to neural networks. Let y = σ(w₂ · σ(w₁x + b₁) + b₂), where σ is the sigmoid. To find ∂y/∂w₁:

Notice the product σ'(z₂) · σ'(z₁). Since σ'(z) ≤ 0.25, multiplying many such terms makes the gradient exponentially small. With 50 sigmoid layers: 0.25⁵⁰ ≈ 10⁻³⁰. This is the vanishing gradient problem — and the reason ReLU (whose derivative is 0 or 1) largely replaced sigmoid in deep networks.

Chapter 6: The Jacobian

What if your function has vector input AND vector output? A function f: Rⁿ → R^m maps an n-vector to an m-vector. Its derivative needs to capture how each output changes with each input. That's the Jacobian matrix.

The Jacobian is an m×n matrix. Row i contains the gradient of f_i (the i-th output). Column j shows how all outputs respond to changes in x_j.

Example: f(x₁, x₂) = [x₁² + x₂,   x₁x₂]. The Jacobian is:

At (x₁, x₂) = (3, 2): J = [[6, 1], [2, 3]]. This tells us: a small change δx₁ = 0.01 changes f₁ by 0.06 and f₂ by 0.02. A small change δx₂ = 0.01 changes f₁ by 0.01 and f₂ by 0.03.

The Hessian H is the Jacobian of the gradient: H_ij = ∂²f/∂x_i∂x_j. It's symmetric (by Schwarz's theorem, mixed partials commute). The Hessian tells you about curvature: positive definite H means a local minimum, negative definite means a local maximum, indefinite means a saddle point.

Chapter 7: Gradients of Matrices

In ML, we often need derivatives of scalar-valued functions with respect to matrices. For instance, the loss L(θ) depends on weight matrices W. How do we compute ∂L/∂W?

The key identities you'll use constantly:

Let's derive the most important one: the gradient of the squared loss.

Let L(w) = ||Xw − y||² = (Xw − y)^T(Xw − y). Expand:

Differentiate term by term:

Combining: ∇_wL = 2X^TXw − 2X^Ty = 2X^T(Xw − y).

For the rule ∂tr(AX)/∂X = A^T, the derivation uses the fact that tr(AX) = ∑_i,j A_jiX_ij. Taking ∂/∂X_ij gives A_ji = (A^T)_ij. Stacking all these partial derivatives gives the matrix A^T.

Chapter 8: Backpropagation

Everything we've built leads here. Backpropagation is the algorithm that computes the gradient of a loss function with respect to all parameters in a computational graph. It's just the chain rule, applied systematically from output back to inputs.

The algorithm has two passes:

For L = (wx + b − y)², let's define intermediate variables:

forward pass
z = w * x          # multiply
a = z + b          # add bias
r = a - y          # residual
L = r2             # squared error

The backward pass computes gradients using the chain rule, starting from ∂L/∂L = 1:

backward pass
∂L/∂r = 2r
∂L/∂a = ∂L/∂r · ∂r/∂a = 2r · 1 = 2r
∂L/∂b = ∂L/∂a · ∂a/∂b = 2r · 1 = 2r
∂L/∂z = ∂L/∂a · ∂a/∂z = 2r · 1 = 2r
∂L/∂w = ∂L/∂z · ∂z/∂w = 2r · x = 2(wx+b-y)x

In matrix notation for a full layer with weight matrix W, input x, and ReLU activation:

The upstream gradient ∂L/∂a is passed down from the layer above. Each layer only needs to know its own local derivative and the incoming gradient. This locality is what makes backprop modular — you can swap activation functions, add layers, or change architectures, and the gradient computation "just works."

Chapter 9: Automatic Differentiation

Backpropagation is a specific case of a broader idea: automatic differentiation (autodiff). Autodiff computes exact derivatives of any program by tracing the computation and applying the chain rule mechanically. No approximation, no symbolic manipulation — just exact derivatives, to machine precision.

Reverse-mode autodiff is backpropagation generalized. The "tape" (computation graph) is recorded during the forward pass. The backward pass walks the tape in reverse, accumulating gradients. PyTorch's autograd and JAX's grad implement this.

python
import torch

# Define parameters (requires_grad=True tells PyTorch to track)
w = torch.tensor(1.5, requires_grad=True)
b = torch.tensor(0.5, requires_grad=True)
x = torch.tensor(2.0)
y = torch.tensor(1.0)

# Forward pass (builds computation graph automatically)
L = (w * x + b - y) ** 2

# Backward pass (computes all gradients in one call)
L.backward()

print(w.grad)  # dL/dw = 2(wx+b-y)*x = 2(3.5-1)*2 = 10.0
print(b.grad)  # dL/db = 2(wx+b-y)*1 = 2(3.5-1) = 5.0

The connection between this chapter's ideas and real ML frameworks:

"The chain rule is the most important single idea in differential calculus." — Gilbert Strang