Neural Networks
From Zero

Chapter 1: Modeling Neurons — Biological Inspiration

The name "neural network" comes from the brain. A biological neuron receives electrical signals through its dendrites, integrates them in the cell body, and if the total signal exceeds a threshold, it fires an output pulse down its axon to the next neuron's dendrites.

The mathematical analogy: inputs x_i arrive along "dendrites," each multiplied by a weight w_i (the "synapse strength"). The cell body sums them: w · x + b. If the result is large enough, the activation function "fires" — producing a nonzero output.

This simplification is useful, not accurate. The mathematical model gives us a composable building block: something small and simple that, when wired together in large numbers, produces powerful behavior. That's the real insight — not the biology.

One more historical note: the perceptron, proposed by Frank Rosenblatt in 1958, was the first artificial neuron. It could only learn linearly separable patterns. When Minsky and Papert proved this limitation in 1969, interest in neural networks collapsed for over a decade — the first "AI winter." The fix, as we'll see, was adding hidden layers.

Chapter 2: A Single Neuron — It's a Linear Classifier

Let's formalize the single neuron. It takes n inputs x₁, x₂, ..., x_n, computes a weighted sum, adds a bias, and passes the result through an activation function σ:

If we use a sigmoid activation, the output is between 0 and 1. That's a probability. A single neuron with a sigmoid is literally logistic regression — binary classification with a linear decision boundary. If we use a threshold activation (output 1 if sum > 0, else 0), that's the original perceptron.

Notice: no matter how you adjust the sliders, the boundary is always a straight line. That's the limit of one neuron. To get curves, we need layers of neurons.

Chapter 3: Activation Functions — Sigmoid, Tanh, ReLU

The activation function is what makes a neural network nonlinear. Without it, stacking layers would be pointless: a linear function of a linear function is still linear. W₂(W₁x) = (W₂W₁)x = W'x. You'd just have a single linear layer with extra steps.

Sigmoid: σ(z) = 1 / (1 + e^−z). Squashes any input to the range (0, 1). Historically popular because it looks like a neuron "firing rate." But it has serious problems: gradients vanish when inputs are very large or very small (the curve is flat), and the output is not zero-centered.

Tanh: tanh(z) = 2σ(2z) − 1. Squashes input to (−1, 1). Zero-centered, which is better than sigmoid. But it still saturates — gradients vanish for large |z|.

ReLU: f(z) = max(0, z). Dead simple: output z if positive, zero if negative. Doesn't saturate for positive inputs. Computationally trivial. This one function revolutionized deep learning. Proposed for neural networks by Nair and Hinton in 2010, it made training deep networks drastically faster.

Chapter 4: ReLU and Friends — The Dead Neuron Problem

ReLU has one nasty failure mode. If a neuron's pre-activation is always negative (say, because a large gradient update pushed the weights too far), the output is permanently zero. Zero output means zero gradient flowing back. The neuron will never update again. It's dead.

Leaky ReLU: f(z) = z if z > 0, else αz (typically α = 0.01). Instead of outputting zero for negative inputs, it outputs a small negative slope. Dead neurons can't happen because there's always a nonzero gradient.

ELU (Exponential Linear Unit): f(z) = z if z > 0, else α(e^z − 1). Smoother than Leaky ReLU. Outputs approach −α for large negative inputs, which pushes the mean activation closer to zero.

GELU (Gaussian Error Linear Unit): f(z) = z · Φ(z), where Φ is the standard Gaussian CDF. Used in BERT and GPT. It's smooth, and unlike ReLU, it doesn't have a hard kink at zero — it gradually "turns on." Think of it as a soft gate: inputs that are very negative get zeroed out, very positive pass through, and inputs near zero get a smooth blend.

Chapter 5: Layers — Stacking Neurons Together

A single neuron draws one boundary. To draw complex boundaries, we wire neurons together in layers. The most common arrangement is the fully-connected layer (also called a dense layer): every neuron in one layer connects to every neuron in the next.

For a 2-layer network, the forward pass looks like this:

Where f is an activation function (like ReLU), W₁ maps inputs to hidden units, W₂ maps hidden units to output scores, and h is the hidden representation — the network's internal encoding of the input.

For a 3-layer network, we add another layer:

python
import numpy as np

def forward(x, W1, b1, W2, b2):
    # 2-layer neural network forward pass
    h = np.maximum(0, W1 @ x + b1)  # ReLU activation
    scores = W2 @ h + b2              # output layer (no activation)
    return scores

# Example: 2D input, 4 hidden neurons, 3 classes
W1 = np.random.randn(4, 2) * 0.01
b1 = np.zeros((4, 1))
W2 = np.random.randn(3, 4) * 0.01
b2 = np.zeros((3, 1))

x = np.array([[0.5], [-0.3]])
scores = forward(x, W1, b1, W2, b2)

Chapter 6: Representational Power — Universal Approximation

How powerful are neural networks? A stunning mathematical result tells us: very. The Universal Approximation Theorem (Cybenko, 1989; Hornik, 1991) proves that a neural network with just one hidden layer and enough neurons can approximate any continuous function to any desired accuracy.

This sounds like "just use one hidden layer with lots of neurons and you're done." But there's a catch. The theorem says a solution exists — it doesn't say gradient descent will find it. A very wide, shallow network might need an astronomically large number of neurons, while a deeper network can represent the same function with exponentially fewer parameters.

Think of it like building with LEGO. A one-layer-deep structure can theoretically build anything, but you'd need millions of pieces all laid flat. With depth, you can stack and reuse pieces, building the same thing far more efficiently. Deep networks compose features hierarchically: edges → textures → parts → objects.

Chapter 7: Playground — Neural Network Classifier

Time to see everything in action. Below is a 2D classification task. You control the network architecture — how many hidden layers, how many neurons per layer, and which activation function to use. Hit "Train" and watch the decision boundary evolve in real time as the network learns.

Notice how a linear classifier (0 hidden layers) can only draw a straight line — it fails on circles and spirals. Adding just one hidden layer with a few ReLU neurons lets the network bend the boundary. More layers and neurons let it carve out increasingly intricate regions.

Chapter 8: How Many Neurons? — Capacity and Overfitting

With more neurons, the network can represent more complex functions. So why not use a massive network for every problem? Because of overfitting: a network with too much capacity memorizes the training data — including its noise — instead of learning the underlying pattern.

The cs231n notes make a counterintuitive recommendation: don't reduce overfitting by using fewer neurons. Smaller networks have fewer local minima, but those minima tend to be worse. Larger networks have many more local minima, but they tend to be equivalent and have lower loss. The solutions found by large networks are more varied, but on average they're better.

Chapter 9: Beyond — From Architecture to Training

You now know how to set up a neural network: choose the number of layers, the number of neurons per layer, and the activation function. You understand why ReLU is the default, why depth matters, and why regularization beats architecture shrinkage for controlling overfitting.

But architecture is only half the story. A beautifully designed network with random weights is useless. The next step is training: using gradient descent and backpropagation to find good weights. That's where the real magic happens.

Related lessons:

• Linear Classification — the foundation this lesson builds on
• Optimization & Backpropagation — how neural networks learn
• GPT — neural networks scaled to billions of parameters
• Transformer — the architecture that powers modern AI