What if a network had not 50 layers, but infinitely many infinitely thin ones? A residual block is one step of solving a differential equation — take the step size to zero and depth becomes a continuous flow you hand to an ODE solver.
A ResNet works by adding a small correction at each layer: the new hidden state is the old one plus whatever the layer computes. Stack 50 of these and the data takes 50 little steps from input to output. Each step nudges the representation a bit closer to something useful.
Now ask a strange question. What if instead of 50 steps we took 500 tiny ones? Or 5,000 even tinier ones? In the limit — infinitely many, infinitely small steps — the discrete staircase of layers becomes a smooth curve. The hidden state stops jumping from layer to layer and instead flows continuously. That continuous flow is described by a differential equation, and a network defined this way is a Neural ODE (Chen et al., 2018).
The orange staircase is a ResNet: each step is one layer’s update. Add more layers (smaller steps) and the staircase converges to the smooth teal curve — the solution of a differential equation.
Here is the bridge that started it all. A residual update looks like this: next hidden state equals current state plus the layer’s function of the current state. Now write the simplest way to solve a differential equation numerically — Euler’s method: the next value equals the current value plus a small step size times the derivative.
They are the same equation. A ResNet is Euler’s method with the step size baked in as 1, where f — the residual block — plays the role of the derivative of the hidden state. So a ResNet isn’t just like solving an ODE; it literally is a crude ODE solve, taking one clumsy step per layer.
Let the derivative be f(h) = 0.5·h, start at h = 1.0, and integrate from t=0 to t=1. Take it in two Euler steps (step size 0.5):
| step | h | f(h)=0.5h | h + 0.5·f(h) |
|---|---|---|---|
| 0→0.5 | 1.000 | 0.500 | 1.000 + 0.5·0.5 = 1.250 |
| 0.5→1.0 | 1.250 | 0.625 | 1.250 + 0.5·0.625 = 1.563 |
Two steps give 1.563. The true answer (this ODE has solution h(t)=e0.5t) is e0.5 = 1.649. We’re close but low — Euler with big steps undershoots curvature. Take 10 steps and you’d get 1.628; take 1000 and you’d nail 1.649. More layers = smaller step = less error. The Neural ODE just takes this to the continuous limit and lets a smart solver pick the steps.
The teal curve is the exact solution; the orange path is Euler’s method with your chosen step size. Big steps drift off the curve (that’s a shallow ResNet’s approximation error); small steps hug it.
If f is the derivative of the hidden state, then f is a vector field: at every point in the state space, it tells you which direction to move and how fast. The hidden state is a particle dropped into this field, drifting along wherever the arrows point. Running the network = letting the particle flow for a fixed amount of time.
This is a genuinely different mental picture from stacked layers. The network is the field. The same field is applied continuously the whole way through (in the simplest version, f uses one shared set of weights for all time) — what changes is where the particle has drifted to. Learning means shaping the field so that particles starting at different inputs flow to the right outputs.
Arrows show the learned velocity field f. Click anywhere to drop a particle — watch it flow along the field. That path is the hidden state moving through “depth.” Change the field to see different flows.
In a normal network, you decide the number of layers. In a Neural ODE, you hand the differential equation to a black-box ODE solver and it decides how many steps to take. Modern solvers (like Dormand–Prince) are adaptive: they estimate their own error at each step and shrink the step where the field changes fast, grow it where the field is calm — spending computation exactly where it’s needed.
The count of times the solver calls f is the number of function evaluations, or NFE — the continuous analogue of “how many layers.” You don’t set it; it emerges from the dynamics and your chosen error tolerance. Loosen the tolerance and the solver takes big, cheap, sloppy steps; tighten it and it takes many small, accurate ones.
The solver places steps (dots) along the trajectory — denser where the field bends sharply. Tighten the tolerance and watch the solver add evaluations to stay accurate; loosen it and it coasts on a few big steps.
Because the solver chooses its own steps, a Neural ODE spends more compute on harder inputs — for free. An input whose trajectory passes through a calm region of the field resolves in a few steps; one that threads a sharply curving region forces the solver to take many small steps to stay accurate. The model literally thinks longer about hard examples.
This is a property fixed-depth networks simply don’t have: a 50-layer ResNet does exactly 50 layers of work whether the input is trivial or brutal. The Neural ODE’s NFE rises and falls with difficulty, an early and elegant form of adaptive computation.
Two particles start in different regions. The one in the calm zone needs few steps (low NFE); the one crossing the turbulent zone forces many. Same model, different compute — automatically.
Training needs gradients. The naive approach: record every operation the solver performed and backpropagate through all of them. But an adaptive solver might take hundreds of steps — storing every intermediate state is as memory-hungry as a hundred-layer network. The Neural ODE’s signature trick avoids this entirely: the adjoint sensitivity method.
The idea: to get gradients, solve a second ODE backward in time, from the output back to the input. This backward ODE tracks how sensitive the loss is to the state at each moment (the “adjoint”). Crucially, you don’t need to have stored the forward trajectory — you reconstruct what you need on the fly during the backward solve. Memory cost becomes constant in depth: O(1), whether the solver took 10 steps or 10,000.
Top: backprop-through-solver stores every step (memory grows with depth, orange bars pile up). Bottom: the adjoint solves backward, holding only the current state (flat teal memory). Drag the step count and watch only the orange bars grow.
Here’s where Neural ODEs become a generative tool. Suppose you flow not a single point but a whole cloud of points — a probability distribution. As the cloud flows along the field, it stretches and compresses. If you start from a simple distribution (a Gaussian blob) and flow it through a learned field, you can morph it into a complex data distribution. This is a continuous normalizing flow (CNF).
The magic is that you can track exactly how the probability density changes as it flows — the rate of change of (log) density is governed by how much the field is locally expanding or contracting (its divergence). So you get a generative model with exact likelihoods: sample a Gaussian, flow it forward to generate data; flow data backward to score its probability. No fixed architecture of invertible layers needed — the ODE is invertible by running it in reverse.
A cloud of points starts as a simple blob (left) and flows along the field, stretching into a target shape (right). Drag the flow time to watch the distribution transform continuously — that’s a continuous normalizing flow generating data.
Let’s watch a Neural ODE untangle two interleaved classes. The points start mixed; the learned field flows them apart over integration time until a straight line can separate them. You control how long to flow, the field strength, and the solver tolerance — and the readout shows the function evaluations (NFE) the solver needed.
Two classes (orange, teal) start interleaved. Press Flow to integrate the field forward; watch the classes separate as integration time grows. Increase tolerance for a coarse, cheap solve; decrease it for a fine, expensive one. The readout tracks NFE and separation quality.
Notice the trade-off you control at inference: a looser tolerance gives a few-step, slightly jagged flow that still separates the classes; a tight tolerance gives a silky-smooth flow at many times the NFE. The model didn’t change — only how carefully you solved it. That run-time accuracy dial is the Neural ODE’s signature gift.
Neural ODEs are elegant but not free. Knowing their costs tells you when to reach for them.
For one trained model, tightening tolerance trades NFE (compute, orange) for solution accuracy (teal). A fixed-depth net is a single point; the Neural ODE is the whole curve — you choose where to sit, after training.
| Model | Depth | Memory in depth | Adaptive compute? |
|---|---|---|---|
| ResNet | fixed layers | O(depth) | no |
| Neural ODE | continuous (solver picks steps) | O(1) via adjoint | yes |
| RNN | fixed per step, discrete time | O(time) | no |
| Latent ODE | continuous time | O(1) | yes (irregular series) |
→ Flow Matching — the cheap way to train these continuous flows
→ Diffusion — the cousin that moves noise to data over time
→ Skip Connections — the ResNet that started the analogy
→ SSM & Mamba — another continuous-time dynamical view of sequences