What if a network defined a continuous-time field instead of a fixed layer stack? Consistently scaled residual updates can approximate its flow; an ODE solver computes the endpoint.
A residual block adds a learned correction: the new hidden state is the old one plus whatever the block computes. Stack 50 of these and the data undergoes 50 discrete updates. Training aims to make the final representation useful; an individual update need not be small or improve the task.
Now ask a more precise question. Suppose every residual update is scaled by a smaller time step and its learned rule changes smoothly across depth. As we add steps while holding a fixed integration interval, that family of discretizations can approach a continuous hidden-state trajectory. A finite, independently parameterized ResNet does not automatically have this limit. In a Neural ODE (Chen et al., 2018), a neural network specifies the derivative dh/dt = f(h,t;θ), and an ODE solver computes the endpoint from an initial state.
For this specified toy ODE, the orange path uses n Euler updates over the same interval; as n grows, it approaches the teal exact curve. This demonstrates the limiting idea, not a guarantee for every trained ResNet.
Here is the bridge that started it all. A residual update looks like this: next hidden state equals current state plus the layer’s function of the current state. Now write the simplest way to solve a differential equation numerically — Euler’s method: the next value equals the current value plus a small step size times the derivative.
These updates share an Euler form when a residual block represents Δt times a chosen vector field. Writing an unscaled ResNet update as Euler with Δt=1 is an algebraic interpretation of that one step; it does not make an arbitrary finite trained network the accurate solution of one underlying ODE. A Neural ODE instead defines the derivative field explicitly and calls a numerical solver to integrate it.
Let the derivative be f(h) = 0.5·h, start at h = 1.0, and integrate from t=0 to t=1. Take it in two Euler steps (step size 0.5):
| step | h | f(h)=0.5h | h + 0.5·f(h) |
|---|---|---|---|
| 0→0.5 | 1.000 | 0.500 | 1.000 + 0.5·0.5 = 1.250 |
| 0.5→1.0 | 1.250 | 0.625 | 1.250 + 0.5·0.625 = 1.563 |
Two steps give 1.563. The true answer (this ODE has solution h(t)=e0.5t) is e0.5 = 1.649. We’re close but low — Euler with big steps undershoots curvature. With 10 equal Euler steps the endpoint is about 1.629. With 1000 it is 1.648515, close to e0.5 ≈ 1.648721 but still not exact. For this smooth scalar equation, shrinking the step over the same interval reduces Euler error; adding arbitrary ResNet layers does not by itself give that guarantee.
The teal curve is the exact solution; the orange path is Euler’s method with your chosen step size. Here big Euler steps depart from this exact scalar solution; smaller steps reduce this toy solver error. The plot does not measure a ResNet classifier’s error.
If f is the derivative of the hidden state, then f is a vector field: at every point in the state space, it tells you which direction to move and how fast. The hidden state is a particle dropped into this field, drifting along wherever the arrows point. Running the network = letting the particle flow for a fixed amount of time.
This is a genuinely different mental picture from stacked layers. The network is the field. The same field is applied continuously the whole way through (in the simplest version, f uses one shared set of weights for all time) — what changes is where the particle has drifted to. Learning means shaping the field so that particles starting at different inputs flow to the right outputs.
Arrows show one of three hand-coded toy fields. Click to draw a 120-step Euler path from that point; the path appears at once. Switch fields to compare trajectories. No model is trained in this widget.
A Neural ODE specifies a derivative field and an integration interval, then uses a numerical solver to compute the endpoint. A solver can be fixed-step or adaptive. An adaptive method estimates local numerical error and accepts, rejects, or resizes trial steps to meet the selected relative and absolute tolerances. Smaller steps may be needed where the numerical problem demands them; the rule does not identify what is semantically difficult for a classifier.
NFE means the number of calls to the derivative function f, including calls in rejected trials. It is not the count of accepted steps: a Runge–Kutta step may call f several times. With an adaptive solver, the dynamics and numerical tolerances influence NFE; a fixed-step solver follows its chosen schedule. Tightening tolerance usually costs more evaluations and reduces estimated numerical integration error, but does not promise a better classifier.
This widget runs an error-controlled toy integrator. Dots mark accepted steps, while the NFE readout counts every derivative call, including rejected trials. Tightening the tolerance targets local numerical error; it does not test classifier accuracy.
With an adaptive solver, two initial states can require different numbers of derivative evaluations to satisfy the same numerical tolerance. One trajectory may be inexpensive to integrate while another needs smaller trial steps or more rejected trials. That is input-dependent computational work, but it is neither free nor a reliable measure of which example is hard to classify.
An ordinary 50-block ResNet evaluates the same scheduled blocks for each input. An adaptively solved Neural ODE can vary its derivative-call count with the trajectory and tolerance. Other networks can also use conditional execution or early exits. In a batched solve, a shared error controller may couple the work across examples.
Two particles start in one hand-coded field. Both particles use the same error-controlled solver, and the readout counts its derivative calls. A higher count means this toy path was numerically costlier at the chosen tolerance, not that it was harder to classify.
Training needs gradients. The naive approach: record every operation the solver performed and backpropagate through all of them. But an adaptive solver might take hundreds of steps — storing every intermediate state is as memory-hungry as a hundred-layer network. One way to avoid storing that trajectory is the adjoint sensitivity method.
The idea: solve an augmented ODE backward from the output. It carries the current state, the loss sensitivity (the “adjoint”), and parameter-gradient information. It can reconstruct the forward path instead of caching every solver step. In this formulation, trajectory storage is constant with respect to the number of solver steps; it still depends on state size, parameters, batch, and solver workspace.
Schematic memory comparison: storing each solver step grows with step count (orange); a reverse adjoint solve keeps a current augmented state and bounded solver workspace (teal). Drag the step count; this is not total model memory.
Here’s where Neural ODEs become a generative tool. Suppose you flow not a single point but a whole cloud of points — a probability distribution. As the cloud flows along the field, it stretches and compresses. If you start from a simple distribution (a Gaussian blob) and flow it through a learned field, you can morph it into a complex data distribution. This is a continuous normalizing flow (CNF).
For a regular enough field with unique solutions, the instantaneous change-of-variables rule says log density changes at the negative divergence of the field along a trajectory. Integrating that rule lets a trained flow evaluate model likelihoods and generate samples. The rule is exact mathematically; numerical solves and any stochastic trace estimate introduce approximation. Reverse integration can also accumulate error.
This fixed example solves x-dot = 0.9 sin(2y), y-dot = 0 exactly: x(t) = x(0) + 0.9t sin(2y(0)). Drag time to see a Gaussian shear. It is an invertible ODE flow, not a trained target distribution or a likelihood measurement.
Train a small, shared neural derivative field and a linear logistic head on 240 labeled two-moon points. The 120 colored points below are a separate, fixed test set. Their labels are used only for color and held-out scoring, never to choose an individual trajectory or prediction. The plotted path uses a finite Euler approximation to the learned field.
Press Train once, then Flow or drag time in either direction. The dashed line is the trained head's actual boundary. Changing Euler steps keeps the learned weights fixed and changes only the numerical approximation.
The readout counts actual derivative calls on the frozen test set and reports its classification score. More fixed Euler steps usually refine this same approximate flow; they do not guarantee a better label. This is not an adaptive tolerance control, and a finite Euler plot does not inherit every uniqueness or noncrossing property of a sufficiently regular exact ODE flow.
Neural ODEs are elegant but not free. Knowing their costs tells you when to reach for them.
Concept sketch, not measured data: an adaptive solver often spends more derivative evaluations for a tighter local-error tolerance, but NFE is not guaranteed monotonic and task accuracy need not improve. The teal line depicts numerical precision only.
| Model | Depth | Typical training cache | Adaptive ODE solver? |
|---|---|---|---|
| ResNet | fixed layers | O(depth) | no |
| Neural ODE | continuous (numerically solved) | step-independent with adjoint* | if adaptive solver |
| RNN | fixed per step, discrete time | O(time) | no |
| Latent ODE | continuous latent time | depends on training method* | if adaptive solver |
*Model state, parameters, batch, observations, and solver workspace still use memory.
→ Flow Matching — a related way to train continuous velocity fields
→ Diffusion — the cousin that moves noise to data over time
→ Skip Connections — the ResNet that started the analogy
→ SSM & Mamba — another continuous-time dynamical view of sequences