Kolmogorov–Arnold Networks — flip the MLP inside out. Replace each scalar edge weight with a learnable 1-D function, and let the nodes just add. Plotting those edge functions can make a small, sparse KAN easier to inspect, although readability is not automatic.
Train an MLP and you get millions of numbers — the weights. Each one is a multiplier on a connection. A single weight usually has little standalone meaning; behavior depends on patterns across many parameters at once. The network can work well, but a single weight is not a readable explanation. For discovering a compact physical law, that distributed representation can make interpretation difficult.
KANs (Liu et al., 2024) ask: what if a network’s learnable parts were functions you could look at, not opaque scalars? What if you could plot a connection’s learned curve and test whether it resembles a sine or a square? That is the interpretability interface KANs offer — and it begins with one structural change to the MLP, which the next chapters build up carefully.
Left: an MLP edge carries a learned scalar. Right: a KAN edge carries a learned 1-D function that you can plot. Toggle to compare the two inspection surfaces; a plotted curve still needs interpretation and validation.
To appreciate the flip, pin down exactly what an MLP neuron does. It takes each incoming value, multiplies it by a learnable weight (the weight lives on the edge), sums all those products, adds a bias, and then passes the sum through a fixed activation function — ReLU, say — that lives on the node.
Two things to notice. The learnable parts are the weights, and they’re on the edges. The nonlinearity is fixed (you chose ReLU; the network can’t change its shape), and it’s on the node. This split — learnable linear weights on edges, fixed nonlinearity on nodes — is the DNA of every MLP. Its theoretical license is the universal approximation theorem: with suitable activations, enough hidden units can approximate any continuous function on a compact domain arbitrarily well.
Inputs scaled by edge weights (drag them), summed at the node, then squashed by a fixed activation. The weights learn; the activation’s shape never does.
Here is the core KAN move. Replace each scalar weight with a learnable 1-D function on the edge, and make every node sum its incoming transformed values — with no separate fixed activation at that node. The nonlinearity is learned as part of each edge function.
| edges | nodes | |
|---|---|---|
| MLP | learnable numbers (weights) | fixed nonlinearity (ReLU) |
| KAN | learnable functions (splines) | just sum |
So where an MLP edge does “multiply by 0.7,” a KAN edge does “apply this whole curve to the input.” The learned curve can approximate shapes such as a sine or parabola, subject to the chosen spline basis and grid resolution; a smooth finite-grid spline can only approximate a discontinuous step. The node downstream simply adds up the curve-outputs.
KANs are inspired by the Kolmogorov–Arnold representation theorem. For a continuous function f on the unit cube [0,1]n, the theorem gives an exact two-layer form with 2n+1 outer terms: sums of continuous univariate functions applied to sums of continuous univariate functions. That form is a [n, 2n+1, 1] KAN. The practical architecture generalizes it to arbitrary widths and depths, but the theorem does not itself guarantee that the required edge functions are smooth or easy to learn.
Same two-input, one-output unit, drawn both ways. Toggle: in the MLP the edges are weights and the node squashes; in the KAN the edges are curves and the node only sums.
“A learnable 1-D function” sounds vague — how do you store and train a curve? The original KAN implementation parameterizes each edge with a B-spline plus a residual base function. On a fixed grid, the spline is a linear combination of local B-spline basis functions, and gradient descent learns their coefficients as well as two edge-wise scale factors by default.
Each bump is local — it only affects a small piece of the curve — so raising one coefficient bends the curve in just that region, leaving the rest alone. That locality is what makes splines so controllable. In the original implementation, each edge also includes a fixed-shape SiLU base function multiplied by a trainable scale; the spline has its own trainable scale. The grid has G intervals (G+1 grid points), and a k-th order spline uses G+k basis coefficients, so both grid resolution and spline order control the function class.
Faint curves are weighted cubic B-spline basis functions on five grid intervals; the bold teal curve is their sum. Eight coefficients control this spline component. Drag a coefficient to see its local support. The original edge also has trainable spline and SiLU-branch scales.
Now assemble a layer. Say it has 3 inputs and 2 outputs. An affine MLP layer has a 3×2 weight matrix plus 2 output biases: 8 trainable scalars. A KAN layer has 6 learnable edge functions, one for each input-output pair. In the original pykan parameterization, each edge contains G+k spline coefficients plus two trainable scales by default. Trace one input value through: it hits the edge going to output 1 and gets transformed by that edge’s curve; the same input hits the edge to output 2 and gets a different curve. Each output node then sums the transformed values arriving on its incoming edges. That sum is the output. No activation afterwards — the nonlinearity already happened, on the edges.
Stacking these layers composes learned 1-D edge functions through summing nodes. The original theorem corresponds specifically to a two-layer [n, 2n+1, 1] KAN; deeper stacks are the paper’s architectural generalization. For G grid intervals and spline order k, the spline contributes G+k coefficients per edge, plus two trainable scale factors in the default implementation. The paper reports that smaller KANs matched or beat larger MLPs on some small-scale function-fitting and PDE tasks; this is empirical and task-dependent, not a general edge-count guarantee.
Watch an input value travel each edge: every edge applies its own learned curve, and the destination node sums the results. Drag the input and see each edge’s output (dot on its curve) update, then the node totals.
KANs expose an internal-resolution knob that ordinary MLP layers do not. If a spline uses G intervals and order k, it has G + k spline coefficients per edge. Grid extension builds a finer-grid model by interpolating the learned edge curves, then continues training. The layer’s connectivity can stay fixed, but its parameter count grows. Training error can fall after refinement; test error need not, and very fine grids can overfit or become harder to optimize.
This gives a resolution-versus-cost path you can test deliberately: start coarse, measure held-out error, and refine while added capacity helps. It resembles grid refinement in numerical analysis, but finer grids do not guarantee smaller test error or easier optimization.
The teal line is the target. The orange line joins noiseless target samples with straight segments. This illustrates how denser sampling can resolve more detail, but it is not a B-spline, does not train a KAN, and cannot demonstrate overfitting because it has no noisy training set or held-out test set.
This is the payoff. Because every edge function can be plotted, a trained KAN offers a direct interpretation workflow, although useful interpretation still requires sparsity, fitting choices, and validation:
For scientific discovery this can produce a candidate relation that a human can inspect and test. In the original paper, KAN workflows recovered known relations among knot invariants and mobility-edge formulas for Anderson-localization models. Those examples used sparsity, symbolic fitting, domain assumptions, and human validation; the learned curve alone was not a publishable law.
This toy adds deterministic wiggles to a sine and toggles them off. A real KAN workflow scores a library of candidates after fitting input/output shifts and scales, lets the user fix a candidate, retrains the remaining parameters, and validates the loss.
This browser toy shows a sum of Gaussian radial-basis bumps approaching a target. Press train and each coefficient moves toward a target sample at its bump center. The control changes the number of toy basis functions. This is useful intuition for local basis expansions, but it is not the B-spline basis, gradient-based training, adaptive grid, or full parameter count used by pykan.
Teal = target function; orange = this toy basis expansion. Press Train to move each Gaussian coefficient toward one sampled target value. The readout counts only those toy coefficients; it is not a KAN-versus-MLP parameter comparison.
On some structured function-fitting tasks, small KANs achieve favorable accuracy per parameter and expose edge functions for inspection. Neither exact recovery nor a KAN advantage follows from this toy: both depend on the target, basis, optimizer, parameter or FLOP budget, and validation data.
KANs are exciting but young, and honesty about their costs matters.
These two exponential curves are hand-drawn to illustrate a possible accuracy-per-parameter advantage on a KAN-friendly target. They are not measurements and do not identify a dataset, architecture, optimizer, parameter count, FLOP count, or uncertainty. Use published Pareto frontiers for evidence.
| MLP | KAN | |
|---|---|---|
| learnable part | weights (numbers) | edge functions (splines) |
| nodes | fixed activation | sum |
| theory link | universal approximation | inspired by Kolmogorov–Arnold; deep KAN is a generalization |
| interpretability route | usually indirect | edge plots + sparsity/pruning + validated symbolic fits |
| large-scale recipes | mature | less established |
| strong current use | scaled vision/language + dense throughput | structured function fitting + interactive science workflows |
→ Activation Functions — compare fixed node activations with learned edge functions
→ Initialization — why parameter structure matters for training
→ Neural ODEs — another rethink of what a “layer” is
→ Embedding Layers — more on what networks actually store