Mobile & NAS Backbones — Vision Models That Fit in a Phone

Chapter 1: Depthwise Separable Convolution

To make convolution cheap we first have to see exactly where its cost comes from. A standard convolution does two jobs at once, and the trick of MobileNetV1 is to notice that and split them apart. Let us count, because the count is the insight.

Take one convolution layer. It reads an input with C_in channels and produces an output with C_out channels, using filters of size K×K (say 3×3), over an output feature map of spatial size H×W. A standard convolution's cost in MACs is the product of all of them: K · K · C_in · C_out · H · W. Every output pixel, in every output channel, is a weighted sum over a K×K patch across all input channels. That "across all input channels" is the expensive part: each output channel mixes spatially and across channels in one shot.

MobileNetV1's idea, the depthwise separable convolution, is to factor that single operation into two stages that, together, do the same two jobs but at a tiny fraction of the cost:

Stage 1 — the depthwise convolution. Apply one K×K filter to each input channel separately. Channel 0's filter only sees channel 0; channel 1's only sees channel 1. No mixing across channels. This does the spatial job (finding edges, textures) per channel. Its cost is K · K · C_in · H · W — the C_out factor is gone because we are not producing C_out outputs here, just filtering the C_in we have.

Stage 2 — the pointwise convolution. Now mix across channels with a 1×1 convolution: for each output pixel, take a weighted sum over all C_in channels to produce each of C_out channels. This does the channel-mixing job. Because the filter is 1×1, there is no K×K spatial extent, so its cost is 1 · 1 · C_in · C_out · H · W = C_in · C_out · H · W.

Where does that ~8× come from in general? Divide the separable cost by the standard cost: (K·K·C_in·H·W + C_in·C_out·H·W) / (K·K·C_in·C_out·H·W) = 1/C_out + 1/K². With C_out=64 and K=3 that is 1/64 + 1/9 ≈ 0.0156 + 0.111 ≈ 0.127, i.e. about 7.9× cheaper — matching the worked numbers. For a 3×3 filter the term 1/K² = 1/9 dominates, so the speedup tends toward roughly K² = 9× for wide layers. That single algebraic ratio is the entire economic case for MobileNetV1.

Drag the sliders below to feel the factorization. Watch the standard-conv bar tower over the separable bar, and watch the ratio settle near K² as the channel count grows. The visual is the arithmetic.

Here is the factorization as code — first counting the MACs from scratch so nothing is hidden, then the library one-liner that builds the same block.

python
def standard_conv_macs(K, Cin, Cout, H, W):
    return K * K * Cin * Cout * H * W            # spatial AND channel mixing in one shot

def separable_conv_macs(K, Cin, Cout, H, W):
    depthwise = K * K * Cin * H * W              # one KxK filter per channel — spatial only
    pointwise = Cin * Cout * H * W               # 1x1 conv — channel mixing only
    return depthwise + pointwise

std = standard_conv_macs(3, 64, 64, 56, 56)   # 115.6M
sep = separable_conv_macs(3, 64, 64, 56, 56)   # 14.66M
print(std / sep)                                  # ~7.9x — equals 1/(1/Cout + 1/K**2)

python
import torch.nn as nn
# The library version: a depthwise-separable block is just two Conv2d layers.
def separable(Cin, Cout, K=3):
    return nn.Sequential(
        nn.Conv2d(Cin, Cin, K, padding=K//2, groups=Cin),  # groups=Cin == depthwise
        nn.BatchNorm2d(Cin), nn.ReLU(),
        nn.Conv2d(Cin, Cout, 1),                            # 1x1 == pointwise (channel mix)
        nn.BatchNorm2d(Cout), nn.ReLU())

The single line that makes it depthwise is groups=Cin: it tells PyTorch to give each input channel its own filter with no cross-channel mixing — exactly Stage 1. The following 1×1 conv is Stage 2. This little two-layer pattern is the atom from which MobileNetV1 (and, in evolved form, every network after it) is built.

Chapter 2: Inverted Residuals & Linear Bottlenecks

MobileNetV1 was a flat stack of separable blocks. MobileNetV2 asked a sharper question: how do you add residual connections — the skip-and-add shortcuts that let very deep networks train — to a depthwise world, without paying for them in memory and compute? The answer, the inverted residual block, is so good it became the default block of mobile vision for years.

Recall the classic ResNet bottleneck: it is wide–narrow–wide. You take a wide tensor (many channels), squeeze it to few channels with a 1×1 conv, do the expensive 3×3 spatial work on the narrow tensor, then expand back to wide, and add the skip connection between the two wide ends. The skip connects the fat tensors; the work happens on the thin one.

MobileNetV2 inverts this to narrow–wide–narrow, which is why it is called an inverted residual. You start narrow (few channels), expand with a 1×1 conv by an expansion factor t (typically 6×) to a wide tensor, do a cheap depthwise 3×3 on the wide tensor, then project back down to narrow with another 1×1 — and the skip connection now joins the two narrow ends.

Why invert? Memory. The skip connection has to hold a tensor alive in memory across the block. In the classic block the skip is wide, so you keep a fat tensor resident. In the inverted block the skip is narrow, so the tensor you keep resident is small — the expensive wide tensor exists only transiently inside the block and is never carried across. On a memory-starved phone, that is the difference between fitting and not fitting. The depthwise conv being so cheap is what makes it affordable to do the spatial work on the wide tensor at all.

The second idea, the linear bottleneck, is subtle and important. The final projection (wide → narrow) is followed by no activation function — it is linear. Why remove the ReLU there? Because a ReLU destroys information when the tensor is low-dimensional: clamping negatives to zero in a narrow space collapses structure you cannot recover, whereas in a wide space the same information survives in other channels. MobileNetV2's authors argued the useful signal lives on a low-dimensional manifold, and you must not ReLU it after squeezing into that low-dimensional bottleneck. So: ReLU after the expand and the depthwise (both wide), but linear after the project (narrow).

Trace the tensor through the block in the widget. Watch the channel count balloon at the expand, the depthwise do its cheap spatial work on the wide tensor, the projection squeeze it back, and the narrow skip connection arc over the top. Toggle the ReLU on the projection to see the "linear bottleneck" choice highlighted.

Chapter 3: Squeeze-and-Excitation & Hard-Swish

MobileNetV3 kept the inverted residual and added two near-free upgrades that buy accuracy for almost no latency: a tiny attention-like module called squeeze-and-excitation and a cheaper activation called hard-swish. Both are small, both are clever, and both teach the same lesson — on mobile, the best upgrades cost almost nothing to run.

Squeeze-and-excitation (SE) gives the network a way to say "this channel matters more for this image; turn it up." A convolution treats all channels democratically. But for a picture of a dog, the "fur texture" channel should probably shout while the "sky gradient" channel should whisper. SE learns to re-weight channels per image, in three steps.

Squeeze. Collapse each channel's whole H×W map to a single number by global average pooling. A (C, H, W) tensor becomes a (C,) vector — one scalar summarizing "how much of this channel fired anywhere." This is the same global-pooling move scene classifiers use, here serving as a cheap global descriptor.

Excite. Feed that C-vector through a tiny two-layer network: squeeze it down to C/r channels (a reduction ratio r, often 4 or 16), apply ReLU, expand back to C, and pass through a sigmoid to get C values in [0,1]. These are the channel gates: a learned importance weight per channel.

Scale. Multiply each original channel's entire feature map by its gate. Channels the network deemed important keep their full strength; unimportant ones are dialed down. The whole module adds a negligible number of MACs (a couple of small dense layers on a C-vector) but consistently lifts accuracy — it is the cheapest attention you will meet.

The second upgrade is the activation. MobileNetV3 noticed that the swish function, x·sigmoid(x), trains better than ReLU but the sigmoid and exponential are slow on mobile. So it swapped in hard-swish: x · ReLU6(x+3) / 6, where ReLU6 clamps to [0,6]. This is a piecewise-linear approximation of swish that uses only clamps and a multiply — no exponential — so it runs fast on integer-friendly mobile hardware while keeping most of swish's accuracy benefit. MobileNetV3 uses hard-swish in the deeper layers (where its benefit is largest) and plain ReLU early (where activations are cheaper and the gain is smaller).

Compare the curves and the gates in the widget. Slide a channel's importance and watch SE scale its activation; overlay ReLU, swish, and hard-swish to see how closely the cheap hard-swish tracks the expensive swish.

Chapter 4: Compound Scaling (EfficientNet)

Suppose you have a good mobile block and you want a bigger, more accurate model. You have three knobs you can turn: depth (more layers), width (more channels per layer), and resolution (larger input images). Before EfficientNet, people turned one knob at a time, somewhat arbitrarily. EfficientNet's contribution was to show that these three are coupled, and to scale them together with a single coefficient. This is compound scaling.

Why are the three coupled? Intuition: if you feed a bigger image (more resolution) but keep the network shallow, the deeper features cannot integrate the extra detail — the receptive field does not reach across the larger image. If you make the network deeper but keep the image tiny, you have more layers analyzing a starved input. And more width without more depth or resolution gives you fine-grained channels with nothing new to look at. The knobs only pay off when turned in balance.

EfficientNet formalizes this. Pick a single compound coefficient φ (your overall compute budget). Then scale depth by α^φ, width by β^φ, and resolution by γ^φ, where α, β, γ are constants found by a small grid search on the baseline, subject to the constraint that they keep total FLOPs growing predictably.

The constraint is the elegant part. FLOPs scale linearly with depth (twice the layers, twice the work), but quadratically with width (channels appear as C_in·C_out, so doubling width quadruples a layer's cost) and quadratically with resolution (H·W appears, so doubling each side quadruples the pixels). So total FLOPs grow as α · β² · γ², all raised to the φ. EfficientNet constrains α · β² · γ² ≈ 2, which means each unit increase of φ doubles the FLOPs — a clean, predictable dial.

EfficientNet-B0 (the φ=0 baseline, itself found by NAS — the topic of Chapter 5) through B7 are exactly this single block scaled up the compound ladder. B0 is mobile-sized; B7 is a server model — same architecture, one coefficient apart. EfficientNet matched the accuracy of much larger hand-designed networks at a fraction of the FLOPs precisely because it never wasted budget over-scaling a single dimension.

Turn the φ dial below. Watch all three knobs rise together along the constrained ratios, and watch accuracy climb while FLOPs double per step. Then try the "unbalanced" toggle — dump all the budget into one knob — and see accuracy lag behind the balanced curve for the same FLOPs. That gap is the entire argument for compound scaling.

Chapter 5: Hardware-Aware NAS (MobileNetV4 / UIB)

So far humans designed these blocks. But the design space is enormous — for each of dozens of layers you could pick a filter size, an expansion factor, whether to add SE, which activation, whether to downsample — and the best choice depends on the chip. Neural architecture search (NAS) hands this combinatorial design problem to an algorithm. The key modern twist is hardware-aware NAS: the search optimizes not for FLOPs, but for measured latency on the actual target device.

Why is FLOPs not enough? Because FLOPs are a poor predictor of real latency. Two layers with identical FLOP counts can have wildly different wall-clock times: depthwise convs are memory-bound (they move a lot of data per multiply) and often run slower than their FLOP count suggests, while big dense 1×1 convs are compute-bound and run efficiently on the same hardware that crawls on depthwise. Memory access, cache behavior, kernel availability, and accelerator quirks all break the FLOPs–latency link. The only honest cost is a stopwatch on the target chip.

Hardware-aware NAS makes the cost function literal: the search runs candidate architectures (or estimates them from a measured lookup table of per-block latencies) on the phone, and the objective rewards accuracy while penalizing measured milliseconds. The result is a network tuned to one device's reality — the same search targeting a CPU, a GPU, and a neural accelerator can return three different winning architectures, because each chip rewards different blocks. This is how MnasNet, MobileNetV3, and the EfficientNet baseline were born.

MobileNetV4 (2024) pushed this further with a single flexible block, the Universal Inverted Bottleneck (UIB). UIB generalizes the inverted residual by making the placement of depthwise convolutions optional and configurable: it can be a plain inverted bottleneck, an inverted bottleneck with an extra depthwise before the expansion, one with an extra depthwise after, or a lightweight variant with no extra depthwise at all. The NAS picks, per layer, which of these UIB shapes runs best on the target. One superset block, many instantiations, search chooses.

The widget lets you be the hardware-aware search. Choose, per stage, between a few block types; each carries a (FLOPs, measured-latency, accuracy) triple that you can read live. Try to assemble the network that maximizes accuracy under a latency cap — and notice that the FLOP-cheapest choice is often not the latency-cheapest one.

Chapter 6: The Pareto Frontier — Why One Number Lies

You now have a zoo of models: MobileNetV1/2/3, EfficientNet-B0…B7, MobileNetV4 variants, ResNets, ViTs. How do you compare them honestly? Not by accuracy alone (that ignores latency) and not by latency alone (that ignores accuracy). The honest comparison is a two-dimensional picture: the latency–accuracy Pareto frontier.

Plot every model as a point with latency on the x-axis and accuracy on the y-axis. A model is Pareto-optimal (or "on the frontier") if no other model is both faster and more accurate than it. A model that some other model beats on both axes is Pareto-dominated — strictly worse, never the right choice. The frontier is the upper-left boundary of the cloud of points: the set of best-possible trade-offs.

This reframes the whole field. There is no single "best mobile backbone." There is a frontier, and where you sit on it depends on your latency budget. If you have 5 ms, you pick the most accurate model at 5 ms. If you have 30 ms, you pick the most accurate model at 30 ms — a different model. The job of architecture research (MobileNet, EfficientNet, NAS) is to push the whole frontier up and to the left, so that at every budget you get more accuracy.

And one number lies in a second way: the number depends on the device. A model that is Pareto-optimal on a datacenter GPU can be dominated on a phone CPU, because the GPU loves the wide dense ops that the CPU runs slowly. The frontier is per-hardware. This is the deep reason hardware-aware NAS exists: it pushes the frontier on your chip, not on a generic FLOP axis that no real device obeys.

Explore the frontier below. Each dot is a model. The frontier line connects the Pareto-optimal ones; dominated models sit below-and-right of it, greyed out. Switch the device and watch points shuffle — some that were on the frontier fall off it, because the new chip rewards different blocks. This shuffling is the reason "best model" is a device-specific question.

Chapter 7: Showcase — The Pareto Explorer

Everything converges here. This is a live mobile backbone you design with sliders, and a Pareto plot that places your design among reference models in real time. You choose the block type, the width multiplier, the input resolution, and the depth; the panel computes predicted FLOPs, predicted on-device latency, and predicted accuracy, and drops your model onto the latency–accuracy plane. The goal: build a point that lands on the frontier — or pushes past it.

The block choices encode the whole lesson. Separable is MobileNetV1's bare depthwise-separable block — cheapest, least accurate per layer. Inverted residual is MobileNetV2 — more accurate, slightly costlier. Inv-res + SE + hard-swish is MobileNetV3 — the accuracy bump for near-free latency. UIB (searched) is MobileNetV4's universal block — the best trade-off, as if a hardware-aware NAS tuned it for the chip. As you change blocks you are literally walking through Chapters 1–5.

Four things to try, each a chapter made tangible:

• Start with Separable, then upgrade the block to Inv-res + SE. Accuracy jumps for a small latency cost — you just felt why MobileNetV3 happened.
• Crank the width multiplier alone. Watch latency rise quadratically (width enters as C_in·C_out) while accuracy gains taper — the unbalanced-scaling penalty of Chapter 4.
• Raise resolution alone. Latency climbs with the pixel count (also quadratic) — another single-knob trap.
• Switch the target device. The same design moves on the plane because the latency model changes — Pareto is per-chip, exactly Chapter 6.

If you removed this simulation, you would lose something real: reading "FLOPs grow quadratically with width" is abstract until you drag the width slider and watch your star slide right far faster than it rises. The napkin drawing of this whole lesson is this plane — latency out, accuracy up, a frontier to chase.

Chapter 8: Quantization & Deployment to the Edge

You have a great architecture. There is one more lever, often the biggest single speedup of all, that lives below the architecture: quantization. The model you trained stores its weights and activations in 32-bit floating point (FP32). Quantization replaces those with 8-bit integers (INT8) — and on phone hardware that is roughly a 4× smaller model and 2–4× faster inference, because integer math is cheaper, moves less memory, and maps onto dedicated mobile accelerators.

How do you cram a float into 8 bits? An INT8 value holds 256 levels (−128 to 127). Quantization maps a float range onto those levels with two numbers: a scale s (the size of one step) and a zero-point z (which integer represents 0.0). The rule is q = round(x / s) + z, and you recover an approximate float with x ≈ s · (q − z). You are snapping every float to the nearest rung of a 256-rung ladder. The art is choosing s and z so the ladder covers the values the tensor actually takes.

There are two ways to do this. Post-training quantization (PTQ) takes an already-trained float model and quantizes it, using a small calibration set of images to measure each tensor's typical range and pick good scales. It is fast and needs no retraining, but can lose a point or two of accuracy on sensitive models. Quantization-aware training (QAT) simulates the rounding during training (the forward pass uses fake-quantized values, the backward pass flows gradients through), so the network learns weights that are robust to being snapped to the ladder. QAT recovers most of the lost accuracy at the cost of a retraining run.

Why does this interact with everything before it? Because some architectural choices are quantization-friendly and some are not. ReLU6 (used throughout MobileNet) bounds activations to [0,6], which keeps their range tight and easy to quantize — that bound was chosen partly for quantization. The linear bottleneck and hard-swish were likewise designed with integer hardware in mind. Mobile architecture and quantization co-evolved: the blocks are shaped so that snapping them to 8 bits barely hurts.

Slide the bit-width below and watch the weight histogram snap to a coarser ladder. At 8 bits the quantized curve hugs the original; at 4 bits it visibly stair-steps and accuracy drops. The readout shows model size and the round-trip error — the quantitative version of the trade-off you are watching.

python
import numpy as np
def quantize(x, s, z, qmin=-128, qmax=127):
    q = np.round(x / s) + z                  # snap to the integer ladder
    return np.clip(q, qmin, qmax).astype(np.int8)

def dequantize(q, s, z):
    return s * (q.astype(np.float32) - z)     # back to approximate float

w = np.array([-1.0, 0.42, 0.005, 0.99])
s = 1.0 / 127; z = 0                         # scale = range/levels, zero-point at 0
q  = quantize(w, s, z)                        # int8: [-127, 53, 1, 126]
wr = dequantize(q, s, z)                      # ~[-1.0, 0.417, 0.0079, 0.992]
print(np.abs(w - wr).max())                   # tiny — well under one step

Chapter 9: Connections & Where It's Used

Mobile backbones are not a niche; they are the substrate of on-device intelligence. The pattern you built — factor the expensive operation, invert the block to save memory, add near-free attention, scale with one balanced dial, then let a search tune it to the chip, and finally quantize — recurs everywhere computation must be small. Naming the connections turns this lesson into a map.

The shared idea is efficiency-as-design. Depthwise separable convolution is the same "factor a big operation into cheaper pieces" move that low-rank factorization plays in language-model adapters and that separable filters play in classical signal processing. The squeeze-and-excitation gate is a lightweight cousin of the attention mechanism you meet in transformers — a learned, content-dependent re-weighting, here over channels instead of tokens.

Where it lands in the corpus. The convolution itself is taught from zero in CNN Classification; the whole-image-to-one-label task these backbones feed is Visual Scene Classification and Acoustic Scene Classification, whose edge constraint (a battery-powered sensor classifying its environment) is exactly the budget this lesson optimizes. The attention machinery the SE gate foreshadows is in Vision Transformer and The Transformer, and the modern blend of convolution and attention — relevant to MobileNetV4's hybrid stages — is in Conv-Transformer Hybrids. For the broader low-resource framing, see TinyML.

Cheat sheet

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." — Antoine de Saint-Exupéry, which is, quite literally, the design philosophy of every network in this lesson.