MobileNetV4: Universal Models for the Mobile Ecosystem

Chapter 1: The Roofline and the Real Cost of a Layer

Before we can understand why MNv4's blocks are shaped the way they are, we need the right cost model. FLOPs alone lie. The honest model is the roofline: a layer's achievable speed is capped by whichever runs out first — the chip's arithmetic throughput, or its memory bandwidth.

Define two quantities for a layer:

Their ratio is the operational intensity:

A chip has a peak compute rate P (ops/s) and a peak bandwidth B (bytes/s). The roofline says the attainable rate for this layer is:

If I is large, the layer is compute-bound (you saturate the math units). If I is small, it is memory-bound (you saturate the bus and the math units idle). The crossover happens at the ridge point I* = P / B. Below I*, adding more FLOPs is nearly free — you have spare math. Above I*, more FLOPs cost time.

Worked example: 1×1 conv vs depthwise conv

Take a feature map of H = W = 28, C = 128 channels. Compare two layers a mobile network uses constantly.

(a) A 1×1 (pointwise) conv mixing 128 → 128 channels. Per output pixel it does C·C = 128·128 = 16,384 multiply-adds, over H·W = 784 pixels:

Memory: weights = 128·128 = 16,384 values; activations in/out ≈ 2·784·128 = 200,704 values. At 1 byte/value (int8): Q ≈ 217,088 bytes. So

(b) A 3×3 depthwise conv on the same map. Each channel is convolved independently, so per output pixel it does only 3·3 = 9 MACs per channel:

Memory: weights = 128·9 = 1,152 values; activations ≈ 2·784·128 = 200,704 values. Q ≈ 201,856 bytes.

The depthwise conv has 14× fewer FLOPs than the 1×1 conv — and yet its operational intensity is 13× lower. On a Pixel EdgeTPU the ridge point I* is high (it is a matrix engine with enormous P), so the depthwise layer sits deep in the memory-bound regime and runs at a tiny fraction of peak. The 1×1 conv, despite its larger FLOP count, runs near peak. The cheap-looking layer is the slow one. That single inversion of intuition explains most of MNv4.

Chapter 2: The Inverted Bottleneck — the block MNv4 generalizes

To understand MNv4's new block we first need the one it extends: the inverted bottleneck (IB) from MobileNetV2. The name is a deliberate joke. A classic ResNet bottleneck goes wide → narrow → wide; the inverted bottleneck goes narrow → wide → narrow. Why invert it?

The block takes an input with C channels and:

Plus a residual add when input and output shapes match. The genius is that the expensive spatial work (the depthwise conv) happens in the wide expanded space where it is most expressive, while the data that travels between blocks is kept narrow — small to store and to move. The block is "fat in the middle, thin at the ends."

Worked cost: one IB block

Let C = 64, e = 4 (so eC = 256), k = 3, stride 1, H = W = 28, C' = 64.

Expand 1×1: 28·28·(64·256) = 784·16,384 = 12,845,056 MACs.
Depthwise 3×3: 28·28·(256·9) = 784·2,304 = 1,806,336 MACs.
Project 1×1: 28·28·(256·64) = 784·16,384 = 12,845,056 MACs.

Total ≈ 27.5M MACs, of which the two 1×1 convs are ~93%. The depthwise — the part that does the actual spatial reasoning — is only ~7% of the FLOPs but, per Chapter 1, can be a much larger fraction of the latency on a matrix engine. Notice the asymmetry: the block's expressiveness comes from the cheap part, its FLOPs from the expensive parts, and its latency from a chip-dependent mix.

One more crucial detail: the linear bottleneck. MobileNetV2 removed the activation after the final projection. The argument: a ReLU on a low-dimensional tensor destroys information (it zeroes out roughly half of each narrow vector, and there is no redundant width to recover it). In the wide expanded space ReLU is fine — there is plenty of room — but on the narrow output you keep it linear to preserve the manifold. MNv4 inherits this.

Chapter 3: The Universal Inverted Bottleneck (UIB)

Here is the paper's central architectural contribution. The inverted bottleneck has exactly one depthwise conv, in a fixed position (after the expansion). Other strong blocks place depthwise convs elsewhere — a ConvNext block puts a large-kernel depthwise first. Each is good for different chips and stages. Instead of committing to one, MNv4 builds a superset with two optional depthwise convolutions and lets the search decide which to keep.

The UIB block, in order:

Two binary switches give four named instantiations — a single primitive that covers the whole zoo:

The ExtraDW instantiation (both depthwise convs on) is the new and surprisingly powerful member. Why does adding a second depthwise help?

• Receptive field for free-ish. Two stacked k×k depthwise convs give an effective receptive field of (2k−1)×(2k−1) at almost no FLOP cost (depthwise is cheap in FLOPs). A 3×3 start-DW plus a 3×3 mid-DW reaches a 5×5 footprint; mix kernel sizes (e.g. 3×3 and 5×5) and you reach further still.
• Spatial mixing on both sides of the expansion. The start-DW mixes spatially in the narrow space (cheap, before channels balloon); the mid-DW mixes in the wide space (more expressive). You get two bites at the spatial structure for the price of two cheap layers.
• Fusion-friendly. On accelerators that fuse depthwise into neighboring ops, the extra depthwise's real cost is far below its standalone roofline — so on those chips ExtraDW is nearly free accuracy.

Worked example: the four configs on one input

Input [28, 28, 64], e = 4 (eC = 256), C' = 64, k = 3, stride 1. Compare MAC cost (the always-on 1×1 convs are shared by all four):

The four configs differ by at most ~9% in FLOPs — they are nearly interchangeable on a FLOP budget — but they place spatial mixing very differently and therefore land in different spots on each chip's roofline. That is the point: the search has cheap freedom to choose where the receptive field comes from. Note the start-DW is 4× cheaper than the mid-DW here because it runs on 64 channels instead of the expanded 256.

One more subtlety the paper handles carefully: stride placement. When a UIB block downsamples (stride 2), the stride lives on whichever depthwise conv is present (preferring the mid-DW), keeping the 1×1 convs at full resolution-independent matmul shapes. If both depthwise convs are off (FFN config), the block cannot downsample on its own — the search simply won't place an FFN where a stride is required.

Chapter 4: Mobile MQA — attention that is actually fast on-device

Convolutions have a local receptive field. To reach genuinely global context (helpful for the largest, highest-accuracy MNv4 models) you want attention. But standard multi-head attention is brutal on mobile: it is memory-bound (low operational intensity, per Chapter 1) and its keys/values blow up memory traffic. MNv4 introduces Mobile MQA, an attention block engineered around the roofline.

Start from standard attention on a feature map flattened to N = H·W tokens of dimension d:

In multi-head attention (MHA) with h heads, each head has its own Q, K, and V projections. Keys and values therefore cost h separate [N, d_h] tensors. The K and V tensors dominate the memory traffic, and on a mobile accelerator that traffic — not the matmul FLOPs — is the bottleneck.

Idea 1: Multi-Query Attention (MQA)

MQA (Shazeer 2019) keeps h separate query heads but uses a single shared key and a single shared value across all heads:

This collapses the K/V memory from h projections to 1. The query is still per-head (cheap to compute, the part that gives heads their diversity), but the expensive-to-move K and V are shared. On-device this is a large win precisely because the bottleneck was K/V traffic, not query FLOPs. The paper reports MQA gives a substantial speedup on mobile accelerators with negligible accuracy loss versus MHA.

Idea 2: Spatial reduction of K and V (SR)

Even one K and one V over all N = H·W positions is a lot when the feature map is large. Mobile MQA adds an optional spatial-reduction: downsample the keys and values (a stride-s depthwise conv, typically s = 2) before the attention, so K and V have N/s² rows while the queries keep full resolution N. The attention matrix shrinks from N×N to N×(N/s²):

Queries still attend globally (every position gets a global receptive field), but to a coarser summary of the scene. For classification this is nearly lossless — fine detail is already captured by the convolutional stages; attention only needs the global gist.

Worked example: the memory MQA + SR saves

Take a 14×14 feature map, so N = 196, with h = 8 heads and d_h = 32, spatial reduction s = 2 (so N/s² = 49).

K/V tensor sizes:

• MHA: K and V are each h·N·d_h = 8·196·32 = 50,176 values. Two of them: ~100k values of K/V to read.
• MQA (shared K/V): each is N·d_h = 196·32 = 6,272 values — an 8× reduction from sharing across heads.
• MQA + SR (s=2): each is (N/4)·d_h = 49·32 = 1,568 values — another 4× reduction, so 32× smaller than MHA's K/V.

Attention matrix size: MHA computes h·N·N = 8·196·196 = 307,328 scores; MQA+SR computes h·N·(N/4) = 8·196·49 = 76,832 — a 4× shrink in the score matrix too. The combined effect: a global-context block whose cost is dominated by the small per-head query matmuls, with the memory-heavy K/V slashed by ~32×. That is what makes attention viable on a phone.

In MNv4 the attention is added only in the deepest, lowest-resolution stages (where N is small, so even attention is affordable) and only in the Hybrid variants. The pure-conv variants skip it entirely — many accelerators run convolution far better than attention, so for those targets a conv-only model is on the frontier. This is the universality philosophy again: attention is one tool offered to the search, not a mandate.

Chapter 5: Two-Phase Hardware-Aware NAS

We now have a rich search space: a stack of UIB blocks, each with two depthwise switches, kernel sizes, expansion ratios, and channel counts, plus optional Mobile MQA in the deep stages. The combinatorics are enormous. MNv4 searches it with two-phase neural architecture search guided by measured on-device latency.

Why two phases? A flat search over everything at once is sample-inefficient: the search wastes most of its budget on architectures that are the wrong overall size. MNv4 factorizes the decision:

The objective is a Pareto search: maximize accuracy subject to a latency target measured by actually running each candidate on the device (or a calibrated latency model of it), not by a FLOP proxy. Per Chapter 1, FLOPs and latency disagree on mobile chips, so a FLOP-guided search would systematically pick the wrong blocks.

Why factorizing helps — a back-of-envelope

Suppose the space has M macro configurations and, for each, F micro configurations per block over B blocks (so F^B micro choices). A flat search faces M·F^B joint candidates. The two-phase scheme first explores ~M macro candidates (with a fixed cheap block), picks the best skeleton, then explores micro choices around that one skeleton — roughly M + F^B work instead of M·F^B. With M ≈ 1000 and F^B ≈ 10⁶, that is ~10⁶ evaluations instead of ~10⁹. The factorization is a multiplicative-to-additive saving — the same trick as separating concerns in any large search.

The latency model and "robust" search

Measuring every candidate on real hardware is slow, so NAS systems build a latency lookup table / cost model: benchmark each primitive (a UIB of a given config, kernel, width, resolution) once on the device, store the measured latency, and estimate a whole network as the sum of its blocks. This makes the inner loop a fast table lookup — which is exactly the "latency/accuracy lookup" we build in the Chapter 8 explorer and the second Code Lab. MNv4 also designs for robustness: it seeks architectures that are good across several target devices, not overfit to one latency table, which is what produces the universal frontier.

Chapter 6: Distillation and the Training Recipe

A great architecture is only half of a great model — the training recipe matters as much. MNv4's largest model (Hybrid-Large) reaches ~87% ImageNet-1K top-1, and a big chunk of that final accuracy comes from knowledge distillation, plus a refined data-augmentation and regularization recipe.

Distillation in one equation

A large, accurate teacher network produces soft probability vectors over the 1000 classes. The small student (the MNv4 model) is trained to match the teacher's softened distribution, not just the hard one-hot label. With temperature T, the teacher's softened probabilities are:

where z are the teacher logits. The student's loss blends a hard-label cross-entropy with a soft-target term that pulls the student's softened distribution q^T toward the teacher's:

The T² factor rescales the soft-target gradients (which shrink like 1/T²) so they stay comparable to the hard-label term as you raise the temperature. The dark knowledge idea: the teacher's "wrong" probabilities carry information — that a cat image looks 8% like a lynx and 0.001% like a truck tells the student about class geometry that a one-hot label never could.

Why distillation helps small models so much

A mobile-sized student has limited capacity. Hard labels give it one bit of supervision per example ("this is class 7"). Soft targets give it a full 1000-dimensional gradient signal per example — a much richer training set without any new data. Empirically this is worth several points of top-1 for compact models, which is why the headline ~87% figure relies on it. MNv4 also uses an offline-generated set of teacher predictions so distillation is cheap to run at scale.

The rest of the recipe

Chapter 7: Results and Ablations

The headline result is the Pareto frontier: across a suite of mobile accelerators — Pixel CPU, Pixel EdgeTPU, Samsung/Qualcomm-class GPUs and Hexagon DSPs, and Apple Neural Engine — the MNv4 family sits on or near the best achievable accuracy at each latency. No single competitor backbone matches it on all targets simultaneously; MNv4 is described as "mostly Pareto-optimal across the mobile ecosystem." The Hybrid-Large model reaches around 87% ImageNet-1K top-1, with hybrid models running in the low-single-digit milliseconds on a Pixel EdgeTPU.

The family spans Small / Medium / Large, each in a Conv (pure UIB) and a Hybrid (UIB + Mobile MQA in the deep stages) form. Conv variants win on chips that run attention poorly; Hybrid variants squeeze out the top accuracy where attention is affordable.

What the ablations show

Reading the frontier like an engineer

Two networks A and B with the same accuracy but different latency: B dominates if it is faster. A whole family Pareto-dominates another if, for every accuracy level, it offers a lower-latency model (and vice versa). Formally, family F dominates family G when for each model g in G there exists f in F with accuracy(f) ≥ accuracy(g) and latency(f) ≤ latency(g), strictly on at least one axis. MNv4's claim is precisely this kind of dominance — replicated across many devices, which is the part nobody had achieved before.

Chapter 8: UIB Explorer — build a stage, read its cost on every chip

This is the payoff. You are now the NAS. Configure a small stack of UIB blocks — choose each block's type (IB / ConvNext / ExtraDW / FFN), kernel size, and whether the deepest block uses Mobile MQA — and the explorer estimates accuracy contribution and latency on four different accelerators from a lookup table, exactly as the hardware-aware search does in its inner loop. Watch how the best configuration differs by target chip.

The estimate combines the per-primitive latency table (Chapter 5) with a roofline weighting (Chapter 1): on the matrix-engine EdgeTPU, depthwise convs are penalized, so ExtraDW costs more latency; on the DSP/CPU they are cheap, so ExtraDW is nearly free accuracy. Mobile MQA adds global context (accuracy) but is only affordable in the deep, low-resolution block. Try to find the configuration on the frontier for each device — you will discover the search's lesson: the best block assignment is device-dependent.

The second Code Lab below implements the engine behind this explorer: a latency lookup that sums per-block measured costs and returns the accuracy/latency pair a NAS would read off. That is the literal inner loop of hardware-aware search — and the thing that makes the whole approach tractable.

Chapter 9: Limitations and Connections

MNv4 is a strong, practical result, but it is honest about its boundaries — and placing it among neighboring ideas is the best way to remember it.

Limitations

• Classification-centric headline. The Pareto claims and the ~87% figure are for ImageNet-1K classification. Detection/segmentation transfer is shown, but the universality story is strongest for the backbone-as-classifier setting; downstream heads have their own cost models.
• The latency table must exist. Hardware-aware NAS needs a measured latency model for each target accelerator. For a brand-new chip with no benchmark data, you must re-profile the primitives — the method is universal across known devices, not magically across unmeasured ones.
• Recipe-dependent accuracy. The headline numbers fold in distillation and a heavy augmentation recipe. The architecture alone, trained plainly, lands several points lower — reproducing the paper means reproducing the recipe.
• Attention only in the deep stages. Mobile MQA buys global context only where the feature map is already small; the bulk of the network is still local convolution, so very-long-range early-layer interactions are not captured.
• Quantization assumed. Much of the on-device speed assumes int8 execution; some accuracy/latency points shift under different numeric formats.

Connections — related Veanors

• Attention Is All You Need — the multi-head self-attention that Mobile MQA simplifies (shared K/V) for the deep stages.
• An Image is Worth 16×16 Words (ViT) — pure-attention vision; MNv4-Hybrid is the conv-first counterpoint that keeps attention only where it is cheap.
• Veanors index — more architecture and systems papers in the same interactive format.

Connections — related Gleams

• Convolutional Neural Networks — the convolution and depthwise-separable foundations MNv4 builds on.
• Vision Transformers — global attention for vision, the contrast to MNv4's local-conv-plus-light-attention design.
• Quantization — int8 execution, which underlies MNv4's on-device latency numbers.
• Discovery — search the full catalog for related material.

Cheat sheet