A convolution is cheap and sees only a tiny patch. Attention sees the whole image but its cost explodes. A phone has a millisecond budget and a thermal limit. The entire field of mobile vision backbones is one question made concrete: can a single network be local and global, on real silicon, in real time? MobileViT, FastViT, and EfficientViT are three different answers.
You are building the camera feature that segments a person from the background in a video call. The user holds a three-year-old phone. Every frame must be processed in under 16 milliseconds or the video stutters. The chip heats up if you push too hard, and the OS throttles you the moment it does. This is the world your vision model lives in: a fixed millisecond budget, a thermal ceiling, and a user who notices a single dropped frame.
Your first instinct is the network that wins every accuracy leaderboard: a big Vision Transformer (a model that splits the image into patches and lets every patch attend to every other patch). You port it. On the phone it runs at 4 frames per second, the device gets hot in a minute, and the feature is unusable. The leaderboard winner is, on this hardware, a brick.
So you reach for the opposite: a convolutional network (a model that slides small filters over the image, each filter seeing only a tiny local patch). It is fast and cool. It runs at 60 frames per second. But its accuracy at the segmentation task is mediocre — it keeps cutting off the top of the person's head or leaking background through their hair, because to decide "this pixel belongs to the person" it needs to consider parts of the image far away from that pixel, and a small filter cannot see that far.
You are caught between two failures. The accurate model is too slow. The fast model is too short-sighted. This is not a bug in either; it is the fundamental tension of on-device vision. A convolution gives you locality — it processes nearby pixels together, cheaply, with a built-in assumption that what matters is local. Attention gives you globality — it lets distant parts of the image inform each other, at a cost that grows with the square of the image size. Locality is cheap and myopic. Globality is far-sighted and expensive.
The breakthrough idea of this entire lesson is refusing to choose. A conv-transformer hybrid is a single backbone that uses cheap convolutions to do the local work — clean up textures, mix nearby pixels, downsample — and reserves expensive attention for the few places where genuine long-range reasoning is worth its cost. Convolution handles the 90% that is local; attention handles the 10% that is global. You pay for globality only where it earns its keep.
Drag the slider below. At the far left the network is pure convolution: fast, cheap, but its receptive field (the region of the input that can influence one output) stays small, so it cannot connect distant parts of the image. At the far right it is pure attention: the receptive field is the whole image instantly, but the cost bar shoots up. Somewhere in the middle — mostly conv with a little attention — you get a wide receptive field at a cost a phone can afford. That middle is where MobileViT, FastViT, and EfficientViT all live.
Slide from pure convolution (left) to pure attention (right). Watch the receptive field grow and the cost bar climb. The phone-affordable sweet spot is mostly conv with a touch of attention — not either extreme.
Why should you care beyond a video-call background? The same constraint — "be accurate AND fit in a fixed compute budget on weak hardware" — governs on-device photo classification, real-time AR, robot perception on an embedded board, drone obstacle avoidance, and any vision that must run without a round-trip to a data center. Whenever the model lives on the device instead of the cloud, the conv-transformer hybrid is the dominant design pattern. Over the next nine chapters we will build it piece by piece: pin down the locality/globality tradeoff precisely, dissect MobileViT and the hybrid block, learn the train-time trick (structural reparameterization) that makes FastViT fast for free, see how EfficientViT cuts attention's redundancy, confront why FLOPs lie about latency, and finish with a hybrid block you assemble yourself.
To design a hybrid, you have to measure the two things it trades. Those two things are receptive field — how far across the image one output can "see" — and cost — how many multiply-add operations it takes to compute. A convolution and an attention layer sit at opposite corners of this measurement, and getting the numbers in your bones is what makes every later chapter click.
How a convolution's receptive field grows. A single 3×3 convolution lets one output pixel depend on a 3×3 = 9-pixel patch of its input. Stack a second 3×3 conv and that output now depends on a 5×5 patch (each side grows by 2). Stack a third and it is 7×7. The rule: with a kernel of size k, each layer adds (k−1) to the receptive field width. So a 3×3 conv adds 2 per layer. To reach a receptive field of, say, 200 pixels — enough to connect a person's head to their feet — you need about (200−1)/2 ≈ 100 layers of plain 3×3 convolution. That is why CNNs grow context laboriously, through depth.
Downsampling helps: every time you halve the spatial resolution (stride-2), each later conv covers twice as much of the original image. That is how real CNNs reach a useful receptive field in dozens of layers instead of a hundred — but it is still a slow climb, and the receptive field is local by construction until you have stacked enough of it.
How attention's receptive field grows. It doesn't grow — it starts at maximum. In one self-attention layer, every token attends to every other token, so the receptive field is the entire image on layer one. There is no climb. For a task whose signal is spread across the frame, this is exactly the right tool. The catch is the cost.
The cost, with real arithmetic. Self-attention compares every token to every token, so its cost scales with the number of tokens squared. Let us put numbers on it. Split a 224×224 image into 16×16 patches: that is 14×14 = 196 tokens. The attention score matrix is 196×196 ≈ 38,000 entries — bearable. Now run attention at higher resolution, say 56×56 = 3,136 tokens (common in the early stages of a backbone). The score matrix is 3,136×3,136 ≈ 9.8 million entries. The token count went up 16×; the attention cost went up 256×. This quadratic blow-up is the single fact that shapes every mobile transformer: you cannot afford global attention at high resolution.
Now the design insight that the rest of the lesson exploits: attention's quadratic cost depends on token count, and token count depends on resolution. If you run attention only at low resolution — after the convolutions have already downsampled the image to, say, 7×7 = 49 tokens — the score matrix is just 49×49 ≈ 2,400 entries, trivially cheap. So the hybrid recipe writes itself: use convolutions to do all the early, high-resolution local work and to shrink the spatial size, then let cheap attention reason globally over the handful of remaining low-resolution tokens. You get globality where it is affordable and locality everywhere else.
The widget below makes the two cost curves concrete. As you raise the resolution at which a layer operates, the convolution's cost rises linearly (one orange line) while attention's rises quadratically (the steep teal curve). The gap between the lines is the budget you save by keeping attention at low resolution. Watch where attention overtakes conv and becomes unaffordable — that crossover is exactly the resolution below which hybrids put their attention blocks.
Slide the operating resolution (token-grid side). Convolution cost (orange) rises gently; attention cost (teal) explodes. The vertical marker shows where you sit. Hybrids keep attention left of the crossover — low resolution, cheap.
MobileViT (Mehta & Rastegari, Apple, 2021) is the cleanest first hybrid to study because its design is a literal sandwich: convolutions on the outside, a transformer in the middle, and a clever trick to feed pixels to the transformer and back. It was built to answer "how do I give a MobileNet a global view without paying ViT prices?" and its answer is worth tracing tensor-by-tensor.
Step 1 — the conv stem. MobileViT begins like any mobile CNN: a stack of cheap convolutions, including the depthwise separable convolutions (a convolution split into a per-channel spatial filter plus a 1×1 channel mixer, which costs a fraction of a full conv) that MobileNet made famous. This stem does the local work — edges, textures, color blobs — and downsamples the image. After the stem you hold a feature map of shape, say, (C, H, W) = (96, 32, 32): 96 channels over a 32×32 grid. Cheap, local, and already smaller than the input.
Step 2 — the unfold trick (this is the heart of it). A transformer wants a sequence of tokens, but a feature map is a 2D grid. MobileViT does not flatten the whole grid into one giant sequence (that would be expensive and would lose 2D structure). Instead it unfolds the grid into small patches — say 2×2 — and runs the transformer across the patches, with each of the 4 pixel-positions inside a patch handled as its own parallel sequence. Concretely: a 32×32 grid of 2×2 patches gives a 16×16 = 256 grid of patches, and the transformer attends among those 256 positions. So the transformer sees 256 tokens, not 1,024 — and it operates at low resolution by design.
Step 3 — the transformer block. Standard self-attention runs over those unfolded tokens. Because every token attends to every token, a patch in the top-left can now directly influence a patch in the bottom-right — the global view a pure CNN lacked. This is the one place MobileViT pays for globality, and it pays at low resolution where it is affordable.
Step 4 — fold back and fuse. The globally-mixed tokens are folded back into the original 2D grid shape (the inverse of unfold), then concatenated with the original pre-transformer features and fused with a convolution. This fusion is deliberate: it keeps the cheap local features and adds the new global information, rather than discarding one for the other. The output is again (C, H, W), ready for the next stage of convolutions.
Let us put one number on the savings. Suppose the feature map is 32×32 = 1,024 spatial positions. Flattening all of them into one attention sequence would cost 1,0242 ≈ 1.05 million score entries. MobileViT's unfold into 2×2 patches instead runs attention over 16×16 = 256 patch-positions (with the 4 intra-patch positions batched in parallel), costing 2562 ≈ 65,500 entries per intra-patch position × 4 ≈ 262,000 — about 4× cheaper than naive full attention, while still giving every region a global path. The unfold is not a detail; it is the cost-control mechanism.
The widget below animates the MobileViT data flow. Step through stem → unfold → transformer → fold → fuse and watch the tensor shape and the "what kind of mixing is happening" label change at each stage. Notice that attention appears exactly once in the middle, sandwiched by convolution on both sides.
Step through the pipeline. The shape and the mixing type (local conv vs global attention) update at each stage. Attention is the single guest in the middle of a convolutional sandwich.
MobileViT showed one specific sandwich. Step back and you find the general pattern that nearly every modern hybrid block follows, whether it is MobileViT, FastViT, EfficientViT, or EdgeNeXt. The pattern is so consistent it is worth naming as a recipe: local mixing, then global mixing, with a channel mixer to finish. Once you see the three slots, every hybrid block becomes a filling-in of the same template.
Slot 1 — local token mixing (convolution). First, mix each position with its spatial neighbors using a cheap convolution — usually a depthwise convolution (one spatial filter per channel). This is the token mixer for nearby positions: it cleans up local structure, sharpens edges, and crucially injects a locality prior that pure attention lacks. It is cheap because depthwise convs cost almost nothing.
Slot 2 — global token mixing (attention or an efficient stand-in). Then mix every position with every other position. In MobileViT this is full attention; in EfficientViT it is a cheaper cascaded-group attention (Chapter 5); in some blocks it is a large-kernel conv approximating globality. Whatever fills it, this slot is where long-range information flows. It runs at the block's (low) resolution so its quadratic cost is contained.
Slot 3 — channel mixing (the MLP / 1×1 conv). Finally, mix information across channels at each position independently, using a small two-layer MLP (equivalently, two 1×1 convolutions with an activation between). The first two slots moved information across space; this slot moves it across features. Every transformer and every hybrid has this slot; it is where much of the parameters and FLOPs actually live.
Tie them together with residual connections (add the input back to each slot's output) and you have a hybrid block. The data flow for one position is: take its feature vector, add neighborhood context (local conv), add global context (attention), then re-mix its channels (MLP), keeping a skip path around each step so gradients flow and the block can learn to do nothing where nothing is needed.
The widget below builds one hybrid block and lets you toggle each slot on or off, showing how information propagates through a small token grid. With only the local slot, a token's influence stays in its neighborhood (a small spreading blob). Add the global slot and one token's information jumps across the whole grid instantly. Add the channel slot and you see the per-token feature re-mixing. Turning slots on and off makes the division of labor concrete.
A pulse starts at the center token. Toggle the three slots and watch how far and how its influence spreads. Local conv = neighborhood blob; global attention = instant jump across the grid; channel MLP = per-token re-mix (shown as recoloring).
Here is a trick that feels like cheating and isn't. FastViT (Apple, 2023) and the RepVGG line before it use structural reparameterization: train the network with a complicated multi-branch block for accuracy, then algebraically collapse that block into a single simple convolution for inference, with identical outputs. You get the training benefits of a rich block and the inference speed of a plain one. No accuracy is lost; the speedup is exact and free.
Why multiple branches help training. A block with several parallel paths — a 3×3 conv, a parallel 1×1 conv, and an identity skip, all added together — gives the optimizer more routes for gradients and a richer function class, which trains to higher accuracy than a single 3×3 conv. But at inference those parallel branches mean multiple memory reads and kernel launches, which is slow on a phone. Reparameterization keeps the accuracy and deletes the slowness.
The key fact that makes it possible: convolution is a linear operation, and the sum of several linear operations is itself one linear operation. If branch A computes conv(x, WA) and branch B computes conv(x, WB) over the same input, then A(x) + B(x) = conv(x, WA + WB) — provided we first express both kernels at the same size. So three branches added together equal one convolution whose kernel is the sum of the three branches' kernels, after padding them to a common shape. We never approximate; we just add the weights.
Let us fold a block by hand. Our block has three parallel branches applied to a single-channel input, all summed:
Read what just happened. The 1×1 conv and the identity skip both only affect the center tap of a 3×3 kernel, because they have no spatial reach — they touch only the pixel under them. So "merge a 1×1 and an identity into a 3×3" reduces to "add their scalars to the center of the 3×3." That is the entire algebra. (In real networks there is a second wrinkle: each branch usually has a BatchNorm, which is also linear at inference and folds into the conv's weights and bias first — same principle, fold the linear BN into the linear conv, then sum branches.)
The widget below performs this fold live. Adjust the three branches' contributions and watch the single fused 3×3 kernel update — and verify that applying the fused kernel to a test patch gives the identical result to running the three branches separately and summing. The "match" indicator stays green because the fold is exact, not approximate.
Set each branch's strength. The fused 3×3 kernel (right) is the cell-wise sum. The readout proves three-branches-summed equals one-fused-conv on a test patch — the same number, always.
FastViT layers this idea on top of the hybrid template from Chapter 3: its token mixer is a reparameterizable convolution that is multi-branch during training and a single conv at inference, and it removes some normalization and activation cost too. The result — reported in the FastViT paper — is a model that matches the accuracy of heavier hybrids while running markedly faster on an iPhone, because at inference its blocks are structurally as cheap as plain CNN blocks. The accuracy came from the rich train-time block; the speed came from collapsing it.
FastViT made the convolutions cheaper. EfficientViT (Liu et al., 2023) attacks the other slot: it makes the attention cheaper, by noticing that standard multi-head attention is wastefully redundant. Understanding its cascaded group attention (CGA) means first seeing the waste, then seeing the fix.
The waste: redundant heads. Multi-head attention runs several attention "heads" in parallel, each over the full feature vector, then concatenates them. The EfficientViT authors found that many heads learn nearly the same attention pattern — they are computing redundant maps over the same input, burning compute to rediscover the same thing. If four heads all attend roughly the same way, three of them are wasted work.
Fix part one — feature grouping. Instead of feeding every head the whole feature vector, CGA splits the channels into groups and gives each head only its own group. Head 1 sees channels 0–31, head 2 sees channels 32–63, and so on. Each head now operates on a smaller slice, so each head is cheaper, and because the heads see different inputs they are forced to learn different patterns — the redundancy is structurally removed, not just discouraged.
Fix part two — cascading. Grouping alone would make heads independent and slightly weaken them (each sees less). So CGA cascades: the output of head 1 is added into the input of head 2, head 2's output into head 3, and so on. Information flows down the chain of heads, so a later head benefits from earlier heads' work even though it only directly sees its own channel group. This recovers the representational power of full attention at a fraction of the cost — each head is small, but the cascade lets the group act like a deeper attention.
Put one number on the saving. Suppose C = 128 channels and 4 heads. Standard multi-head attention projects the full 128-dim input for each head's query/key/value — the projection cost scales with C per head, so ≈ 4 × (work over 128 dims). CGA gives each head only 128/4 = 32 channels, so each head's projection is over 32 dims — ≈ 4 × (work over 32 dims), roughly a 4× reduction in the QKV projection cost, with the cascade adding only cheap additions. You traded a little per-head width for a lot of saved compute, and the diversity-by-grouping pays the accuracy back.
The widget below contrasts the two. On the left, standard multi-head attention: four heads, all fed the full feature, and watch how similar their attention maps become (redundancy). On the right, cascaded group attention: four heads, each fed a different channel group, with the cascade arrows showing information chaining head-to-head — the maps stay diverse. Toggle and compare the total cost bars.
Switch between the two. Standard heads all see the full feature and their maps collapse to near-duplicates (wasted compute). Cascaded group attention splits channels per head, chains them, and keeps the maps diverse at lower cost.
Beyond CGA, EfficientViT makes two more mobile-minded choices worth knowing: it uses attention sparingly (sandwiched between cheap convolutions, like every hybrid in this lesson) and it favors operations that are friendly to hardware — avoiding the reshape-heavy, memory-bound patterns that make naive attention slow even when its FLOPs look fine. That second point — "friendly to hardware" — is the entire subject of the next chapter, because it is the gap between the theory you've learned and the milliseconds you actually measure.
Here is the trap that has sunk more mobile models than any other. You optimize your network to have fewer FLOPs (floating-point operations — the raw count of multiply-adds), you ship it, and it runs slower than the model with more FLOPs that you replaced. The leaderboard FLOPs number lied. Understanding why is the difference between a model that looks fast on paper and one that is fast in the user's hand.
FLOPs measure arithmetic; latency measures arithmetic plus everything else. A processor spends real time on three things the FLOP count ignores: moving data between memory and compute units (memory bandwidth), launching kernels (each operation has fixed overhead), and waiting when one of these stalls the others. A layer can have tiny FLOPs yet be slow because it is memory-bound — the chip spends its time shuttling data, not computing.
The concept that explains it: arithmetic intensity. Define arithmetic intensity as FLOPs divided by bytes moved — how much compute you do per byte you fetch. A chip can do, say, 100 multiply-adds in the time it takes to fetch one byte from memory. If a layer's arithmetic intensity is below that ratio, the compute units sit idle waiting for data — you are memory-bound, and reducing FLOPs does nothing because FLOPs were never the bottleneck. If it is above, you are compute-bound, and FLOPs matter.
This is exactly why mobile-vision papers stopped reporting FLOPs as the headline and started reporting measured latency on a specific device (FastViT reports iPhone latency; EfficientViT reports throughput on real GPUs and edge chips). It is also why certain operations are quietly avoided: frequent reshapes and transposes (attention is full of them) are memory traffic with near-zero FLOPs; activation functions like GELU are cheaper in FLOPs than they are in latency because of how they are computed; and depthwise convolutions, beloved for their low FLOPs, are often memory-bound and underperform their FLOP count on real chips. The FLOP-optimal model and the latency-optimal model are frequently not the same model.
There is a second, brutal reality on top of arithmetic intensity: the hardware's actual support. Mobile chips have a fixed-function accelerator (Apple's Neural Engine, others' NPUs) that runs convolutions blazingly fast but may not support some attention operations, forcing them onto the slower CPU or GPU — or splitting the model across processors, which adds expensive hand-offs. A theoretically cheap attention block can be slow simply because the fast accelerator can't run it. Hybrids that lean on convolution win partly because convolution is the operation every accelerator is built to love.
The widget below is a roofline-style explorer. Pick an operation type; the dot lands at its arithmetic intensity. To the left of the chip's "ridge" you are memory-bound (latency set by bytes moved, lowering FLOPs won't help); to the right you are compute-bound (latency set by FLOPs). Drag operations around and watch a layer with fewer FLOPs sit in the slow memory-bound zone while a higher-FLOP layer runs faster. Seeing this kills the "lower FLOPs = faster" reflex for good.
Drag arithmetic intensity. Left of the ridge = memory-bound (cutting FLOPs is wasted effort). Right of the ridge = compute-bound (FLOPs matter). The colored dots are typical mobile ops — notice low-FLOP ops can land in the slow zone.
Everything converges here. This is a hybrid backbone you assemble and tune, watching receptive field, cost, and on-device latency move in real time. You control the mix of conv and attention, the resolution at which attention runs, whether reparameterization is applied, and whether attention uses cascaded group attention — every lever from Chapters 1 through 6, in one panel. The goal: find a configuration that reaches a wide receptive field while staying inside a phone's latency budget.
The left side shows a token grid with a pulse at the center; the spreading glow is the network's effective receptive field — how far one token's information reaches. The right side shows three live bars: receptive field (want it large), FLOPs, and measured latency (want it under the red budget line). Watch how the levers trade against each other — and watch FLOPs and latency diverge, exactly as Chapter 6 promised.
Things to try, each replaying a chapter:
• Slide the conv/attention mix toward attention. Receptive field jumps (Chapter 1) — but watch latency climb, especially if attention runs at high resolution.
• Now drop the attention resolution. Same receptive-field benefit, far less cost — the core hybrid trick (Chapters 1–2): cheap globality at low resolution.
• Toggle reparameterization ON. FLOPs barely move but latency drops — the multi-branch conv collapsed to one kernel (Chapter 4). Free speed.
• Toggle cascaded group attention ON. The attention slot gets cheaper without shrinking the receptive field (Chapter 5).
• Watch FLOPs and latency disagree. Push attention resolution up: FLOPs and latency both rise, but latency rises faster because high-resolution attention is memory-bound (Chapter 6).
Set the mix, the attention resolution, and the two efficiency toggles. The glow is effective receptive field; the bars are RF, FLOPs, and measured latency. Stay under the red latency budget while keeping RF high — that's the whole job.
If you removed this simulation, would you lose understanding? Yes — because "mostly conv with a little low-res attention is the sweet spot" is abstract until you watch the latency bar punch through the red budget the instant you raise attention resolution, then drop back under it when you reparameterize. The napkin drawing of this entire lesson is this panel: levers in, receptive field and latency out, and the budget line you must respect.
You now know how hybrids work; a competent engineer also knows when to use one and when something simpler or something heavier is the right call. Hybrids are not free lunches in every regime, and pretending otherwise leads to over-engineering. Three regimes, three answers.
Regime 1 — on-device, real-time, mid-accuracy: hybrids win, decisively. When you must run in real time on a phone or embedded chip and you need better accuracy than a plain MobileNet can give, the hybrid is the dominant choice. The convolutions keep it fast and accelerator-friendly; the sprinkle of attention buys the long-range reasoning that pushes accuracy past a pure CNN of the same speed. This is the home turf: video-call segmentation, on-device photo tagging, AR, robot perception. MobileViT, FastViT, and EfficientViT all target exactly here.
Regime 2 — tiny budget, simple task: pure conv often wins. If your task is easy (few classes, objects that are locally distinguishable) and your budget is brutal (a microcontroller, sub-millisecond), the attention slot is overhead you cannot afford and may not need. A well-tuned MobileNet or a tiny RepVGG can beat a hybrid here, because the task's signal is local and attention's globality buys nothing while costing latency and accelerator headaches. Don't pay for globality you won't use.
Regime 3 — huge data, huge compute, accuracy is everything: pure ViT (or a giant model) often wins. In the data center with billions of pretraining images and no latency limit, a large pure Vision Transformer or a foundation model edges out hybrids on top accuracy — attention's flexibility, given enough data, beats the conv locality prior. Hybrids trade a sliver of peak accuracy for a mountain of efficiency; when efficiency is free (cloud) and data is unlimited, that trade no longer favors the hybrid. The hybrid's locality prior is a head start most valuable when data or compute is scarce.
One more axis decides it in practice: your deployment target's hardware. If the chip's accelerator runs convolution but stumbles on attention ops, even a theoretically-cheap attention block can be slow — nudging you toward conv-heavier hybrids or pure conv. If the chip handles attention well (newer NPUs increasingly do), you can afford a more attention-rich hybrid. The "right" model is not an abstract leaderboard rank; it is the best accuracy-per-millisecond on the specific silicon you ship to, which is why the same team will pick different backbones for different phones.
The widget below is a decision map. Set your data scale, your latency budget, and your accelerator's attention support; the map points to the regime — pure conv, hybrid, or pure ViT — and names a representative model. Move the sliders and watch the recommendation cross regime boundaries. There is no single best backbone; there is a best backbone for your point in this space.
Set your data scale and latency budget. The shaded region and the label show the recommended regime (pure conv / hybrid / pure ViT) and a representative model. Watch the boundary move as constraints change.
Conv-transformer hybrids are a crossroads, not a dead end. The two ideas you fused here — convolution's cheap locality and attention's expensive globality — recur everywhere in efficient deep learning, and the tricks you learned (reparameterization, efficient attention, the FLOPs-vs-latency reality) generalize far beyond mobile vision. Naming the connections turns one lesson into a map.
The two halves, taught in depth elsewhere. The convolutional half — what a filter does, depthwise separable convs, how receptive fields grow — lives in the broader vision lessons. The attention half — queries, keys, values, multi-head, the quadratic cost — is the engine of the Vision Transformer and the Transformer. This lesson is, in one sense, the marriage of those two lessons under a hardware constraint; if either half felt shaky, those are the places to firm it up.
The same backbone, different head. Everything here was about the backbone — the feature extractor. Bolt a classification head on it and you have on-device image classification; bolt a segmentation or detection head on it and you have the video-call segmenter from Chapter 0. The whole-image-understanding cousin of this task — reading a global label from the whole frame — is exactly Scene Classification (see its Chapter 5, "From CNN to Vision Transformer," for the same conv→attention arc from the accuracy side rather than the latency side). The mobile backbones here are precisely the engines that make scene classification, detection, and segmentation run on a phone.
The efficiency tricks generalize. Structural reparameterization (Chapter 4) is used far beyond FastViT — any time a train-time-rich, inference-cheap block helps. Efficient-attention ideas (Chapter 5) connect to the entire literature on making attention sub-quadratic. And the FLOPs-vs-latency lesson (Chapter 6) is the single most transferable thing here: it governs LLM inference, training throughput, and any time someone shows you a FLOP count and calls it speed. Whenever you see "fewer FLOPs," ask "but what's the arithmetic intensity, and what does it measure on the real device?"
Where the field is going. Newer mobile backbones (the EfficientViT and FastViT successors, EdgeNeXt, MobileOne, and the steady stream after them) keep refining the same recipe: more conv where conv is cheap, cheaper attention where globality is needed, more reparameterization, and ever-more hardware-aware operator choices. The frontier is co-designing the model with the chip — neural architecture search that optimizes for measured latency on a specific accelerator, which is where this lesson connects to architecture-search-driven backbone design. The template is stable; the filling keeps improving.
| Thing | What to remember |
|---|---|
| The tension | Conv = cheap + local (small receptive field); attention = expensive + global (full RF at layer 1, cost quadratic in tokens) |
| Hybrid recipe | Conv does high-res local work + downsampling; attention runs only at LOW resolution where it's affordable |
| MobileViT | Conv stem → unfold to patches → transformer → fold → fuse with original features; attention is a guest in a conv sandwich |
| Hybrid block | Three slots: local mix (depthwise conv) → global mix (attention) → channel mix (MLP), each with a residual |
| FastViT / reparam | Multi-branch (3×3 + 1×1 + identity) at train time, summed to ONE conv at inference — exact, free speed |
| EfficientViT / CGA | Split channels per head (cheaper, forced-diverse) + cascade head outputs — cuts attention redundancy |
| FLOPs ≠ latency | Latency = compute + memory + launch overhead; low arithmetic intensity = memory-bound = cutting FLOPs won't help. MEASURE on device. |
| When to use | Hybrid = constrained compute + need global reasoning. Pure conv = tiny budget, local task. Pure ViT = huge data + no latency limit. |
"Make it work, make it right, make it fast." — Kent Beck. The conv-transformer hybrid is the third clause taken seriously: a model that is already right, engineered to be fast on the hardware in someone's pocket.