Efficient Model
Architectures

Chapter 1: Standard vs Depthwise Separable

The single most important efficiency trick in deep learning is the depthwise separable convolution. It splits one expensive operation into two cheap ones — and loses almost nothing in accuracy.

Let's understand what a standard convolution actually computes. At each spatial position, a K×K×C_in patch is dot-producted with a filter of the same size, producing one output value. You need C_out such filters to produce all output channels.

The key insight: spatial mixing and channel mixing don't have to happen at the same time. We can separate them.

Step 1: Depthwise Convolution (spatial only)

Apply a separate K×K filter to EACH input channel independently. No cross-channel interaction. Each channel gets its own spatial filter.

Step 2: Pointwise Convolution (channel only)

Apply 1×1 convolutions to mix channels. This is just a matrix multiply at each spatial position — a linear projection from C_in dimensions to C_out dimensions.

Total depthwise separable cost:

Savings ratio:

For K=3, C_out=256: savings = 9×256 / (9+256) = 2304/265 ≈ 8.7×

python
import torch.nn as nn

# Standard convolution: mixes spatial + channels together
standard = nn.Conv2d(256, 256, kernel_size=3, padding=1)
# Parameters: 3*3*256*256 = 589,824

# Depthwise separable: two steps
depthwise = nn.Conv2d(256, 256, kernel_size=3, padding=1, groups=256)
pointwise = nn.Conv2d(256, 256, kernel_size=1)
# Parameters: 3*3*256 + 256*256 = 2,304 + 65,536 = 67,840
# That's 8.7x fewer parameters too!

def depthwise_separable(x):
    x = depthwise(x)   # spatial filtering per channel
    x = pointwise(x)   # channel mixing
    return x

Chapter 2: MobileNet & Inverted Residuals

In 2017, Google published MobileNetV1: stack depthwise separable convolutions, add batch normalization and ReLU between each step, and you get a network that's 8× cheaper than VGG with similar accuracy. Simple and beautiful.

But V1 had a problem. The depthwise step operates on each channel independently — it can't learn cross-channel features at that stage. If you have a narrow bottleneck (few channels), the depthwise filter is starved of information. You need enough channels for the spatial filter to work with.

MobileNetV2: The Inverted Residual Block

The solution is counterintuitive. A standard residual block (ResNet) goes wide→narrow→wide: compress channels with 1×1, do the expensive 3×3 in the narrow space, then expand back. MobileNetV2 does the opposite.

Why "inverted"? Because the residual connection skips across the narrow bottleneck, not the wide inner representation. The data lives in the skinny state; the fat state is temporary, just for spatial filtering.

The expansion factor t (typically 6) controls how wide the internal representation gets. With t=6 and a 24-channel bottleneck, the depthwise step works on 144 channels — plenty of capacity for rich spatial features.

Worked Example: FLOPs for One Inverted Residual Block

Input: 24 channels, 56×56 spatial, expansion t=6.

Compare to a standard 3×3 conv with 144 input and 144 output channels at 56×56:

The inverted residual is 23× cheaper — while maintaining similar representational capacity.

python
class InvertedResidual(nn.Module):
    def __init__(self, c_in, c_out, stride, expand_ratio):
        super().__init__()
        c_mid = c_in * expand_ratio
        self.use_residual = (stride == 1 and c_in == c_out)

        layers = []
        if expand_ratio != 1:
            # Expand: 1x1 conv to widen channels
            layers += [nn.Conv2d(c_in, c_mid, 1),
                       nn.BatchNorm2d(c_mid), nn.ReLU6()]
        # Depthwise: 3x3 spatial filtering
        layers += [nn.Conv2d(c_mid, c_mid, 3, stride,
                             padding=1, groups=c_mid),
                   nn.BatchNorm2d(c_mid), nn.ReLU6()]
        # Project: 1x1 conv to narrow channels (NO activation!)
        layers += [nn.Conv2d(c_mid, c_out, 1),
                   nn.BatchNorm2d(c_out)]
        self.conv = nn.Sequential(*layers)

    def forward(self, x):
        if self.use_residual:
            return x + self.conv(x)
        return self.conv(x)

Chapter 3: EfficientNet & Compound Scaling

You have a baseline network that works. Now you want to make it bigger for higher accuracy. You have three knobs to turn: make it deeper (more layers), wider (more channels per layer), or increase resolution (bigger input images). Which do you turn?

Most people pick one. ResNet scales depth (18→34→50→101→152). WideResNet scales width. But the EfficientNet paper (Tan & Le, 2019) discovered something profound: scaling all three together, in a specific ratio, is dramatically better than scaling any one dimension alone.

Why those exponents? FLOPs scale as d × w² × r² (depth is linear, width and resolution are quadratic in their effect on compute). The constraint α·β²·γ² ≈ 2 ensures that each unit increase in φ roughly doubles the total FLOPs.

EfficientNet's specific values:

Check: 1.2 × 1.1² × 1.15² = 1.2 × 1.21 × 1.3225 = 1.92 ≈ 2 ✓

Worked Example: Scaling B0 to B3

EfficientNet-B0 baseline: 18 layers, 32 base width, 224×224 input.

For B3, φ = 3:

The result: EfficientNet-B3 achieves 81.6% ImageNet accuracy with only 1.8B FLOPs — compared to ResNet-152 at 78.3% accuracy with 11.6B FLOPs. Higher accuracy, 6.4× less compute.

python
import math

# EfficientNet compound scaling
alpha = 1.2   # depth coefficient
beta  = 1.1   # width coefficient
gamma = 1.15  # resolution coefficient

def scale_model(phi, base_depth=18, base_width=32, base_res=224):
    d = math.ceil(base_depth * alpha**phi)
    w = math.ceil(base_width * beta**phi)
    r = math.ceil(base_res * gamma**phi)
    flop_mult = 2**phi  # approximately
    return {'depth': d, 'width': w, 'resolution': r,
            'flop_mult': flop_mult}

# EfficientNet-B3
b3 = scale_model(phi=3)
print(b3)  # {'depth': 32, 'width': 43, 'resolution': 341, 'flop_mult': 8}

Chapter 4: Neural Architecture Search

What if instead of a human designing the architecture, we let an algorithm search for one? That's Neural Architecture Search (NAS) — automated discovery of optimal network structures.

The setup: define a search space (what operations are allowed? how many layers? which connections?), a search strategy (how do we explore?), and an evaluation method (how do we score each candidate?).

Search Space

A typical cell-based search space (NASNet) defines:

• Operations: 3×3 conv, 5×5 conv, 3×3 depthwise, max pool, avg pool, skip, zero
• Connections: which pairs of nodes have edges
• Cell structure: how to wire 5-7 nodes into a computational cell
• Macro structure: how many cells, where to downsample

The Cost Problem

Naive NAS (Zoph & Le, 2017): train each candidate architecture to convergence. Evaluate thousands. At 1 GPU-hour per evaluation, that's thousands of GPU-hours — absolutely impractical for most teams.

One-Shot NAS (Weight Sharing)

The breakthrough: train a single supernet that contains ALL possible architectures as sub-networks. Each candidate is a subset of the supernet's weights. To evaluate a candidate, just mask the supernet to that subset — no retraining needed.

python
# Simplified one-shot NAS supernet
class SupernetCell(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.ops = nn.ModuleList([
            nn.Conv2d(channels, channels, 3, padding=1),     # op 0: 3x3 conv
            nn.Conv2d(channels, channels, 5, padding=2),     # op 1: 5x5 conv
            nn.Conv2d(channels, channels, 3, padding=1,      # op 2: depthwise
                      groups=channels),
            nn.MaxPool2d(3, stride=1, padding=1),          # op 3: max pool
            nn.Identity(),                                    # op 4: skip
        ])

    def forward(self, x, arch_choice):
        # arch_choice selects which operation to use
        return self.ops[arch_choice](x)

# During search: sample arch_choice randomly per step
# After search: fix arch_choice to best found architecture

Chapter 5: Mobile Deployment

You've designed an efficient architecture. You've trained it in PyTorch or TensorFlow on a beefy GPU server. Now you need to run it on a phone, a drone, or an embedded sensor. The gap between "works in my notebook" and "runs at 30fps on an iPhone" is enormous.

The problem: different hardware requires different representations. Your GPU training framework stores the model as a Python object graph with floating-point weights. A mobile chip needs a compact binary with quantized weights and fused operations.

Export Formats

Graph Optimizations

Before deploying, the compiler performs transformations that don't change the math but dramatically speed up execution:

• Operator fusion: Conv + BatchNorm + ReLU becomes one kernel call instead of three
• Constant folding: Pre-compute anything that doesn't depend on input
• Dead code elimination: Remove unused branches
• Layout optimization: Rearrange memory for target hardware (NCHW vs NHWC)

Quantization

The biggest single deployment optimization. Convert FP32 weights and activations to INT8:

This gives 4× memory reduction, 2-4× speedup on INT8-capable hardware, and typically less than 1% accuracy loss with proper calibration.

python
import torch
import onnx
import onnxruntime as ort

# Step 1: Export PyTorch model to ONNX
model.eval()
dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, dummy, "model.onnx",
                  input_names=["image"],
                  output_names=["logits"],
                  dynamic_axes={"image": {0: "batch"}})

# Step 2: Optimize with ONNX Runtime
sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
session = ort.InferenceSession("model.onnx", sess_options)

# Step 3: Quantize to INT8
from onnxruntime.quantization import quantize_dynamic
quantize_dynamic("model.onnx", "model_int8.onnx",
                 weight_type=QuantType.QInt8)

Chapter 6: Hardware Acceleration

Architecture efficiency doesn't exist in a vacuum — it exists relative to hardware. An operation that's "efficient" in FLOPs might be slow if the hardware can't execute it well. Understanding hardware is how you design architectures that are fast in practice, not just in theory.

Hardware Taxonomy

The Roofline Model

Performance is limited by either compute (how fast the hardware can multiply) or memory bandwidth (how fast data can be moved). The arithmetic intensity (FLOPs per byte loaded) determines which bottleneck dominates.

Dense matrix multiply has high arithmetic intensity (~O(N) FLOPs per byte) — it's compute-bound on modern hardware. Depthwise convolution has low arithmetic intensity (only K² FLOPs per element loaded) — it's often memory-bound. This is why depthwise conv is "efficient" in FLOPs but not always fast in practice.

python
# Arithmetic intensity examples

# Dense matmul: C = A @ B, A is MxK, B is KxN
# FLOPs: 2*M*K*N
# Bytes: (M*K + K*N + M*N) * bytes_per_elem
# Intensity: 2*M*K*N / ((M*K + K*N + M*N) * 4) ≈ N/2 for large square

# Depthwise 3x3 conv: C channels, HxW spatial
# FLOPs: 9 * C * H * W
# Bytes: (C*H*W input + 9*C weights + C*H*W output) * 4
# Intensity: 9*C*H*W / (2*C*H*W + 9*C) * 4 ≈ 9/8 ≈ 1.1
# Very low! Memory-bound on most hardware.

def arithmetic_intensity(op_type, **kwargs):
    if op_type == "matmul":
        M, K, N = kwargs['M'], kwargs['K'], kwargs['N']
        flops = 2 * M * K * N
        bytes_moved = (M*K + K*N + M*N) * 4
    elif op_type == "depthwise":
        C, H, W, K = kwargs['C'], kwargs['H'], kwargs['W'], kwargs['K']
        flops = K*K * C * H * W
        bytes_moved = (2*C*H*W + K*K*C) * 4
    return flops / bytes_moved

Chapter 7: Architecture × Hardware Co-Design (Showcase)

This is the capstone. Everything comes together: depthwise separable blocks, compound scaling, hardware awareness, and deployment constraints. You're the architect now. Your job: design a network that meets a specific latency budget on a specific device while maximizing accuracy.

The key insight of hardware-aware design: there's no single "best" architecture. The best architecture depends on:

• Target hardware: GPU favors large matmuls; NPU favors depthwise ops; CPU is flexible but slow
• Latency budget: 1ms = tiny model; 10ms = MobileNet-class; 100ms = EfficientNet-class
• Task complexity: Classification needs fewer features than detection or segmentation
• Memory limit: Mobile has 1-4GB; must fit model + activations + framework overhead

Design Heuristics by Hardware

Chapter 8: Emerging Architectures

Efficient convolutions were the story from 2017-2020. The frontier has moved. Vision Transformers, state-space models, and hardware-software co-optimizations are defining the next generation of efficient architectures.

Vision Transformers: Making Attention Efficient

The Vision Transformer (ViT) splits an image into 16×16 patches, projects each to an embedding, and applies transformer layers. The problem: self-attention is O(N²) in sequence length. For a 224×224 image with 16×16 patches, N = 196. Manageable. But for higher resolution or dense prediction, N explodes.

Efficiency tricks:

• Window attention (Swin): Compute attention only within local windows of 7×7 patches. Shift windows between layers for cross-window communication. Complexity: O(N × W²) instead of O(N²).
• Token pruning: Remove uninformative tokens mid-network. If a patch is "boring" (low attention from CLS token), drop it. Saves 30-50% compute with <0.5% accuracy loss.
• Linear attention: Replace softmax(QK^T)V with φ(Q)(φ(K)^TV). The kernel trick makes attention O(N) instead of O(N²).

State-Space Models (Mamba)

Mamba (Gu & Dao, 2023) offers an entirely different approach to sequence modeling. Instead of attention (O(N²)) or even linear attention (O(N) but with reduced capacity), Mamba uses a selective state-space model that processes sequences in O(N) time with O(1) memory per step during inference.

The key innovation: A, B, C are input-dependent (selective), allowing the model to decide what to remember and what to forget — similar to a gated RNN, but with the parallelizable structure of SSMs during training.

FlashAttention: Hardware-Aware Algorithm Design

FlashAttention doesn't change the math of attention — it changes how it's computed to exploit the GPU memory hierarchy. Standard attention materializes the N×N attention matrix in GPU HBM (slow global memory). FlashAttention tiles the computation so that it stays in SRAM (fast on-chip memory).

The speedup: 2-4× faster, 5-20× less memory. This isn't approximation — it's exact attention, just computed more cleverly relative to hardware.

python
# FlashAttention: exact same math, hardware-aware implementation
# Standard (slow, O(N²) memory):
# attn = softmax(Q @ K.T / sqrt(d)) @ V

# FlashAttention (fast, O(N) memory):
# Tiles Q, K, V into blocks that fit in SRAM
# Computes attention block-by-block using online softmax
# Never materializes the full N×N matrix

from flash_attn import flash_attn_func

# Drop-in replacement: same input/output, 2-4x faster
output = flash_attn_func(q, k, v, causal=True)

# Memory comparison for sequence length 4096:
# Standard: 4096 × 4096 × 2 bytes = 32 MB per head
# FlashAttention: O(block_size) ≈ 256 KB per head

Chapter 9: Mastery & Connections

You now understand the full stack of efficient architecture design: from the fundamental compute savings of depthwise separable convolutions, through compound scaling and automated search, to hardware-aware deployment and emerging paradigms.

Architecture Selection Flowchart

FLOP/Parameter/Latency Cheat Sheet

Derivation: Depthwise Separable Savings

Let's formally prove the savings ratio. For a layer with kernel K, C_in input channels, C_out output channels, and H×W spatial:

As C_out → ∞, ratio → K². For K=3: maximum savings is 9×. For typical C_out=256: savings = 9×256/(9+256) = 8.7×. For C_out=64: savings = 9×64/(9+64) = 7.9×. The savings improve with more output channels.

Connections

This lesson connects to many others in the series:

• Transformers — the architecture that FlashAttention and window attention make efficient
• SSM/Mamba — the O(N) alternative to attention for sequence modeling
• Diffusion Models — efficiency here means faster sampling (fewer steps, distillation)
• VLAs — robot policies that MUST run efficiently in real-time control loops