InternImage: Deformable Convolutions at Scale

Chapter 1: From Convolution to Deformable

Write a standard convolution as a sum over a fixed set of offsets. For an output location p₀ and a 3×3 kernel, define the grid R = {(-1,-1), (-1,0), …, (1,1)} — nine offsets. With kernel weights w_k and input feature map x:

Here K = 9, p_k ranges over R, and w_k is a learned weight (one per kernel tap, per output channel). Two things about this equation are frozen: the offsets p_k are integers fixed at every location, and the weights w_k are the same for every image.

Deformable convolution v1 (Dai et al., 2017) unfroze the first frozen thing: it added a learned offset Δp_k to each tap, predicted from the input by a small offset-network:

Now each tap can move. Because Δp_k is generally fractional, x(·) at a non-integer location is read by bilinear interpolation (Chapter 2). DCNv2 (Zhu et al., 2019) unfroze a second thing by adding a per-tap scalar modulation m_k ∈ [0,1], letting the operator turn a tap down or off:

InternImage's DCNv3 makes three more changes — and the cleanest way to see them is to notice what is still wrong with DCNv2 at scale. We'll preview them here and derive each carefully:

A 1×1 worked example: why the offset matters

Take a tiny 1-D signal x = [0, 0, 9, 0, 0] (a spike at index 2), and a single output at p₀ = 0 with one tap (K=1, w=1). A rigid conv with offset 0 reads x(0) = 0 — it completely misses the spike. Give the tap a learned offset Δp = +2: it now reads x(0 + 0 + 2) = x(2) = 9. The same weight, the same compute — but by moving the sample, the operator captured the signal a rigid kernel was blind to. That single degree of freedom is the entire idea; the rest is bookkeeping for fractional offsets, channels, and stability.

Chapter 2: Deformable Sampling & Bilinear Reads

An offset Δp_k is a real-valued 2-vector, so p₀ + p_k + Δp_k almost never lands on an integer pixel. We need to read x at a fractional location, and we need it differentiable so gradients can flow back to the offset network. The answer is bilinear interpolation, and deriving it by hand is the most important skill in this whole paper.

Let the target location be p = (px, py), fractional. It sits inside a unit square whose four integer corners are the floor and ceil of each coordinate. Write x0 = ⌊px⌋, y0 = ⌊py⌋, x1 = x0+1, y1 = y0+1, and the fractional parts a = px − x0, b = py − y0 (both in [0,1)). The four corner values are the actual stored pixels. Bilinear interpolation is a weighted average where each corner's weight is the area of the opposite sub-rectangle:

Read it as: the closer p is to a corner (small distance → the opposite area is large), the more that corner contributes. The four weights are non-negative and sum to 1 — (1−a)(1−b) + a(1−b) + (1−a)b + ab = (1−a+a)(1−b+b) = 1. It is a convex combination, so x(p) always lies in the convex hull of the four corners (no overshoot).

Fully worked numerical example

Suppose the four corners of the unit cell are

and we want the value at p = (0.25, 0.75). Then a = 0.25, b = 0.75, so 1−a = 0.75, 1−b = 0.25. Plug in term by term:

• corner (0,0): weight 0.75×0.25 = 0.1875, contributes 0.1875×10 = 1.875
• corner (1,0): weight 0.25×0.25 = 0.0625, contributes 0.0625×20 = 1.25
• corner (0,1): weight 0.75×0.75 = 0.5625, contributes 0.5625×30 = 16.875
• corner (1,1): weight 0.25×0.75 = 0.1875, contributes 0.1875×40 = 7.5

Sum = 1.875 + 1.25 + 16.875 + 7.5 = 27.5. Sanity check: p is near the top-left region (high y, low x), so the value should lean toward x(0,1)=30 and x(1,1)=40 — and 27.5 does. The weights summed to 0.1875+0.0625+0.5625+0.1875 = 1.0, as required.

Why bilinear, and why it is differentiable

Each corner weight is a smooth (piecewise-linear) function of a and b, hence of px and py, hence of the predicted offset Δp. So ∂x(p)/∂Δp exists almost everywhere — the offset network receives a gradient that says "shift the sample this way to increase the loss-reducing signal." This is the hinge that lets the network learn where to look by gradient descent. A nearest-neighbor read would be piecewise-constant (zero gradient everywhere), and the offsets could never train.

python
# Bilinear read of feature map x at one fractional point p=(px,py)
import numpy as np

def bilinear(x, px, py):
    H, W = x.shape
    x0, y0 = int(np.floor(px)), int(np.floor(py))
    x1, y1 = x0 + 1, y0 + 1
    a, b = px - x0, py - y0                # fractional parts
    def get(yy, xx):                        # zero-pad out of bounds
        if 0 <= yy < H and 0 <= xx < W: return x[yy, xx]
        return 0.0
    return ((1-a)*(1-b)*get(y0,x0) + a*(1-b)*get(y0,x1)
          + (1-a)*b*get(y1,x0) + a*b*get(y1,x1))

x = np.array([[10,20],[30,40]], dtype=float)
print(bilinear(x, 0.25, 0.75))   # 27.5  (matches the hand calc)

Chapter 3: The DCNv3 Operator

Now we assemble the full DCNv3 equation. With G groups (we cover groups in Chapter 4; here read G=1 for clarity), K sampling points, depthwise weights and softmax-normalized modulations, the operator computing one output location p₀ is:

Compare this to DCNv2 (Chapter 1) and read off the three deliberate differences DCNv3 introduces. Each is a specific engineering decision with a reason:

Worked example: one DCNv3 output with K=3

Single group, K=3 taps. Suppose the three taps (after offsets & bilinear reads) returned sampled values v = [2, 8, 4], the depthwise weight w = 1, and the modulation network produced pre-softmax logits z = [0, 2, 1]. First, softmax the logits:

• exp(z) = [e⁰, e², e¹] = [1.000, 7.389, 2.718]; sum = 11.107
• m = [0.090, 0.665, 0.245] (sums to 1.000 — check)

Then the output is w · ∑_k m_k v_k = 1 · (0.090·2 + 0.665·8 + 0.245·4) = 0.180 + 5.320 + 0.980 = 6.480. Because the modulations sum to 1, the output is a weighted average of the sampled values — it always lies in [min v, max v] = [2, 8]. The tap that the network deemed most relevant (the one reading 8, logit 2) dominates, exactly as a high attention weight would.

Data flow with real tensor shapes

Concretely, for an input feature map of shape [B, H, W, C] (channels-last, as the official kernel uses), a DCNv3 layer with G groups and K=9 produces, from a small projection of the input:

The output spatial size equals the input (stride-1 DCNv3 inside a block; downsampling happens in separate stem/downsample layers). The operator is O(H·W·C·K) — linear in the number of pixels, because each output reads only K=9 points regardless of image size. That is the decisive contrast with self-attention's O((HW)²).

python
# One DCNv3 output location, single group, NumPy (no autograd, for clarity)
import numpy as np

def softmax(z):
    e = np.exp(z - z.max()); return e / e.sum()

# Suppose the sub-network already gave us, for this location:
sampled = np.array([2., 8., 4.])   # K=3 bilinear reads at bent taps
logits  = np.array([0., 2., 1.])   # pre-softmax modulation logits
w       = 1.0                        # depthwise weight (shared over taps)

m = softmax(logits)               # [0.090, 0.665, 0.245], sums to 1
y = w * np.dot(m, sampled)       # convex combo  ->  6.48
print(round(float(y), 3))           # 6.48

Chapter 4: Groups & Weight Sharing

A single offset field forces every channel to sample the same K locations. But different channels encode different things — one channel might respond to vertical edges, another to skin texture, another to large flat regions. They should be allowed to look in different places. DCNv3 solves this exactly the way multi-head attention does: split the channels into G groups, and give each group its own offsets and its own modulations.

Within a group, all C/G channels share that group's K offsets and K modulations (this is the weight sharing). Across groups, the patterns differ. The parallel is exact:

Why share weights within a group at all?

Parameter economy and a useful prior. If every channel had its own offset field, the offset sub-network would output C×K×2 numbers per location — enormous and prone to overfitting. Sharing a group's offsets across its C/G channels assumes those channels want similar spatial structure (a reasonable prior, since grouped channels tend to co-activate), and it shrinks the offset head to G×K×2 outputs. G is a small knob: the paper finds more groups help, because they add sampling diversity at modest cost — the same reason multi-head beats single-head attention.

Worked example: counting offset outputs

Take C=64 channels, K=9 taps. With G=1 the offset head must emit 1×9×2 = 18 offset numbers and 1×9 = 9 modulation logits per location — one sampling pattern shared by all 64 channels. With G=4 it emits 4×9×2 = 72 offsets and 4×9 = 36 logits — four independent patterns, each shared by 16 channels. The cost grew 4× on the tiny offset head (negligible vs. the value path), but the model gained four distinct ways to look around. That is a great trade, which is why InternImage configurations use multiple groups and increase G with model size.

Chapter 5: The Basic Block & Stem

DCNv3 is the operator; InternImage is the network built around it. The design deliberately mirrors a modern Transformer block (like ViT/Swin) but with DCNv3 in the mixing slot — so we get a fair, apples-to-apples comparison where only the token-mixer differs.

The basic block

Each InternImage block is a residual stack with two sub-layers, each preceded by LayerNorm (pre-norm), exactly like a Transformer:

Both sub-layers use residual connections. The first replaces self-attention with DCNv3; the second is the ordinary position-wise FFN found in every Transformer. This is the key structural claim: swap the mixer, keep everything else — so any performance difference is attributable to DCNv3 vs. attention, not to other architectural tricks.

The stem and downsampling (hierarchical, 4 stages)

Like Swin and ConvNeXt, InternImage is hierarchical: it produces feature maps at 4 resolutions (strides 4, 8, 16, 32) so it can serve as a drop-in backbone for FPN-based detectors and segmentation heads, which need multi-scale features. The pipeline:

Worked example: tracing one image through the stem

Input image 224×224×3. Stem conv #1 (stride 2, pad 1, 3×3) → 112×112×(C/2). Stem conv #2 (stride 2) → 56×56×C. That is stride 4 (224/56 = 4), matching the standard backbone convention — a 56×56 = 3136-token grid at width C entering stage 1. After stage 1's downsample: 28×28 (stride 8). Stage 2: 14×14 (stride 16). Stage 3: 7×7 (stride 32). The four stage outputs are precisely the pyramid levels P2–P5 that an FPN expects. No special head surgery is needed — InternImage plugs into the existing detection/segmentation ecosystem unchanged.

Chapter 6: Scaling Deformable Conv to 1B

Designing one block is easy; making a family of models that scales predictably from tens of millions to ~1 billion parameters is the hard part, and historically the part CNNs lost to ViTs. InternImage scales along two axes with explicit rules so the family stays well-behaved.

Two scaling rules

Depth (number of blocks per stage) and width (channels C) are grown together, with the group count G tied to width so each group keeps a sensible channel slice. The InternImage variants span a wide range:

What breaks at a billion params, and the two fixes

Two specific failure modes appear only at extreme scale, and InternImage-H addresses each:

The pretraining recipe (briefly)

InternImage-H is not trained from scratch on the target task. It is pretrained on large-scale image data (the authors assemble a sizable classification corpus, ~400M+ images in their largest setting) and then fine-tuned on detection (COCO) and segmentation (ADE20K). This is the standard foundation-model playbook — pretrain broad, fine-tune narrow — and is part of why InternImage is framed as a vision foundation model, not just an architecture. The linear O(N) cost is what makes pretraining at high resolution on hundreds of millions of images affordable; an O(N²) attention backbone at the same resolution would be far costlier.

Chapter 7: Results & Ablations

The paper's empirical case rests on three pillars: classification, detection, and segmentation — plus controlled ablations isolating why DCNv3 works.

Headline numbers

The crucial comparison is matched-budget: at equal parameter count, InternImage-T/S/B beat Swin-T/S/B and ConvNeXt-T/S/B on detection and segmentation. The gains are larger on dense-prediction tasks (detection, segmentation) than on classification — which is exactly what the theory predicts, since long-range adaptive sampling matters most when you must localize objects, not just label whole images.

Ablations — isolating the mechanism

The paper removes one ingredient at a time. The qualitative findings (the ones to remember):

• Remove learnable offsets (freeze to the rigid grid): performance drops sharply — this is the long-range/adaptive-sampling property, the single biggest contributor.
• Replace softmax modulation with sigmoid (DCNv2-style): training becomes less stable and final accuracy drops, especially at large scale — confirming the normalization's role.
• Reduce the number of groups G → 1: accuracy falls, mirroring single-head vs. multi-head attention — sampling diversity matters.
• Use dense (non-separable) weights: params/FLOPs balloon for little gain — justifying the depthwise+pointwise factorization.

Effective receptive field (the visual proof)

A striking qualitative result: the effective receptive field (ERF) of InternImage is large and adaptive — it expands to cover the relevant object, not a fixed circular blob centered on the pixel. Visualizations show the sampled points clustering on the foreground object (a cat's whole body, a car's full outline) rather than spreading uniformly. This is direct evidence that the learned offsets do what the math promised: gather evidence from where the content is.

Chapter 8: DCNv3 Explorer

This is the payoff: a live, multi-control DCNv3 operator on a synthetic feature map. You control the input content, the learned offset strength, the modulation temperature, and the number of groups — and watch the sampled taps move, the modulation weights redistribute, and the aggregated output change. Everything below is computed from the actual equations of Chapters 2–4.

Things to try, and what they reveal:

• Move the object far from center, set offset strength to 0. The taps stay on the rigid grid and miss the object — the output collapses. This is the rigid-conv blindness from Chapter 0.
• Now raise offset strength. The taps stream toward the object; the output climbs as the deformable sampler captures the signal a fixed kernel could not reach — long-range adaptive sampling, live.
• Sharpen the modulation temperature. The softmax concentrates on the few taps sitting on the brightest part of the object (near one-hot); soften it and the weights spread out toward a uniform average.
• Add groups. Each group's constellation drifts to a different part of the scene — coverage diversifies, just like multi-head attention.

Chapter 9: Limitations & Connections

DCNv3 and InternImage are not the end of the story. Knowing where the operator strains is as important as knowing where it shines.

Limitations

• Custom CUDA dependency. The deformable sample-and-aggregate is a fused custom kernel. It is fast, but it is not a vanilla op — deploying to exotic accelerators or compiling with some graph toolchains is harder than for pure matmul/attention. Hardware support is improving but remains a friction point.
• Sparse, not global. Each output still reads only K=9 points. If a task genuinely needs to integrate all positions densely (rare, but possible), attention's global view can win. DCNv3 bets that K well-placed points suffice — usually true, not always.
• Offset-network overhead & training subtlety. Predicting offsets and modulations adds a small sub-network and can be finicky to train (offsets can collapse or wander early in training); the softmax/group design mitigates but does not erase this.
• Interpretability. Attention maps are a familiar lens; deformable sampling is harder to inspect, though ERF and tap visualizations help.

Where DCNv3 sits in the lineage

Related lessons on Engineermaxxing

Cheat sheet