3D scene understanding has been an island of specialized backbones — sparse-conv U-Nets and local-attention point transformers — cut off from the Transformer ecosystem. Volt asks: what if you just use a plain ViT-style encoder on 3D, with global attention and a 3D rotary positional embedding? It works — and once you feed it data, it scales past the specialists.
Here is a strange fact about deep learning in 2026. If you work on language, images, audio, or video, you almost certainly use the same architecture: a Transformer. A trick that helps language models — a better attention kernel, a new positional encoding, a smarter way to scale — usually transfers to vision within months. Everyone shares one toolbox, one set of hardware optimizations, one research ecosystem.
And then there is 3D scene understanding — the task of taking a point cloud of a room or a street (from a depth camera or a LiDAR) and labeling every point: this is floor, that is chair, that is car. This field lives on its own island. Its best models are not Transformers. They are sparse convolutional U-Nets and hand-engineered local-attention networks with names like MinkUNet and Point Transformer v3.
The specialists exist for a good reason. A 3D scene is huge and mostly empty — a single room can be hundreds of thousands of points. Treating each point as a token and running global attention would cost O(N²) with N over 100,000, which is hopeless. So the field built architectures that bake in strong assumptions about 3D structure:
These assumptions are inductive biases: built-in beliefs about what the answer should look like. A good inductive bias is a gift when you have little data — it tells the model "nearby points matter most" so it doesn't have to learn that from scratch. The whole 3D field is tuned for today's small datasets, where these biases pay off.
So why rock the boat? Two reasons, and they compound.
First, isolation. Because 3D backbones are not plain Transformers, they cannot ride the wave of Transformer progress. FlashAttention — a fused attention kernel that makes attention dramatically faster and lighter — was written for standard attention. PTv3's custom local attention needs its own custom kernels and indexing logic. Every hardware generation that is "optimized for Transformers" passes 3D by. Every advance in pretraining, multi-modality, or scaling that the rest of the field enjoys has to be painstakingly re-derived for 3D.
Second, the ceiling. Strong inductive biases help in the low-data regime, but they become a limitation as data grows. We have seen this movie before. In 2D vision, convolutional networks dominated for years — until the Vision Transformer (ViT), with almost no built-in bias, was fed enough data and surpassed them. The lesson: hand-crafted priors are training wheels. They keep you upright when you're slow, but they cap your top speed.
The figure below is the soul of the entire paper, made interactive. Drag the supervision slider. Two models compete: a strong-prior specialist (like PTv3) and a weak-prior Transformer (Volt). Watch who's ahead in the data-scarce regime versus the data-rich regime — and find the crossover point where the bet pays off.
Accuracy vs. amount of training supervision. The specialist starts ahead — its priors substitute for data. The plain Transformer starts behind (it overfits when data is scarce) but climbs faster and eventually overtakes. This is the ViT story, and Volt's thesis is that it replays in 3D.
The insight is almost embarrassingly simple, which is exactly why it's interesting: do to 3D scenes what ViT did to images.
Recall the ViT recipe. An image is a big grid of pixels — too many to treat each as a token. So you chop it into a grid of small square patches (say 16×16 pixels), flatten each patch into a vector, linearly project it to the model dimension, and now you have a short sequence of tokens. Feed that sequence to a completely standard Transformer encoder. No convolutions in the body, no image-specific machinery — just patchify, then attend.
Volt = ViT with exactly three changes for 3D, and not one more:
| Component | ViT (2D) | Volt (3D) |
|---|---|---|
| Patchify | square pixel patches | cubic voxel patches (Ch. 3) |
| Encoder | vanilla Transformer, global attention | identical — vanilla, global (Ch. 4) |
| Positions | learned 1D/2D embeddings or 2D RoPE | 3D rotary embeddings (Ch. 5) |
| Decode | class token / linear | one transposed conv + linear (Ch. 6) |
That's the whole architecture. The encoder is byte-for-byte a ViT encoder — same block, same LayerNorm placement, same attention. This is the point: by keeping it vanilla, Volt can use FlashAttention out of the box, inherit every future Transformer improvement for free, and slot into any Transformer tooling.
This is the objection everyone raises, so let's kill it immediately with arithmetic. The fear is the O(N²) cost of attention. But N here is not the number of points — it's the number of tokens after patchification, and that number is small.
Watch the collapse. Start with a raw point cloud of, say, 200,000 points. Voxelize it (snap points to a grid, keep one per cell): now maybe 40,000 occupied voxels. Group voxels into cubic patches of 5×5×5: each patch becomes one token, and because the scene is sparse, you end up with roughly 5,000 tokens. At 5,000 tokens, global attention is not just feasible — with FlashAttention it's fast.
Below, build your own scene budget. Set the raw point count, the voxel size, and the patch size. Watch how a hopeless O(N²) per-point cost collapses into a comfortable per-token cost. The whole feasibility argument is in this one widget.
Before we can patchify a scene, we have to confront what a 3D scene actually is as data — and why it's so much messier than an image.
An image is a gift: a perfectly regular 2D grid. Pixel (i, j) always exists, always has neighbors at (i±1, j) and (i, j±1), and the grid is dense — every cell is filled. A convolution can slide a fixed kernel across it because the structure is guaranteed.
A point cloud is the opposite. It's a set of points X = {(pᵢ, fᵢ)}, where each point has a 3D coordinate pᵢ ∈ ℝ³ and a feature vector fᵢ ∈ ℝᶜ (typically color). Two headaches:
This is why early methods (PointNet) treated clouds as sets with symmetric pooling, and why later methods leaned on expensive nearest-neighbor searches. Both routes get awkward at the scale of a whole room.
The fix that unlocks everything: voxelize. Lay an imaginary 3D grid of cell size δ over space and snap each point to its cell:
Every point with coordinate pᵢ is integer-divided by the voxel size δ and floored, giving an integer cell index zᵢ. Keep one representative point per occupied cell (with its feature f̃ᵡ). Suddenly you have a regular integer grid again — order restored, neighbors well-defined. Volt uses δ = 2 cm indoors and 5 cm outdoors.
But here's the thing: a 3D scene is almost entirely empty space. Surfaces are thin shells. In a typical room voxelization, well under 10% of cells are occupied — the rest is air. Sparse convolution engines (Minkowski Engine and friends) exploit this: they store only occupied voxels in a hash table and compute convolutions only there. That's how MinkUNet runs at all — and, as we'll see in Chapter 3, Volt borrows the exact same sparse-conv trick to build its tokens.
Below is a 2D analog (a slice through a scene, for clarity). Toggle between the dense grid — every cell allocated, mostly wasted — and the sparse representation, where only the occupied surface cells exist. Watch the cell count plummet. This gap is the entire reason scene-scale 3D is tractable.
This is where a messy point cloud becomes a tidy sequence of tokens. It is the 3D counterpart of ViT's patch embedding, and getting the data flow exact is the whole game.
Start from the voxelized scene from Chapter 2: a sparse set of occupied voxels Xᵛ = {(zⲟ, f̃ⲟ)}, M cells, each with an integer position and a C-dim feature. Now:
The result is a token sequence X = [x₁, …, xₜ]. Crucially, unlike ViT where the image is a fixed size, here T varies per scene — it depends on scene size, density, and how the occupied voxels are distributed. The Transformer doesn't care: it eats variable-length sequences natively.
Let's make it concrete. Say C = 3 (RGB color) and P = 5.
Why is this better than the alternative — grouping points by k-nearest-neighbors, as point transformers do? Because kNN grouping is density-dependent and slow: you must search neighbors for every point, and dense regions blow up the cost. Voxelization + sparse conv is density-invariant and runs in one fast pass. Tokenization simultaneously aggregates local features and downsamples the scene to keep the sequence short.
Below, a 2D slice of a voxelized scene. Drag the patch size slider to tile it differently. Occupied patches (containing at least one filled voxel) light up and become tokens; empty patches are skipped. Watch the token count — and notice the trade-off: small patches = more tokens = finer detail but longer sequence; big patches = fewer tokens = faster but coarser. (Chapter 10 quantifies exactly this.)
Here is the chapter with the least to explain, and that is the entire point. Volt's encoder is a stack of L identical Transformer blocks, each one a textbook ViT block: multi-head self-attention, then an MLP, each wrapped in a residual connection with LayerNorm applied before the sublayer (pre-norm). No hierarchical stages, no local windows, no domain-specific grouping. Just global attention and MLPs.
That's a block. Read it twice and you've read the whole encoder. LN is LayerNorm, Attn is multi-head self-attention over the full token sequence (every token can attend to every other), and the two + x terms are the residual skips. Volt stacks L of these. Volt-S uses the ViT-S configuration; Volt-B uses ViT-B — literally the same widths and depths as the image models.
Volt adds QK-Normalization in every attention layer — it L2-normalizes the query and key vectors before their dot product. This is a now-standard trick (borrowed straight from modern LLM/ViT training) that keeps attention logits from exploding during training, stabilizing the optimization of a deep, weakly-regularized model. It is not a 3D-specific invention; it's another piece inherited from the ecosystem.
Why insist on global attention when the specialists work so hard to avoid it? Because global attention lets any token talk to any other token in a single layer. The far corner of the room can directly inform a token across the scene — useful when "is this a table or a desk?" depends on the whole room's context.
PTv3 cannot do this directly. It serializes points along a space-filling curve and attends only within local windows, so long-range interaction must hop through many layers. Worse, that custom local attention needs bespoke CUDA kernels and indexing. Volt's plain global attention runs on FlashAttention-2 — the same fused, memory-efficient kernel the entire Transformer world uses for variable-length sequences. Same kernel as your favorite LLM.
Below: a grid of scene tokens. Click any token to make it the query. In global mode it attends to every token (the whole scene at once). In local mode (PTv3-style) it only attends within a serialized window. Pick a token in one corner and a target object in the far corner — see how global reaches it in one step while local cannot.
Self-attention has a famous blind spot: it is permutation-equivariant. Shuffle the input tokens and the output just shuffles too — attention has no idea where any token is. For text that's a problem; for 3D it's catastrophic, because in 3D the meaning of a token is almost entirely about its position and geometry. We must inject position. This chapter is the one genuinely novel architectural piece, and it rewards a slow read.
The early-ViT answer was absolute learned positional embeddings: a lookup table with one learned vector per grid position, added to each token. This assumes a fixed, dense, bounded grid — which images have. 3D scenes do not. Positions are sparse, the scene is unbounded, and the number of possible token positions runs into the millions. You cannot learn a table that big, and it won't generalize to positions unseen in training. Absolute learned embeddings are a dead end for 3D.
The modern answer, standard in LLMs and recent ViTs, is Rotary Positional Embedding (RoPE). Instead of adding a position vector to the token, RoPE rotates the query and key vectors by an angle proportional to their position, before the attention dot product. The beautiful consequence: the dot product of a rotated query and a rotated key depends only on their relative offset, not their absolute positions.
Here's the 2D core. Take a query component and rotate it by angle θ·m where m is its position; rotate the key by θ·n for its position n. A rotation by angle α is the matrix:
Now the key identity — rotations compose by adding angles, and the dot product of two rotated vectors satisfies:
Stare at the right-hand side: the absolute positions m and n have vanished, leaving only (n − m) — the relative offset. So two tokens 3 apart produce the same positional effect on attention no matter where in the scene they sit. That's exactly what you want: geometry is about relative arrangement, and RoPE bakes that in. It also handles variable lengths, extrapolates beyond training positions, and is compatible with FlashAttention.
A token in 3D has a position p = (pᶩ, pᶻ, pᶻ) — wait, three coordinates: pₓ, pₐ (call them x, y, z). RoPE was built for 1D. Volt's extension (shared with concurrent work LitePT) is clean: split each query/key vector into three axis-blocks and apply RoPE independently per axis, then concatenate:
Read it: chop the query q into three chunks qₓ, qₘ, q⎷; rotate the first chunk by the token's x-position, the second by its y-position, the third by its z-position; glue them back together. Same for the key. Now the attention score depends on the relative offset along all three axes simultaneously.
Two design details that the ablations prove matter:
1. Asymmetric axis allocation. Volt does not split the dimensions evenly. It gives 12, 12, and 8 frequency pairs to x, y, and z respectively — more capacity to the horizontal axes, less to the gravity-aligned vertical z. Why? Scenes vary much more horizontally (a room sprawls in x and y) than vertically (floor-to-ceiling is a short, predictable range), especially outdoors. Spend the positional budget where the variation is.
2. Metric-consistent positions. What number do we plug in as the position? Volt uses the discrete patch index from tokenization (the ⌊ z / P ⌋ we saved in Ch. 3). These indices are proportional to true 3D coordinates by the fixed scale factor δP. So a physical distance of, say, 30 cm always maps to the same index gap, in every scene. The ablation shows that if you instead normalize each scene's coordinates to [0,1], you destroy this cross-scene consistency (the same physical distance becomes a different number depending on room size) and accuracy drops.
This is the heart of the architecture, made tangible. Two tokens sit on a line; each carries a query/key vector drawn as a blade. As you drag a token, its blade rotates by an angle proportional to its position. The attention score (the dot product) is read out live. The lesson to feel: slide both tokens together — keeping their gap fixed — and the score doesn't change. That's relative encoding. Then toggle the asymmetric z-axis to see how vertical position is encoded more coarsely.
The encoder outputs one feature vector per patch token. But the task needs a label for every point. There's a resolution gap — each token covers a P×P×P cube of voxels — and the decoder's only job is to close it.
Volt upsamples the tokens back to voxel resolution with a single transposed convolution whose kernel size equals the patch size P. This is the exact inverse of tokenization: tokenization was a strided conv that folded a P³ cube into one token; the transposed conv unfolds one token back into a P³ cube of per-voxel features:
where M′ is the number of voxels and D the feature width. Then:
Below, watch a single patch travel the full round trip: a P³ cube of voxels → folded into one token → processed by the encoder → unfolded by the transposed conv back into a cube of per-voxel features → classified → each point labeled. Toggle the decoder size to see the ablation's lesson.
Now the twist that makes this a real research paper and not a press release. You build this clean vanilla Transformer, you train it on ScanNet the normal way, and… it overfits badly. Naive Volt scores 31.0 mIoU on ScanNet200 while the specialist PTv3 scores 35.2. Out of the box, the plain Transformer loses.
Why does it overfit? Exactly because we stripped out the inductive biases. A sparse-conv U-Net is constrained — it can essentially only express functions built from local neighborhoods. That constraint is a prior: it shrinks the space of hypotheses the model can entertain, and in that small space, the right answer is easy to find from little data.
A global-attention Transformer has a much larger hypothesis space. It can express almost anything, including ugly shortcuts that fit the training set but don't generalize. With enough data, that flexibility is a superpower (it can discover patterns the conv can't). With too little data — and 3D benchmarks are tiny, orders of magnitude smaller than ImageNet, let alone the billion-image sets behind foundation models — that flexibility becomes a curse. The model memorizes shortcuts. This is the same thing that happened to ViT in low-data 2D.
If you don't have enough scenes, manufacture variety from the ones you have. Volt piles on 3D augmentations that fall into two camps:
Together these force the model to be invariant to things that shouldn't change a label — a chair is a chair whether it's rotated, half-cropped, or sitting in a mashed-up scene.
Borrowed wholesale from ViT best practice:
Below: training and validation curves. Toggle augmentation and regularization on and off and watch the overfitting gap — the chasm between a high training score and a low validation score — shrink as you add each ingredient. With everything off, train soars while val stalls. With the recipe on, they converge.
This is the most counterintuitive ingredient, and the most interesting. Volt gets a boost by learning to imitate a convolutional teacher that is worse than itself. A weaker model teaching a stronger one — and it works. Let's understand why.
Normal knowledge distillation (Hinton et al.) has a strong teacher guiding a weak student — you compress a big model into a small one. Volt inverts this. The teacher is a humble MinkUNet (a sparse-conv U-Net) that, on the validation set, underperforms Volt. Yet distilling from it makes Volt better. How can a worse model improve a better one?
Because the teacher isn't transferring accuracy — it's transferring inductive bias. The MinkUNet, being convolutional, has exactly the local-geometry prior Volt deliberately lacks. By mimicking the conv's predictions, Volt absorbs a dose of that prior — just enough to curb its overfitting — without hard-coding it into the architecture. It's like a freewheeling improviser taking a few lessons from a disciplined classical teacher: the teacher isn't a better musician overall, but the discipline rubs off.
Following DeiT, Volt uses the teacher's hard predictions (the teacher's argmax labels), not soft logits. Concretely, the voxel features F feed into two linear classification heads:
The loss is a half-and-half blend:
Every symbol: F = the per-voxel features from the backbone+decoder; Wsegᵀ F and Wdistillᵀ F = the two heads' logits; ygt = true labels; yteacher = the conv's predicted labels; Lseg = the usual segmentation loss (cross-entropy + Lovász). The 0.5/0.5 means the backbone must satisfy both the ground truth and the teacher simultaneously, so gradients from both flow into the shared features.
Below: the two-head training setup on the left (toggle the distillation head on/off and watch where gradients flow), and the three training curves on the right. Turn distillation on to see the student's curve lift above the teacher it's learning from.
Everything so far has been setup. This chapter is where the bet from Chapter 0 gets settled with real numbers. The claim was: a weak-prior Transformer should benefit more from data than a strong-prior specialist. Does it?
The augmentation + regularization + distillation recipe makes Volt competitive under single-dataset training. But to test scaling, you need more data. So Volt is trained jointly across multiple datasets at once — indoor RGB-D sets together (ScanNet, ScanNet200, ScanNet++, ARKitScenes), and outdoor LiDAR sets together (nuScenes, Waymo, SemanticKITTI).
This is harder than it sounds. Each dataset uses a different sensor, a different reconstruction pipeline, a different label space — so point density, coverage, and noise vary wildly. Volt's accommodation is minimal: a per-dataset linear classification head (to handle different label sets) while the entire backbone is shared. The backbone learns one general 3D representation; only the thin output layer is dataset-specific.
Now the verdict. Train every model on 50% of the data, then 100%, then the full multi-dataset pool, and plot accuracy vs. data:
Below is Figure 4, interactive. Drag the training data axis from 50% → 100% → multi-dataset and watch six curves move: Volt-B, Volt-S, PTv3-B, PTv3-S, MinkUNet, and Naive-Volt-S. Find the data scale where Volt-S overtakes the 5×-larger PTv3-B, and watch Volt-B pull away at the top.
Time to read the scoreboard — and, more importantly, the ablations that reveal why the design choices were made.
| Task / Benchmark | Metric | Volt (best) | Prior SOTA |
|---|---|---|---|
| Indoor semseg — ScanNet | mIoU | 80.5 | PTv3/PPT 79.8 |
| Indoor semseg — ScanNet200 | mIoU | 41.6 | PTv3/PPT 37.5 |
| Indoor semseg — ScanNet++ | mIoU | 49.5 | PTv3 48.8 |
| Outdoor semseg — nuScenes | mIoU | 82.2 | PPT-based ~83 / 81 |
| Instance seg — ScanNet | mAP50 | 82.7 | SGIFormer 79.9 |
| Instance seg — ScanNet200 | mAP50 | 47.5 (test) | SPFormer baseline lower |
Two results deserve emphasis. (1) Under single-dataset training, Volt-S already matches or beats prior work while being 2× smaller and faster. (2) For instance segmentation, the authors did a controlled swap — take the established SPFormer pipeline, change nothing but the backbone (MinkUNet → Volt), and accuracy jumps, with a +14.6 mAP50 leap on ScanNet200 val. The lesson: strengthening the raw 3D representation beats years of increasingly clever decoder designs.
The single most important hyperparameter is the patch size P, because it directly trades accuracy for speed. Smaller patches = more tokens = finer detail (small objects, thin structures) but a longer, slower sequence. Bigger patches = fewer tokens = fast but coarse.
| Patch P | ScanNet200 mIoU | Speed | Sem.KITTI mIoU | Speed |
|---|---|---|---|---|
| 3×3×3 | 37.7 | 34 ms (slow) | 71.0 | 113 ms |
| 5×5×5 | 36.2 | 14 ms | 70.8 | 49 ms |
| 7×7×7 | 33.5 | 13 ms | 69.6 | 34 ms |
P = 3 is most accurate but ~2.4× slower; P = 7 is fast but small objects suffer most (coarse tokenization swallows thin structures). P = 5 is the chosen sweet spot — nearly all the accuracy at a fraction of the cost.
From Chapter 5, the proof that the RoPE design choices matter — Asymmetric RoPE 36.2/70.8 > Symmetric 35.4/69.5 > Asymmetric+normalized-coords 35.7/69.3 >> Fourier 28.6/66.0. The Fourier cliff is the headline: how you inject position (rotate into the QK interaction vs. add to embeddings) matters more than almost any other single choice.
Below: drag the patch size to walk the accuracy/speed frontier (token count, latency, and mIoU all update together), then flip to the positional-encoding ablation to see the four variants side by side.
Step back. What did Volt actually teach the field?
3D scene understanding does not need its own architecture. With three minimal changes — cubic patch tokenization, 3D rotary positions, a one-layer decoder — the vanilla Transformer encoder becomes a state-of-the-art, efficient, general-purpose 3D backbone. The price of admission was a data-efficient training recipe to tame overfitting on today's small benchmarks. The reward is membership in the Transformer ecosystem: FlashAttention, future architectural advances, multi-modal fusion, and scaling all come for free. As 3D data grows, the training scaffolding can fall away and the architecture's scaling advantage takes over.
| Symbol / term | Meaning |
|---|---|
| pᵢ ∈ ℝ³, fᵢ ∈ ℝᶜ | a point's 3D coordinate and C-dim feature (e.g. RGB) |
| δ | voxel size (2 cm indoor, 5 cm outdoor); zᵢ = ⌊pᵢ/δ⌋ is the voxel index |
| P | patch side in voxels (= 5); patch index = ⌊z/P⌋, used as RoPE position |
| uₜ ∈ ℝP³C | flattened patch (empty voxels zero-padded) |
| xₜ = W uₜ ∈ ℝᵀ | token = linear projection of a patch (a strided sparse 3D conv) |
| T | number of tokens (~5k; varies per scene) |
| RΘ(p) | rotary transform; applied per axis with 12/12/8 freq pairs for x/y/z |
| F ∈ ℝM′×D | per-voxel features after the transposed-conv decoder |
| L = 0.5 Lseg(WsegF, ygt) + 0.5 Lseg(WdistillF, yteacher) | joint GT + distillation loss (teacher discarded after training) |
| Volt-S / Volt-B | ViT-S (23.7M) / ViT-B (94.3M) encoder configurations |