PointNet: Learning on Unordered Sets

Chapter 1: Symmetry by Max

We need a function of a set. Let us derive its shape rather than guess it.

Write the thing we want as f({x₁, …, x_n}), and demand that swapping any two arguments leaves the value alone. A function with that property has a name in mathematics: it is symmetric. You already know several. Sum is symmetric: 2 + 5 + 9 does not care about order. Product is symmetric. Minimum and maximum are symmetric. Average is symmetric.

So one legal design is: apply some symmetric function to the raw points and be done. But max over raw coordinates of a chair gives you the bounding-box corner, which is a terrible chair descriptor. Sum over raw coordinates gives you n times the centroid, which is barely better. Raw symmetric functions throw away everything.

The move: transform first, then symmetrise

Here is the idea, and it is the only genuinely creative step in the paper. Do not apply the symmetric function to the coordinates. Apply it to learned features of the coordinates.

where h: R^N → R^K is applied to every point independently, with shared weights, and g is symmetric. Now check the invariance, because it should be immediate and it is:

Permute the input. Each x_i still gets the same h — h has no idea which slot a point arrived in, because it only ever sees one point. So the multiset {h(x₁), …, h(x_n)} is unchanged; only its listing order changed. And g is symmetric, so it ignores listing order. Therefore f is unchanged. There is no training involved in that guarantee. It is algebra.

Which symmetric function? The paper actually measured

Sum, average, max and an attention-weighted sum are all symmetric and all legal. The paper ran the control experiment on ModelNet40 and found max pooling wins "by a large winning margin" over average pooling and over an attention-based weighted sum of the style used in Order Matters. Two intuitions for why:

Max does not dilute. If one point in ten thousand is the tip of an aeroplane's nose, average pooling divides its evidence by ten thousand. Max pooling passes it through untouched. Detection of a rare local pattern survives; under averaging it drowns.

Max is stable under resampling. Sample the same surface twice with different random seeds and the average changes with the sample density; the maximum barely moves, because it is a property of the surface's extremes under h, not of how many samples you drew.

The cost of max — and it is a real cost that Chapter 2 is entirely about — is that the gradient flows to exactly one point per output dimension. Everything else gets zero. We will make that concrete in a moment.

A tiny PointNet, worked entirely by hand

Everything above is easier to believe once you have pushed numbers through it. Let us build the smallest possible PointNet and run it.

Drop to two dimensions so the arithmetic fits on a page — nothing in the argument depends on the dimension. Our "cloud" is four points:

Our per-point network h is a single linear layer with a ReLU, mapping R² → R³. In real PointNet this is a five-layer MLP with widths 64, 64, 64, 128, 1024; here it is one layer with width 3, so that K = 3 and we can write everything down.

Run point A = (0.2, 0.9) through it, one row at a time:

So h(A) = (0, 1.6, 0.5). Doing the same for the other three:

Now pool. Take the maximum down each column, across points:

Now do the thing the whole chapter is about. Feed the points in the order D, C, B, A. Every row of the table above is unchanged — h never knew the order. The columns get maximised over the same four numbers in a different sequence, and maximum does not care. u = (1.5, 1.6, 0.5), exactly. Try B, D, A, C. Same. Try all 24 orderings. Same.

See it hold, and see it break

The simulation runs exactly the four points above. Shuffle them and watch the pooled bars sit perfectly still. Then switch the aggregator to concat — which is what a flattened MLP effectively does, reading features slot by slot — and watch the same shuffle wreck it.

Three things worth noticing while you play. First, the middle band never reorders its contents — each point carries its own feature wherever it lands, which is what "shared weights, applied independently" means in pictures. Second, under max the winning point for each output dimension is marked; for these four points, B wins dimension 1 and A wins dimensions 2 and 3, and C and D win nothing at all. Hold that thought — it is Chapter 2. Third, sum is also perfectly invariant; it is not wrong, it just measures worse, for the dilution reason above.

Where the gradient goes

Forward, max pooling is obvious. Backward, it has a property you should know about before you train anything, and the toy makes it visible.

The derivative of max(a₁, …, a_n) with respect to a_i is 1 if a_i is the maximum and 0 otherwise. So when the loss sends a gradient back through u_j, that gradient is delivered entirely to the single point that won dimension j and nowhere else.

In our toy, with K = 3 and four points: B receives the gradient from dimension 1, A receives the gradients from dimensions 2 and 3, and C and D receive exactly zero. On this step they might as well not exist — their coordinates get no update signal at all.

Two consequences that matter in practice. First, with n = 1024 points and K = 1024 pooled dimensions, at most 1024 of 1024 points could get gradient — but in practice one point wins many dimensions, so the number of points touched per step is far smaller. The learning signal is sparse. Second, this is not a problem the way sparse gradients usually are, because h's weights are shared: every point contributes to the weight gradient through whichever dimensions it won, and across a batch of clouds and thousands of steps, every region of the input space gets visited. Weight sharing rescues what winner-takes-all would otherwise starve.

It also explains why the network converges to the behaviour Chapter 2 describes. If a point only receives gradient when it wins a dimension, the optimiser has a strong incentive to make the winners informative — to place the decision boundaries of h so that the points which win are the ones that distinguish shapes. That is the mechanism behind the "skeleton" visualisation, and it was not designed in.

Concept and realization: PointNet in twenty lines

Everything above is one small module. Here it is with the shapes annotated, so you can see that the invariance argument and the code are the same object.

python# PointNet classification, vanilla (no T-Nets). B = batch, N = points, K = pooled width.
class PointNet(nn.Module):
    def __init__(self, n_classes=40, K=1024):
        # Conv1d with kernel_size=1 IS a shared per-point MLP. The "convolution"
        # never mixes neighbouring columns, because the kernel is one wide — it
        # is just an efficient way to apply the same Linear to every point.
        self.h = nn.Sequential(
            nn.Conv1d(3,   64,  1), nn.BatchNorm1d(64),  nn.ReLU(),
            nn.Conv1d(64,  64,  1), nn.BatchNorm1d(64),  nn.ReLU(),
            nn.Conv1d(64,  64,  1), nn.BatchNorm1d(64),  nn.ReLU(),
            nn.Conv1d(64,  128, 1), nn.BatchNorm1d(128), nn.ReLU(),
            nn.Conv1d(128, K,   1), nn.BatchNorm1d(K),   nn.ReLU())
        self.gamma = nn.Sequential(
            nn.Linear(K, 512), nn.BatchNorm1d(512), nn.ReLU(), nn.Dropout(0.3),
            nn.Linear(512, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(0.3),
            nn.Linear(256, n_classes))

    def forward(self, x):            # x: (B, 3, N)  — channels first
        f = self.h(x)               # (B, K, N)   every column computed in isolation
        u = f.max(dim=2)[0]         # (B, K)      <-- THE ENTIRE INVARIANCE IS THIS LINE
        return self.gamma(u)        # (B, n_classes)

Three things in that listing are load-bearing and easy to miss.

Conv1d with kernel size 1 is not a convolution in any interesting sense. It applies the same linear map to every column independently — which is exactly "shared MLP applied to each point". It is written as a convolution purely because that is how the framework lays out an efficient batched implementation. If you replace it with a real kernel of width 3, you have silently made the model order-dependent, because column i would then depend on columns i−1 and i+1, which are arbitrary neighbours in the array and not neighbours in space. That is a real bug people ship.

max(dim=2) is the whole paper. Change it to mean(dim=2) and you have average pooling, which is still invariant and measurably worse. Change it to flatten() and you have Chapter 0's broken model. One line separates the three.

BatchNorm1d over a (B, C, N) tensor normalises over batch and points. That is the right choice here and worth understanding: the statistics of channel c are estimated across all points of all clouds in the batch, which is precisely the pooled population that the shared weights operate on. Normalising per-point instead would be incoherent — there is nothing to average over. It does mean the model's behaviour is mildly batch-dependent at training time, which is one more reason inference uses running statistics.

Theorem 1: this is not a restriction

A reasonable objection at this point: yes, it is invariant — but have we given up expressive power to buy that invariance? Surely a function that must be blind to order is a weaker function.

The paper's Theorem 1 says no, in a specific and strong sense. Let X be the sets of exactly n points inside the unit cube [0,1]^m, and let f be any set function on X that is continuous with respect to Hausdorff distance — meaning that nudging the points a little nudges f a little. (Hausdorff distance between two sets is, roughly, the worst distance from any point in one set to the nearest point of the other. It is the natural "how different are these two clouds" metric when the clouds have no correspondence.)

So per-point MLP plus max plus MLP can approximate any continuous set function arbitrarily well, given enough neurons at the pooling layer. The invariance is free: you gave up nothing in exchange for it.

The proof sketch is worth carrying around because it also tells you the worst case. In the limit, the network can simply learn to voxelize: partition the cube into small cells, let dimension j of h be an indicator that fires when a point is in cell j, and the max over points becomes exactly the occupancy grid. So PointNet is at least as expressive as the voxel approach it replaces — that is the floor. The paper's own comment is the interesting part: "in practice, however, the network learns a much smarter strategy to probe the space."

From the toy to the real thing: shapes and parameter counts

Now scale the toy up to the network in Figure 2 of the paper, tracking tensors. Take the classification branch, n = 1024 points, k = 40 classes:

Look hard at the max-pool row. Up to that line, the 1024 points have never interacted — each is a 1024-dim feature computed in total ignorance of the others. Then in one operation, all 1,048,576 numbers collapse to 1024. That single line is the entire communication budget of PointNet, and it is why Chapter 4 has to exist.

Count the parameters of the shared MLP by hand; a linear layer a→b has ab + b parameters:

That is exactly the "PointNet (vanilla) 0.8M" row of the paper's Table 6, reproduced from first principles. Note where the parameters are: 82% of them are in the classifier head, on a 1024-dim vector. The part that touches the points costs 0.149M. Per-point processing is nearly free; it is the global vector that is expensive. Keep that in mind in Chapter 3 when we meet a module that costs 1.85M on its own.

Chapter 2: The Critical Points

Go back to the toy table. The pooled descriptor was u = (1.5, 1.6, 0.5), and dimension by dimension it came from B, A, A.

Now ask a question that should make you slightly uneasy: what happens if we delete point D?

h(D) = (0, 0.3, 0). It is not the maximum in any column. Remove it and recompute: u₁ = max(0, 1.5, 0.5) = 1.5, u₂ = max(1.6, 0, 0.5) = 1.6, u₃ = max(0.5, 0.3, 0.4) = 0.5. Identical. Not approximately identical — bitwise identical.

Delete C as well. h(C) = (0.5, 0.5, 0.4), also nowhere a maximum. Recompute over {A, B} alone: (1.5, 1.6, 0.5). Identical again.

Theorem 2, stated and unpacked

This is not an artefact of a toy. It is the paper's Theorem 2, and it is the most interesting result in either paper.

Define u(S) = MAX_x∈S {h(x)}, the pooled vector, and f = γ ∘ u. Then:

In words. There are two special sets attached to every shape.

C_S, the critical point set: the points that actually win at least one dimension of the max. Keep these and you keep the descriptor, no matter what else you throw away. In our toy, C_S = {A, B}.

N_S, the upper-bound shape: every point in space whose feature vector is componentwise no larger than u. Add any of them and the max does not move, because none of them can win anything. Any cloud T sandwiched between the two — contains all of C_S, contained in N_S — produces exactly the same global feature.

And part (b) is the sharp bit: C_S has at most K elements, where K is the pooled width. One dimension of the max can only be won by one point. With K = 1024, a 100,000-point lidar sweep is summarised by at most 1024 points. Usually far fewer, because one point tends to win many dimensions at once.

Computing N_S for the toy, by hand

The upper-bound shape sounds abstract until you write it down. For our h it is three inequalities. A point (x, y) fails to change u if and only if each of its three pre-activations is at most the corresponding u_j — note ReLU(z) ≤ u is the same as z ≤ u whenever u ≥ 0, which it is:

Three half-planes. Their intersection is N_S — a convex polygon containing the original four points. Test a candidate outlier E = (0.4, 0.6):

All three hold, so E is inside N_S. Add it to the cloud and the descriptor is unchanged: h(E) = (0.2, 0.8, 0.4), and every entry loses its column. A spurious return — a dust mote, a rain droplet, a multipath ghost — that lands anywhere inside that polygon is invisible to the model.

Now test F = (1.0, 0.0), further out:

So outliers are not harmless in general — they are harmless exactly when they fall inside the upper-bound shape, and damaging exactly when they push past one of its faces. That is a precise, testable statement about a model's failure mode, derived from three inequalities, and it is the sort of thing you almost never get from a deep network.

What the robustness numbers actually mean

The paper measures the consequences on ModelNet40, and the numbers are the theorem in experimental clothing.

Sit with the VoxNet comparison. Forty points of accuracy versus under four, on identical corruption. That is not a tuning difference. It is the difference between a representation whose output depends on all of the input and one whose output provably depends on at most K of it.

How many critical points, really?

Theorem 2 gives an upper bound of K. The interesting question is what the number actually is, and the answer follows from a simple counting argument you can do yourself.

There are K dimensions and each is won by exactly one point, so the number of wins is exactly K. The number of winners is at most K and equals K only if no point ever wins twice. But points do not win at random — a point that is extremal in one direction is usually extremal in nearby directions too. The wing tip of an aeroplane wins every dimension whose h is sensitive to "far out along this axis", and there are many such dimensions.

So the effective critical count is K divided by the average number of wins per winner, and that divisor grows with how correlated the point functions are. The simulation above makes this visible: at K = 4 you see at most four critical points and usually four; push K to 16 and the count rises but by less than a factor of four, because the same lobe tips keep winning adjacent directions.

This is also why the paper's ablation on K shows diminishing returns rather than a linear improvement: raising K from 64 to 1024 is a 16× increase in probes and buys 2–4% accuracy. Each extra probe overlaps more with existing ones. And it is the mathematical version of the visual intuition — a shape's silhouette is described well by a few dozen well-chosen extremal points, and the hundredth one adds very little.

Gradient sparsity is the same fact from the other side

Chapter 1 established that gradient flows only to the argmax of each dimension. Theorem 2 says the forward pass depends only on those same points. These are one statement, read forwards and backwards.

The practical consequence bites during training, not inference. Early in training, before h has learned anything useful, the winners are essentially arbitrary — whichever points happen to sit far out along the random directions the initialisation produced. Those arbitrary points get all the gradient. As h reshapes, the winners migrate, and the set of points receiving learning signal migrates with them.

This is why the model benefits from the jitter augmentation the paper applies — Gaussian noise at σ = 0.02 on every coordinate, every epoch. Jitter continuously reshuffles which point is marginally extremal in each direction, so the network cannot latch onto specific sampled points and must learn a function of the underlying surface. Without it you would be fitting the sampling, not the shape.

The visualisation the paper is famous for

When the authors rendered C_S for real shapes, the critical points did not look random. They looked like the skeleton of the object: wing tips and tail fins on an aeroplane, chair legs and the seat's four corners, the rim and handle of a mug. The network had, without being asked, learned to summarise a shape by its extremal and distinctive structure — which is what a human sketching the object from memory would also do.

Two consequences the paper draws out. First, C_S and N_S generalise: rendered for objects from categories never seen in training — a teapot, a bunny, a hand, a human body — the critical sets still trace the skeleton, so the per-point functions h learned something about geometry rather than about ModelNet's furniture. Second, because two similar shapes activate overlapping dimensions, you can build a crude shape correspondence by matching points that win the same dimension in both clouds — a free by-product of the architecture.

Push the buttons in order and read the number. Deleting half the non-critical points leaves the descriptor at distance 0.000. Adding a hundred points inside the shaded polygon leaves it at 0.000. One point past the boundary and it jumps. Then raise K from 4 to 16 and watch the critical set grow and the shaded region tighten — more probes means a finer summary means less slack for corruption to hide in, which is Theorem 2(b) and the "K = 64 to 1024 buys 2–4%" measurement, both visible at once.

Reading the bottleneck ablation properly

The paper reports that raising the pooled width K from 64 to 1024 buys 2–4% accuracy, and that accuracy grows with the number of input points but saturates around 1K. Those two curves say something specific when you put them next to Theorem 2.

K is the number of probes. It is your descriptive vocabulary — how many independent questions the network is allowed to ask about the shape before the answers get concatenated into a vector. Doubling K doubles the questions and, because the questions overlap, delivers less than double the information. Hence the diminishing curve.

n, the number of input points, is the resolution of the surface being asked about. And here is why it saturates: once n is large enough that every one of the K probes has a well-determined winner — that is, once the surface is sampled finely enough that adding more points does not change any extremum — extra points contribute nothing at all. They are all non-critical by construction. The paper measures the saturation point at roughly 1000 for ModelNet40 objects, and the number is not universal: it depends on K and on how convoluted the shapes are.

The by-product nobody planned: shape correspondence

Here is a consequence of Theorem 2 that the paper puts in the supplementary and that deserves better placement.

Take two chairs. Each has a critical set, and each critical point is associated with the specific dimensions of the max that it won. Now match points across the two shapes by pairing those that activate the same dimensions. Because dimension j corresponds to one learned point function h_j — "the thing that responds to this kind of local extremity" — the point that wins dimension j on chair one and the point that wins dimension j on chair two are, by construction, playing the same structural role.

The paper's Figures 13 and 14 show this producing sensible correspondences between pairs of chairs and pairs of tables: back-left leg to back-left leg, seat corner to seat corner. Nothing in the training objective asked for correspondence. It falls out because the max's argmax is an index, and indices are comparable across clouds.

This matters beyond the pretty picture. Correspondence is the input to registration, to shape morphing, to transferring annotations from one model to another. Getting a rough version of it for free, out of a classifier, is the kind of thing that tells you the representation is doing something structurally right rather than fitting a benchmark.

The same result is also the flaw

Everything above is a virtue when you are worrying about missing returns and sensor noise. Turn it around and it is the central weakness.

If a shape is summarised by at most K extremal points, then two shapes that agree on those points are indistinguishable, no matter how they differ elsewhere. Fine texture, small parts, the difference between a smooth surface and a lightly ridged one — all of it lives in the non-critical points, and the non-critical points do not reach the descriptor. The paper is candid about the symptom on ModelNet40: the gap to MVCNN "is due to the loss of fine geometry details that can be captured by rendered images."

And there is a second, deeper version of the problem, which Chapter 4 is about. Because pooling is global, the descriptor knows that a wing-tip response fired somewhere in the cloud, but not where, and not what was near it. Nothing in the architecture ever asks "what does the neighbourhood around this point look like?" — and neighbourhoods are where geometry lives.

Chapter 3: T-Net — Alignment and Its Limits

Property 3 from Chapter 0 is still unpaid. Rotate the chair thirty degrees about the vertical axis and it is still a chair, but every coordinate in the array has changed, so every pre-activation of h has changed, so the descriptor has changed. Permutation is handled. Rigid motion is not.

The classical fix is canonicalisation: before doing anything, rotate the object into a standard pose. Run PCA on the point coordinates, take the principal axes, rotate so that they align with the coordinate frame. This works until it does not — a symmetric object has ambiguous principal axes, a partial scan has axes determined by which half you happened to see, and a flat object has two nearly equal eigenvalues so the axis choice flips under noise.

Learn the alignment instead

PointNet's answer is a small network that predicts the alignment from the data. The authors call it a T-Net, and they are explicit that they are borrowing the spatial transformer idea from Jaderberg et al. and getting it much cheaper. In images, applying a predicted transform means resampling a grid with interpolation, which introduces aliasing and needs a custom layer. In point clouds, applying a transform is a matrix multiply on the coordinates. No resampling. No alias. No new layer type.

And the T-Net itself is a mini-PointNet, which is a pleasing recursion. Its architecture, from the supplementary material: shared MLP with widths 64, 128, 1024 on each point; max pool across points; two fully connected layers of width 512 and 256; a final layer emitting the nine numbers of a 3×3 matrix, initialised to the identity. Batch norm and ReLU everywhere except the last layer.

Initialising the output at the identity matters. At step zero the module is a no-op, so training starts from plain PointNet and the alignment is learned as a correction rather than fought for from noise.

Watch it undo a rotation, numerically

Make the mechanism concrete on two points. Suppose the canonical pose of a small object has points p = (1, 0) and q = (0, 1), and the scan arrives rotated by 30 degrees. The rotation matrix is

so the observed points are Rp = (0.866, 0.500) and Rq = (−0.500, 0.866). Any per-point function h now receives entirely different inputs than it did in the canonical pose, and the pooled descriptor moves.

What the T-Net must learn is to output T = R⁻¹ = R^T (rotations are orthogonal, so the inverse is the transpose) for this cloud — and something different for every other pose. Apply it:

Canonical pose recovered exactly. Now look at what that requires of the module. The T-Net must produce a different output for every input pose, and the mapping from "cloud as seen" to "the rotation that undoes it" is exactly the pose-estimation problem you were trying to avoid solving. The T-Net does not sidestep it — it learns an approximation of it, and the quality of that approximation is the quality of your rotation invariance.

And here is the recursive charm of it: the T-Net is a mini-PointNet, so it inherits Theorem 2. Its output also depends on at most K critical points of the cloud. The predicted alignment of a chair is being decided by a handful of extremal points — which is defensible (the extremal points do carry most of the pose information) and also fragile in exactly the situation you would predict: when the critical points are near-symmetric and the true pose is ambiguous.

The second T-Net, and why it needs a regulariser

If aligning coordinates helps, why not align features? After the first shared MLP the points carry 64-dim vectors; insert a second T-Net that predicts a 64×64 matrix and multiply. Same idea, one level up.

Except it does not work out of the box, and the reason is dimensional. A 3×3 matrix has 9 free numbers. A 64×64 matrix has 4,096. That is a much larger, much less constrained thing to regress, and the optimisation becomes unstable. So the paper adds a term to the loss that pushes the predicted matrix toward orthogonality:

with weight 0.001 alongside the softmax classification loss. The justification is one sentence in the paper and worth expanding: an orthogonal transformation does not lose information. Orthogonal means the rows are unit-length and mutually perpendicular, so the map is a rotation or reflection — it can reorient the feature space but cannot squash it. If the matrix were allowed to be, say, rank 20, the module would silently destroy 44 dimensions of feature before pooling ever saw them.

Work it out on a 2×2. Suppose the mini-network predicts

Then A A^T = [[1² + 0.5²,  1(0) + 0.5(1)], [0(1) + 1(0.5),  0² + 1²]] = [[1.25, 0.5], [0.5, 1]]. Subtract from the identity:

The squared Frobenius norm is the sum of squared entries: 0.0625 + 0.25 + 0.25 + 0 = 0.5625. Times the weight 0.001 gives a penalty of 0.00056 added to the loss — small, deliberately, because this is a nudge and not a constraint. Now try a genuine rotation, A = [[cosθ, −sinθ], [sinθ, cosθ]]: A A^T = I exactly, for any θ, so L_reg = 0. The penalty is zero on the whole rotation group and grows with shear and scaling. That is precisely the shape you want.

The ablation, and the part everyone skips

Table 5 of the paper is four lines and a lesson.

Line three is the one people skip: the feature transform, added on its own without the orthogonality penalty, makes the model worse. Adding capacity hurt. The regulariser converts it from −0.2 to +0.3 — and only in combination with the input transform does the whole thing produce +2.1.

What the alignment costs

Now count, because this is where the parameter budget actually goes. Recall a linear layer a→b costs ab + b.

Input T-Net (3-dim input, 9-dim output): 3→64 is 256; 64→128 is 8,320; 128→1024 is 132,096; then 1024→512 is 524,800; 512→256 is 131,328; 256→9 is 2,313. Total 799,113.

Feature T-Net (64-dim input, 4096-dim output): 64→64 is 4,160; 64→128 is 8,320; 128→1024 is 132,096; 1024→512 is 524,800; 512→256 is 131,328; and the last layer, 256→4096, is 256(4096) + 4096 = 1,052,672 on its own. Total 1,853,376.

Which reproduces the "PointNet 3.5M" row of Table 6 exactly. Read the split again: the two alignment modules are 77% of the entire network, and a single output layer — the one that emits 4,096 numbers — is 30% of the whole model by itself. You are paying 2.65M parameters, more than three times the base network, to buy 2.1 points of accuracy.

That is a defensible trade in a 2016 benchmark paper. It is a much less obvious trade in a robot that has a 2 ms budget per sweep, and it is the first thing most practitioners drop. Table 6 tells you what dropping it buys: vanilla PointNet is 0.8M params and 148M FLOPs against the full model's 3.5M and 440M — a third of the compute for 87.1% instead of 89.2%.

Why identity initialisation, precisely

"Initialise the output at the identity" sounds like a nicety. It is closer to a necessity, and the reason is a chicken-and-egg problem in the gradient.

Suppose the final layer of the T-Net were initialised the usual way, with small random weights around zero. Then at step one the predicted matrix T is approximately the zero matrix, so X·T ≈ 0, so every point in every cloud collapses to the origin. The main network then sees an identical, information-free input for every example, produces an identical output, and the gradient with respect to T carries no signal about which alignment would have been better — because no alignment was tried.

The standard implementation adds a constant identity to the regressed matrix:

python# The T-Net's last layer regresses a *correction*, not the matrix itself.
def forward(self, x):              # x: (B, 3, N)
    f = self.h(x).max(dim=2)[0]     # (B, 1024)  a mini-PointNet — same recipe
    d = self.fc(f).view(-1, 3, 3)     # (B, 3, 3)  initialised near zero
    I = torch.eye(3, device=x.device)
    return d + I                   # (B, 3, 3)  starts AT the identity, learns a delta

Now step one is a no-op, training begins from exactly the vanilla model's behaviour, and every gradient step asks the meaningful question — which small departure from doing nothing reduces the loss? This "regress a residual from the identity" pattern shows up everywhere a network predicts a transformation, and the failure mode it avoids is always the same: a randomly initialised transform destroys the signal it was supposed to reorient.

Note also what the mini-PointNet inside the T-Net implies. The predicted alignment is itself a permutation-invariant function of the cloud — it has to be, or the alignment would depend on the row order and you would have reintroduced the bug you were fixing. Every module in this architecture that touches the whole cloud is either symmetric or broken.

The alternatives, and when each is right

Read the guarantee column. The two rows with exact guarantees achieve them the same way PointNet achieves permutation invariance — by removing the degree of freedom from the function class rather than training it away. The T-Net sits in the awkward middle: a learned approximation with a real parameter cost and no theorem. It won its ablation in 2016 and it is the piece of PointNet that aged least well.

The honest limits of learned alignment

T-Net is a good engineering answer, and it is not the same thing as rotation invariance. Four caveats worth carrying:

It is canonicalisation, not equivariance. The network predicts a pose correction and hopes it is right. There is no theorem saying the same shape at two orientations gets mapped to the same canonical frame. In practice it is approximately true for orientations resembling the training distribution and degrades outside it — and PointNet's ModelNet40 augmentation only rotates about the up-axis, so the model is never asked to handle arbitrary SO(3) at training time.

The predicted matrix is affine, not rigid. Nine free numbers can shear and scale, not just rotate. For the 3×3 case the paper accepts this without a regulariser; the module is free to apply a mild affine warp if that happens to reduce loss.

It costs more than the network it corrects. See the arithmetic above.

It does not address Property 2 at all. Aligning the whole cloud does nothing about local structure. You can canonicalise a chair perfectly and the model still has no mechanism for asking what a chair leg looks like up close. Which brings us to the sequel.

Chapter 4: The Flaw, and Set Abstraction

Six months after PointNet, the same four authors published a paper whose second sentence is a critique of their own: "However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes."

Let us be precise about what is missing, because "no local structure" is a slogan until you can point at the line of code.

Where the locality went

Return to the tensor table of Chapter 1. From the input to the max pool, every point is processed alone. Point 17's 1024-dim feature is a function of point 17's three coordinates and nothing else. Then one max pool collapses everything. So the network has exactly two scales: the individual point, and the entire cloud. There is nothing in between.

Compare a CNN, which is nothing but in-between scales. A 3×3 kernel at layer 1 sees a 3×3 pixel patch. At layer 2, after a stride-2 pool, its 3×3 kernel sees an effective 7×7 of the original. By layer 5 it sees most of the image. That growing receptive field is the whole reason CNNs generalise: low layers learn edges that are reusable everywhere, high layers compose them into objects.

The practical symptom shows up worst in scenes. On a single ModelNet40 object, normalised into a unit sphere, "global" is a reasonable scale. On a scanned room, the global pool covers a 1.5-metre cube containing a table, part of a chair, some floor and a wall — and one max over that is a very blunt instrument. The paper's Figure 6 is exactly this: PointNet "captures the overall layout of the room correctly but fails to discover the furniture."

The fix, in one sentence

Apply PointNet recursively on a nested partition of the point set.

That is genuinely the whole idea. If PointNet is a good way to summarise a set of points into a vector, then use it to summarise small neighbourhoods into vectors, treat those vectors as new points at their neighbourhood centres, and repeat. You have built a hierarchy with growing receptive fields, out of the module you already have.

The unit of that hierarchy is called a set abstraction level, and it has three layers.

where d is the coordinate dimension (3) and C, C' are feature widths. The level takes a big set of thin points and returns a smaller set of fat points. Stack three of these and you have PointNet++.

Sampling: why farthest point, not random

Farthest point sampling (FPS) is greedy and one line to state: start from a point, then repeatedly add whichever remaining point is farthest from the set you have already chosen.

Run it by hand. Six points in the plane:

Step 1. Seed with p₁. Distances from p₁: p₂ = 1, p₃ = 1, p₄ = √2 = 1.414, p₅ = √0.5 = 0.707, p₆ = √(0.01+0.0025) = 0.112. The largest is p₄. Pick p₄.

Step 2. Recompute each remaining point's distance to the nearest chosen point. p₂: min(1, 1) = 1. p₃: min(1, 1) = 1. p₅: min(0.707, 0.707) = 0.707. p₆: min(0.112, 1.309) = 0.112. The largest is 1, a tie between p₂ and p₃. Pick p₂ (ties broken by index).

Step 3. p₃: min(1, 1, 1.414) = 1. p₅: 0.707. p₆: 0.112. Pick p₃.

The FPS order is p₁, p₄, p₂, p₃, p₅, p₆ — the four corners first, then the middle, then the near-duplicate last. Now do it randomly and you may well draw p₁, p₆, p₅: two of your three centroids are 0.112 apart and a whole corner of the shape is unrepresented.

The quantity FPS is actually greedily minimising is the covering radius: after m picks, the largest distance from any point to its nearest centroid. In the run above, that number goes 1.414 → 1.0 → 1.0 → 0.707. Every input point is guaranteed to be within the covering radius of some centroid, and that is exactly the guarantee you need before you go asking about neighbourhoods.

FPS costs O(N N') — each of N' picks scans all N remaining distances. For 1024 → 512 that is about half a million distance comparisons, which is why it is written as a CUDA kernel and why it shows up in profiles. And because the first seed is arbitrary, the whole thing is randomised. Does that matter? The authors measured it: over ten different seeds on ModelNet40, the standard deviation of the global feature norm is 0.021 (a 2.1% deviation) and the standard deviation of test accuracy is 0.0017 — 0.17%. Stable enough to ignore.

Grouping: ball query, and why not kNN

You have centroids. Now decide who belongs to whom. Two obvious options:

k-nearest neighbours: take the K closest points to each centroid. Guarantees a fixed group size. The scale of the group varies with density — in a dense region the 32 nearest points span 2 cm, in a sparse region they span 40 cm.

Ball query: take all points within radius r, capped at K. Guarantees a fixed physical scale. The group size varies with density — and can be zero.

PointNet++ chooses ball query, and the reasoning is exactly about weight sharing. The mini-PointNet applied to every group has shared weights. Shared weights only make sense if the groups are comparable objects. Under ball query, "a group" always means "a 20 cm ball of surface", everywhere in the cloud, so a learned detector for "20 cm of curved edge" is meaningful. Under kNN, "a group" means a different physical thing in every part of the cloud, and the shared detector has no consistent quantity to detect. In the paper's words, ball query "guarantees a fixed region scale thus making local region feature more generalizable across space, which is preferred for tasks requiring local pattern recognition."

They measured it too, on uniformly-sampled ModelNet40 with 1024 points:

Two readings. First, on uniform data the gap is small — 90.7 versus 90.3 — and the authors say so honestly, adding "we speculate in very non-uniform point set, kNN based query will result in worse generalization ability." Second, look at the r = 0.1 row. A smaller neighbourhood hurt, which contradicts the CNN folklore that smaller kernels are better. Chapter 5 explains why, with arithmetic.

Note also the cap. Ball query returns a variable number of points, but GPUs want rectangular tensors, so implementations pad or subsample to a fixed K per group. This is harmless in a way that is worth appreciating: the mini-PointNet ends in a max pool over the group, and duplicating a point in a max pool changes nothing.

The PointNet layer, and the one line that does the real work

Each group arrives as a tensor of shape N' × K × (d + C). Before anything else, the coordinates are re-expressed relative to the centroid:

This tiny line is doing three jobs at once, and it is worth naming all three.

It makes the group translation invariant, exactly. Move the whole cloud a metre to the left and every centroid moves with it, so every relative coordinate is unchanged, so every group feature is unchanged. No learned module, no regulariser, no approximation — it is a subtraction. Compare the T-Net of Chapter 3, which spends 0.8M parameters to approximate a global alignment.

It makes weight sharing meaningful. Every group now lives in its own little box centred at the origin with radius r. A shared MLP sees comparable inputs everywhere in the cloud, which is the whole point of sharing.

It encodes the geometry rather than the position. "Point at (3.4, 0.2, 1.1)" is a fact about where the sensor was. "Point at (+0.03, −0.01, +0.02) from my centroid, alongside 31 others" is a fact about shape. The second one generalises.

Then a shared mini-PointNet — MLP on each of the K points, max pool over the K — turns each group into one C'-dim vector, and the level outputs N' × (d + C'): the centroid coordinates, in the original frame, plus the new feature.

Ball query in practice: the cap, the padding, and a bias nobody mentions

Ball query returns "all points within r". GPUs want rectangles. The reconciliation has three details that will bite you if you do not know them.

The cap. Implementations set an upper limit K on group size — typically 32 or 64. Once K neighbours are found, the search stops. In a dense region the ball genuinely contains more than K points and you keep a subset.

The padding. If the ball contains fewer than K points, the remaining slots are filled by repeating the first found neighbour (in the reference implementation, index 0 — usually the centroid itself). This looks like a hack and is almost harmless, for a specific reason: the group ends in a max pool, and duplicating an entry in a max pool changes nothing. Had the aggregator been a mean, padding would have systematically biased every sparse group toward its first neighbour, and the whole density story of Chapter 5 would be worse than it is. The choice of max in 2016 quietly paid off in 2017.

The bias. The reference kernel takes the first K points it encounters within the radius, in input array order — not the K nearest. So in a dense region, which subset of the ball you keep depends on the ordering of the input array. That is a mild permutation dependence smuggled back in through the implementation, and it is the reason two runs on the same cloud with rows shuffled can differ by a hair. It does not matter much in practice — the group is max-pooled and the kept subset is a random sample of a well-populated ball — but it is worth knowing that the model's exact invariance stops at the layer boundary.

Concept and realization: one set abstraction level

python# One set abstraction level: sample -> group -> mini-PointNet.
# xyz:   (B, N, 3)        coordinates in the ORIGINAL frame
# feat:  (B, N, C)        per-point features from the previous level (or None)
def set_abstraction(xyz, feat, n_centroids, radius, k_cap, mlp):
    idx      = farthest_point_sample(xyz, n_centroids)   # (B, N')   O(N*N') greedy
    new_xyz  = gather(xyz, idx)                          # (B, N', 3)

    grp_idx  = ball_query(radius, k_cap, xyz, new_xyz)   # (B, N', K) padded with idx 0
    grp_xyz  = gather(xyz, grp_idx)                      # (B, N', K, 3)

    # THE line: express each neighbour relative to its centroid.
    # Exact translation invariance, zero parameters, one subtraction.
    grp_xyz  = grp_xyz - new_xyz.unsqueeze(2)            # (B, N', K, 3)

    if feat is not None:                              # carry features up the hierarchy
        grp  = cat([grp_xyz, gather(feat, grp_idx)], dim=-1)  # (B, N', K, 3+C)
    else:
        grp  = grp_xyz                                   # (B, N', K, 3)

    grp      = mlp(grp)                                  # (B, N', K, C')  shared over N' AND K
    new_feat = grp.max(dim=2)[0]                       # (B, N', C')    a PointNet per group
    return new_xyz, new_feat                             # (B, N', 3), (B, N', C')

Read the last three lines and compare them with the PointNet listing in Chapter 1. mlp then max(dim=2) — the pooling axis is now the within-group axis K instead of the whole-cloud axis N, and that is the entire difference. PointNet++ is PointNet with the max moved from global to local, wrapped in a sampler and a grouper, and repeated.

Note also which coordinates are returned. new_xyz is in the original frame, unshifted — the centroid subtraction applies only to the grouped neighbours, inside the level. Each level therefore hands the next one a legitimate, smaller point cloud, in world coordinates, with fatter features. That is what makes the recursion type-check.

And note the memory. At level 1 with N' = 512, K = 32, C' = 128, the intermediate grouped tensor for a batch of 16 is 16 × 512 × 32 × 128 = 33.5 million activations before the pool collapses it to 16 × 512 × 128 = 1.05 million. The grouping step inflates the tensor by a factor of K and then throws it away. That factor of 32 is where PointNet++'s memory goes, and it is why the sensible knob when you run out of memory is K, not the MLP width.

The actual classification hierarchy, with arithmetic

Here is the network from the supplementary material, in the paper's own notation, where SA(N', r, [widths]) is a set abstraction level:

Walk the shapes for a 1024-point cloud, and keep an eye on the physical scale, since the cloud is normalised into a unit ball:

Notice the last set abstraction level has no radius and no centroid count. SA([256, 512, 1024]) is a global set abstraction — it takes all 128 remaining points as one group and pools them to a single vector. Which is to say: PointNet++ ends in PointNet. The whole architecture is two levels of local abstraction feeding the original 2016 model.

And notice the receptive-field column. A level-2 group has radius 0.4 in the coordinates of level-1 centroids, but each of those already summarises a ball of radius 0.2. So the level-2 feature depends on original points up to roughly 0.6 away — radii compose exactly the way strided convolutions compose. The 1024→512→128→1 point counts are the analogue of pooling strides, and 0.2→0.4 is the analogue of a growing kernel.

Three things to watch. The covering radius readout drops fast under FPS and erratically under random sampling — that is the coverage guarantee, visible. The empty groups counter is what happens when the radius is smaller than the local spacing; those centroids produce a feature computed from nothing but padding. And the group size spread readout shows the density problem directly: at a single radius, groups on the dense side hit the cap while groups on the sparse side hold two or three points. That spread is what Chapter 5 exists to fix.

Chapter 5: Density, and Multi-Scale Grouping

Every experiment so far has quietly assumed something false: that points are spread evenly over the surface. On a CAD model sampled uniformly by face area, they are. On a real scan, they are emphatically not.

The paper opens with a photograph of a Structure Sensor scan for exactly this reason. Density in a real capture varies because of perspective (a surface at 2 m gets far more returns per square centimetre than the same surface at 20 m), incidence angle (a wall face-on catches a dense grid; the same wall at a glancing angle catches a smear), radial pattern (a spinning lidar has fixed angular spacing, so vertical point spacing grows linearly with range), and motion (a moving platform stretches and compresses the sweep).

How many points are in a ball? Compute it.

This is the arithmetic that makes the whole chapter click, and it is two lines.

Points sampled from a shape do not fill a volume; they lie on a surface. So the expected number in a ball of radius r scales with the ball's cross-sectional area, not its volume. For a cloud of N points spread evenly over a surface of total area A, and a small patch that is locally flat:

Take the paper's own normalisation: the cloud is scaled into a unit sphere. Use the unit sphere's own surface, A = 4π, as a stand-in. The π cancels beautifully:

Now plug in N = 1024, the standard input size, and the exact radii PointNet++ uses:

Read the first row twice. At training density, a radius-0.1 ball holds about two and a half points — already marginal. Drop the cloud to a quarter of its density and the same ball holds less than one point on average. A "local pattern detector" whose input is, most of the time, nothing at all.

This is the mechanism behind the r = 0.1 → 89.1% versus r = 0.2 → 90.7% row of Chapter 4's table, and it is why the paper writes that its experiments "give counter evidence" to the CNN rule that smaller kernels are better. In an image, a 3×3 kernel always sees exactly nine values. In a point cloud, a small radius sees however many points happened to land there — and the answer is often zero.

Multi-scale grouping: refuse to choose

If no single radius is right everywhere, do not pick one. At each centroid, run the grouping and the mini-PointNet at several radii in parallel, and concatenate the results.

Here is the actual MSG classification network from the supplementary:

Level 1 groups the same 512 centroids three times, at radii 0.1, 0.2 and 0.4, with three separate mini-PointNets producing 64, 128 and 128 channels. Concatenated, each centroid carries 64 + 128 + 128 = 320 features instead of the single-scale version's 128. Level 2 does the same at 0.2 / 0.4 / 0.8, producing 128 + 256 + 256 = 640.

Note the width allocation: the smallest radius gets the fewest channels (64) and the larger radii get more. That is a sensible prior — a two-point neighbourhood cannot support 128 independent descriptors, and the authors are not pretending otherwise.

Why the small radius gets the fewest channels. This looks like a detail and it is a statement about information. A radius-0.1 ball at training density holds about 2.56 points; after centroid subtraction those points span 3 × 2.56 ≈ 8 numbers of geometric content. Asking a network to extract 128 independent descriptors from roughly eight numbers is asking for 120 dimensions of noise. Sixty-four is still generous. Meanwhile the radius-0.4 ball holds about 41 points, or 123 numbers, and can genuinely support a wider description. The channel allocation tracks the information available at each scale, and if you change the radii you should change the widths with them.

What MSG costs, counted. The expensive object in a set abstraction level is the grouped tensor, of size N' × K × C. Single-scale grouping at level 1 builds one such tensor. MSG builds three, and the largest radius has the most neighbours to gather. Using the occupancy formula, the three balls hold about 2.56, 10.24 and 40.96 points, so if the cap K allows it the gathered volume is roughly 21× the smallest branch alone and about 5× the single-scale r = 0.2 branch. The measured cost is a clean 2× overall (163.2 ms versus 82.4 ms), because the caps truncate the large balls and because the levels above are unchanged. Still: MSG is not free, and it is most expensive exactly where you have the most centroids — the first level.

Random input dropout: teaching the model that density varies

Concatenating scales is only half the job. If you train exclusively on 1024 uniformly sampled points, the network never encounters a situation where the small radius is useless, so it never learns to discount it. At test time on a sparse scan it will happily trust a feature computed from noise.

So the paper adds an augmentation with a slightly unusual shape. For each training cloud, draw a dropout ratio

then drop each point independently with probability θ. Two sources of variation, deliberately: θ varies the sparsity from cloud to cloud, and the per-point coin flip varies the uniformity within a cloud. The cap at 0.95 rather than 1.0 exists only to avoid producing an empty set. At test time nothing is dropped.

The effect is that the network sees, across a training run, clouds ranging from 1024 points down to about 50, and it must produce a good answer for all of them from the same weights. The only way to do that is to learn a combination rule — effectively, to weight the small-radius branch down when it looks starved and up when it looks rich. The paper calls this learning "to adaptively weight patterns detected at different scales."

The result: MSG+DP loses less than 1% accuracy going from 1024 test points down to 256. Ablated single-scale grouping (SSG) "fails to generalize to sparse sampling density", and SSG+DP fixes most of it. Interestingly, plain PointNet is also fairly robust here — because its only scale is global, and a global summary degrades gracefully — but it starts from a lower ceiling.

The density regimes you actually meet

The occupancy formula is more useful once you attach real situations to its regimes. Here is the same arithmetic, translated.

The third row is the one production systems trip over. Both papers' experiments use object scans and room scans with roughly isotropic local sampling; a spinning lidar does not have that property at all. Points along a laser ring are 2–5 cm apart at 20 m while adjacent rings are 30 cm apart, so a spherical ball query at any radius either grabs a one-dimensional string of points from a single ring or reaches across three rings and includes a great deal of range. Neither is the neighbourhood you wanted. This anisotropy, more than anything in the papers, is why outdoor-lidar perception drifted toward sparse voxel convolutions, where the neighbourhood is defined by the grid rather than by a metric ball.

What multi-scale grouping does not fix

MSG makes the model robust to density variation. It does not create information that the sensor did not capture. Three limits worth stating plainly:

It cannot resolve detail below the sampling. If the far half of your scan has 2 cm point spacing, no grouping rule recovers a 5 mm feature there. MSG's honest contribution is to degrade to a coarser description rather than produce a confident wrong one.

It does not adapt the radii to the data. The radii are hyperparameters, fixed at 0.1 / 0.2 / 0.4 for every centroid in every cloud. A genuinely adaptive scheme would pick a radius per centroid from the local spacing — and then lose the fixed-physical-scale guarantee that motivated ball query over kNN in the first place. The tension is real and PointNet++ resolves it by refusing to choose, at 2× the cost.

It doubles the compute for a benefit that is zero on uniform data. If your deployment genuinely has uniform density — some industrial inspection setups do — MSG buys you nothing and costs you 80 ms per batch. Measure your density distribution before you pay for this.

Multi-resolution grouping: the cheaper cousin

MSG is expensive, and the paper says why in a sentence worth quoting: it "runs local PointNet at large scale neighborhoods for every centroid point", and at the lowest level the centroid count is highest. Running a radius-0.4 PointNet at all 512 level-1 centroids is a lot of gathered points.

Multi-resolution grouping (MRG) gets a similar effect for less. The feature of a region at level L_i is the concatenation of two vectors:

When the region is dense, vector 1 carries real detail and should dominate. When it is sparse, vector 1 was computed from sub-regions containing almost nothing and vector 2 should dominate. The network learns that weighting. And the saving is real: no large-radius grouping at the lowest, most populous level.

Batch size 8, TensorFlow 1.1, a single GTX 1080. Two observations that matter for anyone shipping this. First, PointNet++ SSG is the smallest model in the table at 8.7 MB and still seven times slower than vanilla PointNet — because the cost has moved out of the matrix multiplies and into FPS and neighbourhood gathering, which are memory-bound scatter/gather operations, not FLOPs. A model that is small and slow is a strong hint that your bottleneck is data movement. Second, MSG doubles the time for its density robustness; MRG buys most of the same robustness for about 5% over SSG. If you are latency-bound on a robot, MRG is the line to read.

Slide it down to about 25% and read the r = 0.1 ring. In SSG mode that is the entire feature, and it is empty. In MSG mode the two larger rings are still populated, and the concatenated feature keeps carrying signal at a coarser granularity. That is the whole argument of Section 3.3, in one picture.

Chapter 6: Segmentation by Feature Propagation

Classification wants one vector for the whole cloud, and both architectures deliver it. Segmentation wants one label per point — and both architectures have just spent their entire forward pass throwing points away.

PointNet's answer is simple and worth understanding before the harder one. After the global max pool produces a 1024-dim shape descriptor, copy it back onto every point and concatenate it with that point's own 64-dim local feature:

Then run another shared MLP down to m class scores. Each point now decides its label knowing two things: what it is locally, and what the whole object is. That is enough to say "this is a leg-like local structure, and the object is a chair, therefore chair leg" — and the paper notes it is enough to predict per-point normals accurately, which is a real check that local information survived. It reached 83.7% mean IoU on ShapeNet part segmentation, a 2.3-point improvement on the previous best.

But notice what the global vector is: one 1024-dim summary, identical for every point. The only per-point signal is the 64-dim feature computed from that point's raw coordinates alone. There is still nothing about the neighbourhood.

PointNet++ has a harder problem

In PointNet++ the situation is worse, because subsampling is the architecture. After three levels you have 1024 → 512 → 128 → 1 points. Even if the final features are excellent, they live at 128 centroids and you need labels at all 1024 originals.

One option is to never subsample — sample all points as centroids at every level. That works and costs a fortune. The chosen option is to propagate features back up the hierarchy, in a U-Net-shaped decoder made of feature propagation (FP) levels.

Each FP level moves features from N_l points back to the N_l−1 points of the level below, in three steps.

Repeat until you are back at the original points, then a final MLP produces per-point class scores.

The interpolation, written out and worked

with the paper's defaults p = 2 and k = 3. This is classic inverse distance weighting, also known as Shepard interpolation: nearby sources dominate, and the weights are normalised so that if all three neighbours agree, the answer is that value exactly.

Work one channel by hand. Target point x has three nearest coarse points at distances 0.1, 0.2 and 0.4, carrying values 2.0, 5.0 and 9.0 in the channel we are tracking.

The answer is 2.905, very close to the nearest neighbour's 2.0 and barely influenced by the value 9.0 sitting four times further away. That is the p = 2 exponent doing its job: halving the distance quadruples the weight. The nearest source at 0.1 carries 76% of the total weight; the farthest at 0.4 carries 4.8%.

Change p to 1 and recompute to see the sensitivity: weights 10, 5, 2.5, sum 17.5, giving (20 + 25 + 22.5)/17.5 = 3.857. Much more blended. Push p toward infinity and it becomes pure nearest-neighbour copying, with a hard discontinuity everywhere the nearest neighbour changes. p = 2 is a compromise between blur and blockiness, and it is the same reason inverse-square weighting shows up everywhere from radial basis functions to point-cloud rendering.

Concept and realization: a feature propagation level

python# Move features from the COARSE set (xyz2, feat2) back onto the FINE set (xyz1),
# then fuse with the encoder's own features for those fine points.
# xyz1: (B, N1, 3)   xyz2: (B, N2, 3) with N2 <= N1
def feature_propagation(xyz1, xyz2, skip_feat, feat2, unit_mlp):
    if xyz2.shape[1] == 1:                      # the global-vector special case:
        interp = feat2.repeat(1, xyz1.shape[1], 1)   # one source -> pure copy,
    else:                                        # which is PointNet's own trick.
        d, i   = three_nn(xyz1, xyz2)             # (B, N1, 3) dists + source indices
        w      = 1.0 / (d + 1e-8)**2              # p = 2; epsilon guards a coincident point
        w      = w / w.sum(dim=2, keepdim=True)  # normalise so identical sources -> that value
        interp = (gather(feat2, i) * w.unsqueeze(-1)).sum(dim=2)   # (B, N1, C2)

    # The skip link: the ONLY thing in the decoder that knows where boundaries are.
    fused = cat([interp, skip_feat], dim=-1) if skip_feat is not None else interp
    return unit_mlp(fused)                       # (B, N1, C') — shared, per-point, 1x1

Three details in that listing earn their place.

The epsilon. If a fine point coincides exactly with a coarse point — which happens constantly, because the coarse points are a subset of the fine ones, chosen by FPS — then d = 0 and 1/d² is infinite. The guard turns that into an enormous but finite weight, so after normalisation the coincident source gets essentially all of it. Which is correct behaviour: a point that is a coarse point should receive that coarse point's feature.

The normalisation. Dividing by the weight sum is what makes the interpolation an average rather than a sum. Without it, a fine point surrounded by three very close sources would receive a feature three hundred times too large, and a fine point in a sparse gap would receive one that is far too small. Normalisation makes the operation density-blind, which after Chapter 5 you should recognise as the recurring theme.

The unit PointNet is literally a 1×1 convolution. The paper says so — "similar to one-by-one convolution in CNNs" — and it is the same object as the shared MLP from Chapter 1: one linear map applied to every point independently. It is what fuses the interpolated coarse feature with the skip-linked fine feature into a single vector, and it is permutation-safe for the same reason everything else here is.

The real segmentation architectures

The ScanNet semantic labelling network, from the supplementary material, using FP(widths) for a feature propagation level:

Four levels down, four levels up, perfectly symmetric — an hourglass, with skip links across the waist. Read the radii on the way down: 0.1, 0.2, 0.4, 0.8, doubling each time, exactly like a stride-2 CNN doubling its effective kernel. Read the point counts: 1024, 256, 64, 16 — a factor of four each level. And note that the deepest level keeps 16 points rather than collapsing to one, because in a scene you want several coarse regions, not a single summary.

The ShapeNet part segmentation network is shallower, and it does collapse to a global vector, because a single object does have a meaningful global identity:

Note the first FP level here interpolates from a single point — the global vector — back onto 128 centroids. With one source, inverse distance weighting degenerates to copying, so this level is exactly PointNet's "concatenate the global vector onto every point" trick, recovered as a special case of the general mechanism.

Why not simply never subsample?

The paper mentions the alternative in one clause — "always sample all points as centroids in all set abstraction levels, which however results in high computation cost" — and it is worth pricing, because the number is startling.

Take the ScanNet configuration: 8192 input points per cube, four levels. With subsampling, the levels carry 1024, 256, 64 and 16 centroids. Without it, all four levels carry 8192. The grouped tensor at each level is N' × K × C, so level 1 alone goes from

and the deeper levels are far worse: level 4 goes from 16 × K × 512 to 8192 × K × 512, a factor of 512. Summing across levels, the encoder becomes roughly two orders of magnitude more expensive, and FPS — already O(N N') — becomes O(N²) at every level.

So the decoder is not an aesthetic choice. Downsampling is what makes the deep levels affordable, and feature propagation is the bill for it. This is the same trade a U-Net makes in images, and the interpolation-plus-skip structure is the same answer, arrived at for the same reason.

The ScanNet protocol, and the class imbalance underneath it

The ScanNet setup is worth knowing because it is a template for scene-scale point learning, and because two of its details are about a problem the architecture cannot solve.

The dataset is 1,513 scanned and reconstructed indoor scenes, split 1,201 train / 312 test. Training cubes are 1.5 m × 1.5 m × 3 m, sampled on the fly from a scene and randomly rotated about the vertical axis. A cube is kept only if at least 2% of its voxels are occupied and at least 70% of its surface voxels carry valid annotations — a filter, not a nicety, because a cube of empty air or unlabelled geometry contributes gradient noise and nothing else. Each cube is padded with augmented points to a fixed cardinality of 8,192. At test time the scene is split into cubes, every point gets a prediction, and points that appear in several cubes are resolved by majority vote.

Notice what that last step implies: the same physical point can get different labels from different cubes, because its position within the cube — and therefore its neighbourhood context — differs. Majority voting over overlapping windows is the point-cloud version of test-time augmentation, and it is doing real work.

The imbalance is the part the architecture does not address. In an indoor scan, floor and wall points outnumber chair and lamp points by a large factor, and a per-point cross-entropy will happily reach high overall accuracy by getting the structure right and the furniture wrong. That is exactly the failure the paper's Figure 6 shows for PointNet. Overall point accuracy and mean IoU diverge for this reason — PointNet's S3DIS numbers are 78.62% overall accuracy against 47.71 mean IoU — and if you report only the first you are reporting the floor.

What it bought

On ShapeNet part segmentation the jump is 83.7 → 85.1 mean IoU, which sounds modest until you note that PointNet++ used the same data with normals added and beat a spectral graph CNN without any eigendecomposition. On ScanNet the paper's Figure 6 makes the qualitative point that the table cannot: PointNet gets the room layout right and misses the furniture; PointNet++ segments the furniture. That is the local-structure argument, visible.

The S3DIS setup is worth knowing as an engineering pattern in its own right. The scans are split by room, then into 1 m × 1 m blocks; each point is a 9-dimensional vector — XYZ, RGB, and the point's normalised position within the room, 0 to 1 — and 4096 points are sampled per block on the fly during training, with all points used at test. That normalised-room-position channel is a quiet trick: it hands the network a global positional prior (ceilings are at the top, floors at the bottom) that the local block would otherwise have no way to know.

Chapter 7: The Numbers Then, the Lineage Now

It is easy, nine years on, to read PointNet as obviously correct. It is worth reconstructing what it was actually competing against, and by how much it won, before tracing where the idea went.

ModelNet40, 2016 to 2017

The benchmark: 12,311 CAD models across 40 mostly man-made categories, split 9,843 train / 2,468 test. PointNet's protocol — worth copying if you ever reproduce this — samples 1024 points uniformly on mesh faces proportional to face area (so large flat surfaces get more points, which is what a scanner would do), normalises into a unit sphere, and augments with random rotation about the up-axis plus per-point Gaussian jitter at σ = 0.02.

Three readings. First, PointNet ties the best volumetric method (89.2 overall) using one pass over raw points against twenty rendered subvolumes, at 8× fewer FLOPs. Second, MVCNN's 90.1 average-class is still ahead, and the paper attributes the gap to "loss of fine geometry details that can be captured by rendered images" — the Chapter 2 critical-points limitation, admitted. Third, PointNet++ closes and passes it: 90.7 on coordinates alone, 91.9 with normals and 5000 points.

The MNIST experiment in the PointNet++ paper is a good sanity check on the whole framing. Convert each digit image to a 2D point cloud of its lit pixel locations and classify the set. PointNet vanilla gets 1.30% error, PointNet 0.78%, PointNet++ 0.51% — against LeNet-5's 0.80% and Network-in-Network's 0.47%. A method that never learned that pixels have neighbours matches a purpose-built image CNN, on images.

And the non-Euclidean result deserves more attention than it usually gets. On SHREC15, 1200 non-rigid shapes in 50 categories, the authors built the metric space from geodesic distances along the surface rather than Euclidean distance through space, so that a horse's nose and its tail are far apart even when the horse is curled up:

The 60.18 → 94.49 jump from swapping the input feature, and the further 94.49 → 96.09 from swapping the distance function used for sampling and grouping, together make a structural point: PointNet++ is not tied to R³. Sampling and grouping only need a metric. Give them a geodesic one and the same architecture learns intrinsic surface structure. Give them a learned or graph metric and you have something close to a graph neural network.

The two architectures, side by side

Read the last row as the summary of the pair. PointNet is fast and blunt; PointNet++ is sharp and slow, and the slowness is not in the arithmetic. Every architecture since has been an attempt to keep the second column's accuracy at the first column's speed, and the levers they pull are always the neighbourhood machinery, never the MLPs.

Notice one more thing: PointNet++ dropped the T-Nets entirely. It is not that alignment stopped mattering; it is that centroid subtraction handles translation exactly and for free, and the remaining rotation problem was judged not worth 2.65M parameters. That is a good example of a later paper deleting its predecessor's most conspicuous module because a structural change made it redundant.

Part segmentation, and the partial-scan result that matters most

ShapeNet part segmentation is 16,881 shapes across 16 categories, annotated with 50 parts in total — most categories have two to five. The metric is mean IoU over parts, averaged within a category and then across categories, with the convention that an empty union counts as IoU 1 (otherwise a category whose part is genuinely absent would be punished for correctly predicting nothing).

PointNet's biggest per-category gains are on ear phone (+9.5 over Yi et al.) and cap (+4.8) — categories with few training shapes, 69 and 55 respectively. That is a signature of a method that is learning transferable geometry rather than memorising category-specific priors, and it is the segmentation-side echo of the Chapter 2 finding that critical sets generalise to unseen categories.

But the experiment worth taking away is the one people skip. The authors used a Kinect simulator to generate incomplete point clouds from six random viewpoints for every CAD model — the kind of one-sided, self-occluded scan a real depth camera produces — retrained with the same architecture and settings, and measured the loss:

Five points, for throwing away everything the camera could not see. That number is Theorem 2 again: a partial scan is a subset of the full cloud, and if the surviving subset still contains most of the critical points, the descriptor barely moves. The whole robustness story — deletion, outliers, occlusion, partial views — is one theorem seen through four different experiments.

What the first-layer kernels learned

PointNet++'s Figure 8 is the point-cloud equivalent of the famous Gabor-filter visualisation from the first layer of an image CNN, and it is done in a way worth copying. The authors laid a voxel grid over space, found for each first-level neuron the local point sets that activated it most strongly (the top 100), aggregated those into the grid, kept the cells with the most votes, and converted back to points. The result is a rendering of "what pattern does this neuron detect".

What came out, on a model trained on ModelNet40's mostly-furniture categories: planes, double planes, lines and corners. Which is exactly the vocabulary you would design by hand if you were writing a classical 3D feature detector, and exactly the analogue of edges and blobs in an image CNN's first layer. Set abstraction did not just improve accuracy — it produced the interpretable low-level vocabulary that PointNet, having no local layer, could not have.

Where the idea went: the lineage in four branches

Branch 1 — sparse convolution wins the scene-scale fight. The voxel idea did not die; it got fixed. Instead of allocating a dense tensor, store only occupied cells in a hash map and define convolution over that sparse structure — Choy et al.'s Minkowski networks generalised this to arbitrary dimensions, including a fourth time axis for lidar sequences. This keeps the CNN's clean local receptive fields and its highly optimised kernels while paying only for occupied space, and for large outdoor scenes it became the default backbone for perception stacks. PointNet's critique of voxels was about the dense tensor, and sparse convolution answered it directly.

Branch 2 — attention replaces the mini-PointNet. Zhao et al.'s Point Transformer kept PointNet++'s sampling-and-grouping skeleton and swapped the per-group max-pooled MLP for a self-attention layer over the neighbourhood, reaching 70.4% mIoU on S3DIS Area 5 — the first crossing of the 70% line, 3.3 points over the previous best. The successor Point Transformer V3 made the opposite kind of move, arguing that at scale, simplicity and throughput beat mechanism: it replaced exact kNN neighbourhoods with a serialised neighbour mapping, which let it widen the receptive field from 16 to 1024 points while running 3× faster and using 10× less memory than its predecessor. Notice what survived every one of those redesigns — sample centroids, gather neighbourhoods, pool. That is PointNet++'s set abstraction, with the interior swapped.

Branch 3 — self-supervision, and PointNet as a component. Point-MAE (Pang et al., 2022) is the clearest example of the idea being absorbed rather than replaced. It masks most of a cloud and asks a transformer to reconstruct the missing bits, in the style of masked autoencoders for images. The pipeline: 1024 input points, FPS to n = 64 patch centres, kNN with k = 32 to form patches, mask 60–80% of the patches, encode only the visible ones with 12 transformer blocks of width 384 and 6 heads, decode with 4 blocks, and reconstruct masked patch coordinates under an l₂ Chamfer distance.

The detail that matters for this lesson is the patch embedding. The authors considered flattening each 32×3 patch and applying a linear projection, as ViT does to image patches — then rejected it in one sentence: "we argue that linear embedding fails to follow the principle of permutation invariance." A patch is still a set. So they embed each patch with "a lightweight PointNet, which mainly consists of MLPs and max pooling layers." Six years on, PointNet is the tokeniser.

The results: pretrained on ShapeNet (about 51,300 models, 55 categories, 300 epochs), Point-MAE reaches 85.18% on the hardest ScanObjectNN variant — real scans with cluttered backgrounds — beating Point-BERT by 2.11 points, and 94.04% on ModelNet40 with 8192 points (93.8% at 1024, against 91.4% training from scratch). Few-shot accuracy advanced by 1.5–2.3 points. The same masking-and-reconstruction recipe that worked for text and images works for point sets, once you have a permutation-invariant tokeniser.

Branch 4 — alignment to language and images. ULIP (Xue et al., 2022) attacks the other bottleneck: 3D datasets are tiny and their label sets are fixed. It builds (image, text, point cloud) triplets automatically, freezes a pretrained vision-language model, and trains a 3D encoder to land in the space that model already occupies. Crucially it is backbone agnostic — PointNet++, Point-BERT, PointMLP all slot in — so the 3D architecture question and the supervision question separate cleanly. The consequence is open-vocabulary 3D: you can ask about a category by typing it, exactly as CLIP did for images.

Chapter 8: Point-Cloud Embeddings in Practice

Both papers are written as classification and segmentation papers. But the object they actually produce — a fixed-length vector that describes a set of points and does not care how that set was ordered — is useful for a family of problems the papers barely mention. This chapter is about those, because in a real robotics or mapping stack you will use the embedding far more often than the classifier head.

The descriptor is a database key

Strip the classification head off and what you have is a function from a point set to a vector in R¹⁰²⁴. Precompute it once for every object in a library, store the vectors, and retrieval becomes a dot product. The paper does this implicitly when it embeds the ModelNet40 test split's 1024-dim global signatures with t-SNE and observes that "similar shapes are clustered together according to their semantic categories" — the geometry of the embedding space carries category structure that nobody supervised directly.

Work a retrieval by hand, on four-dimensional descriptors so the arithmetic is visible. A query cloud embeds to

and three library shapes embed to unit vectors a = (0.5, 0.5, 0.5, 0.5), b = (0.8, 0.6, 0, 0), c = (0, 0, 0.6, 0.8). Cosine similarity is the dot product of unit vectors:

Rank: b, then a, then c. Nothing exotic — but notice that this is the entire inference cost of shape retrieval over a library of any size, once the library is embedded. Nearest-neighbour search over a million 1024-dim vectors is a solved engineering problem; running a network a million times is not.

A gotcha you will hit within a day: max-pooled descriptors are not length-normalised

Here is a property of PointNet's descriptor that follows directly from Chapter 2, and that no one warns you about.

Every component of u is a maximum over the points. Adding a point can raise a maximum and can never lower one. So u is componentwise monotone non-decreasing in the point set:

So a 100,000-point sweep of an object produces a systematically longer descriptor than a 1,000-point sweep of the same object — not a different direction, a different length. If you rank by raw dot product, close-range dense objects will beat far-range sparse ones for reasons that have nothing to do with shape. Normalise to unit length before you index, always. And if you cannot, at least fix the input point count, which is exactly why every experiment in both papers uses a constant N.

Registration: aligning two scans

Registration is the problem of finding the rigid transform that maps one scan onto another. Classical pipelines do it in four stages, and PointNet-style networks slot into stage two.

Stage two is where Chapter 4's centroid subtraction earns its keep: because the group is expressed relative to its centre, the descriptor is exactly translation invariant, which is half of the rigid-motion invariance registration needs. The other half — rotation — is not free, and you have three honest options: augment heavily with random rotations, compute a local reference frame from the neighbourhood (typically from the covariance's eigenvectors) and rotate the group into it before embedding, or use rotation-invariant inputs such as point-pair distances and angles rather than raw relative coordinates. The second is the most common in practice and is worth knowing about, because it is exactly the T-Net idea moved from global to local, where it is far better posed.

Place recognition and loop closure

A mapping robot revisits a corridor it drove through twenty minutes ago. It has to notice, because that recognition is what lets the pose graph close the loop and correct accumulated drift. Doing this by matching raw point clouds is hopeless; doing it by comparing one vector per location is easy.

The recipe is the retrieval pipeline with the semantics changed: embed each submap into a descriptor, store it with its pose, and on every new submap query the database for nearest neighbours. A hit above threshold becomes a loop-closure candidate, which then gets geometrically verified by registration — the pipeline above — before it is trusted enough to add a constraint to the graph.

Two design pressures shape what descriptor you want here, and both push away from plain max pooling. First, viewpoint: you may traverse the corridor in the opposite direction, so the descriptor must survive a yaw change of 180 degrees. Second, partial overlap: the two submaps share 60% of their content and each has 40% the other never saw. Max pooling is unhelpfully sensitive to the second, because a single strong response in the non-overlapping 40% can set a component of the descriptor. This is why the place-recognition literature moved to aggregation schemes with soft assignment — VLAD-style pooling over local descriptors, where every local feature contributes to a residual against a learned codebook rather than winner-takes-all. The set-abstraction front end usually stays; it is the pooling that gets replaced.

Sizing a retrieval system, end to end

Suppose you are building shape search over a library of one million CAD parts. Here is the whole system, priced.

Index build. One forward pass per part. PointNet classifies roughly 1,000 objects a second on a 1080-class GPU, so a million parts is about 17 minutes of compute — a coffee, not a project. Store the descriptor at 1024 float32s, or 4 KB per part: 4 GB for the library. Quantise to int8 and it is 1 GB, which fits in memory on a laptop.

Query. One forward pass (about 1 ms) plus a nearest-neighbour search. Exact search over a million 1024-dim vectors is a single matrix multiply of 1M × 1024 by 1024 × 1, roughly 2 GFLOPs — a couple of milliseconds on a GPU, tens of milliseconds on a CPU. An approximate index brings that under a millisecond if you need it.

The dimension question. 1024 is the pooled width, chosen for classification accuracy, not for retrieval. For search you usually want it smaller: memory, index build time and search time all scale linearly with it. A learned projection down to 128 or 256 — trained with a metric-learning objective on top of the frozen encoder — typically loses very little retrieval quality and cuts the index by 4–8×. This is the same move CLIP-style systems make, and the same reason.

The thing that will actually cost you. Not compute. Pose. Every caveat in the previous section is a source of false negatives, and the one that bites first is rotation: if the library was embedded from canonically-oriented CAD files and your queries are scans in arbitrary poses, recall collapses and no amount of index tuning helps. The two fixes are to embed each library item at several orientations (multiplying your index by the number of orientations, which is cheap and dumb and works), or to use a rotation-invariant descriptor and accept the information it discards.

Choosing keypoints, and why FPS is a reasonable default

The registration pipeline above begins with "pick repeatable locations". Repeatability is the whole game: a keypoint detector that fires at different physical locations on two scans of the same scene produces descriptors that cannot be matched, no matter how good the descriptor is.

Classical detectors look for curvature extrema or local maxima of some saliency function — the 3D analogue of corner detection. They are genuinely repeatable on clean data and fragile under noise, because curvature is a second derivative and second derivatives amplify noise.

FPS is a different kind of answer: it is not looking for salient points at all, it is looking for a covering. Its repeatability argument is not "the same points get chosen" — they will not be, since FPS starts from an arbitrary seed — but "the chosen points are spread evenly enough that every part of the surface is near some centroid in both scans". Combined with a descriptor computed over a fixed radius, that is often enough: the two centroids need not coincide, only to be close enough that their balls overlap substantially. This is the same insight that made ball query the right grouping rule, applied one level up.

It also makes the randomness measurement from Chapter 4 relevant here in a way the paper did not intend. A 0.17% accuracy standard deviation across FPS seeds says the downstream features barely notice which points were chosen — which is precisely the stability property a keypoint scheme needs.

Latency arithmetic, so the numbers are not a surprise

Take a 10 Hz lidar producing 100,000 returns per sweep — a million points a second, and 100 ms of budget per frame if you want to keep up.

PointNet's own benchmark: more than one million points per second for classification on a 1080X, which is roughly 1,000 objects a second or 2 rooms a second for segmentation. So a single global PointNet pass over a full sweep is genuinely real-time, and that is the reason it survived in production for pre-processing and proposal scoring long after better classifiers existed.

PointNet++ is a different story. From the timing table: SSG runs 82.4 ms for a batch of 8 clouds of 1024 points, so about 10 ms per 1024-point cloud, and MSG about 20 ms. To cover 100,000 points at 1024 per tile you would need roughly 100 tiles — a second of compute per frame, ten times over budget. The way out in practice is exactly what the S3DIS setup does: process spatial blocks, batch them, and accept that the hierarchy costs an order of magnitude more than the flat model. Chapter 5's table already told you where that time goes — SSG is the smallest model in the comparison at 8.7 MB and one of the slowest, because FPS and ball query are gather-bound, not FLOP-bound.

Chapter 9: Connections and Cheat Sheet

Two papers, six months apart, by the same four people, in which the second one's job is to fix the first one's stated flaw. Here is the whole arc on one page.

Cheat sheet

A debugging table, because these models fail in recognisable ways

Every entry below is a symptom that has a specific cause in something derived in this lesson, which is unusual and worth exploiting.

Reproducing this today, if you want to

Both papers are old enough that the exact protocols are worth restating, because half of the reproduction difficulty is data preparation rather than modelling.

ModelNet40 classification. 9,843 train / 2,468 test. Sample 1024 points on mesh faces proportional to face area — area-weighted, not per-vertex, or flat surfaces will be under-represented and detailed regions over-represented. Normalise to zero mean and unit sphere. Augment with random rotation about the up-axis and per-point Gaussian jitter at σ = 0.02. PointNet++ adds random scaling and translation, uses Adam at learning rate 0.001, and reports about 20 hours to convergence on a GTX 1080 or Titan X.

ShapeNet part segmentation. 16,881 shapes, 16 categories, 50 parts, official split. Category label is assumed known at test time — you are predicting parts within a known category, not category and parts jointly. IoU convention: an empty ground-truth-and-prediction union counts as 1.

S3DIS semantic segmentation. 271 rooms across 6 areas, 13 classes. Split by room, then into 1 m × 1 m blocks. Each point is 9-dimensional: XYZ, RGB, and normalised position within the room. Sample 4096 points per block on the fly during training; evaluate on all points; use the standard k-fold protocol across areas rather than a random split, or you will leak.

The two things most likely to cost you a point or two if you get them wrong: area-weighted sampling, and remembering that "normalise into a unit sphere" is per-shape, not dataset-wide.

The three things worth arguing about

Max pooling is lossy in a way nobody fully fixed. Theorem 2 is a robustness result and a capacity ceiling at the same time. Later work replaced the pool with attention, with VLAD-style soft assignment, with sorted-k pooling — each recovering some of the discarded points at some cost. There is still no aggregator that is simultaneously as cheap as max, as robust as max, and as expressive as attention.

Neighbourhood search, not arithmetic, is the bottleneck. An 8.7 MB model that takes 82 ms is telling you plainly where the time is. Every major speedup since has come from changing how neighbourhoods are found — hashing, voxel binning, space-filling-curve serialisation — and not from changing the network.

Rotation is still not solved cleanly. The T-Net is a learned approximation with no guarantee, trained under up-axis rotation only. Local reference frames help and add fragility of their own; rotation-invariant input features help and discard information. This remains a live design decision in every point-cloud system, and if you are debugging a model that works on your dataset and fails on someone else's, check the pose distribution first.

References worth reading next

1. Qi, C. R., Su, H., Mo, K., Guibas, L. J. "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation," CVPR 2017 — arXiv:1612.00593. Chapters 0–3 and 6–7 are built on this.
2. Qi, C. R., Yi, L., Su, H., Guibas, L. J. "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space," NeurIPS 2017 — arXiv:1706.02413. Chapters 4–6. Read the supplementary for the architecture strings.
3. Pang, Y., Wang, W., Tay, F. E. H., Liu, W., Tian, Y., Yuan, L. "Masked Autoencoders for Point Cloud Self-supervised Learning" (Point-MAE), ECCV 2022 — arXiv:2203.06604. Where PointNet became the tokeniser.
4. Zhao, H., Jiang, L., Jia, J., Torr, P., Koltun, V. "Point Transformer," ICCV 2021 — arXiv:2012.09164. Attention in place of the group-wise max.
5. Wu, X. et al. "Point Transformer V3: Simpler, Faster, Stronger," CVPR 2024 — arXiv:2312.10035. Neighbourhood search is the bottleneck, argued at scale.
6. Choy, C., Gwak, J., Savarese, S. "4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks," CVPR 2019 — arXiv:1904.08755. The sparse-convolution answer to the voxel critique.
7. Xue, L. et al. "ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding," CVPR 2023 — arXiv:2212.05171. Backbone-agnostic alignment to a frozen vision-language space.
8. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K. "Spatial Transformer Networks," NeurIPS 2015. The module the T-Net simplifies.
9. Vinyals, O., Bengio, S., Kudlur, M. "Order Matters: Sequence to Sequence for Sets," ICLR 2016. The prior work PointNet cites for why the RNN route does not scale.