Native and Compact Structured Latents for 3D Generation

Chapter 1: The Key Insight

TRELLIS.2's insight is deceptively simple: don't use fields at all. Instead, represent 3D assets directly as sparse voxels that store both geometry and materials — then compress those voxels into a compact latent space.

Chapter 2: O-Voxel Representation

O-Voxel ("omni-voxel") is the foundation of everything. It's a sparse voxel structure on an N×N×N grid where each active voxel stores a feature tuple:

where p_i ∈ {0, 1, ..., N-1}³ is the voxel coordinate, and only voxels intersecting the mesh surface are active. Let's unpack each component.

Shape: The Flexible Dual Grid

For geometry, each active voxel stores three things:

The idea comes from Dual Contouring (DC), but with a crucial difference: DC requires a signed distance field to detect sign changes across edges. O-Voxel skips the field entirely. It directly tests whether each voxel edge intersects the mesh surface. If an edge crosses a triangle, the adjacent dual face is activated, and Hermite data (intersection point + normal) positions the dual vertex.

The dual vertex position is computed by minimizing a Quadratic Error Function (QEF):

Three terms, each serving a purpose:

• Plane distance d²_Π: The original DC objective — place v close to the tangent planes defined by intersection points and normals. This captures smooth surfaces.
• Boundary distance d²_L: New in O-Voxel — penalizes distance from v to boundary edges of the mesh. This handles open surfaces where there's no sign change to guide placement.
• Regularization d²_q̂: Keeps v near the average of intersection points. Stabilizes the QEF against degenerate configurations.

Material: Volumetric Surface Attributes

Each active voxel also stores PBR material attributes:

Materials are sampled by projecting each voxel center onto intersected triangles and reading from the texture map using UV coordinates. The inverse is equally simple: to reconstruct a material at any surface point, trilinear interpolation of neighboring voxel attributes gives the answer. No baking, no optimization.

Chapter 3: Sparse Compression VAE

O-Voxel gives us a native 3D representation. But a 1024³ grid with ~600K active voxels is too large for a generative model to work with directly. We need to compress it into a compact latent space — that's the job of the Sparse Compression VAE (SC-VAE).

Architecture: Fully Sparse-Convolutional

Unlike TRELLIS (which used transformers), SC-VAE is a fully sparse-convolutional U-shaped network. It processes only active voxels, skipping the ~99.4% of the grid that's empty. The encoder downsamples hierarchically through residual blocks; the decoder mirrors this for reconstruction.

The Residual Autoencoding Trick

The key innovation enabling 16× compression is Sparse Residual Autoencoding, adapted from DC-AE (originally designed for 2D images). The problem: at 16× downsampling, you're squeezing 16³ = 4096 spatial voxels into a single latent. Standard pooling destroys too much information.

The solution: before each 2× downsample, rearrange the 8 child voxels of each parent into the channel dimension:

This is a non-parametric shortcut: spatial information is preserved in channels, then averaged across groups. The learnable conv layers only need to refine a residual on top of this estimate. During upsampling, the symmetric operation distributes channels back to spatial positions:

Optimized ConvNeXt-style Blocks

Sparse convolutions are computationally expensive at high sparsity. SC-VAE replaces the standard two-conv residual block with a ConvNeXt-inspired design: one sparse conv layer followed by a wide point-wise MLP (analogous to a Transformer FFN). This doesn't change runtime but improves reconstruction quality: MD drops 16%, PSNR gains 0.6dB.

Early-Pruning Upsampler

During decoding, not all child voxels of a parent should be active. Before each upsample step, the network predicts a binary mask ρ̂ ∈ {0,1}⁸ for each parent, specifying which children to activate. Inactive children are pruned, saving both computation and memory.

Two-Stage Training

SC-VAE is trained in two stages:

1. Stage 1 (low-resolution): Direct O-Voxel reconstruction losses. MSE on dual vertex positions, BCE on edge flags and pruning masks, L1 on materials, plus KL divergence.
2. Stage 2 (high-resolution): Add rendering-based perceptual supervision. Render mask, depth, normal, and material maps from random camera positions. Supervise with L1 + SSIM + LPIPS. Cameras are placed with shallow near planes to slice through the surface, forcing the model to capture internal structures too.

Chapter 4: Latent Space (Showcase)

Let's put the compression in perspective. How does TRELLIS.2's latent space compare to every other 3D generation method?

The Token Efficiency Frontier

Every 3D latent representation faces a tradeoff: more tokens give better reconstruction quality, but make generation slower and harder. The ideal method sits in the top-left corner: few tokens, high quality.

At 512³, TRELLIS.2 uses only 2.2K tokens — 25× fewer than SparseFlex. At 1024³, it uses 9.6K tokens — 23× fewer than SparseFlex and nearly 2× fewer than Direct3D-S2. Yet its reconstruction quality is substantially better across every metric.

What Happens at Different Compression Ratios

The ablation on Sketchfab assets at 256³ tells the full story:

At 16×, residual AE keeps quality high. At 32×, even with residual AE, quality degrades. Without it, 32× is catastrophic. The 16× sweet spot delivers production-quality reconstruction with a manageable token count.

Chapter 5: Flow Matching for 3D

With the latent space defined, we need a generative model to sample from it. TRELLIS.2 uses flow matching — a cleaner alternative to diffusion that learns straight-line transport from noise to data.

Three-Stage Generation Pipeline

Generation unfolds in three sequential stages, each with its own DiT (Diffusion Transformer):

DiT Architecture Details

Each DiT module uses:

• AdaLN-single modulation: Adaptive Layer Normalization with a single timestep+condition embedding modulating all layers. More parameter-efficient than per-layer conditioning.
• Rotary Position Embedding (RoPE): 3D positional encoding that generalizes across resolutions. Crucial for the cascaded inference that scales from 512³ to 1024³ to 1536³.
• Image conditioning: DINOv3-L features extracted from the input image, injected via cross-attention.
• Classifier-free guidance: 10% condition drop rate during training. At inference, guidance scale controls fidelity-diversity tradeoff.

Progressive Resolution Training

The models are trained progressively:

1. Stage 1 (sparse structure) trains with 512×512 conditioning images to learn coarse occupancy priors.
2. Stages 2-3 start at 512³ output (32³ latent) and scale to 1024³ output (64³ latent), with conditioning images increasing to 1024×1024.

This progressive strategy transfers learned priors across resolutions, enabling efficient training of the large sparse DiTs while maintaining fidelity.

Chapter 6: Training

Training a 4B-parameter 3D generation system requires careful data curation, infrastructure, and a multi-stage training protocol.

Training Data

Training Infrastructure

• SC-VAE training: 16 H100 GPUs, batch size 128. Uses optimized Triton implementation for submanifold convolutions.
• DiT training: 32 H100 GPUs, batch size 256. AdamW optimizer (lr = 1e-4, weight decay 0.01). Classifier-free guidance with 10% condition drop rate.

SC-VAE Training Protocol

Two-stage training, as described in Chapter 3:

• Stage 1: Low-resolution data, direct reconstruction losses (MSE on vertices, BCE on flags, L1 on materials, KL divergence). Fast convergence to stabilize the latent space.
• Stage 2: High-resolution data, add rendering-based perceptual losses (L1 + SSIM + LPIPS on rendered normals, depth, masks, and PBR attribute maps). Random camera placement with shallow near planes to capture internal structures.

Chapter 7: Results

TRELLIS.2 is evaluated on reconstruction fidelity, generation quality, and inference speed. The results are consistently dominant.

Reconstruction Quality

On the Toys4K benchmark (shape reconstruction):

At 1024³, TRELLIS.2 achieves 8× lower mesh distance than SparseFlex (which uses 23× more tokens). Even at 512³ with only 2.2K tokens, it outperforms methods using 10-100× more tokens.

Generation Quality

Image-to-3D generation compared against TRELLIS, Hi3DGen, Direct3D-S2, Step1X-3D, and Hunyuan3D 2.1:

• CLIP alignment: Highest across all methods (0.894 CLIP, 0.477 CLIP-N)
• Geometric similarity: Highest on ULIP-2 (0.758) and Uni3D (0.436)
• User study: Preferred by ~40 participants across 100 AI-generated prompts, outperforming all baselines in visual realism, geometric detail, and prompt alignment

Speed

All timings on a single NVIDIA H100 GPU. These are significantly faster than existing large 3D generation models, despite producing higher quality outputs with PBR materials.

Test-Time Scaling

The compact latent space enables two forms of test-time scaling:

• Resolution scaling: Generate at 1024³, downsample the O-Voxel to a higher-res sparse structure (e.g., 96³), re-apply Stage 2 to get 1536³ output. Finer details without retraining.
• Compute scaling: Generate at 512³, downsample to 64³ structure, re-apply Stage 2 at 1024³. The cascaded inference corrects local errors and yields cleaner geometry. Adds ~3s for meaningfully finer details.

Chapter 8: Applications

TRELLIS.2's native 3D output with PBR materials opens several practical application paths that prior methods cannot support.

Image-to-3D Asset Generation

The primary use case: given a single image, generate a complete 3D asset ready for rendering. The output includes a mesh with proper topology (open surfaces, enclosed interiors supported) plus full PBR materials (base color, metallic, roughness, opacity). The asset is immediately usable in game engines, film pipelines, and AR applications without post-processing.

Shape-Conditioned Texture Generation

Stage 3 (material generation) can run independently. Given any input mesh and a reference image, it produces PBR materials aligned to the geometry. This beats alternatives:

• Multi-view PBR methods (Hunyuan3D-Paint): Suffer from view inconsistencies — ghosting and blurred textures where views disagree.
• UV-based methods (TEXGen): Suffer from ambiguous UV charts and seam artifacts.
• TRELLIS.2 Stage 3: Reasons natively in 3D, producing sharp textures with consistent shape-material alignment. Can also texture internal surfaces — crucial for complex assets like vehicles with visible interiors.

Translucent and Complex Materials

The opacity channel α in O-Voxel enables translucent surface modeling — glass, ice, thin fabric. No prior 3D generation method handles this. The generated assets can be rendered with proper alpha blending in any standard pipeline.

Cascaded Super-Resolution

Because the latent space is resolution-agnostic (thanks to RoPE), a model trained at 1024³ can be applied to generate at 1536³ through cascaded inference. This requires no retraining — just re-running Stage 2 on a downsampled higher-res structure. The result: finer geometric details and sharper materials at the cost of additional inference time (~60s total for 1536³).

Chapter 9: Connections

TRELLIS.2 sits at the intersection of 3D representation learning, latent compression, and large-scale generative modeling. Let's map where it fits.

Relation to TRELLIS (v1)

TRELLIS introduced the Structured Latent (SLAT) representation for joint geometry-appearance modeling. But SLAT was built from multiview 2D features with rendering-based supervision, limiting its ability to capture complex structures and materials. TRELLIS.2 replaces this with native 3D data (O-Voxel), achieving 16× vs 4× compression, better quality, and proper PBR support. The generative pipeline (sparse DiT + flow matching) is evolutionary; the representation is revolutionary.

Relation to NeRF / 3D Gaussian Splatting

NeRFs and 3DGS represent appearance as view-dependent radiance — great for novel view synthesis, but the appearance is baked to specific lighting. O-Voxel stores intrinsic material properties (base color, metallic, roughness) that can be re-lit in any environment. The tradeoff: O-Voxel requires PBR-annotated training data, while NeRF/3DGS can learn from posed images alone.

Relation to DC-AE (Deep Compression Autoencoder)

DC-AE demonstrated that residual autoencoding enables extreme spatial compression in 2D image VAEs. TRELLIS.2 adapts this principle to sparse 3D voxels — the space-to-channel rearrangement and non-parametric shortcuts are directly inherited. The key extension: handling sparsity (most voxels are empty, children may not exist).

Relation to Flow Matching / Rectified Flow

The generation backbone uses flow matching rather than DDPM-style diffusion. Flow matching learns straight-line transport from noise to data, enabling fewer sampling steps. The DiT architecture follows the standard recipe (AdaLN, RoPE), with the novelty being its application to structured sparse 3D latents rather than 2D image tokens.

Cheat Sheet