Neural Harmonic Textures for High-Quality Primitive Based Neural Reconstruction

Abstract

Primitive-based methods such as 3D Gaussian Splatting have recently become the state-of-the-art for novel-view synthesis and related reconstruction tasks. Compared to neural fields, these representations are more flexible, adaptive, and scale better to large scenes. However, the limited expressivity of individual primitives makes modeling high-frequency detail challenging.

We introduce Neural Harmonic Textures, a neural representation approach that anchors latent feature vectors on a virtual scaffold surrounding each primitive. These features are interpolated within the primitive at ray intersection points. Inspired by Fourier analysis, we apply periodic activations to the interpolated features, turning alpha blending into a weighted sum of harmonic components. The resulting signal is then decoded in a single deferred pass using a small neural network, significantly reducing computational cost.

Neural Harmonic Textures yield state-of-the-art results in real-time novel view synthesis while bridging the gap between primitive- and neural-field-based reconstruction. Our method integrates seamlessly into existing primitive-based pipelines such as 3DGUT, Triangle Splatting, and 2DGS. We further demonstrate its generality with applications to 2D image fitting and semantic reconstruction.

Method Overview

We present our approach using 3D Gaussian primitives in the context of novel view synthesis, following 3DGUT. Our method has three core components: particle-bound feature embedding (latent features on a virtual scaffold per primitive), harmonic texturing (periodic encoding of interpolated features before alpha blending), and neural deferred decoding (a single MLP evaluation per pixel to reconstruct color from the accumulated harmonics).

Neural Harmonic Textures applied to novel-view synthesis. We attach feature vectors \(\mathbf{f}_i\) to the vertices of tetrahedra inscribing the Gaussian primitives. We evaluate the point along the ray where the projected Gaussian has maximum response, barycentrically interpolate vertex features there, encode them with sine and cosine, alpha-blend along the ray, and decode the sum of harmonics with a shallow MLP in a single image-space pass.

Particle-bound feature embedding

Each primitive is bounded by an ellipsoid in world space, which becomes a sphere in whitened canonical space. We define a virtual bounding tetrahedron in this canonical space and attach one \(N_f\)-dimensional feature vector \(\mathbf{f}^j \in \mathbb{R}^{N_f}\) to each of its four vertices (\(j \in \{0,1,2,3\}\)). For each ray–primitive intersection, we take the point \(\mathbf{p}^*\) where the projected Gaussian has maximum response along the ray (following 3DGUT). The feature at that location is given by barycentric interpolation of the vertex features:

\[ \mathbf{f} = \sum_{j=0}^{3} w_j \, \mathbf{f}^j, \qquad \text{with } \sum_{j=0}^{3} w_j = 1 \text{ and } w_j \geq 0, \] where \(w_j\) are the barycentric coordinates of \(\mathbf{p}^*\) in the tetrahedron.

Harmonic texturing

Rather than decoding each sample with an MLP before blending (which would require many evaluations per ray), we encode the interpolated features with periodic functions and blend the encodings along the ray. Concretely, we map \(\mathbf{f}_i\) to \(\bigl[\sin(\mathbf{f}_i); \cos(\mathbf{f}_i)\bigr]\), turning the signal into a sum of harmonic components—we call this harmonic texturing. The primitive opacity \(\alpha_i\) and transmittance \(T_i\) act as the amplitude of each harmonic. Large differences between vertex features within a primitive yield rapidly varying (high-frequency) textures; the interpolation thus acts as a frequency modulator.

(a) Latent vectors on virtual tetrahedron

(b) Interpolation at \(\mathbf{p}^*\)

(c) Harmonic decomposition

Neural deferred decoding

After alpha-compositing these harmonic textures along the ray, we decode the final pixel color in a single image-space pass—neural deferred decoding. The rendering equation is

\[ \mathbf{c} = \mathrm{MLP}_\theta \left( \; \sum_{i \in \mathcal{G}} \alpha_i \, T_i \, \begin{bmatrix} \sin(\mathbf{f}_i) \\ \cos(\mathbf{f}_i) \end{bmatrix}, \;\; k \cdot \mathrm{SH}_2(\mathbf{d}) \; \right), \] where \(\mathcal{G}\) is the set of primitives intersected by the ray, \(\mathbf{f}_i\) is the interpolated feature at \(\mathbf{p}_i^*\), and \(\mathrm{SH}_2(\mathbf{d})\) encodes the ray direction for view-dependent effects.