Real-Time Neural Appearance Models

Close-up renderings of a teapot with our neural material shaders. The AI material model accurately learns the details of the ceramic, the imperfect clear-coat glaze, fingerprints, smudges, and dust. Our neural model is capable of preserving these complex material properties while being faster to evaluate than traditional multilayered materials.

We show rendered images of five materials that we approximate with neural models for representing the BRDF and importance sampling. The illustrations show stacks of scattering layers that each material consists of.

We use our neural BRDFs in a renderer as follows: for each ray that hits a surface with a neural BRDF, we perform standard $(u,\; v)$ and MIP level $l$ computation, and query the latent texture of the neural material. Then we input the latent code $z(x)$ into one or two neural decoders, depending on the needs of the rendering algorithm. The BRDF decoder (top box) first extracts two shading frames from $z(x)$, transforms directions ${𝝎}_{\mathrm{i}}$ and ${𝝎}_{\mathrm{o}}$ into each of them, and passes the transformed directions and $z(x)$ to an MLP that predicts the BRDF value (and optionally the directional albedo). The importance sampler (bottom box) extracts parameters of an analytical, two-lobe distribution, which is then sampled for an outgoing direction ${𝝎}_{\mathrm{o}}$, and/or evaluated for PDF $p(x,{𝝎}_{\mathrm{i}},{𝝎}_{\mathrm{o}})$.

A scene with four materials that we approximate using the proposed neural BRDF. The first three columns correspond to different configurations of the BRDF decoder, from fastest to the most accurate. FLIP error images are in the corners, timings quantify the cost of rendering a 1 SPP image of the scene; we show images with 8192 samples to suppress path tracing noise.

Highly detailed materials will alias significantly when rendered without supersampling (left columns, unfiltered). Supersampling averages high frequency glints and produces a filtered material, but at impractical sample cost for real-time (right columns, ground truth at 512 SPP). Our neural material can render filtered materials without aliasing at any distance, without supersampling (middle columns, ours).

The importance sampler (top row) reduces noise levels compared to a simpler variant only supporting isotropic specular reflections (bottom row), in the spirit of Sztrajman et al. [2021] and Fan et al. [2022]. Left: Fine details of a normal map are captured using a non-centered microfacet NDF. Right: At coarser MIP levels, the filtered distribution is strongly anisotropic. The zoomed views are rendered using 4SPP. False-color images show the pixel-wise standard deviation and the mean across the entire inset.