Toronto AI Lab

DIB-R++: Learning to Predict Lighting and
Material with a Hybrid Differentiable Renderer

Wenzheng Chen^1,2,3

Joey Litalien⁴

Jun Gao^1,2,3

Zian Wang^1,2,3

Clement Fuji Tsang ¹

Sameh Khamis¹

Or Litany¹

Sanja Fidler^1,2,3

¹NVIDIA

²University of Toronto

³Vector Institute

⁴McGill University

NeurIPS 2021

We propose a new hybrid renderer which combines rasterization and ray tracing together. Given a 3D mesh M, we employ (a) a rasterization-based renderer to obtain diffuse albedo, surface normals and mask maps. In the shading pass (b), we then use these buffers to compute the incident radiance by sampling or by representing lighting and the specular BRDF using a spherical Gaussian basis. Depending on the representation used in (c), we can render with advanced lighting and material effect (d).

We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning-based approaches for inverse graphics adopt rasterization-based renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIB-R++, a hybrid differentiable renderer which supports these photorealistic effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths—speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIB-R++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry, reflectance and lighting prediction from a single image without requiring any ground-truth. We experimentally demonstrate that our approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches and showcase several artistic applications including material editing and relighting.

Paper

Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang,
Clement Fuji Tsang, Sameh Khamis, Or Litany, Sanja Fidler

DIB-R++: Learning to Predict Lighting and
Material with a Hybrid Differentiable Renderer

NeurIPS 2021

[Arxiv Paper]

[Supplementary Material]

[Bibtex Citation]

[Code Coming Soon!]

Results

To align all the videos, please refresh the website.

Jointly Predicting Geometry, Texture, Light and Material
from a Single-view Image without any 3D Supervision

GT
DIB-R++
DIB-R
	Pred. Texture	Render w. Light	Render. w.o. Light	Pred. Spherical Gaussian Light

3D Reconstruction on Glossy Surfaces with Spherical Gaussian Shading: Given an input image (1st row, 1st column), we predict 3D geometry, texture(1st column), spherical gaussian light(4th column) and material and render them with and without lighting effect(2nd & 3rd columns). We compare our model with GT(1st row) and our previous work, DIB-R(3rd row).
DIB-R++ adopts SG shading, which can correctly disentangle reflectance from texture, as evidenced by the absence of white highlights in the predict textures. DIBR cannot capture these bright regions due to a diffuse-only shading model and predict textures which "bake in" specular lighting effect.

GT
DIB-R++
DIB-R
	Pred. Texture	Rend. w. Light	Rend. w.o. Light	Pred. Environment Map Light

3D Reconstruction on Metallic Surfaces with Monte Carlo Shading: Given an input image (1st row, 1st column), we predict 3D geometry, texture(1st column), environment map light(4th column) and render them with and without lighting effect(2nd & 3rd columns). We compare our model with GT(1st row) and our previous work, DIB-R(3rd row).
DIB-R++ correctly recovers the high frequency details of the sky light maps, while DIBR fails to separate the lighting from the albedo, resulting in the incorrect texture maps incorporating the ground dominant color.

Real Imagery Reconstruction

GT
DIB-R++
DIB-R
	Pred. Texture	Rend. w. Light	Rend. w.o. Light	Pred. Spherical Gaussian Light

3D Reconstruction on GAN-generated Dataset with Spherical Gaussian Shading: We further verify our model on realistic image dataset generated by StyleGAN. DIBR++ can recover a meaningful decomposition of texture and light, as shown by cleaner texture maps and directional highlights in the re-rendered videos. On the other side, DIB-R fails to capture specular lighting effect due to its naive shading model.

3D Reconstruction on LSUN Dataset with Spherical Gaussian Shading: Lastly, we verify our model on real images. DIB-R++, trained on StyleGAN dataset, can generalize well to real images since the distributions are similar. Moreover, it also predicts correct high specular lighting directions and usable, clean textures.