1. [Publications](/index.php/publications)
2. Extracting Triangular 3D Models, Materials, and Lighting From Images
 
 # Extracting Triangular 3D Models, Materials, and Lighting From Images

  ![](/sites/default/files/styles/wide/public/publications/system_0.JPG?itok=93wIaD9b)

 **Abstract**

We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, which typically produce entangled 3D representations encoded in neural networks, we output triangle meshes with spatially-varying materials and environment lighting that can be deployed in any traditional graphics engine unmodified. We leverage recent work in differentiable rendering, coordinate-based networks to compactly represent volumetric texturing, alongside differentiable marching tetrahedrons to enable gradient-based optimization directly on the surface mesh. Finally, we introduce a differentiable formulation of the split sum approximation of environment lighting to efficiently recover all-frequency lighting. Experiments show our extracted models used in advanced scene editing, material decomposition, and high quality view interpolation, all running at interactive rates in triangle-based renderers (rasterizers and path tracers).



 ## Authors



[Jacob Munkberg](/index.php/person/jacob-munkberg)

[Jon Hasselgren](/index.php/person/jon-hasselgren)

Tianchang Shen (NVIDIA)

Jun Gao (NVIDIA)

Wenzheng Chen (NVIDIA)

Alex Evans (NVIDIA)

[Thomas Müller](/index.php/person/thomas-muller)

Sanja Fidler (NVIDIA)

 

 

 ## Publication Date



Thursday, November 25, 2021

 

 ## Published in



[CVPR 2022 (Oral)](https://openaccess.thecvf.com/content/CVPR2022/papers/Munkberg_Extracting_Triangular_3D_Models_Materials_and_Lighting_From_Images_CVPR_2022_paper.pdf)

 

 ## Research Area



[Artificial Intelligence and Machine Learning ](/index.php/research-area/machine-learning-artificial-intelligence)

[Computer Graphics](/index.php/research-area/computer-graphics)

[Computer Vision](/index.php/research-area/computer-vision)

[Real-Time Rendering](/index.php/research-area/real-time-rendering)

 

 

 ## External Links



[Project page](https://nvlabs.github.io/nvdiffrec/)

[Arxiv preprint](https://arxiv.org/abs/2111.12503)

[Video](https://nvlabs.github.io/nvdiffrec/assets/video.mp4)