Neuralangelo: High-Fidelity Neural Surface Reconstruction

¹NVIDIA Corporation ²Johns Hopkins University

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

TIME's Best Inventions of 2023

Paper (arXiv) Paper Supplementary Code Colab notebook

Neuralangelo is a framework for high-fidelity 3D surface reconstruction from RGB video captures. Using ubiquitous mobile devices, we enable users to create digital twins of both object-centric and large-scale real-world scenes with highly detailed 3D geometry.

For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing.

(best viewed with Google Chrome on a desktop/laptop)

Abstract

Neural surface reconstruction has shown to be powerful for recovering dense 3D surfaces via image-based neural rendering. However, current methods struggle to recover detailed structures of real-world scenes. To address the issue, we present Neuralangelo, which combines the representation power of multi-resolution 3D hash grids with neural surface rendering. Our approach is enabled by two key ingredients: (1) numerical gradients for computing higher-order derivatives as a smoothing operation and (2) coarseto-fine optimization on the hash grids controlling different levels of details. Even without auxiliary depth, Neuralangelo can effectively recover dense 3D surface structures from multi-view images with a fidelity that significantly surpasses previous methods, enabling detailed large-scale scene reconstruction from RGB video captures.

Video

Object-centric Reconstruction

Approach

Neuralangelo builds on top of multi-resolution hash encoding and SDF-based volume rendering.

1. Numerical gradients to compute higher-order derivatives

Using numerical gradients with step size matching the spatial resolutions of hash grid optimizes beyond the local cells. The numerical gradients act as a smoothing operation on the SDF in comparison to the analytical gradients.

2. Progressive level of details

By progressively decreasing the step size for numerical gradient and enabling higher resolution hash grids, the optimization landscape is better shaped to recover both large smooth surfaces and fine geometric details. Such a learning curriculum enables progressive level of details.

3. Optimization

Neuralangelo uses three optimization objectives: $$\mathcal{L} = \mathcal{L}_{rgb} + w_\text{eik} \mathcal{L}_{eik} + w_\text{curv} \mathcal{L}_{curv}.$$

RGB synthesis loss $ \mathcal{L}_{rgb} $ : RGB reconstruction loss between the input image and synthesized images.
Eikonal loss $ \mathcal{L}_{eik} $ : regularize underlying SDF such that the surface normals are unit-norm.
Curvature loss $ \mathcal{L}_{curv} $ : regularize underlying SDF such that the mean-curvature is not arbitrarily large.

Presentation

Poster

(Click image to enlarge)

Citation

@inproceedings{li2023neuralangelo,
  title={Neuralangelo: High-Fidelity Neural Surface Reconstruction},
  author={Li, Zhaoshuo and M\"uller, Thomas and Evans, Alex and Taylor, Russell H and Unberath, Mathias and Liu, Ming-Yu and Lin, Chen-Hsuan},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2023}
}