Abstract

Neural surface reconstruction has shown to be powerful for recovering dense 3D surfaces via image-based neural rendering. However, current methods struggle to recover detailed structures of real-world scenes. To address the issue, we present Neuralangelo, which combines the representation power of multi-resolution 3D hash grids with neural surface rendering. Our approach is enabled by two key ingredients: (1) numerical gradients for computing higher-order derivatives as a smoothing operation and (2) coarseto-fine optimization on the hash grids controlling different levels of details. Even without auxiliary depth, Neuralangelo can effectively recover dense 3D surface structures from multi-view images with a fidelity that significantly surpasses previous methods, enabling detailed large-scale scene reconstruction from RGB video captures.

Video

Results

Large-scale Reconstruction

RGB view synthesis 3D surface reconstruction Surface normal

Object-centric Reconstruction

NeuS Neuralangelo (Ours) NeuralWarp

Approach

Neuralangelo builds on top of multi-resolution hash encoding and SDF-based volume rendering.

Numerical Gradients to Compute Higher-order Derivatives

Using numerical gradients with step size matching the spatial resolutions of hash grid optimizes beyond the local cells. The numerical gradients act as a smoothing operation on the SDF in comparison to the analytical gradients.

Progressive Level of Details

By progressively decreasing the step size for numerical gradient and enabling higher resolution hash grids, the optimization landscape is better shaped to recover both large smooth surfaces and fine geometric details. Such a learning curriculum enables progressive level of details.

Optimization

Neuralangelo uses three optimization objectives: $$\mathcal{L} = \mathcal{L}_{rgb} + w_\text{eik} \mathcal{L}_{eik} + w_\text{curv} \mathcal{L}_{curv}.$$
  • RGB synthesis loss \( \mathcal{L}_{rgb} \) :  RGB reconstruction loss between the input image and synthesized images.
  • Eikonal loss \( \mathcal{L}_{eik} \) :  regularize underlying SDF such that the surface normals are unit-norm.
  • Curvature loss \( \mathcal{L}_{curv} \) :  regularize underlying SDF such that the mean-curvature is not arbitrarily large.

Presentation

Citation

@inproceedings{li2023neuralangelo,
  title={Neuralangelo: High-Fidelity Neural Surface Reconstruction},
  author={Li, Zhaoshuo and M\"uller, Thomas and Evans, Alex and Taylor, Russell H and Unberath, Mathias and Liu, Ming-Yu and Lin, Chen-Hsuan},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023}
}