Left: real LiDAR scan demonstrating key LiDAR return properties: a single return and two returns (first return shown in blue and second return in orange). Right: NFL models the waveform and accurately reproduces these properties. (a) Top: the LiDAR energy is fully scattered by the first surface. Bottom: NFL estimates range via peak detection on the computed weights w followed by volume rendering based range refinement. (b) Top: secondary returns resulting from a beam hitting two surfaces. bottom: NFL employs beam divergence r(o, d) and a truncated volume rendering to estimate the second return. (c) Top: beams that do not hit a surface do not return detectable signal. Bottom: NFL utilizes geometric and semantic features to predict the ray drop probability.
Abstract
We present Neural Fields for LiDAR (NFL), a method to optimise a neural field scene representation from LiDAR measurements, with the goal of synthesizing realistic LiDAR scans from novel viewpoints. NFL combines the rendering power of neural fields with a detailed, physically motivated model of the LiDAR sensing process, thus enabling it to accurately reproduce key sensor behaviors like beam divergence, secondary returns, and ray dropping. We evaluate NFL on synthetic and real LiDAR scans and show that it outperforms explicit reconstruct-then-simulate methods as well as other NeRF-style methods on LiDAR novel view synthesis task. Moreover, we show that the improved realism of the synthesized views narrows the domain gap to real scans and translates to better registration and semantic segmentation performance.
LiDAR NVS by adjusting the sensor properties
NFL enables to synthesize LiDAR novel views by adjusting the sensor properties, such as upsampling and downsampling the beams, and reducing azimuth angle resolution. For each example, we show NVS following original configuration (left) and NVS after adjusting the sensor properties (right).
Upsample the beams
Downsample the beams
Downsample the rays
LiDAR NVS by adjusting the sensor poses
NFL enables to synthesize LiDAR novel views by adjusting the sensor poses, such as lifting, tilting, and shifting the sensor in-plane. For each example, the first row shows NVS following raw trajectory (left) and NVS after adjusting the sensor poses (right), the second row shows three individual examples which gradually apply the adjustments.
Lift the sensor
Tilt the sensor
Shift the sensor in-plane
Qualitative results of LiDAR range, intensity and ray drop estimation
LiDAR novel view synthesis on Waymo Interp. dataset. On the left, we color-code rays with and without return using blue and gray colors. On the right side, LiDAR intensity values are color-coded as :0 0.25.
Comparison of first range estimation. Regions with gross errors (-100 100 cm) are highlighted.
LiDAR novel view synthesis by changing the sensor elevation angle θ, pose (x, y, z) and number of beams. Zoom-in points are color-coded by intensity values as :0 0.25.
Citation
@inproceedings{Huang2023nfl,
title={Neural LiDAR Fields for Novel View Synthesis},
author={Huang, Shengyu and Gojcic, Zan and Wang, Zian and Williams, Francis and Kasten, Yoni and Fidler, Sanja and Schindler, Konrad and Litany, Or},
journal={Proceedings of the IEEE/CVF International Conference on Computer Vision},
year={2023}
}
Acknowledgments
We sincerely thank Benjamin Naujoks, Steven Butrimas, Tomislav Medic, Yu Han, and Prof. Dr. Andreas Wieser for helpful discussions around LiDAR models. We are grateful for the feedback on figures from Rodrigo Caye Daudt. This appreciation extends to Zvi Greenstein for organisation support.
: Neural LiDAR Fields for Novel View Synthesis
0.25.
100 cm) are highlighted.
