1. [Publications](/publications)
2. MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
 
 # MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views 

  ![](/sites/default/files/styles/wide/public/publications/mvlidarnet_0.png?itok=SvqfDsxq)

 Autonomous driving requires the inference of actionable information such as detecting and classifying objects, and determining the drivable space. To this end, we present Multi-View LidarNet (MVLidarNet), a two-stage deep neural network for multi-class object detection and drivable space segmentation using multiple views of a single LiDAR point cloud. The first stage processes the point cloud projected onto a perspective view in order to semantically segment the scene. The second stage then processes the point cloud (along with semantic labels from the first stage) projected onto a bird's eye view, to detect and classify objects. Both stages use an encoder-decoder architecture. We show that our multi-view, multi-stage, multi-class approach is able to detect and classify objects while simultaneously determining the drivable space using a single LiDAR scan as input, in challenging scenes with more than one hundred vehicles and pedestrians at a time. The system operates efficiently at 150 fps on an embedded GPU designed for a self-driving car, including a postprocessing step to maintain identities over time. We show results on both KITTI and a much larger internal dataset, thus demonstrating the method's ability to scale by an order of magnitude.



 ## Authors



Ke Chen (NVIDIA)

Ryan Oldja (NVIDIA)

Nikolai Smolyanskiy (NVIDIA)

[Stan Birchfield](/person/stan-birchfield)

Alexander Popov (NVIDIA)

David Wehr (NVIDIA)

Ibrahim Eden (NVIDIA)

Joachim Pehserl (NVIDIA)

 

 

 ## Publication Date



Thursday, June 11, 2020

 

 ## Published in



IROS 2020

 

 ## Research Area



[Applied Perception](/research-area/applied-perception)

[Autonomous Vehicles](/research-area/autonomous-vehicles)

[Computer Vision](/research-area/computer-vision)

[Robotics](/research-area/robotics)

 

 

 ## External Links



[arXiv paper](https://arxiv.org/abs/2006.05518)

[video](https://youtu.be/2ck5_sToayc)