Autonomous Vehicle Research Group

Autonomous Vehicle Research Group


Welcome to the homepage of the NVIDIA Research Autonomous Vehicle Research Group led by Dr. Marco Pavone.

We are a newer team within NVIDIA Research that brings together a diverse and interdisciplinary set of researchers to address core topics in vehicle autonomy, ranging from perception and prediction to planning and control, as well as advance the state of the art in a number of critical related fields such as decision making under uncertainty, self-supervised learning, and the verification and validation of safety-critical AI systems.


February 2024 - Welcome to the team Zhiwen Fan!

January 2024 - Welcome to the team Yurong You and Ziqi Lu! Also, our paper on Partial-View Object View Synthesis via Filtered Inversion will be featured as a spotlight at 3DV!

November 2023 - Welcome to the team Maximilian Igl! We’ve also had Ran (Thomas) Tian join our group as an intern!

October 2023 - 2 papers accepted to NeurIPS! We’ve also organized 1 workshop, presented 1 paper, and given 6 invited talks at ICCV + IROS! We’ve also welcomed Siming Yan to our group as an intern!

Research Areas

Our group addresses challenges in AI-powered autonomous vehicles through a multi-pronged research agenda which, collectively, is aimed at laying the foundation for the next generation of AV autonomy.

  • Next-Generation AV Architectures: While modular architectures provide unparalleled reusability and interpretability, they suffer from integration challenges and information bottlenecks between modules. Accordingly, we are investigating tools and methods to design much more integrated autonomy stacks along the following axes:

    • System-level, including differentiable, semi-modular, and end-to-end stacks.
    • Representation-level: leveraging neural and prior-augmented representations (standard-definition maps, map priors, NeRF priors, etc.).
    • Component-level: investigating new methods for joint prediction/planning, online mapping, neural reconstruction, and policy planning, among others.
  • AV Foundation Models: Large, generally-trained models contain a wealth of information. Harnessing such a breadth of knowledge in autonomous driving may help AVs generalize to new domains and navigate smoothly in rare scenarios. Towards this end, our group investigates vision-language foundation models, vision-language-action modeling, and scene understanding in the wild.

  • Simulation: Simulation is paramount for accelerating AV development. Towards developing more realistic and controllable simulation enviroments, our group is pursuing behavior/traffic modeling to produce realistic and interactive agent motion, language-based simulation generation to leverage text for specifying and editing simulated scenarios, and neural simulators to reconstruct and simulate logged scenes and their sensor data.

  • AI Safety for AVs: Data-driven methods based on ML have enabled tasks beyond what was considered possible with their traditional, non-learning-based counterparts. However, ML-based algorithms can suffer from unpredictability and erratic behavior. To enable the confident infusion of ML models within the decision-making loop of an autonomous vehicle, we research tools and methods for providing both offline and online algorithmic assurances for components of a learning-enabled autonomy stack, including uncertainty quantification and calibration, online monitoring of AI components, rule reasoning in neural architectures, and validation (safety KPIs, evaluation, etc.).

In pursuit of this research agenda, we leverage and combine expertise from a number of fields, including optimal control, decision-making, reinforcement learning, deep learning, machine learning, robotics, computer vision, and formal methods.

Our research efforts are enriched by close collaborations with NVIDIA’s AV product teams.


Quickly discover relevant content by filtering publications.