Research Areas

Autonomous Vehicles

Associated Publications

2025

Cosmos Transfer 1: World-to-World Transfer with Adaptive Multi-Control for Physical AI
Ming-Yu Liu
Arxiv
Cosmos-Reason 1: From Physical AI Common Sense to Embodied Decisions
Tsung-Yi Lin, Ming-Yu Liu
NVIDIA Isaac GR00T N1: An Open Foundation Model for Humanoid Robots
Yuke Zhu, Linxi "Jim" Fan, NVIDIA GEAR Team
Cosmos World Foundation Model Platform for Physical AI
Ming-Yu Liu, Many other contributors at https://d1qx31qr3h6wln.cloudfront.net/publications/NVIDIA%20Cosmos_4.pdf

2024

Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models
Alexander Popov, Alperen Degirmenci, David Wehr, Shashank Hegde , Ryan Oldja, Alexey Kamenev, Bertrand Douillard, David Nistér, Urs Muller, Ruchi Bhargava, Stan Birchfield, Nikolai Smolyanskiy
arXiv
Real-Time Anomaly Detection and Reactive Planning with Large Language Models
Rohan Sinha, Amine Elhafsi, Christopher Agia, Matthew Foutter, Edward Schmerling, Marco Pavone
RSS

2023

Learning for CasADi: Data-driven Models in Numerical Optimization
Tim Salzmann, Jon Arrizabalaga, Joel Andersson, Marco Pavone, Markus Ryll
Arxiv
PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction
Apoorva Sharma, Sushant Veer, Asher Hancock, Heng Yang, Marco Pavone, Anirudha Majumdar
Arxiv
Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Maps
Katie Z Luo, Xinshuo Weng, Yan Wang, Shuang Wu, Jie Li, Kilian Q Weinberger, Yue Wang, Marco Pavone
Arxiv
Interpretable Trajectory Prediction for Autonomous Vehicles via Counterfactual Responsibility
Kai-Chieh Hsu, Karen Leung, Yuxiao Chen, Jaime Fernandez Fisac, Marco Pavone
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
trajdata: A Unified Interface to Multiple Human Trajectory Datasets
Boris Ivanovic, Guanyu Song, Igor Gilitschenski, Marco Pavone
Arxiv Paper
Language Conditioned Traffic Generation
Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, Philipp Krähenbühl
Arxiv
Task-Aware Risk Estimation of Perception Failures for Autonomous Vehicles
Pasquale Antonante, Sushant Veer, Karen Leung, Xinshuo Weng, Luca Carlone, Marco Pavone
Robotics: Science and Systems (RSS) 2023
Refining Obstacle Perception Safety Zones via Maneuver-Based Decomposition
Sever Topan, Yuxiao Chen, Ed Schmerling, Karen Leung, Jonas Nilsson, Michael Cox, Marco Pavone
IEEE Intelligent Vehicles Symposium (IV)
Learning Autonomous Vehicle Safety Concepts from Demonstrations
Karen Leung, Sushant Veer, Edward Schmerling, Marco Pavone
American Control Conference (ACC) 2023
Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models
Filippos Christianos, Peter Karkus, Boris Ivanovic, Stefano V. Albrecht, Marco Pavone
IEEE International Conference on Robotics and Automation (ICRA) 2023
BITS: Bi-level Imitation for Traffic Simulation
Danfei Xu, Yuxiao Chen, Boris Ivanovic, Marco Pavone
IEEE International Conference on Robotics and Automation (ICRA) 2023
Tree-structured Policy Planning with Learned Behavior Models
Yuxiao Chen, Peter Karkus, Boris Ivanovic, Xinshuo Weng, Marco Pavone
IEEE International Conference on Robotics and Automation (ICRA) 2023
Guided Conditional Diffusion for Controllable Traffic Simulation
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Gerry Che, Baishakhi Ray, Marco Pavone
IEEE International Conference on Robotics and Automation (ICRA) 2023
Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles
Sushant Veer, Karen Leung, Ryan Cosner, Yuxiao Chen, Marco Pavone
IEEE International Conference on Robotics and Automation (ICRA) 2023
Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning
Boris Ivanovic, James Harrison, Marco Pavone
IEEE International Conference on Robotics and Automation (ICRA) 2023
Semantic Anomaly Detection with Large Language Models
Amine Elhafsi, Rohan Sinha, Christopher Agia, Edward Schmerling, Issa Nesnas, Marco Pavone
Arxiv
Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models
Wenhao Ding, Tong Che, Ding Zhao, Marco Pavone
Arxiv
Task-Aware Risk Estimation of Perception Failures for Autonomous Vehicles
Pasquale Antonante, Sushant Veer, Karen Leung, Xinshuo Weng, Luca Carlone, Marco Pavone
Arxiv
FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization
Heng Yang, Marco Pavone
Arxiv

2022

Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles
Alec Farid, Sushant Veer, Boris Ivanovic, Karen Leung, Marco Pavone
Conference on Robot Learning (CoRL) 2022
DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles
Peter Karkus, Boris Ivanovic, Shie Mannor, Marco Pavone
Conference on Robot Learning (CoRL) 2022
Robust Trajectory Prediction against Adversarial Attacks
Yulong Cao, Danfei Xu, Xinshuo Weng, Z. Morely Mao, Anima Anandkumar, Chaowei Xiao, Marco Pavone
Conference on Robot Learning (CoRL) 2022
Selected for Oral Presentation
Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty
Boris Ivanovic, Kuan-Hui Lee, Pavel Tokmakov, Blake Wulfe, Adrien Gaidon, Marco Pavone
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2022
AdvDO: Realistic Adversarial Attacks for Trajectory Prediction
Yulong Cao, Chaowei Xiao, Anima Anandkumar, Danfei Xu, Marco Pavone
European Conference on Computer Vision (ECCV) 2022
Robust and Controllable Object-Centric Learning through Energy-based Models
Ruixiang Zhang, Tong Che, Boris Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, Liam Paull
Arxiv
Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning
Boris Ivanovic, James Harrison, Marco Pavone
Arxiv
Zhuyi: Perception Processing Rate Estimation for Safety in Autonomous Vehicles
Yu-Shun Hsiao, Siva Hari, Michał Filipiuk, Timothy Tsai, Michael B. Sullivan, Vijay Janapa Reddi, Vasu Singh, Steve Keckler
Design Automation Conference (DAC)
Exploiting Temporal Data Diversity for Detecting Safety-critical Faults in AV Compute Systems
Saurabh Jha, Shengkun Cui, Timothy Tsai, Siva Hari, Michael B. Sullivan, Zbigniew T. Kalbarczyk, Steve Keckler, Ravishankar K. Iyer
International Conference on Dependable Systems and Networks (DSN)
ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning
Yuxiao Chen, Boris Ivanovic, Marco Pavone
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022
Whose Track Is It Anyway? Improving Robustness to Tracking Errors with Affinity-Based Prediction
Xinshuo Weng, Boris Ivanovic, Kris Kitani, Marco Pavone
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022
MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation
Xinshuo Weng, Boris Ivanovic, Marco Pavone
IEEE Intelligent Vehicles Symposium (IV) 2022
Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation
Sever Topan, Karen Leung, Yuxiao Chen, Pritish Tupekar, Edward Schmerling, Jonas Nilsson, Michael Cox, Marco Pavone
IEEE Intelligent Vehicles Symposium (IV) 2022
Injecting Planning-Awareness into Prediction and Detection Evaluation
Boris Ivanovic, Marco Pavone
IEEE Intelligent Vehicles Symposium (IV) 2022
Propagating State Uncertainty Through Trajectory Forecasting
Boris Ivanovic, Yifeng Lin, Shubham Shrivastava, Punarjay Chakravarty, Marco Pavone
IEEE International Conference on Robotics and Automation (ICRA) 2022
Zhuyi: Perception Processing Rate Estimation for Safety in Autonomous Vehicles
Yu-Shun Hsiao, Siva Hari, Michał Filipiuk, Timothy Tsai, Michael B. Sullivan, Vijay Janapa Reddi, Vasu Singh, Steve Keckler
arXiv
PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation
Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nister, Nikolai Smolyanskiy
ICRA 2022

2021

Suraksha: A Framework to Analyze the Safety Implications of Perception Design Choices in AVs
Hengyu Zhao, Siva Hari, Timothy Tsai, Michael B. Sullivan, Steve Keckler, Jishen Zhao
International Symposium on Software Reliability Engineering (ISSRE)
MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation
Xinshuo Weng, Boris Ivanovic, Marco Pavone
CVPR
Sample-Efficient Safety Assurances using Conformal Prediction
Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, Marco Pavone
Arxiv
Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles
Zahra Ghodsi, Siva Hari, Iuri Frosio, Timothy Tsai, Alejandro Troccoli, Steve Keckler, Siddharth Garg, Anima Anandkumar
IEEE Intelligent Vehicles Symposium (IV)
Suraksha: A Quantitative AV Safety Evaluation Framework to Analyze Safety Implications of Perception Design Choices
Hengyu Zhao, Siva Hari, Timothy Tsai, Michael B. Sullivan, Steve Keckler, Jishen Zhao
Workshop on Safety and Security of Intelligent Vehicles (SSIV)
Binary TTC: A Temporal Geofence for Autonomous Navigation
Abhishek Badki, Orazio Gallo, Jan Kautz, Pradeep Sen
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021
Best Student Paper Honorable Mention, CVPR 2021
Simulation Driven Design and Test for Safety of AI Based Autonomous Vehicles
Vasu Singh, Siva Hari, Timothy Tsai, Mandar Pitale
Workshop on Safe Artificial Intelligence for Automated Driving (SAIAD)
Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles
Zahra Ghodsi, Siva Hari, Iuri Frosio, Timothy Tsai, Alejandro Troccoli, Steve Keckler, Siddharth Garg, Anima Anandkumar
arXiv

2020

Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles
Zahra Ghodsi, Siva Hari, Iuri Frosio, Timothy Tsai, Alejandro Troccoli, Steve Keckler, Siddharth Garg, Anima Anandkumar
IEEE International Workshop on Automotive Reliability & Test (ART)
AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems
Guangpeng Li, Yiran Li, Saurabh Jha, Timothy Tsai, Michael B. Sullivan, Siva Hari, Zbigniew Kalbarczyk, Ravishankar Iyer
International Symposium on Software Reliability Engineering (ISSRE)
Best Paper award
ML-driven Malware that Targets AV Safety
Saurabh Jha, Shengkun Cui, Subho S. Banerjee, James Cyriac, Timothy Tsai, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer
International Conference on Dependable Systems and Networks (DSN)
MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
Ke Chen, Ryan Oldja, Nikolai Smolyanskiy, Stan Birchfield, Alexander Popov, David Wehr, Ibrahim Eden, Joachim Pehserl
IROS 2020
ML-driven Malware that Targets AV Safety
Saurabh Jha, Shengkun Cui, Subho S. Banerjee, James Cyriac, Timothy Tsai, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer
arXiv

2019

ML-based Fault Injection for Autonomous Vehicles: A Case for Bayesian Fault Injection
Saurabh Jha, Subho S. Banerjee, Timothy Tsai, Siva Hari, Michael B. Sullivan, Zbigniew T. Kalbarczyk, Steve Keckler, Ravishankar K. Iyer
arXiv
Kayotee: A Fault Injection-based System to Assess the Safety and Reliability of Autonomous Vehicles to Faults and Errors
Saurabh Jha, Timothy Tsai, Siva Hari, Michael B. Sullivan, Zbigniew Kalbarczyk, Steve Keckler, Ravishankar K. Iyer
arXiv
ML-based Fault Injection for Autonomous Vehicles: A Case for Bayesian Fault Injection
Saurabh Jha, Subho Banerjee, Timothy Tsai, Siva Hari, Michael B. Sullivan, Zbigniew T. Kalbarczyk, Steve Keckler, Ravishankar K. Iyer
International Conference on Dependable Systems and Networks (DSN)

2018

Kayotee: A Fault Injection-based System to Assess the Safety and Reliability of Autonomous Vehicles to Faults and Errors
Saurabh Jha, Timothy Tsai, Siva Hari, Michael B. Sullivan, Zbigniew Kalbarczyk, Steve Keckler, Ravishankar K. Iyer
Third IEEE International Workshop on Automotive Reliability & Test
On the Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach
Nikolai Smolyanskiy, Alexey Kamenev, Stan Birchfield
CVPR 2018 Workshop on Autonomous Driving

2016

Towards Selecting Robust Hand Gestures for Automotive Interfaces
Shalini Gupta, Pavlo Molchanov, Xiaodong Yang, Stephen Tyree, Jan Kautz
IEEE Intelligent Vehicles Symposium (IV) 2016
Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictors
Sushant Veer, Apoorva Sharma, Marco Pavone
Arxiv

Researchers

Apoorva Sharma
Bertrand Douillard
Boris Ivanovic
Danfei Xu
Ed Schmerling
Gal Dalal
Gerry Che
Heng Yang
Karen Leung
Marco Pavone
Maximilian Igl
Michael Watson
Muya Chang
Peter Karkus
Qi Wang
Rachel Luo
Siva Hari
Sushant Veer
Thomas Tian
Vinu Joseph
Wenhao Ding
Wenjie Luo
Xiaodong Yang
Xinshuo Weng
Yan Wang
Yue Wang
Yulong Cao
Yuxiao Chen
Zhiding Yu