Autonomous Systems and Physical AI Research (ASPIRE) Group

Autonomous Systems and Physical AI Research (ASPIRE) Group

Welcome!

Welcome to the homepage of the NVIDIA Research Autonomous Systems and Physical AI Research (ASPIRE) Group, led by Dr. Marco Pavone.

Our mission is to develop the algorithmic foundations for intelligent autonomous systems that perceive, reason, plan, and act in the physical world. Our research spans foundation models, embodied reasoning, navigation and long-horizon planning, agentic development workflows, simulation and reinforcement learning, and safety evaluation and assurance. We study these capabilities with a particular focus on autonomous vehicles, while expanding to a growing range of embodiments, including trucks, off-road vehicles, drones, quadrupeds, humanoids, and more.


If this sounds interesting to you, come work with us! We are currently accepting applications for the following positions:

Research Areas

The ASPIRE Group develops general methods for autonomy across Physical AI. We study these capabilities with a particular focus on autonomous vehicles, while increasingly extending to a broad range of embodiments, including trucks, off-road vehicles, drones, quadrupeds, humanoids, and beyond.

  • Foundation Models for Embodied Autonomy: We develop multimodal and omnimodal models that connect language, perception, world understanding, and action. Our work includes vision-language-action models, world-action models, embodied and counterfactual reasoning, navigation and long-horizon planning across embodiments, transferable action representations, and efficient models for deployment.

  • Agentic Physical AI Development: We build agentic workflows that accelerate the full autonomy-development loop: hypothesis generation, data curation and labeling, model training and post-training, closed-loop evaluation, deployment, and continuous improvement. A key goal is to make these workflows reusable across domains and embodiments while retaining the expertise and tools each system requires.

  • Simulation and Reinforcement Learning: We investigate realistic, controllable simulation and scalable closed-loop learning. This includes neural reconstruction and sensor simulation, behavior and environment modeling, real-to-sim-to-real workflows, high-throughput reinforcement learning, and evaluation in diverse, safety-critical scenarios.

  • Safety Evaluation and Assurance: Through our Physical AI safety research, we develop design-time, run-time, and validation-time guardrails for learning-enabled autonomous systems. Our research spans safety reasoning and risk assessment, uncertainty quantification and calibration, online monitoring, interpretable and rule-aware models, stress testing, and simulation-accelerated validation.

Our research is strengthened by collaborations across NVIDIA and with the broader Physical AI community.

Highlights

  • Throughout the summer, we have released:
    • Alpamayo 2 Super (video), an open 34B reasoning vision-language-action foundation model for autonomous driving.
    • AlpaGym, a closed-loop reinforcement learning framework for training end-to-end driving policies in simulation.
    • CoC Autolabeler, an open-source pipeline that generates Meta-Actions and Chain-of-Causation reasoning labels for driving clips.

Meet the ASPIRE Group

Members

Avatar

Marco Pavone

Avatar

Yulong Cao

Avatar

Gerry Che

Avatar

Xiangyu Chen

Avatar

Yuxiao Chen

Avatar

Wenyan Cong

Avatar

Wenhao Ding

Avatar

Maximilian Igl

Avatar

Bertrand Douillard

Avatar

Boris Ivanovic

Avatar

Peter Karkus

Avatar

Boyi Li

Avatar

Zheng Lian

Avatar

Rachel Luo

Avatar

Wenjie Luo

Avatar

Jef Packer

Avatar

Apoorva Sharma

Avatar

Jack Shi

Avatar

Shuhan Tan

Avatar

Thomas Tian

Avatar

Sushant Veer

Avatar

Yan Wang

Avatar

Yu Wang

Avatar

Xinshuo Weng

Avatar

Michael Watson

Avatar

Yurong You

Faculty Scientists

Avatar

Karen Leung

Avatar

Chaowei Xiao

Avatar

Heng Yang

Current Interns

Avatar

Yuxin Chen

Avatar

Minkyoung Cho

Avatar

Daniel Dauner

Avatar

Lan Feng

Avatar

Asher Hancock

Avatar

Jack Lu

Avatar

Kensuke Nakamura

Avatar

Anjali Parashar

Avatar

Chensheng Peng

Avatar

Han Qi

Avatar

William Sharpless

Avatar

Kai Yan

Avatar

Chenyu Zhang

Avatar

Shuhong Zheng

Avatar

Zewei Zhou

Alumni

Avatar

Fahmy Adan

Avatar

Pasquale Antonante

Avatar

Andrea Bajcsy

Avatar

Kaustav Chakraborty

Avatar

Ed Schmerling

Avatar

Ziyu Chen

Avatar

Kashyap Chitta

Avatar

Vincent Cho

Avatar

Filippos Christianos

Avatar

Ryan Cosner

Avatar

Zhiwen Fan

Avatar

Alec Farid

Avatar

Tarun Gupta

Avatar

Kai-Chieh Hsu

Avatar

Zhiyu Huang

Avatar

Lingdong Kong

Avatar

Haohong Lin

Avatar

Yiming Li

Avatar

Zhixuan Liang

Avatar

Ziqi Lu

Avatar

Katie Luo

Avatar

Jiageng Mao

Avatar

Allen Ren

Avatar

Gautam Singh

Avatar

Fan-Yun Sun

Avatar

Sander Tonkens

Avatar

Letian Wang

Avatar

Yue Wang

Avatar

Danfei Xu

Avatar

Jiawei Yang

Avatar

Cunjun Yu

Avatar

Jiawei Zhang

Avatar

Zhejun Zhang

Avatar

Ziyuan Zhong

Publications

Quickly discover relevant content by filtering publications.

SafeVL: Driving Safety Evaluation via Meticulous Reasoning in Vision Language Models

Submitted

Safety remains a fundamental challenge in autonomous driving, with a key step being the development of a safety evaluator that can …
PDF Project
SafeVL: Driving Safety Evaluation via Meticulous Reasoning in Vision Language Models

Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail

arXiv

End-to-end architectures trained via imitation learning have advanced autonomous driving by scaling model size and data, yet …
PDF Code Project
Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail

LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving

Submitted

Simulators can generate virtually unlimited driving data, yet imitation learning policies in simulation still struggle to achieve …
PDF Code
LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving

Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving

Submitted

We present Flex, an efficient and effective scene encoder that addresses the computational bottleneck of processing high-volume …
PDF Project
Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving

Latent Chain-of-Thought World Modeling for End-to-End Driving

Submitted

Recent Vision-Language-Action (VLA) models for autonomous driving explore inference-time reasoning as a way to improve driving …
PDF
Latent Chain-of-Thought World Modeling for End-to-End Driving

Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions

NeurIPS 2025

Synthesizing realistic Martian landscape videos is crucial for mission rehearsal and robotic simulation. However, this task poses …
PDF Code Project
Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions

Optimization-Guided Diffusion for Interactive Scene Generation

Submitted

Realistic and diverse multi-agent driving scenes are crucial for evaluating autonomous vehicles, but safety-critical events which are …
PDF Project
Optimization-Guided Diffusion for Interactive Scene Generation

Beyond Behavior Cloning in Autonomous Driving: a Survey of Closed-Loop Training Techniques

Submitted

Behavior cloning, the dominant approach for training autonomous vehicle (AV) policies, suffers from a fundamental gap: policies trained …
PDF Project
Beyond Behavior Cloning in Autonomous Driving: a Survey of Closed-Loop Training Techniques

ReSim: Reliable World Simulation for Autonomous Driving

NeurIPS 2025

How can we reliably simulate future driving scenarios under a wide range of ego driving behaviors? Recent driving world models, …
PDF Code Project
ReSim: Reliable World Simulation for Autonomous Driving

RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies

Submitted

Autonomous driving policies are typically trained via open-loop behavior cloning of human demonstrations. However, such policies suffer …
PDF
RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies

Model-Based Policy Adaptation for Closed-Loop End-to-end Autonomous Driving

NeurIPS 2025

End-to-end (E2E) autonomous driving models have demonstrated strong performance in open-loop evaluations but often suffer from …
PDF
Model-Based Policy Adaptation for Closed-Loop End-to-end Autonomous Driving

Extrapolated Urban View Synthesis Benchmark

ICCV 2025

Photorealistic simulators are essential for the training and evaluation of vision-centric autonomous vehicles (AVs). At their core is …
PDF Code Project
Extrapolated Urban View Synthesis Benchmark

Wolf: Dense Video Captioning with a World Summarization Framework

TMLR 2025

We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a …
PDF Code Project
Wolf: Dense Video Captioning with a World Summarization Framework

CaRL: Learning Scalable Planning Policies with Simple Rewards

CoRL 2025

We investigate reinforcement learning (RL) for privileged planning in autonomous driving. State-of-the-art approaches for this task are …
PDF Code
CaRL: Learning Scalable Planning Policies with Simple Rewards

Can Test-Time Scaling Improve World Foundation Model?

COLM 2025

World foundation models, which simulate the physical world by predicting future states from current observations and inputs, have …
PDF Code Project
Can Test-Time Scaling Improve World Foundation Model?

Safety Evaluation of Motion Plans Using Trajectory Predictors as Forward Reachable Set Estimators

Submitted

The advent of end-to-end autonomy stacks - often lacking interpretable intermediate modules - has placed an increased burden on …
PDF
Safety Evaluation of Motion Plans Using Trajectory Predictors as Forward Reachable Set Estimators

Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation

CoRL 2025

Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often …
PDF
Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation

RAMEN: Real-time Asynchronous Multi-agent Neural Implicit Mapping

RSS 2025

Multi-agent neural implicit mapping allows robots to collaboratively capture and reconstruct complex environments with high fidelity. …
PDF
RAMEN: Real-time Asynchronous Multi-agent Neural Implicit Mapping

Diagnostic Runtime Monitoring with Martingales

RSS 2025

Machine learning systems deployed in safety-critical robotics settings must be robust to distribution shifts. However, system designers …
PDF
Diagnostic Runtime Monitoring with Martingales

Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving

RA-L 2025

Autoregressive Transformers are increasingly being deployed as end-to-end robot and autonomous vehicle (AV) policy architectures, owing …
PDF
Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving

Data Scaling Laws for End-to-End Autonomous Driving

CVPRW 2025

Autonomous vehicle (AV) stacks have traditionally relied on decomposed approaches, with separate modules handling perception, …
PDF
Data Scaling Laws for End-to-End Autonomous Driving

Pseudo-Simulation for Autonomous Driving

CoRL 2025

Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations. Real-world evaluation is often challenging due …
PDF Code
Pseudo-Simulation for Autonomous Driving

E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models

Submitted

Spatial intelligence, encompassing 3D reconstruction, perception, and reasoning, is fundamental to applications such as robotics, …
PDF Code Project
E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models

SHRED: Synthesizing Rule-Based Environments for Driving

CVPRW 2025

We provide several simple, interpretable rule-based baselines for the 2025 Waymo Scenario Generation Challenge. The challenge addresses …
PDF Project
SHRED: Synthesizing Rule-Based Environments for Driving

Open X-AV: Unifying End-to-End Autonomous Driving Datasets

CVPRW 2025

The fragmentation of existing autonomous vehicle (AV) datasets hinders the development of generalizable driving policies that can …
PDF Project
Open X-AV: Unifying End-to-End Autonomous Driving Datasets

LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning

NAACL 2025

This paper presents an advanced mathematical problem-solving framework, LLaMA-Berry, for enhancing the mathematical reasoning ability …
PDF
LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning

Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion

ICLR 2025

An important paradigm in 3D object detection is the use of multiple modalities to enhance accuracy in both normal and challenging …
PDF Project
Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion

OmniRe: Omni Urban Scene Reconstruction

ICLR 2025

We introduce OmniRe, a comprehensive system for efficiently creating high-fidelity digital twins of dynamic real-world scenes from …
PDF Code Project
OmniRe: Omni Urban Scene Reconstruction

LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation

ICLR 2025

Visual prompting has gained popularity as a method for adapting pre-trained models to specific tasks, particularly in the realm of …
PDF Code
LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation

Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models

CVPR 2025

Traffic simulation aims to learn a policy for traffic agents that, when unrolled in closed-loop, faithfully recovers the joint …
PDF Code Project
Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

Submitted

While neural 3D reconstruction has advanced substantially, it typically requires densely captured multi-view data with carefully …
PDF Code Project
InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

Online Aggregation of Trajectory Predictors

ICRA 2025

Trajectory prediction, the task of forecasting future agent behavior from past data, is central to safe and efficient autonomous …
PDF
Online Aggregation of Trajectory Predictors

Surprise Potential as a Measure of Interactivity in Driving Scenarios

Submitted

Validating the safety and performance of an autonomous vehicle (AV) requires benchmarking on real-world driving logs. However, typical …
PDF
Surprise Potential as a Measure of Interactivity in Driving Scenarios

DreamDrive: Generative 4D Scene Modeling from Street View Images

ICRA 2025

Synthesizing photo-realistic visual observations from an ego vehicle’s driving trajectory is a critical step towards scalable …
PDF Project
DreamDrive: Generative 4D Scene Modeling from Street View Images

STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes

ICLR 2025

We present STORM, a spatio-temporal reconstruction model designed for reconstructing dynamic outdoor scenes from sparse observations. …
PDF Code Project
STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes

LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models

ICLR 2025

Emerging 3D geometric foundation models, such as DUSt3R, offer a promising approach for in-the-wild 3D vision tasks. However, due to …
PDF Project
LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models

Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data

NeurIPS 2024

Open-vocabulary 3D object detection has recently attracted considerable attention due to its broad applications in autonomous driving …
PDF Code Project
Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data

Learning Multiple Initial Solutions to Optimization Problems

Submitted

Sequentially solving similar optimization problems under strict runtime constraints is essential for many applications, such as robot …
PDF Code
Learning Multiple Initial Solutions to Optimization Problems

DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features

NeurIPS 2024

We propose DistillNeRF, a self-supervised learning framework addressing the challenge of understanding 3D environments from limited 2D …
PDF Project
DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features

Large Spatial Model: End-to-end Unposed Images to Semantic 3D

NeurIPS 2024

Reconstructing and understanding 3D structures from a limited number of images is a classical problem in computer vision. Traditional …
PDF Code Project
Large Spatial Model: End-to-end Unposed Images to Semantic 3D

LoRD: Adapting Differentiable Driving Policies to Distribution Shifts

ICRA 2025

Distribution shifts between operational domains can severely affect the performance of learned models in self-driving vehicles (SDVs). …
PDF Code
LoRD: Adapting Differentiable Driving Policies to Distribution Shifts

System-Level Analysis of Module Uncertainty Quantification in the Autonomy Pipeline

CDC 2024

We present a novel perspective on the design, use, and role of uncertainty measures for learned modules in an autonomous system. While …
PDF
System-Level Analysis of Module Uncertainty Quantification in the Autonomy Pipeline

Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning

ICRA 2025

Autonomous driving necessitates the ability to reason about future interactions between traffic agents and to make informed evaluations …
PDF Project
Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning

System-Level Safety Monitoring and Recovery for Perception Failures in Autonomous Vehicles

ICRA 2025

The safety-critical nature of autonomous vehicle (AV) operation necessitates development of task-relevant algorithms that can reason …
PDF Project
System-Level Safety Monitoring and Recovery for Perception Failures in Autonomous Vehicles

Promptable Closed-loop Traffic Simulation

CoRL 2024

Simulation stands as a cornerstone for safe and efficient autonomous driving development. At its core a simulation system ought to …
PDF Code Project
Promptable Closed-loop Traffic Simulation

PARA-Drive: Parallelized Architecture for Real-time Autonomous Driving

CVPR 2024

Recent works have proposed end-to-end autonomous vehicle (AV) architectures comprised of differentiable modules achieving …
PDF
PARA-Drive: Parallelized Architecture for Real-time Autonomous Driving

Accelerating Online Mapping and Behavior Prediction via Direct BEV Feature Attention

ECCV 2024

Understanding road geometry is a critical component of the autonomous vehicle (AV) stack. While high-definition (HD) maps can readily …
PDF
Accelerating Online Mapping and Behavior Prediction via Direct BEV Feature Attention

Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving

CoRL 2024

The autonomous driving industry is increasingly adopting end-to-end learning from sensory inputs to minimize human biases in system …
PDF Project
Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving

NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

NeurIPS 2024

Benchmarking vision-based driving policies is challenging. On one hand, open-loop evaluation with real data is easy, but these results …
PDF Code
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

Driving Everywhere with Large Language Model Policy Adaptation

CVPR 2024

Adapting driving behavior to new environments, customs, and laws is a long-standing problem in autonomous driving, precluding the …
PDF Project
Driving Everywhere with Large Language Model Policy Adaptation

Parallelized Spatiotemporal Binding

ICML 2024

While modern best practices advocate for scalable architectures that support long-range interactions, object-centric models are yet to …
PDF
Parallelized Spatiotemporal Binding

Producing and Leveraging Online Map Uncertainty in Trajectory Prediction

CVPR 2024 (Best Paper Award Finalist)

High-definition (HD) maps have played an integral role in the development of modern autonomous vehicle (AV) stacks, albeit with high …
PDF
Producing and Leveraging Online Map Uncertainty in Trajectory Prediction

DiffuBox: Refining 3D Object Detection with Point Diffusion

NeurIPS 2024

Ensuring robust 3D object detection and localization is crucial for many applications in robotics and autonomous driving. Recent …
PDF Code Project
DiffuBox: Refining 3D Object Detection with Point Diffusion

RuleFuser: An Evidential Bayes Approach for Rule Injection in Imitation Learned Planners and Predictors for Robustness under Distribution Shifts

ISRR 2024

Modern motion planners for autonomous driving frequently use imitation learning (IL) to draw from expert driving logs. Although IL …
PDF
RuleFuser: An Evidential Bayes Approach for Rule Injection in Imitation Learned Planners and Predictors for Robustness under Distribution Shifts

Language-Image Models with 3D Understanding

ICLR 2025

Multi-modal large language models (MLLMs) have shown incredible capabilities in a variety of 2D vision and language tasks. We extend …
PDF Project
Language-Image Models with 3D Understanding

RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios

arXiv

Simulation plays a crucial role in the development of autonomous vehicles (AVs) due to the potential risks associated with real-world …
PDF Project
RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios

A Language Agent for Autonomous Driving

arXiv

Human-level driving is an ultimate goal of autonomous driving. Conventional approaches formulate autonomous driving as a …
PDF Project
A Language Agent for Autonomous Driving

Dolphins: Multimodal Language Model for Driving

arXiv

The quest for fully autonomous vehicles (AVs) capable of navigating complex real-world scenarios with human-like understanding and …
PDF Project
Dolphins: Multimodal Language Model for Driving

Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent

arXiv

Adept traffic models are critical to both planning and closed-loop simulation for autonomous vehicles (AV), and key design objectives …
PDF
Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent

EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

arXiv

We present EmerNeRF, a simple yet powerful approach for learning spatial-temporal representations of dynamic driving scenes. Grounded …
PDF Code Dataset Project
EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

Partial-View Object View Synthesis via Filtered Inversion

3DV 2024 (Spotlight)

We propose Filtering Inversion (FINV), a learning framework and optimization process that predicts a renderable 3D object …
PDF Code Project
Partial-View Object View Synthesis via Filtered Inversion

trajdata: A Unified Interface to Multiple Human Trajectory Datasets

NeurIPS 2023 Datasets and Benchmarks Track

The field of trajectory forecasting has grown significantly in recent years, partially owing to the release of numerous large-scale, …
PDF Code
trajdata: A Unified Interface to Multiple Human Trajectory Datasets

PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction

NeurIPS 2023

Inductive Conformal Prediction (ICP) provides a practical and effective approach for equipping deep learning models with uncertainty …
PDF
PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction

Language-Guided Traffic Simulation via Scene-Level Diffusion

CoRL 2023 (Oral)

Realistic and controllable traffic simulation is a core capability that is necessary to accelerate autonomous vehicle (AV) development. …
PDF
Language-Guided Traffic Simulation via Scene-Level Diffusion

Language Conditioned Traffic Generation

CoRL 2023

Simulation forms the backbone of modern self-driving development. Simulators help develop, test, and improve driving systems without …
PDF Code Project Video
Language Conditioned Traffic Generation

Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictors

CoRL 2023

Trajectory prediction modules are key enablers for safe and efficient planning of autonomous vehicles (AVs), particularly in highly …
PDF
Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictors

DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving

arXiv

Motion prediction and cost evaluation are vital components in the decision-making system of autonomous vehicles. However, existing …
PDF
DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving

Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Maps

arXiv

Autonomous driving has traditionally relied heavily on costly and labor-intensive High Definition (HD) maps, hindering scalability. In …
PDF
Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Maps

Interactive Joint Planning for Autonomous Vehicles

arXiv

In highly interactive driving scenarios, the actions of one agent greatly influences those of its neighbors. Planning safe motions for …
PDF
Interactive Joint Planning for Autonomous Vehicles

Reinforcement Learning with Human Feedback for Realistic Traffic Simulation

arXiv

In light of the challenges and costs of real-world testing, autonomous vehicle developers often rely on testing in simulation for the …
PDF
Reinforcement Learning with Human Feedback for Realistic Traffic Simulation

Verification and Synthesis of Robust Control Barrier Functions: Multilevel Polynomial Optimization and Semidefinite Relaxation

CDC 2023

We study the problem of verification and synthesis of robust control barrier functions (CBF) for control-affine polynomial systems with …
PDF
Verification and Synthesis of Robust Control Barrier Functions: Multilevel Polynomial Optimization and Semidefinite Relaxation

Interpretable Trajectory Prediction for Autonomous Vehicles via Counterfactual Responsibility

IROS 2023

The ability to anticipate surrounding agents' behaviors is critical to enable safe and seamless autonomous vehicles (AVs). While …
Project
Interpretable Trajectory Prediction for Autonomous Vehicles via Counterfactual Responsibility

Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models

ICML 2023

Recently, reward-conditioned reinforcement learning (RCRL) has gained popularity due to its simplicity, flexibility, and off-policy …
PDF
Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models

Task-Aware Risk Estimation of Perception Failures for Autonomous Vehicles

RSS 2023

Safety and performance are key enablers for autonomous driving: on the one hand we want our autonomous vehicles (AVs) to be safe, while …
PDF Code
Task-Aware Risk Estimation of Perception Failures for Autonomous Vehicles

Refining Obstacle Perception Safety Zones via Maneuver-Based Decomposition

IV 2023

A critical task for developing safe autonomous driving stacks is to determine whether an obstacle is safety-critical, i.e., poses an …
PDF
Refining Obstacle Perception Safety Zones via Maneuver-Based Decomposition

FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization

CVPR 2023

Novel view synthesis with sparse inputs is a challenging problem for neural radiance fields (NeRF). Recent efforts alleviate this …
PDF Code Project
FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization

Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation

CVPR 2023 (Highlight)

The two-stage object pose estimation paradigm first detects semantic keypoints on the image and then estimates the 6D pose by …
PDF
Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation

Learning Autonomous Vehicle Safety Concepts from Demonstrations

ACC 2023

Evaluating the safety of an autonomous vehicle (AV) depends on the behavior of surrounding agents which can be heavily influenced by …
PDF
Learning Autonomous Vehicle Safety Concepts from Demonstrations

BITS: Bi-level Imitation for Traffic Simulation

ICRA 2023

Simulation is the key to scaling up validation and verification for robotic systems such as autonomous vehicles. Despite advances in …
PDF
BITS: Bi-level Imitation for Traffic Simulation

Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning

ICRA 2023

Learning-based behavior prediction methods are increasingly being deployed in real-world autonomous systems, e.g., in fleets of …
PDF Code
Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning

Guided Conditional Diffusion for Controllable Traffic Simulation

ICRA 2023

Controllable and realistic traffic simulation is critical for developing and verifying autonomous vehicles. Typical heuristic-based …
PDF Project
Guided Conditional Diffusion for Controllable Traffic Simulation

Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models

ICRA 2023

Reasoning with occluded traffic agents is a significant open challenge for planning for autonomous vehicles. Recent deep learning …
PDF
Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models

Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles

ICRA 2023

Autonomous vehicles must often contend with conflicting planning requirements, e.g., safety and comfort could be at odds with each …
PDF
Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles

Tree-structured Policy Planning with Learned Behavior Models

ICRA 2023

Autonomous vehicles (AVs) need to reason about the multimodal behavior of neighboring agents while planning their own motion. Many …
PDF
Tree-structured Policy Planning with Learned Behavior Models

Robust and Controllable Object-Centric Learning through Energy-based Models

ICLR 2023

Humans are remarkably good at understanding and reasoning about complex visual scenes. The capability to decompose low-level …
PDF
Robust and Controllable Object-Centric Learning through Energy-based Models

Robust Trajectory Prediction against Adversarial Attacks

CoRL 2022 (Oral)

Trajectory prediction using deep neural networks (DNNs) is an essential component of autonomous driving (AD) systems. However, these …
PDF Code Project
Robust Trajectory Prediction against Adversarial Attacks

DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles

CoRL 2022

Autonomous vehicle (AV) stacks are typically built in a modular fashion, with explicit components performing detection, tracking, …
PDF Code Project
DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles

Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles

CoRL 2022

In modern autonomy stacks, prediction modules are paramount to planning motions in the presence of other mobile agents. However, …
PDF
Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles

AdvDO: Realistic Adversarial Attacks for Trajectory Prediction

ECCV 2022

Trajectory prediction is essential for autonomous vehicles (AVs) to plan correct and safe driving behaviors. While many prior works aim …
PDF
AdvDO: Realistic Adversarial Attacks for Trajectory Prediction

Towards Data-Driven Synthesis of Autonomous Vehicle Safety Concepts

arXiv

As safety-critical autonomous vehicles (AVs) will soon become pervasive in our society, a number of safety concepts for trusted AV …
PDF
Towards Data-Driven Synthesis of Autonomous Vehicle Safety Concepts

Propagating State Uncertainty Through Trajectory Forecasting

ICRA 2022

Uncertainty pervades through the modern robotic autonomy stack, with nearly every component (e.g., sensors, detection, classification, …
PDF Video
Propagating State Uncertainty Through Trajectory Forecasting

ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning

CVPR 2022

Trajectory prediction is a critical functionality of autonomous systems that share environments with uncontrolled agents, one prominent …
PDF Code
ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning

Whose Track Is It Anyway? Improving Robustness to Tracking Errors with Affinity-Based Prediction

CVPR 2022

Multi-agent trajectory prediction is critical for planning and decision-making in human-interactive autonomous systems, such as …
PDF Project
Whose Track Is It Anyway? Improving Robustness to Tracking Errors with Affinity-Based Prediction

Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation

IV 2022

To enable safe autonomous vehicle (AV) operations, it is critical that an AV’s obstacle detection module can reliably detect …
PDF
Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation

MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation

IV 2022

Recently, there has been tremendous progress in developing each individual module of the standard perception-planning robot autonomy …
PDF Project Video
MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation

Injecting Planning-Awareness into Prediction and Detection Evaluation

IV 2022

Detecting other agents and forecasting their behavior is an integral part of the modern robotic autonomy stack, especially in …
PDF Video
Injecting Planning-Awareness into Prediction and Detection Evaluation

Rethinking Trajectory Forecasting Evaluation

arXiv

Forecasting the behavior of other agents is an integral part of the modern robotic autonomy stack, especially in safety-critical …
PDF
Rethinking Trajectory Forecasting Evaluation