Evaluating and improving planning for autonomous vehicles requires scalable generation of
long-tail traffic scenarios. To be useful, these scenarios must be realistic and challenging,
but not impossible to drive through safely. In this work, we introduce STRIVE, a method to
automatically generate challenging scenarios that cause a given planner to produce undesirable
behavior, like collisions. To maintain scenario plausibility, the key idea is to leverage a
learned model of traffic motion in the form of a graph-based conditional VAE. Scenario generation
is formulated as an optimization in the latent space of this traffic model, perturbing
an initial real-world scene to produce trajectories that collide with a given planner. A subsequent
optimization is used to find a "solution" to the scenario, ensuring it is useful to improve the given
planner. Further analysis clusters generated scenarios based on collision type. We attack two planners
and show that STRIVE successfully generates realistic, challenging scenarios in both cases. We additionally
"close the loop" and use these scenarios to optimize hyperparameters of a rule-based planner.
Scenario Generation for Rule-based Planner
From an initial nuScenes scenario (left) STRIVE uses an adversarial optimization to attack a
given planner (middle).
Furthermore, a solution optimization (right) ensures the generated scenario is "solvable" and
thus useful for downstream tasks.
In all examples, the planner and
adversary are colored accordingly.
Original replay
STRIVE attack
STRIVE solution
Original replay
STRIVE attack
STRIVE solution
Original replay
STRIVE attack
STRIVE solution
Original replay
STRIVE attack
STRIVE solution
Collision Scenario Clusters
Generated collision scenarios are clustered and labeled based on the relative position
and heading of the adversary and planner at the time of collision. Representative examples
from these clusters are shown below.
T-Bone Right
Merge Right
Head On
Front Right
Cutoff Right
Behind
T-Bone Left
Merge Left
Front Left
Adversarial Optimization Comparison on Replay Planner
Qualitative examples of scenarios generated by STRIVE compared to the Bicycle baseline which does not use the learned traffic model in any way.
In each example, the original scene used for initialization is shown on the left followed by the output of Bicycle adversarial optimization and ours.
Note the unrealistic attacks containing sharp turns from Bicycle.
Original replay
Bicycle attack
STRIVE attack
The adversary turns sharply to block the planner in a seemingly malicious fashion.
The adversary suddenly changes lanes in a more realistic way, since STRIVE uses the learned prior.
Original replay
Bicycle attack
STRIVE attack
Original replay
Bicycle attack
STRIVE attack
The adversary unnecessarily loops around to collide, since only its dynamics are regularized in optimization.
The adversary instead slows down to cause a rear-end collision, which is more likely under the prior.
Original replay
Bicycle attack
STRIVE attack
Our adversarial optimization chooses a different vehicle to collide than the heuristically pre-chosen attacker of Bicycle, resulting in a more plausible collision.
Second-order Effects
One interesting ability of adversarial optimization is to produce scenarios that cause collisions with the planner using
so-called ``second-order effects", where multiple adversaries act in conjunction to cause a collision.
In the specific examples below, the planner is obstructed by one adversary, causing the planner to slam on its brakes and get hit by a different vehicle.
Original replay
STRIVE attack
Original replay
STRIVE attack
Original replay
STRIVE attack
Pedestrian and Cyclist Adversaries
We train our traffic model on all categories in the nuScenes dataset and
use it to generate scenarios for the Replay planner. In the examples below,
a specific pedestrian or cyclist adversary is chosen before optimization to cause the
collision.
Pedestrian adversary
Pedestrian adversary
Cyclist adversary
STRIVE in 3D Simulation
We used STRIVE within NVIDIA's DRIVE Sim simulator to generate multiple collision scenarios based
on a scene reconstructed from a real-world video. For more details, see the GTC keynote talk.
Also, check out NVIDIA Drive Labs Ep. 27, which features STRIVE and gives a breakdown of how the method can be used to generate many accident variations of a real-world traffic scenario.
Citation
@inproceedings{rempe2022strive,
author={Rempe, Davis and Philion, Jonah and Guibas, Leonidas J. and Fidler, Sanja and Litany, Or},
title={Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior},
booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2022}
}