NVIDIA Spatial Intelligence Lab (SIL) NVIDIA Research

Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems

Tolga Dimlioglu1†,  Nadine Chang2,  Maying Shen2,  Rafid Mahmood2,3,  Jose M. Alvarez2
1New York University
2NVIDIA
3University of Ottawa
†Work completed during an internship at NVIDIA.
CVPR 2026
history_edu arXiv description Paper format_quote BibTeX
Teaser showing clustered driving domains, evaluation tasks, and context-specific scaling behavior.

Mixture Optimization via Scaling Aware Iterative Collection (MOSAIC) is a data mining strategy for complex unstructured data pools such as in autonomous driving. (a) The pool is partitioned into discrete domains. (b) Those domains can contribute to different evaluation tasks at different rates. (c) In autonomous driving, clusters can correspond to distinct contexts such as Pittsburgh's curvy suburban roads and Las Vegas's dense city traffic. (d) Because these contexts improve rule-compliance metrics differently, the best next sample depends on which cluster still offers the largest expected marginal gain.

61%
less data than naive baselines
42%
fewer samples to reach the performance of full training on Navtrain

Abstract

Large-scale deep learning models for physical AI applications depend on diverse training data collection efforts. These models and correspondingly, the training data, must address different evaluation criteria necessary for the models to be deployable in real-world environments. Data selection policies can guide the development of the training set, but current frameworks do not account for the ambiguity in how data points affect different metrics.

We propose Mixture Optimization via Scaling-Aware Iterative Collection (MOSAIC), a general data selection framework that operates by: (i) partitioning the dataset into domains; (ii) fitting neural scaling laws from each data domain to the evaluation metrics; and (iii) optimizing a data mixture by iteratively adding data from domains that maximize the change in metrics. We apply MOSAIC to autonomous driving, where an end-to-end planner model is evaluated on the Extended Predictive Driving Model Score (EPDMS). MOSAIC outperforms a diverse set of baselines on EPDMS with up to 80% less data.

Method Overview

MOSAIC turns dataset growth into a structured allocation over mixtures of data subsets that individually obey scaling laws. It groups the pool into domains that reflect different driving contexts, estimates how performance scales when clips are added from each domain, and then iteratively collects from the cluster with the largest predicted marginal gain.

MOSAIC pipeline showing clustering, scaling estimation, and scaling-aware iterative collection.

MOSAIC clusters the data pool, ranks samples within each cluster, fits cluster-wise scaling laws from pilot runs, and iteratively selects the next sample from the domain with the largest expected utility gain.

1. Cluster and rank the pool

Partition the available clips into structured domains that capture different driving contexts, then prioritize the most informative samples within each domain.

2. Fit cluster scaling laws

Estimate how much EPDMS improves as more data is added from each cluster, rather than assuming every source contributes at the same rate.

3. Select by marginal utility

At each step, collect from the cluster whose next clip is predicted to yield the strongest gain on the aggregate driving objective.

Results

We evaluate MOSAIC on OpenScene and Navtrain by clustering data from VLM-generated captions and training on Hydra-MDP. We compare against multiple baselines from scaling laws, mixtures, and active learning. Our results show: (a) MOSAIC consistently outperforms all baselines; behavior across clusters produces stronger EPDMS and substantially better data efficiency; (b) each of the components of clustering and ranking data within MOSAIC are necessary; and (c) MOSAIC can learn the impact of heterogenous data scaling at different budgets.

Caption-based clustering

Validation EPDMS versus number of clips for different selection methods.

Under caption-based clustering, MOSAIC remains strongest across budgets, requires 61% and 52% fewer clips than random selection to match its performance at 1,600 and 2,400 clips, and reaches full-training performance with 42% fewer samples.

Ablation

Ablation comparing MOSAIC against versions without clustering and without ranking.

Ranking helps in the low-data regime, but the full combination of clustering, ranking, and scaling-aware allocation is what sustains the lead as the collection budget grows.

What data domains improve training?

OpenScene scaling curves and resulting clip allocation ratios across city clusters.

In the geolocation-based OpenScene setting, Boston and Singapore provide the strongest gains early, Pittsburgh improves more steadily at larger budgets, and Las Vegas saturates earlier. MOSAIC exploits those heterogeneous curves to move the budget across domains as the marginal gains change.

BibTeX

@misc{dimlioglu2026scalingaware,
  title         = {Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems},
  author        = {Tolga Dimlioglu and Nadine Chang and Maying Shen and Rafid Mahmood and Jose M. Alvarez},
  year          = {2026},
  eprint        = {2604.08366},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2604.08366},
}