World Simulation - NVIDIA SIL

Overview

We develop simulation-first methodologies that enable intelligent agents to safely learn, plan, and act in rich, physically realistic worlds. Our work spans three complementary areas: world modeling, autonomous vehicle agents, and humanoid agents. Together, these efforts advance robust decision making, control, and interactive intelligence across domains.

Focus Areas

World modeling • Policy learning • Physics-based control • Evaluation at scale

World Modeling

We study models that learn predictive, controllable representations of the world across space and time. This includes video/3D‑aware diffusion world models, temporal reasoning for physically consistent edits, controllable multi‑view generation for scalable synthetic data, and inverse/forward rendering to connect generative models with scene geometry and lighting.

Points of Contact: Zian Wang, Jun Gao

OmniDreams

Real-time generative closed-loop autonomous vehicle simulation built on NVIDIA Cosmos.

ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation

Reframes image editing as video generation with temporal reasoning tokens for physically consistent edits.

Cosmos-Drive-Dreams

Scalable synthetic driving data with controllable, multi‑view, temporally consistent videos and LiDAR from world models.

DiffusionRenderer

Neural inverse and forward rendering with video diffusion models to couple generative models with scene geometry and lighting.

WeatherWeaver

Controllable manipulation of weather and scene conditions in videos while preserving physical and spatial coherence.

Resources & Publications

Autonomous Vehicle Agents

We investigate learned world models, policy learning, and closed-loop simulation for autonomous driving. This thrust focuses on training and evaluating AV agents with realistic sensor models, traffic scenarios, and safety-critical behaviors.

Points of Contact: Jose Alvarez, Seung Wook Kim, Xiuming Zhang

Hydra-MDP

End-to-end multimodal planner via multi-teacher distillation and multihead trajectories; NaviSim winner.

GTRS

Unified trajectory scoring with diffusion proposals and vocabulary generalization.

Resources & Publications

Humanoid Agents

We build physically simulated humanoids capable of robust, versatile behavior, leveraging reinforcement learning and generative modeling. Our research targets skill acquisition, imitation, and interactive control for human-centered applications.

Points of Contact: Jason Peng, Yifeng Jiang, Davis Rempe

ProtoMotions

Training physically simulated humanoids to reproduce a broad repertoire of human motor skills.