NVIDIA Seattle Robotics Lab

At the NVIDIA Seattle Robotics Lab, we strive to fulfill NVIDIA’s robotics mission: developing the essential technology that can enable any company to become a robotics company. We conduct fundamental and applied robotics research across the full robotics stack, including perception, planning, control, reinforcement learning, imitation learning, simulation, and vision-language-action models. Through our work, we aim to transform research paradigms, transfer technology into NVIDIA’s robotics and simulation products, and create new robotics markets for the world.

Research Areas

Perception

Perception is essential for robots to understand the world. Our lab has advanced vision-based policy learning through techniques such as Perceiver-Actor, RVT, and RVT-2, as well as multimodal sensor simulation through simulation and learning frameworks such as TacSL. A major challenge ahead is scaling visuomotor policies to open-world environments with diverse objects and dynamic scenes, and integrating multimodal sensing (vision, touch, force-torque) to achieve robust, dexterous manipulation.

Task and Motion Planning

Task and motion planning (TAMP) enables robots to coordinate and sequence high-level, long-horizon goals with low-level, short-horizon skills. We have developed GPU-accelerated planners such as cuTAMP and cuRobo to rapidly solve complex, long-horizon tasks and generate collision-free motion plans. Key challenges ahead include achieving real-time TAMP in dynamic, partially observable environments and extending planning algorithms to reason about contact-rich, deformable, and uncertain settings.

Control

Control allows robots to satisfy pose, velocity, and force targets with high precision, robustness, and safety. Our team is developing high-performance control libraries for manipulators and humanoids, and we have explored applying these algorithms to settings involving real-world reinforcement learning and high-precision assembly. An open challenge is developing controllers that unify model-based precision and learning-based adaptability, enabling robust performance across varying tasks and environments, especially in contact-rich and safety-critical applications.

Reinforcement Learning

Reinforcement learning (RL) empowers robots to learn complex skills through trial and error, reducing human effort and enabling superhuman performance. We explore simulation-driven model-based and model-free RL for robotics through works such as SHAC and IndustReal, as well as principled domain randomization approaches such as BayesSim. A major challenge ahead is bridging the sim-to-real gap by combining scalable simulation-driven RL with efficient real-world adaptation, and incorporating priors and constraints to ensure safety and generalization.

Imitation Learning

Imitation learning enables robots to rapidly acquire new skills quickly by learning from human demonstrations. Our research has focused on automated data generation through frameworks such as MimicGen, SkillMimicGen, and DexMimicGen, as well as developing hierarchical vision-language-action (VLA) models such as HAMSTER that can learn from off-domain data. Opportunities ahead include generating diverse, high-fidelity datasets for contact-rich and long-horizon tasks, and advancing beyond current behavior cloning paradigms (e.g., diffusion models) to methods capable of structured, hierarchical skill acquisition.

Simulation

Simulation enables large-scale training, testing, and benchmarking in virtual environments. We have developed GPU-accelerated simulation frameworks like Factory, differentiable simulators like DiSECT, and sim-to-real frameworks like AutoMate and FORGE for contact-rich tasks. Grand challenges ahead include improving physical realism (e.g., contact interactions, deformables, fluids) while maintaining GPU-accelerated performance, and closing the sim-to-real loop through online real-world feedback and continual refinement.

Human-Robot Interaction

Human-Robot Interaction (HRI) is critical for building robots that safely and intuitively collaborate with people in shared environments. Our research has included model-predictive control for fluent human-robot handovers, as well as ITPS for real-time policy steering. Major opportunities ahead involve leveraging foundation models and multimodal learning for intuitive, context-aware interaction, and enabling fluid, anticipatory collaboration with humans in industrial and household settings.

Intern Alumni

Discover our talented former interns and their contributions to robotics research.

View Intern Alumni

Publications

Quickly discover relevant content by filtering publications.

Clemens Eppner, Adithyavairavan Murali, Caelan Garrett, Rowland O'Flaherty, Tucker Hermans, Wei Yang, Dieter Fox

November, 2025 JOSS

scene_synthesizer: A Python Library for Procedural Scene Generation in Robot Manipulation

Shengyi Qian, Kaichun Mo, Valts Blukis, David Fouhey, Dieter Fox, Ankit Goyal

June, 2025 Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)

3D-MVP: 3D Multiview Pretraining for Robotic Manipulation

William Shen, Caelan Reed Garrett, Nishanth Kumar, Ankit Goyal, Tucker Hermans, Leslie Pack Kaelbling, Tomás Lozano-Pérez, Fabio Ramos

June, 2025 Proceedings of Robotics: Science and Systems (RSS)

Differentiable GPU-Parallelized Task and Motion Planning

Abhiram Maddukuri*, Zhenyu Jiang, Lawrence Yunliang Chen*, Soroush Nasiriany*, Yuqi Xie, Yu Fang, Wenqi Huang, Zu Wang, Zhenjia Xu, Nikita Chernyadev, Scott Reed, Ken Goldberg, Ajay Mandlekar, Linxi Fan*, Yuke Zhu

June, 2025 Proceedings of Robotics: Science and Systems (RSS)

Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation

Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan*, Yuke Zhu