Robotic Perception and Learning Research

Team

Stan Birchfield

Senior Research Manager

Balakumar Sundaralingam

Senior Research Scientist

Bowen Wen

Senior Research Scientist

Erwin Coumans

Senior Distinguished Engineer

Jeff Smith

Senior Research Engineer

Jonathan Tremblay

Senior Research Scientist

Mikaela Angelina Uy

Research Scientist

Stephen Tyree

Research Scientist

Valts Blukis

Senior Research Scientist

Publications

• BOP-ASK: Object-Interaction Reasoning for Vision-Language Models
  Vineet Bhat, Sungsu Kim, Valts Blukis, Greg Heinrich, Prashanth Krishnamurthy, Ramesh Karri, Stan Birchfield, Farshad Khorrami, Jonathan Tremblay
  CVPR
• CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction
  Xianghui Xie, Bowen Wen, Yan Chang, Hesam Rabeti, Jiefeng Li, Ye Yuan, Gerard Pons-Moll, Stan Birchfield
  CVPR
• Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching
  Bowen Wen, Shaurya Rajat Dewan, Stan Birchfield
  CVPR
• Grasp-MPC: Closed-Loop Visual Grasping via Value-Guided Model Predictive Control
  Jun Yamada, Adithyavairavan Murali, Ajay Mandlekar, Clemens Eppner, Ingmar Posner, Balakumar Sundaralingam
  ICRA
• GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping
  Beining Han, Yu-Wei Chao, Erwin Coumans, Clemens Eppner, Jia Deng, Stan Birchfield, Adithyavairavan Murali
  CVPR
• GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training
  Adithyavairavan Murali, Balakumar Sundaralingam, Yu-Wei Chao, Wentao Yuan, Jun Yamada, Mark Carlson, Fabio Ramos, Stan Birchfield, Dieter Fox, Clemens Eppner
  ICRA
• L4P: Low-Level 4D Vision Perception Unified
  Abhishek Badki, Hang Su, Bowen Wen, Orazio Gallo
  3DV
• Refinery: Active Fine-tuning and Deployment-time Optimization for Contact-Rich Policies
  Bingjie Tang, Iretiayo Akinola, Jie Xu, Bowen Wen, Michael Andres Lin, Dieter Fox, Gaurav S. Sukhatme, Fabio Ramos, Abhishek Gupta, Yashraj Narang
  ICRA
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
  Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, Jonathan Tremblay
  RSS
• SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL
  Siyi Chen, Mikaela Angelina Uy, Chan Hee Song, Faisal Ladhak, Adithyavairavan Murali, Qing Qu, Stan Birchfield, Valts Blukis, Jonathan Tremblay
  CVPR

• 3D-MVP: 3D Multiview Pretraining for Robotic Manipulation
  Shengyi Qian, Kaichun Mo, Valts Blukis, David Fouhey, Dieter Fox, Ankit Goyal
  CVPR
• Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learning
  Neel Anand Jawale, Byron Boots, Balakumar Sundaralingam, Mohak Bhardwaj
  ICRA
• FoundationStereo: Zero-Shot Stereo Matching
  Bowen Wen, Matthew Trepte, Oluwaseun Joseph Aribido, Jan Kautz, Orazio Gallo, Stan Birchfield
  CVPR
• HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction
  Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, Yu Xiang
  NeurIPS
• Inference-Time Policy Steering with Human Interactions
  Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Pérez-D'Arpino, Dieter Fox, Julie A. Shah
  ICRA
• RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion
  Bardienus Pieter Duisterhof, Jan Oberst, Bowen Wen, Stan Birchfield, Deva Ramanan, Jeffrey Ichnowski
  NeurIPS
• RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies
  Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, Artur Kuramshin, Clemens Eppner, Cyrus Neary, Edward Hu, Fabio Ramos, Jonathan Tremblay, Kanav Arora, Kirsty Ellis, Luca Macesanu, Matthew Leonard, Meedeum Cho, Ozgur Aslan, Shivin Dass, Jie Wang, Xingfang Yuan, Xuning Yang, Abhishek Gupta, Dinesh Jayaraman, Glen Berseth, Kostas Daniilidis, Roberto Martin-Martin, Youngwoon Lee, Percy Liang, Chelsea Finn, Sergey Levine
  CoRL
• RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
  Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, Stan Birchfield
  CVPR
• Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces
  Mauro Comi, Alessio Tonioni, Max Yang, Jonathan Tremblay, Valts Blukis, Yijiong Lin, Nathan F. Lepora, Laurence Aitchison
  3DV
• SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation
  Cheng-Chun Hsu, Bowen Wen, Jie Xu, Yashraj Narang, Xiaolong Wang, Yuke Zhu, Joydeep Biswas, Stan Birchfield
  ICRA

• AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries
  Bingjie Tang, Iretiayo Akinola, Jie Xu, Bowen Wen, Ankur Handa, Karl Van Wyk, Dieter Fox, Gaurav S. Sukhatme, Fabio Ramos, Yashraj Narang
  RSS
• Avoid Everything: Model-Free Collision Avoidance with Expert-Guided Fine-Tuning
  Adam Fishman, Aaron Walsman, Mohak Bhardwaj, Wentao Yuan, Balakumar Sundaralingam, Byron Boots, Dieter Fox
  CoRL
• DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation
  Bardienus P. Duisterhof, Mandi Zhao, Yunchao Yao, Jia-Wei Liu, Jenny Seidenschwarz, Mike Zheng Shou, Deva Ramanan, Shuran Song, Stan Birchfield, Bowen Wen, Jeffrey Ichnowski
  WAFR
• DiffusionSeeder: Seeding Motion Optimization with Diffusion for Rapid Motion Planning
  Huang Huang, Balakumar Sundaralingam, Arsalan Mousavian, Adithyavairavan Murali, Ken Goldberg, Dieter Fox
  CoRL
• FACTORSIM: Generative Simulation via Factorized Representation
  Fan-Yun Sun, Harini S I, Angela Yi, Yihan Zhou, Alex Zook, Jonathan Tremblay, Logan Cross, Jiajun Wu, Nick Haber
  NeurIPS
• FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
  Bowen Wen, Wei Yang, Jan Kautz, Stan Birchfield
  CVPR
• NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows
  Zhenggang Tang, Zhongzheng Ren, Xiaoming Zhao, Bowen Wen, Jonathan Tremblay, Stan Birchfield, Alex Schwing
  CVPR
• Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects
  Yijia Weng, Bowen Wen, Jonathan Tremblay, Valts Blukis, Dieter Fox, Leonidas Guibas, Stan Birchfield
  CVPR
• Partial-View Object View Synthesis via Filtering Inversion
  Fan-Yun Sun, Jonathan Tremblay, Valts Blukis, Kevin Lin, Danfei Xu, Boris Ivanovic, Peter Karkus, Stan Birchfield, Dieter Fox, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Marco Pavone, Nick Haber
  3DV
• RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
  Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, Dieter Fox
  CoRL
• RVT-2: Learning Precise Manipulation from Few Demonstrations
  Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, Dieter Fox
  RSS
• SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment
  Caelan Reed Garrett, Ajay Mandlekar, Bowen Wen, Dieter Fox
  CoRL

• Affordance Diffusion: Synthesizing Hand-Object Interactions
  Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, Sifei Liu
  CVPR
• BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
  Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, Stan Birchfield
  CVPR
• HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions
  Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, Stan Birchfield
  IROS
• MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
  Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox
  CoRL
• One-Shot Neural Fields for 3D Object Understanding
  Valts Blukis, Taeyeop Lee, Jonathan Tremblay, Bowen Wen, In So Kweon, Kuk-Jin Yoon, Dieter Fox, Stan Birchfield
  CVPR Workshop on Advances in NeRF for the Metaverse (XRNeRF)
• Parallel Inversion of Neural Radiance Fields for Robust Pose Estimation
  Yunzhi Lin, Thomas Muller, Jonathan Tremblay, Bowen Wen, Stephen Tyree, Alex Evans, Patricio A. Vela, Stan Birchfield
  ICRA
• PROGPROMPT: Generating Situated Robot Task Plans using Large Language Models
  Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg
  ICRA
• RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control
  Zhenggang Tang, Balakumar Sundaralingam, Jonathan Tremblay, Bowen Wen, Ye Yuan, Stephen Tyree, Charles Loop, Alexander Schwing, Stan Birchfield
  ICRA
• TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation
  Taeyeop Lee, Jonathan Tremblay, Valts Blukis, Bowen Wen, Byeong-Uk Lee, Inkyu Shin, Stan Birchfield, In So Kweon, Kuk-Jin Yoon
  CVPR
• Vicinity Vision Transformer
  Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, Yiran Zhong
  TPAMI

• 6-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark
  Stephen Tyree, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Jeffrey Smith, Stan Birchfield
  IROS
• Audio-Visual Segmentation
  Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, Yiran Zhong
  ECCV
• Displacement-Invariant Cost Computation for Efficient Stereo Matching
  Yiran Zhong, Charles Loop, Wonmin Byeon, Stan Birchfield, Yuchao Dai, Kaihao Zhang, Alexey Kamenev, Thomas Breuel, Hongdong Li, Jan Kautz
  IJCV
• Efficient Geometry-aware 3D Generative Adversarial Networks
  Eric Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein
  CVPR
• Keypoint-Based Category-Level Object Pose Tracking from an RGB Sequence with Uncertainty Estimation
  Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
  ICRA
• MegaPose: 6D Pose Estimation of Novel Objects via Render and Compare
  Yann Labbè, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, Josef Sivic
  CoRL
• PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation
  Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, Nikolai Smolyanskiy
  ICRA
• RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis
  Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Thomas Müller, Charles Loop, Nathan Morrical, Koki Nagano, Towaki Takikawa, Stan Birchfield
  ECCV Workshop on Learning to Generate 3D Shapes and Scenes
• Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image
  Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
  ICRA
• Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects
  Atsuhiro Noguchi, Umar Iqbal, Jonathan Tremblay, Tatsuya Harada, Orazio Gallo
  CVPR

• Deep Two-View Structure-from-Motion Revisited
  Jianyuan Wang, Yiran Zhong, Yuchao Dai, Stan Birchfield, Kaihao Zhang, Nikolai Smolyanskiy, Hongdong Li
  CVPR
• DexYCB: A Benchmark for Capturing Hand Grasping of Objects
  Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, Dieter Fox
  CVPR
• Fast Uncertainty Quantification for Deep Object Pose Estimation
  Guanya Shi, Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Fabio Ramos, Animashree Anandkumar, Yuke Zhu
  ICRA
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
  Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Yuke Zhu
  ICRA
• Joint Space Control via Deep Reinforcement Learning
  Visak Kumar, David Hoeller, Balakumar Sundaralingam, Jonathan Tremblay, Stan Birchfield
  IROS
• Multi-View Fusion for Multi-Level Robotic Scene Understanding
  Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
  IROS
• NViSII: A Scriptable Tool for Photorealistic Image Generation
  Nathan Morrical, Jonathan Tremblay, Yunzhi Lin, Stephen Tyree, Stan Birchfield, Valerio Pascucci, Ingo Wald
  ICLR Workshop on Synthetic Data Generation
• RMPflow: A Geometric Framework for Generation of Multi-Task Motion Policies
  Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff
  TASE
• Self-Supervised Real-to-Sim Scene Generation
  Aayush Prakash, Shoubhik Debnath, Jean-Francois Lafleche, Eric Cameracci, Gavriel State, Stan Birchfield, Marc T. Law
  ICCV

• Camera-to-robot pose estimation from a single image
  Timothy E Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, Stan Birchfield
  ICRA
• DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
  Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox
  ICRA
• Indirect Object-to-Robot Pose Estimation from an External Monocular RGB Camera
  Jonathan Tremblay, Stephen Tyree, Terry Mosier, Stan Birchfield
  IROS
• MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
  Ke Chen, Ryan Oldja, Nikolai Smolyanskiy, Stan Birchfield, Alexander Popov, David Wehr, Ibrahim Eden, Joachim Pehserl
  IROS
• Toward Sim-to-Real Directional Semantic Grasping
  Shariq Iqbal, Jonathan Tremblay, Thang To, Jia Cheng, Erik Leitch, Andy Campbell, Kirby Leung, Duncan McKay, Stan Birchfield
  ICRA