Research Areas

Robotics

Associated Publications

2022

Learning Robust Real-World Dexterous Grasping Policies via Implicit Shape Augmentation
Qiuyu Chen, Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, Dieter Fox
The Conference on Robot Learning (CoRL) 2022
MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
Yann Labbe, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, et al.
CoRL 2022
6-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark
Stephen Tyree, Jonathan Tremblay, Stan Birchfield, et al.
IROS 2022
Learning Perceptual Concepts by Bootstrapping from Human Queries
Andreea Bobu, Chris Paxton, Wei Yang, Balakumar Sundaralingam, Yu-Wei Chao, Maya Cakmak, Dieter Fox
RA-L / IROS 2022
Correcting Robot Plans with Natural Language Feedback
Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, Dieter Fox
Robotics Science and Systems (RSS) 2022
Ifor: Iterative flow minimization for robotic object rearrangement
Ankit Goyal, Arsalan Mousavian, Chris Paxton, Yu-Wei Chao, Brian Okorn, Jia Deng, Dieter Fox
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022
HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers
Yu-Wei Chao, Chris Paxton, Yu Xiang, Wei Yang, Balakumar Sundaralingam, Tao Chen, Adithya Murali, Maya Cakmak, Dieter Fox
IEEE International Conference on Robotics and Automation (ICRA) 2022
Accelerated Policy Learning with Parallel Differentiable Simulation
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, Miles Macklin
International Conference on Learning Representations (ICLR) 2022
Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, Kris Kitani
International Conference on Learning Representations (ICLR) 2022 (Oral)
Model Predictive Control for Fluid Human-to-Robot Handovers
Wei Yang, Balakumar Sundaralingam, Chris Paxton, Iretiayo Akinola, Yu-Wei Chao, Maya Cakmak, Dieter Fox
IEEE International Conference on Robotics and Automation (ICRA) 2022
DefGraspSim: Physics-Based Simulation of Grasp Outcomes on 3D Deformable Objects
Isabella Huang, Yashraj Narang, Clemens Eppner, Balakumar Sundaralingam, Miles Macklin, Ruzena Bajcsy, Tucker Hermans, Dieter Fox
IEEE Robotics and Automation Letters
PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation
Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nister, Nikolai Smolyanskiy
ICRA 2022
Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
ICRA 2022
Keypoint-Based Category-Level Object Pose Tracking from an RGB Sequence with Uncertainty Estimation
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
ICRA 2022

2021

Interpreting and predicting tactile signals for the SynTouch BioTac
Yashraj Narang, Balakumar Sundaralingam, Karl Van Wyk, Arsalan Mousavian, Dieter Fox
International Journal of Robotics Research (IJRR) 2021
Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, Dieter Fox
Conference on Robot Learning 2021
STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation
Mohak Bhardwaj, Balakumar Sundaralingam, Arsalan Mousavian, Nathan Ratliff, Dieter Fox, Fabio Ramos, Byron Boots
Conference on Robot Learning (CoRL) 2021
Selected for Oral Presentation
Joint Space Control via Deep Reinforcement Learning
Visak Kumar, David Hoeller, Balakumar Sundaralingam, Jonathan Tremblay, Stan Birchfield
IROS 2021
Multi-View Fusion for Multi-Level Robotic Scene Understanding
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
IROS 2021
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, Dieter Fox
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021
Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections
Yashraj Narang, Balakumar Sundaralingam, Miles Macklin, Arsalan Mousavian, Dieter Fox
International Conference on Robotics and Automation (ICRA) 2021
Alternative Paths Planner (APP) for Provably Fixed-time Manipulation Planning in Semi-structured Environments
Fahad Islam, Chris Paxton, Clemens Eppner, Bryan Peele, Maxim Likhachev, Dieter Fox
2021 IEEE International Conference on Robotics and Automation (ICRA)
ACRONYM: A Large-Scale Grasp Dataset Based on Simulation
Clemens Eppner, Arsalan Mousavian, Dieter Fox
2021 IEEE International Conference on Robotics and Automation (ICRA)
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Yuke Zhu
ICRA 2021
Fast Uncertainty Quantification for Deep Object Pose Estimation
Guanya Shi, Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Fabio Ramos, Anima Anandkumar, Yuke Zhu
ICRA 2021
NViSII: A Scriptable Tool for Photorealistic Image Generation
Nathan Morrical, Jonathan Tremblay, Yunzhi Lin, Stephen Tyree, Stan Birchfield, Valerio Pascucci, Ingo Wald
SDG Workshop at ICLR 2021
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
Luyang Zhu, Arsalan Mousavian, Yu Xiang, Hammad Mazhar, Jozef van Eenbergen, Shoubhik Debnath, Dieter Fox
CVPR 2021
Reactive Human-to-Robot Handovers of Arbitrary Objects
Wei Yang, Chris Paxton, Arsalan Mousavian, Yu-Wei Chao, Maya Cakmak, Dieter Fox
ICRA 2021
Best Paper in Human-Robot Interaction, ICRA 2021
Object Rearrangement Using Learned Implicit Collision Functions
Michael Danielczuk, Arsalan Mousavian, Clemens Eppner, Dieter Fox
ICRA 2021
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes
Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter Fox
ICRA 2021
Transferable Task Execution from Pixels through Deep Planning Domain Learning
Kei Kase, Chris Paxton, Hammad Mazhar, Tetsuya Ogata, Dieter Fox
ICRA 2020

2020

Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies
Visak Kumar, Tucker Hermans, Dieter Fox, Stan Birchfield, Jonathan Tremblay
NeurIPS Workshop on Robot Learning
RMPflow: A Geometric Framework for Generation of Multi-Task Motion Policies
Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff
IEEE Transactions on Automation Science and Engineering (T-ASE)
RMPflow: A Computational Graph for Automatic Motion Policy Generation
Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff
International Workshop on the Algorithmic Foundations of Robotics (WAFR) 2018
Collaborative Interaction Models for Optimized Human-Robot Teamwork
Adam Fishman, Chris Paxton, Wei Yang, Nathan Ratliff, Byron Boots, Dieter Fox
International Conference on Intelligent Robots and Systems (IROS) 2020
Human Grasp Classification for Reactive Human-to-Robot Handovers
Wei Yang, Chris Paxton, Maya Cakmak, Dieter Fox
International Conference on Intelligent Robots and Systems (IROS), 2020
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer
Andrew Hundt, Benjamin Killeen, Nicholas Greene, Hongtao Wu, Heeyeon Kwon, Chris Paxton, Gregory D Hager
IEEE Robotics and Automation Letters
Weakly-Supervised 3D Hand Pose Estimation via Biomechanical Constraints
Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, Jan Kautz
European Conference on Computer Vision, 2020
OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
Hongyu Ren, Yuke Zhu, Jure Leskovec, Anima Anandkumar, Animesh Garg
Conference on Uncertainty in Artificial Intelligence (UAI) 2020
Indirect Object-to-Robot Pose Estimation from an External Monocular RGB Camera
Jonathan Tremblay, Stephen Tyree, Terry Mosier, Stan Birchfield
IROS 2020
Automated Synthetic-to-Real Generalization
Wuyang Chen, Zhiding Yu, Zhangyang Wang, Anima Anandkumar
International Conference on Machine Learning (ICML) 2020
MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
Ke Chen, Ryan, Nikolai, Stan Birchfield, Alexander Popov, David Wehr, Ibrahim Eden, Joachim Pehserl
IROS 2020
6-DOF Grasping for Target-driven Object Manipulation in Clutter
Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner, Chris Paxton, Dieter Fox
ICRA 2020
Best Paper Finalist in Robot Manipulation, ICRA 2020, Best Student Paper Finalist, ICRA 2020
Online replanning in belief space for partially observable task and motion problems
Caelan Reed Garrett, Chris Paxton, Tomas Lozano-Perez, Leslie Kaebling, Dieter Fox
ICRA 2020
Self-supervised 6D Object Pose Estimation for Robot Manipulation
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, Dieter Fox
2020 IEEE International Conference on Robotics and Automation (ICRA)
Motion reasoning for goal-based imitation learning
De-An Huang, Yu-Wei Chao, Chris Paxton, Xinke Deng, Li Fei-Fei, Juan Carlos Niebles, Animesh Garg, Dieter Fox
ICRA 2020
Camera-to-Robot Pose Estimation from a Single Image
Timothy E. Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, Stan Birchfield
ICRA 2020
How to close sim-real gap? transfer with segmentation!
Mengyuan Yan, Qingyun Sun, Iuri Frosio, Stephen Tyree, Jan Kautz
arxiv
Toward Sim-to-Real Directional Semantic Grasping
Shariq Iqbal, Jonathan Tremblay, Thang To, Jia Cheng, Erik Leitch, Andy Campbell, Kirby Leung, Duncan McKay, Stan Birchfield
ICRA 2020
DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox
ICRA 2020

2019

6-DOF GraspNet: Variational Grasp Generation for Object Manipulation
Arsalan Mousavian, Clemens Eppner, Dieter Fox
ICCV 2019
A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set
Clemens Eppner, Arsalan Mousavian, Dieter Fox
The International Symposium of Robotics Research (ISSR)
Representing robot task plans as robust logical-dynamical systems
Chris Paxton, Clemens Eppner, Nathan Ratliff, Dieter Fox
IROS 2019
Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments
Xueting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, Jan Kautz
CVPR 2019
Robust Learning of Tactile Force Estimation through Robot Interaction
Balakumar Sundaralingam, Alexander Lambert, Ankur Handa, Byron Boots, Tucker Hermans, Stan Birchfield, Nathan Ratliff, Dieter Fox
ICRA 2019

2018

Fluidic Elastomer Actuators for Haptic Interactions in Virtual Reality
Jose Barreiros, Houston Claure, Bryan Peele, Omer Shapira, Josef Spjut, David Luebke, Malte Jung, Robert Shepherd
IEEE Robotics and Automation Letters
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects
Jonathan Tremblay, Thang To, Bala Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield
Conference on Robot Learning (CoRL) 2018
Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli, Deqing Sun, James M. Rehg, Jan Kautz
European Conference on Computer Vision (ECCV 2018)
Simultaneous Edge Alignment and Learning
Zhiding Yu, Weiyang Liu, Yang Zou, Chen Feng, Srikumar Ramalingam, B. V. K. Vijaya Kumar, Jan Kautz
European Conference on Computer Vision (ECCV) 2018
HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration
Ben Eckart, Kihwan Kim, Jan Kautz
European Conference on Computer Vision (ECCV 2018)
Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, Jinsong Wang
European Conference on Computer Vision (ECCV) 2018
EOE: Expected Overlap Estimation over Unstructured Point Cloud Data
Ben Eckart, Kihwan Kim, Jan Kautz
International Conference on 3D Vision (3DV) 2018
Region Growing Curriculum Generation for Reinforcement Learning
Artem Molchanov, Karol Hausman, Stan Birchfield, Gaurav Sukhatme
arXiv
Geometry-Aware Learning of Maps for Camera Localization
Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz
CVPR 2018 (Spotlight)
Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation
Jonathan Tremblay, Thang To, Stan Birchfield
CVPR 2018 Workshop on Real World Challenges and New Benchmarks for Deep Learnin…
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox
Robotics: Science and Systems (RSS)
Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations
Jonathan Tremblay, Thang To, Artem Molchanov, Stephen Tyree, Jan Kautz, Stan Birchfield
IEEE International Conference on Robotics and Automation (ICRA) 2018
A Variable Shape and Variable Stiffness Controller for Haptic Virtual Interactions
Benjamin C. Mac Murray, Bryan N. Peele, Patricia Xu, Josef Spjut, Omer Shapira, David Luebke, Robert F. Shepherd
RoboSoft
On the Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach
Nikolai Smolyanskiy, Alexey Kamenev, Stan Birchfield
CVPR 2018 Workshop on Autonomous Driving
Riemannian Motion Policies
Nathan Ratliff, Jan Issac, Daniel Kappler, Stan Birchfield, Dieter Fox
arXiv

2017

Sim-to-Real Transfer of Accurate Grasping with Eye-In-Hand Observations and Continuous Control
Mengyuan Yan, Iuri Frosio, Stephen Tyree, Jan Kautz
NIPS 2017 Workshop on Acting and Interacting in the Real World: Challenges in …
Toward Low-Flying Autonomous MAV Trail Navigation using Deep Neural Networks for Environmental Awareness
Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith, Stan Birchfield
IROS 2017

2016

Accelerated Generative Models for 3D Point Cloud Data
Ben Eckart, Kihwan Kim, Alejandro Troccoli, Alonzo Kelly, Jan Kautz
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016

Researchers

Ajay Mandlekar
Animesh Garg
Ankit Goyal
Ankur Handa
Arsalan Mousavian
Balakumar Sundaralingam
Benjamin Eckart
Boris Ivanovic
Bowen Wen
Caelan Garrett
Chen-Hsuan Lin
Claudia Pérez D’Arpino
Clemens Eppner
Danfei Xu
De-An Huang
Fabio Ramos
Heng Yang
Iuri Frosio
Jeff Smith
Jie Xu
Linxi "Jim" Fan
Jonathan Tremblay
Kaichun Mo
Karen Leung
Li Xu
Marco Pavone
Peter Karkus
Rowland O'Flaherty
Stan Birchfield
Stephen Tyree
Sushant Veer
Tucker Hermans
Valts Blukis
Wei Yang
Wonmin Byeon
Xinshuo Weng
Xuning Yang
Ye Yuan
Yijie Guo
Yu-Wei Chao
Yue Wang
Yuke Zhu
Yuxiao Chen
Zhiding Yu