Research Areas

Robotics

Associated Publications

2023

Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planning
Zhutian Yang, Caelan Garrett, Tomás Lozano-Pérez, Leslie Pack Kaelbling, Dieter Fox
Robotics: Science and Systems (RSS)
StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects
Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, Chris Paxton
Robotics: Science and Systems (RSS) 2023
Learning Human-to-Robot Handovers from Point Clouds
Sammy Christen, Wei Yang, Claudia Pérez D’Arpino , Otmar Hilliges, Dieter Fox, Yu-Wei Chao
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023
Highlight
23 DoF Grasping Policies from a Raw Point Cloud
Martin Matak, Karl Van Wyk, Tucker Hermans
IEEE International Conference on Robotics and Automation (ICRA) Workshop on Geo…
Imitating Task and Motion Planning with Visuomotor Transformers
Murtaza Dalal, Ajay Mandlekar, Caelan Garrett, Ankur Handa, Ruslan Salakhutdinov, Dieter Fox
Workshop on Learning from Diverse, Offline Datasets at the IEEE International C…
Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers
Yixuan Huang, Adam Conkey, Tucker Hermans
IEEE International Conference on Robotics and Automation (ICRA)
DeXtreme: Transfer of Agile In-Hand Manipulation from Simulation to Reality
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Jean-Francois Lafleche, Dieter Fox, Gavriel State
https://arxiv.org/abs/2210.13702
Human-In-The-Loop Task and Motion Planning for Imitation Learning
Ajay Mandlekar, Caelan Garrett, Danfei Xu, Dieter Fox
Workshop on effective Representations, Abstractions, and Priors for Robot Learn…
FewSOL: A Dataset for Few-Shot Object Learning in Robotic Environments
Jishnu Jaykumar P, Yu-Wei Chao, Yu Xiang
IEEE International Conference on Robotics and Automation (ICRA) 2023
CuRobo: Parallelized Collision-Free Robot Motion Generation
Balakumar Sundaralingam, Siva Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, Nathan Ratliff, Dieter Fox
IEEE International Conference on Robotics and Automation (ICRA)
Global and Reactive Motion Generation with Geometric Fabric Command Sequences
Weiming Zhi, Iretiayo Akinola, Karl Van Wyk, Nathan Ratliff, Fabio Ramos
IEEE International Conference on Robotics and Automation (ICRA) 2023
Parallel Inversion of Neural Radiance Fields for Robust Pose Estimation
Yunzhi Lin, Thomas Müller, Jonathan Tremblay, Bowen Wen, Stephen Tyree, Alex Evans, Patricio A. Vela, Stan Birchfield
IEEE International Conference on Robotics and Automation (ICRA) 2023
ProgPrompt: Generating Situated Robot Task Plans Using Large Language Models
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg
The International Conference on Robotics and Automation (ICRA)
DefGraspNets: Grasp Planning on 3D Fields with Graph Neural Nets
Isabella Huang, Yashraj Narang, Ruzena Bajcsy, Fabio Ramos, Tucker Hermans, Dieter Fox
IEEE International Conference on Robotics and Automation (ICRA)
IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Reality
Bingjie Tang, Michael A. Lin, Iretiayo Akinola, Ankur Handa, Gaurav S. Sukhatme, Fabio Ramos, Dieter Fox, Yashraj Narang
Robotics: Science and Systems
CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generation
Adithya Murali, Arsalan Mousavian, Clemens Eppner, Adam Fishman, Dieter Fox
International Conference on Robotics and Automation (ICRA)

2022

Planning Visual-Tactile Precision Grasps via Complementary Use of Vision and Touch
Martin Matak, Tucker Hermans
IEEE Robotics and Automation Letters
Learning Robust Real-World Dexterous Grasping Policies via Implicit Shape Augmentation
Qiuyu Chen, Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, Dieter Fox
The Conference on Robot Learning (CoRL) 2022
MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
Yann Labbe, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, et al.
CoRL 2022
Motion Policy Networks
Adam Fishman, Adithya Murali, Clemens Eppner, Bryan Peele, Byron Boots, Dieter Fox
Conference on Robot Learning (CoRL), 2022
6-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark
Stephen Tyree, Jonathan Tremblay, Stan Birchfield, et al.
IROS 2022
Learning Perceptual Concepts by Bootstrapping from Human Queries
Andreea Bobu, Chris Paxton, Wei Yang, Balakumar Sundaralingam, Yu-Wei Chao, Maya Cakmak, Dieter Fox
RA-L / IROS 2022
Correcting Robot Plans with Natural Language Feedback
Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, Dieter Fox
Robotics Science and Systems (RSS) 2022
Ifor: Iterative flow minimization for robotic object rearrangement
Ankit Goyal, Arsalan Mousavian, Chris Paxton, Yu-Wei Chao, Brian Okorn, Jia Deng, Dieter Fox
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022
HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers
Yu-Wei Chao, Chris Paxton, Yu Xiang, Wei Yang, Balakumar Sundaralingam, Tao Chen, Adithya Murali, Maya Cakmak, Dieter Fox
IEEE International Conference on Robotics and Automation (ICRA) 2022
Accelerated Policy Learning with Parallel Differentiable Simulation
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, Miles Macklin
International Conference on Learning Representations (ICLR) 2022
Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, Kris Kitani
International Conference on Learning Representations (ICLR) 2022 (Oral)
Model Predictive Control for Fluid Human-to-Robot Handovers
Wei Yang, Balakumar Sundaralingam, Chris Paxton, Iretiayo Akinola, Yu-Wei Chao, Maya Cakmak, Dieter Fox
IEEE International Conference on Robotics and Automation (ICRA) 2022
DefGraspSim: Physics-Based Simulation of Grasp Outcomes on 3D Deformable Objects
Isabella Huang, Yashraj Narang, Clemens Eppner, Balakumar Sundaralingam, Miles Macklin, Ruzena Bajcsy, Tucker Hermans, Dieter Fox
IEEE Robotics and Automation Letters
PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation
Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nister, Nikolai Smolyanskiy
ICRA 2022
Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
ICRA 2022
Keypoint-Based Category-Level Object Pose Tracking from an RGB Sequence with Uncertainty Estimation
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
ICRA 2022

2021

Interpreting and predicting tactile signals for the SynTouch BioTac
Yashraj Narang, Balakumar Sundaralingam, Karl Van Wyk, Arsalan Mousavian, Dieter Fox
International Journal of Robotics Research (IJRR) 2021
Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, Dieter Fox
Conference on Robot Learning 2021
STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation
Mohak Bhardwaj, Balakumar Sundaralingam, Arsalan Mousavian, Nathan Ratliff, Dieter Fox, Fabio Ramos, Byron Boots
Conference on Robot Learning (CoRL) 2021
Selected for Oral Presentation
Joint Space Control via Deep Reinforcement Learning
Visak Kumar, David Hoeller, Balakumar Sundaralingam, Jonathan Tremblay, Stan Birchfield
IROS 2021
Multi-View Fusion for Multi-Level Robotic Scene Understanding
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, Stan Birchfield
IROS 2021
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, Dieter Fox
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021
Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections
Yashraj Narang, Balakumar Sundaralingam, Miles Macklin, Arsalan Mousavian, Dieter Fox
International Conference on Robotics and Automation (ICRA) 2021
ACRONYM: A Large-Scale Grasp Dataset Based on Simulation
Clemens Eppner, Arsalan Mousavian, Dieter Fox
2021 IEEE International Conference on Robotics and Automation (ICRA)
Alternative Paths Planner (APP) for Provably Fixed-time Manipulation Planning in Semi-structured Environments
Fahad Islam, Chris Paxton, Clemens Eppner, Bryan Peele, Maxim Likhachev, Dieter Fox
2021 IEEE International Conference on Robotics and Automation (ICRA)
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Yuke Zhu
ICRA 2021
Fast Uncertainty Quantification for Deep Object Pose Estimation
Guanya Shi, Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Fabio Ramos, Anima Anandkumar, Yuke Zhu
ICRA 2021
NViSII: A Scriptable Tool for Photorealistic Image Generation
Nathan Morrical, Jonathan Tremblay, Yunzhi Lin, Stephen Tyree, Stan Birchfield, Valerio Pascucci, Ingo Wald
SDG Workshop at ICLR 2021
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
Luyang Zhu, Arsalan Mousavian, Yu Xiang, Hammad Mazhar, Jozef van Eenbergen, Shoubhik Debnath, Dieter Fox
CVPR 2021
Reactive Human-to-Robot Handovers of Arbitrary Objects
Wei Yang, Chris Paxton, Arsalan Mousavian, Yu-Wei Chao, Maya Cakmak, Dieter Fox
ICRA 2021
Best Paper in Human-Robot Interaction, ICRA 2021
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes
Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter Fox
ICRA 2021
Object Rearrangement Using Learned Implicit Collision Functions
Michael Danielczuk, Arsalan Mousavian, Clemens Eppner, Dieter Fox
ICRA 2021
Transferable Task Execution from Pixels through Deep Planning Domain Learning
Kei Kase, Chris Paxton, Hammad Mazhar, Tetsuya Ogata, Dieter Fox
ICRA 2020

2020

RMPflow: A Computational Graph for Automatic Motion Policy Generation
Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff
International Workshop on the Algorithmic Foundations of Robotics (WAFR) 2018
Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies
Visak Kumar, Tucker Hermans, Dieter Fox, Stan Birchfield, Jonathan Tremblay
NeurIPS Workshop on Robot Learning
RMPflow: A Geometric Framework for Generation of Multi-Task Motion Policies
Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff
IEEE Transactions on Automation Science and Engineering (T-ASE)
Human Grasp Classification for Reactive Human-to-Robot Handovers
Wei Yang, Chris Paxton, Maya Cakmak, Dieter Fox
International Conference on Intelligent Robots and Systems (IROS), 2020
Collaborative Interaction Models for Optimized Human-Robot Teamwork
Adam Fishman, Chris Paxton, Wei Yang, Nathan Ratliff, Byron Boots, Dieter Fox
International Conference on Intelligent Robots and Systems (IROS) 2020
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer
Andrew Hundt, Benjamin Killeen, Nicholas Greene, Hongtao Wu, Heeyeon Kwon, Chris Paxton, Gregory D Hager
IEEE Robotics and Automation Letters
Weakly-Supervised 3D Hand Pose Estimation via Biomechanical Constraints
Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, Jan Kautz
European Conference on Computer Vision, 2020
OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
Hongyu Ren, Yuke Zhu, Jure Leskovec, Anima Anandkumar, Animesh Garg
Conference on Uncertainty in Artificial Intelligence (UAI) 2020
Indirect Object-to-Robot Pose Estimation from an External Monocular RGB Camera
Jonathan Tremblay, Stephen Tyree, Terry Mosier, Stan Birchfield
IROS 2020
Automated Synthetic-to-Real Generalization
Wuyang Chen, Zhiding Yu, Zhangyang Wang, Anima Anandkumar
International Conference on Machine Learning (ICML) 2020
MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
Ke Chen, Ryan, Nikolai, Stan Birchfield, Alexander Popov, David Wehr, Ibrahim Eden, Joachim Pehserl
IROS 2020
6-DOF Grasping for Target-driven Object Manipulation in Clutter
Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner, Chris Paxton, Dieter Fox
ICRA 2020
Best Paper Finalist in Robot Manipulation, ICRA 2020, Best Student Paper Finalist, ICRA 2020
Online replanning in belief space for partially observable task and motion problems
Caelan Garrett, Chris Paxton, Tomas Lozano-Perez, Leslie Kaebling, Dieter Fox
ICRA 2020
Self-supervised 6D Object Pose Estimation for Robot Manipulation
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, Dieter Fox
2020 IEEE International Conference on Robotics and Automation (ICRA)
Motion reasoning for goal-based imitation learning
De-An Huang, Yu-Wei Chao, Chris Paxton, Xinke Deng, Li Fei-Fei, Juan Carlos Niebles, Animesh Garg, Dieter Fox
ICRA 2020
Camera-to-Robot Pose Estimation from a Single Image
Timothy E. Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, Stan Birchfield
ICRA 2020
How to close sim-real gap? transfer with segmentation!
Mengyuan Yan, Qingyun Sun, Iuri Frosio, Stephen Tyree, Jan Kautz
arxiv
Toward Sim-to-Real Directional Semantic Grasping
Shariq Iqbal, Jonathan Tremblay, Thang To, Jia Cheng, Erik Leitch, Andy Campbell, Kirby Leung, Duncan McKay, Stan Birchfield
ICRA 2020
DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox
ICRA 2020

2019

6-DOF GraspNet: Variational Grasp Generation for Object Manipulation
Arsalan Mousavian, Clemens Eppner, Dieter Fox
ICCV 2019
A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set
Clemens Eppner, Arsalan Mousavian, Dieter Fox
The International Symposium of Robotics Research (ISSR)
Representing robot task plans as robust logical-dynamical systems
Chris Paxton, Clemens Eppner, Nathan Ratliff, Dieter Fox
IROS 2019
Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments
Xueting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, Jan Kautz
CVPR 2019
Robust Learning of Tactile Force Estimation through Robot Interaction
Balakumar Sundaralingam, Alexander Lambert, Ankur Handa, Byron Boots, Tucker Hermans, Stan Birchfield, Nathan Ratliff, Dieter Fox
ICRA 2019

2018

Fluidic Elastomer Actuators for Haptic Interactions in Virtual Reality
Jose Barreiros, Houston Claure, Bryan Peele, Omer Shapira, Josef Spjut, David Luebke, Malte Jung, Robert Shepherd
IEEE Robotics and Automation Letters
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects
Jonathan Tremblay, Thang To, Bala Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield
Conference on Robot Learning (CoRL) 2018
Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli, Deqing Sun, James M. Rehg, Jan Kautz
European Conference on Computer Vision (ECCV 2018)
Simultaneous Edge Alignment and Learning
Zhiding Yu, Weiyang Liu, Yang Zou, Chen Feng, Srikumar Ramalingam, B. V. K. Vijaya Kumar, Jan Kautz
European Conference on Computer Vision (ECCV) 2018
HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration
Ben Eckart, Kihwan Kim, Jan Kautz
European Conference on Computer Vision (ECCV 2018)
Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, Jinsong Wang
European Conference on Computer Vision (ECCV) 2018
EOE: Expected Overlap Estimation over Unstructured Point Cloud Data
Ben Eckart, Kihwan Kim, Jan Kautz
International Conference on 3D Vision (3DV) 2018
Region Growing Curriculum Generation for Reinforcement Learning
Artem Molchanov, Karol Hausman, Stan Birchfield, Gaurav Sukhatme
arXiv
Geometry-Aware Learning of Maps for Camera Localization
Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz
CVPR 2018 (Spotlight)
Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation
Jonathan Tremblay, Thang To, Stan Birchfield
CVPR 2018 Workshop on Real World Challenges and New Benchmarks for Deep Learnin…
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox
Robotics: Science and Systems (RSS)
Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations
Jonathan Tremblay, Thang To, Artem Molchanov, Stephen Tyree, Jan Kautz, Stan Birchfield
IEEE International Conference on Robotics and Automation (ICRA) 2018
A Variable Shape and Variable Stiffness Controller for Haptic Virtual Interactions
Benjamin C. Mac Murray, Bryan N. Peele, Patricia Xu, Josef Spjut, Omer Shapira, David Luebke, Robert F. Shepherd
RoboSoft
On the Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach
Nikolai Smolyanskiy, Alexey Kamenev, Stan Birchfield
CVPR 2018 Workshop on Autonomous Driving
Riemannian Motion Policies
Nathan Ratliff, Jan Issac, Daniel Kappler, Stan Birchfield, Dieter Fox
arXiv

2017

Sim-to-Real Transfer of Accurate Grasping with Eye-In-Hand Observations and Continuous Control
Mengyuan Yan, Iuri Frosio, Stephen Tyree, Jan Kautz
NIPS 2017 Workshop on Acting and Interacting in the Real World: Challenges in …
Toward Low-Flying Autonomous MAV Trail Navigation using Deep Neural Networks for Environmental Awareness
Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith, Stan Birchfield
IROS 2017

2016

Accelerated Generative Models for 3D Point Cloud Data
Ben Eckart, Kihwan Kim, Alejandro Troccoli, Alonzo Kelly, Jan Kautz
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016

Researchers

Adithya Murali
Ajay Mandlekar
Animesh Garg
Ankit Goyal
Ankur Handa
Arsalan Mousavian
Balakumar Sundaralingam
Benjamin Eckart
Boris Ivanovic
Bowen Wen
Caelan Garrett
Claudia Pérez D’Arpino
Clemens Eppner
Danfei Xu
De-An Huang
Fabio Ramos
Heng Yang
Iuri Frosio
Jeff Smith
Jie Xu
Linxi "Jim" Fan
Jonathan Tremblay
Kaichun Mo
Karen Leung
Li Xu
Marco Pavone
Peter Karkus
Rowland O'Flaherty
Siva Hari
Stan Birchfield
Stephen Tyree
Sushant Veer
Tucker Hermans
Valts Blukis
Wei Yang
Wonmin Byeon
Xinshuo Weng
Xuning Yang
Ye Yuan
Yijie Guo
Yu-Wei Chao
Yue Wang
Yuke Zhu
Yuxiao Chen
Zhiding Yu