1. [Publications](/publications)
2. Affordance Diffusion: Synthesizing Hand-Object Interactions
 
 # Affordance Diffusion: Synthesizing Hand-Object Interactions

  ![](/sites/default/files/styles/wide/public/publications/pic2.png?itok=l26GDQJt)

 Recent successes in image synthesis are powered by large-scale diffusion models. However, most methods are currently limited to either text- or image-conditioned generation for synthesizing an entire image, texture transfer or inserting objects into a user-specified region. In contrast, in this work we focus on synthesizing complex interactions (i.e., an articulated hand) with a given object. Given an RGB image of an object, we aim to hallucinate plausible images of a human hand interacting with it. We propose a two-step generative approach: a LayoutNet that samples an articulation-agnostic hand-object-interaction layout, and a ContentNet that synthesizes images of a hand grasping the object given the predicted layout. Both are built on top of a large-scale pretrained diffusion model to make use of its latent representation. Compared to baselines, the proposed method is shown to generalize better to novel objects and perform surprisingly well on out-of-distribution in-the-wild scenes of portable-sized objects. The resulting system allows us to predict descriptive affordance information, such as hand articulation and approaching orientation.



 ## Authors



Yufei Ye (Carnegie Mellon University)

[Xueting Li](/person/xueting-li)

Abhinav Gupta (Carnegie Mellon University)

[Shalini De Mello](/person/shalini-de-mello)

[Stan Birchfield](/person/stan-birchfield)

Jiaming Song (NVIDIA)

Shubham Tulsiani (NVIDIA)

[Sifei Liu](/person/sifei-liu)

 

 

 ## Publication Date



Tuesday, June 20, 2023

 

 ## Published in



[IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2023](https://cvpr2023.thecvf.com/)

 

 ## Research Area



[Computer Vision](/research-area/computer-vision)

[Human Computer Interaction](/research-area/human-computer-interaction)

[Robotics](/research-area/robotics)

 

 

 ## External Links



[Project Page](https://judyye.github.io/affordiffusion-www/)

[Code](https://github.com/NVlabs/affordance_diffusion)

[ArXiv](https://arxiv.org/abs/2303.12538)

 

 

 ## Uploaded Files



[Paper](https://d1qx31qr3h6wln.cloudfront.net/publications/2023_affordance_diffusion_synthesiz-Camera-ready%20PDF.pdf "Open file in new window")4.3 MB

[Supplementary](https://d1qx31qr3h6wln.cloudfront.net/publications/2023_affordance_diffusion_synthesiz-Camera-ready%20Supplemental%20Material.pdf "Open file in new window")12.33 MB