Generating Human Interaction Motions in Scenes
with Text Control

Hongwei Yi1,2 Justus Thies2,3 Michael J. Black2 Xue Bin Peng1,4 Davis Rempe1
1 NVIDIA 2 Max Planck Institute for Intelligent Systems, Tübingen, Germany3 Technical University of Darmstad4 Simon Fraser University
arXiv, 2024


TeSMo Code


We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Previous text-to-motion methods focus on characters in isolation without considering scenes due to the limited availability of datasets that include motion, text descriptions, and interactive scenes. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model, emphasizing goal-reaching constraints on large-scale motion-capture datasets. We then enhance this model with a scene-aware component, fine-tuned using data augmented with detailed scene information, including ground plane and object shapes. To facilitate training, we embed annotated navigation and interaction motions within scenes. The proposed method produces realistic and diverse human-object interactions, such as navigation and sitting, in different scenes with various object shapes, orientations, initial body positions, and poses. Extensive experiments demonstrate that our approach surpasses prior techniques in terms of the plausibility of human-scene interactions, as well as the realism and variety of the generated motions. Code will be released upon publication of this work on the project webpage.

    author={Yi, Hongwei and Thies, Justus and Black, Michael J. and Peng, Xue Bin and Rempe, Davis},
    title={Generating Human Interaction Motions in Scenes with Text Control},
    journal = {arXiv:2404.10685},

Thanks to Yangyi Huang and Yifei Liu for technical support. Thanks to Mathis Petrovich and Nikos Athanasiou for the fruitful discussion about text-to-motion synthesis. Thanks to Tomasz Niewiadomski, Taylor McConnell, and Tsvetelina Alexiadis for running the user study. This project page template is based on this page.

For any questions, please contact Hongwei Yi and Davis Rempe.