PlaMo animates a humanoid in a complex 3D scene. The input consists of a physically-simulated 3D scene and a series of text instructions describing high-level navigation landmarks and locomotion type ("Crouch-walk from the tree to the swing"). The scene may contain diverse terrain (gravel, stairs), 3D obstacles and dynamic obstacles (here, a green ball). The output is a sequence of motor actuations controlling a humanoid character. PlaMo produces a planned path together with a head-height and speed profile that match the textual guidance (locomotion type) and the constraints of the movement controller. Images from top left, counter-clockwise: 1. The controller adapts to the 3D environment, producing a crawl locomotion under the obstacle. 2. Supports various motion types. 3. continuously re-planning the path allows to avoid moving obstacles. 4. Locomotion speed is adjusted depending on terrain and obstacles.
Abstract
Controlling humanoids in complex physically simulated worlds is a long-standing challenge with numerous applications in gaming, simulation, and visual content creation. In our setup, given a rich and complex 3D scene, the user provides a list of instructions composed of target locations and locomotion types. To solve this task we present PlaMo, a scene-aware path planner and a robust physics-based controller. The path planner produces a sequence of motion paths, considering the various limitations the scene imposes on the motion, such as location, height, and speed. Complementing the planner, our control policy generates rich and realistic physical motion adhering to the plan. We demonstrate how the combination of both modules enables traversing complex landscapes in diverse forms while responding to real-time changes in the environment.
Overview
A simulated scene is provided to the path planner, together with a series of textual instructions, requesting a humanoid to reach landmarks in the scene using various locomotion types. The high-level planner computes a path that is fed to a reinforcement-learning-based low-level motion controller, which controls a humanoid to follow the path using the request locomotion type.
The three stages of our dynamic path planner: (i) A* solver, (ii) path refiner, and (iii) speed controller.
The locomotion policy observes the character state, the height map of the terrain, and the requested path. During training, the trajectories are randomly sampled. In inference, the trajectories consider the terrain characteristics, such as obstacles. After simulating the predicted action the resulting state is used for providing the style reward and for the path following reward.
Results
Random Static Playgrounds
Dynamic Obstacles
Dynamic Playgrounds
Failures
Citation
@article{hallak2024plamo,
title={PlaMo: Plan and Move in Rich 3D Physical Environments},
author={Hallak, Assaf and Dalal, Gal and Tessler, Chen and Guo, Kelly
and Mannor, Shie and Chechik, Gal},
journal={arXiv preprint arXiv:2406.18237},
year={2024}}
Paper
PlaMo: Plan and Move in Rich 3D Physical Environments
Assaf Hallak, Gal Dalal, Chen Tessler, Yunrong Guo, Shie Mannor and Gal Chechik
NVIDIA Research
