ARDY:
Autoregressive Diffusion with Hybrid Representation
for Interactive Human Motion Generation
ACM Transactions on Graphics · SIGGRAPH 2026
Paper (PDF)
Code
Models
ARDY is an autoregressive diffusion model designed for interactive motion generation, supporting online text prompting and flexible long-horizon kinematic constraints (root paths/waypoints, full-body keyframes, and sparse joint positions/rotations) with real-time responsiveness.
Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches like Kimodo offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations demonstrate strong motion quality and constraint adherence, and we present an interactive demo with dynamic text control, keyframe constraints, path following, and real-time locomotion control.
Key Capabilities of ARDY
Online Text-to-Motion Generation
ARDY supports interactive text-conditioned motion generation across a wide range of behaviors.
Kinematically Constrained Motion Generation
ARDY supports flexible kinematic constraints, including root trajectories or waypoints, full-body keyframes, end-effector joint positions and rotations, as well as arbitrary combinations of these constraints. Constraints can also be specified far into the future (beyond the current generation window) to enable long-horizon goal reaching.
- Root Trajectory 1
- Root Trajectory 2
- Root Waypoint 1
- Root Waypoint 2
- Fullbody Keyframe 1
- Fullbody Keyframe 2
- End Effector Position
- End Effector Position+Orientaion
- Long-Horizon Goal 1
- Long-Horizon Goal 2
- Compositionality
- Kinematic Constraints Chain
Application: Interactive Humanoid Control
ARDY enables online motion synthesis for interactive applications, which can be valuable for game character control, downstream robotics, and simulation workflows. It supports real-time locomotion control via mouse waypoint editing and keyboard velocity commands.
Humanoid Robot Control
By combining ARDY’s real-time humanoid motion generation with the SONIC physical tracking policy, we enable interactive robot motion control with streaming constraints and user inputs. We demonstrate applications on the Unitree G1 robot.
Method
ARDY is an autoregressive diffusion model for interactive motion generation. It is built around two key ideas: (1) a hybrid motion representation that combines explicit global root motion with a compact latent embedding of body motion, and (2) an autoregressive two-stage transformer denoiser that generates motion in a streaming fashion while conditioning on online text prompts and flexible, spatiotemporally sparse kinematic constraints over long horizons.
Motion Tokenizer. The encoder first embeds the patchified body motion into a latent representation. This latent body motion is concatenated
with the patchified global root motion to form our hybrid representation, which is decoded back to reconstruct the body motion.
This hybrid representation balances precise, interpretable root control (useful for global scene-space constraints) with efficient generative
learning in a lower-dimensional latent space for body motion.
Autoregressive Two-Stage Transformer Denoiser. (Left) Conditioned on a variable-length history context and optional spatial goal constraints, the
autoregressive denoiser predicts a sequence of C clean motion tokens within the current generation window. Spatial goal constraints can be arbitrarily sparse
and may be located within or beyond the current motion generation window. (Right) The two-stage denoiser first predicts clean global root motion, which
then conditions the second stage to predict clean latent body tokens, together forming the complete hybrid motion prediction.
The denoiser supports a variable-length history context to capture longer-term semantics, and conditions on masked kinematic constraints that can be
sparse in both time and joints. These constraints can span beyond the current generation window to enable long-horizon goals (e.g., far-future waypoints).
Our interleaved two-stage design predicts root first and then body conditioned on root, helping maintain motion fidelity while satisfying online text prompts and
diverse spatial controls such as root trajectories/waypoints, full-body keyframes, and sparse joint positions/rotations.
More details can be found in the paper.
Humanoid Motion at NVIDIA
ARDY is part of a broader effort to support interactive humanoid motion generation and downstream robotics/animation applications. Related projects include:
SOMA Body Model
A parametric body model used across NVIDIA humanoid motion projects.
BONES-SEED Dataset
A public dataset with production-quality motion capture data.
ProtoMotions
A framework for training physics-based humanoid policies.
SOMA Retargeter
Tools for retargeting motion data across humanoid skeletons.
GEM
A motion diffusion model for reconstructing motion from monocular videos.
Kimodo
An offline controllable motion diffusion model for high-quality 3D motion authoring.
MotionBricks
Scalable real-time motions with modular latent generative model and smart primitives.
GEAR SONIC
A framework for humanoid whole-body tracking and control.
Acknowledgments
We would like to thank Edy Lim, Eugene Jeong, Sam Wu, Ehsan Hassani, Michael Huang, and Jin-Bey Yu for their help with data processing and cleaning, and Cyrus Hogg, Simon Yuen, Lindsey Pavao, Jenna Diamond, Rizwan Khan, Samantha Shinagawa, and Akanksha Shukla for their efforts on data acquisition and labeling. We also thank the anonymous reviewers for their valuable feedback.
BibTeX
@article{zhao2026ardy,
title = {ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation},
author = {Zhao, Kaifeng and Petrovich, Mathis and Zhang, Haotian and Wang, Tingwu and Tang, Siyu and Rempe, Davis},
journal = {ACM Transactions on Graphics (TOG)},
year = {2026},
volume = {45},
number = {4},
articleno = {86},
doi = {10.1145/3811284}
}