NVIDIAResearch

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, Yuval Atzmon

NVIDIA Research

View results ↓ Paper

TL;DR: Joints, Camera, Action! (JCA) jointly creates video and synchronized 3D motion for rigs of any topology from a text prompt. An input image enables motion that responds to its surroundings and can be registered within the 3D scene. Adding a few chosen keyframes provides control over the motion trajectory. JCA also turns rig motion into video and video into rig motion.

Abstract

Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video–motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion.

Architecture

JCA architecture: video and motion denoisers exchange features through bidirectional cross-attention; joint localization provides image positions for camera recovery.
Joints, Camera, Action! (JCA) combines a rig-conditioned motion model with a pretrained video generator. A graph encoder represents the supplied skeleton, while bidirectional attention bridges let the two branches exchange information as they generate motion and video together. The motion model first learns from motion-only data, and bridge guidance strengthens their interaction at inference. A joint-localization head predicts where the generated joints appear in the video, letting us fit a camera that aligns the 3D motion with the depicted scene.

Scene-aware animation

Given a rig in its rest pose, an input image, and a text prompt, JCA generates video and 3D rig motion that responds to the scene.

Keyframe-Controlled Motion

Given a rig in its rest pose, a text prompt, and RGB keyframes, JCA generates the video and 3D rig motion between them. Playback briefly stops at each input keyframe for comparison.

Video-to-Motion

Given a video and the subject’s rig in its rest pose, JCA recovers the subject’s 3D motion.

Motion-to-Video

Without a reference image

Generate a video from a rig in its rest pose, a 3D motion sequence, and a text prompt, without conditioning on a first frame.

With a reference image

Generate a video from a rig in its rest pose, a 3D motion sequence, and a text prompt, using a reference image to specify the first frame.

Training datasets

Examples of the video-and-motion and motion-only training data.

Rig Animation Comparisons

Text-driven rig animation with JCA, AnimateAnyMesh, Puppeteer, and ActionMesh.
Animations are generated from the same assets and action prompts.

BibTeX

@misc{rotstein2026jointscameraactioncogenerating,
      title={Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs},
      author={Noam Rotstein and Yoad Tewel and Chen Tessler and Haggai Maron and Gal Chechik and Or Litany and Yoni Kasten and Yuval Atzmon},
      year={2026},
      eprint={2610.28632},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.28632},
}