Video generation models can now produce visually convincing motion, but that motion has remained trapped in pixels: there has been no reliable way to reuse it to animate a specific 3D asset and replay it within its original scene. NVIDIA researchers present a system that addresses this gap by jointly generating, for any supplied skeletal rig, both the video and its corresponding 3D motion, along with a camera that registers that motion within the scene.
Prior approaches typically handled only human skeletons, often at the cost of billions of additional parameters, or attempted to reconstruct 3D motion by triangulating generated 2D pose maps. The NVIDIA team takes a different route: a separate motion model, capable of supporting arbitrary skeletons, is linked to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is considerably easier to obtain and augment than paired video-motion datasets.
The system also learns to locate each joint directly within video frames, which allows it to recover camera position and anchor the 3D motion within the filmed scene. To train across a wide variety of skeletons, the researchers used a generative video editor to place animated 3D assets into scenes suited to their actions, curating the edits so the underlying known motion stays valid after editing. This process yielded nearly 200,000 training pairs, combining human and robot video.
According to the evaluations presented, the model animates diverse rigs from text prompts more faithfully than existing video-first pipelines, and its generated motion responds to the surrounding scene. It can also convert video into rig motion, reverse the process to turn rig motion into video, and support keyframe-based control of the 3D motion. This ability to connect video generation with usable 3D animation is relevant for animation pipelines, but also for robotics, where turning a video demonstration into a reusable joint trajectory is a concrete practical challenge.