NVIDIA model co-generates video and 3D motion for arbitrary rigs
Original title: Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
Cross-attention links a skeleton model to a pretrained video model, trained on nearly 200K pairs.
NVIDIA researchers present a model that co-generates video and replayable 3D motion for a supplied rig, plus a camera that places that motion in the scene. A separate motion model supports arbitrary skeletons and connects to a pretrained video model through a small set of cross-attention layers. Training uses motion-only data, human and robot video, and edited scenes, yielding nearly 200K pairs. The authors say it animates diverse rigs from text more faithfully than video-first pipelines and also converts between video and rig motion.
- The motion model learns skeletons before connecting to the video model.
- Joint localization in frames is used to recover the camera.
- Training pairs come from human video, robot video, and edited 3D assets.
- The model also supports keyframe control of 3D motion.
- Vendors
- NVIDIA
From the original article
1 min read at arxiv.orgVideo generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene.
By Noam Rotstein · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.