Model co-generates video and 3D motion for arbitrary rigs
Original title: Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
Cross-attention links a skeleton motion model to a pretrained video model, trained on nearly 200K pairs.
Researchers present a model that co-generates video and replayable 3D motion for any supplied rig, plus a camera that places the motion in the scene. A separate motion model learns arbitrary skeletons and connects to a pretrained video model through a small set of cross-attention layers. The authors report nearly 200K training pairs and say the model beats video-first pipelines, supports video-to-rig and rig-to-video conversion, and allows keyframe control.
- The motion model learns from motion-only data before linking to video.
- Joint localization in frames is used to recover the camera.
- A generative video editor places animated 3D assets into suitable scenes.
- Training combines those edits with human and robot video.
- The project page is hosted by NVIDIA research labs.
- Organizations
- NVIDIA
From the original article
1 min read at huggingface.co
Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene.
By Noam Rotstein, Yoad Tewel, Chen Tessler · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.