Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation
PRISM turns a few real videos into counterfactual data so a humanoid carries novel objects without real-world fine-tuning.
PRISM is a real-to-sim-to-real framework that expands a few exemplar human-object videos into hundreds of counterfactual interaction clips using video-to-video generation. A contact-anchored pipeline reconstructs human and object motion and retargets imperfect video into physically plausible trajectories. A single policy trained on that variety generalizes to unseen objects within each category. Deployed on a real humanoid with only onboard depth and no real-world fine-tuning, it picks up, carries, and drops boxes, barrels, bins, and balls across novel instances, sizes, and initial poses.
- PRISM amplifies a few real videos into hundreds of counterfactual clips.
- Contact-anchored real-to-sim retargets video into plausible trajectories.
- One policy generalizes to unseen objects within each category.
- Deployed on a real humanoid using onboard depth, with no real-world fine-tuning.
From the original article
1 min read at huggingface.co
Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical barrier to scaling this approach.
By Zihan Wang, Zhen Wu, Pieter Abbeel · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.