Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation
Pipeline pairs generated video with audio-derived force profiles to enable zero-shot, force-aware robot manipulation on Franka Panda for contact-rich tasks.
The paper leverages generated video and audio jointly: loudness of generated contact sounds shapes a bounded, time-varying desired-force profile from a natural-language task prompt. Trajectories execute on a Franka Panda robot with a closed-loop force regulator tracking the audio-shaped profile, succeeding where a kinematic-only baseline fails. The pipeline also serves as a data generation engine to train closed-loop manipulation policies.
- Audio loudness from generated video shapes desired contact-force profiles
- Closed-loop force regulator executes trajectories on a Franka Panda robot
- Pipeline doubles as a data generation engine for training manipulation policies
- Products
- Franka Panda
From the original article
1 min read at arxiv.orgRecent advances in video generation allow robots to learn manipulation trajectories from generated videos. However, these approaches produce purely kinematic trajectories that lack force information, causing failures in contact-rich tasks where appropriate contact forces are essential for success.
By Guanhua Ji · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.