Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation
Pipeline pairs generated video with audio-derived force profiles to enable zero-shot, force-aware robot manipulation on Franka Panda for contact-rich tasks.
The paper leverages generated video and audio jointly: loudness of generated contact sounds shapes a bounded, time-varying desired-force profile from a natural-language task prompt. Trajectories execute on a Franka Panda robot with a closed-loop force regulator tracking the audio-shaped profile, succeeding where a kinematic-only baseline fails. The pipeline also serves as a data generation engine to train closed-loop manipulation policies.