Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
Researchers present Puppeteer, a posture-aware, object-grounded diffusion model generating physically consistent co-speech gestures with temporal control.
Puppeteer decomposes long gestures into structured primitives encoded by a causal variational autoencoder into temporally ordered latent tokens. Conditional diffusion in the causal latent space conditions on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. The authors also introduce new evaluation metrics and release SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures with corresponding 3D objects.
- Decomposes gestures into primitives encoded by a causal VAE into temporally ordered latent tokens.
- Causal latent diffusion conditions on speech, motion history, posture reference, and object geometry.
- Enables gesture in-betweening, completion, and explicit temporal control with new evaluation metrics.
- SceneGes is the first curated synthetic 3D dataset of embodied co-speech gestures and objects.
From the original article
1 min read at huggingface.co
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space.
By Vida Adeli, Soroush Mehraban, Jacob Rommann · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.