Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
Researchers present Puppeteer, a posture-aware, object-grounded diffusion model generating physically consistent co-speech gestures with temporal control.
Puppeteer decomposes long gestures into structured primitives encoded by a causal variational autoencoder into temporally ordered latent tokens. Conditional diffusion in the causal latent space conditions on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. The authors also introduce new evaluation metrics and release SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures with corresponding 3D objects.
- Decomposes gestures into primitives encoded by a causal VAE into temporally ordered latent tokens.
- Causal latent diffusion conditions on speech, motion history, posture reference, and object geometry.
- Enables gesture in-betweening, completion, and explicit temporal control with new evaluation metrics.
- SceneGes is the first curated synthetic 3D dataset of embodied co-speech gestures and objects.
Full article199 words · extracted from huggingface.co · click to collapse
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-speech gesture diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent tokens, each depending only on the past. We then perform conditional diffusion directly in the causal latent space, conditioning on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.00369