REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening
REALM generates reactive listener facial motion with coarse trajectories plus stochastic expression residuals, including Ameca deployment.
REALM is a coarse-to-fine audio-driven model for embodied reactive listening. A Reactive Gated Speaker-Listener Fusion module combines listener motion history with speaker audio through a delay-centered attention prior and adaptive gating. A coarse decoder predicts base motion, then audio-conditioned stochastic residuals refine the expression subspace while retaining pose. Evaluations on ViCo and L2L improve over baselines, and the behavior was deployed on an Ameca humanoid with a perceptual user study.
- Gated fusion aligns speaker audio with listener motion history using delay-centered attention.
- Stochastic residuals refine expressions while coarse pose parameters stay fixed.
- Motion-quality metrics improve on ViCo and L2L versus evaluated baselines.
- Generated behavior was deployed on an Ameca humanoid and studied with users.
From the original article
1 min read at huggingface.co
Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory.
By Peizhen Li, Longbing Cao, Yang Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.