COSMI: COmpositional Synthesis of Multi-object Interactions
COSMI composes single-object captures into 222k multi-object sequences and trains a text-to-interaction diffusion model.
COSMI builds multi-object human-object interaction data by composing contact-consistent single-object clips, mirroring hands, transferring motions between bodies, and filtering pairings with a language model and geometric checks. The dataset contains 222k sequences and 275 hours with up to five objects, nearly thirty times the largest prior multi-object capture. A text-to-interaction diffusion transformer uses weight-shared object slots to generate a variable number of objects relative to the body parts that move them. Models trained on COSMI generalize to unseen objects and interaction combinations and outperform baselines on text alignment and contact accuracy.
- Composes single-object clips into plausible multi-object interactions.
- Dataset contains 222k sequences and 275 hours with up to five objects.
- Shared object slots generate a variable number of objects relative to the body.
- Largest contact-accuracy margin appears on an unseen object.
From the original article
1 min read at huggingface.co
Generative models of human-object interaction are bounded by the data that exists: everyday activities involve several objects, but most captured datasets record one at a time, as multi-object capture is combinatorially expensive. Our observation is that interactions are local, so single-object captures already contain the parts of multi-object activities.
By Daniel Eskandar, Ilya A. Petrov, Gerard Pons-Moll · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.