DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination
DeCAL, a contact-aware dexterous vision-language-action model with visuo-tactile fusion, reports 71% average task success.
DeCAL is a physically-grounded dexterous vision-language-action (VLA) model built on a Mixture-of-Transformers architecture with specialized experts for understanding, imagination, and action generation. It introduces Adaptive Visuo-Tactile Fusion with contact-aware gating and Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics. It reports state-of-the-art results with a 71% average success rate and 83.4% progress success rate, plus generalization to unseen scenarios.
- Mixture-of-Transformers design separates understanding, imagination, and action experts
- Contact-aware gating dynamically regulates tactile interaction in fusion
- Visuo-Tactile Latent Co-Imagination injects implicit physical world knowledge
- Reports 71% average success rate and strong generalization to unseen scenarios
- Organizations
- Peking University
- AI models
- DeCAL
From the original article
1 min read at arxiv.orgDexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics.
By Yankai Fu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.