DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination
DeCAL, a contact-aware dexterous vision-language-action model with visuo-tactile fusion, reports 71% average task success.
DeCAL is a physically-grounded dexterous vision-language-action (VLA) model built on a Mixture-of-Transformers architecture with specialized experts for understanding, imagination, and action generation. It introduces Adaptive Visuo-Tactile Fusion with contact-aware gating and Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics. It reports state-of-the-art results with a 71% average success rate and 83.4% progress success rate, plus generalization to unseen scenarios.
- Mixture-of-Transformers design separates understanding, imagination, and action experts
- Contact-aware gating dynamically regulates tactile interaction in fusion
- Visuo-Tactile Latent Co-Imagination injects implicit physical world knowledge
- Reports 71% average success rate and strong generalization to unseen scenarios
Full article177 words · extracted from arxiv.org · click to collapse
Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics. While recent works have incorporated tactile sensing into robotic manipulation, most approaches still rely on homogeneous multimodal fusion, lacking adaptive tactile integration and explicit modeling of physical dynamics. In this work, we present DeCAL, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation. Built upon a Mixture-of-Transformers (MoT) architecture, DeCAL leverages specialized experts for each capability while enabling efficient information flow among them. To effectively leverage tactile information, we introduce Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy. Furthermore, we propose Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics, equipping the policy with implicit physical world knowledge. Experimental results show that DeCAL consistently achieves state-of-the-art performance across all tasks, attaining a 71% average success rate and an 83.4% progress success rate, while also demonstrating strong generalization to unseen scenarios. The website is available at https://aureleopku.github.io/DeCAL.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.09119