DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation
DexTacWAM injects fingertip touch into a video world model, scoring 70.6 versus 38.0 on dexterous tasks.
DexTacWAM is a visuo-tactile world-action model that encodes each fingertip, compresses those features with a finger- and pose-aware module, and injects the tactile latent into a video diffusion world model. On six contact-rich dexterous tasks using a 22-DoF bimanual platform, it averages 70.6 versus 38.0 for the strongest baseline. Ablations show modeling contact evolution, not tactile conditioning alone, drives the gain, and continual vision-to-touch learning extends a pretrained video model with roughly 100 demonstrations per task.
- Each fingertip is encoded and compressed into a tactile latent.
- Average score is 70.6 versus 38.0 for the strongest baseline.
- Removing tactile world modeling drops a four-task mean from 74.7 to 26.6.
- About 100 demonstrations per task adapt a frozen vision VAE to touch.
- The compressor keeps 89.4% contact recall and speeds training 2.26x.
Full article226 words · extracted from arxiv.org · click to collapse
Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics. We present DexTacWAM, a visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile world modeling. Across six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform, DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline. Ablations attribute the gain to modeling contact evolution as part of the predicted world state rather than tactile conditioning alone: removing tactile world modeling reduces the four-task mean from 74.7 to 26.6 while keeping the same tactile features and action expert. After four hours of tactile-encoder adaptation with a frozen pretrained vision VAE, our continual vision-to-touch learning extends the pretrained video model to touch using roughly 100 demonstrations per task without tactile midtraining, while retaining visual prediction quality within 0.5 dB of vision-only counterparts. The compressor retains 89.4% of pre-fusion contact recall while enabling 2.26x faster training and 1.29x faster inference. Together, these results show that pretrained video priors can be extended to distributed multi-finger contact dynamics in a data- and compute-efficient manner.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.24976