ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation
ReMoMask-2 rebuilds retrieval in the generator's latent space for text-to-motion generation, achieving lowest FID on KIT-ML and SnapMoGen.
ReMoMask-2 is a retrieval-augmented text-to-motion framework that constructs its retrieval database directly in the generator's pre-quantization latent space and aligns text queries through a distilled lightweight projector, eliminating the representation gap. The framework combines Hierarchical Bidirectional Momentum contrastive learning, Semantic Spatial-Temporal Attention, and Topology Structured Masking to handle hierarchical motion structure. The retriever achieves state-of-the-art accuracy, and ReMoMask-2 attains the lowest FID on KIT-ML and SnapMoGen, with a single mask-transformer stage outperforming the previous two-stage pipeline while delivering the fastest inference.
- Latent-space retrieval database closes representation gap in RAG-T2M
- Retriever achieves state-of-the-art accuracy on HumanML3D, KIT-ML, SnapMoGen
- Lowest FID on KIT-ML and SnapMoGen benchmarks
- Single mask-transformer stage beats prior two-stage pipeline at fastest inference
Full article190 words · extracted from huggingface.co · click to collapse
Text-to-motion (T2M) generation maps natural language to human joint movements, aiding gaming, VR, and robotics. Retrieval-Augmented Text-to-Motion (RAG-T2M) improves generation on complex descriptions by conditioning on retrieved motion-text pairs. However, existing RAG-T2M models face two challenges: coarse-grained retrieval and fusion mechanisms overlook the hierarchical, spatial-temporal topology of human motion, and a representation gap exists because retrieved evidence resides in a semantic space separate from the generator's latents. To address the first, we present ReMoMask, a structure-aware RAG framework coupling Hierarchical Bidirectional Momentum (HBM) contrastive learning to align global and part-level features with text; Semantic Spatial-Temporal Attention (SSTA) for topology-aware fusion; and Topology Structured Masking (TSM) to force robust part-level grounding via adaptive masking. To address the second, we introduce ReMoMask-2, which rebuilds the retrieval database directly within the generator's pre-quantization latent space and aligns text queries via a distilled lightweight projector, allowing the generator to directly consume the retrieved motion's semantic content. Extensive experiments on HumanML3D, KIT-ML, and SnapMoGen demonstrate our retriever achieves state-of-the-art accuracy, while ReMoMask-2 attains the lowest FID on KIT-ML and SnapMoGen; notably, its single mask-transformer stage surpasses ReMoMask's full two-stage pipeline and delivers the fastest inference.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.08365