RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning
RLHND turns the Cosmos 3 video model into a tracker estimating hand pose, contact, and force for robot learning.
RLHND jointly estimates hand pose and realistic tactile information from monocular egocentric video for robot policy training. It turns a pretrained Cosmos 3 video diffusion backbone into a deterministic clip-level feature extractor via clean-latent conditioning. Pose outputs use anatomically plausible joint angles, with optional shape conditioning for consistency, while a separate expert predicts dense contact and force. The authors report state-of-the-art pose, contact, and force results and show retargeting in real-world robot experiments.
- Converts the Cosmos 3 video diffusion backbone into a deterministic clip-level feature extractor.
- Predicts anatomically plausible hand poses with optional consistent shape conditioning.
- A separate tactile expert estimates dense contact and force over the hand surface.
- Reports state-of-the-art pose, contact, and force results plus robot retargeting.
From the original article
1 min read at huggingface.co
Recently, approaches that leverage human video datasets for robot policy training have become increasingly prevalent. However, most existing hand trackers regress pose from cropped frames with limited priors on hand motion and object interaction, resulting in inaccurate and physically inconsistent estimates. Moreover, the lack of physical cues, e.
By Seungjun Moon, Subin Jeon, Sangwoo Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.