EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras
EyeRobot 2.0 matches wrist-camera manipulation using active gaze and only one stereo camera.
EyeRobot 2.0 performs fine-grained bimanual manipulation with a single stereo camera by actively fixating a 3D point and allocating more visual tokens to image centers. A low-level gaze-servoing policy and a task-progress target selector are trained with reinforcement learning on real data, and gripper actions are represented in a fixation-relative SE(3) frame. Across 7 real and 6 simulated tasks, more than 1,000 physical and 1,800 simulated trials show passive-stereo success dropping from 52% to 27% without wrist cameras. EyeRobot 2.0 outperforms passive stereo by 40% in the real world and 20% in simulation, matching clear wrist-camera policies (69% vs 64%) and more than doubling them when grasped objects occlude wrist views (48% vs 22%).
- Active gaze uses one stereo camera and foveal tokens.
- Gaze servoing and a target selector are trained with RL.
- Passive stereo success falls from 52% to 27% without wrist cameras.
- EyeRobot beats passive stereo by 40% in real trials.
- It doubles success versus occluded wrist-camera policies, 48% vs 22%.
- Products
- EyeRobot 2.0
From the original article
1 min read at huggingface.co
Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2. 0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it.
By Kush Hari, Justin Kerr, Nidhya Shivakumar · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.