Reflex: Visually Grounded Reactive Humanoid Control
Reflex enables a Unitree G1 to catch thrown boxes from egocentric RGB-D, with 65% success at 2.5 meters.
Reflex is a three-stage framework for whole-body reactive catching on humanoids using egocentric vision. A state-based policy learns catching, then infers box dynamics from delayed observations, and an RGB-D encoder recovers that representation while the controller stays fixed. In simulation it reaches 85.7% catch success from RGB-D, close to 87.9% with privileged box state. On a Unitree G1 it catches human-thrown boxes with 65% and 40% success at 2.5 m and 5 m.
- Three-stage training: RL catching, dynamics inference, then a frozen visual encoder.
- Egocentric RGB-D catch success is 85.7% in simulation versus 87.9% with privileged observations.
- Removing temporal visual history drops success by 35.6 and 39.8 points.
- Real Unitree G1 success is 65% at 2.5 m and 40% at 5 m.
Full article265 words · extracted from huggingface.co · click to collapse
Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading. These everyday actions require anticipating a moving target and reflexively responding with the whole body within about a second. Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric vision. We study catching thrown boxes as a representative task that demands all three capabilities; existing systems cover only part of it, relying on end-effectors, motion capture, or marked objects. We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics. First, a state-based policy learns whole-body catching with reinforcement learning. Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed. This decomposition provides each capability with a direct training signal and makes visual learning substantially more scalable. In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared with 87.9% when given privileged box observations. Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 points on centered and laterally offset throws. On a real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 and 5 m, reaching 76% and 59% of human catchers' success at the same distances.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.12772