EmbodiedLab benchmarks robots that experiment in order to learn
Original title: EmbodiedLab: Benchmarking Embodied Agents That Act to Know
Across 30 simulated science experiments, even the strongest of six frontier agents succeeds less than half the time.
EmbodiedLab asks a robot to choose its own procedure and operate a physics-based laboratory until instrument readings show it has learned the resampled result. The benchmark covers 30 experiments in five disciplines on 14 apparatus platforms, and every task is verified solvable. Six frontier models were tested in their developers’ coding agents. With native control, most failures come from interacting with the apparatus; distilled manipulation skills help weaker agents more than added perception and planning.
- System parameters are resampled for every instance so the answer is not known ahead.
- Scoring uses instrument readings recorded during the episode and the required physical state.
- Agents were also tested with perception, motion planning, and distilled manipulation skills.
- Benchmark and evaluation code are at github.com/HexaFuture/EmbodiedLab.
From the original article
1 min read at huggingface.co
Embodied agents are expected to accomplish real-world goals by deciding for themselves how and when to interact with their surroundings. For many such goals, changing the physical state of the world is the means rather than the end: the agent changes the state of a system through interaction, observes the feedback, and explores until it either achieves the goal itself (e. g.
By Hongcheng Gao, Zelin Zheng, Alex Zhou · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.