FrontierML tests whether AI agents can buy better model scores
Original title: Import AI 481: AI sub-contractors; a wet lab for AI; and subterfuge during development
Opus 5.5 gained 58.8 points with a $200 budget versus 66.1 for a human researcher baseline.
Import AI describes FrontierML, a benchmark of 21 long-horizon machine-learning tasks in which agents get a 12-hour CPU sandbox, limited validation samples, and $10, $50, or $200 to spend on compute and data. Stronger agents improved more as spending rose; Opus 5.5 led reported systems, and Kimi K3.1 Pro was the top open-weight model cited. The newsletter also notes Y Combinator startup Petrichor, a wet lab meant to take experiment requests from AI agents, and Scale AI’s RSI Control Arena for testing covert side tasks during self-development. The arena write-up is cut off before results.
Unconfirmed: The RSI Control Arena section ends before describing outcomes, so its findings are not established here.
- Tasks cover LLM training, efficiency, vision and audio, computer use, applied NLP, and math and games.
- A human baseline rose 56 points at $10 and 66.1 points at $200.
- Kimi K3.1 Pro scored 25.5 at $10 and 29.6 at $200.
- In 28% of trajectories, agents trained new components or a model from scratch.
- Petrichor says models can submit experiments in plain language and receive results without a person relaying them.
- Products
- FrontierMLRSI Control Arena
- AI models
- Opus 5.5Kimi K3.1 Pro
From the original article
12 min read at importai.substack.com
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.
By Jack Clark · importai.substack.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.