XStack benchmark evaluates ML repository migration between PyTorch and JAX
Best agent passes 75% of migrations while others perform worse, showing migration remains unreliable.
Stories per day · 30 days
2 RSS
Relatedbenchmarks2ai-agents1ai-benchmarks1benchmark1cli-agents1depth-metrics1embedded-evaluators1gdpx1hugging-face1human-agent-collaboration1jax-pytorch1long-horizon-tasks1ml-repo-migration1model-evals1open-weights1semianalysis1steerability1terminal-tasks1
7 stories, newest first · one card per story
Stories tagged #agent-evals
No AI agent scores above 12 percent on SemiAnalysis GDPx
Opus 5.5 leads across 206 simulated tasks; 98 percent of cyber runs broke a behavioral rule.
1
Paper finds most long-horizon benchmarks lack depth
Audit reveals 94% of synthetic tasks pass in three turns yet depth predicts difficulty better; authors propose DEPTH method and release 105 harder tasks.
Hugging Face policy chief says embedded AI evaluators need standards
Irene Solaiman said evals are hard to reproduce or compare and agent tests remain scarce and costly.
2
REALCLI scores frontier CLI agents poorly on collaboration
The best configuration averaged 49.2%, and the top joint collaboration score was 37.6%.
Language Models Are "Insecure" Reporters
GPT-5.5 hid a planted negative result in 198 of 200 reports unless told to be honest.
Pion, an agent designed to run any company autonomously
Andon Labs opens Pion, a platform for running real businesses with autonomous AI agents, citing Vending-Bench findings of collusion and power-seeking in frontier models.
3