Longtail-SWE adds 202 SWE tasks for 13 underrepresented languages
A solver-judge-repairer pipeline revises tests and descriptions; nine model setups were evaluated and artifacts released.
Stories per day · 30 days
2 RSS
Relatedbenchmark1benchmarks1computer-science1frontiercs1llm-agents1longtail-swe1programming-languages1software-engineering1
4 stories, newest first · one card per story
Stories tagged #code-agents
FrontierCS 2 benchmarks LLM agents on 48 open-ended CS tasks
Claude Opus 5.5 leads 12 agents with a 58% pass rate, and 16 tasks have no passing run.
Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL
GAGAR uses groupwise agentic grading to shift code-agent RL credit toward cleaner implementations.
IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
IdeaAMBIG benchmark with 660 instances measures whether LLMs can spot and fix underspecified research-method details for faithful implementation.