Longtail-SWE adds 202 SWE tasks for 13 underrepresented languages
Original title: Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
A solver-judge-repairer pipeline revises tests and descriptions; nine model setups were evaluated and artifacts released.
Researchers introduce Longtail-SWE, a repository-level benchmark of 202 issue-resolution tasks in 13 languages with limited coverage in existing benchmarks. A solver attempts each candidate, and a judge uses the trajectory and test results to revise tests and descriptions so valid alternatives are not rejected and incorrect solutions are not accepted. Nine model configurations, including frontier models, were evaluated for performance, consistency, and resource use. Instances, environments, tooling, trajectories, and an easier task set are released.
Unconfirmed: Benchmark scores are the authors' reported results and are not independently confirmed here.
- Languages include Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift.
- The pipeline aims to keep hard tasks while preserving intended requirements.
- Evaluation covers task-solving performance, repeated-attempt consistency, and resource usage.
- An extra easier task set is released to evaluate smaller models.
From the original article
1 min read at huggingface.co
Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems.
By Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.