Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
SWE-Flux finds repository-level execution reasoning hard: the best LLM scores only 37%.
SWE-Flux is a repository-level benchmark of dynamic execution reasoning with 480 instances across 12 real Python repositories. Gold answers are harvested automatically from instrumented test runs rather than written by hand or scored by an LLM. Questions cover single-test and multi-test control flow, loops, program state, dataflow, exceptions, and invariants. Five LLMs were evaluated; the best reached 37% accuracy, doing better on local behavior than on dataflow, inter-procedural execution, precise state, and suite-level aggregation. Input perturbation produced valid fresh variants for almost 90% of selected instances, and those variants were substantially harder.
- SWE-Flux has 480 execution-grounded instances from 12 Python repositories.
- Gold answers come from instrumented tests, not LLM judges.
- Best of five evaluated LLMs reaches only 37% accuracy.
- Perturbed variants are harvested for nearly 90% of selected instances and are harder.
Full article182 words · extracted from arxiv.org · click to collapse
Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs. The benchmark covers singletest and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs shows that this task remains challenging. The best model achieves only 37% accuracy. Models perform better on localized behavior such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation. Finally, we show that the oracle-harvesting pipeline can generate fresh benchmark variants using input perturbation. It successfully harvests valid variants for almost 90% of the selected instances, and the resulting variants are substantially more challenging for the evaluated models.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.28449