ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents
ProgramGym supplies 1,648 verifiable environments and raises Qwen3.8-27B program-reproduction accuracy from 34.87% to 42.84%.
ProgramGym pairs Program-Env, which builds verifiable training environments from repositories, with Program-Agent, a role-isolated analysis, implementation, and review workflow. The pipeline yields 1,648 environments in C, C++, Go, and Rust. Fine-tuning on the resulting trajectories lifts Qwen3.8-27B from 34.87% to 42.84% on ProgramBench. Program-Agent further raises the base and supervised-fine-tuned models to 55.71% and 56.25%.
- Program-Env produces 1,648 environments spanning C, C++, Go, and Rust.
- Fine-tuning raises Qwen3.8-27B from 34.87% to 42.84% on ProgramBench.
- Program-Agent reaches 55.71% on the base model and 56.25% after SFT.
Full article219 words · extracted from huggingface.co · click to collapse
Coding agents are increasingly used to build programs from scratch as software engineers do rather than to resolve issues within a repository or to generate a repository from a given requirements document. However, scalable verifiable training environments that force agents to discover what is required and to decide how the system should be designed remain scarce. Moreover, existing agent workflows also cannot sustain long-horizon development in which the agent explores the reference program before implementation. To address these challenges, we introduce ProgramGym, a framework consisting of Program-Env and Program-Agent for scalable training environments of long-horizon program reproduction agents. Specifically, Program-Env constructs verifiable training environments from repositories through a feedback-driven loop that repairs runtime sandbox builds and the quality of black-box tests. Using this novel pipeline, we obtain 1,648 training environments spanning C, C++, Go, and Rust. Program-Agent combines a role-isolated workflow of analysis, implementation, and review with revision management for compilation checks and failure recovery. Experiments on ProgramBench show that both Program-Env and Program-Agent improve performance on program reproduction. Fine-tuning on trajectories with these constructed environments raises the performance of Qwen3.8-27B from 34.87% to 42.84%. Program-Agent further raises the base and SFT models to 55.71% and 56.25%, both exceeding some flagship models. We will release the ProgramGym to support further research on coding agents that build programs from scratch.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.13395