Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models
Empirical study finds direct whole-file generation beats iterative diff-based editing for Flutter/Dart code models on about 1,790 held-out tasks.
Researchers trained Rainbow-Pony-100M from scratch and fine-tuned Qwen2.5-Coder-0.5B in both direct-generation and diff-based regimes, then evaluated four resulting models on roughly 1,790 Flutter/Dart tasks. Direct generation outperformed diff-based generation on compilation pass rate, bits-per-byte, character-level similarity, and blinded LLM-judge ratings. Diff-based editing is competitive only on short, localized edits in refactoring and error-handling tasks, a property the authors call task locality.
- Four models evaluated across direct-generation and diff-based regimes.
- Direct generation wins on every metric, including blinded LLM-judge scores.
- Diff-based wins concentrate in low-step-count refactoring and error-handling tasks.
- Authors define task locality to guide regime selection for code-editing models.
- Vendors
- Alibaba
- AI models
- Rainbow-Pony-100MQwen2.5-Coder-0.5B
From the original article
1 min read at huggingface.co
Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted.
By Andrej Andrejev · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.