What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation
A new benchmark shows LLMs reach 68.3-93% accuracy propagating local revisions across conversationally generated artifacts, with parallel-sample selection most cost-effective.
The paper introduces a benchmark for revision propagation: when users request a local change, LLMs must identify dependencies and update all affected parts of an artifact generated through conversation, where context lives in the chat history. Nine revision methods, including sequential reflection and parallel sampling variants, were evaluated on gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b. Baselines scored 68.3-93% accuracy, and selecting among three parallel samples via LLM-based or medoid selection improved accuracy by 2.2-9.7% as the most cost-effective test-time compute strategy. Code and dataset are released.