What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation
A new benchmark shows LLMs reach 68.3-93% accuracy propagating local revisions across conversationally generated artifacts, with parallel-sample selection most cost-effective.
The paper introduces a benchmark for revision propagation: when users request a local change, LLMs must identify dependencies and update all affected parts of an artifact generated through conversation, where context lives in the chat history. Nine revision methods, including sequential reflection and parallel sampling variants, were evaluated on gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b. Baselines scored 68.3-93% accuracy, and selecting among three parallel samples via LLM-based or medoid selection improved accuracy by 2.2-9.7% as the most cost-effective test-time compute strategy. Code and dataset are released.
- New benchmark targets dependency-aware revision propagation in conversational artifacts
- Nine methods evaluated on gpt-oss, gpt-5.4-mini, and qwen3.5 model families
- Best cost-accuracy tradeoff: select from three parallel samples (LLM or medoid)
- Gains of 2.2-9.7% accuracy over baselines
- Code and dataset public on GitHub (ntt-dkiku)
- Vendors
- NTT
- Products
- gpt-ossgpt-5.4-miniqwen3.5
From the original article
1 min read at huggingface.co
Large Language Models (LLMs) often help users generate artifacts through iterative cycles of generation and revision in conversation. A challenge here is that, when users specify only a local change during revision, LLMs must instead identify the relevant dependencies and propagate the revision to all affected parts of the artifact.
By Daisuke Kikuta · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.