Benchmark introduced for LLM agent rule compliance after context compaction
Original title: Do Agents Still Follow the Rules? Benchmarking Rule Compliance under Agent Context Compaction
ARCBench measures violations in agent actions on the pi harness under different compaction strategies.
The paper introduces ARCBench, a benchmark using the pi agent harness to evaluate rule compliance in LLM agents. Compaction rarely deletes rules, with most violations occurring while rules remain in history. Different configurations affect violation sources, with adversarial attacks increasing violations.
- Violations more than double with targeted attacks
- Retention of rules does not guarantee compliance
- Compaction strategy decides rule survival
- Strongest compactor attack corrupts rule in summary
From the original article
1 min read at arxiv.orgLLM agents compact long histories to fit their context windows, but the compacted context must still carry the operating rules that govern later actions. Prior work shows that violations rise after compaction, but measures compliance outside a complete agent harness and does not identify where each violation comes from.
By Yiwei Zha · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.