Short contexts maintain LLM performance on long-horizon tasks
Original title: You don't need long context
Frontier language models sacrifice little to no task accuracy when maintaining short contexts across long-horizon tasks.
Later tokens are costlier to generate than earlier tokens. On frontier open weight models the disparity may rise to as much as 7x. Through frequent compaction frontier language models maintain performance on long-horizon and long-input tasks. LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context.
- Disparity in cost between early and later tokens up to 7x
- Performance maintained across broad range of long-horizon tasks
- Performance does not collapse on long-horizon software engineering tasks
- Outlines concept of semantic state space models (SSSMs)
From the original article
1 min read at arxiv.orgLater tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts.
By Peter Belcak · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.