Instruction capacity metric measures how models hold rules under load
Original title: Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load
On GPT-4o mini, capacity was about 37-38 rules at 0.95 accuracy and 18-19 at full replacement density.
The paper defines instruction capacity as the number of independent word-replacement rules a model holds at a chosen success share, with uncertainty. Load uses meaningless single-token substitutions scored against a computable reference. A GPT-4o mini run of 2,958 requests and 642,430 trials produced regular laws for each load factor and for answer collapse. Code, data and a recomputable passport are published as a calibration for future prompt-load meters.
Unconfirmed: Figures describe one published measurement of GPT-4o mini, not a cross-model benchmark.
- Capacity was about 37-38 rules at a 0.95 right-replacement share.
- At full replacement density, capacity fell to about 18-19 rules.
- The measurement used 2,958 requests and 642,430 Bernoulli trials.
- Each rule outcome is checked against a computable reference answer.
- Vendors
- OpenAI
- AI models
- GPT-4o mini
From the original article
1 min read at huggingface.co
The quality with which language models follow instructions falls as the load grows, yet there is still no quantity that expresses the limit of a model in units of load with an uncertainty.
By Hennadii Kutomanov · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.