Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs
SAGO scores LLM generalization as stability across input variants, not single-format accuracy.
The paper argues LLM generalization should be judged by whether outputs stay semantically stable when the same input is rephrased, rather than by aggregate accuracy on one prompt format. It introduces the Stability-Aware Generalization Objective (SAGO), which measures per-example change across generation consistency, internal activations, confidence, and response mirroring. The authors report statistically significant instability in commonly used models, independent failure modes across behavioral axes, and ranking reversals when the dataset changes.
- Defines generalization as stability under rephrased inputs, not accuracy.
- SAGO tracks generation, activations, confidence, and response mirroring.
- Common models show significant, consistent generalization instability.
- Axes capture separate failures, and datasets can reverse rankings.
From the original article
1 min read at huggingface.co
Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed in different ways. Existing work typically evaluates generalization through aggregate accuracy on a single prompt format, task, or set of variations, which conflates robustness with overall benchmark performance.
By Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.