StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?
Researchers introduce StudyBench, a physics benchmark showing self-evolution gains on textbook problems rarely transfer to olympiad-level questions.
StudyBench is a controlled physics benchmark splitting test data into an Application Set of difficult textbook problems and a Transfer Set of olympiad-level problems. Across three base models, representative self-evolution methods improved on the Application Set but rarely transferred to the harder Transfer Set. A guidance ablation reveals a Guidance Gap, and every method hits a Compute Plateau, indicating the remaining limits are method problems rather than data or compute problems.
- Measures how efficiently self-evolution converts raw material into transferable problem-solving capability.
- Textbook-problem gains rarely transfer to olympiad-level questions across three base models.
- Guidance Gap: best method closes only a fraction of what in-context guidance unlocks.
- All methods hit a Compute Plateau, so limits are method problems, not data or compute.
Full article209 words · extracted from huggingface.co · click to collapse
Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolution method should share the same property, that is autonomously learning from raw training material for transferable problem-solving capability. However, we still lack a direct measurement for it. We introduce StudyBench, a controlled physics benchmark that directly measures how efficiently a self-evolution method converts training material into capability. We organise the test set into an Application Set, consisting of difficult textbook problems and evaluating absorption ability, and a Transfer Set, consisting of olympiad-level problems and evaluating transfer ability. Benchmarking representative self-evolution methods across three base models, we find that improvements on the Application Set rarely translate to the harder Transfer Set. A guidance ablation exposes a Guidance Gap: even the strongest method closes only a small fraction of what the same material unlocks when supplied as in-context guidance. Besides, every method hits a Compute Plateau, saturating well before exhausting its compute budget. The remaining gap is therefore a method problem rather than a data or compute problem. By offering a clean and controlled benchmark, StudyBench turns self-evolution progress from an open-ended pursuit into a measurable target for future research. Our code is released at https://github.com/thunlp/StudyBench.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.00787