Per-option encoders beat joint scoring on closed-set decisions
Original title: Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters
At 32 million parameters, per-option scoring gains 5.8 skill points but needs 5 to 13 times more passes.
A three-seed study asks how small an encoder for closed-set decisions, such as tool selection, can be and whether options should be read jointly or separately. At 32 million parameters, per-option scoring beats joint scoring by 5.8 chance-corrected skill points at matched optimizer steps, and by 2.8 at roughly matched compute. The gain requires a reranker initialization. Among six released joint encoders from 17 million to 1 billion parameters, the 68- and 150-million models beat a 435-million-parameter DeBERTa-v3 zero-shot classifier on overlap-audited tasks.
Unconfirmed: Two external benchmarks agree with per-option scoring, but not significantly on their unseen tasks.
- The skill gap is 5.8 points at matched steps and 2.8 at matched compute.
- Per-option scoring uses at most twice the median latency but over four times the mean.
- Option reordering changes 12 to 23% of joint answers and none for per-option scoring.
- The smallest model’s median request is 14 ms on 16 threads, versus 728 ms for the classifier.
- Gains over DeBERTa-v3 are 4.9 and 5.9 accuracy points on 209 System1-Arena tasks.
- Products
- System1-Arena
- AI models
- DeBERTa-v3
From the original article
1 min read at huggingface.co
Many language-model pipelines end in a closed-set decision, such as selecting a tool. Prompting a large language model for it is slow and order-sensitive. We ask how small an encoder for such decisions can be and whether it should read the options jointly or separately.
By Tardelli Ronan Coelho Stekel · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.