Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Keyword benchmarks falsely credit a 1.1B model with tool use; cheap diagnostics and targeted SFT expose and repair it.
Keyword-matching benchmarks can credit small models for tool use they never perform. A matched pair of Spanish security language models—about 662M parameters with no dedicated tool SFT, and 1,109M parameters after a web-heavy curriculum and 6B-token tool SFT—score almost identically on a lenient metric (B4 0.660 vs 0.650). Verbatim checks separate them: the smaller model emits valid generalized tool calls on 6 of 6 training examples, while the 1B model does so on 0 of 6. A first-token probe traces the failure to a missing prior erased during web-heavy training. Targeted SFT (2,202 steps, about 3.3 GPU-hours) raises valid emission on 269 corpus rows from 0.100 to 0.959, and the repaired 1B passes 0.536 of 238 unseen prompts versus 0.428 for the 600M model.
- A 662M and a 1.1B Spanish model score nearly the same on lenient tool-use metrics.
- Verbatim checks: the 600M emits valid calls on 6/6 examples; the 1B emits none.
- The 1B failure is a missing first-token prior erased by web-heavy training.
- Targeted SFT of about 3.3 GPU-hours lifts valid emission from 0.100 to 0.959.
- Both models over-trigger and rarely answer negative prompts without a call.
From the original article
1 min read at arxiv.orgKeyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A 661.
By Juan S. Santillana · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.