Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Keyword tool-use scores falsely credit small Spanish security models; a cheap diagnostic ladder separates real tool calls.
Researchers show that keyword-matching benchmarks can credit small models for tool use they never perform, using a matched pair of Spanish security language models. A 661.6M model and a 1,109M model scored almost the same on lenient metrics (B4 0.660 vs 0.650), but verbatim checks separated them: the smaller model emitted valid tool calls on 6/6 examples and the larger on 0/6. A first-token probe tied the failure to a missing <|tool_call|> prior erased by web-heavy training. A short targeted SFT run, about 3.3 GPU-hours, lifted valid emission on 269 rows from 0.100 to 0.959 and beat the 600M model on 238 unseen prompts (0.536 vs 0.428).
- A 662M and 1.1B model scored nearly identically on lenient tool-use metrics.
- Verbatim checks: the 600M passed 6/6 examples; the 1B passed 0/6.
- The 1B failure is a missing tool-call token prior after web-heavy training.
- Targeted SFT of about 3.3 GPU-hours raised valid emission from 0.100 to 0.959.
- On 238 unseen prompts the repaired 1B scored 0.536 versus the 600M's 0.428.
From the original article
1 min read at huggingface.co
Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A 661.
By Juan S. Santillana · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.