First-token cueing recovers closed-book facts LLMs fail to start
Original title: On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs
Prefilling only the gold answer's first token fixes a fifth to a half of multi-token English QA errors across 16 open models.
The paper argues many closed-book factual errors are first-token recall failures: models encode answers they do not produce. First-Token Cueing prefills the gold answer's first token, after which models often complete and recognize the answer. Uncertainty peaks at that token, and it recovers more errors than longer gold prefixes. Across 16 open models, the cue corrects between a fifth and a half of multi-token English QA errors and improves accuracy in all seven languages of a multilingual benchmark.
Unconfirmed: Whether corrected errors count as encoded knowledge depends on the authors' sampled-reasoning criterion.
- The first gold token accounts for the largest share of recovered errors.
- Accuracy improves in all seven languages of the multilingual benchmark.
- Most corrected errors are also solved by the model's sampled reasoning.
- Models continue a correct start more confidently than a matched wrong start.
- Models rarely produce that correct start without the gold answer.
From the original article
1 min read at huggingface.co
Large language models often answer factual questions incorrectly in closed-book settings, and such errors are usually attributed to missing knowledge. Yet models often encode answers they fail to produce. We show that many closed-book errors are first-token recall failures: the model commits to a wrong first token, but once given the right one it completes the answer and often recognizes it as correct.
By Jaewon Sok, Jewon Yeom, Seonghyeon Park · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.