Prompting versus token probabilities tested on semantic focus
Original title: Prompts and probabilities in the linguistic assessment of language models: Evidence from semantic focus-sensitivity
Token probabilities beat chance across sizes, while prompting sometimes wins widely in larger models.
The paper compares explicit metalinguistic prompting with token-probability scoring for semantic focus-sensitivity, using contrasts such as which word is focused. Across models of various sizes, token probabilities were universally above chance. Prompting mostly matched earlier findings, but in some larger models it outperformed probabilities by a wide margin. Replacing standard boldface or italics markers with random markers reduced token-probability performance more than prompting.
Unconfirmed: The explanation that models avoid out-of-the-blue focus marking is the authors' proposal.
- Focus was marked in text with boldface, italics, or randomly generated markers.
- Authors attribute the probability gap to models rarely producing focus marking unprompted.
- They argue prompting can be more robust on stylistically atypical data.
From the original article
1 min read at arxiv.orgAs language models (LMs) have improved, an increasingly common means of evaluating their linguistic competence is through explicit prompting (e. g. , "Which sentence is more acceptable?")
By Robert Pasternak · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.