LLMs Show Variable Reliance on Supplied Lexical Knowledge in Translation Tasks
Original title: Are These Programmers Ripe Enough to Eat? Evaluating How LLMs Handle Lexical Knowledge Conflicts in Translation
Study finds seven of nine models favor supplied meanings over prior knowledge even when instructed to ignore them.
Researchers construct controlled Mandarin-to-English translation task with conflicting lexical meanings. Evaluate effects of word frequency, context, and prompt framing on nine models. Larger models more likely to rely on supplied context. Prompt framing reduces but does not eliminate reliance. Motivates methods to detect lexical conflicts.
- Controlled task isolates lexical knowledge conflicts
- Nine models tested with varying CK reliance
- Larger models utilize CK more than smaller ones
- Prompt framing reduces but does not eliminate CK reliance
From the original article
1 min read at arxiv.orgExternal lexical resources can help large language models (LLMs) translate texts in low-resource languages and specialized domains, but these resources may conflict with models' prior lexical knowledge. To isolate this conflict, we construct a controlled Mandarin-to-English translation task in which familiar source words are assigned new meanings within and across semantic categories.
By Dongxu Wu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.