Component translation does not reliably change LLM refusal rates
Original title: Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation
On 190 harmful behaviours, only an exploratory Mistral Small 4 Spanish swap stayed significant after correction.
Researchers compared translating either the request component or the operation component into the same target language while holding behaviour, the English frame and translated-span count fixed. English, Chinese and Spanish assignments produced nine configurations on 190 of 300 sampled harmful behaviours. Language-model judges scored refusal, with unsafe compliance reported separately. Pre-specified swaps on Gemini 2.5 Flash and DeepSeek did not survive Holm correction, while GPT-5.4-mini estimates stayed near zero.
Unconfirmed: The Mistral Small 4 Spanish effect is from an exploratory extension, not the pre-specified primary tests.
- Three Gemini 2.5 Flash and DeepSeek refusal swaps were about 5.3 points; one was -0.5.
- Mistral Small 4 showed a 13.7-point Spanish swap, interval [8.4, 18.9].
- That exploratory result was 12.1 points under a second judge.
- GPT-5.4-mini estimates were within 2 points of zero.
From the original article
1 min read at arxiv.orgWhole-prompt translation evaluates a request in different languages but does not compare which component carries a language. We introduce a paired reversal: translate the complete request component or the complete operation component into the same target language, holding behaviour, English frame and translated-span count fixed. Component identity and non-English text quantity change together.
By Yongxi Zhou · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.