LLMs tested for sentiment robustness on Spanish political posts
Original title: Evaluating Cultural Reasoning and Robustness in LLMs for Spanish Political Discourse
GPT-4.1-mini stayed most stable, while two smaller open-weight models degraded under sentiment flips.
The paper evaluates LLM sentiment analysis of Spanish political social-media discourse using culturally situated cues rather than labels alone. A protocol merges three human annotation views into Gold and Ambiguous subsets and adds sentiment-flipped synthetic counterfactuals. Proprietary and open-weight models are compared in zero-shot and few-shot settings. Ambiguity and counterfactuals expose failures that evaluation on the original corpus hides, and few-shot prompting helps only modestly.
- Gold, Ambiguous, and Synthetic subsets structure the diagnostic protocol.
- Synthetic items are sentiment-flipped counterfactuals selected by perplexity and an LLM judge.
- GPT-4.1-mini was the most stable model evaluated.
- Two smaller open-weight models degraded more under synthetic perturbations.
- The Spanish dataset with three-way human annotations is public.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) are increasingly used to interpret, generate, and mediate public-facing language, which makes their reliability especially consequential in politically sensitive settings.
By Jorge Valverde-Rebaza · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.