Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
Expert-validated 1,042-item benchmark finds LLMs never exceed 0.45 free-text correctness on Colombian law, with about half of cited legal norms wrong or nonexistent.
Researchers introduce an expert-validated benchmark of 1,042 items across ten areas of Colombian law and three question formats, evaluated on 15 proprietary and open-weight models. Closed-question accuracy ranges from 0.905 (Gemini 3.1 Pro) down to 0.577, while free-text factual correctness never exceeds 0.45 for any model, and only about half of cited norms are correct. Answer relevancy dissociates from correctness (Spearman rho = -0.46), so models sound responsive while being wrong. The authors conclude expert supervision remains necessary and grounding in authoritative sources is needed.
- 15 models tested; closed-question accuracy 0.905 (Gemini 3.1 Pro) to 0.577
- Free-text factual correctness never exceeds 0.45 for any model
- Only about half of cited legal norms are correct; rest wrong or nonexistent
- Relevancy and correctness negatively correlated (rho = -0.46), misleading non-experts
- Cheap multiple-choice screening predicts ranking (rho = 0.94) but overstates reliability
- AI models
- Gemini 3.1 Pro
From the original article
1 min read at arxiv.orgLarge language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems outside the United States remains largely undocumented. We introduce an expert-validated benchmark for evaluating LLM reliability on the Colombian legal system.
By Rubén Manrique · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.