DiagSafe scores LLM safety with prompt-level rubrics, not binary labels
Original title: Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics
The bilingual benchmark reaches 95.34% agreement with humans and finds models weaker in Chinese and multi-turn settings.
Researchers introduce DiagSafe, a diagnostic LLM safety evaluation that scores prompts with explicit safety and usefulness rubrics instead of binary safe or unsafe labels. Its English-Chinese benchmark uses an 84-label taxonomy and two subsets: DiagBase and DiagAttack. They report 95.34% agreement with human annotations, above the 69%–87% of baselines, and Cohen's kappa of 0.864. Leading models differ in unsafe rates and look more susceptible in Chinese and multi-turn contexts.
- DiagBase covers single- and multi-turn prompts plus template-based jailbreaks.
- DiagAttack embeds harmful intents in complex instruction-following tasks.
- Human agreement is 95.34%, versus 69%–87% for the baselines.
- Inter-judge consistency is Cohen's kappa of 0.864.
- The taxonomy has 84 labels and covers English and Chinese.
From the original article
1 min read at arxiv.orgCurrent LLM safety benchmarks predominantly rely on binary safe or unsafe labels, lacking explanatory depth and suffering from low consistency across automated judges. To address this, we introduce DiagSafe, a diagnostic evaluation framework that moves beyond binary classification by employing prompt-level rubrics.
By Hongyu Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.