TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors
TIER benchmark shows LLM safety behaviors shift gradually across threat implicitness levels, with jailbreaks exposing the largest robustness gaps.
The TIER benchmark evaluates LLM safety behaviors across four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks, using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show safety behaviors evolve gradually across threat levels rather than flipping from refusal to compliance. Models with similar Attack Success Rates can exhibit distinct response distributions, arguing for behavior-aware safety evaluation.
- Benchmark spans four risk domains and four threat-implicitness levels
- Six-label behavior scale scored by two independent LLM judges
- Contextual prompts yield the most diverse behaviors; jailbreaks reveal robustness gaps
- Similar Attack Success Rates can mask very different response distributions
From the original article
1 min read at arxiv.orgCurrent LLM safety benchmarks largely rely on binary metrics, overlooking how models respond to harmful prompts with varying threat implicitness. We introduce TIER, a Threat Implicitness Benchmark for behavioral safety evaluation of LLMs. TIER covers four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks.
By Thu-Hien Trinh-Thi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.