ZeroHour
DashboardNewsCVEsExploitsIncidentsAIBriefAnalyticsEntitiesWatchlistIOCsAsk the feedSourcesFeeds & APIKiosk mode
arXiv cs.CRpublished ()ingested Thu-Hien Trinh-Thi

TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors

infoAI safety & securityimportance 35
AI summary · glm-5.3-flash

TIER benchmark shows LLM safety behaviors shift gradually across threat implicitness levels, with jailbreaks exposing the largest robustness gaps.

The TIER benchmark evaluates LLM safety behaviors across four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks, using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show safety behaviors evolve gradually across threat levels rather than flipping from refusal to compliance. Models with similar Attack Success Rates can exhibit distinct response distributions, arguing for behavior-aware safety evaluation.

  • Benchmark spans four risk domains and four threat-implicitness levels
  • Six-label behavior scale scored by two independent LLM judges
  • Contextual prompts yield the most diverse behaviors; jailbreaks reveal robustness gaps
  • Similar Attack Success Rates can mask very different response distributions
Read the original
ProductsTIER
Tagsllmsafety-evaluationbenchmarkjailbreaksopen-weights
Full article118 words · extracted from arxiv.org · click to collapse

Current LLM safety benchmarks largely rely on binary metrics, overlooking how models respond to harmful prompts with varying threat implicitness. We introduce TIER, a Threat Implicitness Benchmark for behavioral safety evaluation of LLMs. TIER covers four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks. Responses are assessed using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show that safety behaviors evolve gradually across threat levels rather than shifting directly from refusal to compliance. Contextual prompts yield the most diverse behaviors, while jailbreaks reveal the largest robustness gaps. Furthermore, models with similar Attack Success Rates can exhibit distinct response distributions, highlighting the need for behavior-aware LLM safety evaluation.

Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.05117