SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation
SSP-Bench dynamically generates safety, security, and privacy evals for 24 LLMs, exposing static-benchmark failures like near-zero safety ranking correlation.
The paper introduces SSP-Bench, a dynamic benchmarking framework that generates LLM safety, security, and privacy evaluation instances on demand, ensuring label validity through externally grounded sources, enforcing scope via service-specific validation, and calibrating difficulty with a multi-model steering panel. Benchmark construction is formulated as multi-objective optimization over difficulty, separability, novelty, and diversity. Across 24 models and four SSP services, SSP-Bench found near-zero correlation in safety rankings due to construct mixing, strong safety-over-refusal coupling, and hidden within-family regressions, showing static benchmarks can misrepresent model behavior.
- On-demand instance generation counters saturation, contamination, and aggregation artifacts of static benchmarks
- Label validity grounded in external sources; difficulty calibrated via multi-model steering panel
- Evaluated 24 models across four safety/security/privacy services
- Reveals near-zero safety ranking correlation and hidden within-family model regressions
From the original article
1 min read at arxiv.orgEvaluation of large language models (LLMs) for safety, security, and privacy (SSP) relies heavily on static benchmarks, which suffer from score saturation, data contamination, and aggregation artifacts, and fail to capture sensitivity to linguistic variation. As a result, models that perform well on fixed test sets often fail under semantically equivalent rephrasings.
By Fatih Deniz · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.