An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
An open dashboard scores 18 models on EU GPAI systemic risks, showing averages hide 14–37 point worst-case drops.
Researchers introduce the Systemic Risk Index, an open pipeline and dashboard that traces model risk scores to public benchmark evidence under the EU GPAI Code of Practice. It groups 19 benchmarks into CBRN, cyber offense, harmful manipulation, and loss of control, and tests models with harm-preserving perturbations and simulated deployment contexts. Across 18 models, worst-case aggregation lowers scores by 14 to 37 points compared with averages. LLM judges agree with human graders at kappa 0.78–0.82, and 83 percent of audited transformations preserve the original harm.
- Organizes 19 public benchmarks into four EU GPAI systemic-risk categories.
- Worst-case aggregation lowers scores by 14 to 37 points versus averages.
- LLM-judge agreement with humans matches human-human kappa of 0.78 to 0.82.
- Blind audit finds 83 percent of sampled transformations preserve original harm.
- Products
- Systemic Risk Index
- Organizations
- European Union
From the original article
1 min read at arxiv.orgClaims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is accessible at all. We present the Systemic Risk Index, an open evaluation pipeline and dashboard built to make empirical evidence more transparent and traceable to the public.
By Jacob T. Emmerson · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.