VAST compiles vulnerability scenarios into ranges for LLM red teams
Original title: VAST: Compiling Grounded Vulnerability Scenarios into Verifiable, Multi-Step Environments for Evaluating LLM Red-Team Agents
Budget-matched tests show 12 of 90 solves without the protocol and 76 of 90 when it is supplied in full.
VAST is a declarative range compiler that separates vulnerability scenarios from a workflow-hardening layer for LLM red-team evaluation. The retained benchmark holds 109 seed edges, 81 topic configurations and 1,387 instances. On TopicServer, three models solved 12 of 90 instances with contract-only prompts and 76 of 90 with the full workflow guide. A ranges_v2 ledger solved 88 of 126, and LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct stayed low on 33 held-out instances.
- Forty-five seed edges carry strict upstream CVE annotations.
- full_guide completion was 28/30, 23/30 and 25/30 across the three models.
- Disclosing worker modules lifted Redis chaining solves from 0/18 to 18/18.
- Header-Recovery SFT scored 4.67 of 33; Base scored 0/33.
- Products
- VAST
From the original article
1 min read at arxiv.orgEvaluating Large Language Model (LLM) security agents requires reproducible environments whose workflow difficulty can be varied and whose failures can be diagnosed. We present VAST (Verifiable Adversarial Scenario Training & Evaluation), a declarative range compiler separating vulnerability scenarios from a compiler-introduced workflow-hardening layer.
By Zhen Wan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.