EpiSEM-Bench tests LLM agents when safety evidence is omitted
Original title: EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence
The benchmark has 300 matched unsafe/safe pairs across six domains and compares ten models.
EpiSEM-Bench is an executable benchmark of unsafe execution, safety boundaries, and evidence recovery when tool responses omit safety-relevant facts. It contains 300 matched unsafe/safe task pairs across six domains and suppresses critical facts to measure unsafe execution and the minimum observed failure cardinality. An independent audit checks whether agents seek missing facts and withhold prohibited actions. Recovery access is reported to reduce unsafe execution for all ten models, though agents differ in whether they obtain and use the missing facts.
- The suite covers 300 matched unsafe/safe task pairs in six domains.
- It measures unsafe execution under critical evidence omission.
- An audit tool checks whether agents seek omitted facts before acting.
- Recovery access reduced unsafe execution for all ten models tested.
- Omission resistance did not always produce useful evidence recovery.
From the original article
1 min read at arxiv.orgTool-using language model agents rely on environmental facts to decide whether an action is permitted. When a tool response omits a safety-relevant fact, an agent may carry out a prohibited action even though the user request is benign and every returned statement is true.
By Qijin Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.