Paper defines accountable partial understanding for agentic systems
Original title: Accountable partial understanding: securing the epistemic attack surface of agentic systems
It specifies six conditions and five design principles for security decisions made without full understanding.
The paper argues that agentic systems widen the gap between system behavior and human comprehension by coupling partial understanding to delegated authority. It describes an epistemic attack surface with four interacting categories covering intent interpretation, evidence, delegated authority, and supervisor comprehension. It defines accountable partial understanding through six necessary conditions and derives five design principles, plus probe dimensions for evidence linkage and invalidation latency. Running examples include an AI coding agent, an autonomous cyber-defense agent, and an AI-mediated insulin delivery update.
- Four attack categories target intent, evidence, authority, and supervision.
- Probe dimensions are evidence linkage and invalidation latency.
- APU is distinguished from explainable AI and human-in-the-loop oversight.
- Examples include coding, cyber-defense, and insulin-delivery agents.
From the original article
1 min read at arxiv.orgAttackers profit from what defenders do not understand. Software security has treated the gap between system behavior and human comprehension as a condition to narrow through analysis and assurance, booking the remainder as accepted risk.
By Josiah Dykstra · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.