Hybrid scanner finds malicious AI agent skills with fewer false positives
Original title: The Devil Wears Markdown: Detecting Malicious Intent Hidden in AI Agent Skill Instructions through Hybrid Static, Dynamic, and Agentic Analysis
The full pipeline detects 68 of 80 malicious skills and cuts false positives from 40 to five versus Cisco's LLM mode.
Researchers evaluate detection of malicious intent spread across agent-skill instructions, code, metadata, and runtime behavior. On 140 held-out skills, default Cisco Skill Scanner recalls 0.91 but flags 33 of 60 benign skills; its Qwen3:8B configuration recalls 0.85 with 40 false positives. A hybrid static, sandboxed, and LLM framework raises F1 from 0.70 to 0.89 when static and dynamic evidence are combined. Its full pipeline reaches 0.88 accuracy, 0.93 precision, and 0.85 recall.
Unconfirmed: The system-level comparison does not isolate effects of the model or architecture.
- The test set contains 80 malicious and 60 benign skills.
- Dynamic evidence alone raises F1 from 0.70 to 0.85.
- Combined evidence yields five false positives among 60 benign skills.
- The comparison does not isolate model or architecture effects.
- Vendors
- Cisco
- Products
- Cisco Skill Scanner
- AI models
- Qwen3:8B
From the original article
1 min read at arxiv.orgAgentic AI ecosystems increasingly rely on reusable agent skills that combine executable code, natural-language instructions, configuration artifacts, and external tool integrations. These hybrid artifacts create software supply-chain risks because malicious intent may be distributed across instructions, code, metadata, and runtime behavior, making isolated alerts difficult to interpret.
By Thanh M. Truong-Le · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.