An Evaluation of the Semantic Understanding Capabilities of Large Language Models for Web Attack Payloads
PayloadSemBench evaluates how deeply large language models understand 240 web attack payloads across four semantic layers.
Researchers built PayloadSemBench, a four-layer benchmark of 240 web attack payloads whose labels were set through two rounds of anchor calibration and checked by a fourth expert. Type identification and intent understanding were generally strong, while severity assessment was the main weakness. Obfuscation degraded models unevenly, especially intent explanation and reconstruction of specific obfuscation methods, and only 12.5% to 50% of missed detections were tied to semantic-understanding failures. An external check on 180 production WAF alerts and real application requests reproduced the same uneven capability profile.
- PayloadSemBench scores 240 payloads on four semantic layers.
- Severity assessment was weaker than type and intent recognition.
- Obfuscation hurt intent explanation more than other layers.
- Semantic misses explained only 12.5% to 50% of detection failures.
- A 180-record WAF and application set reproduced the profile.
From the original article
1 min read at arxiv.orgComputer vision services delivered through Web interfaces and APIs process textual requests for image-resource acquisition, inference-task configuration, and result management, making Web attack-payload analysis relevant to their deployment security. Large language models (LLMs) can identify payload types and explain attack intent.
By Hao Sun · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.