SecLog-Bench measures LLM reasoning over cybersecurity logs
Original title: SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence
With scaffolded evidence, the best of eight models reached 44.44% accuracy and 45.07% Macro-F1.
SecLog-Bench is a benchmark for post-detection attack-progress reasoning by large language models. Its simulation platform covers 29 attack scenarios and generates multi-source logs from audit, application, and network data, with attack-stage labels and step-by-step evidence chains. Across eight mainstream models, the best accuracy was 44.44% and the highest Stage Reasoning F1 was 54.87%, with correct answers sometimes paired to flawed reasoning.
- The benchmark targets scale, single-source evidence, and missing reasoning annotations.
- Best Macro-F1 was 45.07%.
- Highest Stage Reasoning F1 was 54.87%.
- Prediction accuracy often diverged from reasoning quality.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process.
By Yifan Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.