What researchers learned about building an LLM security workflow
Oslo and FFI researchers show structured agentic workflows lift LLM alert-triage accuracy from 0% to about 93% on malicious cases.
Researchers at the University of Oslo and the Norwegian Defence Research Establishment tested GPT-5-mini, Claude 3 Haiku, Qwen3:30B, and Gemma 3:27B on alerts from the AIT Log Data Set V1.1; given only alert descriptions and log summaries, all four models correctly flagged zero percent of true-positive cases involving reconnaissance, brute-force logins, and initial access. Wrapping the same models in a workflow with constrained SQL queries over Suricata logs, an evidence summarizer, and a verdict stage with revision loops raised malicious-case accuracy to an average of 93 percent, with GPT-5-mini identifying every malicious case across 100 runs. The authors flag it as a proof-of-concept on one synthetic scenario and note models skewed conservative on benign alerts, with GPT-5-mini marking every benign case uncertain.
- Unstructured summaries produced zero true-positive detections across all four tested models.
- Constrained tools and defined investigation steps, not larger prompts, drove the accuracy gain.
- GPT-5-mini caught all 100 malicious runs but classified every benign case as uncertain.
- Lesson generalizes: evaluate the system built around the model, not just the model.
- Products
- SuricataAIT Log Data Set V1.1
- Countries
- Norway
From the original article
3 min read at helpnetsecurity.com
Security operations centers are running into the same wall everywhere. Detection tools generate more alerts than analysts can work through, and the early stages of any investigation involve pulling together logs from several sources to decide whether something is worth escalating.
By @helpnetsecurity · helpnetsecurity.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.