Where the Numbers Come From: Auditing Evaluation in Provenance-Based Intrusion Detection
Audit finds provenance IDS scores overstate alarm precision; ThreaTrace labeled only 7 of 994 alarms as attacks.
The paper audits evaluation of provenance-based intrusion detectors, reviewing nine released implementations and running four with their own code. Neighbourhood credit inflated ThreaTrace precision to 0.938 even though only seven of 994 alarms carried the detector's own attack label. Dropping test-label checkpoint selection reduced attack-detection precision by 0.16 to 0.33 across forty-member word2vec configurations without changing detector order. A buffer-reuse defect also supplied a linear encoder with unintended degree-dependent inputs; the authors propose six STRICT reporting requirements and say the comparisons do not establish a universal ranking.
- Authors audited nine provenance-IDS implementations and executed four detectors.
- ThreaTrace precision was 0.938, but only 7 of 994 alarms carried its attack label.
- Removing test-label checkpoint selection cut detection precision by 0.16–0.33.
- A buffer-reuse defect gave a linear encoder unintended degree-dependent inputs.
- STRICT proposes six reporting checks and does not universally rank detectors.
- Products
- ThreaTrace
From the original article
1 min read at arxiv.orgReproducing a provenance-based intrusion detector's score does not establish what that score says about its emitted alarms or the information its encoder uses. We audit nine released implementations, execute four detectors using their own code, and isolate three measurement effects. First, a fixed-alert comparison separates label choice from neighbourhood credit: ThreaTrace reports precision 0.
By Jihwan Moon · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.