Audit says CAGE 4 defender scores measure less than assumed
Original title: What a CAGE 4 Score Measures: An Audit of a Learned Cyber-Defense Benchmark, Its Reference Defenders, and the Tables Built on Them
Busy-agent handling, unseeded policies, and unpublished scoring choices make published CAGE Challenge 4 comparisons hard to reproduce.
An audit of CAGE Challenge 4 reproduces reference defenders KEEP and H-MARL Expert and finds three measurement gaps. The simulator discards actions while an agent is busy and still charges for discarded Restores, while the documented observation hides the busy flag. Evaluation seeds the environment but not the policy, and five published tables omit the details needed to compare scores. The authors release their instrument, patches, traces, checkpoints, and a disclosure checklist.
- Disabling the busy check costs KEEP 99.58 points and H-MARL 145.98 per episode.
- Holding interventions until idle is worth 25.3 and 27.6 points per episode.
- 88.9% of per-episode outcome variance comes from the policy's own sampling.
- Repaired training ends short on two of three seeds, by 27 and 125 points.
From the original article
1 min read at arxiv.orgCAGE Challenge 4 is the benchmark on which learned multi-agent cyber defenders are compared. We reproduce its two reference defenders, KEEP and H-MARL Expert, in a pinned container and show that three things read from it measure less than they are taken to measure.
By James Rice · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.