HardEval grades models on live RHEL 8 STIG remediation
Original title: HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation
Models emit Bash scripts run on a VM and scored by hidden OpenSCAP/OVAL checks; fourteen setups differ widely.
HardEval is an execution-verified benchmark for language-model-assisted operating-system hardening, built from Red Hat Enterprise Linux 8 STIG controls that fail on an unhardened baseline. Models see only the required security outcome, not expert reference remediations or OVAL checks, and must produce a Bash script. The script runs on a live virtual machine and is graded fail-to-pass with the matching OpenSCAP/OVAL check. Fourteen frontier and open-weight configurations show large, uneven differences, and a common failure is plausible security code that misses the exact compliance condition.
- Tasks cover SSH, authentication policy, audit rules and kernel parameters.
- Reference remediations and OVAL checks are hidden from the model.
- For four open-weight configurations, multi-run Wilson intervals keep the observed order.
- HardEval is publicly available.
- Vendors
- Red Hat
From the original article
1 min read at arxiv.orgImplementing Security Technical Implementation Guide (STIG) controls remains a demanding operating-system administration task. Each natural-language security requirement must be translated into a precise and persistent configuration change. These changes span Secure Shell (SSH) configuration, authentication policy, audit rules, and kernel parameters.
By Manohar Vellala · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.