MARS: Malware Analysis with Rule-Based Scoring of LLM Claims
MARS study finds direct LLM malware verdicts outperform externally scored behavioral claims.
MARS compares direct LLM malware classification with single-pass behavioral claims scored by an external policy, using the same evidence collector and identical static bundles. The evaluation covers 1,195 PE and ELF binaries in 1,001 near-duplicate clusters and six language models, with deterministic rules as a baseline. Direct classification was more accurate for every model, by 3.7 to 20.9 percentage points on paired usable outputs, and had higher malicious-alert recall in ten of twelve platform-model combinations. Saved claims still support exact verdict reproduction and later policy revision, including outputs from two models later withdrawn by their provider.
- Tested 1,195 PE and ELF binaries in 1,001 near-duplicate clusters.
- Direct classification beat claim scoring by 3.7 to 20.9 percentage points.
- Malicious-alert recall was higher for direct classification in ten of twelve settings.
- Retained claims allow later policy revision without another model call.
Full article246 words · extracted from arxiv.org · click to collapse
Large language models can triage malware through direct verdicts or behavioral claims scored by an external policy. We present MARS, a malware triage framework, and compare direct classification with single-pass claim scoring using the same evidence collector and identical static evidence bundles for each model. The evaluation covers 1,195 PE and ELF binaries grouped into 1,001 near-duplicate clusters and six language models, with deterministic rules providing a baseline. Direct classification is more accurate for all six models. On samples with usable outputs from both paths, its accuracy advantage ranges from 3.7 to 20.9 percentage points, with all 95% cluster-bootstrap confidence intervals for the differences above zero. It also achieves higher malicious alert recall in ten of twelve platform and model combinations. Claim mediation provides no consistent reduction in performance variation across models. Separate subset studies find more consistent alert decisions for direct classification and a larger recall loss for the claim path when predefined indicator fields are removed. In a family identification probe, claims yield higher accuracy than verdict labels but lower accuracy than evidence text. Retained claims expose the inputs to verdict computation and permit policy revision without another model call. We reproduce archived verdicts exactly and apply a revised policy to the same records, including outputs from two additional models withdrawn by their provider. Under the evaluated claim taxonomy and additive policy, these results favor direct classification when only a verdict is required, while demonstrating that retained claims support explicit policy inspection and revision.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.09553