Malware-as-image benchmarks leak near-duplicate samples into tests
Original title: Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks
Content-aware deduplication cuts top-1 accuracy by up to 8 points and macro-F1 by up to 15 on leaky benchmarks.
The paper audits ten benchmarks used to classify malware binaries rendered as images, a setting where published work reports 97-99% accuracy. Deduplicating by file hash misses leakage because distinct binaries can render as identical images. On leaky released benchmarks, image-content deduplication lowers top-1 accuracy by up to 8 points and macro-F1 by up to 15. Byteplots contain 19.7% near-duplicates versus 2.2% for static feature vectors, and changing wrap width drops accuracy by 20 to 33 points without changing bytes.
- Ten malware-as-image benchmarks were audited for duplicates and leakage.
- Published papers on these benchmarks report accuracies of 97-99%.
- Byteplot near-duplicates are 19.7%, against 2.2% for static features.
- Changing row wrap width reduces accuracy by 20 to 33 points.
- Authors recommend reporting content-level duplication and a non-image baseline.
From the original article
1 min read at arxiv.orgClassifying malware binaries rendered as images is a popular approach to malware family identification, and papers on the standard benchmarks for this task report accuracies of 97-99%. In this paper we audit ten of these benchmarks for near-duplicate images and test-train leakage, and retrain standard classifiers under naive and content-aware splits.
By Stefan Larson · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.