Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks

Abstract

Classifying malware binaries rendered as images is a popular approach to malware family identification, and papers on the standard benchmarks for this task report accuracies of 97-99%. In this paper we audit ten of these benchmarks for near-duplicate images and test-train leakage, and retrain standard classifiers under naive and content-aware splits. Deduplicating by file hash, which is the standard practice, does not prevent leakage, because distinct binaries render to identical images. Deduplicating on image content reduces top-1 accuracy by up to 8 points and macro-F1 by up to 15 points on the released benchmarks we find leaky. The rendering step also adds duplication, since byteplots of the same binaries contain 19.7% near-duplicates where static feature vectors contain 2.2%, and changing the width at which bytes are wrapped into rows reduces accuracy by 20 to 33 points, although no byte changes. We recommend that content-level duplication and a non-image baseline be reported alongside accuracy.