AndroTruth: A Reliable Benchmark Android Malware Dataset Derived from Technical Expert Reports

Abstract

Reliable family labels are essential for Android malware analysis, yet most widely used benchmarks derive such labels from aggregated VirusTotal engine outputs. Because antivirus vendors differ in detection logic, naming conventions, and signature updates, these labels are often inconsistent across engines and unstable over time, which introduces substantial noise into downstream empirical evaluation. To address this problem, we construct AndroTruth, an Android malware family benchmark whose labels are derived exclusively from traceable expert technical analysis reports rather than AV-consensus voting. AndroTruth spans 2016 to 2025 and contains 8,172 malware samples from 187 families. Our statistical results show that automated labeling tools can exhibit a misleading consensus failure mode in which AVClass2 and ClarAVy agree with each other yet jointly disagree with expert ground truth on 25.38% of samples with explicit labels from both tools. Experimental results show that, under expert-verified supervision, representative classifiers such as Meta-MAMC and AndMFC achieve accuracy above 96%. When trained with real-world AV-derived labels and evaluated against expert ground truth, however, their performance drops to only about 60% accuracy and about 35% macro-F1. ClarAVy confidenceaware filtering can improve family grouping quality, but cannot replace expert-verified labels for exact family naming. Together, these results demonstrate that label reliability is a first-order factor in Android malware family evaluation and highlight the need for expert-verified benchmarks.