SAGEGAN: Style-Based Anomaly Detection with Gaussian Embeddings using Generative Adversarial Networks

Abstract

Malware evolves faster than rule-based and signature-driven detection pipelines. This paper presents SAGEGAN, a benign-only trained malware anomaly detection framework that converts portable executable files into compact three-channel images and models benign structure through style-conditioned adversarial reconstruction. The representation combines Hilbert-mapped byte values, benign-referenced byte-transition surprise, and entropy deviation from benign software. The model encodes each image into a layer-wise style tensor aligned with a seven-stage modulated generator, rather than a single latent bottleneck. A Gaussian style prior, moment-based prior alignment, and latent consistency are used to reduce mismatch between encoded benign styles and the generator's sampled manifold. For interpretation, a deterministic encoder pathway maps each executable to a fixed style tensor, enabling repeatable layer-wise family distance, gradient sensitivity, principal component, and class-behaviour analyses. On a self-collected portable executable corpus containing malware from 214 families, the Gaussian variant achieves 89.76% area under the receiver operating characteristic curve and 88.19% balanced accuracy, while the genome-style variant reaches 88.03% and 84.09%, respectively. Without refitting model weights, benign reference statistics, or decision thresholds, the same checkpoints are evaluated on DIKE, Microsoft BIG 2015, and Lester malware subsets. The results suggest that layer-wise style modelling supports both anomaly ranking and structured post hoc analysis of how malware families depart from the benign manifold.