Frequency-Domain Membership Inference Does Not Beat Confidence Thresholding: A Band-Resolved Analysis of Why

Abstract

A widely repeated account of membership inference holds that models lean on fine-grained detail for samples they have memorized, so degrading an input's resolution should expose membership through how quickly a prediction falls apart. We test that account band by band. A monotone family of zero-phase low-pass filters is applied to each query image, the target's confidence on its original prediction is read at every step, and the resulting decay is scored in a dozen ways. Two federated ResNet-18 targets are studied, one trained normally and one pushed to a 45.1-point generalization gap. The account does not survive. On the normally trained target the best profile score reaches an AUC of 0.513 against 0.537 for a single confidence threshold, and on the overfit target 0.735 against 0.943. A search over 1206 candidate scores, with direction chosen on a held-out half and a fitted combination as an upper bound, produces no score that beats the threshold in either regime. The failure has a mechanism: the mean confidence gap between members and non-members grows as detail is removed, exactly as the hypothesis predicts, but the spread grows faster, so discriminability falls from the first step onward. Decomposing each image into phase and magnitude shows why no band-localized operation can work, since the advantage travels with image structure, surviving at 0.567 when the phase is kept and collapsing to 0.492 when only the magnitude is. Membership is a whole-image identity effect rather than an accumulation of spectral energy in any band, and defenses aimed at high-frequency reliance are aimed at the wrong target.