Optimal Data Reduction under Information-Theoretic Criteria

Abstract

Selecting an optimal subset of features or instances under an information theoretic criterion has become an effective preprocessing strategy for reducing data complexity while preserving essential information. This study investigates two representative problems within this paradigm: feature selection based on the maximum relevance minimum redundancy criterion, and instance selection grounded in the Kullback Leibler divergence. To address the intrinsic nonconvexities of these problems, we develop polyhedral relaxations that yield exact mixed integer linear programming formulations, thereby enabling globally optimal data reduction. By leveraging modern optimization techniques, we further design efficient algorithmic implementations capable of solving practically sized instances. Extensive numerical experiments on both real world and synthetic datasets demonstrate that our method efficiently solves data reduction problems to global optimality, significantly outperforming existing benchmark approaches.