Feature Selection in Nonlinear SVMs via Local Search and Submodular Optimization

Abstract

Embedded feature selection is a classical approach to interpretable machine learning, aiming to iden- tify the most informative variables while simultaneously training the predictive model. We address the problem of enforcing an exact limit on the number of features used in nonlinear Support Vector Machines (SVMs) by means of a hard cardinality constraint. Whereas existing hard-constraint formulations apply only to the primal formulation of linear SVMs, we embed the cardinality constraint directly into the dual formulation of a nonlinear SVM. This ensures strict control over the number of selected features while still leveraging kernelization. We model the problem as a mixed-integer nonlinear program (MINLP). Our first contribution is a local-search metaheuristic applicable to any nonlinear kernel, which exploits the decomposable structure of the formulation to efficiently explore the binary decision space. Our second and main contribution is a decomposition framework that alternates optimization over continuous and binary variables. For polynomial kernels, we reformulate the binary subproblem as the maximization of a monotone submodular function under a cardinality constraint, enabling the integration of scalable greedy algorithms into the alternating scheme. Computational experiments on benchmark datasets show that the proposed algorithms significantly outperform existing MINLP solvers and standard heuristics, delivering higher-quality solutions within practical runtimes.