Enhancing Model Based Derivative Free Optimization using Direct Search

Abstract

We consider single and multiobjective simulation-based optimization problems. Simulation-based optimization has traditionally used both model-based and search-based methods, often in isolation. Model-based methods include trust region approaches and Bayesian optimization, while search methods include genetic algorithms and Direct Search-type techniques. In this work, we propose a switching framework that leverages Direct Search methods to enhance the performance of any model-based optimizer. Our contributions are twofold. First, in the single-objective setting, we analyze and prove the asymptotic convergence of the proposed switching approach. Second, motivated by applications in machine learning, we consider both classification and regression problems, where the objectives span accuracy, computational time, algorithmic bias, and sparsity. The models range from complex neural networks and decision trees to simpler KNN-type baselines. For machine learning in particular, we also introduce a warm-starting mechanism using weights from previous hyperparameter or architectural configurations to exploit problem structure and accelerate training. Finally, beyond ML tasks, we evaluate the method on the standard CUTEr test problems and compare its performance against classical Bayesian and trust region solvers. We observe consistently strong numerical performance, suggesting promise for the proposed switching-based approach.