Automated Design Optimization via Strategic Search with Large Language Models

Abstract

Optimization methods have long advanced many fields, yet they struggle when faced with design problems where the search space and design parameters are difficult to define. Large language models (LLMs) offer a promising alternative by dynamically interpreting design spaces and leveraging encoded domain knowledge. To this end, we present AUTO: an iterative optimization framework that treats design optimization as a strategic search guided by LLM reasoning. The framework separates high-level planning by a Strategist agent from low-level implementation by concurrent Implementor agents, iteratively refining designs through explore-exploit strategies. We demonstrate AUTO on three GPU code optimization problems. For chemical kinetics, AUTO outperforms in-lab-optimized code by up to 1.74$\times$ for problem sizes up to $10^5$ cells. For matrix multiplication, AUTO achieves up to 94\% of cuBLAS double-precision performance. For KernelBench, we achieve speedups of up to 118$\times$ over PyTorch baselines across 29 problems spanning individual operators and full neural network architectures; however, cheating was frequently observed. A posteriori analysis reveals 50~--~70\% alignment with Bayesian optimization sampling strategies. All AUTO simulations ran within 100 iterations (about 10 hours), with estimated costs of \$15~--~159 per run for proprietary models. Furthermore, AUTO is built entirely on open-source LLMs and libraries, demonstrating affordability and data privacy. Given AUTO's generizability and flexibility, future work will explore domains beyond GPUs and supercomputing.