EvoMaestro: Toward Interpretable and Steerable LLM-Driven Program Evolution

Abstract

Recent program evolution systems use large language models (LLMs) to generate and iteratively improve populations of programs, producing striking advances in mathematics, algorithm design, and scientific computing. Yet these systems largely operate as fully automated black boxes. As populations grow, domain experts must make sense of the scores, code changes, reasoning, and algorithmic ideas across many programs, while current interfaces provide limited support for understanding or redirecting the evolution. We characterize this need to understand and steer populations of evolving algorithmic ideas as semantic oversight. A formative study with 8 domain experts yields six design requirements for this emerging human-computer interaction problem. We then propose a steerable program evolution framework that lets expert judgments shape subsequent evolution. Built on this framework, EvoMaestro is an interactive visual analytics system that organizes evolution information from population overview to source code. It helps experts locate and compare noteworthy programs, guide evolution with natural language, combine promising ideas, and prune unproductive directions. A seven-day system demonstration illustrates how an expert applied these capabilities throughout a long-running evolution process. A within-subjects study with 12 participants shows that EvoMaestro improves users' understanding of evolution processes, reduces cognitive workload, and supports expert steering. These findings suggest that preserving expert agency in open-ended LLM-driven search requires support for both informed judgment and the ability to shape subsequent automation.