Exploiting the Flexibility of LLM Inference for Frequency Regulation Service

Abstract

While the widespread deployment of Large Language Models (LLMs) has significantly driven up power demands in computing clusters, the unique execution characteristics of LLM inference offer a novel opportunity to utilize these clusters as controllable loads. In this paper, we investigate the potential of LLM inference cluster to provide frequency regulation services through inference request scheduling. First, we model the problem as a hierarchical Markov decision process that includes request-to-replica dispatching and token-level resource allocation. Then, we introduce a reward redistribution network to address the temporal asynchrony between millisecond-level scheduling decisions and second-level frequency regulation signals. Furthermore, we develop an optimization framework based on a mean-field approximation to enable coordination among a large number of replicas under high request concurrency. Simulations using real-world data demonstrate that the proposed approach accurately tracks frequency regulation signals without compromising service-level objectives.