arXiv subjects

David Simchi-Levi

Publications and source records attributed to David Simchi-Levi.

3 recordsLinked to original sources

arXiv2026

Multi-Turn LLM Conversations under the Least-Recently-Used Policy: Mean-Field Asymptotics and Hit Ratio Approximation

The major workloads in modern large language model (LLM) serving systems have shifted from single-shot LLM calls to multi-turn conversations, where new responses are generated based on the whole conversation history across all previous turns. The hit ratio, i.e., the average fraction of KV caches accessed directly from existing caches stored in high-bandwidth memory (HBM), is hence a crucial metric that governs system performance. Estimating the hit ratio is a highly nontrivial task due to the complex system dynamics, where the KV cache prefixes grow with turns and some must be evicted due to finite memory capacity. We formulate the system as a multi-turn conversation model under the least-recently-used (LRU) policy. Through a mean-field asymptotic framework, we prove that as the conversation arrival rate and the memory capacity grow proportionally to infinity, the hit ratio converges to a closed-form limit. Based on the characterization of the limit, we further propose a practical hit ratio estimator, and validate its accuracy by real LLM serving experiments on the Qwen3-8B model implemented on Ascend NPUs. Our results provide a theoretical foundation for the analysis of multi-turn LLM serving systems and a practical guideline for memory capacity provisioning.

cs.PF↗

arXiv2026

Refined Thompson Learning for Adaptive Bandits: Power-Efficient Flexibility Scheduling Across Data Centers

The rapid growth of large-scale AI workloads in data centers has placed increasing pressure on power grids in recent years. Since power systems must continuously balance supply and demand, there is growing interests in leveraging data-center workload flexibility as a grid service. We propose a contextual restless multi-armed bandit (CRMAB) framework in which a grid operator requests load reductions without observing internal job-scheduling decisions. Under index-ability guarantee, each data center or physical machine is modeled as a Markov decision process (MDP) over a cyclic virtual-machine (VM) job queue, with unknown rewards and transition dynamics learned online using Thompson sampling and Whittle-index policies. To improve learning under sparse and noisy observations, the framework augments an adaptive Thompson--Whittle (TW) policy with domain-informed transition priors and gated prior mixing. In baseline experiments, the best adaptive refined variant achieves 91.4\% of the oracle reward after 100 rounds and 96.8\% after 1,000 rounds. Across a 16-setting stress test spanning different state-space sizes and levels of contextual noise, the best refined variant consistently outperforms the original TW policy with high confidence while remaining competitive with EXP4. A graph-based prior further incorporates data-center hardware constraints, including computing-resource limits. Overall, the results demonstrate the economic potential of data-center flexibility as a grid service and highlight the importance of high-quality, open-source AI workload traces for developing and evaluating such services.

cs.CE↗

arXiv2026

AI-Generated Measurements for Identification and Inference with Missing Data: A Weak Shadow Variable Approach

Across business and social science applications, outcomes are often missing in ways that depend on the unobserved outcomes themselves. In service systems, for example, whether a customer submits a rating depends on the rating they would have provided. Such missing-not-at-random (MNAR) mechanisms make population quantities difficult to identify without strong assumptions on the observation process. Meanwhile, rich unstructured data, such as customer interaction histories, are increasingly available and can be used to construct structured measurements using tools such as large language models (LLMs). In this work, we develop an assumption-lean partial identification framework that uses such measurements as weak shadow variables, defined as outcome-informative proxies that are conditionally independent of missingness given the true outcome and observed covariates. Importantly, they need not accurately predict missing outcomes or satisfy the completeness requirement in the classical shadow variable literature. For identification, we characterize sharp bounds on population quantities through a pair of linear programs. For estimation and inference, we propose a localized penalized estimator that remains feasible under sampling error, and a subsampling algorithm for constructing confidence intervals. In semi-synthetic experiments using real customer-service dialogues, weak-shadow-variable intervals are about 89\% narrower than those without auxiliary information, while their midpoints have around 41\% lower estimation error than classical MNAR methods.

stat.ML↗