Search arXiv⌕ Search

arXiv · 2611.10178

The Storage View Of KV Cache: Characterizing Cache Behavior Across LLM Workloads

Abstract

Modern LLM serving increasingly extends KV caches beyond GPU memory into host DRAM, SSDs, and remote storage, yet these stores remain largely governed by workload-agnostic policies such as LRU and fixed TTLs. We ask what execution context is needed by storage to make retention, reclamation, and placement decisions across heterogeneous workloads. We characterize KV-cache reuse, lifetime, and footprint at block granularity across 11 production and public traces spanning chat, API serving, RAG, coding agents, tool-using agents, and multi-lineage workflows. We examine reuse sources, recurrence timing and frequency, useful lifetimes, and storage consumption. Across workloads, KV state takes two recurring forms. Single-session state is reused within one execution: its footprint grows as sessions deepen and accumulates across concurrent sessions; it can remain useful through long idle periods but loses value at session completion. Multi-session state recurs across sessions of the same workload, outlives individual sessions, and spans hot prefixes and a large infrequent tail. While prior systems often exploit application-, agent-, or workflow-specific knowledge, our results suggest that three generic signals (a workload identifier, a session identifier, and session completion) expose much of the structure needed for KV-storage management. Trace-driven comparisons show that workload-specific recurrence estimates retain up to 22% more within-session hits and reduce dead state by up to 40%; a session completion signal eliminates the fundamental idle-time reclamation tradeoff for single-session state; and session-aware reuse classification identifies shared state 4-700x more selectively than request-level access counts. These results motivate a small cross-layer interface for workload-aware KV storage without application-specific semantics.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Talia Q. Chen, Daniel Waddington, Narangerelt Batsoyol, Vijay Chidambaram. 2026-11-06. The Storage View Of KV Cache: Characterizing Cache Behavior Across LLM Workloads. https://arxiv.org/abs/2611.10178

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

Towards Fully-fledged GPU Multitasking via Proactive Memory Scheduling

Limited HBM capacity has become the primary bottleneck for hosting a growing number of increasingly larger-scale GPU tasks. While demand paging extends capacity via host DRAM, it can incur order-of-magnitude slowdowns due to the massive working sets of GPU workloads. However, we observe that GPU memory accesses are inherently predictable from kernel launch arguments, prior executions, and the asynchronous execution nature. Leveraging this opportunity, we propose MSched, an OS-level scheduler that extends GPU context switching to include proactive working set preparation. This approach coalesces fragmented, eventual, and expensive page faults into a single efficient migration. MSched combines parameterized templates with page fault feedback to accurately predict working sets, and co-designs the task scheduler and memory manager to resolve global access order and enforce near-optimal page replacement. Across scientific, deep learning, and LLM workloads, MSched outperforms demand paging by up to 186.69x and whole-process swapping by up to 8.63x. MSched is open-source and has been integrated into https://github.com/XpuOS/nvxsched.

cs.OS↗

Memory Fit: Architectural Support for Page Placement by Learning Models of Good Behavior

Server main memory is increasingly heterogeneous, incorporating technologies such as zswap, CXL memory expanders, and emerging phase-change memories (PCM). To decide where to place pages, operating systems have to choose between efficient but coarse-grained access bits, out-of-band profiling, or costly runtime instrumentation. This paper proposes memory fit, a hardware abstraction that provides precise and accurate page placement information to an operating system. Memory fit measures how well a page "fits" each type of memory in the system. Memory fit separates the mechanism of measuring affinity between pages and memories from the OS policy of choosing which pages to move when. The paper describes soft trace matching (STM), a computationally and space efficient algorithm to compute memory fit. The key technical insight behind STM is that it can estimate memory fit by comparing accesses to a page with a learned model of "good" traces for a memory. We have implemented memory fit and STM in a BOOMv3 RISC-V processor running Linux 6.6. On extended GUPS, memory fit placements recover 2.2-3.8 times as much of the performance gap to optimal placement as Linux TPP. For Redis with equal DRAM and PCM capacities, placements selected by memory fit recover 80% of the performance gap, versus 19% for TPP, in a placement replay. Memory fit requires only 64 bits of state per 4 KiB page and can be updated on every main-memory access with only a 1.2% overhead of OS kernel notification processing.

cs.OS↗

MORTIS: Compiler-Guided KV-Cache Management for LLM Agent Programs

Existing agent serving systems divide each prompt into a reusable prefix and an opaque suffix, managing the KV-cache at this coarse granularity without distinguishing the components within each part. We observe that prompt components differ in which LLM calls share their values and how long those values remain reusable, a property we term intra-prompt heterogeneity. Our key insight is that, in frameworks that generate prompts from typed LLM function calls, these components correspond to program variables, allowing compiler analysis to recover their sharing patterns and lifetimes. We present MORTIS, a compiler-serving co-design that combines this analysis with runtime observations to improve KV-cache management. MORTIS reorders prompt components to expose additional prefix reuse and uses their predicted reuse and lifetimes to guide cache retention, eviction, and prefetching. Our evaluation shows that MORTIS achieves up to 1.99x job completion time (JCT) speedup and 1.97x throughput improvement over SGLang, with geometric-mean speedups of 1.45x and 1.49x, respectively.

cs.OS↗