Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.08378

Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth Flash

Jaehoon Yang·Jeongmin Lee·Haneul Park·Seung Yul Lee·Nam Sung Kim·Jae W. Lee
View original source ↗Read source document ↗Export metadata ↓

Abstract

Large language model (LLM) serving is increasingly dominated by agentic workloads, in which agents and their sub-agents accumulate context as KV cache across many requests, consuming substantial memory. High-bandwidth flash (HBF) is a promising solution, providing an order of magnitude greater capacity at HBM-class read bandwidth, but its finite write endurance is the key limiting factor. Our key insight is that KV cache should be placed across HBM and HBF by its lifetime. Placing shorter-lived data in HBM lets HBM absorb more of an agent run's writes and sends less of them to HBF. As the lifetime of KV cache in agentic serving is dictated by the harness, the program that orchestrates the agents, we analyze its behavior and identify three axes along which lifetime diverges, temporal, structural, and inter-worker. Guided by these observations, we present Lachesis, a lifetime-aware KV cache placement layer between the agent harness and the serving engine. At write time, it places each segment in HBM or HBF according to its lifetime, and frees its blocks once the segment is no longer read. In trace-driven simulation, Lachesis extends HBF lifetime by 1.19-3.13x over HBM-first placement, reaching 3.3-12.2 device-years. Even under continuous 24x7 operation at the full load a tight SLO admits, HBF outlasts its five-year warranty on the multi-agent trace.

Explore related subjects

cs.ARcs.DC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jaehoon Yang, Jeongmin Lee, Haneul Park, Seung Yul Lee, Nam Sung Kim, Jae W. Lee. 2026-10-06. Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth Flash. https://arxiv.org/abs/2610.08378

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

An Adaptive Heterogeneous Architecture for High-Ratio, High-Throughput Lossless Compression

Modern lossless compression enforces an acute dichotomy: industrial streaming codecs (e.g., Zstandard) prioritize throughput (10 to 1000 MiB/s) at the expense of ratio, while context-mixing algorithms achieve superior density at serial speeds (0.1 to 1 MiB/s). We present GPX, an adaptive multi-tier compression architecture for structured data streams. GPX operates transparently across arbitrary stream lengths (N >= 1,024 B, identity fallback below 1 KiB), combining an autonomous structural decision probe (6.7--486.2 us across 49 benchmark instances, median 31.6 us, <0.05% overhead), cache-tiled AVX2 SIMD reversible domain transforms, multi-hypothesis sequence optimization with in-flight frame MDL gating, and adaptive entropy boundaries. On canonical Silesia (202.12 MiB) under a 7-repeat protocol (W = 6 pinned P-cores), GPX Track A produces standard RFC 8878-compliant .zst streams compressing to 56.20 MiB at 191.44 MiB/s with native line-speed decompression (5,182.79 MiB/s at W=6, 1,706.52 MiB/s at W=1 on unmodified libzstd, representing 124.1% single-core speed of Stock Level 9). GPX Track B achieves 54.49 MiB at 299.23 MiB/s (CI [295.3, 305.7]) with 4,919.14 MiB/s decode, strictly dominating Stock Levels 9--14 across size and speed (1.04x faster and 1.95 MB smaller than Level 9; 151.78 KB smaller and 8.24x faster than Level 14). GPX Track A+B achieves 54.37 MiB at 204.87 MiB/s (CI [199.6, 210.3]) with 4,599.00 MiB/s decode, strictly dominating Levels 10--14 under 95% Bootstrap CIs (276.07 KB smaller than Level 14, 1.19x to 5.64x faster). Generalization is confirmed across 6 suites (49 instances, 48 streams): Canterbury (-0.65%), Calgary (-1.66%), PE binaries (-14.10%), and Transformer BFloat16 tensors (-10.61%, saving 3.28 MB vs Stock L9). Downstream SIMT nibble unpacking sustains 103.62 GiB/s (1.05x PCIe speedup). All outputs are bit-exact verified via SHA-256.

cs.AR↗
arXiv2026

Terracotta: Enabling the Adoption of New DRAM Techniques via a Flexible DRAM Interface and Memory Controller

DRAM continues to limit the performance, energy efficiency, and robustness of modern systems. Many prior works propose DRAM techniques that support in-DRAM computation, improve memory access latency and parallelism, and enhance DRAM maintenance and reliability. However, adopting each new technique requires repeated modifications to the rigid DRAM interface and memory controller, hindering its deployment. Our goal is to reduce these repeated modifications. We observe that the DRAM commands and memory controller structures of many DRAM techniques are similar. Our key idea is to use these similarities to compose a set of primitives for implementing diverse DRAM techniques. We propose Terracotta, a new framework with two flexible components: (i) custom command extensions that let DRAM vendors define new commands within a single, standardized interface, and (ii) a programmable memory controller that system designers can program to support new DRAM techniques post-silicon. Together, these enable deployment by configuring the memory controller instead of modifying the interface and controller. We design Terracotta for a DDR5-based system and evaluate its performance, energy, and hardware complexity. For four DRAM techniques from four distinct domains (processing-using-DRAM, low-cost DRAM maintenance, subarray-level parallelism, and latency reduction), Terracotta retains almost all of the performance benefits (>96%) of custom implementations. A Terracotta-based composition of two techniques outperforms the Terracotta-based implementation of each technique alone, demonstrating the benefits of adding techniques without repeated interface and controller modifications. Terracotta incurs low DRAM energy (0.6-3.2%), area (0.03%), and power (0.56%) overheads in a high-end server-grade processor. Terracotta's source code is freely available at https://github.com/CMU-SAFARI/Terracotta.

cs.AR↗
arXiv2026

Beyond No-Good Benders Cuts: Exact Realizability for Discretely Tunable Clock Trees

Useful-skew schedules can satisfy timing constraints yet remain unrealizable by a fixed clock tree with discrete tuning choices. We formulate this mismatch as exact membership in a finite relative-latency set and develop a checkable feedback interface between the scheduler and the tree model. Arithmetic certificates explain unrealizable targets, while tree-specific contraction reduces the structural size of the exact relations projected onto selected sinks. These relations become realizability cuts that can exclude more candidates than a no-good on the same certificate support, including discrete holes that linear inequalities cannot separate. Controlled experiments confirm fewer oracle calls and faster projection construction. They also expose important limits: globally minimum certificates can cost more than they save, compact arithmetic feedback can fail on non-parity obstructions, and direct monolithic optimization remains faster on the tested additive models. The contribution is an exact, independently verifiable scheduler--tree interface, rather than a claim of universal solver acceleration or physical signoff.

cs.AR↗