Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2512.09427

ODMA: On-Demand Memory Allocation Strategy for LLM Serving on LPDDR-Class Accelerators

Guoqiang Zou·Wanyu Wang·Hao Zheng·Longxiang Yin·Yinhe Han
View original source ↗Read source document ↗Export metadata ↓

Abstract

Existing memory management techniques severely hinder efficient Large Language Model serving on accelerators constrained by poor random-access bandwidth.While static pre-allocation preserves memory contiguity,it incurs significant overhead due to worst-case provisioning.Conversely,fine-grained paging mitigates this overhead but relies on HBM's high random-access tolerance, making it unsuitable for LPDDR systems where non-sequential access rapidly degrades bandwidth. Furthermore, prior works typically assume static distributions and HBM characteristics, thereby failing to resolve the critical fragmentation and bandwidth constraints inherent to LPDDR hardware. We present ODMA, an on-demand memory allocation strategy tailored for random-access-constrained accelerators, such as the Cambricon MLU series.ODMA advances generation-length prediction by addressing two critical limitations in production workloads: (i) distribution drift that invalidates static bucket boundaries, and (ii) performance fragility under heavy-tailed request patterns. ODMA integrates a lightweight length predictor with adaptive bucket partitioning and a fallback safety pool. Bucket boundaries are dynamically recalibrated via online histograms to maximize utilization, while the safety pool ensures robustness against prediction errors. On Alpaca and Google-NQ benchmarks, ODMA improves S3's prediction accuracy from 98.60% to 99.55% and 82.68% to 93.36%, respectively. Deployment with DeepSeek-R1-Distill-Qwen-7B on Cambricon MLU370-X4 accelerators demonstrates that ODMA increases KV-cache utilization by up to 19.25% (absolute) and throughput (TPS) by 23-27% over static baselines, validating the efficacy of predictor-driven contiguous allocation for LPDDR-class devices.

Explore related subjects

cs.ARcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Guoqiang Zou, Wanyu Wang, Hao Zheng, Longxiang Yin, Yinhe Han. 2026-04-21. ODMA: On-Demand Memory Allocation Strategy for LLM Serving on LPDDR-Class Accelerators. https://arxiv.org/abs/2512.09427

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A System Architecture for Low Latency Multiprogramming Quantum Computing

As quantum systems scale, multiprogramming quantum computing (MPQC) provides a practical way to improve device utilization and throughput. However, because quantum executables are device-dependent, non-portable across qubit regions, and highly susceptible to noise and crosstalk, current MPQC pipelines rely on expensive online compilation to co-optimize concurrently running programs. This online step dominates runtime and impedes low-latency deployments for practical, real-world workloads in the future, such as repeatedly invoked quantum neural network (QNN) services. We present FLAMENCO, a fidelity-aware multi-version compilation system that enables independent offline compilation and low-latency multiprogramming at runtime. \textbf{At the architecture level}, the system abstracts devices into compute units to reduce the search space of region allocation. \textbf{At compile time}, it generates diverse executable versions for each program---each bound to a distinct qubit region---allowing dynamic region selection at runtime and overcoming non-portability. \textbf{At runtime}, it employs a lightweight orchestrator that uses post-compilation fidelity metrics to avoid conflicts and mitigate crosstalk, supporting conflict-free co-execution without online co-optimization. Evaluations show that FLAMENCO achieves over 5$\times$ runtime speedup in post-scheduling execution while maintaining comparable execution fidelity on common-success workloads. When integrated into existing scheduler-coupled systems, it raises workload-level conflict-free orchestration ratio from 0.183 to 1.000 for HyperQ and from 0.050 to 0.400 for QOS.

cs.AR↗
arXiv2026

Microflow: Microarchitectural Causal Observability for Deep Cross-Layer Analysis and Optimization

Modern computer architecture relies heavily on simulation to identify bottlenecks and evaluate optimizations. However, existing microarchitectural performance analysis methods are fundamentally limited by an instruction-centric paradigm that captures only downstream symptoms while leaving the true microarchitectural root cause opaque. Because modern processors are governed by complex interactions across non-instruction entities like prefetchers, replacement policies, and shared queue occupancies, instruction-centric frameworks miss the mechanisms that dictate performance. To eliminate this blind spot, we present Microflow, a framework that achieves causal observability in microarchitectural simulation. To address this, we introduce the Microflow intermediate representation (MFIR), which models execution through microarchitecture-tailored core abstractions such as flows, resource tenancies, and causal edges. By compiling simulation runs into a relational causal database, Microflow decouples tracing from analytical processing. This transforms complex diagnostics into expressive queries, enabling architects to trace performance symptoms directly to hardware root causes without developing bespoke analysis scripts or running costly re-simulations for every new question. We demonstrate that Microflow solves pathologies opaque to conventional tools. Across CVP-1 benchmarks, Microflow decomposes a 22% prefetcher oracle headroom by attributing 61.4% of stall mass to specific hardware prefetcher decisions, yielding a 2.31% average speedup (peaking at 25.11%). Furthermore, it exposes the hidden pipeline-blocking residue of wrong-path execution with high portability and precision across simulators.

cs.AR↗
arXiv2026

On the Limits of Machine-Learned Ranking for Modern Microarchitectural Policies

Machine-learning predictors estimate processor performance far faster than cycle-level simulation. For design-space exploration, however, the valuable test is not merely reproducing the usual hardware ordering, but identifying how different hardware configurations rank on individual program phases. We evaluate four ML-predictors in two design regimes: \emph{Structural Parameters} (SP), varying hardware resources such as issue width, ROB size, and cache capacity; and \emph{Behavioral Policies} (BP), varying prefetching and replacement algorithms. In the SP regime, aggregate ranking is strong, yet counter-intuitive windows(CIW)---where the configuration expected to be slower is faster---constitute $22.4\%$ of non-tied windows across five pairs with a clear architectural prior. CIW match across these pairs is only $23.3$--$39.9\%$; every point estimate is below the $50\%$ random strict-ordering reference. The BP regime presents a different failure: ground-truth ties cover $37.8\%$ of pair-windows, most strict pairs have margins of only a few cycles, and no model family reliably beats a feature-free majority baseline. NeuroScalar and SimNet fall below that baseline, Concorde is statistically tied with it, and the best selected OneDSE head improves by only $2.1$ percentage points. Accuracy rises mainly at large margins. We further show that this failure is not a matter of model capacity: an information-theoretic analysis reveals that when ranking outcomes depend on hidden microarchitectural state absent from the instruction stream, no trace-based predictor can exceed the Bayes accuracy determined by observable inputs alone. Thus high cycle or aggregate ranking accuracy can reflect mastery of easy, high-margin cases while missing the local reversals that carry the most architectural insight and for which cycle-level simulation remains indispensable.

cs.AR↗