Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2606.25091

Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off

Yuan Lyu·Bharath Irukulapati·Jaya Prakash Champati
View original source ↗Read source document ↗Export metadata ↓

Abstract

Speculative decoding (SD) accelerates LLM inference by $1.5$-$3$ times when the draft and target models are co-located. This has motivated a distributed variant (DSD) that places the draft model on an edge device while the target stays in the cloud. We show with closed-form inequalities that DSD's per-request latency benefit is limited under WAN edge-cloud communication. If the server can host both models, co-located SD has lower latency and communication than synchronous DSD, with the same per-output FLOPs and model-weight memory. Pipelining can make DSD competitive with co-located SD only in low-RTT regimes where the round trip is shorter than the edge drafting time window; at WAN RTTs, the cloud round trip remains too large for pipelined DSD to beat co-located SD. Against cloud autoregressive decoding, DSD can reduce latency only inside a bounded window given the target-model speed, acceptance rate, and RTT. DSD is also infeasible against closed-source APIs without a verifier-only interface. The main case for DSD appears in multi-tenant capacity. Under cross-client overlap, offloading draft compute lets a saturated cloud server sustain $(1 + γ\,t_d/t_v)$ times more concurrent clients at the same per-client rate, where $γ$ is the speculation length and $t_d, t_v$ are the per-step draft and verification times. DSD should therefore be evaluated primarily by multi-tenant capacity and server throughput, not only by single-request latency.

Explore related subjects

cs.DC

Keep this discovery

BibTeXRIS

Yuan Lyu, Bharath Irukulapati, Jaya Prakash Champati. 2026-09-05. Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off. https://arxiv.org/abs/2606.25091

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.DC subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

Beyond Lemma Sharing -- Novel Parallelization Strategies for Property Directed Reachability

Property Directed Reachability (PDR) is a commonly used technique for automated hardware model checking, yet efficiently parallelizing it remains a significant challenge. Existing approaches, such as lemma sharing, often suffer from limited scalability as processor counts increase. In this work, we present two novel sharing-based parallelization strategies, preemptive propagation and ARPOS, and compare their performance with classical lemma sharing. To this end, we develop an asynchronous MPI-based message passing framework for the state-of-the-art rIC3 hardware model checker. Experimental results on the 2025 Hardware Model Checking competition benchmark demonstrate that our preemptive propagation strategy yields a significant performance boost over classical lemma sharing.

cs.DC↗
arXiv2026

Towards Decentralized Registries for Assets Metadata Information

The effort to tokenize non-currency assets faces several hurdles, including the lack of a scalable decentralized computing infrastructure to manage asset-related metadata. While the centralized securities depository model has served the financial industry well for several decades, the vision of tokenization at a global scale requires new infrastructure that enables distributed control while protecting the integrity of asset-related metadata, regardless of where it is stored. In this paper, we discuss the decentralized artifacts metadata registry model for tokenized assets as a possible direction for the financial industry seeking to embrace tokenization. The artifacts metadata registries extend the function of the traditional CSD, and could in fact be a new type of service offered by CSDs around the world.

cs.DC↗
arXiv2026

JAXMg: A multi-GPU linear solver in JAX

Solving large dense linear systems and eigenvalue problems is a core requirement in many areas of scientific computing, but scaling these operations beyond a single GPU remains challenging within modern programming frameworks. While highly optimized multi-GPU solver libraries exist, they are typically difficult to integrate into composable, just-in-time (JIT) compiled Python workflows. JAXMg provides distributed dense linear algebra for JAX, enabling linear solves and decompositions for matrices that exceed single-GPU memory limits. By interfacing JAX with NVIDIA's cuSOLVERMp through an XLA Foreign Function Interface, JAXMg exposes distributed GPU routines as JIT-compatible JAX primitives. This design allows scalable linear algebra to be embedded directly within JAX programs, preserving composability with JAX transformations and enabling multi-GPU and multi-node execution in end-to-end scientific workflows.

cs.DC↗