Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2508.12961

WANify: Gauging and Balancing Runtime WAN Bandwidth for Geo-distributed Data Analytics

Anshuman Das Mohapatra·Kwangsung Oh
View original source ↗Read source document ↗Export metadata ↓

Abstract

Accurate wide area network (WAN) bandwidth (BW) is essential for geo-distributed data analytics (GDA) systems to make optimal decisions such as data and task placement to improve performance. Existing GDA systems, however, measure WAN BW statically and independently between data centers (DCs), while data transfer occurs dynamically and simultaneously among DCs during workload execution. Also, they use a single connection WAN BW that cannot capture actual WAN capacities between distant DCs. Such inaccurate WAN BWs yield sub-optimal decisions, inflating overall query latency and cost. In this paper, we present WANify, a new framework that precisely and dynamically gauges achievable runtime WAN BW using a machine learning prediction scheme, decision tree-based Random Forest. This helps GDA systems make better decisions yielding reduced latency and costs including WAN BW monitoring costs. Based on predicted runtime WAN BW, WANify determines the optimal number of heterogeneous parallel connections for data transfer among DCs. This approach improves performance without additional, or even at reduced cost, by fully exploiting available WAN capacities. In addition, WANify considers dynamics like network and workloads, and heterogeneity like skewed data, heterogeneous compute resources, and a varying number of DCs while making decisions. The WANify prototype running on state-of-the-art GDA systems is evaluated on AWS with 8 geo-distributed DCs. Results show that WANify enhances WAN throughput by balancing between the strongest and weakest WAN links, enabling GDA systems to reduce latency and cost by up to 26% and 16% respectively with minimal effort, all while handling dynamics and heterogeneity efficiently.

Explore related subjects

cs.DC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Anshuman Das Mohapatra, Kwangsung Oh. 2025-08-18. WANify: Gauging and Balancing Runtime WAN Bandwidth for Geo-distributed Data Analytics. https://doi.org/10.1109/iiswc66894.2025.00015

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

The HEAL Data Platform

Objective: The objective was to develop a cloud-based, federated system to serve as a single point of search, discovery and analysis for data generated under the NIH Helping to End Addiction Long-term (HEAL) Initiative. Materials and methods: The HEAL Data Platform is built on the open source Gen3 platform, utilizing a small set of framework services and exposed APIs to interoperate with both NIH and non-NIH data repositories. Framework services include those for authentication and authorization, creating persistent identifiers for data objects, and adding and updating metadata. Results: The HEAL Data Platform serves as a single point of discovery of over one thousand studies funded under the HEAL Initiative. With hundreds of users per month, the HEAL Data Platform provides rich metadata and currently interoperates with nineteen data repositories and commons to provide access to shared datasets. Secure, cloud-based compute environments that are integrated with STRIDES facilitate secondary analysis of HEAL data. Discussion: Studies funded under the HEAL Initiative generate a wide variety of data types, which are deposited across multiple NIH and third-party data repositories. The mesh architecture of the HEAL Data Platform provides a single point of discovery of these data resources, accelerating and facilitating secondary use. Conclusion: The HEAL Data Platform enables search, discovery, and analysis of data that are deposited in connected data repositories and commons. By ensuring that these data are fully Findable, Accessible, Interoperable and Reusable (FAIR), the HEAL Data Platform maximizes the value of data generated under the HEAL Initiative.

cs.DC↗
arXiv2026

Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlap

Mixture-of-Experts (MoE) inference under expert parallelism (EP) turns each MoE layer into a distributed computation with costly dispatch and combine communication. State-of-the-art systems reduce this cost through communication-computation overlap, splitting the GPU's SMs for communication and computation respectively. However, this approach still leaves GPU resources wasted along two dimensions. Spatially, the best SM split is determined by each layer's routing result and varies across layers and GPUs, so fixed policies mismatch the workload and waste either NVLink bandwidth or compute throughput. Temporally, complex MoE data dependencies introduce bubbles that leave SMs idle. We present Weave, to our knowledge the first MoE overlap system that performs fine-grained dynamic SM scheduling - deciding per layer and per GPU by routing results at runtime. Once routing completes, each layer's communication and computation volumes become known; Weave exploits this predictability through a lightweight cost model running inside the persistent megakernel: a spatial scheduler partitions SMs into communication workers and computation workers to match the communication/computation throughput ratio, and a temporal scheduler coordinates the two worker groups to minimize SM idleness. On 4x H100 SXM GPUs across six mainstream MoE models, Weave achieves a 2.89x geometric-mean MoE-layer speedup and a 1.33x geometric-mean end-to-end speedup over five state-of-the-art baselines.

cs.DC↗
arXiv2026

pytest-gpu-proof: Enabling Cloud-CPU Continuous Integration for GPU Code with Local GPU Attestation

GPU acceleration is now routine across robotics, but cloud-hosted GPU continuous integration (CI) runners are expensive, resulting in severe under-testing of GPU-accelerated code. We present pytest-gpu-proof, an open-source pytest plugin offering a practical middle ground. Tests can be run on a local machine, signed with a receipt of exactly what ran and what it produced, and integrated into standard CPU CI workflows (e.g., GitHub Actions). The tool is open source and on PyPI, and we are actively integrating it across our lab's software stack.

cs.DC↗