Design and Optimization of Heterogeneous Coded Distributed Computing with Nonuniform File Popularity

Yong Deng; Min Dong

Home/Archive/arXiv

arXiv · 2310.03142

Design and Optimization of Heterogeneous Coded Distributed Computing with Nonuniform File Popularity

Yong Deng·Min Dong

View original source ↗Read source document ↗Export metadata ↓

Abstract

This paper studies MapReduce-based heterogeneous coded distributed computing (CDC) where, besides different computing capabilities at workers, input files to be accessed by computing jobs have nonuniform popularity. We propose a file placement strategy that can handle an arbitrary number of input files. Furthermore, we design a nested coded shuffling strategy that can efficiently manage the nonuniformity of file popularity to maximize the coded multicasting opportunity. We then formulate the joint optimization of the proposed file placement and nested shuffling design variables to optimize the proposed CDC scheme. To reduce the high computational complexity in solving the resulting mixed-integer linear programming (MILP) problem, we propose a simple two-file-group-based file placement approach to obtain an approximate solution. Numerical results show that the optimized CDC scheme outperforms other alternatives. Also, the proposed two-file-group-based approach achieves nearly the same performance as the conventional branch-and-cut method in solving the MILP problem but with substantially lower computational complexity that is scalable over the number of files and workers. For computing jobs with aggregate target functions that commonly appear in machine learning applications, we propose a heterogeneous compressed CDC (C-CDC) scheme to further improve the shuffling efficiency. The C-CDC scheme uses a local data aggregation technique to compress the data to be shuffled for the shuffling load reduction. We again optimize the proposed C-CDC scheme and explore the two-file-group-based low-complexity approach for an approximate solution. Numerical results show the proposed C-CDC scheme provides a considerable shuffling load reduction over the CDC scheme, and also, the two-file-group-based file placement approach maintains good performance.

Explore related subjects

cs.IT math.IT

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Yong Deng, Min Dong. 2024-10-19. Design and Optimization of Heterogeneous Coded Distributed Computing with Nonuniform File Popularity. https://arxiv.org/abs/2310.03142

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Radiance-Field Guided Pretraining: Scaling Localization Models with Unlabeled Wireless Signals

Radio frequency (RF)-based indoor localization offers significant promise for applications such as indoor navigation, augmented reality, and pervasive computing. While deep learning has greatly enhanced localization accuracy and robustness, existing localization models still face major challenges in cross-scene generalization due to their reliance on scene-specific labeled data. To address this, we introduce Radiance-Field Reinforced Pretraining (RFRP). This novel self-supervised pretraining framework couples a large localization model (LM) with a neural radio-frequency radiance field (RF-NeRF) in an asymmetrical autoencoder architecture. In this design, the LM encodes received RF spectra into latent, position-relevant representations, while the RF-NeRF decodes them to reconstruct the original spectra. This alignment between input and output enables effective representation learning using large-scale, unlabeled RF data, which can be collected continuously with minimal effort. To this end, we collected RF samples at 7,327,321 positions across 100 diverse scenes using four common wireless technologies--RFID, BLE, WiFi, and IIoT. Data from 75 scenes were used for training, and the remaining 25 for evaluation. Experimental results show that the RFRP-pretrained LM reduces localization error by over 40% compared to non-pretrained models and by 21% compared to those pretrained using supervised learning.

cs.IT↗

arXiv2026

Uniform Recovery of Structured Signals from Nonlinear Observations: Improved Error Rates

Consider the recovery of structured signals from nonlinear observations. Under Gaussian matrix and a large class of unknown nonlinear link functions, Plan and Vershynin (2016) showed that generalized Lasso achieves accurate nonuniform recovery of a fixed signal. More recently, Genzel and Stollenwerk (2023) showed that generalized Lasso is indeed capable of accurately recovering all structured signals. However, in some canonical settings with discontinuous link functions, their uniform recovery error rate is essentially slower than the nonuniform one. Specifically, in the recovery of $n$-dimensional $k$-sparse vectors from $m$ measurements, generalized Lasso with a perfectly tuned $\ell_1$ constraint achieves nonuniform error rate $ O(\sqrt{k\log(en/k)/m})$, while the uniform error rate of Genzel and Stollenwerk is no faster than $O((k\log(en/k)/m)^{1/4})$. In this paper, we narrow this gap by establishing improved uniform recovery guarantees under piecewise Lipschitz link functions with well-separated jump discontinuities. We analyze a projected gradient descent (PGD) algorithm whose projection can be onto a convex set or a cone, and our results for the PGD with a convex set are also valid for the generalized Lasso. In sparse recovery, the improved uniform error rates match the nonuniform rate $O(\sqrt{k\log(en/k)/m})$ up to logarithmic factors. Under the sign link function, we further show that iterative hard thresholding (a specific instance of the PGD) achieves uniform recovery error rate $O(\sqrt{k\log(en/k)/m})$, matching the nonuniform rate up to a universal constant. Technically, the uniform guarantees for the PGD are obtained by showing that the gradient maps satisfy the restricted approximate invertibility condition uniformly over all signals. We demonstrate that this is a general approach to uniform recovery under nonlinear observations.

cs.IT↗

arXiv2026

Enhanced Feedback Mechanisms for Resource-Efficient Incremental Redundancy

Incremental redundancy (IR) can reduce error rates by spreading coded bits across multiple transmission attempts. However, conventional stop-and-wait operation with coarse feedback often over-provisions retransmissions, triggers unnecessary decoding attempts, and increases end-to-end latency. This paper develops enhanced feedback and scheduling mechanisms that predict the additional redundancy needed for successful decoding and allocate only the required resources. We study two complementary strategies. First, using channel statistics, we learn a one- or two-shot mapping from channel quality to the minimum redundancy budget. As a byproduct, we derive an achievable reliability lower bound on the error probability of hybrid automatic repeat request (HARQ) systems. Numerical results with polar-coded IR-HARQ scheme show that the bound can be closely approached by appropriately selecting the second-transmission redundancy over a wide SNR range with savings up to 60\% in retransmission size. Second, we propose a realization-aware early-feedback mechanism that uses first-transmission reliability information to make per-codeword decisions before decoding: whether the codeword is already decodable, if not, how many additional redundancy versions are needed, or whether decoding is unlikely and rate adaptation is preferable. Link-level simulations with 5G NR LDPC codes show that both predictors achieve high accuracy (about 96\% in our study), increasing the probability of successful decoding within at most two transmission occasions.

cs.IT↗