Search arXiv⌕ Search

arXiv subjects

Xin Wang

Publications and source records attributed to Xin Wang.

At least 955 records · Page 53Linked to original sources
arXiv2024

PrivSGP-VR: Differentially Private Variance-Reduced Stochastic Gradient Push with Tight Utility Bounds

In this paper, we propose a differentially private decentralized learning method (termed PrivSGP-VR) which employs stochastic gradient push with variance reduction and guarantees $(ε, δ)$-differential privacy (DP) for each node. Our theoretical analysis shows that, under DP Gaussian noise with constant variance, PrivSGP-VR achieves a sub-linear convergence rate of $\mathcal{O}(1/\sqrt{nK})$, where $n$ and $K$ are the number of nodes and iterations, respectively, which is independent of stochastic gradient variance, and achieves a linear speedup with respect to $n$. Leveraging the moments accountant method, we further derive an optimal $K$ to maximize the model utility under certain privacy budget in decentralized settings. With this optimized $K$, PrivSGP-VR achieves a tight utility bound of $\mathcal{O}\left( \sqrt{d\log \left( \frac{1}δ \right)}/(\sqrt{n}Jε) \right)$, where $J$ and $d$ are the number of local samples and the dimension of decision variable, respectively, which matches that of the server-client distributed counterparts, and exhibits an extra factor of $1/\sqrt{n}$ improvement compared to that of the existing decentralized counterparts, such as A(DP)$^2$SGD. Extensive experiments corroborate our theoretical findings, especially in terms of the maximized utility with optimized $K$, in fully decentralized settings.

cs.LG↗
arXiv2024

Automated Graph Machine Learning: Approaches, Libraries, Benchmarks and Directions

Graph machine learning has been extensively studied in both academic and industry. However, as the literature on graph learning booms with a vast number of emerging methods and techniques, it becomes increasingly difficult to manually design the optimal machine learning algorithm for different graph-related tasks. To tackle the challenge, automated graph machine learning, which aims at discovering the best hyper-parameter and neural architecture configuration for different graph tasks/data without manual design, is gaining an increasing number of attentions from the research community. In this paper, we extensively discuss automated graph machine learning approaches, covering hyper-parameter optimization (HPO) and neural architecture search (NAS) for graph machine learning. We briefly overview existing libraries designed for either graph machine learning or automated machine learning respectively, and further in depth introduce AutoGL, our dedicated and the world's first open-source library for automated graph machine learning. Also, we describe a tailored benchmark that supports unified, reproducible, and efficient evaluations. Last but not least, we share our insights on future research directions for automated graph machine learning. This paper is the first systematic and comprehensive discussion of approaches, libraries as well as directions for automated graph machine learning.

cs.LG↗
arXiv2024

The UV luminosity function at 0.6 < z < 1 from UVCANDELS

UVCANDELS is a HST Cycle-26 Treasury Program awarded 164 orbits of primary ultraviolet (UV) F275W imaging and coordinated parallel optical F435W imaging in four CANDELS fields: GOODS-N, GOODS-S, EGS, and COSMOS, covering a total area of $\sim426$ arcmin$^2$. This is $\sim2.7$ times larger than the area covered by previous deep-field space UV data combined, reaching a depth of about 27 and 28 ABmag ($5σ$ in $0.2"$ apertures) for F275W and F435W, respectively. Along with the new photometric catalogs, we present an analysis of the rest-frame UV luminosity function (LF), relying on our UV-optimized aperture photometry method yielding a factor of $1.5\times$ increase than the H-isophot aperture photometry in the signal-to-noise ratios of galaxies in our F275W imaging. Using well tested photometric redshift measurements we identify 5810 galaxies at redshifts $0.6<z<1$, down to an absolute magnitude of $M_\text{UV} = -14.2$. In order to minimize the effect of uncertainties in estimating the completeness function, especially at the faint-end, we restrict our analysis to sources above $30\%$ completeness, which provides a final sample of 4726 galaxies at $-21.5<M_\text{UV}<-15.5$. We performed a maximum likelihood estimate to derive the best-fit parameters of the UV LF. We report a best-fit faint-end slope of $α= -1.359^{+0.041}_{-0.041}$ at $z \sim 0.8$. Creating sub-samples at $z\sim0.7$ and $z\sim0.9$, we observe a possible evolution of $α$ with redshift. The unobscured UV luminosity density at $M_\text{UV}<-10$ is derived as $ρ_\text{UV}=1.339^{+0.027}_{-0.030}\ (\times10^{26} \text{ergs/s/Hz/Mpc}^3)$ using our best-fit LF parameters. The new F275W and F435 photometric catalogs from UVCANDELS have been made publicly available on the Barbara A. Mikulski Archive for Space Telescopes (MAST).

astro-ph.GA↗
arXiv2024

Intraspecific predator interference promotes biodiversity in ecosystems

Explaining biodiversity is a fundamental issue in ecology. A long-standing puzzle lies in the paradox of the plankton: many species of plankton feeding on a limited variety of resources coexist, apparently flouting the competitive exclusion principle (CEP), which holds that the number of predator (consumer) species cannot exceed that of the resources at a steady state. Here, we present a mechanistic model and demonstrate that intraspecific interference among the consumers enables a plethora of consumer species to coexist at constant population densities with only one or a handful of resource species. This facilitated biodiversity is resistant to stochasticity, either with the stochastic simulation algorithm or individual-based modeling. Our model naturally explains the classical experiments that invalidate the CEP, quantitatively illustrates the universal S-shaped pattern of the rank-abundance curves across a wide range of ecological communities, and can be broadly used to resolve the mystery of biodiversity in many natural ecosystems.

q-bio.PE↗
arXiv2024

Tele-FLM Technical Report

Large language models (LLMs) have showcased profound capabilities in language understanding and generation, facilitating a wide array of applications. However, there is a notable paucity of detailed, open-sourced methodologies on efficiently scaling LLMs beyond 50 billion parameters with minimum trial-and-error cost and computational resources. In this report, we introduce Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgment capabilities. Tele-FLM demonstrates superior multilingual language modeling abilities, measured by BPB on textual corpus. Besides, in both English and Chinese foundation model evaluation, it is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B. In addition to the model weights, we share the core designs, engineering practices, and training details, which we expect to benefit both the academic and industrial communities.

cs.CL↗
arXiv2024

Optimizing OOD Detection in Molecular Graphs: A Novel Approach with Diffusion Models

The open-world test dataset is often mixed with out-of-distribution (OOD) samples, where the deployed models will struggle to make accurate predictions. Traditional detection methods need to trade off OOD detection and in-distribution (ID) classification performance since they share the same representation learning model. In this work, we propose to detect OOD molecules by adopting an auxiliary diffusion model-based framework, which compares similarities between input molecules and reconstructed graphs. Due to the generative bias towards reconstructing ID training samples, the similarity scores of OOD molecules will be much lower to facilitate detection. Although it is conceptually simple, extending this vanilla framework to practical detection applications is still limited by two significant challenges. First, the popular similarity metrics based on Euclidian distance fail to consider the complex graph structure. Second, the generative model involving iterative denoising steps is time-consuming especially when it runs on the enormous pool of drugs. To address these challenges, our research pioneers an approach of Prototypical Graph Reconstruction for Molecular OOD Detection, dubbed as PGR-MOOD and hinges on three innovations: i) An effective metric to comprehensively quantify the matching degree of input and reconstructed molecules; ii) A creative graph generator to construct prototypical graphs that are in line with ID but away from OOD; iii) An efficient and scalable OOD detector to compare the similarity between test samples and pre-constructed prototypical graphs and omit the generative process on every new molecule. Extensive experiments on ten benchmark datasets and six baselines are conducted to demonstrate our superiority.

cs.LG↗
arXiv2024

FLARE: A New Federated Learning Framework with Adjustable Learning Rates over Resource-Constrained Wireless Networks

Wireless federated learning (WFL) suffers from heterogeneity prevailing in the data distributions, computing powers, and channel conditions of participating devices. This paper presents a new Federated Learning with Adjusted leaRning ratE (FLARE) framework to mitigate the impact of the heterogeneity. The key idea is to allow the participating devices to adjust their individual learning rates and local training iterations, adapting to their instantaneous computing powers. The convergence upper bound of FLARE is established rigorously under a general setting with non-convex models in the presence of non-i.i.d. datasets and imbalanced computing powers. By minimizing the upper bound, we further optimize the scheduling of FLARE to exploit the channel heterogeneity. A nested problem structure is revealed to facilitate iteratively allocating the bandwidth with binary search and selecting devices with a new greedy method. A linear problem structure is also identified and a low-complexity linear programming scheduling policy is designed when training models have large Lipschitz constants. Experiments demonstrate that FLARE consistently outperforms the baselines in test accuracy, and converges much faster with the proposed scheduling policy.

eess.SP↗
arXiv2024

FedGreen: Carbon-aware Federated Learning with Model Size Adaptation

Federated learning (FL) provides a promising collaborative framework to build a model from distributed clients, and this work investigates the carbon emission of the FL process. Cloud and edge servers hosting FL clients may exhibit diverse carbon footprints influenced by their geographical locations with varying power sources, offering opportunities to reduce carbon emissions by training local models with adaptive computations and communications. In this paper, we propose FedGreen, a carbon-aware FL approach to efficiently train models by adopting adaptive model sizes shared with clients based on their carbon profiles and locations using ordered dropout as a model compression technique. We theoretically analyze the trade-offs between the produced carbon emissions and the convergence accuracy, considering the carbon intensity discrepancy across countries to choose the parameters optimally. Empirical studies show that FedGreen can substantially reduce the carbon footprints of FL compared to the state-of-the-art while maintaining competitive model accuracy.

cs.LG↗
arXiv2024

Mechanisms promoting biodiversity in ecosystems

Explaining biodiversity is a central focus in theoretical ecology. A significant obstacle arises from the Competitive Exclusion Principle (CEP), which states that two species competing for the same type of resources cannot coexist at constant population densities, or more generally, the number of consumer species cannot exceed that of resource species at steady states. The conflict between CEP and biodiversity is exemplified by the paradox of the plankton, where a few types of limiting resources support a plethora of plankton species. In this review, we introduce mechanisms proposed over the years for promoting biodiversity in ecosystems, with a special focus on those that alleviate the constraints imposed by the CEP, including mechanisms that challenge the CEP in well-mixed systems at a steady state or those that circumvent its limitations through contextual differences.

q-bio.PE↗
arXiv2024

Federated Learning Model Aggregation in Heterogenous Aerial and Space Networks

Federated learning offers a promising approach under the constraints of networking and data privacy constraints in aerial and space networks (ASNs), utilizing large-scale private edge data from drones, balloons, and satellites. Existing research has extensively studied the optimization of the learning process, computing efficiency, and communication overhead. An important yet often overlooked aspect is that participants contribute predictive knowledge with varying diversity of knowledge, affecting the quality of the learned federated models. In this paper, we propose a novel approach to address this issue by introducing a Weighted Averaging and Client Selection (WeiAvgCS) framework that emphasizes updates from high-diversity clients and diminishes the influence of those from low-diversity clients. Direct sharing of the data distribution may be prohibitive due to the additional private information that is sent from the clients. As such, we introduce an estimation for the diversity using a projection-based method. Extensive experiments have been performed to show WeiAvgCS's effectiveness. WeiAvgCS could converge 46% faster on FashionMNIST and 38% faster on CIFAR10 than its benchmarks on average in our experiments.

cs.LG↗
arXiv2024

Self-Selected Attention Span for Accelerating Large Language Model Inference

Large language models (LLMs) can solve challenging tasks. However, their inference computation on modern GPUs is highly inefficient due to the increasing number of tokens they must attend to as they generate new ones. To address this inefficiency, we capitalize on LLMs' problem-solving capabilities to optimize their own inference-time efficiency. We demonstrate with two specific tasks: (a) evaluating complex arithmetic expressions and (b) summarizing news articles. For both tasks, we create custom datasets to fine-tune an LLM. The goal of fine-tuning is twofold: first, to make the LLM learn to solve the evaluation or summarization task, and second, to train it to identify the minimal attention spans required for each step of the task. As a result, the fine-tuned model is able to convert these self-identified minimal attention spans into sparse attention masks on-the-fly during inference. We develop a custom CUDA kernel to take advantage of the reduced context to attend to. We demonstrate that using this custom CUDA kernel improves the throughput of LLM inference by 28%. Our work presents an end-to-end demonstration showing that training LLMs to self-select their attention spans speeds up autoregressive inference in solving real-world tasks.

cs.CL↗
arXiv2024

Data Generation-based Operator Learning for Solving Partial Differential Equations on Unbounded Domains

Wave propagation problems are typically formulated as partial differential equations (PDEs) on unbounded domains to be solved. The classical approach to solving such problems involves truncating them to problems on bounded domains by designing the artificial boundary conditions or perfectly matched layers, which typically require significant effort, and the presence of nonlinearity in the equation makes such designs even more challenging. Emerging deep learning-based methods for solving PDEs, with the physics-informed neural networks (PINNs) method as a representative, still face significant challenges when directly used to solve PDEs on unbounded domains. Calculations performed in a bounded domain of interest without imposing boundary constraints can lead to a lack of unique solutions thus causing the failure of PINNs. In light of this, this paper proposes a novel and effective operator learning-based method for solving PDEs on unbounded domains. The key idea behind this method is to generate high-quality training data. Specifically, we construct a family of approximate analytical solutions to the target PDE based on its initial condition and source term. Then, using these constructed data comprising exact solutions, initial conditions, and source terms, we train an operator learning model called MIONet, which is capable of handling multiple inputs, to learn the mapping from the initial condition and source term to the PDE solution on a bounded domain of interest. Finally, we utilize the generalization ability of this model to predict the solution of the target PDE. The effectiveness of this method is exemplified by solving the wave equation and the Schrodinger equation defined on unbounded domains. More importantly, the proposed method can deal with nonlinear problems, which has been demonstrated by solving Burger's equation and Korteweg-de Vries (KdV) equation.

math.NA↗
arXiv2024

Tackling Challenges in 21cm Global Spectrum Experiment: the Impact of Ionosphere and Beam Distortion

The HI 21cm global signal from the Cosmic Dawn and the Epoch of Reionization (EoR) offers critical insights into the evolution of our Universe. Yet, its detection presents significant challenges due to its extremely low signal-to-contamination ratio and complex instrumental systematics. In this paper, we examine the effects of the ionosphere and antenna beam on data analysis. The ionosphere, an ionized plasma layer in the Earth's atmosphere, refracts, absorbs, and emits radio waves in the relevant frequency range. This interaction results in additional spectral distortion of the observed signal, complicating the process of foreground subtraction. Additionally, chromatic variations in the beam can also introduce further contamination into the global spectrum measurement. Notably, the ionospheric effect, being dependent on the direction of incoming light, interacts with the instrumental beam, adding another layer of complexity. To address this, we evaluate three different fitting templates of foreground: the logarithmic polynomial, the physically motivated EDGES template, and a SVD-based template. Our findings indicate that the EDGES and SVD templates generally surpass logarithmic polynomials in performance. Recognizing the significance of beam chromaticity, we further investigate specific beam distortion models and their impacts on the signal extraction process.

astro-ph.CO↗
arXiv2024

Union: An Automatic Workload Manager for Accelerating Network Simulation

With the rapid growth of the machine learning applications, the workloads of future HPC systems are anticipated to be a mix of scientific simulation, big data analytics, and machine learning applications. Simulation is a great research vehicle to understand the performance implications of co-running scientific applications with big data and machine learning workloads on large-scale systems. In this paper, we present Union, a workload manager that provides an automatic framework to facilitate hybrid workload simulation in CODES. Furthermore, we use Union, along with CODES, to investigate various hybrid workloads composed of traditional simulation applications and emerging learning applications on two dragonfly systems. The experiment results show that both message latency and communication time are important performance metrics to evaluate network interference. Network interference on HPC applications is more reflected by the message latency variation, whereas ML application performance depends more on the communication time.

cs.DC↗
arXiv2024

Study of Workload Interference with Intelligent Routing on Dragonfly

Dragonfly interconnect is a crucial network technology for supercomputers. To support exascale systems, network resources are shared such that links and routers are not dedicated to any node pair. While link utilization is increased, workload performance is often offset by network contention. Recently, intelligent routing built on reinforcement learning demonstrates higher network throughput with lower packet latency. However, its effectiveness in reducing workload interference is unknown. In this work, we present extensive network simulations to study multi-workload contention under different routing mechanisms, intelligent routing and adaptive routing, on a large-scale Dragonfly system. We develop an enhanced network simulation toolkit, along with a suite of workloads with distinctive communication patterns. We also present two metrics to characterize application communication intensity. Our analysis focuses on examining how different workloads interfere with each other under different routing mechanisms by inspecting both application-level and network-level metrics. Several key insights are made from the analysis.

cs.NI↗
arXiv2024

Q-adaptive: A Multi-Agent Reinforcement Learning Based Routing on Dragonfly Network

High-radix interconnects such as Dragonfly and its variants rely on adaptive routing to balance network traffic for optimum performance. Ideally, adaptive routing attempts to forward packets between minimal and non-minimal paths with the least congestion. In practice, current adaptive routing algorithms estimate routing path congestion based on local information such as output queue occupancy. Using local information to estimate global path congestion is inevitably inaccurate because a router has no precise knowledge of link states a few hops away. This inaccuracy could lead to interconnect congestion. In this study, we present Q-adaptive routing, a multi-agent reinforcement learning routing scheme for Dragonfly systems. Q-adaptive routing enables routers to learn to route autonomously by leveraging advanced reinforcement learning technology. The proposed Q-adaptive routing is highly scalable thanks to its fully distributed nature without using any shared information between routers. Furthermore, a new two-level Q-table is designed for Q-adaptive to make it computational lightly and saves 50% of router memory usage compared with the previous Q-routing. We implement the proposed Q-adaptive routing in SST/Merlin simulator. Our evaluation results show that Q-adaptive routing achieves up to 10.5% system throughput improvement and 5.2x average packet latency reduction compared with adaptive routing algorithms. Remarkably, Q-adaptive can even outperform the optimal VALn non-minimal routing under the ADV+1 adversarial traffic pattern with up to 3% system throughput improvement and 75% average packet latency reduction.

cs.NI↗
arXiv2024

TeleChat Technical Report

In this technical report, we present TeleChat, a collection of large language models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It includes pretrained language models as well as fine-tuned chat models that is aligned with human preferences. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, including trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves comparable performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat's 7B and 12B variant, along with code and a portion of our pretraining data, to the public community.

cs.CL↗