arXiv subjects

Jiahao Liu

Publications and source records attributed to Jiahao Liu.

At least 127 records · Page 7Linked to original sources

arXiv2023

FedDWA: Personalized Federated Learning with Dynamic Weight Adjustment

Different from conventional federated learning, personalized federated learning (PFL) is able to train a customized model for each individual client according to its unique requirement. The mainstream approach is to adopt a kind of weighted aggregation method to generate personalized models, in which weights are determined by the loss value or model parameters among different clients. However, such kinds of methods require clients to download others' models. It not only sheer increases communication traffic but also potentially infringes data privacy. In this paper, we propose a new PFL algorithm called \emph{FedDWA (Federated Learning with Dynamic Weight Adjustment)} to address the above problem, which leverages the parameter server (PS) to compute personalized aggregation weights based on collected models from clients. In this way, FedDWA can capture similarities between clients with much less communication overhead. More specifically, we formulate the PFL problem as an optimization problem by minimizing the distance between personalized models and guidance models, so as to customize aggregation weights for each client. Guidance models are obtained by the local one-step ahead adaptation on individual clients. Finally, we conduct extensive experiments using five real datasets and the results demonstrate that FedDWA can significantly reduce the communication traffic and achieve much higher model accuracy than the state-of-the-art approaches.

cs.LG↗

arXiv2023

GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices. However, the deployment of knowledge distillation systems faces great challenges in real-world industrial-strength applications, which require the use of complex distillation methods on even larger-scale PLMs (over 10B), limited by memory on GPUs and the switching of methods. To overcome these challenges, we propose GKD, a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods. With GKD, developers can build larger distillation models on memory-limited GPUs and easily switch and combine different distillation methods within a single framework. Experimental results show that GKD can support the distillation of at least 100B-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs.

cs.CL↗

arXiv2023

PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models

While transformer-based pre-trained language models (PLMs) have dominated a number of NLP applications, these models are heavy to deploy and expensive to use. Therefore, effectively compressing large-scale PLMs becomes an increasingly important problem. Quantization, which represents high-precision tensors with low-bit fix-point format, is a viable solution. However, most existing quantization methods are task-specific, requiring customized training and quantization with a large number of trainable parameters on each individual task. Inspired by the observation that the over-parameterization nature of PLMs makes it possible to freeze most of the parameters during the fine-tuning stage, in this work, we propose a novel ``quantize before fine-tuning'' framework, PreQuant, that differs from both quantization-aware training and post-training quantization. PreQuant is compatible with various quantization strategies, with outlier-aware parameter-efficient fine-tuning incorporated to correct the induced quantization error. We demonstrate the effectiveness of PreQuant on the GLUE benchmark using BERT, RoBERTa, and T5. We also provide an empirical investigation into the workflow of PreQuant, which sheds light on its efficacy.

cs.CL↗

arXiv2023

RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank

Unsupervised sentence representation learning is one of the fundamental problems in natural language processing with various downstream applications. Recently, contrastive learning has been widely adopted which derives high-quality sentence representations by pulling similar semantics closer and pushing dissimilar ones away. However, these methods fail to capture the fine-grained ranking information among the sentences, where each sentence is only treated as either positive or negative. In many real-world scenarios, one needs to distinguish and rank the sentences based on their similarities to a query sentence, e.g., very relevant, moderate relevant, less relevant, irrelevant, etc. In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. In particular, we learn semantically discriminative sentence representations by simultaneously ensuring ranking consistency between two representations with different dropout masks, and distilling listwise ranking knowledge from the teacher. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks. Experimental results demonstrate the superior performance of our approach over several state-of-the-art baselines.

cs.CL↗

arXiv2023

Lifting the Curse of Capacity Gap in Distilling Language Models

Pretrained language models (LMs) have shown compelling performance on various downstream tasks, but unfortunately they require a tremendous amount of inference compute. Knowledge distillation finds a path to compress LMs to small ones with a teacher-student paradigm. However, when the capacity gap between the teacher and the student is large, a curse of capacity gap appears, invoking a deficiency in distilling LMs. While a few studies have been carried out to fill the gap, the curse is not yet well tackled. In this paper, we aim at lifting the curse of capacity gap via enlarging the capacity of the student without notably increasing the inference compute. Largely motivated by sparse activation regime of mixture of experts (MoE), we propose a mixture of minimal experts (MiniMoE), which imposes extra parameters to the student but introduces almost no additional inference compute. Experimental results on GLUE and CoNLL demonstrate the curse of capacity gap is lifted by the magic of MiniMoE to a large extent. MiniMoE also achieves the state-of-the-art performance at small FLOPs compared with a range of competitive baselines. With a compression rate as much as $\sim$50$\times$, MiniMoE preserves $\sim$95\% GLUE score of the teacher.

cs.CL↗

arXiv2023

Cosmic ray density variations in nearby giant molecular clouds

In this paper, we analyzed 12 years of Fermi LAT gamma-ray data towards three nearby giant molecular clouds (GMCs), i.e., R~CrA, Chamaeleon, and Lupus. We calibrated the gas column density of these regions by using the Planck dust opacity map as well as the Gaia extinction map. With both the gamma-ray observations and gas column density maps, we derived the cosmic ray densities in the three GMCs. We found the derived CR spectra have almost the same shape but significantly different normalizations, which may reflect that the distributions of CRs in the vicinity of solar systems are inhomogeneous.

astro-ph.HE↗

arXiv2023

Revealing the nature of hidden charm pentaquarks with machine learning

We study the nature of the hidden charm pentaquarks, i.e. the $P_c(4312)$, $P_c(4440)$ and $P_c(4457)$, with a neural network approach in pionless effective field theory. In this framework, the normal $χ^2$ fitting approach cannot distinguish the quantum numbers of the $P_c(4440)$ and $P_c(4457)$. In contrast to that, the neural network-based approach can discriminate them. In addition, we also illustrate the role of each experimental data bin of the invariant $J/ψp$ mass distribution on the underlying physics in both neural network and fitting methods. Their similarities and differences demonstrate that neural network methods can use data information more effectively and directly. This study provides more insights about how the neural network-based approach predicts the nature of exotic states from the mass spectrum.

hep-ph↗

arXiv2023

Triple Structural Information Modelling for Accurate, Explainable and Interactive Recommendation

In dynamic interaction graphs, user-item interactions usually follow heterogeneous patterns, represented by different structural information, such as user-item co-occurrence, sequential information of user interactions and the transition probabilities of item pairs. However, the existing methods cannot simultaneously leverage all three structural information, resulting in suboptimal performance. To this end, we propose TriSIM4Rec, a triple structural information modeling method for accurate, explainable and interactive recommendation on dynamic interaction graphs. Specifically, TriSIM4Rec consists of 1) a dynamic ideal low-pass graph filter to dynamically mine co-occurrence information in user-item interactions, which is implemented by incremental singular value decomposition (SVD); 2) a parameter-free attention module to capture sequential information of user interactions effectively and efficiently; and 3) an item transition matrix to store the transition probabilities of item pairs. Then, we fuse the predictions from the triple structural information sources to obtain the final recommendation results. By analyzing the relationship between the SVD-based and the recently emerging graph signal processing (GSP)-based collaborative filtering methods, we find that the essence of SVD is an ideal low-pass graph filter, so that the interest vector space in TriSIM4Rec can be extended to achieve explainable and interactive recommendation, making it possible for users to actively break through the information cocoons. Experiments on six public datasets demonstrated the effectiveness of TriSIM4Rec in accuracy, explainability and interactivity.

cs.IR↗

arXiv2023

Predicting the Fracture Propensity of Amorphous Silica Using Molecular Dynamics Simulations and Machine Learning

Amorphous silica ($a-SiO_2$) is a widely used inorganic material. Interestingly, the relationship between the local atomic structures of $a-SiO_2$ and their effects on ductility and fracture is seldom explored. Here, we combine large-scale molecular dynamics simulations and machine learning methods to examine the molecular deformations and fracture mechanisms of $a-SiO_2$. By quenching at high pressures, we demonstrate that densifying $a-SiO_2$ increases the ductility and toughness. Through theoretical analysis and simulation results, we find that changes in local bonding topologies greatly facilitate energy dissipation during plastic deformation, particularly if the coordination numbers decrease. The appearance of fracture can then be accurately located based on the spatial distribution of the atoms. We further observe that the static unstrained structure encodes the propensity for local atomic coordination to change during applied strain, hence a distinct connection can be made between the initial atomic configurations before loading and the final far-from-equilibrium atomic configurations upon fracture. These results are essential for understanding how atomic arrangements strongly influence the mechanical properties and structural features in amorphous solids and will be useful in atomistic design of functional materials.

cond-mat.mes-hall↗

arXiv2023

Multi-task Transformer with Relation-attention and Type-attention for Named Entity Recognition

Named entity recognition (NER) is an important research problem in natural language processing. There are three types of NER tasks, including flat, nested and discontinuous entity recognition. Most previous sequential labeling models are task-specific, while recent years have witnessed the rising of generative models due to the advantage of unifying all NER tasks into the seq2seq model framework. Although achieving promising performance, our pilot studies demonstrate that existing generative models are ineffective at detecting entity boundaries and estimating entity types. This paper proposes a multi-task Transformer, which incorporates an entity boundary detection task into the named entity recognition task. More concretely, we achieve entity boundary detection by classifying the relations between tokens within the sentence. To improve the accuracy of entity-type mapping during decoding, we adopt an external knowledge base to calculate the prior entity-type distributions and then incorporate the information into the model via the self and cross-attention mechanisms. We perform experiments on an extensive set of NER benchmarks, including two flat, three nested, and three discontinuous NER datasets. Experimental results show that our approach considerably improves the generative NER model's performance.

cs.CL↗

arXiv2023

An Error-Surface-Based Fractional Motion Estimation Algorithm and Hardware Implementation for VVC

Versatile Video Coding (VVC) introduces more coding tools to improve compression efficiency compared to its predecessor High Efficiency Video Coding (HEVC). For inter-frame coding, Fractional Motion Estimation (FME) still has a high computational effort, which limits the real-time processing capability of the video encoder. In this context, this paper proposes an error-surface-based FME algorithm and the corresponding hardware implementation. The algorithm creates an error surface constructed by the Rate-Distortion (R-D) cost of the integer motion vector (IMV) and its neighbors. This method requires no iteration and interpolation, thus reducing the area and power consumption and increasing the throughput of the hardware. The experimental results show that the corresponding BDBR loss is only 0.47% compared to VTM 16.0 in LD-P configuration. The hardware implementation was synthesized using GF 28nm process. It can support 13 different sizes of CU varying from 128x128 to 8x8. The measured throughput can reach 4K@30fps at 400MHz, with a gate count of 192k and power consumption of 12.64 mW. And the throughput can reach 8K@30fps at 631MHz when only quadtree is searched. To the best of our knowledge, this work is the first hardware architecture for VVC FME with interpolation-free strategies

eess.IV↗

arXiv2023

Personalized Graph Signal Processing for Collaborative Filtering

The collaborative filtering (CF) problem with only user-item interaction information can be solved by graph signal processing (GSP), which uses low-pass filters to smooth the observed interaction signals on the similarity graph to obtain the prediction signals. However, the interaction signal may not be sufficient to accurately characterize user interests and the low-pass filters may ignore the useful information contained in the high-frequency component of the observed signals, resulting in suboptimal accuracy. To this end, we propose a personalized graph signal processing (PGSP) method for collaborative filtering. Firstly, we design the personalized graph signal containing richer user information and construct an augmented similarity graph containing more graph topology information, to more effectively characterize user interests. Secondly, we devise a mixed-frequency graph filter to introduce useful information in the high-frequency components of the observed signals by combining an ideal low-pass filter that smooths signals globally and a linear low-pass filter that smooths signals locally. Finally, we combine the personalized graph signal, the augmented similarity graph and the mixed-frequency graph filter by proposing a pipeline consisting of three key steps: pre-processing, graph convolution and post-processing. Extensive experiments show that PGSP can achieve superior accuracy compared with state-of-the-art CF methods and, as a nonparametric method, PGSP has very high training efficiency.

cs.IR↗

arXiv2022

Parameter-free Dynamic Graph Embedding for Link Prediction

Dynamic interaction graphs have been widely adopted to model the evolution of user-item interactions over time. There are two crucial factors when modelling user preferences for link prediction in dynamic interaction graphs: 1) collaborative relationship among users and 2) user personalized interaction patterns. Existing methods often implicitly consider these two factors together, which may lead to noisy user modelling when the two factors diverge. In addition, they usually require time-consuming parameter learning with back-propagation, which is prohibitive for real-time user preference modelling. To this end, this paper proposes FreeGEM, a parameter-free dynamic graph embedding method for link prediction. Firstly, to take advantage of the collaborative relationships, we propose an incremental graph embedding engine to obtain user/item embeddings, which is an Online-Monitor-Offline architecture consisting of an Online module to approximately embed users/items over time, a Monitor module to estimate the approximation error in real time and an Offline module to calibrate the user/item embeddings when the online approximation errors exceed a threshold. Meanwhile, we integrate attribute information into the model, which enables FreeGEM to better model users belonging to some under represented groups. Secondly, we design a personalized dynamic interaction pattern modeller, which combines dynamic time decay with attention mechanism to model user short-term interests. Experimental results on two link prediction tasks show that FreeGEM can outperform the state-of-the-art methods in accuracy while achieving over 36X improvement in efficiency. All code and datasets can be found in https://github.com/FudanCISL/FreeGEM.

cs.LG↗

arXiv2022

Anomalous Nernst effect in compensated ferrimagnetic CoxGd1-x films

The anomalous Nernst effect (ANE) is one of the most intriguing thermoelectric phenomena which has attracted growing interest both for its underlying physics and potential applications. Typically, a large ANE response is observed in magnets with pronounced magnetizations or nontrivial Berry curvature. Here, we report a significant ANE signal in compensated ferrimagnetic CoxGd1-x alloy films, which exhibit vanishingly small magnetization. In particular, we found that the polarity of ANE signal is dominated by the magnetization orientation of the transition metal Co sublattices, rather than the net magnetization of CoxGd1-x films. This observation is not expected from the conventional understanding of ANE but is analogous to the anomalous Hall effect in compensated ferrimagnets. We attribute the origin of ANE and its Co-dominant property to the Co-dominant Berry curvature. Our work could trigger a more comprehensive understanding of ANE and may be useful for building energy-harvesting devices by employing ANE in compensated ferrimagnets.

cond-mat.mtrl-sci↗

arXiv2022

GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval

Recently, retrieval models based on dense representations are dominant in passage retrieval tasks, due to their outstanding ability in terms of capturing semantics of input text compared to the traditional sparse vector space models. A common practice of dense retrieval models is to exploit a dual-encoder architecture to represent a query and a passage independently. Though efficient, such a structure loses interaction between the query-passage pair, resulting in inferior accuracy. To enhance the performance of dense retrieval models without loss of efficiency, we propose a GNN-encoder model in which query (passage) information is fused into passage (query) representations via graph neural networks that are constructed by queries and their top retrieved passages. By this means, we maintain a dual-encoder structure, and retain some interaction information between query-passage pairs in their representations, which enables us to achieve both efficiency and efficacy in passage retrieval. Evaluation results indicate that our method significantly outperforms the existing models on MSMARCO, Natural Questions and TriviaQA datasets, and achieves the new state-of-the-art on these datasets.

cs.IR↗

arXiv2022

Sampling Gaussian Stationary Random Fields: A Stochastic Realization Approach

Generating large-scale samples of stationary random fields is of great importance in the fields such as geomaterial modeling and uncertainty quantification. Traditional methodologies based on covariance matrix decomposition have the diffculty of being computationally expensive, which is even more serious when the dimension of the random field is large. This paper proposes an effcient stochastic realization approach for sampling Gaussian stationary random fields from a systems and control point of view. Specifically, we take the exponential and Gaussian covariance functions as examples and make a decoupling assumption when there are multiple dimensions. Then a rational spectral density is constructed in each dimension using techniques from covariance extension, and the corresponding autoregressive moving-average (ARMA) model is obtained via spectral factorization. As a result, samples of the random field with a specific covariance function can be generated very effciently in the space domain by implementing the ARMA recursion using a white noise input. Such a procedure is computationally cheap due to the fact that the constructed ARMA model has a low order. Furthermore, the same method is integrated to multiscale simulations where interpolations of the generated samples are achieved when one zooms into finer scales. Both theoretical analysis and simulation results show that our approach performs favorably compared with covariance matrix decomposition methods.

stat.ME↗

arXiv2022

The symbology of Feynman integrals from twistor geometries

We study the symbology of planar Feynman integrals in dimensional regularization by considering geometric configurations in momentum twistor space corresponding to their leading singularities (LS). Cutting propagators in momentum twistor space amounts to intersecting lines associated with loop and external dual momenta, including the special line associated with the point at infinity, which breaks dual conformal symmetry. We show that cross-ratios of intersection points on these lines, especially those on the infinity line, naturally produce symbol letters for Feynman integrals in $D=4-2ε$, which include and generalize their LS. At one loop, we obtain all symbol letters using intersection points from quadruple cuts for integrals up to pentagon kinematics with two massive corners, which agree perfectly with canonical differential equation (CDE) results. We then obtain all two-loop letters, for up to four-mass box and one-mass pentagon kinematics, by considering more intersections arising from two-loop cuts. Finally we comment on how cluster algebras appear from this construction, and importantly how we may extend the method to non-planar integrals.

hep-th↗

arXiv2022

Study of exotic hadrons with machine learning

We analyzed the invariant mass spectrum of near-threshold exotic states for one-channel candidates with a deep neural network. It can extract the scattering length and effective range, which would shed light on the nature of given states, from the experimental mass spectrum. As an application, the mass spectrum of the $X(3872)$ and the $T_{cc}^+$ are studied. The obtained scattering lengths, effective ranges, and most relevant thresholds are consistent with those from fitting to the experimental data. The advantage of the neural network is that it is more stable than the fitting, especially for low-statistic data. The network, which provides another way to analyze the experimental data, can also be applied to other one-channel near-threshold exotic candidates.

hep-ph↗