arXiv subjects

Xinyu Zhang

Publications and source records attributed to Xinyu Zhang.

At least 415 records · Page 23Linked to original sources

arXiv2022

KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models

Previous works show the great potential of pre-trained language models (PLMs) for storing a large amount of factual knowledge. However, to figure out whether PLMs can be reliable knowledge sources and used as alternative knowledge bases (KBs), we need to further explore some critical features of PLMs. Firstly, knowledge memorization and identification abilities: traditional KBs can store various types of entities and relationships; do PLMs have a high knowledge capacity to store different types of knowledge? Secondly, reasoning ability: a qualified knowledge source should not only provide a collection of facts, but support a symbolic reasoner. Can PLMs derive new knowledge based on the correlations between facts? To evaluate these features of PLMs, we propose a benchmark, named Knowledge Memorization, Identification, and Reasoning test (KMIR). KMIR covers 3 types of knowledge, including general knowledge, domain-specific knowledge, and commonsense, and provides 184,348 well-designed questions. Preliminary experiments with various representative pre-training language models on KMIR reveal many interesting phenomenons: 1) The memorization ability of PLMs depends more on the number of parameters than training schemes. 2) Current PLMs are struggling to robustly remember the facts. 3) Model compression technology retains the amount of knowledge well, but hurts the identification and reasoning abilities. We hope KMIR can facilitate the design of PLMs as better knowledge sources.

cs.CL↗

arXiv2022

Multi-Objective Model-based Reinforcement Learning for Infectious Disease Control

Severe infectious diseases such as the novel coronavirus (COVID-19) pose a huge threat to public health. Stringent control measures, such as school closures and stay-at-home orders, while having significant effects, also bring huge economic losses. In the face of an emerging infectious disease, a crucial question for policymakers is how to make the trade-off and implement the appropriate interventions timely given the huge uncertainty. In this work, we propose a Multi-Objective Model-based Reinforcement Learning framework to facilitate data-driven decision-making and minimize the overall long-term cost. Specifically, at each decision point, a Bayesian epidemiological model is first learned as the environment model, and then the proposed model-based multi-objective planning algorithm is applied to find a set of Pareto-optimal policies. This framework, combined with the prediction bands for each policy, provides a real-time decision support tool for policymakers. The application is demonstrated with the spread of COVID-19 in China.

cs.LG↗

arXiv2022

From Model Selection to Model Averaging: A Comparison for Nested Linear Models

Model selection (MS) and model averaging (MA) are two popular approaches when having many candidate models. Theoretically, the estimation risk of an oracle MA is not larger than that of an oracle MS because the former one is more flexible, but a foundational issue is: does MA offer a {\it substantial} improvement over MS? Recently, a seminal work: Peng and Yang (2021), has answered this question under nested models with linear orthonormal series expansion. In the current paper, we further reply this question under linear nested regression models. Especially, a more general nested framework, heteroscedastic and autocorrelated random errors, and sparse coefficients are allowed in the current paper, which is more common in practice. In addition, we further compare MAs with different weight sets. Simulation studies support the theoretical findings in a variety of settings.

math.ST↗

arXiv2022

A Hybrid Cloud ERP Framework For Processing Purchasing Data

Cloud-based enterprise resource planning (cloud ERP) systems have existed in the business market for around ten years. Cloud ERP supports enterprises' daily activities by integrating organizational back-end systems in the cloud environment. One of the critical functions that cloud ERP offers is the purchasing application. The purchasing function of cloud ERP enables enterprises to streamline all the online purchasing transactions in real-time automatically. Even cloud ERP is deployed quite often these days, organizations somehow still lack the knowledge of it; to be specific, there are many issues attached to cloud ERP implementation yet to be solved. Hence, this paper compares four leading cloud ERP platforms in Australia and proposes a hybrid cloud ERP framework to process online purchasing transactions. By adopting a case study approach, a purchasing web-based application is designed and presented in this paper. In general, the proposed hybrid cloud ERP framework and the integrated web-based purchasing application allow user companies to process online purchasing transactions with short operation time and increased business efficiency; in the meantime, the proposed framework also reduces security risks attached to the public cloud.

cs.CY↗

arXiv2022

Model Averaging for Support Vector Machine by Cross-Validation

Support vector machine (SVM) is a well-known statistical technique for classification problems in machine learning and other fields. An important question for SVM is the selection of covariates (or features) for the model. Many studies have considered model selection methods. As is well-known, selecting one winning model over others can entail considerable instability in predictive performance due to model selection uncertainties. This paper advocates model averaging as an alternative approach, where estimates obtained from different models are combined in a weighted average. We propose a model weighting scheme and provide the theoretical underpinning for the proposed method. In particular, we prove that our proposed method yields a model average estimator that achieves the smallest hinge risk among all feasible combinations asymptotically. To remedy the computational burden due to a large number of feasible models, we propose a screening step to eliminate the uninformative features before combining the models. Results from real data applications and a simulation study show that the proposed method generally yields more accurate estimates than existing methods.

stat.ME↗

arXiv2021

$\mathcal{N}=1$ Curves on Generalized Coulomb Branches

We study the low energy effective dynamics of four-dimensional $\mathcal{N}=1$ supersymmetric gauge theories of class $\mathcal{S}_k$ on the generalized Coulomb branch. The low energy effective gauge couplings are naturally encoded in algebraic curves $\mathcal{X}$, which we derive for general values of the couplings and mass deformations. We then recast these IR curves $\mathcal{X}$ to the UV or M-theory form $\mathcal{C}$: the punctured Riemann surfaces on which the six-dimensional $\mathcal{N}=(1,0)$ ${A_{k-1}}$ SCFTs are compactified giving the class $\mathcal{S}_k$ theories. We find that the UV curves $\mathcal{C}$ and their corresponding meromorphic differentials take the same form as those for their mother four-dimensional $\mathcal{N}=2$ theories of class $\mathcal{S}$. They have the same poles, and their residues are functions of all the exactly marginal couplings and the bare mass parameters which we can compute exactly.

hep-th↗

arXiv2021

Orthounimodal Distributionally Robust Optimization: Representation, Computation and Multivariate Extreme Event Applications

This paper studies a basic notion of distributional shape known as orthounimodality (OU) and its use in shape-constrained distributionally robust optimization (DRO). As a key motivation, we argue how such type of DRO is well-suited to tackle multivariate extreme event estimation by giving statistically valid confidence bounds on target extremal probabilities. In particular, we explain how DRO can be used as a nonparametric alternative to conventional extreme value theory that extrapolates tails based on theoretical limiting distributions, which could face challenges in bias-variance control and other technical complications. We also explain how OU resolves the challenges in interpretability and robustness faced by existing distributional shape notions used in the DRO literature. Methodologically, we characterize the extreme points of the OU distribution class in terms of what we call OU sets and build a corresponding Choquet representation, which subsequently allows us to reduce OU-DRO into moment problems over infinite-dimensional random variables. We then develop, in the bivariate setting, a geometric approach to reduce such moment problems into finite dimension via a specially constructed variational problem designed to eliminate suboptimal solutions. Numerical results illustrate how our approach gives rise to valid and competitive confidence bounds for extremal probabilities.

math.OC↗

arXiv2021

Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retrieval techniques in non-English languages, motivated by recent observations that existing techniques for representation learning perform poorly when applied to out-of-distribution data. As a starting point, we provide zero-shot baselines for this new dataset based on a multi-lingual adaptation of DPR that we call "mDPR". Experiments show that although the effectiveness of mDPR is much lower than BM25, dense representations nevertheless appear to provide valuable relevance signals, improving BM25 results in sparse-dense hybrids. In addition to analyses of our results, we also discuss future challenges and present a research agenda in multi-lingual dense retrieval. Mr. TyDi can be downloaded at https://github.com/castorini/mr.tydi.

cs.CL↗

arXiv2021

Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations

Quantization and pruning are core techniques used to reduce the inference costs of deep neural networks. State-of-the-art quantization techniques are currently applied to both the weights and activations; however, pruning is most often applied to only the weights of the network. In this work, we jointly apply novel uniform quantization and unstructured pruning methods to both the weights and activations of deep neural networks during training. Using our methods, we empirically evaluate the currently accepted prune-then-quantize paradigm across a wide range of computer vision tasks and observe a non-commutative nature when applied to both the weights and activations of deep neural networks. Informed by these observations, we articulate the non-commutativity hypothesis: for a given deep neural network being trained for a specific task, there exists an exact training schedule in which quantization and pruning can be introduced to optimize network performance. We identify that this optimal ordering not only exists, but also varies across discriminative and generative tasks. Using the optimal training schedule within our training framework, we demonstrate increased performance per memory footprint over existing solutions.

cs.LG↗

arXiv2021

A Broad-Spectrum Diffractive Network via Ensemble Learning

We proposed a broad-spectrum diffractive deep neural network (BS-D2NN) framework, which incorporates multi-wavelength channels of input lightfields and performs a parallel phase-only modulation utilizing a layered passive mask architecture. A complementary multi-channel base learner cluster is formed in a homogeneous ensemble framework based on the diffractive dispersion during lightwave modulation. In addition, both the optical Sum operation and the Hybrid (optical-electronic) Maxout operation are performed for motivating the BS-D2NN to learn and construct a mapping between input lightfields and truth labels under heterochromatic ambient lighting. The BS-D2NN can be trained using deep learning algorithms so as to perform a kind of wavelength-insensitive high-accuracy object classification.

cs.ET↗

arXiv2021

Towards More Effective and Economic Sparsely-Activated Model

The sparsely-activated models have achieved great success in natural language processing through large-scale parameters and relatively low computational cost, and gradually become a feasible technique for training and implementing extremely large models. Due to the limit of communication cost, activating multiple experts is hardly affordable during training and inference. Therefore, previous work usually activate just one expert at a time to alleviate additional communication cost. Such routing mechanism limits the upper bound of model performance. In this paper, we first investigate a phenomenon that increasing the number of activated experts can boost the model performance with higher sparse ratio. To increase the number of activated experts without an increase in computational cost, we propose SAM (Switch and Mixture) routing, an efficient hierarchical routing mechanism that activates multiple experts in a same device (GPU). Our methods shed light on the training of extremely large sparse models and experiments prove that our models can achieve significant performance gain with great efficiency improvement.

cs.CL↗

arXiv2021

Tuning Confidence Bound for Stochastic Bandits with Bandit Distance

We propose a novel modification of the standard upper confidence bound (UCB) method for the stochastic multi-armed bandit (MAB) problem which tunes the confidence bound of a given bandit based on its distance to others. Our UCB distance tuning (UCB-DT) formulation enables improved performance as measured by expected regret by preventing the MAB algorithm from focusing on non-optimal bandits which is a well-known deficiency of standard UCB. "Distance tuning" of the standard UCB is done using a proposed distance measure, which we call bandit distance, that is parameterizable and which therefore can be optimized to control the transition rate from exploration to exploitation based on problem requirements. We empirically demonstrate increased performance of UCB-DT versus many existing state-of-the-art methods which use the UCB formulation for the MAB problem. Our contribution also includes the development of a conceptual tool called the "Exploration Bargain Point" which gives insights into the tradeoffs between exploration and exploitation. We argue that the Exploration Bargain Point provides an intuitive perspective that is useful for comparatively analyzing the performance of UCB-based methods.

stat.ML↗

arXiv2021

Safe Screening Rules for Generalized Double Sparsity Learning

In a high-dimensional setting, sparse model has shown its power in computational and statistical efficiency. We consider variables selection problem with a broad class of simultaneous sparsity regularization, enforcing both feature-wise and group-wise sparsity at the same time. The analysis leverages an introduction of $εq$-norm in vector space, which is proved to has close connection with the mixture regularization and naturally leads to a dual formulation. Properties of primal/dual optimal solution and optimal values are discussed, which motivates the design of screening rules. We several fast safe screening rules in the general framework, rules that discard inactive features/groups at an early stage that are guaranteed to be inactive in the exact solution, leading to a significant gain in computation speed.

math.OC↗

arXiv2021

YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Ranker

Pre-trained model such as BERT has been proved to be an effective tool for dealing with Information Retrieval (IR) problems. Due to its inspiring performance, it has been widely used to tackle with real-world IR problems such as document ranking. Recently, researchers have found that selecting "hard" rather than "random" negative samples would be beneficial for fine-tuning pre-trained models on ranking tasks. However, it remains elusive how to leverage hard negative samples in a principled way. To address the aforementioned issues, we propose a fine-tuning strategy for document ranking, namely Self-Involvement Ranker (SIR), to dynamically select hard negative samples to construct high-quality semantic space for training a high-quality ranking model. Specifically, SIR consists of sequential compressors implemented with pre-trained models. Front compressor selects hard negative samples for rear compressor. Moreover, SIR leverages supervisory signal to adaptively adjust semantic space of negative samples. Finally, supervisory signal in rear compressor is computed based on condition probability and thus can control sample dynamic and further enhance the model performance. SIR is a lightweight and general framework for pre-trained models, which simplifies the ranking process in industry practice. We test our proposed solution on MS MARCO with document ranking setting, and the results show that SIR can significantly improve the ranking performance of various pre-trained models. Moreover, our method became the new SOTA model anonymously on MS MARCO Document ranking leaderboard in May 2021.

cs.IR↗

arXiv2021

Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking

Context information in search sessions has proven to be useful for capturing user search intent. Existing studies explored user behavior sequences in sessions in different ways to enhance query suggestion or document ranking. However, a user behavior sequence has often been viewed as a definite and exact signal reflecting a user's behavior. In reality, it is highly variable: user's queries for the same intent can vary, and different documents can be clicked. To learn a more robust representation of the user behavior sequence, we propose a method based on contrastive learning, which takes into account the possible variations in user's behavior sequences. Specifically, we propose three data augmentation strategies to generate similar variants of user behavior sequences and contrast them with other sequences. In so doing, the model is forced to be more robust regarding the possible variations. The optimized sequence representation is incorporated into document ranking. Experiments on two real query log datasets show that our proposed model outperforms the state-of-the-art methods significantly, which demonstrates the effectiveness of our method for context-aware document ranking.

cs.IR↗

arXiv2021

Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need

Designing pre-training objectives that more closely resemble the downstream tasks for pre-trained language models can lead to better performance at the fine-tuning stage, especially in the ad-hoc retrieval area. Existing pre-training approaches tailored for IR tried to incorporate weak supervised signals, such as query-likelihood based sampling, to construct pseudo query-document pairs from the raw textual corpus. However, these signals rely heavily on the sampling method. For example, the query likelihood model may lead to much noise in the constructed pre-training data. \blfootnote{$\dagger$ This work was done during an internship at Huawei.} In this paper, we propose to leverage the large-scale hyperlinks and anchor texts to pre-train the language model for ad-hoc retrieval. Since the anchor texts are created by webmasters and can usually summarize the target document, it can help to build more accurate and reliable pre-training samples than a specific algorithm. Considering different views of the downstream ad-hoc retrieval, we devise four pre-training tasks based on the hyperlinks. We then pre-train the Transformer model to predict the pair-wise preference, jointly with the Masked Language Model objective. Experimental results on two large-scale ad-hoc retrieval datasets show the significant improvement of our model compared with the existing methods.

cs.IR↗

arXiv2021

Asymptotic Theory of Principal Component Analysis for Time Series Data with Cautionary Comments

Principal component analysis (PCA) is a most frequently used statistical tool in almost all branches of data science. However, like many other statistical tools, there is sometimes the risk of misuse or even abuse. In this paper, we highlight possible pitfalls in using the theoretical results of PCA based on the assumption of independent data when the data are time series. For the latter, we state with proof a central limit theorem of the eigenvalues and eigenvectors (loadings), give direct and bootstrap estimation of their asymptotic covariances, and assess their efficacy via simulation. Specifically, we pay attention to the proportion of variation, which decides the number of principal components (PCs), and the loadings, which help interpret the meaning of PCs. Our findings are that while the proportion of variation is quite robust to different dependence assumptions, the inference of PC loadings requires careful attention. We initiate and conclude our investigation with an empirical example on portfolio management, in which the PC loadings play a prominent role. It is given as a paradigm of correct usage of PCA for time series data.

stat.ME↗

arXiv2021

MSTC*:Multi-robot Coverage Path Planning under Physical Constraints

For large-scale tasks, coverage path planning (CPP) can benefit greatly from multiple robots. In this paper, we present an efficient algorithm MSTC* for multi-robot coverage path planning (mCPP) based on spiral spanning tree coverage (Spiral-STC). Our algorithm incorporates strict physical constraints like terrain traversability and material load capacity. We compare our algorithm against the state-of-the-art in mCPP for regular grid maps and real field terrains in simulation environments. The experimental results show that our method significantly outperforms existing spiral-STC based mCPP methods. Our algorithm can find a set of well-balanced workload distributions for all robots and therefore, achieve the overall minimum time to complete the coverage.

cs.RO↗