Search arXiv⌕ Search

arXiv subjects

Chao Zhang

Publications and source records attributed to Chao Zhang.

At least 721 records · Page 40Linked to original sources
arXiv2023

Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach

Large Language Models have demonstrated remarkable few-shot performance, but the performance can be sensitive to the selection of few-shot instances. We propose PATRON, a new method that uses prompt-based uncertainty estimation for data selection for pre-trained language model fine-tuning under cold-start scenarios, i.e., no initial labeled data are available. In PATRON, we design (1) a prompt-based uncertainty propagation approach to estimate the importance of data points and (2) a partition-then-rewrite (PTR) strategy to promote sample diversity when querying for annotations. Experiments on six text classification datasets show that PATRON outperforms the strongest cold-start data selection baselines by up to 6.9%. Besides, with 128 labels only, PATRON achieves 91.0% and 92.1% of the fully supervised performance based on vanilla fine-tuning and prompt-based learning respectively. Our implementation of PATRON is available at \url{https://github.com/yueyu1030/Patron}.

cs.CL↗
arXiv2023

Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation

Knowledge distillation is a popular technique to transfer knowledge from large teacher models to a small student model. Typically, the student learns to imitate the teacher by minimizing the KL divergence of its output distribution with the teacher's output distribution. In this work, we argue that such a learning objective is sub-optimal because there exists a discrepancy between the teacher's output distribution and the ground truth label distribution. Therefore, forcing the student to blindly imitate the unreliable teacher output distribution leads to inferior performance. To this end, we propose a novel knowledge distillation objective PTLoss by first representing the vanilla KL-based distillation loss function via a Maclaurin series and then perturbing the leading-order terms in this series. This perturbed loss implicitly transforms the original teacher into a proxy teacher with a distribution closer to the ground truth distribution. We establish the theoretical connection between this "distribution closeness" and the student model generalizability, which enables us to select the PTLoss's perturbation coefficients in a principled way. Extensive experiments on five datasets demonstrate PTLoss can significantly improve the distillation effectiveness for teachers of various scales.

cs.LG↗
arXiv2023

Semi-device-independent certification of indefinite causal order in a photonic quantum switch

We report an experimental certification of indefinite causal order that relies only on the characterization of the operations of a single party. We do so in the semi-device-independent scenario with the fewest possible assumptions of characterization of the parties' local operations in which indefinite causal order can be demonstrated with the quantum switch. To achieve this result, we introduce the concept of semi-device-independent causal inequalities and show that the correlations generated in a photonic quantum switch, in which all parties are able to collect local outcome statistics, achieve a violation of this inequality of 224 standard deviations. This result consists of the experimental demonstration of indefinite causal order with the fewest device-characterization assumptions to date.

quant-ph↗
arXiv2023

Detecting the secondary spin with extreme mass ratio inspirals in Scalar-Tensor theory

In this paper, we investigate the detectability of secondary spin in the extreme mass ratio inspirals system within a modified gravity model coupled with a scalar field. The central black hole, which reduces to a Kerr one, is circularly spiralled by a scalar-charged spinning secondary body on the equatorial plane. The analysis reveals that the presence of the scalar field amplifies the secondary spin effect, allowing for a lower limit of the detectability and an improved resolution of the secondary spin when the scalar charge is sufficiently large. Our findings suggest that the secondary spin detection is more feasible when the primary mass is not large, and TianQin is the optimal choice for detection.

gr-qc↗
arXiv2023

Bruce-Vincent transference numbers from molecular dynamics simulations

Transference number is a key design parameter for electrolyte materials used in electrochemical energy storage systems. However, the determination of the true transference number from experiments is rather demanding. On the other hand, the Bruce-Vincent method is widely used in the lab to measure transference numbers of polymer electrolytes approximately, which becomes exact in the limit of infinite dilution. Therefore, theoretical formulations to treat the Bruce-Vincent transference number and the true transference number on an equal footing are clearly needed. Here we show how the Bruce-Vincent transference number for concentrated electrolyte solutions can be derived in terms of the Onsager coefficients, without involving any extrathermodynamic assumptions. By demonstrating it for the case of PEO-LiTFSI system, this work opens the door to calibrating molecular dynamics (MD) simulations via reproducing the Bruce-Vincent transference number and using MD simulations as a predictive tool for determining the true transference number.

physics.chem-ph↗
arXiv2023

Quantum phases of lattice dipolar bosons coupled to a high-finesse cavity

Two types of long range interactions, dipolar interaction and cavity-mediated interaction lead to exotic quantum phases. Both interactions have been realized and observed in optical lattice setups. Here, we study quantum phases of dipolar bosons trapped in optical lattices and coupled to a high-finesse cavity where both dipolar interaction and cavity-mediated interaction coexist. We perform quantum Monte Carlo simulations, and find that the checkerboard solid is enhanced and the checkerboard supersolid phase can exist in a wide range of densities (e.g. $ 0.27\lesssim n\lesssim0.73 $). Our unbiased numerical results suggest that both solid and supersolid phases can be achieved experimentally with magnetic atoms coupled to a cavity.

cond-mat.quant-gas↗
arXiv2023

Accelerated Stochastic ADMM with Variance Reduction

Alternating Direction Method of Multipliers (ADMM) is a popular method for solving large-scale Machine Learning problems. Stochastic ADMM was proposed to reduce the per iteration computational complexity, which is more suitable for big data problems. Recently, variance reduction techniques have been integrated with stochastic ADMM in order to get a faster convergence rate, such as SAG-ADMM and SVRG-ADMM. However, their convergence rate is still suboptimal w.r.t the smoothness constant. In this paper, we propose an accelerated stochastic ADMM algorithm with variance reduction, which enjoys a faster convergence than all the existing stochastic ADMM algorithms. We theoretically analyse its convergence rate and show its dependence on the smoothness constant is optimal. We also empirically validate its effectiveness and show its priority over other stochastic ADMM algorithms.

math.NA↗
arXiv2023

An Asynchronous Decentralized Algorithm for Wasserstein Barycenter Problem

Wasserstein Barycenter Problem (WBP) has recently received much attention in the field of artificial intelligence. In this paper, we focus on the decentralized setting for WBP and propose an asynchronous decentralized algorithm (A$^2$DWB). A$^2$DWB is induced by a novel stochastic block coordinate descent method to optimize the dual of entropy regularized WBP. To our knowledge, A$^2$DWB is the first asynchronous decentralized algorithm for WBP. Unlike its synchronous counterpart, it updates local variables in a manner that only relies on the stale neighbor information, which effectively alleviate the waiting overhead, and thus substantially improve the time efficiency. Empirical results validate its superior performance compared to the latest synchronous algorithm.

cs.LG↗
arXiv2023

Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization

Nowadays, algorithms with fast convergence, small memory footprints, and low per-iteration complexity are particularly favorable for artificial intelligence applications. In this paper, we propose a doubly stochastic algorithm with a novel accelerating multi-momentum technique to solve large scale empirical risk minimization problem for learning tasks. While enjoying a provably superior convergence rate, in each iteration, such algorithm only accesses a mini batch of samples and meanwhile updates a small block of variable coordinates, which substantially reduces the amount of memory reference when both the massive sample size and ultra-high dimensionality are involved. Empirical studies on huge scale datasets are conducted to illustrate the efficiency of our method in practice.

cs.LG↗
arXiv2023

On the computation of thermo-relaxing multi-component flows with the Baer-Nunziato model

In inertial confinement fusion (ICF) implosions, mixing the ablator into the fuel and the hot spot is one of the most adverse factors that lead to ignition degradation. Recent experiments in the Marble campaign at the Omega laser facility and the National Ignition Facility (NIF) demonstrate the significance of the temperature separation in heterogeneous mixing flows. In the present work we provide an approach to deal with thermally disequilibrium multi-component flows with the ultimate aim to investigate the temperature separation impact on mixing and fusion burn. The present work is two-fold, i.e., (a)we derive a model governing the multi-component flows in thermal disequilibrium with transport terms, (b)we use the derived model to study the Rayleigh-Taylor (RT) instability in thermally relaxing multi-component systems. The model is reduced from the fully disequilibrium multi-phase Baer-Nunziato model in the limit of small Knudsen number $Kn << 1$. Velocity disequilibrium is closed with the diffusion laws and only one mass weighted velocity is retained formally. Thus, the complex wave structure of the original Baer-Nunziato model is simplified to a large extent and the obtained model is much more computationally affordable. Moreover, the capability to deal with finite temperature relaxation is kept. Efficient numerical methods for solving the proposed model are aslo presented. Equipped with the proposed model and numerical methods, we further investigate the impact of the thermal relaxation on the RT instability development at the inertial confinement fusion (ICF) deceleration stage. On the basis of numerical simulations, we have found that for the RT instability at an interface between the high-density low-temperature component and the low-density high-temperature component, the thermal relaxation significantly suppress the development of the instability.

physics.flu-dyn↗
arXiv2023

Cold-Start based Multi-Scenario Ranking Model for Click-Through Rate Prediction

Online travel platforms (OTPs), e.g., Ctrip.com or Fliggy.com, can effectively provide travel-related products or services to users. In this paper, we focus on the multi-scenario click-through rate (CTR) prediction, i.e., training a unified model to serve all scenarios. Existing multi-scenario based CTR methods struggle in the context of OTP setting due to the ignorance of the cold-start users who have very limited data. To fill this gap, we propose a novel method named Cold-Start based Multi-scenario Network (CSMN). Specifically, it consists of two basic components including: 1) User Interest Projection Network (UIPN), which firstly purifies users' behaviors by eliminating the scenario-irrelevant information in behaviors with respect to the visiting scenario, followed by obtaining users' scenario-specific interests by summarizing the purified behaviors with respect to the target item via an attention mechanism; and 2) User Representation Memory Network (URMN), which benefits cold-start users from users with rich behaviors through a memory read and write mechanism. CSMN seamlessly integrates both components in an end-to-end learning framework. Extensive experiments on real-world offline dataset and online A/B test demonstrate the superiority of CSMN over state-of-the-art methods.

cs.IR↗
arXiv2023

Hulk: Graph Neural Networks for Optimizing Regionally Distributed Computing Systems

Large deep learning models have shown great potential for delivering exceptional results in various applications. However, the training process can be incredibly challenging due to the models' vast parameter sizes, often consisting of hundreds of billions of parameters. Common distributed training methods, such as data parallelism, tensor parallelism, and pipeline parallelism, demand significant data communication throughout the process, leading to prolonged wait times for some machines in physically distant distributed systems. To address this issue, we propose a novel solution called Hulk, which utilizes a modified graph neural network to optimize distributed computing systems. Hulk not only optimizes data communication efficiency between different countries or even different regions within the same city, but also provides optimal distributed deployment of models in parallel. For example, it can place certain layers on a machine in a specific region or pass specific parameters of a model to a machine in a particular location. By using Hulk in experiments, we were able to improve the time efficiency of training large deep learning models on distributed systems by more than 20\%. Our open source collection of unlabeled data:https://github.com/DLYuanGod/Hulk.

cs.DC↗
arXiv2023

EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performance

In recent years, pretrained models have been widely used in various fields, including natural language understanding, computer vision, and natural language generation. However, the performance of these language generation models is highly dependent on the model size and the dataset size. While larger models excel in some aspects, they cannot learn up-to-date knowledge and are relatively difficult to relearn. In this paper, we introduce EvoText, a novel training method that enhances the performance of any natural language generation model without requiring additional datasets during the entire training process (although a prior dataset is necessary for pretraining). EvoText employs two models: $G$, a text generation model, and $D$, a model that can determine whether the data generated by $G$ is legitimate. Initially, the fine-tuned $D$ model serves as the knowledge base. The text generated by $G$ is then input to $D$ to determine whether it is legitimate. Finally, $G$ is fine-tuned based on $D$'s output. EvoText enables the model to learn up-to-date knowledge through a self-escalation process that builds on a priori knowledge. When EvoText needs to learn something new, it simply fine-tunes the $D$ model. Our approach applies to autoregressive language modeling for all Transformer classes. With EvoText, eight models achieved stable improvements in seven natural language processing tasks without any changes to the model structure.

cs.CL↗
arXiv2023

Experimental Examination of Entanglement Estimates

Recently a proper genuine multipartite entanglement (GME) measure has been found for three-qubit pure states [see Xie and Eberly, Phys. Rev. Lett. 127, 040403 (2021)], but capturing useful entanglement measures for mixed states has remained an open challenge. So far, it requires not only a full tomography in experiments, but also huge calculational labor. A leading proposal was made by Gühne, Reimpell, and Werner [Phys. Rev. Lett. 98, 110502 (2007)], who used expectation values of entanglement witnesses to describe a lower bound estimation of entanglement. We provide here an extension that also gives genuine upper bounds of entanglement. This advance requires only the expectation value of {\em any} Hermitian operator. Moreover, we identify a class of operators $\A_1$ which not only give good estimates, but also require a remarkably small number of experimental measurements. In this note we define our approach and illustrate it by estimating entanglement measures for a number of pure and mixed states prepared in our recent experiments.

quant-ph↗
arXiv2023

ODDFUZZ: Discovering Java Deserialization Vulnerabilities via Structure-Aware Directed Greybox Fuzzing

Java deserialization vulnerability is a severe threat in practice. Researchers have proposed static analysis solutions to locate candidate vulnerabilities and fuzzing solutions to generate proof-of-concept (PoC) serialized objects to trigger them. However, existing solutions have limited effectiveness and efficiency. In this paper, we propose a novel hybrid solution ODDFUZZ to efficiently discover Java deserialization vulnerabilities. First, ODDFUZZ performs lightweight static taint analysis to identify candidate gadget chains that may cause deserialization vulner-abilities. In this step, ODDFUZZ tries to locate all candidates and avoid false negatives. Then, ODDFUZZ performs directed greybox fuzzing (DGF) to explore those candidates and generate PoC testcases to mitigate false positives. Specifically, ODDFUZZ applies a structure-aware seed generation method to guarantee the validity of the testcases, and adopts a novel hybrid feedback and a step-forward strategy to guide the directed fuzzing. We implemented a prototype of ODDFUZZ and evaluated it on the popular Java deserialization repository ysoserial. Results show that, ODDFUZZ could discover 16 out of 34 known gadget chains, while two state-of-the-art baselines only identify three of them. In addition, we evaluated ODDFUZZ on real-world applications including Oracle WebLogic Server, Apache Dubbo, Sonatype Nexus, and protostuff, and found six previously unreported exploitable gadget chains with five CVEs assigned.

cs.CR↗
arXiv2023

Local behaviour of the mixed local and nonlocal problems with nonstandard growth

We consider the mixed local and nonlocal functionals with nonstandard growth \begin{eqnarray*} u\mapsto\int_Ω(|Du|^p-f(x)u)\,dx+\int_{\mathbb{R}^N}\int_{\mathbb{R}^N}\frac{|u(x)-u(y)|^q}{|x-y|^{N+sq}}\,dxdy \end{eqnarray*} with $1 sq$ in [De Filippis-Mingione, Math. Ann., https://doi.org/10.1007/s00208-022-02512-7].

math.AP↗
arXiv2023

Mutually-paced Knowledge Distillation for Cross-lingual Temporal Knowledge Graph Reasoning

This paper investigates cross-lingual temporal knowledge graph reasoning problem, which aims to facilitate reasoning on Temporal Knowledge Graphs (TKGs) in low-resource languages by transfering knowledge from TKGs in high-resource ones. The cross-lingual distillation ability across TKGs becomes increasingly crucial, in light of the unsatisfying performance of existing reasoning methods on those severely incomplete TKGs, especially in low-resource languages. However, it poses tremendous challenges in two aspects. First, the cross-lingual alignments, which serve as bridges for knowledge transfer, are usually too scarce to transfer sufficient knowledge between two TKGs. Second, temporal knowledge discrepancy of the aligned entities, especially when alignments are unreliable, can mislead the knowledge distillation process. We correspondingly propose a mutually-paced knowledge distillation model MP-KD, where a teacher network trained on a source TKG can guide the training of a student network on target TKGs with an alignment module. Concretely, to deal with the scarcity issue, MP-KD generates pseudo alignments between TKGs based on the temporal information extracted by our representation module. To maximize the efficacy of knowledge transfer and control the noise caused by the temporal knowledge discrepancy, we enhance MP-KD with a temporal cross-lingual attention mechanism to dynamically estimate the alignment strength. The two procedures are mutually paced along with model training. Extensive experiments on twelve cross-lingual TKG transfer tasks in the EventKG benchmark demonstrate the effectiveness of the proposed MP-KD method.

cs.LG↗
arXiv2023

Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition

Although large foundation models pre-trained by self-supervised learning have achieved state-of-the-art performance in many tasks including automatic speech recognition (ASR), knowledge distillation (KD) is often required in practice to transfer the knowledge learned by large teacher models into much smaller student models with affordable computation and memory costs. This paper proposes a novel two-stage KD framework to distil the knowledge from multiple speech foundation models as teachers into a single student neural transducer model for ASR. In the first stage, the student model encoder is pre-trained using the embeddings extracted from multiple teacher models. In the second stage, the student encoder is fine-tuned with the audio-text pairs based on the ASR task. Experiments on the LibriSpeech 100-hour subset show that the proposed KD framework improves the performance of both streaming and non-streaming student models when using only one teacher. The performance of the student model can be further enhanced when multiple teachers are used jointly, achieving word error rate reductions (WERRs) of 17.5% and 10.6%. Our proposed framework can be combined with other existing KD methods to achieve further improvements. Further WERRs were obtained by incorporating extra unlabelled data during encoder pre-training, leading to a total relative WERR of 55.0% on the non-streaming student model.

eess.AS↗