arXiv subjects

Xinyi Wang

Publications and source records attributed to Xinyi Wang.

At least 217 records · Page 12Linked to original sources

arXiv2022

A Comprehensive Review on Deep Supervision: Theories and Applications

Deep supervision, or known as 'intermediate supervision' or 'auxiliary supervision', is to add supervision at hidden layers of a neural network. This technique has been increasingly applied in deep neural network learning systems for various computer vision applications recently. There is a consensus that deep supervision helps improve neural network performance by alleviating the gradient vanishing problem, as one of the many strengths of deep supervision. Besides, in different computer vision applications, deep supervision can be applied in different ways. How to make the most use of deep supervision to improve network performance in different applications has not been thoroughly investigated. In this paper, we provide a comprehensive in-depth review of deep supervision in both theories and applications. We propose a new classification of different deep supervision networks, and discuss advantages and limitations of current deep supervision networks in computer vision applications.

cs.CV↗

arXiv2022

QuSBT: Search-Based Testing of Quantum Programs

Generating a test suite for a quantum program such that it has the maximum number of failing tests is an optimization problem. For such optimization, search-based testing has shown promising results in the context of classical programs. To this end, we present a test generation tool for quantum programs based on a genetic algorithm, called QuSBT (Search-based Testing of Quantum Programs). QuSBT automates the testing of quantum programs, with the aim of finding a test suite having the maximum number of failing test cases. QuSBT utilizes IBM's Qiskit as the simulation framework for quantum programs. We present the tool architecture in addition to the implemented methodology (i.e., the encoding of the search individual, the definition of the fitness function expressing the search problem, and the test assessment w.r.t. two types of failures). Finally, we report results of the experiments in which we tested a set of faulty quantum programs with QuSBT to assess its effectiveness. Repository (code and experimental results): https://github.com/Simula-COMPLEX/qusbt-tool Video: https://youtu.be/3apRCtluAn4

cs.SE↗

arXiv2022

Noise-induce coexisting firing patterns in hybrid-synaptic interacting networks

Synaptic noise plays a major role in setting up coexistence of various firing patterns, but the precise mechanisms whereby these synaptic noise contributes to coexisting firing activities are subtle and remain elusive. To investigate these mechanisms, neurons with hybrid synaptic interaction in a balanced neuronal networks have been recently put forward. Here we show that both synaptic noise intensity and excitatory weights can make a greater contribution than variance of synaptic noise to the coexistence of firing states with slight modification parameters. The resulting statistical analysis of both voltage trajectories and their spike trains reveals two forms of coexisting firing patterns: time-varying and parameter-varying multistability. The emergence of time-varying multistability as a format of metstable state has been observed under suitable parameters settings of noise intensity and excitatory synaptic weight. While the parameter-varying multistability is accompanied by coexistence of synchrony state and metastable (or asynchronous firing state) with slightly varying noise intensity and excitatory weights. Our results offer a series of precise statistical explanation of the intricate effect of synaptic noise in neural multistability. This reconciles previous theoretical and numerical works, and confirms the suitability of various statistical methods to investigate multistability in a hybrid synaptic interacting neuronal networks.

physics.bio-ph↗

arXiv2022

Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

The performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text present in a target language. Thus, the majority of the world's languages cannot benefit from recent progress in NLP as they have no or limited textual data. To expand possibilities of using NLP technology in these under-represented languages, we systematically study strategies that relax the reliance on conventional language resources through the use of bilingual lexicons, an alternative resource with much better language coverage. We analyze different strategies to synthesize textual or labeled data using lexicons, and how this data can be combined with monolingual or parallel text when available. For 19 under-represented languages across 3 tasks, our methods lead to consistent improvements of up to 5 and 15 points with and without extra monolingual text respectively. Overall, our study highlights how NLP methods can be adapted to thousands more languages that are under-served by current technology

cs.CL↗

arXiv2022

Towards Bi-directional Skip Connections in Encoder-Decoder Architectures and Beyond

U-Net, as an encoder-decoder architecture with forward skip connections, has achieved promising results in various medical image analysis tasks. Many recent approaches have also extended U-Net with more complex building blocks, which typically increase the number of network parameters considerably. Such complexity makes the inference stage highly inefficient for clinical applications. Towards an effective yet economic segmentation network design, in this work, we propose backward skip connections that bring decoded features back to the encoder. Our design can be jointly adopted with forward skip connections in any encoder-decoder architecture forming a recurrence structure without introducing extra parameters. With the backward skip connections, we propose a U-Net based network family, namely Bi-directional O-shape networks, which set new benchmarks on multiple public medical imaging segmentation datasets. On the other hand, with the most plain architecture (BiO-Net), network computations inevitably increase along with the pre-set recurrence time. We have thus studied the deficiency bottleneck of such recurrent design and propose a novel two-phase Neural Architecture Search (NAS) algorithm, namely BiX-NAS, to search for the best multi-scale bi-directional skip connections. The ineffective skip connections are then discarded to reduce computational costs and speed up network inference. The finally searched BiX-Net yields the least network complexity and outperforms other state-of-the-art counterparts by large margins. We evaluate our methods on both 2D and 3D segmentation tasks in a total of six datasets. Extensive ablation studies have also been conducted to provide a comprehensive analysis for our proposed methods.

cs.CV↗

arXiv2022

Asynchronous Decentralized Federated Learning for Collaborative Fault Diagnosis of PV Stations

Due to the different losses caused by various photovoltaic (PV) array faults, accurate diagnosis of fault types is becoming increasingly important. Compared with a single one, multiple PV stations collect sufficient fault samples, but their data is not allowed to be shared directly due to potential conflicts of interest. Therefore, federated learning can be exploited to train a collaborative fault diagnosis model. However, the modeling efficiency is seriously affected by the model update mechanism since each PV station has a different computing capability and amount of data. Moreover, for the safe and stable operation of the PV system, the robustness of collaborative modeling must be guaranteed rather than simply being processed on a central server. To address these challenges, a novel asynchronous decentralized federated learning (ADFL) framework is proposed. Each PV station not only trains its local model but also participates in collaborative fault diagnosis by exchanging model parameters to improve the generalization without losing accuracy. The global model is aggregated distributedly to avoid central node failure. By designing the asynchronous update scheme, the communication overhead and training time are greatly reduced. Both the experiments and numerical simulations are carried out to verify the effectiveness of the proposed method.

cs.LG↗

arXiv2021

Numerical Simulation of a Fundamental Mechanism of Solar Eruption with Different Magnetic Flux Distributions

Solar eruptions are explosive release of coronal magnetic field energy as manifested in solar flares and coronal mass ejection. Observations have shown that the core of eruption-productive regions are often a sheared magnetic arcade, i.e., a single bipolar configuration, and, particularly, the corresponding magnetic polarities at the photosphere are elongated along a strong-gradient polarity inversion line (PIL). It remains unclear what mechanism triggers the eruption in a single bipolar field and why the one with a strong PIL is eruption-productive. Recently, using high accuracy simulations, we have established a fundamental mechanism of solar eruption initiation that a bipolar field as driven by quasi-static shearing motion at the photosphere can form an internal current sheet, and then fast magnetic reconnection triggers and drives the eruption. Here we investigate the behavior of the fundamental mechanism with different photospheric magnetic flux distributions, i.e., magnetograms, by combining theoretical analysis and numerical simulation. Our study shows that the bipolar fields of different magnetograms, as sheared continually, all exhibit similar evolutions from the slow storage to fast release of magnetic energy in accordance with the fundamental mechanism, which demonstrates the robustness of the mechanism. We further found that the magnetograms with stronger PIL produce larger eruptions, and the key reason is that the sheared bipolar fields with stronger PIL can achieve more non-potentiality, and their internal current sheet can form at a lower height and with a larger current density, by which the reconnection can be more efficient. This also provides a viable trigger mechanism for the observed eruptions in active region with strong PIL.

astro-ph.SR↗

arXiv2021

The Initiation of Shear Band Formation in Deformed Metallic Glasses from Soft Localized Domains

It has long been thought that shear band (SB) formation in amorphous solids initiates from relatively 'soft' regions in the material in which large-scale non-affine deformations become localized. The test of this hypothesis requires an effective means of identifying 'soft' regions and their evolution as the material is deformed to varying degrees, where the metric of 'softness' must also account for the effect of temperature on local material stiffness. We show that the mean square atomic displacement on a caging timescale , the 'Debye-Waller factor', provides a useful method for estimating the shear modulus of the entire material and, by extension, the material stiffness at an atomic scale. Based on this 'softness' metrology, we observe that SB formation indeed occurs through the strain-induced formation of localized soft regions in our deformed metallic glass free-standing films. Unexpectedly, the critical strain condition of SB formation occurs when the softness ( ) distribution within the emerging soft regions approaches that of the interfacial region in its undeformed state, initiating an instability with similarities to the transition to turbulence. Correspondingly, no SBs arise when the material is so thin that the entire material can be approximately described as being 'interfacial' in nature. We also quantify relaxation in the glass and the nature and origin of highly non-Gaussian particle displacements in the dynamically heterogeneous SB regions at times longer than the caging time.

cond-mat.mtrl-sci↗

arXiv2021

Counterfactual Maximum Likelihood Estimation for Training Deep Networks

Although deep learning models have driven state-of-the-art performance on a wide array of tasks, they are prone to spurious correlations that should not be learned as predictive clues. To mitigate this problem, we propose a causality-based training framework to reduce the spurious correlations caused by observed confounders. We give theoretical analysis on the underlying general Structural Causal Model (SCM) and propose to perform Maximum Likelihood Estimation (MLE) on the interventional distribution instead of the observational distribution, namely Counterfactual Maximum Likelihood Estimation (CMLE). As the interventional distribution, in general, is hidden from the observational data, we then derive two different upper bounds of the expected negative log-likelihood and propose two general algorithms, Implicit CMLE and Explicit CMLE, for causal predictions of deep learning models using observational data. We conduct experiments on both simulated data and two real-world tasks: Natural Language Inference (NLI) and Image Captioning. The results show that CMLE methods outperform the regular MLE method in terms of out-of-domain generalization performance and reducing spurious correlations, while maintaining comparable performance on the regular evaluations.

cs.LG↗

arXiv2021

A Dataset for Answering Time-Sensitive Questions

Time is an important dimension in our physical world. Lots of facts can evolve with respect to time. For example, the U.S. President might change every four years. Therefore, it is important to consider the time dimension and empower the existing QA models to reason over time. However, the existing QA datasets contain rather few time-sensitive questions, hence not suitable for diagnosing or benchmarking the model's temporal reasoning capability. In order to promote research in this direction, we propose to construct a time-sensitive QA dataset. The dataset is constructed by 1) mining time-evolving facts from WikiData and aligning them to their corresponding Wikipedia page, 2) employing crowd workers to verify and calibrate these noisy facts, 3) generating question-answer pairs based on the annotated time-sensitive facts. Our dataset poses challenges in the aspect of both temporal understanding and temporal reasoning. We evaluate different SoTA long-document QA systems like BigBird and FiD on our dataset. The best-performing model FiD can only achieve 46\% accuracy, still far behind the human performance of 87\%. We demonstrate that these models are still lacking the ability to perform consistent temporal reasoning. Therefore, we believe that our dataset could serve as a benchmark to develop NLP models more sensitive to temporal shifts. The dataset and code are released in~\url{https://github.com/wenhuchen/Time-Sensitive-QA}.

cs.CL↗

arXiv2021

Formation of Magnetic Flux Rope during Solar Eruption. I. Evolution of Toroidal Flux and Reconnection Flux

Magnetic flux ropes (MFRs) constitute the core structure of coronal mass ejections (CMEs), but hot debates remain on whether the MFR forms before or during solar eruptions. Furthermore, how flare reconnection shapes the erupting MFR is still elusive in three dimensions. Here we studied a new MHD simulation of CME initiation by tether-cutting magnetic reconnection in a single magnetic arcade. The simulation follows the whole life, including the birth and subsequent evolution, of an MFR during eruption. In the early phase, the MFR is partially separated from its ambient field by a magnetic quasi-separatrix layer (QSL) that has a double-J shaped footprint on the bottom surface. With the ongoing of the reconnection, the arms of the two J-shaped footprints continually separate from each other, and the hooks of the J shaped footprints expand and eventually become closed almost at the eruption peak time, and thereafter the MFR is fully separated from the un-reconnected field by the QSL. We further studied the evolution of the toroidal flux in the MFR and compared it with that of the reconnected flux. Our simulation reproduced an evolution pattern of increase-to-decrease of the toroidal flux, which is reported recently in observations of variations in flare ribbons and transient coronal dimming. The increase of toroidal flux is owing to the flare reconnection in the early phase that transforms the sheared arcade to twisted field lines, while its decrease is a result of reconnection between field lines in the interior of the MFR in the later phase.

astro-ph.SR↗

arXiv2021

Efficient Test Time Adapter Ensembling for Low-resource Language Varieties

Adapters are light-weight modules that allow parameter-efficient fine-tuning of pretrained models. Specialized language and task adapters have recently been proposed to facilitate cross-lingual transfer of multilingual pretrained models (Pfeiffer et al., 2020b). However, this approach requires training a separate language adapter for every language one wishes to support, which can be impractical for languages with limited data. An intuitive solution is to use a related language adapter for the new language variety, but we observe that this solution can lead to sub-optimal performance. In this paper, we aim to improve the robustness of language adapters to uncovered languages without training new adapters. We find that ensembling multiple existing language adapters makes the fine-tuned model significantly more robust to other language varieties not included in these adapters. Building upon this observation, we propose Entropy Minimized Ensemble of Adapters (EMEA), a method that optimizes the ensemble weights of the pretrained language adapters for each test sentence by minimizing the entropy of its predictions. Experiments on three diverse groups of language varieties show that our method leads to significant improvements on both named entity recognition and part-of-speech tagging across all languages.

cs.CL↗

arXiv2021

Modeling Disclosive Transparency in NLP Application Descriptions

Broader disclosive transparency$-$truth and clarity in communication regarding the function of AI systems$-$is widely considered desirable. Unfortunately, it is a nebulous concept, difficult to both define and quantify. This is problematic, as previous work has demonstrated possible trade-offs and negative consequences to disclosive transparency, such as a confusion effect, where "too much information" clouds a reader's understanding of what a system description means. Disclosive transparency's subjective nature has rendered deep study into these problems and their remedies difficult. To improve this state of affairs, We introduce neural language model-based probabilistic metrics to directly model disclosive transparency, and demonstrate that they correlate with user and expert opinions of system transparency, making them a valid objective proxy. Finally, we demonstrate the use of these metrics in a pilot study quantifying the relationships between transparency, confusion, and user perceptions in a corpus of real NLP system descriptions.

cs.CL↗

arXiv2021

Innovations Autoencoder and its Application in One-class Anomalous Sequence Detection

An innovations sequence of a time series is a sequence of independent and identically distributed random variables with which the original time series has a causal representation. The innovation at a time is statistically independent of the history of the time series. As such, it represents the new information contained at present but not in the past. Because of its simple probability structure, an innovations sequence is the most efficient signature of the original. Unlike the principle or independent component analysis representations, an innovations sequence preserves not only the complete statistical properties but also the temporal order of the original time series. An long-standing open problem is to find a computationally tractable way to extract an innovations sequence of non-Gaussian processes. This paper presents a deep learning approach, referred to as Innovations Autoencoder (IAE), that extracts innovations sequences using a causal convolutional neural network. An application of IAE to the one-class anomalous sequence detection problem with unknown anomaly and anomaly-free models is also presented.

stat.ML↗

arXiv2021

BiX-NAS: Searching Efficient Bi-directional Architecture for Medical Image Segmentation

The recurrent mechanism has recently been introduced into U-Net in various medical image segmentation tasks. Existing studies have focused on promoting network recursion via reusing building blocks. Although network parameters could be greatly saved, computational costs still increase inevitably in accordance with the pre-set iteration time. In this work, we study a multi-scale upgrade of a bi-directional skip connected network and then automatically discover an efficient architecture by a novel two-phase Neural Architecture Search (NAS) algorithm, namely BiX-NAS. Our proposed method reduces the network computational cost by sifting out ineffective multi-scale features at different levels and iterations. We evaluate BiX-NAS on two segmentation tasks using three different medical image datasets, and the experimental results show that our BiX-NAS searched architecture achieves the state-of-the-art performance with significantly lower computational cost.

eess.IV↗

arXiv2021

A Deep Learning Approach to Anomaly Sequence Detection for High-Resolution Monitoring of Power Systems

A deep learning approach is proposed to detect data and system anomalies using high-resolution continuous point-on-wave (CPOW) or phasor measurements. Both the anomaly and anomaly-free measurement models are assumed to have unknown temporal dependencies and probability distributions. Historical training samples are assumed for the anomaly-free model, while no training samples are available for the anomaly measurements. By transforming the anomaly-free observations into uniform independent and identically distributed sequences via a generative adversarial network, the proposed approach deploys a uniformity test for anomaly detection at the sensor level. A distributed detection scheme that combines sensor level detections at the control center is also proposed that combines local detections to form more reliable detections. Numerical results demonstrate significant improvement over the state-of-the-art solutions for various bad-data cases using real and synthetic CPOW and PMU data sets.

eess.SY↗

arXiv2021

Optimizing Data Usage via Differentiable Rewards

To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, a machine learning model could potentially be trained better with a scorer that "adapts" to its current learning state and estimates the importance of each training data instance. Training such an adaptive scorer efficiently is a challenging problem; in order to precisely quantify the effect of a data instance at a given time during the training, it is typically necessary to first complete the entire training process. To efficiently optimize data usage, we propose a reinforcement learning approach called Differentiable Data Selection (DDS). In DDS, we formulate a scorer network as a learnable function of the training data, which can be efficiently updated along with the main model being trained. Specifically, DDS updates the scorer with an intuitive reward signal: it should up-weigh the data that has a similar gradient with a dev set upon which we would finally like to perform well. Without significant computing overhead, DDS delivers strong and consistent improvements over several strong baselines on two very different tasks of machine translation and image classification.

cs.LG↗

arXiv2021

RefBERT: Compressing BERT by Referencing to Pre-computed Representations

Recently developed large pre-trained language models, e.g., BERT, have achieved remarkable performance in many downstream natural language processing applications. These pre-trained language models often contain hundreds of millions of parameters and suffer from high computation and latency in real-world applications. It is desirable to reduce the computation overhead of the models for fast training and inference while keeping the model performance in downstream applications. Several lines of work utilize knowledge distillation to compress the teacher model to a smaller student model. However, they usually discard the teacher's knowledge when in inference. Differently, in this paper, we propose RefBERT to leverage the knowledge learned from the teacher, i.e., facilitating the pre-computed BERT representation on the reference sample and compressing BERT into a smaller student model. To guarantee our proposal, we provide theoretical justification on the loss function and the usage of reference samples. Significantly, the theoretical result shows that including the pre-computed teacher's representations on the reference samples indeed increases the mutual information in learning the student model. Finally, we conduct the empirical evaluation and show that our RefBERT can beat the vanilla TinyBERT over 8.1\% and achieves more than 94\% of the performance of $\BERTBASE$ on the GLUE benchmark. Meanwhile, RefBERT is 7.4x smaller and 9.5x faster on inference than BERT$_{\rm BASE}$.

cs.CL↗