arXiv subjects

Di Wu

Publications and source records attributed to Di Wu.

At least 541 records · Page 30Linked to original sources

arXiv2022

FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition

The recently proposed Conformer architecture which combines convolution with attention to capture both local and global dependencies has become the \textit{de facto} backbone model for Automatic Speech Recognition~(ASR). Inherited from the Natural Language Processing (NLP) tasks, the architecture takes Layer Normalization~(LN) as a default normalization technique. However, through a series of systematic studies, we find that LN might take 10\% of the inference time despite that it only contributes to 0.1\% of the FLOPs. This motivates us to replace LN with other normalization techniques, e.g., Batch Normalization~(BN), to speed up inference with the help of operator fusion methods and the avoidance of calculating the mean and variance statistics during inference. After examining several plain attempts which directly remove all LN layers or replace them with BN in the same place, we find that the divergence issue is mainly caused by the unstable layer output. We therefore propose to append a BN layer to each linear or convolution layer where stabilized training results are observed. We also propose to simplify the activations in Conformer, such as Swish and GLU, by replacing them with ReLU. All these exchanged modules can be fused into the weights of the adjacent linear/convolution layers and hence have zero inference cost. Therefore, we name it FusionFormer. Our experiments indicate that FusionFormer is as effective as the LN-based Conformer and is about 10\% faster.

cs.SD↗

arXiv2022

Representation Learning for Resource-Constrained Keyphrase Generation

State-of-the-art keyphrase generation methods generally depend on large annotated datasets, limiting their performance in domains with limited annotated data. To overcome this challenge, we design a data-oriented approach that first identifies salient information using retrieval-based corpus-level statistics, and then learns a task-specific intermediate representation based on a pre-trained language model using large-scale unlabeled documents. We introduce salient span recovery and salient span prediction as denoising training objectives that condense the intra-article and inter-article knowledge essential for keyphrase generation. Through experiments on multiple keyphrase generation benchmarks, we show the effectiveness of the proposed approach for facilitating low-resource keyphrase generation and zero-shot domain adaptation. Our method especially benefits the generation of absent keyphrases, approaching the performance of models trained with large training sets.

cs.CL↗

arXiv2022

Transfer Learning on Electromyography (EMG) Tasks: Approaches and Beyond

Machine learning on electromyography (EMG) has recently achieved remarkable success on a variety of tasks, while such success relies heavily on the assumption that the training and future data must be of the same data distribution. However, this assumption may not hold in many real-world applications. Model calibration is required via data re-collection and label annotation, which is generally very expensive and time-consuming. To address this problem, transfer learning (TL), which aims to improve target learners' performance by transferring the knowledge from related source domains, is emerging as a new paradigm to reduce the amount of calibration effort. In this survey, we assess the eligibility of more than fifty published peer-reviewed representative transfer learning approaches for EMG applications. Unlike previous surveys on purely transfer learning or EMG-based machine learning, this survey aims to provide an insight into the biological foundations of existing transfer learning methods on EMG-related analysis. In specific, we first introduce the physiological structure of the muscles and the EMG generating mechanism, and the recording of EMG to provide biological insights behind existing transfer learning approaches. Further, we categorize existing research endeavors into data based, model based, training scheme based, and adversarial based. This survey systematically summarizes and categorizes existing transfer learning approaches for EMG related machine learning applications. In addition, we discuss possible drawbacks of existing works and point out the future direction of better EMG transfer learning algorithms to enhance practicality for real-world applications.

eess.SP↗

arXiv2022

Exploring Localization for Self-supervised Fine-grained Contrastive Learning

Self-supervised contrastive learning has demonstrated great potential in learning visual representations. Despite their success in various downstream tasks such as image classification and object detection, self-supervised pre-training for fine-grained scenarios is not fully explored. We point out that current contrastive methods are prone to memorizing background/foreground texture and therefore have a limitation in localizing the foreground object. Analysis suggests that learning to extract discriminative texture information and localization are equally crucial for fine-grained self-supervised pre-training. Based on our findings, we introduce cross-view saliency alignment (CVSA), a contrastive learning framework that first crops and swaps saliency regions of images as a novel view generation and then guides the model to localize on foreground objects via a cross-view alignment loss. Extensive experiments on both small- and large-scale fine-grained classification benchmarks show that CVSA significantly improves the learned representation.

cs.CV↗

arXiv2022

AutoMix: Unveiling the Power of Mixup for Stronger Classifiers

Data mixing augmentation have proved to be effective in improving the generalization ability of deep neural networks. While early methods mix samples by hand-crafted policies (e.g., linear interpolation), recent methods utilize saliency information to match the mixed samples and labels via complex offline optimization. However, there arises a trade-off between precise mixing policies and optimization complexity. To address this challenge, we propose a novel automatic mixup (AutoMix) framework, where the mixup policy is parameterized and serves the ultimate classification goal directly. Specifically, AutoMix reformulates the mixup classification into two sub-tasks (i.e., mixed sample generation and mixup classification) with corresponding sub-networks and solves them in a bi-level optimization framework. For the generation, a learnable lightweight mixup generator, Mix Block, is designed to generate mixed samples by modeling patch-wise relationships under the direct supervision of the corresponding mixed labels. To prevent the degradation and instability of bi-level optimization, we further introduce a momentum pipeline to train AutoMix in an end-to-end manner. Extensive experiments on nine image benchmarks prove the superiority of AutoMix compared with state-of-the-art in various classification scenarios and downstream tasks.

cs.CV↗

arXiv2022

Manipulating THz Spin Current Dynamics by the Dzyaloshinskii-Moriya Interaction in Antiferromagnetic Hematite

An important vision of modern magnetic research is to use antiferromagnets as controllable and active ultrafast components in spintronic devices. Hematite is a promising model material in this respect because its pronounced Dzyaloshinskii-Moriya interaction leads to the coexistence of antiferromagnetism and weak ferromagnetism. Here, we use femtosecond laser pulses to drive terahertz spin currents from hematite into an adjacent Pt layer. We find two contributions to the generation of the spin current with distinctly different dynamics: the impulsive stimulated Raman scatting that relies on the antiferromagnetic order and the ultrafast spin Seebeck effect that relies on the net magnetization. The total THz spin current dynamics can thus be manipulated by a medium-strength magnetic field. The controllability of the THz spin current achieved in hematite opens the pathway toward controlling the exact spin current dynamics from ultrafast antiferromagnetic spin sources.

cond-mat.mes-hall↗

arXiv2022

An ICA-Based HVAC Load Disaggregation Method Using Smart Meter Data

This paper presents an independent component analysis (ICA) based unsupervised-learning method for heat, ventilation, and air-conditioning (HVAC) load disaggregation using low-resolution (e.g., 15 minutes) smart meter data. We first demonstrate that electricity consumption profiles on mild-temperature days can be used to estimate the non-HVAC base load on hot days. A residual load profile can then be calculated by subtracting the mild-day load profile from the hot-day load profile. The residual load profiles are processed using ICA for HVAC load extraction. An optimization-based algorithm is proposed for post-adjustment of the ICA results, considering two bounding factors for enhancing the robustness of the ICA algorithm. First, we use the hourly HVAC energy bounds computed based on the relationship between HVAC load and temperature to remove unrealistic HVAC load spikes. Second, we exploit the dependency between the daily nocturnal and diurnal loads extracted from historical meter data to smooth the base load profile. Pecan Street data with sub-metered HVAC data were used to test and validate the proposed methods.Simulation results demonstrated that the proposed method is computationally efficient and robust across multiple customers.

eess.SY↗

arXiv2022

Data-driven Ranking and Selection under Input Uncertainty

We consider a simulation-based Ranking and Selection (R&S) problem with input uncertainty, where unknown input distributions can be estimated using input data arriving in batches of varying sizes over time. Each time a batch arrives, additional simulations can be run using updated input distribution estimates. The goal is to confidently identify the best design after collecting as few batches as possible. We first introduce a moving average estimator for aggregating simulation outputs generated under heterogenous input distributions. Then, based on a Sequential Elimination framework, we devise two major R&S procedures by establishing exact and asymptotic confidence bands for the estimator. In deriving the latter confidence bands, we incorporate the result of "Multiple Comparison with Best" and establish an asymptotic normality result which explicitly characterizes the tradeoff between input uncertainty and stochastic uncertainty in an online environment. We also extend our procedures to the indifference zone setting, which helps save simulation effort for practical usage. Numerical results show the effectiveness and necessity of our procedures. Moreover, the efficiency can be further boosted through optimizing the "drop rate" parameter of the estimator.

math.OC↗

arXiv2022

MALICE: Manipulation Attacks on Learned Image ComprEssion

Deep learning techniques have shown promising results in image compression, with competitive bitrate and image reconstruction quality from compressed latent. However, while image compression has progressed towards a higher peak signal-to-noise ratio (PSNR) and fewer bits per pixel (bpp), their robustness to adversarial images has never received deliberation. In this work, we, for the first time, investigate the robustness of image compression systems where imperceptible perturbation of input images can precipitate a significant increase in the bitrate of their compressed latent. To characterize the robustness of state-of-the-art learned image compression, we mount white-box and black-box attacks. Our white-box attack employs fast gradient sign method on the entropy estimation of the bitstream as its bitrate approximation. We propose DCT-Net simulating JPEG compression with architectural simplicity and lightweight training as the substitute in the black-box attack and enable fast adversarial transferability. Our results on six image compression models, each with six different bitrate qualities (thirty-six models in total), show that they are surprisingly fragile, where the white-box attack achieves up to 56.326x and black-box 1.947x bpp change. To improve robustness, we propose a novel compression architecture factorAtn which incorporates attention modules and a basic factorized entropy model, resulting in a promising trade-off between the rate-distortion performance and robustness to adversarial attacks that surpasses existing learned image compressors.

cs.CV↗

arXiv2022

FedMCSA: Personalized Federated Learning via Model Components Self-Attention

Federated learning (FL) facilitates multiple clients to jointly train a machine learning model without sharing their private data. However, Non-IID data of clients presents a tough challenge for FL. Existing personalized FL approaches rely heavily on the default treatment of one complete model as a basic unit and ignore the significance of different layers on Non-IID data of clients. In this work, we propose a new framework, federated model components self-attention (FedMCSA), to handle Non-IID data in FL, which employs model components self-attention mechanism to granularly promote cooperation between different clients. This mechanism facilitates collaboration between similar model components while reducing interference between model components with large differences. We conduct extensive experiments to demonstrate that FedMCSA outperforms the previous methods on four benchmark datasets. Furthermore, we empirically show the effectiveness of the model components self-attention mechanism, which is complementary to existing personalized FL and can significantly improve the performance of FL.

cs.LG↗

arXiv2022

FedComm: Understanding Communication Protocols for Edge-based Federated Learning

Federated learning (FL) trains machine learning (ML) models on devices using locally generated data and exchanges models without transferring raw data to a distant server. This exchange incurs a communication overhead and impacts the performance of FL training. There is limited understanding of how communication protocols specifically contribute to the performance of FL. Such an understanding is essential for selecting the right communication protocol when designing an FL system. This paper presents FedComm, a benchmarking methodology to quantify the impact of optimized application layer protocols, namely Message Queue Telemetry Transport (MQTT), Advanced Message Queuing Protocol (AMQP), and ZeroMQ Message Transport Protocol (ZMTP), and non-optimized application layer protocols, namely as TCP and UDP, on the performance of FL. FedComm measures the overall performance of FL in terms of communication time and accuracy under varying computational and network stress and packet loss rates. Experiments on a lab-based testbed demonstrate that TCP outperforms UDP as a non-optimized application layer protocol with higher accuracy and shorter communication times for 4G and Wi-Fi networks. Optimized application layer protocols such as AMQP, MQTT, and ZMTP outperformed non-optimized application layer protocols in most network conditions, resulting in a 2.5x reduction in communication time compared to TCP while maintaining accuracy. The experimental results enable us to highlight a number of open research issues for further investigation. FedComm is available for download from https://github.com/qub-blesson/FedComm.

cs.DC↗

arXiv2022

An Atlas of Optimal Low-Thrust Rephasing Solutions in Circular Orbit

In this paper, the time- and propellant-optimal low-thrust rephasing problems in circular orbit are studied to depict their solution spaces in an atlas. The number of key parameters that settle the rephasing problems is reduced by developing a set of linearized equations of motion based on the Sundman transformation and by formulating two reduced shooting functions using the minimum principle and symmetry properties. Only one key parameter is identified for the time-optimal problem, while two key parameters are obtained for the propellant-optimal one. Numerical investigation of the relationships between these parameters and shooting variables reveals that they can be depicted by some curve (or contour) maps and approximated by piecewise functions (or linear interpolations). For the relatively short- or long-term rephasing cases, some analytical time- and propellant-optimal solutions are proposed and consistent with the numerical solutions. Numerical results demonstrate that the proposed solutions can provide good initial guesses to solve the low-thrust rephasing problems with nonlinear dynamics. Moreover, the approximations of the performance indexes can be used in the preliminary mission design.

math.DS↗

arXiv2022

A Latent Feature Analysis-based Approach for Spatio-Temporal Traffic Data Recovery

Missing data is an inevitable and common problem in data-driven intelligent transportation systems (ITS). In the past decade, scholars have done many research on the recovery of missing traffic data, however how to make full use of spatio-temporal traffic patterns to improve the recovery performance is still an open problem. Aiming at the spatio-temporal characteristics of traffic speed data, this paper regards the recovery of missing data as a matrix completion problem, and proposes a spatio-temporal traffic data completion method based on hidden feature analysis, which discovers spatio-temporal patterns and underlying structures from incomplete data to complete the recovery task. Therefore, we introduce spatial and temporal correlation to capture the main underlying features of each dimension. Finally, these latent features are applied to recovery traffic data through latent feature analysis. The experimental and evaluation results show that the evaluation criterion value of the model is small, which indicates that the model has better performance. The results show that the model can accurately estimate the continuous missing data.

eess.SP↗

arXiv2022

Distinguishing two-component anomalous Hall effect from topological Hall effect

In transport, the topological Hall effect (THE) presents itself as non-monotonic features (or humps and dips) in the Hall signal and is widely interpreted as a sign of chiral spin textures, like magnetic skyrmions. However, when anomalous Hall effect (AHE) is also present, the co-existence of two AHEs could give rise to similar artifacts, making it difficult to distinguish between genuine THE with AHE and two-component AHE. Here we confirm genuine THE with AHE by means of transport and magneto-optical Kerr effect (MOKE) microscopy, in which magnetic skyrmions are directly observed, and find that genuine THE occurs in the transition region of the AHE. In sharp contrast, the artifact "THE", or two-component AHE occurs well beyond the saturation of the "AHE component" (under the false assumption of THE+AHE). Furthermore, we distinguish artifact "THE" from genuine THE by three methods: 1. Minor loops, 2. Temperature dependence, 3. Gate dependence. Minor loops of genuine THE with AHE are always within the full loop, while minor loops of the artifact "THE" may reveal a single loop that cannot fit into the "AHE component". Besides, the temperature or gate dependence of the artifact "THE" may also be accompanied by a polarity change of the "AHE component", as the non-monotonic features vanish, while the temperature dependence of genuine THE with AHE reveals no such change. Our work may help future researchers to exercise cautions and use these methods to examine carefully in order to ascertain genuine THE.

cond-mat.mtrl-sci↗

arXiv2022

An Adam-adjusting-antennae BAS Algorithm for Refining Latent Factors

Extracting the latent information in high-dimensional and incomplete matrices is an important and challenging issue. The Latent Factor Analysis (LFA) model can well handle the high-dimensional matrices analysis. Recently, Particle Swarm Optimization (PSO)-incorporated LFA models have been proposed to tune the hyper-parameters adaptively with high efficiency. However, the incorporation of PSO causes the premature problem. To address this issue, we propose a sequential Adam-adjusting-antennae BAS (A2BAS) optimization algorithm, which refines the latent factors obtained by the PSO-incorporated LFA model. The A2BAS algorithm consists of two sub-algorithms. First, we design an improved BAS algorithm which adjusts beetles' antennae and step-size with Adam; Second, we implement the improved BAS algorithm to optimize all the row and column latent factors sequentially. With experimental results on two real high-dimensional matrices, we demonstrate that our algorithm can effectively solve the premature convergence issue.

cs.LG↗

arXiv2022

An Online Sparse Streaming Feature Selection Algorithm

Online streaming feature selection (OSFS), which conducts feature selection in an online manner, plays an important role in dealing with high-dimensional data. In many real applications such as intelligent healthcare platform, streaming feature always has some missing data, which raises a crucial challenge in conducting OSFS, i.e., how to establish the uncertain relationship between sparse streaming features and labels. Unfortunately, existing OSFS algorithms never consider such uncertain relationship. To fill this gap, we in this paper propose an online sparse streaming feature selection with uncertainty (OS2FSU) algorithm. OS2FSU consists of two main parts: 1) latent factor analysis is utilized to pre-estimate the missing data in sparse streaming features before con-ducting feature selection, and 2) fuzzy logic and neighborhood rough set are employed to alleviate the uncertainty between estimated streaming features and labels during conducting feature selection. In the experiments, OS2FSU is compared with five state-of-the-art OSFS algorithms on six real datasets. The results demonstrate that OS2FSU outperforms its competitors when missing data are encountered in OSFS.

cs.LG↗

arXiv2022

Time Series Anomaly Detection via Reinforcement Learning-Based Model Selection

Time series anomaly detection has been recognized as of critical importance for the reliable and efficient operation of real-world systems. Many anomaly detection methods have been developed based on various assumptions on anomaly characteristics. However, due to the complex nature of real-world data, different anomalies within a time series usually have diverse profiles supporting different anomaly assumptions. This makes it difficult to find a single anomaly detector that can consistently outperform other models. In this work, to harness the benefits of different base models, we propose a reinforcement learning-based model selection framework. Specifically, we first learn a pool of different anomaly detection models, and then utilize reinforcement learning to dynamically select a candidate model from these base models. Experiments on real-world data have demonstrated that the proposed strategy can indeed outplay all baseline models in terms of overall performance.

cs.LG↗

arXiv2022

DLME: Deep Local-flatness Manifold Embedding

Manifold learning (ML) aims to seek low-dimensional embedding from high-dimensional data. The problem is challenging on real-world datasets, especially with under-sampling data, and we find that previous methods perform poorly in this case. Generally, ML methods first transform input data into a low-dimensional embedding space to maintain the data's geometric structure and subsequently perform downstream tasks therein. The poor local connectivity of under-sampling data in the former step and inappropriate optimization objectives in the latter step leads to two problems: structural distortion and underconstrained embedding. This paper proposes a novel ML framework named Deep Local-flatness Manifold Embedding (DLME) to solve these problems. The proposed DLME constructs semantic manifolds by data augmentation and overcomes the structural distortion problem using a smoothness constrained based on a local flatness assumption about the manifold. To overcome the underconstrained embedding problem, we design a loss and theoretically demonstrate that it leads to a more suitable embedding based on the local flatness. Experiments on three types of datasets (toy, biological, and image) for various downstream tasks (classification, clustering, and visualization) show that our proposed DLME outperforms state-of-the-art ML and contrastive learning methods.

cs.LG↗