arXiv subjects

Yu Wu

Publications and source records attributed to Yu Wu.

At least 235 records · Page 13Linked to original sources

arXiv2021

Learning Audio-Visual Correlations from Variational Cross-Modal Generation

People can easily imagine the potential sound while seeing an event. This natural synchronization between audio and visual signals reveals their intrinsic correlations. To this end, we propose to learn the audio-visual correlations from the perspective of cross-modal generation in a self-supervised manner, the learned correlations can be then readily applied in multiple downstream tasks such as the audio-visual cross-modal localization and retrieval. We introduce a novel Variational AutoEncoder (VAE) framework that consists of Multiple encoders and a Shared decoder (MS-VAE) with an additional Wasserstein distance constraint to tackle the problem. Extensive experiments demonstrate that the optimized latent representation of the proposed MS-VAE can effectively learn the audio-visual correlations and can be readily applied in multiple audio-visual downstream tasks to achieve competitive performance even without any given label information during training.

cs.CV↗

arXiv2021

Learning to Anticipate Egocentric Actions by Imagination

Anticipating actions before they are executed is crucial for a wide range of practical applications, including autonomous driving and robotics. In this paper, we study the egocentric action anticipation task, which predicts future action seconds before it is performed for egocentric videos. Previous approaches focus on summarizing the observed content and directly predicting future action based on past observations. We believe it would benefit the action anticipation if we could mine some cues to compensate for the missing information of the unobserved frames. We then propose to decompose the action anticipation into a series of future feature predictions. We imagine how the visual feature changes in the near future and then predicts future action labels based on these imagined representations. Differently, our ImagineRNN is optimized in a contrastive learning way instead of feature regression. We utilize a proxy task to train the ImagineRNN, i.e., selecting the correct future states from distractors. We further improve ImagineRNN by residual anticipation, i.e., changing its target to predicting the feature difference of adjacent frames instead of the frame content. This promotes the network to focus on our target, i.e., the future action, as the difference between adjacent frame features is more important for forecasting the future. Extensive experiments on two large-scale egocentric action datasets validate the effectiveness of our method. Our method significantly outperforms previous methods on both the seen test set and the unseen test set of the EPIC Kitchens Action Anticipation Challenge.

cs.CV↗

arXiv2020

Average Doppler Shift of Gamma-ray Spectra of Positron Annihilation Process in Molecules

This paper studies the gamma-ray spectra of positron annihilation processes in a series of molecules. The results show that the average valence electron energy of the molecules has a linear correlation with the full width at half maximum (FWHM) of the gamma-ray spectra. In addition, we defined a new physical quantity Average Doppler Shift (ADS), which can be used as the eigenvalue to describe the characteristics of the gamma-ray spectra. Since ADS contains all the information about the gamma-ray spectra, it can more accurately represent the characteristics of the gamma-ray spectra. For a series of molecules, this paper compares the ADS and FWHM of their gamma-ray spectra and the average valence electron energy. The results show that ADS has a linear correlation with the average valence electron energy and the FWHM. Further, this proves that the annihilation mainly occurs on valence electrons, and it also illustrates that the ADS has certain applicability. It is expected that this will provide us with a deeper understanding of the positron annihilation process.

physics.atm-clus↗

arXiv2020

Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020

This paper describes the Microsoft speaker diarization system for monaural multi-talker recordings in the wild, evaluated at the diarization track of the VoxCeleb Speaker Recognition Challenge(VoxSRC) 2020. We will first explain our system design to address issues in handling real multi-talker recordings. We then present the details of the components, which include Res2Net-based speaker embedding extractor, conformer-based continuous speech separation with leakage filtering, and a modified DOVER (short for Diarization Output Voting Error Reduction) method for system fusion. We evaluate the systems with the data set provided by VoxSRCchallenge 2020, which contains real-life multi-talker audio collected from YouTube. Our best system achieves 3.71% and 6.23% of the diarization error rate (DER) on development set and evaluation set, respectively, being ranked the 1st at the diarization track of the challenge.

eess.AS↗

arXiv2020

Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformer

With its strong modeling capacity that comes from a multi-head and multi-layer structure, Transformer is a very powerful model for learning a sequential representation and has been successfully applied to speech separation recently. However, multi-channel speech separation sometimes does not necessarily need such a heavy structure for all time frames especially when the cross-talker challenge happens only occasionally. For example, in conversation scenarios, most regions contain only a single active speaker, where the separation task downgrades to a single speaker enhancement problem. It turns out that using a very deep network structure for dealing with signals with a low overlap ratio not only negatively affects the inference efficiency but also hurts the separation performance. To deal with this problem, we propose an early exit mechanism, which enables the Transformer model to handle different cases with adaptive depth. Experimental results indicate that not only does the early exit mechanism accelerate the inference, but it also improves the accuracy.

cs.SD↗

arXiv2020

Continuous Speech Separation with Conformer

Continuous speech separation plays a vital role in complicated speech related tasks such as conversation transcription. The separation model extracts a single speaker signal from a mixed speech. In this paper, we use transformer and conformer in lieu of recurrent neural networks in the separation system, as we believe capturing global information with the self-attention based method is crucial for the speech separation. Evaluating on the LibriCSS dataset, the conformer separation model achieves state of the art results, with a relative 23.5% word error rate (WER) reduction from bi-directional LSTM (BLSTM) in the utterance-wise evaluation and a 15.4% WER reduction in the continuous evaluation.

eess.AS↗

arXiv2020

Towards Accurate Predictions of Carrier Mobilities and Thermoelectric Performances in 2D Materials

The interactions between electrons and lattice vibrational modes play the key role in determining the carrier transport properties, thermoelectric performance and other physical quantities related to phonons in semiconductors. However, for two-dimensional (2D) materials, the widely-used models for carrier transport only consider the interactions between electrons and some specific phonon modes, which usually leads to inaccruate predictions of electrons/phonons transport properties. In this work, comprehensive investigations on full electron-phonon couplings and their influences on carrier mobility and thermoelectric performances of 2D group-IV and V elemental monolayers were performed, and we also analyzed in details the selection rules on electron-phonon couplings using group-theory arguments. Our calculations revealed that, for the cases of shallow dopings where only intravalley scatterings are allowed, the contributions from optical phonon modes are significantly larger than those from acoustic phonon modes in group-IV elemental monolayers, and LA and some specific optical phonon modes contribute significantly to the total intravalley scatterings. When the doping increases and intervalley scatterings are allowed, the intervalley scatterings are much stronger than intravalley scatterings, and ZA/TA/LO phonon modes dominate the intervalley scatterings in monolayer Si, Ge and Sn. The dominant contributions to the total intervalley scatterings are ZA/TO in monolayer P, ZA/TO in monolayer As and TO/LO in monolayer Sb. Based on the thorough investigations on the full electron-phonon couplings, we predict accurately the carrier mobilities and thermoelectric figure of merits in these two elemental crystals, and reveal significant reductions when compared with the calculations based on the widely-used simplified model.

cond-mat.mtrl-sci↗

arXiv2020

Effects of intervalley scatterings in thermoelectric performance of band-convergent antimonene

The strategy of band convergence of multi-valley conduction bands or multi-peak valence bands has been widely used to search or improve thermoelectric materials. However, the phonon-assisted intervalley scatterings due to multiple band degeneracy are usually neglected in the thermoelectric community. In this work, we investigate the (thermo)electric properties of non-polar monolayer $β$- and $α$-antimonene considering full mode- and momentum-resolved electron-phonon interactions. We also analyze thoroughly the selection rules on electron-phonon matrix-elements using group-theory arguments. Our calculations reveal strong intervalley scattering between the nearly degenerate valley states in both $β$- and $α$-antimonene, and the commonly-used deformation potential approximation neglecting the dominant intervalley scattering gives inaccurate estimations of the electron-phonon scattering and thermoelectric transport properties. By considering full electron-phonon interactions based on the rigid-band approximation, we find that, the maximum value of the thermoelectric figure of merits $zT$ at room temperature reduces to 0.37 in $β$-antimonene, by a factor of 5.7 comparing to the value predicted based on the constant relaxation-time approximation method. Our work not only provides an accurate prediction of the thermoelectric performances of antimonenes that reveals the key role of intervalley scatterings in determining the electronic part of zT, but also showcases a computational framework for thermoelectric materials.

cond-mat.mtrl-sci↗

arXiv2020

Revisiting EmbodiedQA: A Simple Baseline and Beyond

In Embodied Question Answering (EmbodiedQA), an agent interacts with an environment to gather necessary information for answering user questions. Existing works have laid a solid foundation towards solving this interesting problem. But the current performance, especially in navigation, suggests that EmbodiedQA might be too challenging for the contemporary approaches. In this paper, we empirically study this problem and introduce 1) a simple yet effective baseline that achieves promising performance; 2) an easier and practical setting for EmbodiedQA where an agent has a chance to adapt the trained model to a new environment before it actually answers users questions. In this new setting, we randomly place a few objects in new environments, and upgrade the agent policy by a distillation network to retain the generalization ability from the trained model. On the EmbodiedQA v1 benchmark, under the standard setting, our simple baseline achieves very competitive results to the-state-of-the-art; in the new setting, we found the introduced small change in settings yields a notable gain in navigation.

cs.CV↗

arXiv2020

Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents

With the arising concerns for the AI systems provided with direct access to abundant sensitive information, researchers seek to develop more reliable AI with implicit information sources. To this end, in this paper, we introduce a new task called video description via two multi-modal cooperative dialog agents, whose ultimate goal is for one conversational agent to describe an unseen video based on the dialog and two static frames. Specifically, one of the intelligent agents - Q-BOT - is given two static frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has already seen the entire video, assists Q-BOT to accomplish the goal by providing answers to those questions. We propose a QA-Cooperative Network with a dynamic dialog history update learning mechanism to transfer knowledge from A-BOT to Q-BOT, thus helping Q-BOT to better describe the video. Extensive experiments demonstrate that Q-BOT can effectively learn to describe an unseen video by the proposed model and the cooperative learning method, achieving the promising performance where Q-BOT is given the full ground truth history dialog.

cs.CV↗

arXiv2020

On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition

Recently, there has been a strong push to transition from hybrid models to end-to-end (E2E) models for automatic speech recognition. Currently, there are three promising E2E methods: recurrent neural network transducer (RNN-T), RNN attention-based encoder-decoder (AED), and Transformer-AED. In this study, we conduct an empirical comparison of RNN-T, RNN-AED, and Transformer-AED models, in both non-streaming and streaming modes. We use 65 thousand hours of Microsoft anonymized training data to train these models. As E2E models are more data hungry, it is better to compare their effectiveness with large amount of training data. To the best of our knowledge, no such comprehensive study has been conducted yet. We show that although AED models are stronger than RNN-T in the non-streaming mode, RNN-T is very competitive in streaming mode if its encoder can be properly initialized. Among all three E2E models, transformer-AED achieved the best accuracy in both streaming and non-streaming mode. We show that both streaming RNN-T and transformer-AED models can obtain better accuracy than a highly-optimized hybrid model.

eess.AS↗

arXiv2020

Abnormal activity capture from passenger flow of elevator based on unsupervised learning and fine-grained multi-label recognition

We present a work-flow which aims at capturing residents' abnormal activities through the passenger flow of elevator in multi-storey residence buildings. Camera and sensors (hall sensor, photoelectric sensor, gyro, accelerometer, barometer, and thermometer) with internet connection are mounted in elevator to collect image and data. Computer vision algorithms such as instance segmentation, multi-label recognition, embedding and clustering are applied to generalize passenger flow of elevator, i.e. how many people and what kinds of people get in and out of the elevator on each floor. More specifically in our implementation we propose GraftNet, a solution for fine-grained multi-label recognition task, to recognize human attributes, e.g. gender, age, appearance, and occupation. Then anomaly detection of unsupervised learning is hierarchically applied on the passenger flow data to capture abnormal or even illegal activities of the residents which probably bring safety hazard, e.g. drug dealing, pyramid sale gathering, prostitution, and over crowded residence. Experiment shows effects are there, and the captured records will be directly reported to our customer(property managers) for further confirmation.

cs.CV↗

arXiv2020

Low Latency End-to-End Streaming Speech Recognition with a Scout Network

The attention-based Transformer model has achieved promising results for speech recognition (SR) in the offline mode. However, in the streaming mode, the Transformer model usually incurs significant latency to maintain its recognition accuracy when applying a fixed-length look-ahead window in each encoder layer. In this paper, we propose a novel low-latency streaming approach for Transformer models, which consists of a scout network and a recognition network. The scout network detects the whole word boundary without seeing any future frames, while the recognition network predicts the next subword by utilizing the information from all the frames before the predicted boundary. Our model achieves the best performance (2.7/6.4 WER) with only 639 ms latency on the test-clean and test-other data sets of Librispeech.

eess.AS↗

arXiv2020

Curriculum Pre-training for End-to-End Speech Translation

End-to-end speech translation poses a heavy burden on the encoder, because it has to transcribe, understand, and learn cross-lingual semantics simultaneously. To obtain a powerful encoder, traditional methods pre-train it on ASR data to capture speech features. However, we argue that pre-training the encoder only through simple speech recognition is not enough and high-level linguistic knowledge should be considered. Inspired by this, we propose a curriculum pre-training method that includes an elementary course for transcription learning and two advanced courses for understanding the utterance and mapping words in two languages. The difficulty of these courses is gradually increasing. Experiments show that our curriculum pre-training method leads to significant improvements on En-De and En-Fr speech translation benchmarks.

cs.CL↗

arXiv2020

MuTual: A Dataset for Multi-Turn Dialogue Reasoning

Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. To facilitate the conversation reasoning research, we introduce MuTual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that can handle various reasoning problems. Empirical results show that state-of-the-art methods only reach 71%, which is far behind the human performance of 94%, indicating that there is ample room for improving reasoning ability. MuTual is available at https://github.com/Nealcly/MuTual.

cs.CL↗

arXiv2020

Unsupervised Person Re-identification via Softened Similarity Learning

Person re-identification (re-ID) is an important topic in computer vision. This paper studies the unsupervised setting of re-ID, which does not require any labeled information and thus is freely deployed to new scenarios. There are very few studies under this setting, and one of the best approach till now used iterative clustering and classification, so that unlabeled images are clustered into pseudo classes for a classifier to get trained, and the updated features are used for clustering and so on. This approach suffers two problems, namely, the difficulty of determining the number of clusters, and the hard quantization loss in clustering. In this paper, we follow the iterative training mechanism but discard clustering, since it incurs loss from hard quantization, yet its only product, image-level similarity, can be easily replaced by pairwise computation and a softened classification task. With these improvements, our approach becomes more elegant and is more robust to hyper-parameter changes. Experiments on two image-based and video-based datasets demonstrate state-of-the-art performance under the unsupervised re-ID setting.

cs.CV↗

arXiv2020

Gated Channel Transformation for Visual Recognition

In this work, we propose a generally applicable transformation unit for visual recognition with deep convolutional neural networks. This transformation explicitly models channel relationships with explainable control variables. These variables determine the neuron behaviors of competition or cooperation, and they are jointly optimized with the convolutional weight towards more accurate recognition. In Squeeze-and-Excitation (SE) Networks, the channel relationships are implicitly learned by fully connected layers, and the SE block is integrated at the block-level. We instead introduce a channel normalization layer to reduce the number of parameters and computational complexity. This lightweight layer incorporates a simple l2 normalization, enabling our transformation unit applicable to operator-level without much increase of additional parameters. Extensive experiments demonstrate the effectiveness of our unit with clear margins on many vision tasks, i.e., image classification on ImageNet, object detection and instance segmentation on COCO, video classification on Kinetics.

cs.CV↗

arXiv2020

An alternative indicator of annihilated electrons in atoms: Rahm's electronegativity scale

This paper presents a new explanation of the width of gamma-ray spectra based on Rahm's electronegativity scale. This quantitatively proves, for the first time, that positrophilic electrons in the positron-electron annihilation process are exactly the valence electrons. This suggests the replacement of Full Width at Half Maximum (FWHM) of the gamma-ray spectra with the newly defined physical quantity Average Doppler Shift (ADS). Both FWHM and ADS of the gamma-ray spectra in light elements agree well with the corresponding Rahm's electronegativity values, respectively. Further, ADS provides strong evidence in favor of Rahm's electronegativity scale. It is expected that this will be useful in understanding the mechanism of the positron-electron annihilation process.

physics.atm-clus↗