arXiv subjects

Shinji Watanabe

Publications and source records attributed to Shinji Watanabe.

At least 451 records · Page 25Linked to original sources

arXiv2020

Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition

Recently very deep transformers have outperformed conventional bi-directional long short-term memory networks by a large margin in speech recognition. However, to put it into production usage, inference computation cost is still a serious concern in real scenarios. In this paper, we study two different non-autoregressive transformer structure for automatic speech recognition (ASR): A-CMLM and A-FMLM. During training, for both frameworks, input tokens fed to the decoder are randomly replaced by special mask tokens. The network is required to predict the tokens corresponding to those mask tokens by taking both unmasked context and input speech into consideration. During inference, we start from all mask tokens and the network iteratively predicts missing tokens based on partial results. We show that this framework can support different decoding strategies, including traditional left-to-right. A new decoding strategy is proposed as an example, which starts from the easiest predictions to the most difficult ones. Results on Mandarin (Aishell) and Japanese (CSJ) ASR benchmarks show the possibility to train such a non-autoregressive network for ASR. Especially in Aishell, the proposed method outperformed the Kaldi ASR system and it matches the performance of the state-of-the-art autoregressive transformer with 7x speedup. Pretrained models and code will be made available after publication.

eess.AS↗

arXiv2020

End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification

The most common approach to speaker diarization is clustering of speaker embeddings. However, the clustering-based approach has a number of problems; i.e., (i) it is not optimized to minimize diarization errors directly, (ii) it cannot handle speaker overlaps correctly, and (iii) it has trouble adapting their speaker embedding models to real audio recordings with speaker overlaps. To solve these problems, we propose the End-to-End Neural Diarization (EEND), in which a neural network directly outputs speaker diarization results given a multi-speaker recording. To realize such an end-to-end model, we formulate the speaker diarization problem as a multi-label classification problem and introduce a permutation-free objective function to directly minimize diarization errors. Besides its end-to-end simplicity, the EEND method can explicitly handle speaker overlaps during training and inference. Just by feeding multi-speaker recordings with corresponding speaker segment labels, our model can be easily adapted to real conversations. We evaluated our method on simulated speech mixtures and real conversation datasets. The results showed that the EEND method outperformed the state-of-the-art x-vector clustering-based method, while it correctly handled speaker overlaps. We explored the neural network architecture for the EEND method, and found that the self-attention-based neural network was the key to achieving excellent performance. In contrast to conditioning the network only on its previous and next hidden states, as is done using bidirectional long short-term memory (BLSTM), self-attention is directly conditioned on all the frames. By visualizing the attention weights, we show that self-attention captures global speaker characteristics in addition to local speech activity dynamics, making it especially suitable for dealing with the speaker diarization problem.

eess.AS↗

arXiv2020

Attention-based ASR with Lightweight and Dynamic Convolutions

End-to-end (E2E) automatic speech recognition (ASR) with sequence-to-sequence models has gained attention because of its simple model training compared with conventional hidden Markov model based ASR. Recently, several studies report the state-of-the-art E2E ASR results obtained by Transformer. Compared to recurrent neural network (RNN) based E2E models, training of Transformer is more efficient and also achieves better performance on various tasks. However, self-attention used in Transformer requires computation quadratic in its input length. In this paper, we propose to apply lightweight and dynamic convolution to E2E ASR as an alternative architecture to the self-attention to make the computational order linear. We also propose joint training with connectionist temporal classification, convolution on the frequency axis, and combination with self-attention. With these techniques, the proposed architectures achieve better performance than RNN-based E2E model and performance competitive to state-of-the-art Transformer on various ASR benchmarks including noisy/reverberant tasks.

eess.AS↗

arXiv2020

ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

This paper introduces a new end-to-end text-to-speech (E2E-TTS) toolkit named ESPnet-TTS, which is an extension of the open-source speech processing toolkit ESPnet. The toolkit supports state-of-the-art E2E-TTS models, including Tacotron~2, Transformer TTS, and FastSpeech, and also provides recipes inspired by the Kaldi automatic speech recognition (ASR) toolkit. The recipes are based on the design unified with the ESPnet ASR recipe, providing high reproducibility. The toolkit also provides pre-trained models and samples of all of the recipes so that users can use it as a baseline. Furthermore, the unified design enables the integration of ASR functions with TTS, e.g., ASR-based objective evaluation and semi-supervised learning with both ASR and TTS models. This paper describes the design of the toolkit and experimental evaluation in comparison with other toolkits. The experimental results show that our models can achieve state-of-the-art performance comparable to the other latest toolkits, resulting in a mean opinion score (MOS) of 4.25 on the LJSpeech dataset. The toolkit is publicly available at https://github.com/espnet/espnet.

cs.CL↗

arXiv2020

End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection

This paper integrates a voice activity detection (VAD) function with end-to-end automatic speech recognition toward an online speech interface and transcribing very long audio recordings. We focus on connectionist temporal classification (CTC) and its extension of CTC/attention architectures. As opposed to an attention-based architecture, input-synchronous label prediction can be performed based on a greedy search with the CTC (pre-)softmax output. This prediction includes consecutive long blank labels, which can be regarded as a non-speech region. We use the labels as a cue for detecting speech segments with simple thresholding. The threshold value is directly related to the length of a non-speech region, which is more intuitive and easier to control than conventional VAD hyperparameters. Experimental results on unsegmented data show that the proposed method outperformed the baseline methods using the conventional energy-based and neural-network-based VAD methods and achieved an RTF less than 0.2. The proposed method is publicly available.

eess.AS↗

arXiv2020

Speaker Diarization with Region Proposal Network

Speaker diarization is an important pre-processing step for many speech applications, and it aims to solve the "who spoke when" problem. Although the standard diarization systems can achieve satisfactory results in various scenarios, they are composed of several independently-optimized modules and cannot deal with the overlapped speech. In this paper, we propose a novel speaker diarization method: Region Proposal Network based Speaker Diarization (RPNSD). In this method, a neural network generates overlapped speech segment proposals, and compute their speaker embeddings at the same time. Compared with standard diarization systems, RPNSD has a shorter pipeline and can handle the overlapped speech. Experimental results on three diarization datasets reveal that RPNSD achieves remarkable improvements over the state-of-the-art x-vector baseline.

eess.AS↗

arXiv2020

End-to-End Multi-speaker Speech Recognition with Transformer

Recently, fully recurrent neural network (RNN) based end-to-end models have been proven to be effective for multi-speaker speech recognition in both the single-channel and multi-channel scenarios. In this work, we explore the use of Transformer models for these tasks by focusing on two aspects. First, we replace the RNN-based encoder-decoder in the speech recognition model with a Transformer architecture. Second, in order to use the Transformer in the masking network of the neural beamformer in the multi-channel case, we modify the self-attention component to be restricted to a segment rather than the whole sequence in order to reduce computation. Besides the model architecture improvements, we also incorporate an external dereverberation preprocessing, the weighted prediction error (WPE), enabling our model to handle reverberated signals. Experiments on the spatialized wsj1-2mix corpus show that the Transformer-based models achieve 40.9% and 25.6% relative WER reduction, down to 12.1% and 6.4% WER, under the anechoic condition in single-channel and multi-channel tasks, respectively, while in the reverberant case, our methods achieve 41.5% and 13.8% relative WER reduction, down to 16.5% and 15.2% WER.

eess.AS↗

arXiv2019

High-speed XYZ-nanopositioner for scanning ion conductance microscopy

We describe a tip-scan-type high-speed XYZ-nanopositioner designed for scanning ion conductance microscopy (SICM), with a special care being devoted to the way of nanopipette holding. The nanopipette probe is mounted in the center of a hollow piezoactuator, both ends of which are attached to identical diaphragm flexures, for Z-positioning. This design minimizes the generation of undesirable mechanical vibrations. Mechanical amplification is used to increase the XY-travel range of the nanopositioner. The first resonance frequencies of the nanopositioner are measured as $\sim$100 kHz and $\sim$2.3 kHz for the Z- and XY-displacements, respectively. The travel ranges are $\sim$6 $μ$m and $\sim$34 $μ$m for Z and XY, respectively. When this nanopositioner is used for hopping mode imaging of SICM with a $\sim$10-nm radius tip, the vertical tip velocity can be increased to 400 nm/ms; hence, the one-pixel acquisition time can be minimized to $\sim$1 ms.

physics.ins-det↗

arXiv2019

Development of High-Speed Ion Conductance Microscopy

Scanning ion conductance microscopy (SICM) can image the surface topography of specimens in ionic solutions without mechanical probe--sample contact. This unique capability is advantageous for imaging fragile biological samples but its highest possible imaging rate is far lower than the level desired in biological studies. Here, we present the development of high-speed SICM. The fast imaging capability is attained by a fast Z-scanner with active vibration control and pipette probes with enhanced ion conductance. By the former, the delay of probe Z-positioning is minimized to sub-10 us, while its maximum stroke is secured at 6 um. The enhanced ion conductance lowers a noise floor in ion current detection, increasing the detection bandwidth up to 100 kHz. Thus, temporal resolution 100-fold higher than that of conventional systems is achieved, together with spatial resolution around 20 nm.

physics.app-ph↗

arXiv2019

Non-Divergent Grüneisen Parameter in Quantum Critical Quasicrystal Yb$_{15}$Al$_{34}$Au$_{51}$: Reflection of Robustness of Quantum Criticality under Pressure

The mechanism of not diverging Grüneisen parameter in the quantum critical heavy-fermion quasicrystal (QC) Yb$_{15}$Al$_{34}$Au$_{51}$ is analyzed. We construct the formalism for calculating the specific heat $C_V(T)$, the thermal-expansion coefficient $α(T)$, and the Grüneisen parameter $Γ(T)$ near the quantum critical point of the Yb valence transition. By applying the framework to the QC, we calculate $C_V(T)$, $α(T)$, and $Γ(T)$, which explains the measurements. Not diverging $Γ(T)$ is attributed to the robustness of the quantum criticality in the QC under pressure. The difference in $Γ(T)$ at the lowest temperature between the QC and approximant crystal is shown to reflect the difference in the volume derivative of characteristic energy scales of the critical Yb-valence fluctuation and the Kondo temperature. Possible implications of our theory to future experiments are also discussed.

cond-mat.str-el↗

arXiv2019

Multilingual End-to-End Speech Translation

In this paper, we propose a simple yet effective framework for multilingual end-to-end speech translation (ST), in which speech utterances in source languages are directly translated to the desired target languages with a universal sequence-to-sequence architecture. While multilingual models have shown to be useful for automatic speech recognition (ASR) and machine translation (MT), this is the first time they are applied to the end-to-end ST problem. We show the effectiveness of multilingual end-to-end ST in two scenarios: one-to-many and many-to-many translations with publicly available data. We experimentally confirm that multilingual end-to-end ST models significantly outperform bilingual ones in both scenarios. The generalization of multilingual training is also evaluated in a transfer learning scenario to a very low-resource language pair. All of our codes and the database are publicly available to encourage further research in this emergent multilingual ST topic.

cs.CL↗

arXiv2019

Charge Transfer Effect under Odd-Parity Crystalline Electric Field

Charge-transfer effect under odd-parity crystalline electric field (CEF) is analyzed theoretically. In quantum-critical metal $β$-YbAlB$_4$, seven-fold configuration of B atoms surrounding Yb atom breaks local inversion symmetry at the Yb site, giving rise to the odd-parity CEF. Analysis of the CEF on the basis of hybridization picture shows that admixture of 4f and 5d wave functions at Yb with pure imaginary coefficient occurs, which makes magnetic-toroidal (MT) and electric-dipole (ED) degrees of freedom active. By constructing the minimal model for periodic crystal $β$-YbAlB$_4$, we show that the MT as well as ED fluctuation is divergently enhanced at the quantum critical point of valence transition simultaneously with critical valence fluctuations.

cond-mat.str-el↗

arXiv2019

Towards Online End-to-end Transformer Automatic Speech Recognition

The Transformer self-attention network has recently shown promising performance as an alternative to recurrent neural networks in end-to-end (E2E) automatic speech recognition (ASR) systems. However, Transformer has a drawback in that the entire input sequence is required to compute self-attention. We have proposed a block processing method for the Transformer encoder by introducing a context-aware inheritance mechanism. An additional context embedding vector handed over from the previously processed block helps to encode not only local acoustic information but also global linguistic, channel, and speaker attributes. In this paper, we extend it towards an entire online E2E ASR system by introducing an online decoding process inspired by monotonic chunkwise attention (MoChA) into the Transformer decoder. Our novel MoChA training and inference algorithms exploit the unique properties of Transformer, whose attentions are not always monotonic or peaky, and have multiple heads and residual connections of the decoder layers. Evaluations of the Wall Street Journal (WSJ) and AISHELL-1 show that our proposed online Transformer decoder outperforms conventional chunkwise approaches.

eess.AS↗

arXiv2019

Grüneisen Parameter and Thermal Expansion near Magnetic Quantum Critical Points in Itinerant Electron Systems

Complete expressions of the thermal-expansion coefficient $α$ and the Grüneisen parameter $Γ$ are derived on the basis of the self-consistent renormalization (SCR) theory. By considering zero-point as well as thermal spin fluctuation under the stationary condition, the specific heat for each class of the magnetic quantum critical point (QCP) specified by the dynamical exponent $z=3$ (FM) and $z=2$ (AFM) and the spatial dimension ($d=3$ and $2$) is shown to be expressed as $C_{V}=C_a-C_b$, where $C_a$ is dominant at low temperatures, reproducing the past SCR criticality endorsed by the renormalization group theory. Starting from the explicit form of the entropy and using the Maxwell relation, $α=α_a+α_b$ (with $α_a$ and $α_b$ being related to $C_a$ and $C_b$, respectively) is derived, which is proven to be equivalent to $α$ derived from the free energy. The temperature-dependent coefficient found to exist in $α_b$, which is dominant at low temperatures, contributes to the crossover from the quantum-critical regime to the Curie-Weiss regime and even affects the quantum criticality at 2d AFM QCP. Based on these correctly calculated $C_{V}$ and $α$, Grüneisen parameter $Γ=Γ_a+Γ_b$ is derived, where $Γ_a$ and $Γ_b$ contain $α_a$ and $α_b$, respectively. The inverse susceptibility coupled to the volume $V$ in $Γ_b$ gives rise to divergence of $Γ$ at the QCP for each class even though characteristic energy scale of spin fluctuation $T_0$ is finite at the QCP, which gives a finite contribution in $Γ_a=-\frac{V}{T_0}\left(\frac{\partial T_0}{\partial V}\right)_{T=0}$. General properties of $α$ and $Γ$ including their signs as well as the relation to $T_0$ and the Kondo temperature in temperature-pressure phase diagrams of Ce- and Yb-based heavy electron systems are discussed.

cond-mat.str-el↗

arXiv2019

A practical two-stage training strategy for multi-stream end-to-end speech recognition

The multi-stream paradigm of audio processing, in which several sources are simultaneously considered, has been an active research area for information fusion. Our previous study offered a promising direction within end-to-end automatic speech recognition, where parallel encoders aim to capture diverse information followed by a stream-level fusion based on attention mechanisms to combine the different views. However, with an increasing number of streams resulting in an increasing number of encoders, the previous approach could require substantial memory and massive amounts of parallel data for joint training. In this work, we propose a practical two-stage training scheme. Stage-1 is to train a Universal Feature Extractor (UFE), where encoder outputs are produced from a single-stream model trained with all data. Stage-2 formulates a multi-stream scheme intending to solely train the attention fusion module using the UFE features and pretrained components from Stage-1. Experiments have been conducted on two datasets, DIRHA and AMI, as a multi-stream scenario. Compared with our previous method, this strategy achieves relative word error rate reductions of 8.2--32.4%, while consistently outperforming several conventional combination methods.

cs.CL↗

arXiv2019

Quantum Valence Criticality in Heavy Fermions on Periodic and Aperiodic Crystals

Progress of theories and experiments of the quantum valence criticality is overviewed focusing on recent discoveries of direct evidence of quantum critical point of valence transitions in periodic crystal $α$-YbAl$_{1-x}$Fe$_x$B$_4$ at $x=0.014$ and quasicrystal Yb$_{15}$(Al$_{1-x} $Ga$_{x}$)$_{34}$(Au$_{1-y}$Cu$_{y}$)$_{51}$ at $x=y=0$. The common quantum criticality as well as $T/B$ scaling behavior in the magnetization observed in the periodic crystal $β$-YbAlB$_4$ and quasicrystal Yb$_{15}$Al$_{34}$Au$_{51}$ is shown to be explained from the theory of critical Yb-valence fluctuations starting from the Hamiltonians for describing the low-energy electronic states in both systems.

cond-mat.str-el↗

arXiv2019

Grüneisen Parameter and Thermal Expansion by the Self-Consistent Renormalization Theory of Spin Fluctuations

The thermal expansion coefficient $α$ and the Grüneisen parameter $Γ$ near the magnetic quantum critical point (QCP) are derived on the basis of the self-consistent renormalization (SCR) theory of spin fluctuation. From the SCR entropy, the specific heat $C_{V}$, $α$, and $Γ$ are shown to be expressed in a simple form as $C_{V}=C_{\rm a}-C_{\rm b}$, $α=α_{\rm a}+α_{\rm b}$, and $Γ=Γ_{\rm a}+Γ_{\rm b}$, respectively, where $C_{\rm i}$, $α_{\rm i}$, and $Γ_{\rm i}$ $({\rm i}={\rm a}, {\rm b})$ are related with each other. As the temperature $T$ decreases, $C_{\rm a}$, $α_{\rm b}$, and $Γ_{\rm b}$ become dominant in $C_{V}$, $α$, and $Γ$, respectively. The inverse susceptibility of spin fluctuation coupled to the volume $V$ in $Γ_{\rm b}$ is found to give rise to the divergence of $Γ$ at the QCP for each class of ferromagnetism and antiferromagnetism (AFM) in spatial dimensions $d=3$ and $2$. This $V$-dependent inverse susceptibility in $α_{b}$ and $Γ_{\rm b}$ contributes to the $T$ dependences of $α$ and $Γ$, and even affects their criticality in the case of the AFM QCP in $d=2$. $Γ_{\rm a}$ is expressed as $Γ_{\rm a}(T=0)=-\frac{V}{T_0}\left(\frac{\partial T_0}{\partial V}\right)_{T=0}$ with $T_0$ being the characteristic temperature of spin fluctuation, which has an enhanced value in heavy electron systems.

cond-mat.str-el↗

arXiv2019

Multi-Stream End-to-End Speech Recognition

Attention-based methods and Connectionist Temporal Classification (CTC) network have been promising research directions for end-to-end (E2E) Automatic Speech Recognition (ASR). The joint CTC/Attention model has achieved great success by utilizing both architectures during multi-task training and joint decoding. In this work, we present a multi-stream framework based on joint CTC/Attention E2E ASR with parallel streams represented by separate encoders aiming to capture diverse information. On top of the regular attention networks, the Hierarchical Attention Network (HAN) is introduced to steer the decoder toward the most informative encoders. A separate CTC network is assigned to each stream to force monotonic alignments. Two representative framework have been proposed and discussed, which are Multi-Encoder Multi-Resolution (MEM-Res) framework and Multi-Encoder Multi-Array (MEM-Array) framework, respectively. In MEM-Res framework, two heterogeneous encoders with different architectures, temporal resolutions and separate CTC networks work in parallel to extract complimentary information from same acoustics. Experiments are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative Word Error Rate (WER) reduction of 18.0-32.1% and the best WER of 3.6% in the WSJ eval92 test set. The MEM-Array framework aims at improving the far-field ASR robustness using multiple microphone arrays which are activated by separate encoders. Compared with the best single-array results, the proposed framework has achieved relative WER reduction of 3.7% and 9.7% in AMI and DIRHA multi-array corpora, respectively, which also outperforms conventional fusion strategies.

eess.AS↗