arXiv subjects

Yongqiang Wang

Publications and source records attributed to Yongqiang Wang.

At least 73 records · Page 4Linked to original sources

arXiv2023

Ensuring both Provable Convergence and Differential Privacy in Nash Equilibrium Seeking on Directed Graphs

We study in this paper privacy protection in fully distributed Nash equilibrium seeking where a player can only access its own cost function and receive information from its immediate neighbors over a directed communication network. In view of the non-cooperative nature of the underlying decision-making process, it is imperative to protect the privacy of individual players in networked games when sensitive information is involved. We propose an approach that can achieve both accurate convergence and rigorous differential privacy with finite cumulative privacy budget in distributed Nash equilibrium seeking, which is in sharp contrast to existing differential-privacy solutions for networked games that have to trade convergence accuracy for differential privacy. The approach is applicable even when the communication graph is unbalanced and it does not require individual players to have any global structure information of the communication graph. Since the approach utilizes independent noises for privacy protection, it can combat adversaries having access to all shared messages in the network. It is also encryption-free, ensuring high efficiency in communication and computation. Numerical comparison results with existing counterparts confirm the effectiveness of the proposed approach.

cs.GT↗

arXiv2022

Decentralized Nonconvex Optimization with Guaranteed Privacy and Accuracy

Privacy protection and nonconvexity are two challenging problems in decentralized optimization and learning involving sensitive data. Despite some recent advances addressing each of the two problems separately, no results have been reported that have theoretical guarantees on both privacy protection and saddle/maximum avoidance in decentralized nonconvex optimization. We propose a new algorithm for decentralized nonconvex optimization that can enable both rigorous differential privacy and saddle/maximum avoiding performance. The new algorithm allows the incorporation of persistent additive noise to enable rigorous differential privacy for data samples, gradients, and intermediate optimization variables without losing provable convergence, and thus circumventing the dilemma of trading accuracy for privacy in differential privacy design. More interestingly, the algorithm is theoretically proven to be able to efficiently { guarantee accuracy by avoiding} convergence to local maxima and saddle points, which has not been reported before in the literature on decentralized nonconvex optimization. The algorithm is efficient in both communication (it only shares one variable in each iteration) and computation (it is encryption-free), and hence is promising for large-scale nonconvex optimization and learning involving high-dimensional optimization parameters. Numerical experiments for both a decentralized estimation problem and an Independent Component Analysis (ICA) problem confirm the effectiveness of the proposed approach.

math.OC↗

arXiv2022

Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.

eess.SY↗

arXiv2022

Accelerating RNN-T Training and Inference Using CTC guidance

We propose a novel method to accelerate training and inference process of recurrent neural network transducer (RNN-T) based on the guidance from a co-trained connectionist temporal classification (CTC) model. We made a key assumption that if an encoder embedding frame is classified as a blank frame by the CTC model, it is likely that this frame will be aligned to blank for all the partial alignments or hypotheses in RNN-T and it can be discarded from the decoder input. We also show that this frame reduction operation can be applied in the middle of the encoder, which result in significant speed up for the training and inference in RNN-T. We further show that the CTC alignment, a by-product of the CTC decoder, can also be used to perform lattice reduction for RNN-T during training. Our method is evaluated on the Librispeech and SpeechStew tasks. We demonstrate that the proposed method is able to accelerate the RNN-T inference by 2.2 times with similar or slightly better word error rates (WER).

eess.AS↗

arXiv2022

Gradient-tracking based Distributed Optimization with Guaranteed Optimality under Noisy Information Sharing

Distributed optimization enables networked agents to cooperatively solve a global optimization problem even with each participating agent only having access to a local partial view of the objective function. Despite making significant inroads, most existing results on distributed optimization rely on noise-free information sharing among the agents, which is problematic when communication channels are noisy, messages are coarsely quantized, or shared information are obscured by additive noise for the purpose of achieving differential privacy. The problem of information-sharing noise is particularly pronounced in the state-of-the-art gradient-tracking based distributed optimization algorithms, in that information-sharing noise will accumulate with iterations on the gradient-tracking estimate of these algorithms, and the ensuing variance will even grow unbounded when the noise is persistent. This paper proposes a new gradient-tracking based distributed optimization approach that can avoid information-sharing noise from accumulating in the gradient estimation. The approach is applicable even when the {inter-agent interaction is} time-varying, which is key to enable the incorporation of a decaying factor in inter-agent interaction to gradually eliminate the influence of information-sharing noise. In fact, we rigorously prove that the proposed approach can ensure the almost sure convergence of all agents to the same optimal solution even in the presence of persistent information-sharing noise. The approach is applicable to general directed graphs. It is also capable of ensuring the almost sure convergence of all agents to an optimal solution when the gradients are noisy, which is common in machine learning applications. Numerical simulations confirm the effectiveness of the proposed approach.

math.OC↗

arXiv2022

Quantization enabled Privacy Protection in Decentralized Stochastic Optimization

By enabling multiple agents to cooperatively solve a global optimization problem in the absence of a central coordinator, decentralized stochastic optimization is gaining increasing attention in areas as diverse as machine learning, control, and sensor networks. Since the associated data usually contain sensitive information, such as user locations and personal identities, privacy protection has emerged as a crucial need in the implementation of decentralized stochastic optimization. In this paper, we propose a decentralized stochastic optimization algorithm that is able to guarantee provable convergence accuracy even in the presence of aggressive quantization errors that are proportional to the amplitude of quantization inputs. The result applies to both convex and non-convex objective functions, and enables us to exploit aggressive quantization schemes to obfuscate shared information, and hence enables privacy protection without losing provable optimization accuracy. In fact, by using a {stochastic} ternary quantization scheme, which quantizes any value to three numerical levels, we achieve quantization-based rigorous differential privacy in decentralized stochastic optimization, which has not been reported before. In combination with the presented quantization scheme, the proposed algorithm ensures, for the first time, rigorous differential privacy in decentralized stochastic optimization without losing provable convergence accuracy. Simulation results for a distributed estimation problem as well as numerical experiments for decentralized learning on a benchmark machine learning dataset confirm the effectiveness of the proposed approach.

math.OC↗

arXiv2022

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.

eess.AS↗

arXiv2022

Dynamics based Privacy Preservation in Decentralized Optimization

With decentralized optimization having increased applications in various domains ranging from machine learning, control, sensor networks, to robotics, its privacy is also receiving increased attention. Existing privacy-preserving approaches for decentralized optimization achieve privacy preservation by patching decentralized optimization with information-technology privacy mechanisms such as differential privacy or homomorphic encryption, which either sacrifices optimization accuracy or incurs heavy computation/communication overhead. We propose an inherently privacy-preserving decentralized optimization algorithm by exploiting the robustness of decentralized optimization to uncertainties in optimization dynamics. More specifically, we present a general decentralized optimization framework, based on which we show that privacy can be enabled in decentralized optimization by adding randomness in optimization parameters. We further show that the added randomness has no influence on the accuracy of optimization, and prove that our inherently privacy-preserving algorithm has $R$-linear convergence when the global objective function is smooth and strongly convex. We also rigorously prove that the proposed algorithm can avoid the gradient of a node from being inferable by other nodes. Numerical simulation results confirm the theoretical predictions.

math.OC↗

arXiv2022

Decentralized Stochastic Optimization with Inherent Privacy Protection

Decentralized stochastic optimization is the basic building block of modern collaborative machine learning, distributed estimation and control, and large-scale sensing. Since involved data usually contain sensitive information like user locations, healthcare records and financial transactions, privacy protection has become an increasingly pressing need in the implementation of decentralized stochastic optimization algorithms. In this paper, we propose a decentralized stochastic gradient descent algorithm which is embedded with inherent privacy protection for every participating agent against other participating agents and external eavesdroppers. This proposed algorithm builds in a dynamics based gradient-obfuscation mechanism to enable privacy protection without compromising optimization accuracy, which is in significant difference from differential-privacy based privacy solutions for decentralized optimization that have to trade optimization accuracy for privacy. The dynamics based privacy approach is encryption-free, and hence avoids incurring heavy communication or computation overhead, which is a common problem with encryption based privacy solutions for decentralized stochastic optimization. Besides rigorously characterizing the convergence performance of the proposed decentralized stochastic gradient descent algorithm under both convex objective functions and non-convex objective functions, we also provide rigorous information-theoretic analysis of its strength of privacy protection. Simulation results for a distributed estimation problem as well as numerical experiments for decentralized learning on a benchmark machine learning dataset confirm the effectiveness of the proposed approach.

cs.LG↗

arXiv2022

Unsupervised Data Selection via Discrete Speech Representation for ASR

Self-supervised learning of speech representations has achieved impressive results in improving automatic speech recognition (ASR). In this paper, we show that data selection is important for self-supervised learning. We propose a simple and effective unsupervised data selection method which selects acoustically similar speech to a target domain. It takes the discrete speech representation available in common self-supervised learning frameworks as input, and applies a contrastive data selection method on the discrete tokens. Through extensive empirical studies we show that our proposed method reduces the amount of required pre-training data and improves the downstream ASR performance. Pre-training on a selected subset of 6% of the general data pool results in 11.8% relative improvements in LibriSpeech test-other compared to pre-training on the full set. On Multilingual LibriSpeech French, German, and Spanish test sets, selecting 6% data for pre-training reduces word error rate by more than 15% relatively compared to the full set, and achieves competitive results compared to current state-of-the-art performances.

eess.AS↗

arXiv2022

Algorithm-Level Confidentiality for Average Consensus on Time-Varying Directed Graphs

Average consensus plays a key role in distributed networks, with applications ranging from time synchronization, information fusion, load balancing, to decentralized control. Existing average consensus algorithms require individual agents to exchange explicit state values with their neighbors, which leads to the undesirable disclosure of sensitive information in the state. In this paper, we propose a novel average consensus algorithm for time-varying directed graphs that can protect the confidentiality of a participating agent against other participating agents. The algorithm injects randomness in interaction to obfuscate information on the algorithm-level and can ensure information-theoretic privacy without the assistance of any trusted third party or data aggregator. By leveraging the inherent robustness of consensus dynamics against random variations in interaction, our proposed algorithm can also guarantee the accuracy of average consensus. The algorithm is distinctly different from differential-privacy based average consensus approaches which enable confidentiality through compromising accuracy in obtained consensus value. Numerical simulations confirm the effectiveness and efficiency of our proposed approach.

cs.MA↗

arXiv2021

Effects of Proton Irradiation on Hole Carrier Transport in Hydrogen-Terminated Diamond Surfaces

In this letter, we report the effects of irradiating hydrogen-terminated diamond surfaces with protons. We study the electrical behavior of the two-dimensional hole gas that forms on the diamond surface as a result of hydrogen-termination. Four diamond samples, two of which were passivated with Al2O3 and the other two unpassivated, were irradiated with 2 MeV protons at two fluences, 0.5x10^14 and 1.0x10^14 /cm^2. The hole conductivity degraded at a higher proton fluence, which is attributed to the reduction of the mobility caused by enhanced hole scattering. Both passivated and unpassivated samples exhibited a reduction in the mobility, which can be explained by charging effects of the Al2O3 (for the passivated samples) and degradation of the hydrogen-terminated surface (for the unpassivated samples). To our knowledge, this is the first reported data on 2 MeV proton tolerance of hydrogen-terminated diamond surfaces.

cond-mat.mtrl-sci↗

arXiv2021

Robust Almost Global Splay State Stabilization of Pulse Coupled Oscillators

This technical note deals with the problem of asymptotically stabilizing the splay state configuration of a network of identical pulse coupled oscillators through the design of the their phase response function. The network of pulse coupled oscillators is modeled as a hybrid system. The design of the phase response function is performed to achieve almost global asymptotic stability of a set wherein oscillators' phases are evenly distributed on the unit circle. To establish such a result, a novel Lyapunov function is proposed. Robustness with respect to frequency perturbation is assessed. Finally, the effectiveness of the proposed methodology is shown in an example.

eess.SY↗

arXiv2021

Large topological Hall effect near room temperature in noncollinear ferromagnet LaMn2Ge2 single crystal

Non-trivial spin structures in itinerant magnets can give rise to topological Hall effect (THE) due to the interacting local magnetic moments and conductive electrons. While, in series of materials, THE has mostly been observed at low temperatures far below room temperature (RT) limiting its potential applications. Here, we report the anisotropic anomalous Hall effect (AHE) near RT in LaMn2Ge2, a noncollinear ferromagnetic (FM) with Curie temperature TC=325 K. Large topological Hall resistivity of ~1.0 10-6 ohmcm in broad temperature range (190 K <T< 300 K) is realized as field (H) parallel to the ab-plane (H // ab) and current along c axis (I // c), in contrast to the conventional AHE for H // c and I // ab. The emergence of THE is attributed to the spin chirality of noncoplanar spin configurations stabilized by thermal fluctuation during spin flop process. Moreover, the constructed temperature-field (H-T) phase diagrams based on the isothermal topological Hall resistivity reveal a field-induced transition from the noncoplanar spin configuration to polarized ferromagnetic states. Our experimental realization of large THE near RT highlights LaMn2Ge2 as a promising system for functional applications in novel spintronic devices.

cond-mat.str-el↗

arXiv2020

Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition

This paper proposes an efficient memory transformer Emformer for low latency streaming speech recognition. In Emformer, the long-range history context is distilled into an augmented memory bank to reduce self-attention's computation complexity. A cache mechanism saves the computation for the key and value in self-attention for the left context. Emformer applies a parallelized block processing in training to support low latency models. We carry out experiments on benchmark LibriSpeech data. Under average latency of 960 ms, Emformer gets WER $2.50\%$ on test-clean and $5.62\%$ on test-other. Comparing with a strong baseline augmented memory transformer (AM-TRF), Emformer gets $4.6$ folds training speedup and $18\%$ relative real-time factor (RTF) reduction in decoding with relative WER reduction $17\%$ on test-clean and $9\%$ on test-other. For a low latency scenario with an average latency of 80 ms, Emformer achieves WER $3.01\%$ on test-clean and $7.09\%$ on test-other. Comparing with the LSTM baseline with the same latency and model size, Emformer gets relative WER reduction $9\%$ and $16\%$ on test-clean and test-other, respectively.

cs.SD↗

arXiv2020

Streaming Attention-Based Models with Augmented Memory for End-to-End Speech Recognition

Attention-based models have been gaining popularity recently for their strong performance demonstrated in fields such as machine translation and automatic speech recognition. One major challenge of attention-based models is the need of access to the full sequence and the quadratically growing computational cost concerning the sequence length. These characteristics pose challenges, especially for low-latency scenarios, where the system is often required to be streaming. In this paper, we build a compact and streaming speech recognition system on top of the end-to-end neural transducer architecture with attention-based modules augmented with convolution. The proposed system equips the end-to-end models with the streaming capability and reduces the large footprint from the streaming attention-based model using augmented memory. On the LibriSpeech dataset, our proposed system achieves word error rates 2.7% on test-clean and 5.8% on test-other, to our best knowledge the lowest among streaming approaches reported so far.

cs.CL↗

arXiv2020

Streaming Simultaneous Speech Translation with Augmented Memory Transformer

Transformer-based models have achieved state-of-the-art performance on speech translation tasks. However, the model architecture is not efficient enough for streaming scenarios since self-attention is computed over an entire input sequence and the computational cost grows quadratically with the length of the input sequence. Nevertheless, most of the previous work on simultaneous speech translation, the task of generating translations from partial audio input, ignores the time spent in generating the translation when analyzing the latency. With this assumption, a system may have good latency quality trade-offs but be inapplicable in real-time scenarios. In this paper, we focus on the task of streaming simultaneous speech translation, where the systems are not only capable of translating with partial input but are also able to handle very long or continuous input. We propose an end-to-end transformer-based sequence-to-sequence model, equipped with an augmented memory transformer encoder, which has shown great success on the streaming automatic speech recognition task with hybrid or transducer-based models. We conduct an empirical evaluation of the proposed model on segment, context and memory sizes and we compare our approach to a transformer with a unidirectional mask.

cs.CL↗

arXiv2020

Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications

In this paper, we summarize the application of transformer and its streamable variant, Emformer based acoustic model for large scale speech recognition applications. We compare the transformer based acoustic models with their LSTM counterparts on industrial scale tasks. Specifically, we compare Emformer with latency-controlled BLSTM (LCBLSTM) on medium latency tasks and LSTM on low latency tasks. On a low latency voice assistant task, Emformer gets 24% to 26% relative word error rate reductions (WERRs). For medium latency scenarios, comparing with LCBLSTM with similar model size and latency, Emformer gets significant WERR across four languages in video captioning datasets with 2-3 times inference real-time factors reduction.

cs.CL↗