arXiv subjects

Hai Li

Publications and source records attributed to Hai Li.

At least 127 records · Page 7Linked to original sources

arXiv2021

Gaussian Random Number Generator with Reconfigurable Mean and Variance using Stochastic Magnetic Tunnel Junctions

Generating high-quality random numbers with a Gaussian probability distribution function is an important and resource consuming computational task for many applications in the fields of machine learning and Monte Carlo algorithms. Recently, CMOS-based digital hardware architectures have been explored as specialized Gaussian random number generators (GRNGs). These CMOS-based GRNGs have a large area and require entropy sources at their input which increase the computing cost. Here, we propose a GRNG that works on the principle of the Boltzmann law in a physical system made from an interconnected network of thermally unstable magnetic tunnel junctions. The proposed hardware can produce multi-bit Gaussian random numbers at a gigahertz speed and can be configured to generate distributions with a desired mean and variance. An analytical derivation of the required interconnection and bias strengths is provided followed by numerical simulations to demonstrate the functionalities of the GRNG.

cs.ET↗

arXiv2021

FL-WBC: Enhancing Robustness against Model Poisoning Attacks in Federated Learning from a Client Perspective

Federated learning (FL) is a popular distributed learning framework that trains a global model through iterative communications between a central server and edge devices. Recent works have demonstrated that FL is vulnerable to model poisoning attacks. Several server-based defense approaches (e.g. robust aggregation), have been proposed to mitigate such attacks. However, we empirically show that under extremely strong attacks, these defensive methods fail to guarantee the robustness of FL. More importantly, we observe that as long as the global model is polluted, the impact of attacks on the global model will remain in subsequent rounds even if there are no subsequent attacks. In this work, we propose a client-based defense, named White Blood Cell for Federated Learning (FL-WBC), which can mitigate model poisoning attacks that have already polluted the global model. The key idea of FL-WBC is to identify the parameter space where long-lasting attack effect on parameters resides and perturb that space during local training. Furthermore, we derive a certified robustness guarantee against model poisoning attacks and a convergence guarantee to FedAvg after applying our FL-WBC. We conduct experiments on FasionMNIST and CIFAR10 to evaluate the defense against state-of-the-art model poisoning attacks. The results demonstrate that our method can effectively mitigate model poisoning attack impact on the global model within 5 communication rounds with nearly no accuracy drop under both IID and Non-IID settings. Our defense is also complementary to existing server-based robust aggregation approaches and can further improve the robustness of FL under extremely strong attacks.

cs.LG↗

arXiv2021

A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness

Neural networks are getting better accuracy with higher energy and computational cost. After quantization, the cost can be greatly saved, and the quantized models are more hardware friendly with acceptable accuracy loss. On the other hand, recent research has found that neural networks are vulnerable to adversarial attacks, and the robustness of a neural network model can only be improved with defense methods, such as adversarial training. In this work, we find that adversarially-trained neural networks are more vulnerable to quantization loss than plain models. To minimize both the adversarial and the quantization losses simultaneously and to make the quantized model robust, we propose a layer-wise adversarial-aware quantization method, using the Lipschitz constant to choose the best quantization parameter settings for a neural network. We theoretically derive the losses and prove the consistency of our metric selection. The experiment results show that our method can effectively and efficiently improve the robustness of quantized adversarially-trained neural networks.

cs.LG↗

arXiv2021

Physics-Based Models for Magneto-Electric Spin-Orbit Logic Circuits

Spintronic devices are a promising beyond-CMOS device option thanks to their energy efficiency and compatibility with CMOS. To accurately capture their multi-physics dynamics, a rigorous treatment of both spin and charge and their inter-conversion is required. Here we present physics-based device models based on 4x4 matrices for the spin-orbit coupling part of the magneto-electric spin-orbit (MESO) device. Also, a more rigorous physics model of ferroelectric and magnetoelectric switching of ferromagnets, based on Landau-Lifshitz-Gilbert (LLG) and Landau-Khalatnikov (LK) equations, is presented. With the combined model implemented in a SPICE circuit simulator environment, simulation results were obtained which show feasibility of MESO implementation and functional operation of buffers, oscillators, and majority gates.

cond-mat.mes-hall↗

arXiv2021

Multi-Objective Optimization of ReRAM Crossbars for Robust DNN Inferencing under Stochastic Noise

Resistive random-access memory (ReRAM) is a promising technology for designing hardware accelerators for deep neural network (DNN) inferencing. However, stochastic noise in ReRAM crossbars can degrade the DNN inferencing accuracy. We propose the design and optimization of a high-performance, area-and energy-efficient ReRAM-based hardware accelerator to achieve robust DNN inferencing in the presence of stochastic noise. We make two key technical contributions. First, we propose a stochastic-noise-aware training method, referred to as ReSNA, to improve the accuracy of DNN inferencing on ReRAM crossbars with stochastic noise. Second, we propose an information-theoretic algorithm, referred to as CF-MESMO, to identify the Pareto set of solutions to trade-off multiple objectives, including inferencing accuracy, area overhead, execution time, and energy consumption. The main challenge in this context is that executing the ReSNA method to evaluate each candidate ReRAM design is prohibitive. To address this challenge, we utilize the continuous-fidelity evaluation of ReRAM designs associated with prohibitive high computation cost by varying the number of training epochs to trade-off accuracy and cost. CF-MESMO iteratively selects the candidate ReRAM design and fidelity pair that maximizes the information gained per unit computation cost about the optimal Pareto front. Our experiments on benchmark DNNs show that the proposed algorithms efficiently uncover high-quality Pareto fronts. On average, ReSNA achieves 2.57% inferencing accuracy improvement for ResNet20 on the CIFAR-10 dataset with respect to the baseline configuration. Moreover, CF-MESMO algorithm achieves 90.91% reduction in computation cost compared to the popular multi-objective optimization algorithm NSGA-II to reach the best solution from NSGA-II.

cs.ET↗

arXiv2021

Spin-charge interconversion in KTaO$_3$ two-dimensional electron gases

Oxide interfaces exhibit a broad range of physical effects stemming from broken inversion symmetry. In particular, they can display non-reciprocal phenomena when time reversal symmetry is also broken, e.g., by the application of a magnetic field. Examples include the direct and inverse Edelstein effects (DEE, IEE) that allow the interconversion between spin currents and charge currents. The DEE and IEE have been investigated in interfaces based on the perovskite SrTiO$_3$ (STO), albeit in separate studies focusing on one or the other. The demonstration of these effects remains mostly elusive in other oxide interface systems despite their blossoming in the last decade. Here, we report the observation of both the DEE and IEE in a new interfacial two-dimensional electron gas (2DEG) based on the perovskite oxide KTaO$_3$. We generate 2DEGs by the simple deposition of Al metal onto KTaO$_3$ single crystals, characterize them by angle-resolved photoemission spectroscopy and magnetotransport, and demonstrate the DEE through unidirectional magnetoresistance and the IEE by spin-pumping experiments. We compare the spin-charge interconversion efficiency with that of STO-based interfaces, relate it to the 2DEG electronic structure, and give perspectives for the implementation of KTaO$_3$ 2DEGs into spin-orbitronic devices.

cond-mat.mes-hall↗

arXiv2021

Privacy-Preserving Representation Learning on Graphs: A Mutual Information Perspective

Learning with graphs has attracted significant attention recently. Existing representation learning methods on graphs have achieved state-of-the-art performance on various graph-related tasks such as node classification, link prediction, etc. However, we observe that these methods could leak serious private information. For instance, one can accurately infer the links (or node identity) in a graph from a node classifier (or link predictor) trained on the learnt node representations by existing methods. To address the issue, we propose a privacy-preserving representation learning framework on graphs from the \emph{mutual information} perspective. Specifically, our framework includes a primary learning task and a privacy protection task, and we consider node classification and link prediction as the two tasks of interest. Our goal is to learn node representations such that they can be used to achieve high performance for the primary learning task, while obtaining performance for the privacy protection task close to random guessing. We formally formulate our goal via mutual information objectives. However, it is intractable to compute mutual information in practice. Then, we derive tractable variational bounds for the mutual information terms, where each bound can be parameterized via a neural network. Next, we train these parameterized neural networks to approximate the true mutual information and learn privacy-preserving node representations. We finally evaluate our framework on various graph datasets.

cs.LG↗

arXiv2021

Enriching Source Style Transfer in Recognition-Synthesis based Non-Parallel Voice Conversion

Current voice conversion (VC) methods can successfully convert timbre of the audio. As modeling source audio's prosody effectively is a challenging task, there are still limitations of transferring source style to the converted speech. This study proposes a source style transfer method based on recognition-synthesis framework. Previously in speech generation task, prosody can be modeled explicitly with prosodic features or implicitly with a latent prosody extractor. In this paper, taking advantages of both, we model the prosody in a hybrid manner, which effectively combines explicit and implicit methods in a proposed prosody module. Specifically, prosodic features are used to explicit model prosody, while VAE and reference encoder are used to implicitly model prosody, which take Mel spectrum and bottleneck feature as input respectively. Furthermore, adversarial training is introduced to remove speaker-related information from the VAE outputs, avoiding leaking source speaker information while transferring style. Finally, we use a modified self-attention based encoder to extract sentential context from bottleneck features, which also implicitly aggregates the prosodic aspects of source speech from the layered representations. Experiments show that our approach is superior to the baseline and a competitive system in terms of style transfer; meanwhile, the speech quality and speaker similarity are well maintained.

eess.AS↗

arXiv2021

Neuromorphic Algorithm-hardware Codesign for Temporal Pattern Learning

Neuromorphic computing and spiking neural networks (SNN) mimic the behavior of biological systems and have drawn interest for their potential to perform cognitive tasks with high energy efficiency. However, some factors such as temporal dynamics and spike timings prove critical for information processing but are often ignored by existing works, limiting the performance and applications of neuromorphic computing. On one hand, due to the lack of effective SNN training algorithms, it is difficult to utilize the temporal neural dynamics. Many existing algorithms still treat neuron activation statistically. On the other hand, utilizing temporal neural dynamics also poses challenges to hardware design. Synapses exhibit temporal dynamics, serving as memory units that hold historical information, but are often simplified as a connection with weight. Most current models integrate synaptic activations in some storage medium to represent membrane potential and institute a hard reset of membrane potential after the neuron emits a spike. This is done for its simplicity in hardware, requiring only a "clear" signal to wipe the storage medium, but destroys temporal information stored in the neuron. In this work, we derive an efficient training algorithm for Leaky Integrate and Fire neurons, which is capable of training a SNN to learn complex spatial temporal patterns. We achieved competitive accuracy on two complex datasets. We also demonstrate the advantage of our model by a novel temporal pattern association task. Codesigned with this algorithm, we have developed a CMOS circuit implementation for a memristor-based network of neuron and synapses which retains critical neural dynamics with reduced complexity. This circuit implementation of the neuron model is simulated to demonstrate its ability to react to temporal spiking patterns with an adaptive threshold.

cs.AR↗

arXiv2021

The Multi-speaker Multi-style Voice Cloning Challenge 2021

The Multi-speaker Multi-style Voice Cloning Challenge (M2VoC) aims to provide a common sizable dataset as well as a fair testbed for the benchmarking of the popular voice cloning task. Specifically, we formulate the challenge to adapt an average TTS model to the stylistic target voice with limited data from target speaker, evaluated by speaker identity and style similarity. The challenge consists of two tracks, namely few-shot track and one-shot track, where the participants are required to clone multiple target voices with 100 and 5 samples respectively. There are also two sub-tracks in each track. For sub-track a, to fairly compare different strategies, the participants are allowed to use only the training data provided by the organizer strictly. For sub-track b, the participants are allowed to use any data publicly available. In this paper, we present a detailed explanation on the tasks and data used in the challenge, followed by a summary of submitted systems and evaluation results.

eess.AS↗

arXiv2021

The Untapped Potential of Off-the-Shelf Convolutional Neural Networks

Over recent years, a myriad of novel convolutional network architectures have been developed to advance state-of-the-art performance on challenging recognition tasks. As computational resources improve, a great deal of effort has been placed in efficiently scaling up existing designs and generating new architectures with Neural Architecture Search (NAS) algorithms. While network topology has proven to be a critical factor for model performance, we show that significant gains are being left on the table by keeping topology static at inference-time. Due to challenges such as scale variation, we should not expect static models configured to perform well across a training dataset to be optimally configured to handle all test data. In this work, we seek to expose the exciting potential of inference-time-dynamic models. By allowing just four layers to dynamically change configuration at inference-time, we show that existing off-the-shelf models like ResNet-50 are capable of over 95% accuracy on ImageNet. This level of performance currently exceeds that of models with over 20x more parameters and significantly more complex training procedures.

cs.CV↗

arXiv2021

Can Targeted Adversarial Examples Transfer When the Source and Target Models Have No Label Space Overlap?

We design blackbox transfer-based targeted adversarial attacks for an environment where the attacker's source model and the target blackbox model may have disjoint label spaces and training datasets. This scenario significantly differs from the "standard" blackbox setting, and warrants a unique approach to the attacking process. Our methodology begins with the construction of a class correspondence matrix between the whitebox and blackbox label sets. During the online phase of the attack, we then leverage representations of highly related proxy classes from the whitebox distribution to fool the blackbox model into predicting the desired target class. Our attacks are evaluated in three complex and challenging test environments where the source and target models have varying degrees of conceptual overlap amongst their unique categories. Ultimately, we find that it is indeed possible to construct targeted transfer-based adversarial attacks between models that have non-overlapping label spaces! We also analyze the sensitivity of attack success to properties of the clean data. Finally, we show that our transfer attacks serve as powerful adversarial priors when integrated with query-based methods, markedly boosting query efficiency and adversarial success.

cs.LG↗

arXiv2021

A Case for 3D Integrated System Design for Neuromorphic Computing & AI Applications

Over the last decade, artificial intelligence has found many applications areas in the society. As AI solutions have become more sophistication and the use cases grew, they highlighted the need to address performance and energy efficiency challenges faced during the implementation process. To address these challenges, there has been growing interest in neuromorphic chips. Neuromorphic computing relies on non von Neumann architectures as well as novel devices, circuits and manufacturing technologies to mimic the human brain. Among such technologies, 3D integration is an important enabler for AI hardware and the continuation of the scaling laws. In this paper, we overview the unique opportunities 3D integration provides in neuromorphic chip design, discuss the emerging opportunities in next generation neuromorphic architectures and review the obstacles. Neuromorphic architectures, which relied on the brain for inspiration and emulation purposes, face grand challenges due to the limited understanding of the functionality and the architecture of the human brain. Yet, high-levels of investments are dedicated to develop neuromorphic chips. We argue that 3D integration not only provides strategic advantages to the cost-effective and flexible design of neuromorphic chips, it may provide design flexibility in incorporating advanced capabilities to further benefits the designs in the future.

cs.ET↗

arXiv2021

BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization

Mixed-precision quantization can potentially achieve the optimal tradeoff between performance and compression rate of deep neural networks, and thus, have been widely investigated. However, it lacks a systematic method to determine the exact quantization scheme. Previous methods either examine only a small manually-designed search space or utilize a cumbersome neural architecture search to explore the vast search space. These approaches cannot lead to an optimal quantization scheme efficiently. This work proposes bit-level sparsity quantization (BSQ) to tackle the mixed-precision quantization from a new angle of inducing bit-level sparsity. We consider each bit of quantized weights as an independent trainable variable and introduce a differentiable bit-sparsity regularizer. BSQ can induce all-zero bits across a group of weight elements and realize the dynamic precision reduction, leading to a mixed-precision quantization scheme of the original model. Our method enables the exploration of the full mixed-precision space with a single gradient-based optimization process, with only one hyperparameter to tradeoff the performance and compression. BSQ achieves both higher accuracy and higher bit reduction on various model architectures on the CIFAR-10 and ImageNet datasets comparing to previous methods.

cs.LG↗

arXiv2021

Improving Adversarial Robustness in Weight-quantized Neural Networks

Neural networks are getting deeper and more computation-intensive nowadays. Quantization is a useful technique in deploying neural networks on hardware platforms and saving computation costs with negligible performance loss. However, recent research reveals that neural network models, no matter full-precision or quantized, are vulnerable to adversarial attacks. In this work, we analyze both adversarial and quantization losses and then introduce criteria to evaluate them. We propose a boundary-based retraining method to mitigate adversarial and quantization losses together and adopt a nonlinear mapping method to defend against white-box gradient-based adversarial attacks. The evaluations demonstrate that our method can better restore accuracy after quantization than other baseline methods on both black-box and white-box adversarial attacks. The results also show that adversarial training suffers quantization loss and does not cooperate well with other training methods.

cs.LG↗

arXiv2021

On Provable Backdoor Defense in Collaborative Learning

As collaborative learning allows joint training of a model using multiple sources of data, the security problem has been a central concern. Malicious users can upload poisoned data to prevent the model's convergence or inject hidden backdoors. The so-called backdoor attacks are especially difficult to detect since the model behaves normally on standard test data but gives wrong outputs when triggered by certain backdoor keys. Although Byzantine-tolerant training algorithms provide convergence guarantee, provable defense against backdoor attacks remains largely unsolved. Methods based on randomized smoothing can only correct a small number of corrupted pixels or labels; methods based on subset aggregation cause a severe drop in classification accuracy due to low data utilization. We propose a novel framework that generalizes existing subset aggregation methods. The framework shows that the subset selection process, a deciding factor for subset aggregation methods, can be viewed as a code design problem. We derive the theoretical bound of data utilization ratio and provide optimal code construction. Experiments on non-IID versions of MNIST and CIFAR-10 show that our method with optimal codes significantly outperforms baselines using non-overlapping partition and random selection. Additionally, integration with existing coding theory results shows that special codes can track the location of the attackers. Such capability provides new countermeasures to backdoor attacks.

cs.CR↗

arXiv2021

Anisotropic Andreev reflection in semi-Dirac materials

In the framework of Bogoliubov-de Gennes equation, we theoretically study the Andreev reflection in normal-superconducting junctions based on semi-Dirac materials. Owing to the intrinsic anisotropy of semi-Dirac materials, the configuration of Andreev reflection and differential conductance are strongly orientation-dependent. For the transport along the linear dispersion direction, the differential conductance exhibits a clear crossover from retro Andreev reflection to specular Andreev reflection with an increasing bias-voltage, and the differential conductance oscillates without a decaying profile when the interfacial barrier strength increases. However, for the transport along the quadratic dispersion direction, the boundary between the retro Andreev reflection and specular Andreev reflection is ambiguous, and the differential conductance decays with increasing the momentum mismatch or the interfacial barrier strength. We illustrate the pseudo-spin textures to reveal the underling physics behind the anisotropic coherent transport properties. These results enrich the understanding of the superconducting coherent transport in semi-Dirac materials.

cond-mat.mes-hall↗

arXiv2020

Effect of inter-system coupling on heat transport in a microscopic collision model

In this paper we consider a bipartite system composed of two subsystems each coupled to its own thermal environment. Based on a collision model, we mainly study whether the approximation (i.e., the inter-system interaction is ignored when modeling the system-environment coupling) is valid or not. We also address the problem of heat transport unitedly for both conventional energy-preserving system-environment interactions and non-energy preserving system-environment interactions. For the former interaction, as the inter-system interaction strength increases, at first this approximation gets worse as expected, but then counterintuitively gets better even for a stronger inter-system coupling. For the latter interaction with asymmetry, this approximation gets progressively worse. In this case we realize a perfect thermal rectification, and we can not find apparent rectification effect for the former interaction. Finally and more importantly, our results show that whether this approximation is valid or not is closely related to the quantum correlations between the subsystems, i.e., the weaker the quantum correlations, the more justified the approximation and vice versa.

quant-ph↗