arXiv subjects

Tao Luo

Publications and source records attributed to Tao Luo.

At least 127 records · Page 7Linked to original sources

arXiv2023

Desire Backpropagation: A Lightweight Training Algorithm for Multi-Layer Spiking Neural Networks based on Spike-Timing-Dependent Plasticity

Spiking neural networks (SNNs) are a viable alternative to conventional artificial neural networks when resource efficiency and computational complexity are of importance. A major advantage of SNNs is their binary information transfer through spike trains which eliminates multiplication operations. The training of SNNs has, however, been a challenge, since neuron models are non-differentiable and traditional gradient-based backpropagation algorithms cannot be applied directly. Furthermore, spike-timing-dependent plasticity (STDP), albeit being a spike-based learning rule, updates weights locally and does not optimize for the output error of the network. We present desire backpropagation, a method to derive the desired spike activity of all neurons, including the hidden ones, from the output error. By incorporating this desire value into the local STDP weight update, we can efficiently capture the neuron dynamics while minimizing the global error and attaining a high classification accuracy. That makes desire backpropagation a spike-based supervised learning rule. We trained three-layer networks to classify MNIST and Fashion-MNIST images and reached an accuracy of 98.41% and 87.56%, respectively. In addition, by eliminating a multiplication during the backward pass, we reduce computational complexity and balance arithmetic resources between forward and backward pass, making desire backpropagation a candidate for training on low-resource devices.

cs.NE↗

arXiv2023

On the Splash Singularity for the free-boundary problem of the viscous and non-resistive incompressible magnetohydrodynamic equations in 3D

In this paper, the existence of finite-time splash singularity is proved for the free-boundary problem of the viscous and non-resistive incompressible magnetohydrodynamic (MHD) equations in $ \mathbb{R}^{3}$, based on a construction of a sequence of initial data alongside delicate estimates of the solutions. The result and analysis in this paper generalize those by Coutand and Shkoller in [14, Ann. Inst. H. Poincaré C Anal. Non Linéaire, 2019] from the viscous surface waves to the viscous conducting fluids with magnetic effects for which non-trivial magnetic fields may present on the free boundary. The arguments in this paper also hold for any space dimension $d\ge 2$.

math.AP↗

arXiv2023

Limitation of Characterizing Implicit Regularization by Data-independent Functions

In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself not completely defined and well understood. In this work, we attempt to mathematically define and study implicit regularization. Importantly, we explore the limitations of a common approach to characterizing implicit regularization using data-independent functions. We propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping mechanisms, based on which we provide two recipes for producing classes of one-hidden-neuron NNs that provably cannot be fully characterized by a type of or all data-independent functions. Following the previous works, our results further emphasize the profound data dependency of implicit regularization in general, inspiring us to study in detail the data dependency of NN implicit regularization in the future.

cs.LG↗

arXiv2023

Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance

Winograd is generally utilized to optimize convolution performance and computational efficiency because of the reduced multiplication operations, but the reliability issues brought by winograd are usually overlooked. In this work, we observe the great potential of winograd convolution in improving neural network (NN) fault tolerance. Based on the observation, we evaluate winograd convolution fault tolerance comprehensively from different granularities ranging from models, layers, and operation types for the first time. Then, we explore the use of inherent fault tolerance of winograd convolution for cost-effective NN protection against soft errors. Specifically, we mainly investigate how winograd convolution can be effectively incorporated with classical fault-tolerant design approaches including triple modular redundancy (TMR), fault-aware retraining, and constrained activation functions. According to our experiments, winograd convolution can reduce the fault-tolerant design overhead by 55.77\% on average without any accuracy loss compared to standard convolution, and further reduce the computing overhead by 17.24\% when the inherent fault tolerance of winograd convolution is considered. When it is applied on fault-tolerant neural networks enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in presence of various faults.

cs.LG↗

arXiv2023

Optimistic Estimate Uncovers the Potential of Nonlinear Models

We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models. It yields an optimistic sample size that quantifies the smallest possible sample size to fit/recover a target function using a nonlinear model. We estimate the optimistic sample sizes for matrix factorization models, deep models, and deep neural networks (DNNs) with fully-connected or convolutional architecture. For each nonlinear model, our estimates predict a specific subset of targets that can be fitted at overparameterization, which are confirmed by our experiments. Our optimistic estimate reveals two special properties of the DNN models -- free expressiveness in width and costly expressiveness in connection. These properties suggest the following architecture design principles of DNNs: (i) feel free to add neurons/kernels; (ii) restrain from connecting neurons. Overall, our optimistic estimate theoretically unveils the vast potential of nonlinear models in fitting at overparameterization. Based on this framework, we anticipate gaining a deeper understanding of how and why numerous nonlinear models such as DNNs can effectively realize their potential in practice in the near future.

cs.LG↗

arXiv2023

Phenomenological study of $J/ψ\toΞ^0(Λπ^0) \barΞ^0(\barΛ γ)$ decays

The measurement of decay parameters is one of the important goals of particle physics experiments, and the measurement serves as a probe to search for evidence of CP violation in baryonic decays. The experimental results will help advance existing theoretical research and establish new experimental objectives. In this paper, we formulate the asymmetric parameters that characterize parity violation, and then derive formulas for the measurement of CP violation. The formulae for the joint angular distribution of the full decay chain as well as the polarization observable of $Ξ^ 0 $, $\bar { Ξ} ^ 0 $, $Λ$ and $\bar { Λ} $ are also provided for experiments. Lastly, we evaluated the sensitivity of two asymmetric parameters: $α_ { Ξ^ 0 \to Λπ^ 0 } $ (abbreviated as $α_ { Ξ^ 0 } $) and $α_ { \bar { Ξ} ^ 0 \to \bar { Λ} γ} $ (abbreviated as $α_ {\barΞ^0 } $) for future experimental measurement.

hep-ph↗

arXiv2023

A Class of Smoothing Modulus-Based Iterative Method for Solving Implicit Complementarity Problems

In this paper, a class of smoothing modulus-based iterative method was presented for solving implicit complementarity problems. The main idea was to transform the implicit complementarity problem into an equivalent implicit fixed-point equation, then introduces a smoothing function to obtain its approximation solutions. The convergence analysis of the algorithm was given, and the efficiency of the algorithms was verified by numerical experiment

math.NA↗

arXiv2023

Stochastic Modified Equations and Dynamics of Dropout Algorithm

Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and its impact on achieving good generalization abilities remain poorly understood. In this work, we derive the stochastic modified equations for analyzing the dynamics of dropout, where its discrete iteration process is approximated by a class of stochastic differential equations. In order to investigate the underlying mechanism by which dropout facilitates the identification of flatter minima, we study the noise structure of the derived stochastic modified equation for dropout. By drawing upon the structural resemblance between the Hessian and covariance through several intuitive approximations, we empirically demonstrate the universal presence of the inverse variance-flatness relation and the Hessian-variance relation, throughout the training process of dropout. These theoretical and empirical findings make a substantial contribution to our understanding of the inherent tendency of dropout to locate flatter minima.

cs.LG↗

arXiv2023

DeepFire2: A Convolutional Spiking Neural Network Accelerator on FPGAs

Brain-inspired spiking neural networks (SNNs) replace the multiply-accumulate operations of traditional neural networks by integrate-and-fire neurons, with the goal of achieving greater energy efficiency. Specialized hardware implementations of those neurons clearly have advantages over general-purpose devices in terms of power and performance, but exhibit poor scalability when it comes to accelerating large neural networks. DeepFire2 introduces a hardware architecture which can map large network layers efficiently across multiple super logic regions in a multi-die FPGA. That gives more control over resource allocation and parallelism, benefiting both throughput and energy consumption. Avoiding the use of lookup tables to implement the AND operations of an SNN, prevents the layer size to be limited by logic resources. A deep pipeline does not only lead to an increased clock speed of up to 600 MHz. We double the throughput and power efficiency compared to our previous version of DeepFire, which equates to an almost 10-fold improvement over other previous implementations. Importantly, we are able to deploy a large ImageNet model, while maintaining a throughput of over 1500 frames per second.

cs.NE↗

arXiv2023

Hierarchical Weight Averaging for Deep Neural Networks

Despite the simplicity, stochastic gradient descent (SGD)-like algorithms are successful in training deep neural networks (DNNs). Among various attempts to improve SGD, weight averaging (WA), which averages the weights of multiple models, has recently received much attention in the literature. Broadly, WA falls into two categories: 1) online WA, which averages the weights of multiple models trained in parallel, is designed for reducing the gradient communication overhead of parallel mini-batch SGD, and 2) offline WA, which averages the weights of one model at different checkpoints, is typically used to improve the generalization ability of DNNs. Though online and offline WA are similar in form, they are seldom associated with each other. Besides, these methods typically perform either offline parameter averaging or online parameter averaging, but not both. In this work, we firstly attempt to incorporate online and offline WA into a general training framework termed Hierarchical Weight Averaging (HWA). By leveraging both the online and offline averaging manners, HWA is able to achieve both faster convergence speed and superior generalization performance without any fancy learning rate adjustment. Besides, we also analyze the issues faced by existing WA methods, and how our HWA address them, empirically. Finally, extensive experiments verify that HWA outperforms the state-of-the-art methods significantly.

cs.LG↗

arXiv2023

Phase Diagram of Initial Condensation for Two-layer Neural Networks

The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the earlier work by Luo et al.~\cite{luo2021phase}, we present a phase diagram of initial condensation for two-layer neural networks. Condensation is a phenomenon wherein the weight vectors of neural networks concentrate on isolated orientations during the training process, and it is a feature in non-linear learning process that enables neural networks to possess better generalization abilities. Our phase diagram serves to provide a comprehensive understanding of the dynamical regimes of neural networks and their dependence on the choice of hyperparameters related to initialization. Furthermore, we demonstrate in detail the underlying mechanisms by which small initialization leads to condensation at the initial training stage.

cs.LG↗

arXiv2023

URGLQ: An Efficient Covariance Matrix Reconstruction Method for Robust Adaptive Beamforming

The computational complexity of the conventional adaptive beamformer is relatively large, and the performance degrades significantly due to the model mismatch errors and the unwanted signals in received data. In this paper, an efficient unwanted signal removal and Gauss-Legendre quadrature (URGLQ)-based covariance matrix reconstruction method is proposed. Different from the prior covariance matrix reconstruction methods, a projection matrix is constructed to remove the unwanted signal from the received data, which improves the reconstruction accuracy of the covariance matrix. Considering that the computational complexity of most matrix reconstruction algorithms is relatively large due to the integral operation, we proposed a Gauss-Legendre quadrature-based method to approximate the integral operation while maintaining accuracy. Moreover, to improve the robustness of the beamformer, the mismatch in the desired steering vector is corrected by maximizing the output power of the beamformer under a constraint that the corrected steering vector cannot converge to any interference steering vector. Simulation results and prototype experiments demonstrate that the performance of the proposed beamformer outperforms the compared methods and is much closer to the optimal beamformer in different scenarios.

eess.SP↗

arXiv2023

Insight-HXMT and GECAM-C observations of the brightest-of-all-time GRB 221009A

GRB 221009A is the brightest gamma-ray burst ever detected since the discovery of this kind of energetic explosions. However, an accurate measurement of the prompt emission properties of this burst is very challenging due to its exceptional brightness. With joint observations of \textit{Insight}-HXMT and GECAM-C, we made an unprecedentedly accurate measurement of the emission during the first $\sim$1800 s of GRB 221009A, including its precursor, main emission (ME, which dominates the burst in flux), flaring emission and early afterglow, in the hard X-ray to soft gamma-ray band from $\sim$ 10 keV to $\sim$ 6 MeV. Based on the GECAM-C unsaturated data of the ME, we measure a record-breaking isotropic equivalent energy ($E_{\rm iso}$) of $\bf \sim 1.5 \times 10^{55}$ erg, which is about eight times the total rest-mass energy of the Sun. The early afterglow data require a significant jet break between 650 s and 1100 s, most likely at $\sim950$ s from the afterglow starting time $T_{AG}$, which corresponds to a jet opening angle of $\sim {0.7^\circ} \ (η_γn)^{1/8}$, where $n$ is the ambient medium density in units of $\rm cm^{-3}$ and $η_γ$ is the ratio between $γ$-ray energy and afterglow kinetic energy. The beaming-corrected total $γ$-ray energy $E_γ$ is $\sim 1.15 \times10^{51} \ (η_γn)^{1/4}$ erg, which is typical for long GRBs. These results suggest that this GRB may have a special central engine, which could launch and collimate a very narrowly beamed jet with an ordinary energy budget, leading to exceptionally luminous gamma-ray radiation per unit solid angle. Alternatively, more GRBs might have such a narrow and bright beam, which are missed by an unfavorable viewing angle or have been detected without distance measurement.

astro-ph.HE↗

arXiv2023

In-orbit Performance of ME onboard Insight-HXMT in the first 5 years

Introduction: The Medium Energy X-ray telescope (ME) is a collimated X-ray telescope onboard the Insight hard X-ray modulation telescope (Insight-HXMT) satellite. It has 1728 Si-PIN pixels readout using 54 low noise application-specific integrated circuits (ASICs). ME covers the energy range of 5-30 keV and has a total detection area of 952 cm2. The typical energy resolution of ME at the beginning of the mission is 3 keV at 17.8 keV (Full Width at Half Maximum, FWHM) and the time resolution is 255 us. In this study, we present the in-orbit performance of ME in its first 5 years of operation. Methods: The performance of ME was monitored using onboard radioactive sources and astronomical X-ray objects. ME carries six 241Am radioactive sources for onboard calibration, which can continuously illuminate the calibration pixels. The long-term performance evolution of ME can be quantified using the properties of the accumulated spectra of the calibration pixels. In addition, observations of the Crab Nebula and the pulsar were used to check the long-term evolution of the detection efficiency as a function of energy. Conclusion: After 5 years of operation, 742 cm2 of the Si-PIN pixels were still working normally. The peak positions of 241Am emission lines gradually shifted to the high energy region, implying a slow increase in ME gain of 1.43%. A comparison of the ME spectra of the Crab Nebula and the pulsar shows that the E-C relations and the redistribution matrix file are still acceptable for most data analysis works, and there is no detectable variation in the detection efficiency.

astro-ph.IM↗

arXiv2023

Optimization of Image Transmission in a Cooperative Semantic Communication Networks

In this paper, a semantic communication framework for image transmission is developed. In the investigated framework, a set of servers cooperatively transmit images to a set of users utilizing semantic communication techniques. To evaluate the performance of studied semantic communication system, a multimodal metric is proposed to measure the correlation between the extracted semantic information and the original image. To meet the ISS requirement of each user, each server must jointly determine the semantic information to be transmitted and the resource blocks (RBs) used for semantic information transmission. We formulate this problem as an optimization problem aiming to minimize each server's transmission latency while reaching the ISS requirement. To solve this problem, a value decomposition based entropy-maximized multi-agent reinforcement learning (RL) is proposed, which enables servers to coordinate for training and execute RB allocation in a distributed manner to approach to a globally optimal performance with less training iterations. Compared to traditional multi-agent RL, the proposed RL improves the valuable action exploration of servers and the probability of finding a globally optimal RB allocation policy based on local observation. Simulation results show that the proposed algorithm can reduce the transmission delay by up to 16.1% compared to traditional multi-agent RL.

cs.AI↗

arXiv2022

Study of parity violation in $Λ_c^+\toϕp$ and $Λ_c^+ \to ωp$ decays

Evidence for CP violation in baryonic decays has not been seen in experiments. With large data events accumulated at $e^+e^-$ collider or Large Hadron Collider, the charmed baryon decays would provide a promising laboratory to test the CP symmetry. In this work, we formulate the $Λ_c^+\toϕp$ and $ Λ_c^+\toωp$ decays for the measurements of their asymmetry parameters in weak decays at the BESIII or the LHCb experiments. The polarization transfer is analyzed in the two processes of $e^+ e^- \toΛ_c^+\barΛ_c^-$ and $Λ_b^0\to Λ^+_cπ^-$, and the formulas of the joint angular distribution for these processes are provided. The sensitivity of the measurement of asymmetry parameters for the above two decay processes is estimated for the future experiments.

hep-ph↗

arXiv2022

Linear Stability Hypothesis and Rank Stratification for Nonlinear Models

Models with nonlinear architectures/parameterizations such as deep neural networks (DNNs) are well known for their mysteriously good generalization performance at overparameterization. In this work, we tackle this mystery from a novel perspective focusing on the transition of the target recovery/fitting accuracy as a function of the training data size. We propose a rank stratification for general nonlinear models to uncover a model rank as an "effective size of parameters" for each function in the function space of the corresponding model. Moreover, we establish a linear stability theory proving that a target function almost surely becomes linearly stable when the training data size equals its model rank. Supported by our experiments, we propose a linear stability hypothesis that linearly stable functions are preferred by nonlinear training. By these results, model rank of a target function predicts a minimal training data size for its successful recovery. Specifically for the matrix factorization model and DNNs of fully-connected or convolutional architectures, our rank stratification shows that the model rank for specific target functions can be much lower than the size of model parameters. This result predicts the target recovery capability even at heavy overparameterization for these nonlinear models as demonstrated quantitatively by our experiments. Overall, our work provides a unified framework with quantitative prediction power to understand the mysterious target recovery behavior at overparameterization for general nonlinear models.

cs.LG↗

arXiv2022

An Experimental Comparison Between Temporal Difference and Residual Gradient with Neural Network Approximation

Gradient descent or its variants are popular in training neural networks. However, in deep Q-learning with neural network approximation, a type of reinforcement learning, gradient descent (also known as Residual Gradient (RG)) is barely used to solve Bellman residual minimization problem. On the contrary, Temporal Difference (TD), an incomplete gradient descent method prevails. In this work, we perform extensive experiments to show that TD outperforms RG, that is, when the training leads to a small Bellman residual error, the solution found by TD has a better policy and is more robust against the perturbation of neural network parameters. We further use experiments to reveal a key difference between reinforcement learning and supervised learning, that is, a small Bellman residual error can correspond to a bad policy in reinforcement learning while the test loss function in supervised learning is a standard index to indicate the performance. We also empirically examine that the missing term in TD is a key reason why RG performs badly. Our work shows that the performance of a deep Q-learning solution is closely related to the training dynamics and how an incomplete gradient descent method can find a good policy is interesting for future study.

cs.LG↗