arXiv subjects

Yibo Yang

Publications and source records attributed to Yibo Yang.

113 records · Page 7Linked to original sources

arXiv2018

Optimization Algorithm Inspired Deep Neural Network Structure Design

Deep neural networks have been one of the dominant machine learning approaches in recent years. Several new network structures are proposed and have better performance than the traditional feedforward neural network structure. Representative ones include the skip connection structure in ResNet and the dense connection structure in DenseNet. However, it still lacks a unified guidance for the neural network structure design. In this paper, we propose the hypothesis that the neural network structure design can be inspired by optimization algorithms and a faster optimization algorithm may lead to a better neural network structure. Specifically, we prove that the propagation in the feedforward neural network with the same linear transformation in different layers is equivalent to minimizing some function using the gradient descent algorithm. Based on this observation, we replace the gradient descent algorithm with the heavy ball algorithm and Nesterov's accelerated gradient descent algorithm, which are faster and inspire us to design new and better network structures. ResNet and DenseNet can be considered as two special cases of our framework. Numerical experiments on CIFAR-10, CIFAR-100 and ImageNet verify the advantage of our optimization algorithm inspired structures over ResNet and DenseNet.

cs.CV↗

arXiv2018

Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization

We propose a simple and easy to implement neural network compression algorithm that achieves results competitive with more complicated state-of-the-art methods. The key idea is to modify the original optimization problem by adding K independent Gaussian priors (corresponding to the k-means objective) over the network parameters to achieve parameter quantization, as well as an L1 penalty to achieve pruning. Unlike many existing quantization-based methods, our method uses hard clustering assignments of network parameters, which adds minimal change or overhead to standard network training. We also demonstrate experimentally that tying neural network parameters provides less gain in generalization performance than changing network architecture and connectivity patterns entirely.

stat.ML↗

arXiv2018

Convolutional Neural Networks with Alternately Updated Clique

Improving information flow in deep networks helps to ease the training difficulties and utilize parameters more efficiently. Here we propose a new convolutional neural network architecture with alternately updated clique (CliqueNet). In contrast to prior networks, there are both forward and backward connections between any two layers in the same block. The layers are constructed as a loop and are updated alternately. The CliqueNet has some unique properties. For each layer, it is both the input and output of any other layer in the same block, so that the information flow among layers is maximized. During propagation, the newly updated layers are concatenated to re-update previously updated layer, and parameters are reused for multiple times. This recurrent feedback structure is able to bring higher level visual information back to refine low-level filters and achieve spatial attention. We analyze the features generated at different stages and observe that using refined features leads to a better result. We adopt a multi-scale feature strategy that effectively avoids the progressive growth of parameters. Experiments on image recognition datasets including CIFAR-10, CIFAR-100, SVHN and ImageNet show that our proposed models achieve the state-of-the-art performance with fewer parameters.

cs.CV↗

arXiv2016

Perturbative Renormalization and Mixing of Quark and Glue Energy-Momentum Tensors on the Lattice

We report the renormalization and mixing constants to one-loop order for the quark and gluon energy-momentum (EM) tensor operators on the lattice. A unique aspect of this mixing calculation is the definition of the glue EM tensor operator. The glue operator is comprised of gauge-field tensors constructed from the overlap Dirac operator. The resulting perturbative calculations are performed using methods similar to the Kawai approach using the Wilson action for all QCD vertices and the overlap Dirac operator to define the glue EM tensor. Our results are used to connect the lattice QCD results of quark and glue momenta and angular momenta to the $\overline{\text{MS}}$ scheme at input scale $μ$

hep-lat↗

arXiv2015

Orbital Angular Momentum and Generalized Transverse Momentum Distribution

We show that, when boosted to the infinite momentum frame, the quark and gluon orbital angular momentum operators defined in the nucleon spin sum rule of X. S. Chen et al. are the same as those derived from generalized transverse momentum distributions. This completes the connection between the infinite momentum limit of each term in that sum rule and experimentally measurable observables. We also show that these orbital angular momentum operators can be defined locally, and discuss the strategies of calculating them in lattice QCD.

hep-ph↗