arXiv subjects

Meng Li

Publications and source records attributed to Meng Li.

At least 325 records · Page 18Linked to original sources

arXiv2021

Low-Rank+Sparse Tensor Compression for Neural Networks

Low-rank tensor compression has been proposed as a promising approach to reduce the memory and compute requirements of neural networks for their deployment on edge devices. Tensor compression reduces the number of parameters required to represent a neural network weight by assuming network weights possess a coarse higher-order structure. This coarse structure assumption has been applied to compress large neural networks such as VGG and ResNet. However modern state-of-the-art neural networks for computer vision tasks (i.e. MobileNet, EfficientNet) already assume a coarse factorized structure through depthwise separable convolutions, making pure tensor decomposition a less attractive approach. We propose to combine low-rank tensor decomposition with sparse pruning in order to take advantage of both coarse and fine structure for compression. We compress weights in SOTA architectures (MobileNetv3, EfficientNet, Vision Transformer) and compare this approach to sparse pruning and tensor decomposition alone.

cs.LG↗

arXiv2021

LSTM-RPA: A Simple but Effective Long Sequence Prediction Algorithm for Music Popularity Prediction

The big data about music history contains information about time and users' behavior. Researchers could predict the trend of popular songs accurately by analyzing this data. The traditional trend prediction models can better predict the short trend than the long trend. In this paper, we proposed the improved LSTM Rolling Prediction Algorithm (LSTM-RPA), which combines LSTM historical input with current prediction results as model input for next time prediction. Meanwhile, this algorithm converts the long trend prediction task into multiple short trend prediction tasks. The evaluation results show that the LSTM-RPA model increased F score by 13.03%, 16.74%, 11.91%, 18.52%, compared with LSTM, BiLSTM, GRU and RNN. And our method outperforms tradi-tional sequence models, which are ARIMA and SMA, by 10.67% and 3.43% improvement in F score.Code: https://github.com/maliaosaide/lstm-rpa

cs.IR↗

arXiv2021

Atomistic mechanisms of binary alloy surface segregation from nanoseconds to seconds using accelerated dynamics

Although the equilibrium composition of many alloy surfaces is well understood, the rate of transient surface segregation during annealing is not known, despite its crucial effect on alloy corrosion and catalytic reactions occurring on overlapping timescales. In this work, CuNi bimetallic alloys representing (100) surface facets are annealed in vacuum using atomistic simulations to observe the effect of vacancy diffusion on surface separation. We employ multi-timescale methods to sample the early transient, intermediate, and equilibrium states of slab surfaces during the separation process, including standard MD as well as three methods to perform atomistic, long-time dynamics: parallel trajectory splicing (ParSplice), adaptive kinetic Monte Carlo (AKMC), and kinetic Monte Carlo (KMC). From nanosecond (ns) to second timescales, our multiscale computational methodology can observe rare stochastic events not typically seen with standard MD, closing the gap between computational and experimental timescales for surface segregation. Rapid diffusion of a vacancy to the slab is resolved by all four methods in tens of ns. Stochastic re-entry of vacancies into the subsurface, however, is only seen on the microsecond timescale in the two KMC methods. Kinetic vacancy trapping on the surface and its effect on the segregation rate are discussed. The equilibrium composition profile of CuNi after segregation during annealing is estimated to occur on a timescale of seconds as determined by KMC, a result directly comparable to nanoscale experiments.

cond-mat.mtrl-sci↗

arXiv2021

Propionamide (C2H5CONH2): The largest peptide-like molecule in space

Peptide bonds, as the molecular bridges that connect amino acids, are crucial to the formation of proteins. Searches and studies of molecules with embedded peptide-like bonds are thus important for the understanding of protein formation in space. Here we report the first tentative detection of propionamide (C2H5CONH2), the largest peptide-like molecule detected in space toward Sagittarius B2(N1) at a position called N1E that is slightly offset from the continuum peak. A new laboratory measurements of the propionamide spectrum were carried out in the 9-461 GHz, which provide good opportunity to check directly for the transition frequencies of detected interstellar lines of propionamide. Our observing result indicates that propionamide emission comes from the warm, compact cores in Sagittarius B2, in which massive protostellars are forming. The column density of propionamide toward Sgr B2(N1E) was derived to be 1.5\times 10^{16} cm^-2, which is three fifths of that of acetamide, and one nineteenth of that of formamide. This detection suggests that large peptide-like molecules can form and survive during star-forming process and may form more complex molecules in the interstellar medium. The detection of propionamide bodes well for the presence of polypeptides, as well as other complex prebiotic molecules in the interstellar medium.

astro-ph.GA↗

arXiv2021

Noisy Training Improves E2E ASR for the Edge

Automatic speech recognition (ASR) has become increasingly ubiquitous on modern edge devices. Past work developed streaming End-to-End (E2E) all-neural speech recognizers that can run compactly on edge devices. However, E2E ASR models are prone to overfitting and have difficulties in generalizing to unseen testing data. Various techniques have been proposed to regularize the training of ASR models, including layer normalization, dropout, spectrum data augmentation and speed distortions in the inputs. In this work, we present a simple yet effective noisy training strategy to further improve the E2E ASR model training. By introducing random noise to the parameter space during training, our method can produce smoother models at convergence that generalize better. We apply noisy training to improve both dense and sparse state-of-the-art Emformer models and observe consistent WER reduction. Specifically, when training Emformers with 90% sparsity, we achieve 12% and 14% WER improvements on the LibriSpeech Test-other and Test-clean data set, respectively.

cs.CL↗

arXiv2021

Hash Adaptive Bloom Filter

Bloom filter is a compact memory-efficient probabilistic data structure supporting membership testing, i.e., to check whether an element is in a given set. However, as Bloom filter maps each element with uniformly random hash functions, few flexibilities are provided even if the information of negative keys (elements are not in the set) are available. The problem gets worse when the misidentification of negative keys brings different costs. To address the above problems, we propose a new Hash Adaptive Bloom Filter (HABF) that supports the customization of hash functions for keys. The key idea of HABF is to customize the hash functions for positive keys (elements are in the set) to avoid negative keys with high cost, and pack customized hash functions into a lightweight data structure named HashExpressor. Then, given an element at query time, HABF follows a two-round pattern to check whether the element is in the set. Further, we theoretically analyze the performance of HABF and bound the expected false positive rate. We conduct extensive experiments on representative datasets, and the results show that HABF outperforms the standard Bloom filter and its cutting-edge variants on the whole in terms of accuracy, construction time, query time, and memory space consumption (Note that source codes are available in [1]).

cs.DB↗

arXiv2021

Vision Transformers with Patch Diversification

Vision transformer has demonstrated promising performance on challenging computer vision tasks. However, directly training the vision transformers may yield unstable and sub-optimal results. Recent works propose to improve the performance of the vision transformers by modifying the transformer structures, e.g., incorporating convolution layers. In contrast, we investigate an orthogonal approach to stabilize the vision transformer training without modifying the networks. We observe the instability of the training can be attributed to the significant similarity across the extracted patch representations. More specifically, for deep vision transformers, the self-attention blocks tend to map different patches into similar latent representations, yielding information loss and performance degradation. To alleviate this problem, in this work, we introduce novel loss functions in vision transformer training to explicitly encourage diversity across patch representations for more discriminative feature extraction. We empirically show that our proposed techniques stabilize the training and allow us to train wider and deeper vision transformers. We further show the diversified features significantly benefit the downstream tasks in transfer learning. For semantic segmentation, we enhance the state-of-the-art (SOTA) results on Cityscapes and ADE20k. Our code is available at https://github.com/ChengyueGongR/PatchVisionTransformer.

cs.CV↗

arXiv2021

AlphaNet: Improved Training of Supernets with Alpha-Divergence

Weight-sharing neural architecture search (NAS) is an effective technique for automating efficient neural architecture design. Weight-sharing NAS builds a supernet that assembles all the architectures as its sub-networks and jointly trains the supernet with the sub-networks. The success of weight-sharing NAS heavily relies on distilling the knowledge of the supernet to the sub-networks. However, we find that the widely used distillation divergence, i.e., KL divergence, may lead to student sub-networks that over-estimate or under-estimate the uncertainty of the teacher supernet, leading to inferior performance of the sub-networks. In this work, we propose to improve the supernet training with a more generalized alpha-divergence. By adaptively selecting the alpha-divergence, we simultaneously prevent the over-estimation or under-estimation of the uncertainty of the teacher model. We apply the proposed alpha-divergence based supernets training to both slimmable neural networks and weight-sharing NAS, and demonstrate significant improvements. Specifically, our discovered model family, AlphaNet, outperforms prior-art models on a wide range of FLOPs regimes, including BigNAS, Once-for-All networks, and AttentiveNAS. We achieve ImageNet top-1 accuracy of 80.0% with only 444M FLOPs. Our code and pretrained models are available at https://github.com/facebookresearch/AlphaNet.

cs.CV↗

arXiv2021

2D MXenes: Visible Black but Infrared White Materials

Black materials with low infrared absorption/emission (or IR white) are rare in nature but highly desired in numerous areas, such as solar-thermal energy harvesting, multispectral camouflage, thermal insulation, and anti-counterfeiting. Due to the lack of spectral selectivity in intrinsic materials, such counter-intuitive properties are generally realized by constructing complicated subwavelength metamaterials with costly nanofabrication techniques. Here we report the low mid-IR emissivity (down to 10%) of 2D Ti3C2Tx MXenes. Associated with a high solar absorptance (up to 90%), they embrace the best spectral selectivity among the reported intrinsic black solar absorbing materials. Their appealing potentials in several aforementioned areas are experimentally demonstrated. First-principles calculations reveal that the IR emissivity of MXenes relies on both the nanoflake orientations and terminal groups, indicating great tunability. The calculations also suggest that more MXenes including Ti2CTx, Nb2CTx, and V2CTx are also potential low-emissivity materials. This work opens the avenue to further exploration of a family of intrinsically low-emissivity materials with over 70 members.

cond-mat.mtrl-sci↗

arXiv2021

Gaussian graphical models with graph constraints for magnetic moment interaction in high entropy alloys

This article is motivated by studying the interaction of magnetic moments in high entropy alloys (HEAs), which plays an important role in guiding HEA designs in materials science. While first principles simulations can capture magnetic moments of individual atoms, explicit models are required to analyze their interactions. This is essentially an inverse covariance matrix estimation problem. Unlike most of the literature on graphical models, the inverse covariance matrix possesses inherent structural constraints encompassing node types, topological distance of nodes, and partially specified conditional dependence patterns. The present article is, to our knowledge, the first to consider such intricate structures in graphical models. In particular, we introduce graph constraints to formulate these structures that arise from domain knowledge and are critical for interpretability, which leads to a Bayesian conditional autoregressive model with graph constraints (CARGO) for structured inverse covariance matrices. The CARGO method enjoys efficient implementation with a modified Gauss-Seidel scheme through proximity operators for closed-form posterior exploration. We establish algorithmic convergence for the proposed algorithm under a verifiable stopping criterion. Simulations show competitive performance of CARGO relative to several other methods and confirm our convergence analysis. In a novel real data application to HEAs, the proposed methods lead to data-driven quantification and interpretation of magnetic moment interactions with high tractability and transferability.

stat.ME↗

arXiv2021

Unconditional energy dissipation and error estimates of the SAV Fourier spectral method for nonlinear fractional generalized wave equation

In this paper, we consider a second-order scalar auxiliary variable (SAV) Fourier spectral method to solve the nonlinear fractional generalized wave equation. Unconditional energy conservation or dissipation properties of the fully discrete scheme are first established. Next, we utilize the temporal-spatial error splitting argument to obtain unconditional optimal error estimate of the fully discrete scheme, which overcomes time-step restrictions caused by strongly nonlinear system, or the restrictions that the nonlinear term needs to satisfy the assumption of global Lipschitz condition in all previous works for fractional undamped or damped wave equations. Finally, some numerical experiments are presented to confirm our theoretical analysis.

math.NA↗

arXiv2021

Towards High-Performance Solid-State-LiDAR-Inertial Odometry and Mapping

We present a novel tightly-coupled LiDAR-inertial odometry and mapping scheme for both solid-state and mechanical LiDARs. As frontend, a feature-based lightweight LiDAR odometry provides fast motion estimates for adaptive keyframe selection. As backend, a hierarchical keyframe-based sliding window optimization is performed through marginalization for directly fusing IMU and LiDAR measurements. For the Livox Horizon, a newly released solid-state LiDAR, a novel feature extraction method is proposed to handle its irregular scan pattern during preprocessing. LiLi-OM (Livox LiDAR-inertial odometry and mapping) is real-time capable and achieves superior accuracy over state-of-the-art systems for both LiDAR types on public data sets of mechanical LiDARs and in experiments using the Livox Horizon. Source code and recorded experimental data sets are available at https://github.com/KIT-ISAS/lili-om.

cs.RO↗

arXiv2021

AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling

Neural architecture search (NAS) has shown great promise in designing state-of-the-art (SOTA) models that are both accurate and efficient. Recently, two-stage NAS, e.g. BigNAS, decouples the model training and searching process and achieves remarkable search efficiency and accuracy. Two-stage NAS requires sampling from the search space during training, which directly impacts the accuracy of the final searched models. While uniform sampling has been widely used for its simplicity, it is agnostic of the model performance Pareto front, which is the main focus in the search process, and thus, misses opportunities to further improve the model accuracy. In this work, we propose AttentiveNAS that focuses on improving the sampling strategy to achieve better performance Pareto. We also propose algorithms to efficiently and effectively identify the networks on the Pareto during training. Without extra re-training or post-processing, we can simultaneously obtain a large number of networks across a wide range of FLOPs. Our discovered model family, AttentiveNAS models, achieves top-1 accuracy from 77.3% to 80.7% on ImageNet, and outperforms SOTA models, including BigNAS and Once-for-All networks. We also achieve ImageNet accuracy of 80.1% with only 491 MFLOPs. Our training code and pretrained models are available at https://github.com/facebookresearch/AttentiveNAS.

cs.CV↗

arXiv2021

Electronic transport properties and hydrostatic pressure effect of FeSe$_{0.67}$Te$_{0.33}$ single crystals free of phase separation

FeSe$_{1-x}$Te$_{x}$ superconductors manifest some intriguing electronic properties depending on the value of $x$. In FeSe single crystal, the nematic phase and Dirac band structure have been observed, while topological surface superconductivity with the Majorana bound state was found in the crystal of $x \sim 0.55$. Therefore, the electronic properties of single crystals with $0 < x \leq 0.5$ are crucial for probing the evolution of those intriguing properties as well as their relations. However, this study is still left blank due to the lack of single crystals because of phase separation. Here, we report the synthesis, magnetization, electronic transport properties, and hydrostatic pressure effect of FeSe$_{0.67}$Te$_{0.33}$ single crystals free of phase separation. A structural (nematic) transition is visible at $T_{s} = 39$ K, below which the resistivity exhibits a Fermi-liquid behavior. Analysis of upper critical fields suggests that spin-paramagnetic effect should be taken into account for both $H \parallel c$ axis and $H \parallel ab$ plane. A crossover from the low-$H$ quadratic to the high-$H$ quasi-linear behavior is observed in the magnetoresistance, signifying the possible existence of Dirac-cone state. Besides, the strong temperature dependence of Hall coefficient, violation of (modified) Kohler's rule, and two-band model analysis indicate the multiband effects in FeSe$_{0.67}$Te$_{0.33}$ single crystals. Hydrostatic pressure measurements reveal that $T_{s}$ is quickly suppressed with pressure while $T_{c}$ is monotonically increased up to 2.31 GPa, indicating the competition between nematicity and superconductivity. No signature of magnetic order that has been detected in FeSe$_{1-x}$S$_{x}$ is observed. Our findings fill up the blank of the knowledge on the basic properties of FeSe$_{1-x}$Te$_{x}$ system with low-Te concentrations.

cond-mat.supr-con↗

arXiv2021

Hydrothermal synthesis and complete phase diagram of FeSe$_{1-x}$S$_{x}$ $(0 \leq x \leq 1)$ single crystals

We report the successful synthesis of FeSe$_{1-x}$S$_{x}$ single crystals with $x$ ranging from 0 to 1 via a hydrothermal method. A complete phase diagram of FeSe$_{1-x}$S$_{x}$ has been obtained based on resistivity and magnetization measurements. The nematicity is suppressed with increasing $x$, and a small superconducting dome appears within the nematic phase. Outside the nematic phase, the superconductivity is continuously suppressed and reaches a minimum $T_c$ at $x$ = 0.45; beyond this point, $T_c$ slowly increases until $x$ = 1. Intriguingly, an anomalous resistivity upturn with a characteristic temperature $T^*$ in the intermediate region of $0.31 \leq x \leq 0.71$ is observed. $T^{*}$ shows a dome-like behavior with a maximum value at $x$ = 0.45, which is opposite the evolution of $T_c$, indicating competition between $T^*$ and superconductivity. The origin of $T^*$ is discussed in detail. Furthermore, the normal state resistivity evolves from non-Fermi-liquid to Fermi-liquid behavior with S doping at low temperatures, accompanied by a reduction in electronic correlations. Our study addresses the lack of single crystals in the high-S doping region and provides a complete phase diagram, which will promote the study of relations among nematicity, superconductivity, and magnetism.

cond-mat.supr-con↗

arXiv2021

Aggregate Modeling and Equilibrium Analysis of the Crowdsourcing Market for Autonomous Vehicles

Autonomous vehicles (AVs) have the potential of reshaping the human mobility in a wide variety of aspects. This paper focuses on a new possibility that the AV owners have the option of "renting" their AVs to a company, which can use these collected AVs to provide on-demand ride services without any drivers. We call such a mobility market with AV renting options the "AV crowdsourcing market". This paper establishes an aggregate equilibrium model with multiple transport modes to analyze the AV crowdsourcing market. The modeling framework can capture the customers' mode choices and AV owners' rental decisions with the presence of traffic congestion. Then, we explore different scenarios that either maximize the crowdsourcing platform's profit or maximize social welfare. Gradient-based optimization algorithms are designed for solving the problems. The results obtained by numerical examples reveal the welfare enhancement and the strong profitability of the AV crowdsourcing service. However, when the crowdsourcing scale is small, the crowdsourcing platform might not be profitable. A second-best pricing scheme is able to avoid such undesirable cases. The insights generated from the analyses provide guidance for regulators, service providers and citizens to make future decisions regarding the utilization of the AV crowdsourcing markets for serving the good of the society.

econ.GN↗

arXiv2021

Network-level rhythmic control of heterogeneous automated traffic with buses

Guaranteeing the quality of transit service is of great importance to promote the attractiveness of buses and alleviate urban traffic issues such as congestion and pollution. Emerging technologies of automated driving and V2X communication have the potential to enable the accurate control of vehicles and the efficient organization of traffic to enhance both the schedule adherence of buses and the overall network mobility. This study proposes an innovative network-level control scheme for heterogeneous automated traffic composed of buses and private cars under a full connected and automated environment. Inheriting the idea of network-level rhythmic control proposed by Lin et al. (2020), an augmented rhythmic control scheme for heterogeneous traffic, i.e., RC-H, is established to organize the mixed traffic in a rhythmic manner. Realized virtual platoons are designed for accommodating vehicles to pass through the network, including dedicated virtual platoons for buses to provide exclusive right-of-ways (ROWs) on their trips and regular virtual platoons for private cars along with an optimal assignment plan to minimize the total travel cost. A mixed-integer linear program (MILP) is formulated to optimize the RC-H scheme and a bilevel heuristic solution method is designed to relieve the computational burden of MILP. Numerical examples and simulation experiments are conducted to evaluate the performance of the RC-H scheme under different scenarios. The results show that the bus operation can be guaranteed and the travel delay can be minimized under various demand levels with transit priority. Moreover, compared with traffic signal control strategies, the RC-H scheme has significant advantages in handling massive traffic demand, in terms of both vehicle delay and network throughput.

math.OC↗

arXiv2021

Exploring wav2vec 2.0 on speaker verification and language identification

Wav2vec 2.0 is a recently proposed self-supervised framework for speech representation learning. It follows a two-stage training process of pre-training and fine-tuning, and performs well in speech recognition tasks especially ultra-low resource cases. In this work, we attempt to extend self-supervised framework to speaker verification and language identification. First, we use some preliminary experiments to indicate that wav2vec 2.0 can capture the information about the speaker and language. Then we demonstrate the effectiveness of wav2vec 2.0 on the two tasks respectively. For speaker verification, we obtain a new state-of-the-art result, Equal Error Rate (EER) of 3.61% on the VoxCeleb1 dataset. For language identification, we obtain an EER of 12.02% on 1 second condition and an EER of 3.47% on full-length condition of the AP17-OLR dataset. Finally, we utilize one model to achieve the unified modeling by the multi-task learning for the two tasks.

cs.SD↗